mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 23:38:07 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,987 Commits 32 Branches 296 Tags 152 MiB
6d4f5c94ac
Commit Graph

866 Commits

Author SHA1 Message Date
Chris Lu
ab966d7192 refactoring variable name 2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69 volume: add deletion capability for previously readonly volumes 2019-12-19 00:42:46 -08:00
Chris Lu
deb03ecfa3 remove duplicated checking 
already checked in store.go
 2019-12-18 23:59:53 -08:00
Chris Lu
ef3ae3cd41 clean up sorted index file 2019-12-18 20:59:15 -08:00
Chris Lu
39810a9951 tiered volume is read only 2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu
356bd1b629 adjust text 2019-12-10 22:09:29 -08:00
Chris Lu
10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu
2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
Chris Lu
f38f90b7ea cleanly remove .tier volume file 2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86 show volume info in volume.list in weed shell 2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
ee2072dff9 volume: avoid inifinite loop reading ec volume info if error happens 2019-11-27 23:07:53 -08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu
e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu
9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5 rm ldb and bdb directory 2019-11-19 09:35:06 +08:00
Chris Lu
c6e8225a20 adjust memory mapped file size 
related to https://github.com/chrislusf/seaweedfs/pull/1087
 2019-11-12 21:05:48 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu
85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu
c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu
c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu
84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
zhangsong
1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
Chris Lu
db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu
57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu
b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu
4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Chris Lu
cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu
46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu
f937933b3d refactoring 2019-10-23 22:25:53 -07:00
Chris Lu
3c865ee39b fix missing needle header read 2019-10-22 00:57:47 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu
fec07c829d go fmt 2019-10-21 23:03:48 -07:00
Chris Lu
faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock
2e2fe00dbd Comma, no space? 2019-10-18 11:31:25 +01:00
j.laycock
248f3be6e3 using a space instead of a comma to hopefully fix the build! 2019-10-18 11:23:02 +01:00
j.laycock
2c455841ea Make volumeCreate more consistent between all 3 implementations. 2019-10-18 11:01:45 +01:00
j.laycock
b0ddad6889 Fix volume_create breaking the build 2019-10-18 10:32:07 +01:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu
87e5a02a99 fix logic error 2019-10-09 00:07:18 -07:00
Chris Lu
09874f0d16 volume: return error if superblock is not initialized 
fix https://github.com/chrislusf/seaweedfs/issues/1079
 2019-10-09 00:02:50 -07:00
j.laycock
eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
j.laycock
d5f5acb734 limit locking physical memory to 80% of max physical memory 2019-09-20 12:44:29 +01:00
j.laycock
476140fd6b minor change to setProcessWorkingSetSize function 2019-09-19 11:59:00 +01:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu
5e9c65469e volume: skip readonly checking when Destroy() 
fix https://github.com/chrislusf/seaweedfs/issues/1063
 2019-09-11 09:45:09 -07:00
j.laycock
c1a928887a Add some comments/documentation 2019-09-11 10:42:37 +01:00
joeslay
9ea9a5e97a
Merge pull request #5 from chrislusf/master 
merge seaweed master
 2019-09-11 10:25:27 +01:00
j.laycock
15a4c91c28 Remove Finalizer 2019-09-10 17:05:54 +01:00
j.laycock
9c9dff7386 Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming 2019-09-10 16:18:51 +01:00
Chris Lu
02710f9624 refactoring: remove unused functions 2019-09-09 10:03:41 -07:00
Chris Lu
0f861d23a2 avoid nil needle map 
fix https://github.com/chrislusf/seaweedfs/issues/1061
 2019-09-09 09:48:08 -07:00
j.laycock
63acc8c972 rename mem_buffer to mBuffer 2019-09-05 13:46:29 +01:00
j.laycock
44ae041e80 rename more memorymapped variables to memoryMapMaxSizeMB 2019-09-05 13:42:23 +01:00
j.laycock
5885ab67b3 rename volume property MemoryMap to MemoryMapMaxSizeMB 2019-09-05 11:56:48 +01:00
j.laycock
5a6dfb05fc Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size 2019-09-05 11:46:12 +01:00
j.laycock
f6146d6ad2 use 0 instead of false for max memory flag 2019-09-05 11:43:18 +01:00
j.laycock
4257582db5 Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks 2019-09-05 11:15:01 +01:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock
cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock
de5a7e60a2 remove temp hack 2019-09-03 18:20:09 +01:00
j.laycock
cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock
9a459d984b Do not vacuum memory mapped files 2019-09-03 15:05:43 +01:00
j.laycock
0123c7a898 Change to max 2GB for now 2019-09-02 17:46:22 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
j.laycock
1e62a2b233 Fix memory not being unmapped 2019-08-30 16:55:33 +01:00
j.laycock
840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock
dc50701e7d Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete 2019-08-30 12:30:00 +01:00
j.laycock
523f3a12b3 use Os.File/uintptr instead of windows.Handle in memory map struct 2019-08-30 12:30:00 +01:00
j.laycock
0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu
58168a8c52 volume: truncate dat files when creating 2019-08-26 13:01:57 -07:00
joeslay
36bde84000
Merge pull request #1 from chrislusf/master 
merge changes from chrislusf/seaweed master
 2019-08-23 17:03:37 +01:00
j.laycock
1e89e719b3 Add windows memory map functions to storage package 2019-08-23 16:19:00 +01:00
xushuxun
9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
Chris Lu
f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
Chris Lu
d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu
e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu
f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu
fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu
a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu
e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu
115558e5f5 adjust counters 2019-06-17 21:02:50 -07:00
Chris Lu
935639b908 metrics: disk size for <collection, normal/EC> 2019-06-16 21:56:41 -07:00
Chris Lu
0fdb1e705d collect volume disk usage metrics 2019-06-16 02:44:20 -07:00
Chris Lu
289fd7eb39 count number of volumes and ec shards 2019-06-16 02:24:15 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
0e52862586 avoid the test that depends on ordering in a different folder 2019-06-04 02:22:32 -07:00
Chris Lu
2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu
11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu
689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu
6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu
76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu
8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu
cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu
46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu
4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu
53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error"" 
This reverts commit 359b7c1650.
 2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650 Revert "volume: close request body if any parsing error" 
This reverts commit 5996656141.
 2019-04-15 21:35:12 -07:00
Chris Lu
5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu
3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu
0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
stlpmo-jn
c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu
784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu
ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu
44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu
346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
chenwanli
fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu
5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
bingoohuang
cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
Chris Lu
badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
Chris Lu
6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu
d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu
1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu
42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu
168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu
8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu
4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu
5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu
d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu
df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu
ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu
9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224
8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu
f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu
496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu
eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu
66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu
bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu
1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu
76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu
ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu
d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu
694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu
1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu
75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu
495a776671 support 128 bit NeedleId 2018-07-31 23:25:26 -07:00
Chris Lu
10bb76fe6b go fmt 2018-07-27 02:10:42 -07:00
Chris Lu
4d322df95e avoid reparsing the multipart form 2018-07-27 02:10:10 -07:00
Chris Lu
57ded4fd80 speed up leveldb bolddb loading 2018-07-24 22:17:56 -07:00
Chris Lu
ea8ecf35f4 set needle AppendAtNs value during writes 2018-07-24 02:44:33 -07:00
Chris Lu
77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu
4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu
852af28f91 needle priting format 2018-07-21 17:41:21 -07:00
Chris Lu
7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu
c98df05ed0 support PUT 2018-07-21 15:58:48 -07:00
Chris Lu
feb8eeb830 refactor: separate multipart parsing file 2018-07-21 12:01:35 -07:00
Chris Lu
3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu
a51aa4c586 avoid extra log messages 
fix https://github.com/chrislusf/seaweedfs/issues/689
 2018-07-13 22:07:32 -07:00
Chris Lu
8a12fd9ec4 add error checking when loading dat file 2018-07-09 23:31:25 -07:00
Chris Lu
d0982cafa7 refactoring 2018-07-09 00:22:50 -07:00
Chris Lu
5b1fd374be fix all tests 2018-07-08 02:39:04 -07:00
Chris Lu
d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu
5bfb72d058 faster loading boltdb or leveldb needle map metrics by bloomfilter 
avoid btree
 2018-07-07 00:51:17 -07:00
Chris Lu
aba1fe01b3 use constant NeedleIndexSize instead of the number 16 2018-07-07 00:50:14 -07:00
Chris Lu
defcd4edc5 adjust log level 2018-07-07 00:48:58 -07:00
Chris Lu
9bcaa65574 avoid divided by 0 
fix https://github.com/chrislusf/seaweedfs/issues/650
 2018-06-25 01:20:15 -07:00
Chris Lu
233a9a9b6c fix log error 2018-06-24 16:12:41 -07:00
Chris Lu
49f4216280 fix compilation error 2018-06-24 15:37:54 -07:00
Chris Lu
7f6500ae07 super block reserves a couple of bits for future extension 2018-06-24 15:23:03 -07:00
Chris Lu
f339e9c284 add support for extra super block info 2018-06-24 15:19:57 -07:00
Chris Lu
9ba335a7c4 prepare for flexible super block 2018-06-24 11:37:08 -07:00
Chris Lu
3a6f9c9bcf increase test size 2018-06-23 18:30:13 -07:00
Chris Lu
7f7e4e9885 fix error for deleted files during compaction 
deletion during commit may cause trouble when make up the difference during commitCompact()
 2018-06-23 18:24:59 -07:00
Chris Lu
3699996985 add compaction test 2018-06-23 16:48:19 -07:00
Chris Lu
0467195f07 fix file closing on windows 2018-06-13 00:36:51 -07:00
Chris Lu
eaf000eff1 fix error on go tip 2018-06-11 23:39:42 -07:00
Chris Lu
43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu
458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu
1675243f29 maybe speed up a little when loading index 2018-05-23 02:38:11 -07:00
Chris Lu
69b9d8c3c2 fix boltdb variable usage 2018-05-22 10:18:09 -07:00
Chris Lu
942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu
57c5405657 add todo 2018-05-09 04:18:31 -07:00
Chris Lu
6db1c8bafb Revert "adjust the needle mapper" 
This reverts commit e2c7be3a2e.
 2018-05-09 03:57:07 -07:00
Chris Lu
e2c7be3a2e adjust the needle mapper 
as discussed on
https://github.com/chrislusf/seaweedfs/pull/642#issuecomment-387606960
 2018-05-09 03:50:14 -07:00
WenLong LI
b9068c1304
use volumes variable 
fix leveldb mount bug
 2018-05-03 12:13:53 +08:00
liwenlong05
02a1597642 fix leveldb mount bug 2018-05-03 11:20:38 +08:00
mtolman
e4b1e5c3c8
Fix for issue #635 
Fix for the following issue on master: https://github.com/chrislusf/seaweedfs/issues/635
 2018-04-17 13:26:48 -06:00
Yang Pengcheng
a2112e1be0 fix ldb dir deleting 2017-08-30 22:55:03 +08:00
Chris Lu
58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu
5047bdb4a2 skip bytes cache 2017-06-03 11:44:24 -07:00
Chris Lu
82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00
Chris Lu
bd97cbc523 skip busy reporting back to master 
possible fix for https://github.com/chrislusf/seaweedfs/issues/499
 2017-05-23 21:12:03 -07:00
Chris Lu
043b7a7c65 add locking to access l.volumes 2017-01-20 10:18:43 -08:00
brstgt
0656838fe5 Don't return actual file size from CheckVolumeDataIntegrity, it will be 0 if last needle is a tombstone, so it's not reliable anyway 2017-01-20 16:37:45 +01:00
brstgt
4fb5bb09b6 Remove obsolete property Volume.dataFileSize 2017-01-20 16:31:11 +01:00
brstgt
e074a54a20 Delete volumes online without restarting volume server 2017-01-20 13:02:37 +01:00
brstgt
492f93416d Mount and unmount volumes online without restarting volume server 2017-01-20 12:49:20 +01:00
brstgt
18b3afc97a Log volumeId if dataFileSize != actual data file size - improve log types 2017-01-20 12:22:25 +01:00
brstgt
0c898df430 Log volumeId if dataFileSize != actual data file size 2017-01-20 11:52:23 +01:00
Chris Lu
5b0f5f456e volume server get notified of leader change 2017-01-18 09:34:27 -08:00
Chris Lu
59022b6fe0 fix byte counter on loading index file 
fix https://github.com/chrislusf/seaweedfs/issues/441
 2017-01-18 08:40:39 -08:00
Chris Lu
3065506b38 volume servers always connect to the master leader 2017-01-13 10:20:40 -08:00
Chris Lu
e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu
4beaaa0650 simplify the gzip optimization logic 
fix https://github.com/chrislusf/seaweedfs/issues/436
 2017-01-09 19:31:58 -08:00
Chris Lu
ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
Chris Lu
7b6837cbc2 move back the section 2017-01-08 10:35:47 -08:00
sparklxb
da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb
86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
Chris Lu
13e7069eb9 keep track of total data file size 2017-01-06 10:22:20 -08:00
lixianbin
d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu
dcaf1796fe add option to enable caching 2016-11-16 07:09:57 -08:00
Chris Lu
f54f530ada adjust verbose logging 2016-11-13 14:07:51 -08:00
霍晓栋
b9f385bd66 refactor concurrent loading code logic 2016-11-13 13:24:52 +08:00
霍晓栋
096ffa9744 concurrent loading volume 2016-11-11 11:53:22 +08:00
Chris Lu
36f9633223 add locks for location.volumes 
fix https://github.com/chrislusf/seaweedfs/issues/392
 2016-11-06 20:55:22 -08:00
霍晓栋
7d73bbb073 comment UT case 2016-10-07 16:40:51 +08:00
霍晓栋
7d382ba5fe comment UT case 2016-10-07 16:34:22 +08:00
霍晓栋
ce1f7ab662 makediff func with UT case 2016-10-07 16:22:24 +08:00
霍晓栋
ed848425c7 supplemental data between compacting and commit compacting 2016-09-29 13:57:23 +08:00
Chris Lu
dffad65f2f fix compilation 2016-09-26 22:30:44 -07:00
Chris Lu
7e29218327 add a template for makeupDiff 2016-09-26 22:26:41 -07:00
Chris Lu
1bc041b46d add a new way to manually compact corrupted volume 
fix https://github.com/chrislusf/seaweedfs/issues/371
 2016-09-22 20:31:20 -07:00
Chris Lu
01cbd5cb58 lock 
fix https://github.com/chrislusf/seaweedfs/issues/367
 2016-09-08 09:50:31 -07:00
霍晓栋
b9b3651a98 deleted index entry could not point to deleted needle 2016-07-25 14:56:58 +08:00
霍晓栋
09bd3d015d deleted index entry could not point to deleted needle 2016-07-25 14:54:40 +08:00
霍晓栋
78678f4bcb deleted needle does not need checksum verification 2016-07-25 11:40:35 +08:00
Chris Lu
d981eb282f refactor volume_read_write.go out of volume.go 2016-07-03 00:10:27 -07:00
Chris Lu
576573711a refactoring 2016-07-02 23:58:23 -07:00
Chris Lu
3d8df0f709 refactor volume_loading.go out of volume.go 2016-07-02 23:56:49 -07:00
Chris Lu
582d5d526c refactoring 2016-07-02 23:50:58 -07:00
Chris Lu
3713119ab9 adjust verification logic. 
1. log file name for easier debugging
2. remove unused version in readIndexEntryAtOffset()
3. if indexSize==0, the dataFile usually still has a super block of
size 8. So skipping this case.
 2016-07-02 23:45:37 -07:00
霍晓栋
16f6984d02 refactor data integrity checking code v2 2016-07-03 14:11:25 +08:00
霍晓栋
dda13def2a refactor data integrity checking code 2016-07-03 12:53:03 +08:00
霍晓栋
2e74fb60c8 add data integrity checking 2016-07-02 21:34:30 +08:00
Chris Lu
101e784577 add locking on possible concurrent map access 
fix https://github.com/chrislusf/seaweedfs/issues/328
 2016-06-23 09:10:25 -07:00
霍晓栋
1ef81ac518 filer could detect master nodes healthy status and choose a working one when encountering SPOF 2016-06-07 11:38:40 +08:00
Chris Lu
5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu
5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu
55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu
ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00