Commit Graph

1204 Commits

Author SHA1 Message Date
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
a72cef3c42 encode by collection 2019-05-31 15:48:40 -07:00
Chris Lu
de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu
689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5 resolve concurrent modification problem
n and err can be modified concurrently
2019-05-30 13:59:07 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
866197eee3 print out the ec balancing plan 2019-05-30 01:38:59 -07:00
Chris Lu
5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc WIP
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu
92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu
ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu
8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password
fix https://github.com/chrislusf/seaweedfs/issues/971
2019-05-20 09:00:30 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
ae499fd5aa Merge branch 'master' into erasure_coding 2019-05-19 21:38:18 -07:00
Chris Lu
698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu
693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu
f2c4c888f6 1.34 2019-05-17 22:53:30 -07:00
Chris Lu
8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu
76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2
large filer db may see high CPU and disk usage due to background compaction
2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB
fix https://github.com/chrislusf/seaweedfs/issues/966
2019-05-17 09:11:08 -07:00
Chris Lu
bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
017d0957c3 avoid switching master node if same ip and host name
fix https://github.com/chrislusf/seaweedfs/issues/955
2019-05-11 18:42:46 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
69b7dd398d adjust help message 2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu
cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu
4e42e7b5e7 shell: volume.balance add dataCenter filter 2019-05-06 13:30:14 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
2019-05-06 08:10:18 -07:00
Chris Lu
8656102127 shell: volume.balance runs for each collection, and for ALL at the end 2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de refactoring 2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611 add error checking 2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05 shell: volume.list output sorted voume list 2019-05-05 20:23:50 -07:00
Chris Lu
46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu
4b27f892e6 remove 128 bit needle id build option
fix https://github.com/chrislusf/seaweedfs/issues/949
2019-05-03 22:39:00 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer
addressing https://github.com/chrislusf/seaweedfs/issues/717
2019-05-03 22:33:08 -07:00
Chris Lu
5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu
8c982db25d 1.33 2019-05-03 17:27:50 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav
add WebDav
2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
2019-04-29 10:42:09 -07:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
575926cdec minor 2019-04-26 09:32:37 -07:00
Chris Lu
630cf1e33e fix 2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c fix 2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c logging 2019-04-24 12:30:36 -07:00
Chris Lu
9beea63960 memory alignment
fix https://github.com/chrislusf/seaweedfs/issues/939
2019-04-24 09:23:19 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
067b935763 report unknown commands 2019-04-23 22:29:36 -07:00
Chris Lu
53e2e73de2 fix metrics
fix https://github.com/chrislusf/seaweedfs/issues/935
2019-04-22 12:32:10 -07:00
Chris Lu
0c96cfdb2e minor 2019-04-21 15:50:51 -07:00
Chris Lu
00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu
59d532a8c3 1.31 2019-04-21 14:09:50 -07:00
Chris Lu
5c411f3e5f minor 2019-04-21 13:33:32 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
64a9a0e104 shell: fix volume.copy 2019-04-20 20:48:07 -07:00
Chris Lu
5f3d0e33a1 shell: add commands volume.copy volume.delete volume.mount volume.unmount 2019-04-20 20:31:35 -07:00
Chris Lu
d06d3c57d4 add notes 2019-04-20 20:12:07 -07:00
Chris Lu
2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu
07091c44cd lock btree writes
fix https://github.com/chrislusf/seaweedfs/issues/930
2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
af49aea0c6 weed shell: fs.meta.save adjusts meta data file name 2019-04-16 09:40:27 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
014906ec79 weed shell: add command fs.meta.load to restore meta data 2019-04-16 01:37:11 -07:00
Chris Lu
79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu
d35023c713 weed shell: add command fs.meta.save 2019-04-16 01:06:32 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
967e108b9a increase max idle conn per hosts 2019-04-15 22:00:50 -07:00
Chris Lu
f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error""
This reverts commit 359b7c1650.
2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650 Revert "volume: close request body if any parsing error"
This reverts commit 5996656141.
2019-04-15 21:35:12 -07:00
Chris Lu
5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu
6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00