mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-07 05:09:03 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,974 Commits 32 Branches 296 Tags 149 MiB
3f6e0e34cb
Commit Graph

1581 Commits

Author SHA1 Message Date
Chris Lu
b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu
19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu
fb90d63589 shell: Added minor changes to fs.meta.save 
* added an option to set the output file name
* added an option to see verbose output, default to non-verbose mode

related to: https://github.com/chrislusf/seaweedfs/issues/1019
 2019-07-26 09:35:22 -07:00
divinerapier
9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu
344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu
3f851feb59 1.42 2019-07-21 21:52:48 -07:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu
f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu
4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu
fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu
c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu
aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu
0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu
a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu
9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu
1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu
61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu
476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu
1e2127e261 s3: adding Location for multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/1002
 2019-07-08 12:37:45 -07:00
Chris Lu
62843991f2 s3: adjust object key 2019-07-08 12:37:20 -07:00
Chris Lu
cf2804eebd 1.41 2019-07-07 20:40:20 -07:00
Chris Lu
7872fc38ea filer leveldb2: correct directory listing 
fix https://github.com/chrislusf/seaweedfs/issues/1001
 2019-07-07 20:37:48 -07:00
Chris Lu
fe2882ce7d fix test 2019-07-02 21:42:06 -07:00
Chris Lu
36250154cc remove println 2019-07-02 21:33:23 -07:00
Chris Lu
4d579f7f57 adjust 2019-07-02 21:28:51 -07:00
Chris Lu
23d36dfbaf 1.40 
enable azure replication
 2019-07-02 21:27:05 -07:00
Chris Lu
dcc331cf5a parallelize leveldb 2019-07-02 21:25:53 -07:00
Chris Lu
88fed85bba temp skipping azure 
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
 2019-06-30 23:17:45 -07:00
Chris Lu
ce5aab66e9 add versions 2019-06-30 22:43:10 -07:00
Chris Lu
36022f9416 1.39 2019-06-30 00:50:17 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
ee933cd859 filer: add metrics for go memory, gc stats 2019-06-29 03:02:28 -07:00
Chris Lu
92c7f7e069 fix compilation error 2019-06-28 11:19:08 -07:00
Chris Lu
9f3f2f7c79 protect locations slice 
fix https://github.com/chrislusf/seaweedfs/issues/995
 2019-06-28 09:47:29 -07:00
Chris Lu
327336ecf3 filer: avoid concurrent modification to result slice 
fix https://github.com/chrislusf/seaweedfs/issues/972
 2019-06-28 01:12:41 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Lei Liu
1d9b75b536
weed.go: remove unused parameter 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-06-26 10:46:32 +08:00
Chris Lu
359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu
fdff927b37 1.38 2019-06-24 00:36:46 -07:00
Chris Lu
a8b4fe6eae adjust help text 2019-06-24 00:26:03 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu
cd45ab072a fix compilation error 2019-06-22 12:30:08 -07:00
Chris Lu
c369e5a13b add metrics for filer store 2019-06-22 12:23:25 -07:00
Chris Lu
6bc3dee5b3 refactoring 2019-06-22 10:56:54 -07:00
Chris Lu
95ad56d99d 1.37 2019-06-21 20:56:34 -07:00
Chris Lu
a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu
308102f023 fix cassandra creation problem 
fix https://github.com/chrislusf/seaweedfs/issues/986
 2019-06-21 17:10:38 -07:00
Chris Lu
fe4eb6bffa 1.36 2019-06-21 13:11:40 -07:00
Chris Lu
059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu
ff97acae62 more logs 2019-06-21 12:14:40 -07:00
Chris Lu
d0ce5b020b logging 2019-06-21 12:05:00 -07:00
Chris Lu
78994024ff adjust logging 2019-06-21 11:46:12 -07:00
Chris Lu
f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu
613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu
11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu
4b0f084b5d delete chunks only when file writing is successful 2019-06-20 21:58:35 -07:00
Chris Lu
eebc44cce3 refactoring 2019-06-20 21:57:17 -07:00
Chris Lu
3106065b58 fix startup problem with metrics 2019-06-20 09:56:49 -07:00
Chris Lu
c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu
e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu
115558e5f5 adjust counters 2019-06-17 21:02:50 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
68d1bef236 instance format 2019-06-16 21:57:32 -07:00
Chris Lu
935639b908 metrics: disk size for <collection, normal/EC> 2019-06-16 21:56:41 -07:00
Chris Lu
0fdb1e705d collect volume disk usage metrics 2019-06-16 02:44:20 -07:00
Chris Lu
289fd7eb39 count number of volumes and ec shards 2019-06-16 02:24:15 -07:00
Chris Lu
ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu
d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
be3f1f84a2 1.35 2019-06-10 21:34:06 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
f9d8bd51ad ec shard balancing 2019-06-10 21:32:56 -07:00
Chris Lu
9d9162ca35 ec.balance: collect dc rack info 2019-06-07 00:25:05 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu
7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu
6b08db65b0 fix shard count reporting 2019-06-05 01:58:37 -07:00
Chris Lu
784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
b9e138713c ec.encode: add -fullPercent option 2019-06-05 00:13:13 -07:00
Chris Lu
0e49add2bc adjust help message 2019-06-05 00:11:37 -07:00
Chris Lu
eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
0e52862586 avoid the test that depends on ordering in a different folder 2019-06-04 02:22:32 -07:00
Chris Lu
2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu
06133ae98f correct count for free ec slots, avoid removing generated shards 2019-06-04 01:49:51 -07:00
Chris Lu
6cea23d091 purge duplicated ec shards 2019-06-04 01:32:36 -07:00
Chris Lu
b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu
11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu
d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu
9ce6b949bf adjust help message 2019-06-01 10:37:08 -07:00
Chris Lu
d6e2c877fa ec.balance: use specific collection names 2019-06-01 02:00:18 -07:00
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
a72cef3c42 encode by collection 2019-05-31 15:48:40 -07:00
Chris Lu
de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu
689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
866197eee3 print out the ec balancing plan 2019-05-30 01:38:59 -07:00
Chris Lu
5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu
92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu
ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu
8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
ae499fd5aa Merge branch 'master' into erasure_coding 2019-05-19 21:38:18 -07:00
Chris Lu
698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu
693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu
f2c4c888f6 1.34 2019-05-17 22:53:30 -07:00
Chris Lu
8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu
76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB 
fix https://github.com/chrislusf/seaweedfs/issues/966
 2019-05-17 09:11:08 -07:00
Chris Lu
bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
017d0957c3 avoid switching master node if same ip and host name 
fix https://github.com/chrislusf/seaweedfs/issues/955
 2019-05-11 18:42:46 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
69b7dd398d adjust help message 2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu
cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu
4e42e7b5e7 shell: volume.balance add dataCenter filter 2019-05-06 13:30:14 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls 
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
 2019-05-06 08:10:18 -07:00
Chris Lu
8656102127 shell: volume.balance runs for each collection, and for ALL at the end 2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de refactoring 2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611 add error checking 2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05 shell: volume.list output sorted voume list 2019-05-05 20:23:50 -07:00
Chris Lu
46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu
4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer 
addressing https://github.com/chrislusf/seaweedfs/issues/717
 2019-05-03 22:33:08 -07:00
Chris Lu
5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu
8c982db25d 1.33 2019-05-03 17:27:50 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba 
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
 2019-04-29 10:42:09 -07:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
575926cdec minor 2019-04-26 09:32:37 -07:00
Chris Lu
630cf1e33e fix 2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c fix 2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c logging 2019-04-24 12:30:36 -07:00
Chris Lu
9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
067b935763 report unknown commands 2019-04-23 22:29:36 -07:00
Chris Lu
53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00
Chris Lu
0c96cfdb2e minor 2019-04-21 15:50:51 -07:00
Chris Lu
00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu
59d532a8c3 1.31 2019-04-21 14:09:50 -07:00
Chris Lu
5c411f3e5f minor 2019-04-21 13:33:32 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
64a9a0e104 shell: fix volume.copy 2019-04-20 20:48:07 -07:00
Chris Lu
5f3d0e33a1 shell: add commands volume.copy volume.delete volume.mount volume.unmount 2019-04-20 20:31:35 -07:00
Chris Lu
d06d3c57d4 add notes 2019-04-20 20:12:07 -07:00
Chris Lu
2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu
07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
af49aea0c6 weed shell: fs.meta.save adjusts meta data file name 2019-04-16 09:40:27 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
014906ec79 weed shell: add command fs.meta.load to restore meta data 2019-04-16 01:37:11 -07:00
Chris Lu
79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu
d35023c713 weed shell: add command fs.meta.save 2019-04-16 01:06:32 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
967e108b9a increase max idle conn per hosts 2019-04-15 22:00:50 -07:00
Chris Lu
f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error"" 
This reverts commit 359b7c1650.
 2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650 Revert "volume: close request body if any parsing error" 
This reverts commit 5996656141.
 2019-04-15 21:35:12 -07:00
Chris Lu
5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu
6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00
Chris Lu
e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu
3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu
2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Chris Lu
59a9e15733 1.30 2019-04-11 19:40:36 -07:00
Chris Lu
e11e127d91 close grpc connections during errors 2019-04-11 09:27:08 -07:00
Wine93
361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu
0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu
c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
stlpmo-jn
c3fa50d3b3 remove the health chekcer, because it's the same as command_volume_fix_replication 2019-04-11 13:40:31 +08:00
Chris Lu
2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn
f2031884f0 fix bug : CI build failed 2019-04-11 09:53:31 +08:00
Chris Lu
4df10e50b3 update glide.yaml 2019-04-10 14:27:28 -07:00
stlpmo-jn
c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu
35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu
a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu
6d4c44a85c 1.29 2019-04-08 22:41:59 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu
784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive 
replication: add GoCDK PubSub support
 2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu
a32abda1a3 adjust compression to optimize for speed 2019-04-07 11:01:28 -07:00
Chris Lu
c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu
174bf1e8b2 more efficient client side gzip compression 2019-04-06 14:14:28 -07:00
Chris Lu
868913aa04 printout statistics 2019-04-06 11:12:35 -07:00
Chris Lu
de7626bd22 weed shell: volume.list add summary statistics 2019-04-06 09:25:29 -07:00
Chris Lu
94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu
b2d92a2992 weed shell: fs.tree fix directory tree listing 2019-04-05 23:42:36 -07:00
Chris Lu
24c020104a weed filer.copy: parallelize the copying to increase throughput 
improvement https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 23:35:30 -07:00
Chris Lu
c789b496d8 use cached grpc client 2019-04-05 20:31:58 -07:00
Chris Lu
5808caa2f5 use cached grpc client 2019-04-05 20:26:52 -07:00
Chris Lu
cd6d35aa52 weed shell: fs.tree improvements 2019-04-05 11:40:54 -07:00
Jonathan Amsterdam
72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Jonathan Amsterdam
a6a5d80401 fix typo 2019-04-05 08:13:32 -04:00
Chris Lu
300b39b5af weed filer.copy: use cached filer grpc connection 
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 01:09:06 -07:00
Chris Lu
921231268b weed filer.copy: skip files that can not be opened 
such as socket
 2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
c94823f52c set default http idle connection per host 
possible fix https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-04 20:28:40 -07:00
Chris Lu
0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu
9cc73f4a9a fix test 2019-04-04 19:34:15 -07:00
Chris Lu
af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu
766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Jonathan Amsterdam
c0c9a8bad5 replication: add GoCDK PubSub support 2019-04-04 17:22:45 -04:00
Chris Lu
715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu
20dcb44077 fix tests 2019-04-01 23:59:31 -07:00
Chris Lu
beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu
78b9db34d5 weed s3: generate time format acceptable by aws cli s3 
fix https://github.com/chrislusf/seaweedfs/issues/911
 2019-04-01 16:13:29 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu
c5a3ff1c35 weed s3: multipart upload goes to the right bucket 
fix https://github.com/chrislusf/seaweedfs/issues/908
 2019-03-31 11:10:19 -07:00
Chris Lu
189c890715 weed replicate: replicate atomic rename to other systems 2019-03-31 08:10:47 -07:00
Chris Lu
8c823abe1f 1.28 2019-03-30 23:22:41 -07:00
Chris Lu
78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu
07775998a9 some comments 2019-03-28 00:38:30 -07:00
Chris Lu
53b81fcfcd weed shell: add fs.tree 2019-03-28 00:05:04 -07:00
Chris Lu
c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu
65757ae2fd weed filer: set content-type consistent with filer store 2019-03-27 11:41:11 -07:00
Chris Lu
37ce4c5269 weed shell: add fs.ls 2019-03-26 12:43:51 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
c913583e2d Merge branch 'master' into add_volume_follow 2019-03-25 21:19:34 -07:00
Chris Lu
91b9a68493 udpate glide 2019-03-25 09:39:54 -07:00
Chris Lu
a32797518b weed master: redirect to leader for default admin UI 
fix https://github.com/chrislusf/seaweedfs/issues/898
 2019-03-25 09:20:05 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
eaa42c3865 1.27 2019-03-23 14:14:40 -07:00
Chris Lu
cd8a3b99bb textual changes 2019-03-23 12:57:35 -07:00
Chris Lu
bd1c0735e0 weed shell: adjust help text format 2019-03-23 11:54:26 -07:00
Chris Lu
6b70b36105 weed shell: add "volume.fix.replication" 2019-03-23 11:34:09 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Chris Lu
531add52c2 weed master: skip proxied file read in http read only mode 2019-03-21 09:49:04 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Chris Lu
977b30e992 weed filer: http HEAD response header add ETag 
fix https://github.com/chrislusf/seaweedfs/issues/892
 2019-03-20 10:13:53 -07:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu
88ab932f7d refactoring function parameter 2019-03-19 23:01:23 -07:00
Chris Lu
5ae4b963a4 avoid using global rand 2019-03-19 22:20:14 -07:00
Chris Lu
f3d316a846 weed shell: adding file system disk usage fs.du 
fix https://github.com/chrislusf/seaweedfs/issues/889

echo "fs.du http://localhost:8888/some/path" | weed shell
 2019-03-19 21:58:00 -07:00
Chris Lu
cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu
eb4a54d9fe refactoring 2019-03-19 05:19:37 -07:00
Chris Lu
ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu
104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu
44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
22fbbf023b adjust collection list output 2019-03-17 17:31:05 -07:00
Chris Lu
e48267e287 adjust output 2019-03-17 17:28:29 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu
36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu
42cb05c088 add context.Context 2019-03-15 17:23:02 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu
95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu
346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
Chris Lu
99ce10daeb Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-11 00:58:17 -07:00
Chris Lu
2ce4dae795 move to 1.26 2019-03-11 00:58:12 -07:00
chenwanli
c668e55e07 Revert "fix https://github.com/chrislusf/seaweedfs/issues/780" 
This reverts commit 31038acccf.
 2019-03-11 11:48:11 +08:00
Chris Lu
6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu
ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu
0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli
f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu
219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00