mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-28 11:57:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,171 Commits 32 Branches 296 Tags 153 MiB
12df236def
Commit Graph

1742 Commits

Author SHA1 Message Date
Chris Lu
35d471a818
Revert "use std encoding lib" 2019-08-23 07:59:09 -07:00
Chris Lu
fa160e00f3
Merge pull request #1051 from xushuxun/master 
volume: read dat files' last modified time correctly
 2019-08-20 22:25:37 -07:00
xushuxun
9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
divinerapier
e798e4e9c5
use std encoding lib 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-08-19 18:07:54 +08:00
xushuxun
5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu
f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
xushuxun
46ae35cd7e wdclient: clear client cache when failed to connect to master leader 2019-08-15 15:13:16 +08:00
Chris Lu
d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu
e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu
6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu
0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Chris Lu
b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu
19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu
fb90d63589 shell: Added minor changes to fs.meta.save 
* added an option to set the output file name
* added an option to see verbose output, default to non-verbose mode

related to: https://github.com/chrislusf/seaweedfs/issues/1019
 2019-07-26 09:35:22 -07:00
divinerapier
9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu
344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu
3f851feb59 1.42 2019-07-21 21:52:48 -07:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu
f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu
4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu
fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu
c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu
aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu
0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu
a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu
9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu
1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu
61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu
476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu
1e2127e261 s3: adding Location for multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/1002
 2019-07-08 12:37:45 -07:00
Chris Lu
62843991f2 s3: adjust object key 2019-07-08 12:37:20 -07:00
Chris Lu
cf2804eebd 1.41 2019-07-07 20:40:20 -07:00
Chris Lu
7872fc38ea filer leveldb2: correct directory listing 
fix https://github.com/chrislusf/seaweedfs/issues/1001
 2019-07-07 20:37:48 -07:00
Chris Lu
fe2882ce7d fix test 2019-07-02 21:42:06 -07:00
Chris Lu
36250154cc remove println 2019-07-02 21:33:23 -07:00
Chris Lu
4d579f7f57 adjust 2019-07-02 21:28:51 -07:00
Chris Lu
23d36dfbaf 1.40 
enable azure replication
 2019-07-02 21:27:05 -07:00
Chris Lu
dcc331cf5a parallelize leveldb 2019-07-02 21:25:53 -07:00
Chris Lu
88fed85bba temp skipping azure 
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
 2019-06-30 23:17:45 -07:00
Chris Lu
ce5aab66e9 add versions 2019-06-30 22:43:10 -07:00
Chris Lu
36022f9416 1.39 2019-06-30 00:50:17 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
ee933cd859 filer: add metrics for go memory, gc stats 2019-06-29 03:02:28 -07:00
Chris Lu
92c7f7e069 fix compilation error 2019-06-28 11:19:08 -07:00
Chris Lu
9f3f2f7c79 protect locations slice 
fix https://github.com/chrislusf/seaweedfs/issues/995
 2019-06-28 09:47:29 -07:00
Chris Lu
327336ecf3 filer: avoid concurrent modification to result slice 
fix https://github.com/chrislusf/seaweedfs/issues/972
 2019-06-28 01:12:41 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Lei Liu
1d9b75b536
weed.go: remove unused parameter 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-06-26 10:46:32 +08:00
Chris Lu
359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu
fdff927b37 1.38 2019-06-24 00:36:46 -07:00
Chris Lu
a8b4fe6eae adjust help text 2019-06-24 00:26:03 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu
cd45ab072a fix compilation error 2019-06-22 12:30:08 -07:00
Chris Lu
c369e5a13b add metrics for filer store 2019-06-22 12:23:25 -07:00
Chris Lu
6bc3dee5b3 refactoring 2019-06-22 10:56:54 -07:00
Chris Lu
95ad56d99d 1.37 2019-06-21 20:56:34 -07:00
Chris Lu
a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu
308102f023 fix cassandra creation problem 
fix https://github.com/chrislusf/seaweedfs/issues/986
 2019-06-21 17:10:38 -07:00
Chris Lu
fe4eb6bffa 1.36 2019-06-21 13:11:40 -07:00
Chris Lu
059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu
ff97acae62 more logs 2019-06-21 12:14:40 -07:00
Chris Lu
d0ce5b020b logging 2019-06-21 12:05:00 -07:00
Chris Lu
78994024ff adjust logging 2019-06-21 11:46:12 -07:00
Chris Lu
f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu
613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu
11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu
4b0f084b5d delete chunks only when file writing is successful 2019-06-20 21:58:35 -07:00
Chris Lu
eebc44cce3 refactoring 2019-06-20 21:57:17 -07:00
Chris Lu
3106065b58 fix startup problem with metrics 2019-06-20 09:56:49 -07:00
Chris Lu
c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu
e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu
115558e5f5 adjust counters 2019-06-17 21:02:50 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
68d1bef236 instance format 2019-06-16 21:57:32 -07:00
Chris Lu
935639b908 metrics: disk size for <collection, normal/EC> 2019-06-16 21:56:41 -07:00
Chris Lu
0fdb1e705d collect volume disk usage metrics 2019-06-16 02:44:20 -07:00
Chris Lu
289fd7eb39 count number of volumes and ec shards 2019-06-16 02:24:15 -07:00
Chris Lu
ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu
d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
be3f1f84a2 1.35 2019-06-10 21:34:06 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
f9d8bd51ad ec shard balancing 2019-06-10 21:32:56 -07:00
Chris Lu
9d9162ca35 ec.balance: collect dc rack info 2019-06-07 00:25:05 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu
7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu
6b08db65b0 fix shard count reporting 2019-06-05 01:58:37 -07:00
Chris Lu
784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
b9e138713c ec.encode: add -fullPercent option 2019-06-05 00:13:13 -07:00
Chris Lu
0e49add2bc adjust help message 2019-06-05 00:11:37 -07:00
Chris Lu
eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
0e52862586 avoid the test that depends on ordering in a different folder 2019-06-04 02:22:32 -07:00
Chris Lu
2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu
06133ae98f correct count for free ec slots, avoid removing generated shards 2019-06-04 01:49:51 -07:00
Chris Lu
6cea23d091 purge duplicated ec shards 2019-06-04 01:32:36 -07:00
Chris Lu
b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu
11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu
d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu
9ce6b949bf adjust help message 2019-06-01 10:37:08 -07:00
Chris Lu
d6e2c877fa ec.balance: use specific collection names 2019-06-01 02:00:18 -07:00
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
a72cef3c42 encode by collection 2019-05-31 15:48:40 -07:00
Chris Lu
de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu
689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
866197eee3 print out the ec balancing plan 2019-05-30 01:38:59 -07:00
Chris Lu
5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu
92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu
ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu
8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
ae499fd5aa Merge branch 'master' into erasure_coding 2019-05-19 21:38:18 -07:00
Chris Lu
698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu
693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu
f2c4c888f6 1.34 2019-05-17 22:53:30 -07:00
Chris Lu
8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu
76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB 
fix https://github.com/chrislusf/seaweedfs/issues/966
 2019-05-17 09:11:08 -07:00
Chris Lu
bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
017d0957c3 avoid switching master node if same ip and host name 
fix https://github.com/chrislusf/seaweedfs/issues/955
 2019-05-11 18:42:46 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
69b7dd398d adjust help message 2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu
cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu
4e42e7b5e7 shell: volume.balance add dataCenter filter 2019-05-06 13:30:14 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls 
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
 2019-05-06 08:10:18 -07:00
Chris Lu
8656102127 shell: volume.balance runs for each collection, and for ALL at the end 2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de refactoring 2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611 add error checking 2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05 shell: volume.list output sorted voume list 2019-05-05 20:23:50 -07:00
Chris Lu
46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu
4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer 
addressing https://github.com/chrislusf/seaweedfs/issues/717
 2019-05-03 22:33:08 -07:00
Chris Lu
5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu
8c982db25d 1.33 2019-05-03 17:27:50 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba 
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
 2019-04-29 10:42:09 -07:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
575926cdec minor 2019-04-26 09:32:37 -07:00
Chris Lu
630cf1e33e fix 2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c fix 2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c logging 2019-04-24 12:30:36 -07:00
Chris Lu
9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
067b935763 report unknown commands 2019-04-23 22:29:36 -07:00
Chris Lu
53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00
Chris Lu
0c96cfdb2e minor 2019-04-21 15:50:51 -07:00
Chris Lu
00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu
59d532a8c3 1.31 2019-04-21 14:09:50 -07:00
Chris Lu
5c411f3e5f minor 2019-04-21 13:33:32 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
64a9a0e104 shell: fix volume.copy 2019-04-20 20:48:07 -07:00
Chris Lu
5f3d0e33a1 shell: add commands volume.copy volume.delete volume.mount volume.unmount 2019-04-20 20:31:35 -07:00
Chris Lu
d06d3c57d4 add notes 2019-04-20 20:12:07 -07:00
Chris Lu
2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu
07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
af49aea0c6 weed shell: fs.meta.save adjusts meta data file name 2019-04-16 09:40:27 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
014906ec79 weed shell: add command fs.meta.load to restore meta data 2019-04-16 01:37:11 -07:00
Chris Lu
79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu
d35023c713 weed shell: add command fs.meta.save 2019-04-16 01:06:32 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
967e108b9a increase max idle conn per hosts 2019-04-15 22:00:50 -07:00
Chris Lu
f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error"" 
This reverts commit 359b7c1650.
 2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650 Revert "volume: close request body if any parsing error" 
This reverts commit 5996656141.
 2019-04-15 21:35:12 -07:00
Chris Lu
5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu
6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00
Chris Lu
e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu
3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu
2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Chris Lu
59a9e15733 1.30 2019-04-11 19:40:36 -07:00
Chris Lu
e11e127d91 close grpc connections during errors 2019-04-11 09:27:08 -07:00
Wine93
361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu
0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu
c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
stlpmo-jn
c3fa50d3b3 remove the health chekcer, because it's the same as command_volume_fix_replication 2019-04-11 13:40:31 +08:00
Chris Lu
2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn
f2031884f0 fix bug : CI build failed 2019-04-11 09:53:31 +08:00
Chris Lu
4df10e50b3 update glide.yaml 2019-04-10 14:27:28 -07:00
stlpmo-jn
c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu
35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu
a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu
6d4c44a85c 1.29 2019-04-08 22:41:59 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu
784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive 
replication: add GoCDK PubSub support
 2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu
a32abda1a3 adjust compression to optimize for speed 2019-04-07 11:01:28 -07:00
Chris Lu
c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu
174bf1e8b2 more efficient client side gzip compression 2019-04-06 14:14:28 -07:00
Chris Lu
868913aa04 printout statistics 2019-04-06 11:12:35 -07:00
Chris Lu
de7626bd22 weed shell: volume.list add summary statistics 2019-04-06 09:25:29 -07:00
Chris Lu
94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu
b2d92a2992 weed shell: fs.tree fix directory tree listing 2019-04-05 23:42:36 -07:00
Chris Lu
24c020104a weed filer.copy: parallelize the copying to increase throughput 
improvement https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 23:35:30 -07:00
Chris Lu
c789b496d8 use cached grpc client 2019-04-05 20:31:58 -07:00
Chris Lu
5808caa2f5 use cached grpc client 2019-04-05 20:26:52 -07:00
Chris Lu
cd6d35aa52 weed shell: fs.tree improvements 2019-04-05 11:40:54 -07:00
Jonathan Amsterdam
72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Jonathan Amsterdam
a6a5d80401 fix typo 2019-04-05 08:13:32 -04:00
Chris Lu
300b39b5af weed filer.copy: use cached filer grpc connection 
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 01:09:06 -07:00
Chris Lu
921231268b weed filer.copy: skip files that can not be opened 
such as socket
 2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
c94823f52c set default http idle connection per host 
possible fix https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-04 20:28:40 -07:00
Chris Lu
0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu
9cc73f4a9a fix test 2019-04-04 19:34:15 -07:00
Chris Lu
af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu
766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Jonathan Amsterdam
c0c9a8bad5 replication: add GoCDK PubSub support 2019-04-04 17:22:45 -04:00
Chris Lu
715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu
20dcb44077 fix tests 2019-04-01 23:59:31 -07:00
Chris Lu
beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu
78b9db34d5 weed s3: generate time format acceptable by aws cli s3 
fix https://github.com/chrislusf/seaweedfs/issues/911
 2019-04-01 16:13:29 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu
c5a3ff1c35 weed s3: multipart upload goes to the right bucket 
fix https://github.com/chrislusf/seaweedfs/issues/908
 2019-03-31 11:10:19 -07:00
Chris Lu
189c890715 weed replicate: replicate atomic rename to other systems 2019-03-31 08:10:47 -07:00
Chris Lu
8c823abe1f 1.28 2019-03-30 23:22:41 -07:00
Chris Lu
78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu
07775998a9 some comments 2019-03-28 00:38:30 -07:00
Chris Lu
53b81fcfcd weed shell: add fs.tree 2019-03-28 00:05:04 -07:00
Chris Lu
c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu
65757ae2fd weed filer: set content-type consistent with filer store 2019-03-27 11:41:11 -07:00
Chris Lu
37ce4c5269 weed shell: add fs.ls 2019-03-26 12:43:51 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
c913583e2d Merge branch 'master' into add_volume_follow 2019-03-25 21:19:34 -07:00
Chris Lu
91b9a68493 udpate glide 2019-03-25 09:39:54 -07:00
Chris Lu
a32797518b weed master: redirect to leader for default admin UI 
fix https://github.com/chrislusf/seaweedfs/issues/898
 2019-03-25 09:20:05 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
eaa42c3865 1.27 2019-03-23 14:14:40 -07:00
Chris Lu
cd8a3b99bb textual changes 2019-03-23 12:57:35 -07:00
Chris Lu
bd1c0735e0 weed shell: adjust help text format 2019-03-23 11:54:26 -07:00
Chris Lu
6b70b36105 weed shell: add "volume.fix.replication" 2019-03-23 11:34:09 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Chris Lu
531add52c2 weed master: skip proxied file read in http read only mode 2019-03-21 09:49:04 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Chris Lu
977b30e992 weed filer: http HEAD response header add ETag 
fix https://github.com/chrislusf/seaweedfs/issues/892
 2019-03-20 10:13:53 -07:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu
88ab932f7d refactoring function parameter 2019-03-19 23:01:23 -07:00
Chris Lu
5ae4b963a4 avoid using global rand 2019-03-19 22:20:14 -07:00
Chris Lu
f3d316a846 weed shell: adding file system disk usage fs.du 
fix https://github.com/chrislusf/seaweedfs/issues/889

echo "fs.du http://localhost:8888/some/path" | weed shell
 2019-03-19 21:58:00 -07:00
Chris Lu
cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu
eb4a54d9fe refactoring 2019-03-19 05:19:37 -07:00
Chris Lu
ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu
104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu
44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
22fbbf023b adjust collection list output 2019-03-17 17:31:05 -07:00
Chris Lu
e48267e287 adjust output 2019-03-17 17:28:29 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu
36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu
42cb05c088 add context.Context 2019-03-15 17:23:02 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu
95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu
346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
Chris Lu
99ce10daeb Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-11 00:58:17 -07:00
Chris Lu
2ce4dae795 move to 1.26 2019-03-11 00:58:12 -07:00
chenwanli
c668e55e07 Revert "fix https://github.com/chrislusf/seaweedfs/issues/780" 
This reverts commit 31038acccf.
 2019-03-11 11:48:11 +08:00
Chris Lu
6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu
ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu
0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli
f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu
219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00
Chris Lu
2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
chenwanli
900aad4935 Fix https://github.com/chrislusf/seaweedfs/issues/864 2019-02-28 11:37:44 +08:00
Chris Lu
cb07d15254 add namespace for s3 2019-02-27 00:21:37 -08:00
Chris Lu
70ac2f6ea8 add namespace for ListAllMyBucketsResult 2019-02-26 23:23:04 -08:00
chenwanli
fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu
344caf3cd7 1.25 2019-02-25 00:44:48 -08:00
Chris Lu
7bbe24dd28 volume server directly support https 2019-02-25 00:43:36 -08:00
Chris Lu
c892b898a1 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-25 00:34:25 -08:00
Chris Lu
0d84e32f56 update library versions 2019-02-25 00:34:21 -08:00
chenwanli
0bc3cdd26a Fix https://github.com/chrislusf/seaweedfs/issues/825 2019-02-25 10:47:41 +08:00
Chris Lu
0a106c1757 updated glide 2019-02-20 09:09:47 -08:00
Chris Lu
e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu
0fa1be8c4b adjust formatting and comments 2019-02-19 21:10:10 -08:00
Chris Lu
72c2e6bb8f remove unused file 2019-02-19 14:16:19 -08:00
Chris Lu
07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu
448645203a remove unused variables 2019-02-18 20:05:55 -08:00
Chris Lu
d37c3ab7a5 adjust error message 2019-02-18 18:03:27 -08:00
Chris Lu
2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu
9a4dda3011
Merge pull request #855 from chrislusf/add_jwt 
Add jwt
 2019-02-18 15:05:32 -08:00
Chris Lu
a1c7dc3806 avoid changing attributes for mount directory 2019-02-18 12:14:28 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
55761ae806 skip nil options 2019-02-16 12:49:58 -08:00
Chris Lu
98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu
31aa60ce05 Merge branch 'master' into add_jwt 2019-02-16 08:55:38 -08:00
Chris Lu
ad257ae179 simplify isLeader() logic 2019-02-16 08:55:23 -08:00
Chris Lu
712a0e19c8 Merge branch 'master' into add_jwt 2019-02-15 10:01:16 -08:00
Chris Lu
2ec6a679c2 avoid "fchmod failed" during cp 2019-02-15 10:00:27 -08:00
Chris Lu
157c0f7c01 add jwt for writes 2019-02-15 09:59:22 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
a3b0e39b06 weed mount deletion always goes to weed filer 2019-02-14 07:07:38 -08:00
Chris Lu
215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu
501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Chris Lu
729d71fd4c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-09 12:52:12 -08:00
Chris Lu
8afc632484 raft: use the first master to bootstrap the election 2019-02-09 12:52:09 -08:00
Chris Lu
744abc2690 log raft state changes 2019-02-09 12:51:14 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
1334507595 Revert "randomize based on self address" 
This reverts commit 6230eb28a6.
 2019-01-28 12:12:51 -08:00
Chris Lu
6230eb28a6 randomize based on self address 
fix #851
 2019-01-28 11:55:33 -08:00
Chris Lu
221105eea3 Revert "use the first entry to bootstrap master cluster" 
This reverts commit 40c8725ffa.
 2019-01-28 11:46:46 -08:00
Chris Lu
7a493bbefa better way to detect a master isLeader 2019-01-28 10:36:16 -08:00
Chris Lu
40c8725ffa use the first entry to bootstrap master cluster 
fix https://github.com/chrislusf/seaweedfs/issues/851
 2019-01-28 10:35:28 -08:00
Chris Lu
adcfe66034 1.24 2019-01-26 00:15:50 -06:00
Chris Lu
834f414af9 add a timeout 2019-01-26 00:15:42 -06:00
Chris Lu
09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu
5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
Chris Lu
d3f675abc6 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-01-24 22:39:41 -08:00
Chris Lu
e70f740deb set file attributes 
fix https://github.com/chrislusf/seaweedfs/issues/847
 2019-01-24 19:38:04 -08:00
bingoohuang
cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
Chris Lu
3f56b12ed4 raft: adding idle connection time out 
another attempt to fix https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-22 09:25:25 -08:00
chenwanli
f47f8be912 Fix block-waiting for leader when there are no volumes 2019-01-22 15:58:37 +08:00
Chris Lu
8ec57e18c7 1.23 2019-01-18 14:27:34 -08:00
Chris Lu
a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu
67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
Chris Lu
dfae0f4e9d 1.22 2019-01-18 00:25:20 -08:00
Chris Lu
e84dcb0bfe fix timeout issue, but can not reproduce with local built binaries 
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 00:24:40 -08:00
Chris Lu
1d103e3ed5 timeout http connections 
possible fix for https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-17 23:38:33 -08:00
Chris Lu
b881e11a16 1.21 2019-01-16 20:44:38 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
bingoohuang
219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu
2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli
b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli
80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
Chris Lu
8613e4f558
Merge pull request #827 from suilongfei/master 
change log_dir to logdir for the "flag redefined:log_dir" error when use another package with "github.com/golang/glog"
 2019-01-10 09:10:07 -08:00
Chris Lu
2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli
39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
隋龙飞@视频结构化
b864cf7e07
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:17:32 +08:00
隋龙飞@视频结构化
ea4f29d1e7
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:16:58 +08:00
Chris Lu
2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon
58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu
badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
chenwanli
41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu
e768e64fbc 1.20 2019-01-06 10:52:26 -08:00
Chris Lu
bd32108a90 disable async file deletion 2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu
fe50224ea0 refactoring 2019-01-05 15:21:56 -08:00
Chris Lu
6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu
d9f9aca453 refactoring 2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5 increase batch deletion size to 1024 2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832 reduce directory list cache usage 2019-01-03 22:21:39 -08:00
Chris Lu
7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu
ceafb7d145 1.19 2019-01-02 12:59:20 -08:00
Chris Lu
28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu
d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu
e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu
d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu
3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu
43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu
9539823c35 fix log 2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a async file chunk deletion 2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a get bytes from sync pool 2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu
81904ad336 debugging: track all memory allocations 2018-12-31 14:54:13 -08:00
Chris Lu
03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu
d6181076bb FUSE remove adjust error 2018-12-30 22:29:27 -08:00
Chris Lu
16e8610426 1.18 2018-12-30 00:52:20 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu
d8c7844d2e 1.17 2018-12-29 13:16:38 -08:00
Chris Lu
9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f set attributes include crtime 2018-12-28 22:37:18 -08:00
Chris Lu
ba9b7b62d7 1.16 2018-12-28 15:10:49 -08:00
Chris Lu
79168ab0bb adjust error and log 2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2 reuse buffer by sync pool 2018-12-28 03:27:48 -08:00
Chris Lu
260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu
2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu
d3660ffce8 avoid gzipping files without file extensions 
fix https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 12:17:05 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a avoid error log if not found 2018-12-24 17:45:26 -08:00
Chris Lu
ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu
332668a3fd 1.15 2018-12-23 12:54:41 -08:00
Chris Lu
c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu
d3839fe279 go fmt 2018-12-22 11:11:57 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu
16634235c1 1.14 and HCFS 1.0.3 2018-12-17 10:03:03 -08:00
Chris Lu
52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu
2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu
7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu
6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
66d04cb83f 1.12 2018-12-15 08:17:24 -08:00