mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-15 19:26:42 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,583 Commits 32 Branches 296 Tags 150 MiB
edeea739ed
Commit Graph

789 Commits

Author SHA1 Message Date
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu
a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu
ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu
258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu
2158d4fe4d adjust help message 2021-08-21 02:17:10 -07:00
Chris Lu
ccb3df41f8 elide secret key 2021-08-17 11:27:08 -07:00
Chris Lu
49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu
a539d64896 refactor 2021-08-15 12:09:54 -07:00
Chris Lu
ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu
cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu
889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00
Chris Lu
708debca14 remote.cache and uncache: more flexible options to select files to cache or uncache 2021-08-14 15:11:55 -07:00
Chris Lu
0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu
0f7d4556d8 shell: volume.tier.move makes up changes if volume move failed 2021-08-13 03:09:28 -07:00
Chris Lu
333cdce485 add verbose message 2021-08-13 02:57:14 -07:00
Chris Lu
057ef429ac format 2021-08-10 12:33:29 -07:00
Chris Lu
b63b042afc dedup keeps the largest replica 2021-08-10 12:30:41 -07:00
Chris Lu
0526db12e2 do not treat read only volumes differently 2021-08-10 11:37:12 -07:00
Chris Lu
e50a5b8e28 minor: print disk type 2021-08-10 11:10:09 -07:00
Chris Lu
db6275a0c8 print out balance ratio 2021-08-10 10:43:42 -07:00
Chris Lu
85832d02c0 wait for goroutines 2021-08-10 04:13:12 -07:00
Chris Lu
8ff6c9a0c6 output format 2021-08-10 03:25:18 -07:00
Chris Lu
48f448ee09 parallelize tier move 2021-08-10 03:08:29 -07:00
Chris Lu
69a6da7969 avoid fail on tail error 2021-08-10 02:50:28 -07:00
Chris Lu
18228f3044 fix help message 2021-08-10 02:48:41 -07:00
Chris Lu
02f728cb54 fix bug with remote.uncache 2021-08-09 15:24:21 -07:00
Chris Lu
402315f117 go fmt 2021-08-09 14:37:34 -07:00
Chris Lu
713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu
df85f7a1eb adjust help message 2021-08-08 22:30:12 -07:00
Chris Lu
c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu
882a93dacd fix tests 2021-08-08 17:56:26 -07:00
Chris Lu
eed26af266 Merge branch 'master' into add_remote_storage 2021-08-08 15:48:04 -07:00
Chris Lu
26c222f596 shell: volume.tier.move avoid moving all volumes to one destination 2021-08-08 15:12:39 -07:00
Chris Lu
13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu
679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
Chris Lu
0c0f77e2ae skip not found error on deletion 2021-08-06 19:35:47 -07:00
Chris Lu
8828f485c0 print volume deletion error 2021-08-06 19:30:22 -07:00
Chris Lu
e23ca3845b
Merge pull request #2237 from combineads/add_mtime 
Add crtime and mtime
 2021-08-05 21:27:13 -07:00
Chris Lu
286e5dd375 Merge branch 'master' into add_remote_storage 2021-08-05 21:07:04 -07:00
Chris Lu
1e22166939 adjust error message 2021-08-05 21:06:55 -07:00
byunghwa.yun
e84fad9acf Add crtime and mtime 2021-08-06 12:13:37 +09:00
Chris Lu
1a5d29520c add tests 2021-08-05 14:49:24 -07:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
b9ecf1e3a8 refacotring 2021-08-04 14:56:13 -07:00
Chris Lu
d84c311699 refactoring 2021-08-04 12:30:18 -07:00
Chris Lu
f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu
42969c9c62 Merge branch 'master' into add_remote_storage 2021-08-03 21:20:05 -07:00
Chris Lu
8cf0c515bf shell: volume.fix.replication retries even when there is no error #2235 
fix https://github.com/chrislusf/seaweedfs/issues/2235
 2021-08-03 09:32:55 -07:00
Chris Lu
2ae9705442 adjust text 2021-08-01 22:55:19 -07:00
Chris Lu
3739717092 Revert "adds a test" 
This reverts commit f690643b47.
 2021-08-01 22:54:45 -07:00
Chris Lu
f690643b47 adds a test 2021-08-01 22:53:50 -07:00
Chris Lu
6de786185d volume.balance: balance read only volumes first 2021-08-01 22:32:50 -07:00
byunghwa.yun
9e839cb5cc Add force option in volume.deleteEmpty command 2021-08-02 11:37:20 +09:00
Chris Lu
81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu
2ca1839d77 shell: add volume.deleteEmpty command 2021-08-01 15:33:45 -07:00
Chris Lu
9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu
f5a69a0e44 Merge branch 'master' into add_remote_storage 2021-07-31 09:39:48 -07:00
byunghwa.yun
5d4438a72f Fix typo 2021-07-31 22:23:52 +09:00
Chris Lu
a3290faf17 shell command to calculate size / number of volumes in a collection 
fix https://github.com/chrislusf/seaweedfs/issues/2224
 2021-07-30 15:49:46 -07:00
Chris Lu
c6f992b2a3 remove dead code 2021-07-30 15:18:01 -07:00
Chris Lu
899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu
4deac06da0 log fix 2021-07-28 22:25:04 -07:00
divanikus
5c6270a93a shell: ability to use wildcards for collections, all collections if ommited 2021-07-27 23:53:01 +03:00
Chris Lu
1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu
35f70c51b0 refactor 2021-07-25 03:37:37 -07:00
Chris Lu
72eec84167 shell: add fs.mkdir 2021-07-25 02:37:39 -07:00
Chris Lu
092932af8f shell: faster bucket delete 
avoid waiting for large object metadata
 2021-07-23 11:25:46 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
67537ff05b
Merge pull request #2208 from combineads/add_rm_options 2021-07-21 11:08:43 -07:00
byunghwa.yun
41d559f54e Add fs rm options 2021-07-21 21:10:36 +09:00
Chris Lu
70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu
3138805b33
Merge pull request #2200 from combineads/add_fs_rm 
weed shell: fs.rm remove file or directory
 2021-07-21 00:35:54 -07:00
Chris Lu
7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
byunghwa.yun
b23b307e08 Add fs rm 2021-07-20 08:49:28 +09:00
Chris Lu
99155e4880 add -retry option for volume.fix.replication 
fix https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:13:46 -07:00
Chris Lu
2f209675ab Added -retry option for volumeServer.evacuate 
related to https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:08:21 -07:00
Chris Lu
fb7a1be1c4 refactor 2021-07-16 11:47:37 -07:00
Chris Lu
4be5b4ff5c [volume.check.disk] skip errors and check next volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2129
 2021-07-16 02:03:32 -07:00
Chris Lu
5a838dbe53 fix compilation 2021-07-13 11:40:21 -07:00
Chris Lu
6103649ffb shell: volume.check.disk adds retries in case the volumes are just moved 
related to https://github.com/chrislusf/seaweedfs/issues/2194
 2021-07-13 11:19:56 -07:00
Chris Lu
01adc567aa shell: volume.fsck deletes a volume is the volume has only orphaned data. 
fix https://github.com/chrislusf/seaweedfs/issues/2190
 2021-07-12 11:22:00 -07:00
bingoohuang
9e48bff8ee fix typo in fs.configure help message text 2021-07-12 13:51:21 +08:00
Chris Lu
d013d6d968 shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete 
fix https://github.com/chrislusf/seaweedfs/issues/2188
 2021-07-10 23:16:06 -07:00
Chris Lu
3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu
da7bd62822 /etc files are stored inside metadata store 2021-07-09 02:33:14 -07:00
Chris Lu
08377fecb8 fixes 2021-06-27 23:32:57 -07:00
Chris Lu
1cac2f2278 shell: volume.fsck checks missing chunks in filer 
fix https://github.com/chrislusf/seaweedfs/issues/2154
 2021-06-24 23:56:24 -07:00
Chris Lu
9dd09bbb33 refactor 2021-06-24 17:22:53 -07:00
Daniel Nagy
7ce3bee0c0
Check for history file errors as well 2021-06-08 11:14:49 +02:00
Chris Lu
ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu
615cb24ba6 shell: use tab print style for completion 2021-05-31 03:29:29 -07:00
Chris Lu
b70aeb7585 bucket list only directories 2021-05-23 10:36:22 -07:00
Chris Lu
30c67e3652 minor 2021-05-19 23:59:39 -07:00
Chris Lu
76c48ffe27 optional parallel copy ec shards 
fix https://github.com/chrislusf/seaweedfs/issues/2048
 2021-05-06 01:53:35 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu
46ef1811a1 correct help message 2021-04-22 22:26:38 -07:00
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e avoid moving to another server with the same volume id 2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0 shell: add volume.tier.move 2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986 refactoring 2021-02-22 00:28:42 -08:00
Chris Lu
b961cd6208 add WIP message 2021-02-19 03:39:19 -08:00
Chris Lu
816ef0ed30 add more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1816
 2021-02-18 10:34:49 -08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
0611233f16 adjust printing 2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
Chris Lu
38efc6f572 simplify 2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu
a0c6db361c avoid nil 2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0 move to the empty nodes first 2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1 avoid nil 2021-02-16 04:16:46 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
Chris Lu
876dbe3d26 collect disk types 2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b refactoring 2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0 adjust help message 2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu
770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
f20ec82a28 update help message 
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
 2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07 avoid error message 2021-01-27 00:21:59 -08:00
Chris Lu
b3f66199db shell: volume.fix.replication adds collectionPattern with wildcard characters 
fix https://github.com/chrislusf/seaweedfs/issues/1758
 2021-01-26 22:30:39 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
f20743bb38 remove WIP status 2021-01-13 22:21:34 -08:00
Chris Lu
15da5834e1 Merge branch 'master' into support_ssd_volume 2020-12-23 14:37:37 -08:00
Baptiste Mille-Mathias
385a4dec30 [shell] Clarify bucket replication setting order 2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7 minor 2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0 shell: change bucket.list to s3.bucket.list, same for create and delete 2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb filer first, master second 2020-12-22 17:34:44 -08:00
Baptiste Mille-Mathias
0ed5345954 [shell] Add more information to bucket.create 
State the replication setting that will be honor if not defined.
 2020-12-22 17:18:59 +01:00
Chris Lu
2260864b45 a cleaner fix 2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693 fix when a volume server has only hdd volumes 2020-12-18 10:34:16 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
97dcae827b Merge branch 'master' into support_ssd_volume 2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262 collection.delete requires _default_ as the default empty collection name 
fix https://github.com/chrislusf/seaweedfs/issues/1677
 2020-12-14 01:05:20 -08:00
Chris Lu
5cffddebb9 add logging 2020-12-14 00:11:52 -08:00
Chris Lu
8baba93fce rename parameter name to "disk" 2020-12-13 12:06:21 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
17372ac3da balance ssd volumes 2020-12-13 03:40:33 -08:00
Chris Lu
83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu
57578a6cd1 change parameter help message 2020-12-06 21:54:55 -08:00
Chris Lu
9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06 refactoring 2020-12-06 20:05:06 -08:00
Chris Lu
4c72482536 shell: fix fs.configure 2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af 2.14 2020-12-06 18:56:58 -08:00
Konstantin Lebedev
03620776ec Merge branch 'upstream_master' into store_s3cred 2020-12-01 16:03:34 +05:00
Chris Lu
965413c21b shell: add volume.vacuum command 2020-11-28 23:18:02 -08:00
Chris Lu
536202e4e0 minor 2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d minor 2020-11-28 00:14:11 -08:00
Chris Lu
6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Konstantin Lebedev
4e55baf5b1 s3 config changes 2020-11-26 03:50:53 +05:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78 fix typo 2020-11-20 19:50:46 +05:00
Chris Lu
7385103057 fix volume placement validating 
fix https://github.com/chrislusf/seaweedfs/issues/1626
 2020-11-20 01:12:25 -08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
9add554feb fail fast if user configures collection for a bucket 2020-11-16 19:57:08 -08:00
Chris Lu
ed1ce3f299 adjust help message 2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e filere.conf: prefer to use json format 2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55 adds more help message 2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970 fix typo 2020-11-15 21:16:28 -08:00
Chris Lu
ee2fa14dbe filer conf: delete location specific configuration 2020-11-15 20:15:47 -08:00
Chris Lu
71056dae07 fs.configure: read and local add filer configuration 2020-11-15 18:09:35 -08:00
Konstantin Lebedev
fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu
de3bdd0651 delete old volume replica 
related to https://github.com/chrislusf/seaweedfs/issues/1607

old is:
* older compaction revision
* older modified time
* smaller volume size
 2020-11-10 12:26:05 -08:00
Konstantin Lebedev
fab01f9d8d add readonly 2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1 add shell command volume mark writable 2020-10-28 22:47:09 +05:00
Chris Lu
e73d6c9526 able to print partial file id 
address https://github.com/chrislusf/seaweedfs/issues/1552
 2020-10-21 23:48:07 -07:00
Chris Lu
2bc05cfbc8 fix typo 2020-10-21 22:26:41 -07:00
Chris Lu
1069b325dd shell: volumeServer.evacuate adds printout for ec volumes 2020-10-13 20:26:03 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
f498c71199 shell: move volume operations to use flag parsing arguments 2020-09-20 09:27:34 -07:00
Chris Lu
fcbc520373 fix print out 2020-09-20 08:47:30 -07:00
Chris Lu
56094541c8 shell: adjust command option for collection.delete 2020-09-20 08:46:16 -07:00
Chris Lu
2d3a904a82 shell: volumeServer.evacuate adds option to skip non moveable volumes 2020-09-15 00:33:49 -07:00
Chris Lu
a595916342 shell: add volumeServer.evacuate command 2020-09-14 23:47:11 -07:00
Chris Lu
5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
d15682b4a1 shell: volume.balance plan by ratio of fullness 2020-09-12 04:06:26 -07:00
Chris Lu
e2c741f76f adjust replica placement after move 2020-09-12 01:01:19 -07:00
Chris Lu
12a8f5294d test for multi dc replication 2020-09-11 01:21:17 -07:00
Chris Lu
e60b2117c3 shell: volume balance follows replica placement 2020-09-11 00:29:25 -07:00
Chris Lu
89a62e8007 refactoring 2020-09-10 23:05:00 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
d1b816212f return nil if no need to do anything 2020-09-07 16:03:05 -07:00
Chris Lu
64a621bcc8 shell: volume.fix.replication also purge over replicated volumes 2020-09-07 16:00:10 -07:00
Chris Lu
18b98fdb72 fix test 2020-09-07 12:57:38 -07:00
Chris Lu
d80538a187 refactoring 2020-09-07 12:35:02 -07:00
Chris Lu
1a09bc43d1 refactor 2020-09-07 11:31:33 -07:00
Chris Lu
2b643f477d typo 2020-09-06 12:47:55 -07:00
Chris Lu
05034aade5 printout over replicated locations 2020-09-06 12:44:02 -07:00
Chris Lu
b8f32bcab9 filer: compress stored metadata 2020-09-03 11:00:20 -07:00
Chris Lu
f76a2b2c8a printout meta data size 2020-09-03 09:51:21 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
707192f966 fix help message 2020-08-27 00:13:38 -07:00
Chris Lu
d60bcbf08a sorting chunks 2020-08-23 00:00:36 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
885c624bce volume.fsck: follow manifest chunks 2020-07-20 22:02:05 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
ae1994cbc1 erasure coding: fix cases where there are no .ecj files 2020-06-18 09:52:35 -07:00
Chris Lu
5d3ec22975 refactoring 2020-05-26 00:03:31 -07:00
Chris Lu
d4235afe4d support multiple locks 2020-05-25 18:39:53 -07:00
Chris Lu
8005786d9a logs 2020-05-17 20:27:10 -07:00
Chris Lu
75179544c3 logging 2020-05-17 20:20:21 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu
508ec01ab0 tweaking 2020-04-26 16:11:06 -07:00
Chris Lu
662b5d0cf7 support semicolon seperated command lines 2020-04-23 14:01:46 -07:00
Chris Lu
73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu
57ec7f2ba9 fix builds 2020-04-23 03:43:45 -07:00
Chris Lu
eab9701823 allow lock with an existing lock 2020-04-23 03:35:52 -07:00
Chris Lu
842e3301ee make second lock instant 2020-04-23 03:32:18 -07:00
Chris Lu
69f336e59f shell: add lock/unlock command 2020-04-23 03:11:07 -07:00
Chris Lu
30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu
9f9826e95a refactor 2020-04-21 21:28:47 -07:00
Chris Lu
076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu
8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu
91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu
dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu
c981975627 volume.fix.replication: add test cases for complicated moving 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 02:16:16 -07:00
Chris Lu
ae3d08883f volume.fsck: fix for erasure encoded volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1254
 2020-04-01 22:10:13 -07:00
Chris Lu
8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu
d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu
4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu
d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu
a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu
00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu
0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu
49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu
0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu
3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu
443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00
Chris Lu
bb9b97e2b4 add comments 2020-03-23 23:07:11 -07:00
Chris Lu
8047ec2f51 shell: fs.meta.save add option to export all fileIds for all files 2020-03-23 22:54:46 -07:00
Chris Lu
782d776d2a refactoring 2020-03-23 22:54:02 -07:00
Chris Lu
38e73463f1 fix du block and byte couting 2020-03-23 21:37:04 -07:00
Chris Lu
b51fa81f0e fix directory checking 2020-03-23 21:36:39 -07:00
Chris Lu
e666aeece2 simplify parsing filer host and port 2020-03-23 21:26:15 -07:00
Chris Lu
d151185b7e shell: desupport filer url in the arguments 2020-03-23 20:46:17 -07:00
Chris Lu
7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
410bce3925 go fmt 2020-03-01 22:39:08 -08:00
Chris Lu
6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
86cce3eb58 fix test 2020-02-25 22:29:01 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu
0644d63748 shell: add commands for bucket 2020-02-24 23:30:01 -08:00
Chris Lu
3a1d017de2 shell: ensure dc and rack does not change for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1203
 2020-02-21 21:23:25 -08:00
Chris Lu
40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
9c139e0601 update help message 2020-01-01 12:38:29 -08:00
Chris Lu
88a80ab557 fix help message 2020-01-01 12:37:38 -08:00
Chris Lu
09043c8e5a fs: synchronized meta file writing 
fix https://github.com/chrislusf/seaweedfs/issues/1175
 2019-12-29 20:19:51 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu
3ebeae0c0b ec encode distribute ec data and parity shards evenly 2019-12-24 16:52:21 -08:00
Chris Lu
1ad34a2487 ed.decode prefers servers with most data shards 2019-12-24 00:00:45 -08:00
Chris Lu
a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu
c7d5a0b00c adjust help message 2019-12-23 17:58:47 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
dda5c6d3cb fmt 2019-12-22 04:31:36 -08:00
Chris Lu
71d55ec5cb adjust help message 2019-12-20 08:22:32 -08:00
Chris Lu
d0aa0c5086 shell: add fs.meta.cat to see file meta data content 
related to https://github.com/chrislusf/seaweedfs/issues/1163
 2019-12-20 08:22:18 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
bf4b13612d adjust command name 2019-12-11 23:13:04 -08:00
Chris Lu
f81d43442b filer: speed up filer.meta.save by parallelizing 2019-12-06 22:20:59 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu
5f6b360eb7 adjust error message 2019-11-24 23:13:40 -08:00
Chris Lu
2cbe79acda fix typo in command help 2019-11-24 22:17:43 -08:00
Chris Lu
66acc2c84f passing in averageShardsPerEcNode instead of expectedTotalEcShards 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 15:36:18 -08:00
Chris Lu
40514c5362 add sortEcNodesByFreeslotsDecending and sortEcNodesByFreeslotsAscending 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 13:47:36 -08:00
Chris Lu
717ec47fa2 avoid double counting ec slot 
addressing issue found in https://github.com/chrislusf/seaweedfs/issues/1111

The ec slots are already counted on the server side.
 2019-11-12 09:33:51 -08:00
Chris Lu
9c2f3b1b0f fix rack_not_found case 
fix rack_not_found in https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-10 12:06:41 -08:00
Chris Lu
e6ab75972f use constant 2019-11-10 12:05:58 -08:00
Chris Lu
69de05f6cb go fmt 2019-10-25 07:45:12 -07:00
Chris Lu
288c45a690 fix iitial filer url 2019-10-25 07:44:37 -07:00
Lei Liu
3cc084269c master api: return http 404 when volumeId not exist 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 14:23:39 +08:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
fb90d63589 shell: Added minor changes to fs.meta.save 
* added an option to set the output file name
* added an option to see verbose output, default to non-verbose mode

related to: https://github.com/chrislusf/seaweedfs/issues/1019
 2019-07-26 09:35:22 -07:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
cd45ab072a fix compilation error 2019-06-22 12:30:08 -07:00
Chris Lu
6bc3dee5b3 refactoring 2019-06-22 10:56:54 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
f9d8bd51ad ec shard balancing 2019-06-10 21:32:56 -07:00
Chris Lu
9d9162ca35 ec.balance: collect dc rack info 2019-06-07 00:25:05 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
b9e138713c ec.encode: add -fullPercent option 2019-06-05 00:13:13 -07:00
Chris Lu
0e49add2bc adjust help message 2019-06-05 00:11:37 -07:00
Chris Lu
06133ae98f correct count for free ec slots, avoid removing generated shards 2019-06-04 01:49:51 -07:00
Chris Lu
6cea23d091 purge duplicated ec shards 2019-06-04 01:32:36 -07:00
Chris Lu
b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu
d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu
9ce6b949bf adjust help message 2019-06-01 10:37:08 -07:00
Chris Lu
d6e2c877fa ec.balance: use specific collection names 2019-06-01 02:00:18 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
a72cef3c42 encode by collection 2019-05-31 15:48:40 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
866197eee3 print out the ec balancing plan 2019-05-30 01:38:59 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
69b7dd398d adjust help message 2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
4e42e7b5e7 shell: volume.balance add dataCenter filter 2019-05-06 13:30:14 -07:00
Chris Lu
8656102127 shell: volume.balance runs for each collection, and for ALL at the end 2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de refactoring 2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611 add error checking 2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05 shell: volume.list output sorted voume list 2019-05-05 20:23:50 -07:00