mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-28 13:31:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,678 Commits 32 Branches 296 Tags 148 MiB
90d55cd179
Commit Graph

645 Commits

Author SHA1 Message Date
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu
794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu
5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
e862b2529a refactor 2021-10-01 12:10:11 -07:00
Konstantin Lebedev
5e64b22b45 check that the topology has been updated 2021-10-01 18:51:22 +05:00
Konstantin Lebedev
2cecde89c3 rename opt volumesPerStep 2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b https://github.com/chrislusf/seaweedfs/issues/1846 2021-09-30 20:24:24 +05:00
Chris Lu
ad5099e570 refactor 2021-09-19 12:02:23 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu
6cd1ce8b74 erasure coding: add cleanup step if anything goes wrong 2021-09-13 01:55:49 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
0b4269b6a8 remember commands even if failed 2021-09-08 15:55:19 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu
8e4b43a017 minor 2021-09-06 15:13:38 -07:00
Chris Lu
57a95887d2 remote.cache remote.uncache supports all mounted directories 2021-09-05 14:47:06 -07:00
Chris Lu
c735608685 obfuscate secret keys on display 2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu
d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu
d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu
49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu
df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu
8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu
63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu
da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu
cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu
44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu
18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu
21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu
001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu
975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu
65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu
a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu
0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu
49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu
6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu
a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu
ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu
258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu
2158d4fe4d adjust help message 2021-08-21 02:17:10 -07:00
Chris Lu
ccb3df41f8 elide secret key 2021-08-17 11:27:08 -07:00
Chris Lu
49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu
a539d64896 refactor 2021-08-15 12:09:54 -07:00
Chris Lu
ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu
cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu
889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00
Chris Lu
708debca14 remote.cache and uncache: more flexible options to select files to cache or uncache 2021-08-14 15:11:55 -07:00
Chris Lu
0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu
0f7d4556d8 shell: volume.tier.move makes up changes if volume move failed 2021-08-13 03:09:28 -07:00
Chris Lu
333cdce485 add verbose message 2021-08-13 02:57:14 -07:00
Chris Lu
057ef429ac format 2021-08-10 12:33:29 -07:00
Chris Lu
b63b042afc dedup keeps the largest replica 2021-08-10 12:30:41 -07:00
Chris Lu
0526db12e2 do not treat read only volumes differently 2021-08-10 11:37:12 -07:00
Chris Lu
e50a5b8e28 minor: print disk type 2021-08-10 11:10:09 -07:00
Chris Lu
db6275a0c8 print out balance ratio 2021-08-10 10:43:42 -07:00
Chris Lu
85832d02c0 wait for goroutines 2021-08-10 04:13:12 -07:00
Chris Lu
8ff6c9a0c6 output format 2021-08-10 03:25:18 -07:00
Chris Lu
48f448ee09 parallelize tier move 2021-08-10 03:08:29 -07:00
Chris Lu
69a6da7969 avoid fail on tail error 2021-08-10 02:50:28 -07:00
Chris Lu
18228f3044 fix help message 2021-08-10 02:48:41 -07:00
Chris Lu
02f728cb54 fix bug with remote.uncache 2021-08-09 15:24:21 -07:00
Chris Lu
402315f117 go fmt 2021-08-09 14:37:34 -07:00
Chris Lu
713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu
df85f7a1eb adjust help message 2021-08-08 22:30:12 -07:00
Chris Lu
c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu
882a93dacd fix tests 2021-08-08 17:56:26 -07:00
Chris Lu
eed26af266 Merge branch 'master' into add_remote_storage 2021-08-08 15:48:04 -07:00
Chris Lu
26c222f596 shell: volume.tier.move avoid moving all volumes to one destination 2021-08-08 15:12:39 -07:00
Chris Lu
13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu
679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
Chris Lu
0c0f77e2ae skip not found error on deletion 2021-08-06 19:35:47 -07:00
Chris Lu
8828f485c0 print volume deletion error 2021-08-06 19:30:22 -07:00
Chris Lu
e23ca3845b
Merge pull request #2237 from combineads/add_mtime 
Add crtime and mtime
 2021-08-05 21:27:13 -07:00
Chris Lu
286e5dd375 Merge branch 'master' into add_remote_storage 2021-08-05 21:07:04 -07:00
Chris Lu
1e22166939 adjust error message 2021-08-05 21:06:55 -07:00
byunghwa.yun
e84fad9acf Add crtime and mtime 2021-08-06 12:13:37 +09:00
Chris Lu
1a5d29520c add tests 2021-08-05 14:49:24 -07:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
b9ecf1e3a8 refacotring 2021-08-04 14:56:13 -07:00
Chris Lu
d84c311699 refactoring 2021-08-04 12:30:18 -07:00
Chris Lu
f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu
42969c9c62 Merge branch 'master' into add_remote_storage 2021-08-03 21:20:05 -07:00
Chris Lu
8cf0c515bf shell: volume.fix.replication retries even when there is no error #2235 
fix https://github.com/chrislusf/seaweedfs/issues/2235
 2021-08-03 09:32:55 -07:00
Chris Lu
2ae9705442 adjust text 2021-08-01 22:55:19 -07:00
Chris Lu
3739717092 Revert "adds a test" 
This reverts commit f690643b47.
 2021-08-01 22:54:45 -07:00
Chris Lu
f690643b47 adds a test 2021-08-01 22:53:50 -07:00
Chris Lu
6de786185d volume.balance: balance read only volumes first 2021-08-01 22:32:50 -07:00
byunghwa.yun
9e839cb5cc Add force option in volume.deleteEmpty command 2021-08-02 11:37:20 +09:00
Chris Lu
81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu
2ca1839d77 shell: add volume.deleteEmpty command 2021-08-01 15:33:45 -07:00
Chris Lu
9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu
f5a69a0e44 Merge branch 'master' into add_remote_storage 2021-07-31 09:39:48 -07:00
byunghwa.yun
5d4438a72f Fix typo 2021-07-31 22:23:52 +09:00
Chris Lu
a3290faf17 shell command to calculate size / number of volumes in a collection 
fix https://github.com/chrislusf/seaweedfs/issues/2224
 2021-07-30 15:49:46 -07:00
Chris Lu
c6f992b2a3 remove dead code 2021-07-30 15:18:01 -07:00
Chris Lu
899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu
4deac06da0 log fix 2021-07-28 22:25:04 -07:00
divanikus
5c6270a93a shell: ability to use wildcards for collections, all collections if ommited 2021-07-27 23:53:01 +03:00
Chris Lu
1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu
35f70c51b0 refactor 2021-07-25 03:37:37 -07:00
Chris Lu
72eec84167 shell: add fs.mkdir 2021-07-25 02:37:39 -07:00
Chris Lu
092932af8f shell: faster bucket delete 
avoid waiting for large object metadata
 2021-07-23 11:25:46 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
67537ff05b
Merge pull request #2208 from combineads/add_rm_options 2021-07-21 11:08:43 -07:00
byunghwa.yun
41d559f54e Add fs rm options 2021-07-21 21:10:36 +09:00
Chris Lu
70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu
3138805b33
Merge pull request #2200 from combineads/add_fs_rm 
weed shell: fs.rm remove file or directory
 2021-07-21 00:35:54 -07:00
Chris Lu
7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
byunghwa.yun
b23b307e08 Add fs rm 2021-07-20 08:49:28 +09:00
Chris Lu
99155e4880 add -retry option for volume.fix.replication 
fix https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:13:46 -07:00
Chris Lu
2f209675ab Added -retry option for volumeServer.evacuate 
related to https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:08:21 -07:00
Chris Lu
fb7a1be1c4 refactor 2021-07-16 11:47:37 -07:00
Chris Lu
4be5b4ff5c [volume.check.disk] skip errors and check next volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2129
 2021-07-16 02:03:32 -07:00
Chris Lu
5a838dbe53 fix compilation 2021-07-13 11:40:21 -07:00
Chris Lu
6103649ffb shell: volume.check.disk adds retries in case the volumes are just moved 
related to https://github.com/chrislusf/seaweedfs/issues/2194
 2021-07-13 11:19:56 -07:00
Chris Lu
01adc567aa shell: volume.fsck deletes a volume is the volume has only orphaned data. 
fix https://github.com/chrislusf/seaweedfs/issues/2190
 2021-07-12 11:22:00 -07:00
bingoohuang
9e48bff8ee fix typo in fs.configure help message text 2021-07-12 13:51:21 +08:00
Chris Lu
d013d6d968 shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete 
fix https://github.com/chrislusf/seaweedfs/issues/2188
 2021-07-10 23:16:06 -07:00
Chris Lu
3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu
da7bd62822 /etc files are stored inside metadata store 2021-07-09 02:33:14 -07:00
Chris Lu
08377fecb8 fixes 2021-06-27 23:32:57 -07:00
Chris Lu
1cac2f2278 shell: volume.fsck checks missing chunks in filer 
fix https://github.com/chrislusf/seaweedfs/issues/2154
 2021-06-24 23:56:24 -07:00
Chris Lu
9dd09bbb33 refactor 2021-06-24 17:22:53 -07:00
Daniel Nagy
7ce3bee0c0
Check for history file errors as well 2021-06-08 11:14:49 +02:00
Chris Lu
ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu
615cb24ba6 shell: use tab print style for completion 2021-05-31 03:29:29 -07:00
Chris Lu
b70aeb7585 bucket list only directories 2021-05-23 10:36:22 -07:00
Chris Lu
30c67e3652 minor 2021-05-19 23:59:39 -07:00
Chris Lu
76c48ffe27 optional parallel copy ec shards 
fix https://github.com/chrislusf/seaweedfs/issues/2048
 2021-05-06 01:53:35 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu
46ef1811a1 correct help message 2021-04-22 22:26:38 -07:00
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00