mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-29 14:31:30 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,613 Commits 32 Branches 296 Tags 148 MiB
c387fe957b
Commit Graph

4358 Commits

Author SHA1 Message Date
Chris Lu
e862b2529a refactor 2021-10-01 12:10:11 -07:00
Chris Lu
88e006bc6d
Merge pull request #2349 from kmlebedev/fix_repl_volumes_per_step 
Topology update for every Nth id volume
 2021-10-01 12:05:51 -07:00
Konstantin Lebedev
5e64b22b45 check that the topology has been updated 2021-10-01 18:51:22 +05:00
Chris Lu
f58ea6a2ee add source name to error message 2021-10-01 02:19:30 -07:00
chrislusf
3ffbaaa071 use github.com/linxGnu/grocksdb 2021-10-01 03:16:03 +00:00
Konstantin Lebedev
2cecde89c3 rename opt volumesPerStep 2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b https://github.com/chrislusf/seaweedfs/issues/1846 2021-09-30 20:24:24 +05:00
Chris Lu
a067deaabc avoid possible modified location list 
fix issue 1 of https://github.com/chrislusf/seaweedfs/issues/2345
 2021-09-28 16:54:18 -07:00
Chris Lu
040443e2d1 fix possible error case 2021-09-27 23:59:45 -07:00
Chris Lu
2e9372dcf7 volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
Chris Lu
225b019fe0 stream read multiple volumes in a volume server 2021-09-27 02:51:31 -07:00
Chris Lu
1904448d4e adjust starting offset 2021-09-27 02:01:29 -07:00
Chris Lu
5956a8b05a adjust comment 2021-09-27 01:58:03 -07:00
Chris Lu
aa64f2ac4c send needle data instead of raw needle body 2021-09-27 01:53:41 -07:00
Chris Lu
c4d7ee6c5c volume server: read all files in a volume 2021-09-27 01:45:32 -07:00
Chris Lu
4a1d4d7462 s3: default to allow empty folder for better performance 2021-09-26 22:34:14 -07:00
Chris Lu
cee4d20bc1 2.70 2021-09-26 17:37:46 -07:00
Chris Lu
603ea2db73 avoid looping forever if there are no more metadata updates 2021-09-26 11:55:27 -07:00
Chris Lu
9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu
2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu
a814f3f0a8 adjust metadata tail output 2021-09-25 01:04:51 -07:00
Chris Lu
52fe86df45 use default 10000 for grpc port 2021-09-20 14:05:59 -07:00
Chris Lu
b3d88180ca Merge branch 'master' into bptree 2021-09-19 23:56:59 -07:00
Chris Lu
fa7c65bd4b 2.69 2021-09-19 21:44:06 -07:00
Chris Lu
84fdda85e7 go fmt 2021-09-19 12:06:15 -07:00
Chris Lu
ad5099e570 refactor 2021-09-19 12:02:23 -07:00
Chris Lu
5abdc0be77 s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests 2021-09-19 03:24:47 -07:00
Chris Lu
ede7a65a50 update log 2021-09-19 01:33:50 -07:00
Chris Lu
c1255d30ad add cache error log 2021-09-19 01:33:26 -07:00
Chris Lu
59dd271734 more glog 2021-09-19 00:29:51 -07:00
Chris Lu
4cbba2b1c6 add more glog to s3 2021-09-19 00:28:22 -07:00
Chris Lu
71175461ef add glog for s3 handlers 2021-09-19 00:18:59 -07:00
Chris Lu
e066e2642c add NodeStore 2021-09-18 15:32:17 -07:00
Chris Lu
198fa58e3c Merge branch 'master' into bptree 2021-09-18 14:06:16 -07:00
Chris Lu
49d971e602 filer: redis store needs to clean up batch deleted sub folders 2021-09-18 14:05:16 -07:00
Chris Lu
8f2e4be074 wip 2021-09-18 14:04:30 -07:00
Chris Lu
b751debd31 split node based on the last inserted key 2021-09-18 01:29:47 -07:00
Chris Lu
2226c3c8b6 Merge branch 'master' into bptree 2021-09-17 10:35:21 -07:00
Chris Lu
94a01fcfcb filer.remote.gateway: add options to include or exclude new bucket names to mirror 2021-09-15 23:04:16 -07:00
Chris Lu
b5f4910412 adjust help messages 2021-09-15 22:53:10 -07:00
Chris Lu
f0907eb83c add command filre.remote.gateway 2021-09-15 22:48:04 -07:00
Chris Lu
63da4bbb54 separate filer.remote.gateway command to avoid confusion 2021-09-15 22:47:17 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu
7504be58f9 Avoid xattr printed out as HTTP headers 
fix https://github.com/chrislusf/seaweedfs/issues/2336
 2021-09-13 04:00:57 -07:00
Chris Lu
96514f0f00 fix tests 2021-09-13 02:19:48 -07:00
Chris Lu
20ac710ceb 2.68 2021-09-13 02:16:09 -07:00
Chris Lu
6cd1ce8b74 erasure coding: add cleanup step if anything goes wrong 2021-09-13 01:55:49 -07:00
Chris Lu
f74b29416a better etag matching 2021-09-13 00:31:46 -07:00
Chris Lu
e9760f261e minor 2021-09-13 00:31:06 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
2c9d4c8f43 custom grpc port: filer 2021-09-12 02:28:37 -07:00
Chris Lu
e690a2be16 custom grpc port: volume server 2021-09-12 02:25:15 -07:00
Chris Lu
232ad2fe65 custom grpc port: master follower 2021-09-12 02:25:00 -07:00
Chris Lu
bd1f800efe custom grpc port: master 2021-09-12 02:19:10 -07:00
Chris Lu
7591336a22 log format 2021-09-11 14:27:57 -07:00
Chris Lu
8c6ff55226 add volume not found error type, to reduce error log 2021-09-11 14:26:41 -07:00
Chris Lu
7e600bff5f adjust log level 2021-09-11 12:59:15 -07:00
Chris Lu
a243d7e047 EC: clean up if failed in the middle 2021-09-11 02:07:14 -07:00
Chris Lu
f7f01ac209 return if error 2021-09-11 02:06:43 -07:00
Chris Lu
59cb6146a3 refactor 2021-09-11 02:06:30 -07:00
Chris Lu
5496d68f6a increase counter only if not early terminated 2021-09-11 02:05:55 -07:00
Chris Lu
2b8ab1534a mysql table DDL: minor, not necessary, but a little more strict 
the directory is already hashed into dirhash. So this change is not strictly required.
 2021-09-08 23:55:18 -07:00
joshuafc
5654d0d60d CompactionTableSizeMultiplier of leveldb use default value. #2325 
To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.
 2021-09-09 10:42:34 +08:00
Chris Lu
0b4269b6a8 remember commands even if failed 2021-09-08 15:55:19 -07:00
Chris Lu
0207f5fe9b replicated remote.cache 2021-09-08 15:54:55 -07:00
Chris Lu
00edcf71fc fix compilation 2021-09-08 07:27:03 -07:00
Chris Lu
574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
zhoub
6a7ed1bd0e add bloom filter to leveldb_store to improve fuse performance. 2021-09-07 21:09:10 +08:00
Chris Lu
9fdf02bcda remove detecting ipv6 
Got this error on my local:

transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host

related to https://github.com/chrislusf/seaweedfs/pull/2310
 2021-09-07 02:31:34 -07:00
Chris Lu
889a39a24f ipv6 2021-09-07 02:30:25 -07:00
Chris Lu
8c6d706328 2.67 2021-09-07 00:08:03 -07:00
Chris Lu
2b1feb732c remote.cache supports replication 2021-09-06 18:30:44 -07:00
Chris Lu
3adc3da291 refactor 2021-09-06 16:35:55 -07:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
64f6532fbe update help message 2021-09-06 16:16:22 -07:00
Chris Lu
308d4b2dae filer.remote.sync: createBucketWithRandomSuffix defaults to true 2021-09-06 15:14:26 -07:00
Chris Lu
8e4b43a017 minor 2021-09-06 15:13:38 -07:00
Chris Lu
c218ef20c7 filer.remote.sync: automatically detect the primary remote storage 2021-09-06 15:10:55 -07:00
Chris Lu
1702ce5395 remove unused code 2021-09-06 12:01:44 -07:00
Chris Lu
0cda61d539 return error if not found 2021-09-05 23:43:44 -07:00
Chris Lu
f62df7f695 fix wrong srcPath checking 2021-09-05 23:37:40 -07:00
Chris Lu
e93d4935e3 add other replica locations when assigning volumes 2021-09-05 23:32:25 -07:00
Chris Lu
7a13816e94 refactor 2021-09-05 23:17:15 -07:00
Chris Lu
6022db6d6a 2.66 2021-09-05 16:21:14 -07:00
Chris Lu
006c01a519 fix format 2021-09-05 16:18:50 -07:00
Chris Lu
65af3cf4df master: disconnect only the phantom volume server 
fix https://github.com/chrislusf/seaweedfs/issues/2311
 2021-09-05 15:20:03 -07:00
Chris Lu
57a95887d2 remote.cache remote.uncache supports all mounted directories 2021-09-05 14:47:06 -07:00
Chris Lu
28f45f8fa6 re-org 2021-09-05 14:38:31 -07:00
Chris Lu
c735608685 obfuscate secret keys on display 2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu
2348e8d8da
Merge pull request #2310 from nivekuil/ipv6 
Detect ipv6 addresses
 2021-09-05 10:56:44 -07:00
nivekuil
0fe9d2997b Detect ipv6 addresses 2021-09-05 06:21:40 -07:00
Chris Lu
bdefdee4e6 filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts 2021-09-04 22:46:28 -07:00
Chris Lu
d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu
98c68d7a79 filer.remote.sync fix upload logic 2021-09-04 18:46:28 -07:00
Chris Lu
d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu
c5ee03d6af format 2021-09-04 13:57:55 -07:00
Chris Lu
cc278a5928 Update Makefile 2021-09-04 13:46:49 -07:00
Chris Lu
16a342eb8a re-ordering 2021-09-04 13:46:44 -07:00
Chris Lu
796b2eb929 remember sync time 2021-09-04 13:46:22 -07:00
Chris Lu
53b9b521c9 adjust error message 2021-09-04 13:46:06 -07:00
Chris Lu
49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu
99d4594ee8 update remote storage mount mapping 2021-09-04 05:22:34 -07:00
Chris Lu
df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu
e9ce099453 rename 2021-09-04 05:03:51 -07:00
Chris Lu
8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu
13cc2ef723 when creating bucket, use remote storage client for default storage 2021-09-04 04:54:55 -07:00
Chris Lu
63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu
4b28c5f6c3 filer.remote.sync: split into buckets mode and single directory mode 2021-09-04 04:35:46 -07:00
Chris Lu
da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu
8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu
bbc77f7af4 fix compilation 2021-09-03 22:56:59 -07:00
Chris Lu
0652805236 cloud drive: add createBucket() deleteBucket() 2021-09-03 22:30:55 -07:00
Chris Lu
03a31587ce go fmt 2021-09-03 20:42:28 -07:00
Chris Lu
83cd0fc739 cloud drive: add list buckets 2021-09-03 20:42:02 -07:00
Chris Lu
fbfc90fd1e adjust formatting remote location 2021-09-03 18:52:37 -07:00
Chris Lu
491f7636f8 s3: CopyObject return http Status 400 Bad Request for non-existing source 
fix https://github.com/chrislusf/seaweedfs/issues/2306
 2021-09-03 17:38:39 -07:00
Chris Lu
bca4a9de78 simplify 2021-09-02 23:09:24 -07:00
Chris Lu
958125bd02 conforming to http user agent common practice 2021-09-02 22:55:35 -07:00
Chris Lu
11a496404b reset wait time 2021-09-02 19:55:01 -07:00
Chris Lu
7f2a8246a9 minor 2021-09-02 12:32:59 -07:00
Chris Lu
1b942dfa3c add error message 2021-09-02 12:32:46 -07:00
Chris Lu
9a73b0e3c9 refactor 2021-09-02 07:07:16 -07:00
Chris Lu
e281f0fa82 refactor 2021-09-02 06:53:21 -07:00
Chris Lu
24fda6f47f remove QR code for mobile upload 2021-09-01 16:59:37 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
d1a4e19a3f volume: copy file also copies modification time 
to ensure ttl can work well
 2021-09-01 02:42:57 -07:00
Chris Lu
3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu
3faaa6e360 ensure cached client with updated storage conf 2021-09-01 01:27:45 -07:00
Chris Lu
8e125339d5 line wrap 2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f add additionalPathPrefixes 2021-08-31 23:26:26 -07:00
Chris Lu
43fd11278e support follow additional path prefixes 2021-08-31 23:23:08 -07:00
Chris Lu
cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu
72d4ff6b28
Merge pull request #2280 from blacktear23/filerstore-tikv 
Use TiKV as Filerstore
 2021-08-30 02:21:14 -07:00
Chris Lu
f811fd0903 fix build 2021-08-30 02:03:08 -07:00
yulai.li
b17b81529e Add build tags for TiKV filerstore 2021-08-30 15:59:25 +08:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu
44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu
18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu
21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu
5adfdd7982 change to a new bloom filter library 2021-08-29 21:26:38 -07:00
Chris Lu
05f32376eb add cluster id in filer configuration response 2021-08-29 21:02:10 -07:00
Chris Lu
eacaa44dc2 refactor 2021-08-29 19:13:48 -07:00
Chris Lu
6deee4c0b9 import hdfs 2021-08-29 18:49:43 -07:00
Chris Lu
a31f2907f0 cloud drive: filer.remote.sync supports remove folder 2021-08-29 18:46:28 -07:00
Chris Lu
001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu
05d2774117 refactoring 2021-08-28 22:49:38 -07:00
Chris Lu
975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu
bec3f63298 2.65 2021-08-28 05:27:33 -07:00
Chris Lu
223c1b3571 adjust invalidation logic 2021-08-28 05:21:01 -07:00
nivekuil
8c523d50ad Invalidate mount meta cache more aggressively 2021-08-28 04:26:37 -07:00
Chris Lu
65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu
a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu
a9b41900d8 handle cases when mounted directory does not exist 2021-08-28 01:56:56 -07:00
Chris Lu
0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu
dc481c081c remove unused function 2021-08-27 22:18:54 -07:00
Chris Lu
9242f3aaeb fix build 2021-08-26 17:33:57 -07:00
Chris Lu
49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu
6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac Add deleterange_concurrency to filer configuration file 2021-08-26 18:25:08 +08:00
yulai.li
318757ef8c Change DeleteFolderChildren to DeleteRange api 2021-08-26 17:49:56 +08:00
yulai.li
de8ef28460 Update filer.toml scaffold to support tikv store 2021-08-26 16:26:27 +08:00
yulai.li
546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
yulai.li
2088f28424 init post 2021-08-26 15:20:18 +08:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu
9bcf94b2b1 ensure multi-threaded correctness 2021-08-25 17:28:50 -07:00
Chris Lu
a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu
7c39a18ba5 update azure library 2021-08-24 00:32:35 -07:00
Chris Lu
a78d0227cd adjust package name 2021-08-23 23:19:31 -07:00
Chris Lu
12631a3f5b cloud drive: gcs simplify a little bit 2021-08-23 14:43:01 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Chris Lu
2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu
95e2b83ca5 fix format 2021-08-23 00:49:59 -07:00
Chris Lu
ff7dc3b44c 2.64 2021-08-23 00:39:15 -07:00
Chris Lu
ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu
258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu
df1d6133a8 bptree does not work well for auto-increasing keys 2021-08-22 18:19:26 -07:00
Chris Lu
3b2c39f1be mount: set name when mount path equals mount folder 
fix https://github.com/chrislusf/seaweedfs/issues/2275#issuecomment-903255876
 2021-08-22 04:59:09 -07:00
Chris Lu
51c8f2518f change key type to ItemKey 2021-08-21 15:54:42 -07:00
Chris Lu
b3e49d2758 change value type to ItemValue 2021-08-21 15:52:17 -07:00