mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-15 02:59:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,477 Commits 32 Branches 296 Tags 150 MiB
4dba102137
Commit Graph

526 Commits

Author SHA1 Message Date
Chris Lu
17e08a16f4 fix redis3 deletion 2021-10-09 04:54:48 -07:00
Chris Lu
e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu
0a856241fe avoid int bigger than math.MaxInt32 
fix https://github.com/chrislusf/seaweedfs/issues/2363
 2021-10-07 21:12:57 -07:00
Chris Lu
f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu
2336a397dc use pipeline to save some time 2021-10-06 20:40:18 -07:00
Chris Lu
371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d test with real redis 
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
 2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu
947add39e6 clean up *SkipListElementReference loaded from Redis 2021-10-04 02:31:38 -07:00
Chris Lu
2b9aab3442 use 1000 per batch 2021-10-04 01:03:40 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
chrislusf
3ffbaaa071 use github.com/linxGnu/grocksdb 2021-10-01 03:16:03 +00:00
Chris Lu
040443e2d1 fix possible error case 2021-09-27 23:59:45 -07:00
Chris Lu
9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu
2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu
198fa58e3c Merge branch 'master' into bptree 2021-09-18 14:06:16 -07:00
Chris Lu
49d971e602 filer: redis store needs to clean up batch deleted sub folders 2021-09-18 14:05:16 -07:00
Chris Lu
8f2e4be074 wip 2021-09-18 14:04:30 -07:00
Chris Lu
96514f0f00 fix tests 2021-09-13 02:19:48 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
joshuafc
5654d0d60d CompactionTableSizeMultiplier of leveldb use default value. #2325 
To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.
 2021-09-09 10:42:34 +08:00
Chris Lu
574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
zhoub
6a7ed1bd0e add bloom filter to leveldb_store to improve fuse performance. 2021-09-07 21:09:10 +08:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
28f45f8fa6 re-org 2021-09-05 14:38:31 -07:00
Chris Lu
df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu
e9ce099453 rename 2021-09-04 05:03:51 -07:00
Chris Lu
8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu
8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
yulai.li
b17b81529e Add build tags for TiKV filerstore 2021-08-30 15:59:25 +08:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu
dc481c081c remove unused function 2021-08-27 22:18:54 -07:00
Chris Lu
9242f3aaeb fix build 2021-08-26 17:33:57 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac Add deleterange_concurrency to filer configuration file 2021-08-26 18:25:08 +08:00
yulai.li
318757ef8c Change DeleteFolderChildren to DeleteRange api 2021-08-26 17:49:56 +08:00
yulai.li
546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
yulai.li
2088f28424 init post 2021-08-26 15:20:18 +08:00
Chris Lu
9bcf94b2b1 ensure multi-threaded correctness 2021-08-25 17:28:50 -07:00
Chris Lu
40dc283b2d fix locating data chunks 2021-08-15 23:07:58 -07:00
Chris Lu
72eb6d5b9d ensure no writes to remote storage if content is not changed 2021-08-15 20:23:41 -07:00
Chris Lu
5d5a21ba2d adjust log format 2021-08-15 19:46:45 -07:00
Chris Lu
49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu
8f7d2d317f readerAt need to use the right offset 
fix https://github.com/chrislusf/seaweedfs/issues/2259
 2021-08-15 11:55:58 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
0c66b173a4 fix 2021-08-13 11:31:43 -07:00
Chris Lu
e02a317d3d adjust retry logic in case some data is partially written 2021-08-13 11:30:38 -07:00
Chris Lu
2d519c6cb6 adjust the retry logic 2021-08-13 11:13:30 -07:00
Chris Lu
b961fcd338 filer: stream read from volume server, reduce memory usage 2021-08-13 11:00:11 -07:00
Chris Lu
5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu
bfac55e6c0 avoid integer overflow 
fix https://github.com/chrislusf/seaweedfs/issues/2254
 2021-08-11 22:22:49 -07:00
Chris Lu
9d85569c55 ensure using local quorum consistency 2021-08-10 05:10:57 -07:00
Chris Lu
713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu
c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu
13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu
46b0cb8c86 Merge branch 'master' into add_remote_storage 2021-08-07 15:41:27 -07:00
Chris Lu
de730b079d ChunkStreamReader implenents io.ReaderAt 2021-08-07 15:41:07 -07:00
Chris Lu
59732a0529 refactoring 2021-08-07 15:35:27 -07:00
Chris Lu
ecb234f75a refactor 2021-08-07 14:46:23 -07:00
Chris Lu
679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
byunghwa.yun
f3dc909b21 Change default permissions 2021-08-06 12:24:35 +09:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu
81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu
89933c46d2 s3: skip hidden directories in /buckets folder 2021-08-01 12:28:08 -07:00
Chris Lu
767edd3c08 rename 2021-07-31 23:52:09 -07:00
Chris Lu
9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu
899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu
c090d6bb25 add ReadRemote(), add read remote setup when filer starts 2021-07-28 22:43:12 -07:00
Chris Lu
1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
byunghwa.yun
2b28a818f1 Fix mysql sql for batch delete 2021-07-25 11:06:14 +09:00
Chris Lu
bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter 
add bloom filter for filer leveldb/rocksdb
 2021-07-22 23:35:54 -07:00
qieqieplus
2b46df13f8 add bloom filter for filer leveldb/rocksdb 2021-07-23 14:05:59 -06:00
Chris Lu
182288f860 filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu
a125c8fbe2
Merge pull request #2204 from nivekuil/tokenpolicy 
cassandra: use LocalQuorum for all queries
 2021-07-20 15:37:31 -07:00
nivekuil
b9a67d46c5 cassandra: use LocalQuorum for all queries 
This changes this filer store from eventual to strong consistency at the cost
of read performance.
 2021-07-20 14:47:39 -07:00
Chris Lu
54c8bc8673 Merge branch 'remote_overlay' 2021-07-19 23:18:35 -07:00
Chris Lu
7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
Chris Lu
93c37cfded
Update filechunk_manifest.go 2021-07-19 18:41:41 -07:00
Chris Lu
450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
nivekuil
2faf96f002 cassandra: Use LocalOne instead of One consistency 2021-07-16 04:29:46 -07:00
Chris Lu
3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu
ecce300964 s3 config read via grpc 2021-07-09 02:48:03 -07:00
Chris Lu
2cb8b31ea7
Merge pull request #2179 from nivekuil/tokenpolicy 
Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default
 2021-07-02 13:51:53 -07:00
nivekuil
8425705643 Cassandra: Use TokenAwareHostPolicy by default with fallback 
See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing
 2021-07-02 13:50:01 -07:00
陈杨文
6f683e6572
also failed on mips64 
alpine support mips64, will try to build on next version
 2021-06-30 04:36:00 +08:00
Chris Lu
41db292332 skip s390 ppc64le 
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
 2021-06-29 10:06:38 -07:00
Chris Lu
cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
Konstantin Lebedev
6aa1a56ec8 avoid crashes Galera Cluster 
https://github.com/chrislusf/seaweedfs/issues/2125
 2021-06-15 18:12:39 +05:00
Chris Lu
28a4a1f8d6 fix for mysql2 postgres2 on fast dropping buckets 2021-06-13 07:31:56 -07:00
Chris Lu
9357911a95 remove all bucket metadata 
fix https://github.com/chrislusf/seaweedfs/issues/2118
 2021-06-10 23:37:54 -07:00
Chris Lu
21ad9a4ac2 filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key 2021-06-06 13:12:01 -07:00
Chris Lu
ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu
67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev
0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev
bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Chris Lu
064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu
2d0a0733ec filer: bootstrap from peer filer 
fix https://github.com/chrislusf/seaweedfs/issues/1861
 2021-05-22 03:17:02 -07:00
Konstantin Lebedev
3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev
26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Konstantin Lebedev
42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Konstantin Lebedev
03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu
1737af480a adjust logs 2021-05-10 21:47:51 -07:00
Chris Lu
55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Konstantin Lebedev
c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Chris Lu
79f2e780c1 ensure name pattern checking is case sensitive 2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu
89eb9f6e70 clean up .uploads directory in mysql and postgres tables 
fix https://github.com/chrislusf/seaweedfs/issues/1957

When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
 2021-04-22 23:23:23 -07:00
Chris Lu
d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu
7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Konstantin Lebedev
58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Konstantin Lebedev
4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Chris Lu
5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00