mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-01-09 12:58:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,165 Commits 32 Branches 297 Tags 156 MiB
0aed16a9c4
Commit Graph

735 Commits

Author SHA1 Message Date
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection (#3413) 2022-08-07 00:58:53 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
chrislu
036566629a filer.sync: fix synchronization logic in active-active mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3328
 2022-08-01 00:06:18 -07:00
chrislu
303bd067b5 Revert "rename: delete source entry metadata only, skipping hard links" 
This reverts commit 03466f955e.

fix https://github.com/seaweedfs/seaweedfs/issues/3386
 2022-07-31 22:51:41 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
zzq09494
01b7aa79fa fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted 2022-07-28 12:04:37 +08:00
Konstantin Lebedev
da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu
ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu
fc8241fb5e leveldb3: add instant dropping bucket 2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44 explicit bucket aware declaration 2022-07-21 18:23:53 -07:00
Konstantin Lebedev
b6471ecd75 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
11e393dbe7 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 00:45:13 +05:00
Konstantin Lebedev
7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
chrislu
1db012485f mount: fix bug during busy writes 
fix https://github.com/chrislusf/seaweedfs/issues/3315
 2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
chrislu
fa61074513 add clientId logging 2022-07-14 12:27:34 -07:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00
guosj
354b7bdff0 replace errors.New(fmt.Sprintf(...)) with fmt.Errorf() 2022-07-13 22:49:03 +08:00
guosj
ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0 detect sequential mode with a counter 2022-07-13 02:20:03 -07:00
chrislu
1451b389a4 Avoid fatal error: sync: Unlock of unlocked RWMutex 
fix https://github.com/chrislusf/seaweedfs/issues/3306
 2022-07-13 00:58:15 -07:00
chrislu
860c207c94 typo 2022-07-12 02:56:19 -07:00
guosj
5469efa336 close SkipCheckParentDirectory flag 2022-07-11 09:44:48 +08:00
chrislu
185332b91f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-08 00:29:41 -07:00
chrislu
28add5a534 mount: fix racing conditions 
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
 2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f minor 2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8 remove dead code 2022-07-07 22:26:03 -07:00
chrislu
f2f0482dd3 mount: random read also try to use the local cache first 2022-07-07 11:50:28 -07:00
石昌林
2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
geekboood
fdacd94af5 fix: invalid chunk data when failed to read manifests 2022-06-19 16:54:04 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
chrislu
ebd6097e6f fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 
related to https://github.com/chrislusf/seaweedfs/pull/3074
 2022-06-07 22:33:35 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
9219664505 reset sync offset if peer filer resets 
fix https://github.com/chrislusf/seaweedfs/issues/3089
 2022-05-29 13:11:24 -07:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator. 2022-05-25 01:11:02 -07:00
chrislu
a5cdcc7e63 fix reading files larger than 20GB due to manifest resolving bug 2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd fix reading files larger than 20GB due to manifest resolving bug 
this is introduced since Mar 08 2022
 2022-05-23 01:14:56 -07:00
Konstantin Lebedev
d12a423aa0 refactor 
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
chrislu
bb4beebce3 prevent nil 2022-04-05 10:49:17 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu
86d4c52d6e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-26 13:33:20 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545 
avoid connect to the old filler address
 2022-03-26 13:02:49 -07:00
chrislu
cae998eda1 Revert "Revert "remove duplicated metadata subscription in filer"" 
This reverts commit 34b743c481.
 2022-03-26 13:00:16 -07:00
chrislu
34b743c481 Revert "remove duplicated metadata subscription in filer" 
This reverts commit 34742be029.

Related to https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-26 12:33:45 -07:00
chrislu
4e2388e1b5 mount: set file size if it is only on remote gateway 2022-03-21 02:09:54 -07:00
elee
beb406bbbb fix ls, onBucketDelete still not triggering 2022-03-18 22:21:57 -05:00
elee
411c0df3fe switch to multi collection, change readme 2022-03-18 21:51:16 -05:00
elee
1cea6c73d3 update readme 2022-03-18 00:34:19 -05:00
elee
25be96832a update readme with index info 2022-03-18 00:33:08 -05:00
elee
06f23aa675 put in delete folder children query 2022-03-18 00:29:48 -05:00
elee
0701feeb17 added more to readme 2022-03-18 00:26:11 -05:00
elee
171c27ddf1 zz 2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb update readme 2022-03-17 21:26:16 -05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
elee
bf745bdccb revise comment 2022-03-17 14:09:34 -05:00
elee
a0df993cef cleanup, add a few comments 2022-03-17 14:08:36 -05:00
elee
94883e2fad ok then 2022-03-17 05:23:12 -05:00
elee
f3ab6769e9 make start from indexing work 2022-03-17 05:09:13 -05:00
elee
31571dd96e fix file listing 2022-03-17 05:07:40 -05:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
6886167d5f minor optimization 2022-03-17 00:03:28 -07:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3 fix: ut test error 2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
chrislu
fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu
2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu
53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
Konstantin Lebedev
834210a9dc avoid connect to the old filler address 
https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-10 15:24:45 +05:00
zzq09494
2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu
6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
banjiaojuhao
a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu
da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu
6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu
941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu
551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
chrislu
7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu
3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu
86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu
03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
chrislu
ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu
b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu
56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
chrislu
7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00