mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 08:33:48 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,158 Commits 32 Branches 296 Tags 150 MiB
b048659749
Commit Graph

695 Commits

Author SHA1 Message Date
chrislu
77e4b1376e refactoring 2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps (#3472) 
* Fix FUSE server buffer leaks in file gaps

This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.

* Move zero logic into its own method
 2022-08-21 11:33:58 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
Guo Lei
3172c33fa6
fix redundant type from array (#3462) 2022-08-18 02:25:25 -07:00
chrislu
4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
96caf21d09 less verbose log 2022-08-15 10:03:52 -07:00
chrislu
0aeec04c31 quicker to adapt to pattern change 2022-08-07 10:14:01 -07:00
chrislu
928d29af9e fix wrong logic about reader isRandomMode() 2022-08-07 10:13:04 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection (#3413) 2022-08-07 00:58:53 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
chrislu
036566629a filer.sync: fix synchronization logic in active-active mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3328
 2022-08-01 00:06:18 -07:00
chrislu
303bd067b5 Revert "rename: delete source entry metadata only, skipping hard links" 
This reverts commit 03466f955e.

fix https://github.com/seaweedfs/seaweedfs/issues/3386
 2022-07-31 22:51:41 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
zzq09494
01b7aa79fa fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted 2022-07-28 12:04:37 +08:00
Konstantin Lebedev
da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu
ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu
fc8241fb5e leveldb3: add instant dropping bucket 2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44 explicit bucket aware declaration 2022-07-21 18:23:53 -07:00
Konstantin Lebedev
b6471ecd75 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
11e393dbe7 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 00:45:13 +05:00
Konstantin Lebedev
7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
chrislu
1db012485f mount: fix bug during busy writes 
fix https://github.com/chrislusf/seaweedfs/issues/3315
 2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
chrislu
fa61074513 add clientId logging 2022-07-14 12:27:34 -07:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00
guosj
354b7bdff0 replace errors.New(fmt.Sprintf(...)) with fmt.Errorf() 2022-07-13 22:49:03 +08:00
guosj
ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0 detect sequential mode with a counter 2022-07-13 02:20:03 -07:00
chrislu
1451b389a4 Avoid fatal error: sync: Unlock of unlocked RWMutex 
fix https://github.com/chrislusf/seaweedfs/issues/3306
 2022-07-13 00:58:15 -07:00
chrislu
860c207c94 typo 2022-07-12 02:56:19 -07:00
guosj
5469efa336 close SkipCheckParentDirectory flag 2022-07-11 09:44:48 +08:00
chrislu
185332b91f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-08 00:29:41 -07:00
chrislu
28add5a534 mount: fix racing conditions 
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
 2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f minor 2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8 remove dead code 2022-07-07 22:26:03 -07:00
chrislu
f2f0482dd3 mount: random read also try to use the local cache first 2022-07-07 11:50:28 -07:00
石昌林
2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
geekboood
fdacd94af5 fix: invalid chunk data when failed to read manifests 2022-06-19 16:54:04 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
chrislu
ebd6097e6f fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 
related to https://github.com/chrislusf/seaweedfs/pull/3074
 2022-06-07 22:33:35 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
9219664505 reset sync offset if peer filer resets 
fix https://github.com/chrislusf/seaweedfs/issues/3089
 2022-05-29 13:11:24 -07:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator. 2022-05-25 01:11:02 -07:00
chrislu
a5cdcc7e63 fix reading files larger than 20GB due to manifest resolving bug 2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd fix reading files larger than 20GB due to manifest resolving bug 
this is introduced since Mar 08 2022
 2022-05-23 01:14:56 -07:00
Konstantin Lebedev
d12a423aa0 refactor 
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00