mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 00:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,377 Commits 32 Branches 296 Tags 152 MiB
f08f95ac80
Commit Graph

751 Commits

Author SHA1 Message Date
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
chrislu
fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher (#3473) 
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
 2022-08-21 11:54:02 -07:00
chrislu
388f82f322 minor 2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e refactoring 2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps (#3472) 
* Fix FUSE server buffer leaks in file gaps

This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.

* Move zero logic into its own method
 2022-08-21 11:33:58 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
Guo Lei
3172c33fa6
fix redundant type from array (#3462) 2022-08-18 02:25:25 -07:00
chrislu
4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
96caf21d09 less verbose log 2022-08-15 10:03:52 -07:00
chrislu
0aeec04c31 quicker to adapt to pattern change 2022-08-07 10:14:01 -07:00
chrislu
928d29af9e fix wrong logic about reader isRandomMode() 2022-08-07 10:13:04 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection (#3413) 2022-08-07 00:58:53 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
chrislu
036566629a filer.sync: fix synchronization logic in active-active mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3328
 2022-08-01 00:06:18 -07:00
chrislu
303bd067b5 Revert "rename: delete source entry metadata only, skipping hard links" 
This reverts commit 03466f955e.

fix https://github.com/seaweedfs/seaweedfs/issues/3386
 2022-07-31 22:51:41 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
zzq09494
01b7aa79fa fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted 2022-07-28 12:04:37 +08:00
Konstantin Lebedev
da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu
ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu
fc8241fb5e leveldb3: add instant dropping bucket 2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44 explicit bucket aware declaration 2022-07-21 18:23:53 -07:00
Konstantin Lebedev
b6471ecd75 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
11e393dbe7 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 00:45:13 +05:00
Konstantin Lebedev
7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
chrislu
1db012485f mount: fix bug during busy writes 
fix https://github.com/chrislusf/seaweedfs/issues/3315
 2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
chrislu
fa61074513 add clientId logging 2022-07-14 12:27:34 -07:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00
guosj
354b7bdff0 replace errors.New(fmt.Sprintf(...)) with fmt.Errorf() 2022-07-13 22:49:03 +08:00
guosj
ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0 detect sequential mode with a counter 2022-07-13 02:20:03 -07:00
chrislu
1451b389a4 Avoid fatal error: sync: Unlock of unlocked RWMutex 
fix https://github.com/chrislusf/seaweedfs/issues/3306
 2022-07-13 00:58:15 -07:00
chrislu
860c207c94 typo 2022-07-12 02:56:19 -07:00
guosj
5469efa336 close SkipCheckParentDirectory flag 2022-07-11 09:44:48 +08:00
chrislu
185332b91f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-08 00:29:41 -07:00
chrislu
28add5a534 mount: fix racing conditions 
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
 2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f minor 2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8 remove dead code 2022-07-07 22:26:03 -07:00
chrislu
f2f0482dd3 mount: random read also try to use the local cache first 2022-07-07 11:50:28 -07:00
石昌林
2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
geekboood
fdacd94af5 fix: invalid chunk data when failed to read manifests 2022-06-19 16:54:04 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
chrislu
ebd6097e6f fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 
related to https://github.com/chrislusf/seaweedfs/pull/3074
 2022-06-07 22:33:35 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
9219664505 reset sync offset if peer filer resets 
fix https://github.com/chrislusf/seaweedfs/issues/3089
 2022-05-29 13:11:24 -07:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator. 2022-05-25 01:11:02 -07:00
chrislu
a5cdcc7e63 fix reading files larger than 20GB due to manifest resolving bug 2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd fix reading files larger than 20GB due to manifest resolving bug 
this is introduced since Mar 08 2022
 2022-05-23 01:14:56 -07:00
Konstantin Lebedev
d12a423aa0 refactor 
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
chrislu
bb4beebce3 prevent nil 2022-04-05 10:49:17 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu
86d4c52d6e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-26 13:33:20 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545 
avoid connect to the old filler address
 2022-03-26 13:02:49 -07:00
chrislu
cae998eda1 Revert "Revert "remove duplicated metadata subscription in filer"" 
This reverts commit 34b743c481.
 2022-03-26 13:00:16 -07:00
chrislu
34b743c481 Revert "remove duplicated metadata subscription in filer" 
This reverts commit 34742be029.

Related to https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-26 12:33:45 -07:00
chrislu
4e2388e1b5 mount: set file size if it is only on remote gateway 2022-03-21 02:09:54 -07:00
elee
beb406bbbb fix ls, onBucketDelete still not triggering 2022-03-18 22:21:57 -05:00
elee
411c0df3fe switch to multi collection, change readme 2022-03-18 21:51:16 -05:00
elee
1cea6c73d3 update readme 2022-03-18 00:34:19 -05:00
elee
25be96832a update readme with index info 2022-03-18 00:33:08 -05:00
elee
06f23aa675 put in delete folder children query 2022-03-18 00:29:48 -05:00
elee
0701feeb17 added more to readme 2022-03-18 00:26:11 -05:00
elee
171c27ddf1 zz 2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb update readme 2022-03-17 21:26:16 -05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
elee
bf745bdccb revise comment 2022-03-17 14:09:34 -05:00
elee
a0df993cef cleanup, add a few comments 2022-03-17 14:08:36 -05:00
elee
94883e2fad ok then 2022-03-17 05:23:12 -05:00
elee
f3ab6769e9 make start from indexing work 2022-03-17 05:09:13 -05:00
elee
31571dd96e fix file listing 2022-03-17 05:07:40 -05:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
6886167d5f minor optimization 2022-03-17 00:03:28 -07:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3 fix: ut test error 2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
chrislu
fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu
2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu
53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
Konstantin Lebedev
834210a9dc avoid connect to the old filler address 
https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-10 15:24:45 +05:00
zzq09494
2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu
6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
banjiaojuhao
a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu
da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu
6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00