zemul
0122e022ea
Mount concurrent read ( #4400 )
...
* fix:mount deadlock
* feat: concurrent read
* fix
* Remove useless code
* fix
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
LHHDZ
8e80ce9476
fix key corrupt when fs.configure copy path trie ( #4353 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
chrislu
5db9fcccd4
refactoring
2023-03-21 23:01:49 -07:00
chrislu
9bb0a9e306
clean comments
2023-02-17 01:31:44 -08:00
chrislu
545d5d0cc3
fix for io.ReaderAt used in filer.remote.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/4194
2023-02-17 01:30:55 -08:00
chrislu
84e9934bf9
fix filer.remote.sync on a S3 cloud mount
...
fix https://github.com/seaweedfs/seaweedfs/issues/4175
2023-02-01 20:44:00 -08:00
Neo
b9b613a78e
filter system log dir does not make subscribe event ( #4172 )
2023-01-31 18:56:11 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6
use one readerCache for the whole file
2023-01-16 22:43:02 -08:00
chrislu
12f8251d32
avoid index out of slice range
2023-01-14 13:31:22 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload ( #4130 )
...
* merge chunks during upload
* fix test
2023-01-12 23:07:36 -08:00
chrislu
361d97941b
mysql: Missed escaping column name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
85f1b70207
reduce group size from 256MB to 64MB
2023-01-10 01:35:44 -08:00
chrislu
9e0d526163
fix compilation
2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e
chunk group remove manifestChunks and reset sections in SetChunks()
2023-01-10 00:49:31 -08:00
chrislu
e650c8397d
ensure section.chunks is always garbage free
2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1
refactor
2023-01-10 00:46:46 -08:00
chrislu
296fdc296c
mount: faster add chunks
2023-01-06 01:03:29 -08:00
chrislu
2abf817580
fix for stream reader
...
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586
add notes for full version
2023-01-04 09:52:25 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 ( #4094 )
2023-01-01 05:07:53 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query ( #4095 )
2023-01-01 05:06:41 -08:00
chrislu
3fc3e7083c
metadata skip reading un-available logs
...
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
wusong
549354e324
Fix hardlink counting ( #4042 )
...
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer ( #4025 )
2022-11-30 07:43:30 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
d90aa31d5f
filer store: skip disabled location specific filer store initialization
...
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted ( #3932 )
2022-10-31 11:34:45 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
9d230521a4
report write error
2022-09-18 16:49:48 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var ( #3692 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
chrislu
21c0587900
go fmt
2022-09-14 23:06:44 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive ( #3673 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates ( #3668 )
...
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `SafeRenewInteval` -> `SafeRenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `InitLockInteval` -> `InitLockInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `RenewInteval` -> `RenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability ( #3665 )
2022-09-14 10:11:31 -07:00
gfx
48db56ddad
arangodb s3 bucket name compatibility ( #3588 )
...
* Update arangodb_store.go
* update readme, properly escape queries, add name patching
* use underscore
* use underscore
* better comment
* fix readme
Co-authored-by: a <a@a.a>
2022-09-09 09:43:42 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting ( #3577 )
2022-09-06 08:19:13 -07:00
famosss
131d389fc4
adjust log level ( #3589 )
2022-09-04 22:37:32 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket ( #3587 )
...
* Update helpers.go
* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context ( #3586 )
...
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu
a112cbe44c
filer: rocksdb store report not found correctly
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu
10efdc7aab
align memory
2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch ( #3566 )
...
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu
d81db3c703
s3: fix configuring IAM for the same user
...
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]
s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
chrislu
57a46f46a0
nano level precision
2022-08-26 16:55:15 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount ( #3527 )
2022-08-26 16:41:37 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate ( #3525 )
...
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks ( #3505 )
2022-08-24 20:03:34 -07:00
chrislu
fef9c6a520
filer: do not always save files in "/etc" folder to filer store
...
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher ( #3473 )
...
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu
388f82f322
minor
2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e
refactoring
2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps ( #3472 )
...
* Fix FUSE server buffer leaks in file gaps
This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.
* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu
689b4ecdcc
remove unused collection and replication from upload result
2022-08-20 18:14:57 -07:00
Guo Lei
3172c33fa6
fix redundant type from array ( #3462 )
2022-08-18 02:25:25 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
96caf21d09
less verbose log
2022-08-15 10:03:52 -07:00
chrislu
0aeec04c31
quicker to adapt to pattern change
2022-08-07 10:14:01 -07:00
chrislu
928d29af9e
fix wrong logic about reader isRandomMode()
2022-08-07 10:13:04 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection ( #3413 )
2022-08-07 00:58:53 -07:00
LHHDZ
84ec68e11a
Add download speed limit support ( #3408 )
2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083
...
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 08:23:10 -07:00
Konstantin Lebedev
a98f6d66a3
rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 12:51:41 +05:00
chrislu
036566629a
filer.sync: fix synchronization logic in active-active mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3328
2022-08-01 00:06:18 -07:00
chrislu
303bd067b5
Revert "rename: delete source entry metadata only, skipping hard links"
...
This reverts commit 03466f955ehttps://github.com/seaweedfs/seaweedfs/issues/3386
2022-07-31 22:51:41 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
chrislu
74f60f246f
dynamically connect to a filer
2022-07-28 23:24:38 -07:00
chrislu
68065128b8
add dc and rack
2022-07-28 23:22:51 -07:00
zzq09494
01b7aa79fa
fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted
2022-07-28 12:04:37 +08:00
Konstantin Lebedev
da9d3e8f6c
refactor
2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4
fix logic else brake
2022-07-26 11:47:11 +05:00
chrislu
ec0edb1ac4
filer: fix wrong logic during read
2022-07-25 22:40:00 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
fc8241fb5e
leveldb3: add instant dropping bucket
2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44
explicit bucket aware declaration
2022-07-21 18:23:53 -07:00
Konstantin Lebedev
b6471ecd75
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
11e393dbe7
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev
7b1497ee63
Use BackoffSchedule for getLookupFileId
2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8
Use fallback if urls are not found
2022-07-15 15:29:15 +05:00
chrislu
1db012485f
mount: fix bug during busy writes
...
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4
more logging related to filer metadata subscription
2022-07-15 00:42:13 -07:00
chrislu
fa61074513
add clientId logging
2022-07-14 12:27:34 -07:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak
2022-07-13 09:23:37 -07:00
guosj
354b7bdff0
replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()
2022-07-13 22:49:03 +08:00
guosj
ab1b9697e6
supplement check duplicate accesskey
2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0
detect sequential mode with a counter
2022-07-13 02:20:03 -07:00
chrislu
1451b389a4
Avoid fatal error: sync: Unlock of unlocked RWMutex
...
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
chrislu
ebd6097e6f
fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074
...
related to https://github.com/chrislusf/seaweedfs/pull/3074
2022-06-07 22:33:35 -07:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
chrislu
6adc42147f
fresh filer store bootstrap from the oldest peer
2022-05-30 21:27:48 -07:00
chrislu
c59068d0f3
refactor
2022-05-30 16:28:36 -07:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
aece35a64f
stop when on disk log is done
2022-05-30 15:20:51 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
9219664505
reset sync offset if peer filer resets
...
fix https://github.com/chrislusf/seaweedfs/issues/3089
2022-05-29 13:11:24 -07:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator.
2022-05-25 01:11:02 -07:00
chrislu
a5cdcc7e63
fix reading files larger than 20GB due to manifest resolving bug
2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd
fix reading files larger than 20GB due to manifest resolving bug
...
this is introduced since Mar 08 2022
2022-05-23 01:14:56 -07:00
Konstantin Lebedev
d12a423aa0
refactor
...
add some tests
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9
fixes from comments
...
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-11 21:50:09 +05:00
chrislu
76da688176
adjust log level
2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6
remove deprecated function
2022-05-10 00:02:23 -07:00
chrislu
791c29ae93
remove deprecated function
...
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
2022-05-10 00:01:43 -07:00
chrislu
10207a48a4
how to setup local test env
2022-05-06 02:25:56 -07:00
chrislu
db33b03735
add comments
2022-05-06 02:19:47 -07:00
chrislu
e350ba8896
larger test range
2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5
a bit optimization
2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7
filer: etcd fix pagination
...
fix https://github.com/chrislusf/seaweedfs/issues/3016
2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911
ydb add empty doc file
2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25
build tag ydb since the growth of the binary is 5 mb
2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf
add options to scaffold
2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af
refactor
2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb
add support native ttl
2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb
enable query cache policy instead of prepare
2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3
fix scanNamed
2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83
Merge branch 'new_master' into ydb
...
# Conflicts:
# go.mod
# go.sum
2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5
fix queries
2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48
fix createDB
2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63
ydb do Tx or DB
2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3
ydb BucketAware interface
2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4
ydb-go-sdk move to v3
2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3
refactor use const CountEntryChunksForGzip
2022-05-01 22:28:55 +05:00
