Commit Graph

714 Commits

Author SHA1 Message Date
chrislu
61c42f9991 adjust lock APIs 2023-06-25 20:30:20 -07:00
chrislu
06471dac9d init lock ring 2023-06-25 15:28:16 -07:00
chrislu
868f7875d7 refactor 2023-06-25 14:30:58 -07:00
chrislu
3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
chrislu
3fbf4f6189 Revert "Initial patch for redis version 7 support (#4572)"
This reverts commit a25bca0692.
2023-06-12 22:27:38 -07:00
Aaron Gipson
a25bca0692
Initial patch for redis version 7 support (#4572) 2023-06-12 13:30:03 -07:00
wusong
d6af3a8eb6
fix 404 not found (#4536)
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 09:14:49 -07:00
Mesar Hameed
a90b777ff4
Feat: etcd filer store keys should have customizable prefix (#4484)
An etcd cluster is not necessarily only dedicated to seaweedfs.
This security enhancement adds a customizable key_prefix option to the etcd filer store.
This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the
key_prefix, instead of all keys on the etcd cluster.
2023-05-18 23:08:56 -07:00
chrislu
b7f011f777 fix test 2023-05-15 17:46:56 -07:00
Mesar Hameed
65484e80ef
Feat: support username/password authentication for etcd filer store s… (#4477)
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262

Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
2023-05-15 17:45:14 -07:00
wusong
4867aa03ab
Fix: http rename move dir to subdir (#4432)
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-04-26 20:54:36 -07:00
zemul
0122e022ea
Mount concurrent read (#4400)
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
LHHDZ
8e80ce9476
fix key corrupt when fs.configure copy path trie (#4353)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
chrislu
5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
chrislu
9bb0a9e306 clean comments 2023-02-17 01:31:44 -08:00
chrislu
545d5d0cc3 fix for io.ReaderAt used in filer.remote.sync
fix https://github.com/seaweedfs/seaweedfs/issues/4194
2023-02-17 01:30:55 -08:00
chrislu
84e9934bf9 fix filer.remote.sync on a S3 cloud mount
fix https://github.com/seaweedfs/seaweedfs/issues/4175
2023-02-01 20:44:00 -08:00
Neo
b9b613a78e
filter system log dir does not make subscribe event (#4172) 2023-01-31 18:56:11 -08:00
chrislu
81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu
12f8251d32 avoid index out of slice range 2023-01-14 13:31:22 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload (#4130)
* merge chunks during upload

* fix test
2023-01-12 23:07:36 -08:00
chrislu
361d97941b mysql: Missed escaping column name
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
85f1b70207 reduce group size from 256MB to 64MB 2023-01-10 01:35:44 -08:00
chrislu
9e0d526163 fix compilation 2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e chunk group remove manifestChunks and reset sections in SetChunks() 2023-01-10 00:49:31 -08:00
chrislu
e650c8397d ensure section.chunks is always garbage free 2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1 refactor 2023-01-10 00:46:46 -08:00
chrislu
296fdc296c mount: faster add chunks 2023-01-06 01:03:29 -08:00
chrislu
2abf817580 fix for stream reader
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586 add notes for full version 2023-01-04 09:52:25 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query (#4095) 2023-01-01 05:06:41 -08:00
chrislu
3fc3e7083c metadata skip reading un-available logs
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
wusong
549354e324
Fix hardlink counting (#4042)
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
d90aa31d5f filer store: skip disabled location specific filer store initialization
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
chrislu
f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
chrislu
cea73ac008 serialize sqlite operations
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
chrislu
b2a171632b mount: fix 0 file size in linux 2022-09-18 21:42:13 -07:00
chrislu
9d230521a4 report write error 2022-09-18 16:49:48 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var (#3692)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00