mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 17:07:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,697 Commits 32 Branches 296 Tags 152 MiB
440ebd7b6d
Commit Graph

6265 Commits

Author SHA1 Message Date
chrislu
28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305 filer sink retries reading file chunks, skipping missing chunks 
if the file chunk is not available during replication time, the file is skipped
 2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c metadata skip reading un-available logs 
this means the volumes for metadata histories have been lost.
 2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993 more detailed logs 2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed (#4057) 2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key (#4047) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-12-11 22:49:57 -08:00
chrislu
438146249f 3.37 2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9 rotate log files 2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c volume server logs add url info 
debug https://github.com/seaweedfs/seaweedfs/issues/3964
 2022-12-11 13:15:38 -08:00
chrislu
38c63c429b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb add chunk etag when downloading from remote storage 
fix https://github.com/seaweedfs/seaweedfs/issues/3987
 2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting (#4042) 
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
 2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming (#4039) 2022-12-07 07:47:40 -08:00
chrislu
659f28b855 simple refactoring 2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8 duplicated logic 2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" (#4035) 
* Revert "refactor: moved to locked entry"

This reverts commit 94bc9afd9d.

* only add LockedEntry, no changes to entryLock

* fix compilation
 2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb 3.36 2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d refactor: moved to locked entry 2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory (#4027) 2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories (#4016) 2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip (#4012) 2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata (#4010) 2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles (#4005) 2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo (#3993) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles (#3991) 
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.

Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
 2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57 3.35 2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading (#3958) 
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f filer store: skip disabled location specific filer store initialization 
fix https://github.com/seaweedfs/seaweedfs/issues/3971
 2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f file name escape uri 
fix https://github.com/seaweedfs/seaweedfs/issues/3969
 2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part (#3968) 2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid (#3963) 2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers (#3965) 2022-11-10 07:09:53 -08:00
chrislu
47db75a695 3.34 2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType (#3949) 
* fix filer.remote.sync to azure with ContentType

* fix pass X-Amz-Meta to X-Ms-Meta
 2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration (#3941) 2022-11-04 00:34:40 -07:00
chrislu
4193dafce1 azure metadata: skip metadata prefixed with "X-" 
fix https://github.com/seaweedfs/seaweedfs/issues/3875
 2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete (#3939) 2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
chrislu
0f360862bf 3.33 2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
chrislu
8b9957d461 add back "/" prefix if it is missing in object 
fix https://github.com/seaweedfs/seaweedfs/issues/3737
 2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-28 12:53:23 -07:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048 master: delete partially created replicated volumes during volume growth 
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
 2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic (#3907) 2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit (#3905) 2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase (#3894) 
* fix: set user metadata key to lowercase

* feat: simplify a bit
 2022-10-24 17:29:52 -07:00
chrislu
091f999e8c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-24 16:50:41 -07:00
chrislu
377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo (#3893) 
* types packages is imported more than onece

* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
 2022-10-24 09:25:48 -07:00
chrislu
d5364218b2 adjust help message 2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3 fix merge 2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40 skip reading needle if size is invalid 
related to https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d remove volume server tcp client 2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a fix size variable 2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50 volume server: remote tier volumes only soft delete in local index 
fix https://github.com/seaweedfs/seaweedfs/issues/3889
 2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite (#3881) 2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0 3.32 2022-10-16 20:43:37 -07:00
chrislu
de286fe662 shell: volume.move handles volume moved to cloud tier 
fix https://github.com/seaweedfs/seaweedfs/issues/3803
 2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15 shell: volume.tier.upload clean up replicated copies 
fix https://github.com/seaweedfs/seaweedfs/issues/3804
 2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping (#3859) 2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict (#3858) 
fix conflict

https://github.com/seaweedfs/seaweedfs/pull/3841
 2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9 Revert "mount: remove mount directory from generating cache directory id" 
This reverts commit 109dc7fdf6.
 2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9 ensure monotonic n.AppendAtNs 
fix https://github.com/seaweedfs/seaweedfs/issues/3852
 2022-10-13 23:15:00 -07:00
chrislu
e05637c42c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35 remove unused ReadNeedleBlobRequest.needle_id 
fix https://github.com/seaweedfs/seaweedfs/issues/3853
 2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp (#3855) 
* add acl helper functionalities

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* add tests

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* remove 0 when create map

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* delete when empty at `AssembleEntryWithAcp`

`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp`  shouldn't skip empty value

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece (#3838) 2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb default hasSlowRead to true 2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true (#3710) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit

* volume server:set the default value of "hasSlowRead" to true
 2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities (#3831) 2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata (#3833) 2022-10-12 07:14:49 -07:00
chrislu
e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
chrislu
cea73ac008 serialize sqlite operations 
fix https://github.com/seaweedfs/seaweedfs/issues/3827
 2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-11 21:58:19 -07:00
chrislu
0d817bc347 fix invalid memory address or nil pointer dereference on filer.sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3826
 2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824) 
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency (#3813) 2022-10-10 06:44:29 -07:00
chrislu
0711870f43 3.31 2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6 mount: remove mount directory from generating cache directory id 
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
 2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9 tests add parsing ec shard info 2022-10-09 22:47:55 -07:00
chrislu
0623bf582e include ec shard for capacityByFreeVolumeCount 2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15 move volume: find target volume server by exiting/max ratio 2022-10-09 22:30:59 -07:00
chrislu
049f040c3c refactor 2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7 parsing ec volumes 2022-10-09 19:21:23 -07:00
chrislu
1e27d36576 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-09 18:58:14 -07:00
chrislu
f9383aa726 refactor to change capacity data type 2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie (#3748) 
* fix: compact_map get error

* fix: CompactSection delete lock and move test to compact_map

Co-authored-by: shibinbin <shibinbin@megvii.com>
 2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options (#3807) 
fuse: Ignore "-o" command
 2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not (#3799) 
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452

* fix metric MasterAdminLock
 2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. (#3802) 2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics (#3798) 2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
chrislu
ea271600ec fix parameters 2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c filer.sync: limit concurrency when fetching file chunks 
fix https://github.com/seaweedfs/seaweedfs/issues/3787
 2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels (#3785) 2022-10-04 01:20:01 -07:00
chrislu
29632d5a34 3.30 2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis (#3765) 2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7 better fix for option format 2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb mount: fix extra option format 2022-09-30 08:44:41 -07:00
chrislu
8e81619d02 mount: accept all extra mount options 
fix https://github.com/seaweedfs/seaweedfs/issues/3767
 2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account (#3753) 
associate `Account` and `Identity` by accountId
 2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1 skip debug message 2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server (#3708) 
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer (#3750) 
Adjusted filer response headers
 2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" (#3747) 
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
 2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda skip truncation on error 
fix https://github.com/seaweedfs/seaweedfs/issues/3746
 2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e 3.29 2022-09-25 19:20:45 -07:00
chrislu
640c53cda8 add future list 2022-09-25 17:43:42 -07:00
chrislu
cc570a4477 assert results are sorted 2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1 ordered execution async wait 2022-09-25 13:45:55 -07:00
chrislu
973b8ff067 rename 2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b rename file 2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2 s3 to watch specific directories 2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9 grpc: watch metadata changes by directory 2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f filer replication: compare content changes directly 
Fix https://github.com/seaweedfs/seaweedfs/issues/3714

The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.

For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
 2022-09-20 08:35:10 -07:00
chrislu
b2a171632b mount: fix 0 file size in linux 2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf adjust logs 2022-09-18 18:14:24 -07:00
chrislu
901e8157de 3.28 2022-09-18 17:46:04 -07:00
chrislu
9d230521a4 report write error 2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b volume server: rename readBufferSize to readBufferSizeMB 2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-16 23:54:39 -07:00
chrislu
06748f2771 remove unused code 2022-09-16 23:54:37 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store (#3707) 2022-09-16 09:05:38 -07:00
Ryan Russell
824f7ad9e1
refactor(shell): readability improvements (#3704) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:49 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements (#3703) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35 mark "hasSlowRead" as experimental 2022-09-15 23:33:46 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements (#3696) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… (#3695) 
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:48 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep (#3694) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet (#3693) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var (#3692) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:09:06 -07:00
Ryan Russell
d65bdeef08
docs(meta_cache): readability fixes (#3691) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:04:57 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… (#3690) 
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:04:29 -07:00
Ryan Russell
a635973847
docs(topology): readability improvements (#3688) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:03:10 -07:00
Ryan Russell
0fc242b084
docs: panicing -> panicking (#3687) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:38:04 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes (#3686) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:33:04 -07:00
Ryan Russell
150ce9f3f6
docs(wdclient): readability fix (#3685) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:29:26 -07:00
ImFantuan
b64411bda8
fix 0 size (#3683) 2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default (#3682) 
* simplify a bit

* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
 2022-09-15 00:27:15 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu
c8645fd232 master: implement grpc VolumeMarkWritable 
fix https://github.com/seaweedfs/seaweedfs/issues/3657
 2022-09-14 23:05:30 -07:00
Ryan Russell
c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr (#3679) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 13:10:27 -07:00
Ryan Russell
12914af4d8
Character readability (#3678) 
* refactor(pb): `quote_charactoer` -> `quote_character`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 13:09:53 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime (#3676) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported (#3677) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:16:39 -07:00
Ryan Russell
bd2dc6d641
refactor(shell): Decending -> Descending (#3675) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:06:48 -07:00
Ryan Russell
dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… (#3674) 
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:02:33 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive (#3673) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:00:16 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements (#3672) 
* refactor(net_timeout): `listner` -> `listener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(filer): `localPublicListner` -> `localPublicListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(command): `masterLocalListner` -> `masterLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(net_timeout): `ipListner` -> `ipListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:59:55 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction (#3671) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:45 -07:00
Ryan Russell
c4363c4b5e
refactor(command_fs_rm): entiries -> entries (#3670) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:27 -07:00
Ryan Russell
a36d7b77be
refactor(cipher_test): plantext -> plaintext (#3669) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:09 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates (#3668) 
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `SafeRenewInteval` -> `SafeRenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `InitLockInteval` -> `InitLockInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `RenewInteval` -> `RenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 10:29:55 -07:00
Ryan Russell
e22335ba78
refactor(queue_unbounded): inbountLen -> inboundLen (#3666) 2022-09-14 10:12:45 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
Ryan Russell
d734fff322
docs: replicte -> replicate (#3664) 2022-09-14 10:01:18 -07:00
Ryan Russell
dfaa602661
refactor(notification_kafka): parition -> partition (#3663) 2022-09-14 09:15:21 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662) 2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original (#3661) 2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer (#3654) 2022-09-14 06:33:00 -07:00
Ryan Russell
c9f573eed8
docs: indicates readability comment fix (#3659) 2022-09-14 06:31:07 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename (#3658) 2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix (#3655) 2022-09-14 04:53:04 -07:00
chrislu
58d18b68d8 rename variable 2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas (#3591) 
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
 2022-09-13 22:33:35 -07:00
chrislu
0baf3d38c2 adjust logs 2022-09-13 10:33:28 -07:00
chrislu
ce55922884 fixing 2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero (#3630) 
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629

* fix GrpcPort

* skip url :0

* skip empty DataCenter or Rack

* skip empty heartbeat Ip

* dell msg add DataCenter

* comment todo

* fix
 2022-09-11 22:31:53 -07:00
chrislu
b834027c5a refactor 2022-09-11 21:53:15 -07:00
chrislu
d8ca7d34fe 3.27 2022-09-11 19:47:53 -07:00
chrislu
22064c3425 mount: ensure ordered file handle lock and unlock 2022-09-11 19:44:34 -07:00
chrislu
b9112747b5 volume server: synchronously report volume readonly status to master 
fix https://github.com/seaweedfs/seaweedfs/issues/3628
 2022-09-11 19:29:10 -07:00
chrislu
6690236754 rename variables 2022-09-11 19:18:42 -07:00
chrislu
4af8c427eb refactor 2022-09-10 16:56:03 -07:00
chrislu
4957d8eec6 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 15:36:02 -07:00
chrislu
b1501f61dd Revert "a new test case" 
This reverts commit eb7cf3de81.
 2022-09-10 15:35:42 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
chrislu
2c6b68b40e simplify a bit 2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a
[sync] override amz storage class, None to delete (#3639) 
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
 2022-09-10 14:15:42 -07:00
chrislu
3cb914f7e1 avoid dead lock 2022-09-10 11:26:19 -07:00
chrislu
ebef579ebb Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 11:22:18 -07:00
chrislu
576c113c59 replace PR https://github.com/seaweedfs/seaweedfs/pull/3621 
replace https://github.com/seaweedfs/seaweedfs/pull/3621
 2022-09-10 11:22:16 -07:00
Brian
4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640) 2022-09-10 08:05:28 -07:00
chrislu
205ecb5d03 skip directories under ".uploads" directory 
related to https://github.com/seaweedfs/seaweedfs/issues/3636

skipping all directories under ".uploads" directory.
 2022-09-09 22:57:38 -07:00
chrislu
fc4208d128 volume.balance: default to balance ALL_COLLECTIONS 2022-09-09 16:06:05 -07:00
gfx
48db56ddad
arangodb s3 bucket name compatibility (#3588) 
* Update arangodb_store.go

* update readme, properly escape queries, add name patching

* use underscore

* use underscore

* better comment

* fix readme

Co-authored-by: a <a@a.a>
 2022-09-09 09:43:42 -07:00
VictorSFF
10d545060f
mount: fix new inode allocation (#3633) 2022-09-09 06:32:19 -07:00
famosss
41af5a6ee5
simplify a bit (#3631) 2022-09-08 21:01:21 -07:00
chrislu
64e75a286e simplify a bit 2022-09-08 19:07:57 -07:00
chrislu
d14d029c73 reduce v.dataFileAccessLock lock scope 2022-09-08 19:05:31 -07:00
chrislu
406a80da4b adjust variable names 2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
chrislu
eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu
4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu
d653c5f811 unused 2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43 rename 2022-09-08 12:06:03 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
chrislu
7de112943b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195 return err for processRangeRequest() 2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown (#3618) 2022-09-07 18:57:13 -07:00
Konstantin Lebedev
695e3a909c
avoid data race on currentDiskUsage.maxVolumeCount (#3592) 
https://github.com/seaweedfs/seaweedfs/issues/3590
 2022-09-07 09:51:15 -07:00
Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount (#3594) 
https://github.com/seaweedfs/seaweedfs/issues/3593
 2022-09-07 09:50:17 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error (#3616) 2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown (#3613) 
* fix: volume Deadlock when exception is thrown

* adjust log level
 2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
chrislu
c07ab9c060 3.26 2022-09-06 08:26:20 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket (#3609) 
https://github.com/seaweedfs/seaweedfs/issues/3607
 2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level (#3610) 2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap (#3606) 2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic (#3596) 2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e 3.25 2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level (#3589) 2022-09-04 22:37:32 -07:00
chrislu
f4abefc109 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 22:23:31 -07:00
chrislu
39340f7e42 cloud tier: s3 consume all read response body 
fix https://github.com/seaweedfs/seaweedfs/issues/3584
 2022-09-04 22:23:28 -07:00
chrislu
7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket (#3587) 
* Update helpers.go

* Update arangodb_store_bucket.go
 2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context (#3586) 
Update arangodb_store.go
 2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d minor optimization 2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca filer.backup and filer.sync: include headers during backup and sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3532
 2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level (#3585) 2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move (#3583) 2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync (#3579) 
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
 2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c filer: rocksdb store report not found correctly 
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
 2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e Revert "avoid data race on grace.hooks (#3572)" 
This reverts commit c37d6fc01a.
 2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks (#3572) 
https://github.com/seaweedfs/seaweedfs/issues/3564
 2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab align memory 2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566) 
https://github.com/seaweedfs/seaweedfs/issues/3565
 2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1 add more logs 2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541) 
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0

Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update code

* more code fix

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount (#3562) 
* Add an e2e workflow to test FUSE mount

* Fix deadlocks during concurrent r/w
 2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c upload_content: upload close response.Body 2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced refactor 2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c just a bit safer 2022-08-31 00:10:17 -07:00
chrislu
9c944377dc minor 2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6 s3: close response body with copying object 2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae print process before do the work 2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
chrislu
de5263364b master client: use GetMaster() for latest master address 2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a more detailed error 2022-08-30 09:43:17 -07:00
chrislu
d81db3c703 s3: fix configuring IAM for the same user 
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]

s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
 2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0 rename variable 2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster (#3538) 
https://github.com/seaweedfs/seaweedfs/issues/3510
 2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00
chrislu
570b488572 ensure memory alignment on different CPUs 2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount (#3537) 
https://github.com/seaweedfs/seaweedfs/issues/3511
 2022-08-29 13:23:02 -07:00
Patrick Schmidt
00041f6e08
Fix corrupted mount reads after changes on fh (#3536) 2022-08-29 08:27:13 -07:00
chrislu
b7a887fea1 3.24 2022-08-28 21:53:13 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug (#3534) 2022-08-28 07:51:47 -07:00
chrislu
cc0c8c5f81 simplify 2022-08-27 00:21:57 -07:00
chrislu
3741779144 minor 2022-08-27 00:20:58 -07:00
chrislu
580a7fa685 adjust for data race 
fix https://github.com/seaweedfs/seaweedfs/issues/3512
 2022-08-27 00:19:13 -07:00
chrislu
87b70a6809 clean up 2022-08-27 00:09:04 -07:00
chrislu
27c7611ebf Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-26 23:47:39 -07:00
chrislu
9920d65bc0 gateway to remote object store: adjust upload concurrency 2022-08-26 23:47:37 -07:00
chrislu
c839ce1b19 s3 sink use s3 upload manager 
fix https://github.com/seaweedfs/seaweedfs/issues/3531
 2022-08-26 23:47:12 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children (#3530) 2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu
301b49b63f atomic operation 2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0 nano level precision 2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c more detailed error 
related to https://github.com/seaweedfs/seaweedfs/issues/3528
 2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8 rename varaible 2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
chrislu
6f483a4d36 atomic add 
fix https://github.com/seaweedfs/seaweedfs/issues/3514
 2022-08-25 22:20:34 -07:00
chrislu
7394f7feee minor: adjust ordering 2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1 minor clean up 2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact (#3388) 
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
 2022-08-23 23:53:35 -07:00
chrislu
10414fd81c ping timeout at 15 seconds 
this 72 minute timeout setting seems unreasonably long

15 seconds is around the time when a new raft leader should be elected.
 2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug (#3499) 2022-08-23 08:53:24 -07:00
chrislu
21a933c932 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-23 01:52:32 -07:00
chrislu
f0b4a7659a fix test 2022-08-23 01:52:29 -07:00
Patrick Schmidt
cda8cc22bc
Implement lseek syscall in FUSE (#3491) 
See the man page of lseek:
https://man7.org/linux/man-pages/man2/lseek.2.html
 2022-08-23 01:47:18 -07:00
chrislu
bf88006037 format 2022-08-23 01:20:45 -07:00
chrislu
9fce75607d s3: report http.StatusOK if the directory is explicitly created 
fix https://github.com/seaweedfs/seaweedfs/issues/3457
 2022-08-23 01:16:46 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497) 2022-08-22 23:54:51 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests (#3496) 2022-08-22 23:39:34 -07:00
chrislu
676e27c589 shell: stop long running jobs if lock is lost 2022-08-22 14:12:23 -07:00
chrislu
601ba5fb68 master: disable locking if not leader 2022-08-22 14:11:48 -07:00
chrislu
57e7582c36 refactoring 2022-08-22 14:11:13 -07:00
chrislu
7ea2c24c5d metadata processing use retry instead of retryForEver 2022-08-22 11:25:40 -07:00
Patrick Schmidt
f875031f06
Reuse readDataByFileHandle in Read call (#3482) 2022-08-22 08:24:06 -07:00
famosss
bdba3da2e4
Handle preflight cors requests (#3481) 2022-08-22 08:21:38 -07:00
chrislu
f4b52d4c52 fix format 2022-08-22 00:20:21 -07:00
chrislu
b4131a3a72 filer.remote.sync: parallelize uploading to remove object storage 2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561 refactor 2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-21 23:26:09 -07:00
chrislu
8bac903d29 mount: add locking to prevent nil pointer 
fix https://github.com/seaweedfs/seaweedfs/issues/3479
 2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu
7807f6641a minor 2022-08-21 22:54:56 -07:00
chrislu
c4e862e908 3.23 2022-08-21 19:13:26 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same (#3477) 
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc)
 2022-08-21 19:11:18 -07:00
chrislu
63fbf281c7 remove same file copying rage limitation 2022-08-21 17:16:21 -07:00
Patrick Schmidt
3a75d7f7aa
Implement copy_file_range syscall in FUSE (#3475) 
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
 2022-08-21 17:11:43 -07:00
chrislu
fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
chrislu
2762154130 fix compilation 2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3 EC: with multiple volume locations, the ec rebuilding may fail 2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations 
related to https://github.com/seaweedfs/seaweedfs/issues/3459
 2022-08-21 14:52:37 -07:00