mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-24 17:47:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,154 Commits 33 Branches 296 Tags 153 MiB
11d055bfc1
Commit Graph

6007 Commits

Author SHA1 Message Date
chrislu
438146249f 3.37 2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9 rotate log files 2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c volume server logs add url info 
debug https://github.com/seaweedfs/seaweedfs/issues/3964
 2022-12-11 13:15:38 -08:00
chrislu
38c63c429b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb add chunk etag when downloading from remote storage 
fix https://github.com/seaweedfs/seaweedfs/issues/3987
 2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting (#4042) 
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
 2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming (#4039) 2022-12-07 07:47:40 -08:00
chrislu
659f28b855 simple refactoring 2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8 duplicated logic 2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" (#4035) 
* Revert "refactor: moved to locked entry"

This reverts commit 94bc9afd9d.

* only add LockedEntry, no changes to entryLock

* fix compilation
 2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb 3.36 2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d refactor: moved to locked entry 2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory (#4027) 2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories (#4016) 2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip (#4012) 2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata (#4010) 2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles (#4005) 2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo (#3993) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles (#3991) 
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.

Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
 2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57 3.35 2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading (#3958) 
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f filer store: skip disabled location specific filer store initialization 
fix https://github.com/seaweedfs/seaweedfs/issues/3971
 2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f file name escape uri 
fix https://github.com/seaweedfs/seaweedfs/issues/3969
 2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part (#3968) 2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid (#3963) 2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers (#3965) 2022-11-10 07:09:53 -08:00
chrislu
47db75a695 3.34 2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType (#3949) 
* fix filer.remote.sync to azure with ContentType

* fix pass X-Amz-Meta to X-Ms-Meta
 2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration (#3941) 2022-11-04 00:34:40 -07:00
chrislu
4193dafce1 azure metadata: skip metadata prefixed with "X-" 
fix https://github.com/seaweedfs/seaweedfs/issues/3875
 2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete (#3939) 2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
chrislu
0f360862bf 3.33 2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
chrislu
8b9957d461 add back "/" prefix if it is missing in object 
fix https://github.com/seaweedfs/seaweedfs/issues/3737
 2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-28 12:53:23 -07:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048 master: delete partially created replicated volumes during volume growth 
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
 2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic (#3907) 2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit (#3905) 2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase (#3894) 
* fix: set user metadata key to lowercase

* feat: simplify a bit
 2022-10-24 17:29:52 -07:00
chrislu
091f999e8c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-24 16:50:41 -07:00
chrislu
377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo (#3893) 
* types packages is imported more than onece

* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
 2022-10-24 09:25:48 -07:00
chrislu
d5364218b2 adjust help message 2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3 fix merge 2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40 skip reading needle if size is invalid 
related to https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d remove volume server tcp client 2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a fix size variable 2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50 volume server: remote tier volumes only soft delete in local index 
fix https://github.com/seaweedfs/seaweedfs/issues/3889
 2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite (#3881) 2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0 3.32 2022-10-16 20:43:37 -07:00
chrislu
de286fe662 shell: volume.move handles volume moved to cloud tier 
fix https://github.com/seaweedfs/seaweedfs/issues/3803
 2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15 shell: volume.tier.upload clean up replicated copies 
fix https://github.com/seaweedfs/seaweedfs/issues/3804
 2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping (#3859) 2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict (#3858) 
fix conflict

https://github.com/seaweedfs/seaweedfs/pull/3841
 2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9 Revert "mount: remove mount directory from generating cache directory id" 
This reverts commit 109dc7fdf6.
 2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9 ensure monotonic n.AppendAtNs 
fix https://github.com/seaweedfs/seaweedfs/issues/3852
 2022-10-13 23:15:00 -07:00
chrislu
e05637c42c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35 remove unused ReadNeedleBlobRequest.needle_id 
fix https://github.com/seaweedfs/seaweedfs/issues/3853
 2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp (#3855) 
* add acl helper functionalities

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* add tests

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* remove 0 when create map

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* delete when empty at `AssembleEntryWithAcp`

`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp`  shouldn't skip empty value

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece (#3838) 2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb default hasSlowRead to true 2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true (#3710) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit

* volume server:set the default value of "hasSlowRead" to true
 2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities (#3831) 2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata (#3833) 2022-10-12 07:14:49 -07:00
chrislu
e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
chrislu
cea73ac008 serialize sqlite operations 
fix https://github.com/seaweedfs/seaweedfs/issues/3827
 2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-11 21:58:19 -07:00
chrislu
0d817bc347 fix invalid memory address or nil pointer dereference on filer.sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3826
 2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824) 
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency (#3813) 2022-10-10 06:44:29 -07:00
chrislu
0711870f43 3.31 2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6 mount: remove mount directory from generating cache directory id 
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
 2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9 tests add parsing ec shard info 2022-10-09 22:47:55 -07:00
chrislu
0623bf582e include ec shard for capacityByFreeVolumeCount 2022-10-09 22:31:30 -07:00