chrislu
bfe5d910c6
use one readerCache for the whole file
2023-01-16 22:43:02 -08:00
chrislu
4614e85efa
adjust help message
2023-01-15 21:28:36 -08:00
chrislu
c24678153c
use memory chunks only for sequential writes
2023-01-15 21:10:08 -08:00
chrislu
2885ba0e50
3.40
2023-01-15 20:12:05 -08:00
chrislu
12f8251d32
avoid index out of slice range
2023-01-14 13:31:22 -08:00
chrislu
313fd17ae4
allow random writes for mem chunk
2023-01-13 20:32:18 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload ( #4130 )
...
* merge chunks during upload
* fix test
2023-01-12 23:07:36 -08:00
chrislu
2452f3b2f7
sort chunks by offset
2023-01-12 20:29:07 -08:00
chrislu
361d97941b
mysql: Missed escaping column name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
efcffe4f77
no need to set chunks during flush
2023-01-10 10:01:52 -08:00
chrislu
85f1b70207
reduce group size from 256MB to 64MB
2023-01-10 01:35:44 -08:00
chrislu
86657ed3a8
avoid repeatedly adding manifest chunks
2023-01-10 01:35:10 -08:00
chrislu
28fe578944
avoid possible nil entry
2023-01-10 01:07:34 -08:00
chrislu
47fa5ef979
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-10 00:52:50 -08:00
chrislu
9e0d526163
fix compilation
2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e
chunk group remove manifestChunks and reset sections in SetChunks()
2023-01-10 00:49:31 -08:00
chrislu
e650c8397d
ensure section.chunks is always garbage free
2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1
refactor
2023-01-10 00:46:46 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. ( #4119 )
2023-01-09 06:47:08 -08:00
chrislu
51d4a4b28d
3.39
2023-01-08 18:20:22 -08:00
chrislu
0b52f08226
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-08 18:03:25 -08:00
chrislu
469c959e6a
add missing options for webdav
2023-01-08 18:03:22 -08:00
monchickey
3e2c9ea73d
Add image cropping. ( #4117 )
2023-01-06 09:28:07 -08:00
chrislu
296fdc296c
mount: faster add chunks
2023-01-06 01:03:29 -08:00
chrislu
2abf817580
fix for stream reader
...
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586
add notes for full version
2023-01-04 09:52:25 -08:00
chrislu
5423790b2c
add back previous chunk upload selection algo
2023-01-03 22:50:39 -08:00
chrislu
7bdae5172e
batch delete EC needles
...
fix https://github.com/seaweedfs/seaweedfs/issues/4107
2023-01-03 22:05:26 -08:00
zemul
6b4c033431
add mount log ( #4101 )
...
* filer.backup use replication.source.filer
* add mount log
* Revert "filer.backup use replication.source.filer"
This reverts commit 07bf6f956c
2023-01-03 00:00:45 -08:00
chrislu
48f2edc065
3.38
2023-01-02 23:23:02 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 ( #4094 )
2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax ( #4096 )
2023-01-01 05:06:57 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query ( #4095 )
2023-01-01 05:06:41 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding ( #4092 )
2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
Guo Lei
5f993a5f20
Fix ide warning ( #4082 )
...
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
2022-12-26 00:03:47 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file ( #4060 )
...
* fs.verify needles status of file
* fix parse args
* refactor logging
* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
chrislu
77b7c9f7cc
lock and unlock chunks for read
2022-12-21 14:15:35 -08:00
chrislu
d1797deccb
remove duplicated logic of removing from writableChunks
2022-12-21 14:15:11 -08:00
chrislu
6ede19e825
add a simple file replication progress bar
2022-12-20 19:47:21 -08:00
chrislu
f7beba8515
unlock before submitting the uploading jobs
2022-12-20 12:52:30 -08:00
chrislu
28d479e5c0
mount: adjust locking for upload pipeline
2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305
filer sink retries reading file chunks, skipping missing chunks
...
if the file chunk is not available during replication time, the file is skipped
2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c
metadata skip reading un-available logs
...
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command ( #4061 )
2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993
more detailed logs
2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed ( #4057 )
2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key ( #4047 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-12-11 22:49:57 -08:00
chrislu
438146249f
3.37
2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9
rotate log files
2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c
volume server logs add url info
...
debug https://github.com/seaweedfs/seaweedfs/issues/3964
2022-12-11 13:15:38 -08:00
chrislu
38c63c429b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb
add chunk etag when downloading from remote storage
...
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting ( #4042 )
...
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming ( #4039 )
2022-12-07 07:47:40 -08:00
chrislu
659f28b855
simple refactoring
2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8
duplicated logic
2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" ( #4035 )
...
* Revert "refactor: moved to locked entry"
This reverts commit 94bc9afd9d
2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb
3.36
2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d
refactor: moved to locked entry
2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory ( #4027 )
2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer ( #4025 )
2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn ( #4024 )
2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories ( #4016 )
2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip ( #4012 )
2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata ( #4010 )
2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
...
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57
3.35
2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
...
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f
filer store: skip disabled location specific filer store initialization
...
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f
file name escape uri
...
fix https://github.com/seaweedfs/seaweedfs/issues/3969
2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part ( #3968 )
2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid ( #3963 )
2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers ( #3965 )
2022-11-10 07:09:53 -08:00
chrislu
47db75a695
3.34
2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType ( #3949 )
...
* fix filer.remote.sync to azure with ContentType
* fix pass X-Amz-Meta to X-Ms-Meta
2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration ( #3941 )
2022-11-04 00:34:40 -07:00
chrislu
4193dafce1
azure metadata: skip metadata prefixed with "X-"
...
fix https://github.com/seaweedfs/seaweedfs/issues/3875
2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete ( #3939 )
2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
...
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
...
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted ( #3932 )
2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
...
check needles status from volume server
2022-10-31 11:33:04 -07:00
chrislu
0f360862bf
3.33
2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies ( #3924 )
...
avoid vacuum if not enough replica copies
2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
...
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
8b9957d461
add back "/" prefix if it is missing in object
...
fix https://github.com/seaweedfs/seaweedfs/issues/3737
2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-28 12:53:23 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
...
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error ( #3918 )
2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048
master: delete partially created replicated volumes during volume growth
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit ( #3905 )
2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
...
* fix: set user metadata key to lowercase
* feat: simplify a bit
2022-10-24 17:29:52 -07:00
chrislu
091f999e8c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-24 16:50:41 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo ( #3893 )
...
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2022-10-24 09:25:48 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3
fix merge
2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
...
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
...
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40
skip reading needle if size is invalid
...
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d
remove volume server tcp client
2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a
fix size variable
2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
...
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0
3.32
2022-10-16 20:43:37 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9
Revert "mount: remove mount directory from generating cache directory id"
...
This reverts commit 109dc7fdf6
2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
...
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2022-10-13 23:15:00 -07:00
chrislu
e05637c42c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp ( #3855 )
...
* add acl helper functionalities
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* add tests
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* remove 0 when create map
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* delete when empty at `AssembleEntryWithAcp`
`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb
default hasSlowRead to true
2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true ( #3710 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities ( #3831 )
2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c ( #3834 )
2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata ( #3833 )
2022-10-12 07:14:49 -07:00
chrislu
e55076c46f
cloud tier: add retry when copying data file
...
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-11 21:58:19 -07:00
chrislu
0d817bc347
fix invalid memory address or nil pointer dereference on filer.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
...
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… ( #3824 )
...
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency ( #3813 )
2022-10-10 06:44:29 -07:00
chrislu
0711870f43
3.31
2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6
mount: remove mount directory from generating cache directory id
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
chrislu
f9383aa726
refactor to change capacity data type
2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
...
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options ( #3807 )
...
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not ( #3799 )
...
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452
* fix metric MasterAdminLock
2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics ( #3798 )
2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop ( #3797 )
2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
chrislu
ea271600ec
fix parameters
2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c
filer.sync: limit concurrency when fetching file chunks
...
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels ( #3785 )
2022-10-04 01:20:01 -07:00
chrislu
29632d5a34
3.30
2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis ( #3765 )
2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir ( #3770 )
2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7
better fix for option format
2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb
mount: fix extra option format
2022-09-30 08:44:41 -07:00
chrislu
8e81619d02
mount: accept all extra mount options
...
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
...
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer ( #3759 )
2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account ( #3753 )
...
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
...
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1
skip debug message
2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server ( #3708 )
...
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer ( #3750 )
...
Adjusted filer response headers
2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" ( #3747 )
...
* fs.meta.load load any dirs with prefix "important"
* replace dirPattern to dirPrefix
* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda
skip truncation on error
...
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e
3.29
2022-09-25 19:20:45 -07:00
chrislu
640c53cda8
add future list
2022-09-25 17:43:42 -07:00
chrislu
cc570a4477
assert results are sorted
2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1
ordered execution async wait
2022-09-25 13:45:55 -07:00
chrislu
973b8ff067
rename
2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b
rename file
2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f
filer replication: compare content changes directly
...
Fix https://github.com/seaweedfs/seaweedfs/issues/3714
The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.
For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-20 08:35:10 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf
adjust logs
2022-09-18 18:14:24 -07:00
chrislu
901e8157de
3.28
2022-09-18 17:46:04 -07:00
chrislu
9d230521a4
report write error
2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b
volume server: rename readBufferSize to readBufferSizeMB
2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-16 23:54:39 -07:00
chrislu
06748f2771
remove unused code
2022-09-16 23:54:37 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store ( #3707 )
2022-09-16 09:05:38 -07:00
Ryan Russell
824f7ad9e1
refactor(shell): readability improvements ( #3704 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:49 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements ( #3703 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization ( #3702 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35
mark "hasSlowRead" as experimental
2022-09-15 23:33:46 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements ( #3696 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… ( #3695 )
...
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:48 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep ( #3694 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet ( #3693 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4
volume: add "hasSlowRead" option to customize read optimization
2022-09-15 03:11:32 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var ( #3692 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
Ryan Russell
d65bdeef08
docs(meta_cache): readability fixes ( #3691 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:57 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… ( #3690 )
...
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:29 -07:00
Ryan Russell
a635973847
docs(topology): readability improvements ( #3688 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:03:10 -07:00
Ryan Russell
0fc242b084
docs: panicing -> panicking ( #3687 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:38:04 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes ( #3686 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:33:04 -07:00
Ryan Russell
150ce9f3f6
docs(wdclient): readability fix ( #3685 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:29:26 -07:00
ImFantuan
b64411bda8
fix 0 size ( #3683 )
2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default ( #3682 )
...
* simplify a bit
* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
2022-09-15 00:27:15 -07:00
chrislu
21c0587900
go fmt
2022-09-14 23:06:44 -07:00
chrislu
c8645fd232
master: implement grpc VolumeMarkWritable
...
fix https://github.com/seaweedfs/seaweedfs/issues/3657
2022-09-14 23:05:30 -07:00
Ryan Russell
c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr ( #3679 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:10:27 -07:00
Ryan Russell
12914af4d8
Character readability ( #3678 )
...
* refactor(pb): `quote_charactoer` -> `quote_character`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:09:53 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime ( #3676 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported ( #3677 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:16:39 -07:00
Ryan Russell
bd2dc6d641
refactor(shell): Decending -> Descending ( #3675 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:06:48 -07:00
Ryan Russell
dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… ( #3674 )
...
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:02:33 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive ( #3673 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements ( #3672 )
...
* refactor(net_timeout): `listner` -> `listener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(filer): `localPublicListner` -> `localPublicListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(command): `masterLocalListner` -> `masterLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(net_timeout): `ipListner` -> `ipListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:59:55 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction ( #3671 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:45 -07:00
Ryan Russell
c4363c4b5e
refactor(command_fs_rm): entiries -> entries ( #3670 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:27 -07:00
Ryan Russell
a36d7b77be
refactor(cipher_test): plantext -> plaintext ( #3669 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:09 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates ( #3668 )
...
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `SafeRenewInteval` -> `SafeRenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `InitLockInteval` -> `InitLockInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `RenewInteval` -> `RenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell
e22335ba78
refactor(queue_unbounded): inbountLen -> inboundLen ( #3666 )
2022-09-14 10:12:45 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability ( #3665 )
2022-09-14 10:11:31 -07:00
Ryan Russell
d734fff322
docs: replicte -> replicate ( #3664 )
2022-09-14 10:01:18 -07:00
Ryan Russell
dfaa602661
refactor(notification_kafka): parition -> partition ( #3663 )
2022-09-14 09:15:21 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… ( #3662 )
2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original ( #3661 )
2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer ( #3654 )
2022-09-14 06:33:00 -07:00
Ryan Russell
c9f573eed8
docs: indicates readability comment fix ( #3659 )
2022-09-14 06:31:07 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename ( #3658 )
2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix ( #3655 )
2022-09-14 04:53:04 -07:00
chrislu
58d18b68d8
rename variable
2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas ( #3591 )
...
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578
* put with in the util.Retry
* just purge on any errors
2022-09-13 22:33:35 -07:00
chrislu
0baf3d38c2
adjust logs
2022-09-13 10:33:28 -07:00
chrislu
ce55922884
fixing
2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero ( #3630 )
...
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629
* fix GrpcPort
* skip url :0
* skip empty DataCenter or Rack
* skip empty heartbeat Ip
* dell msg add DataCenter
* comment todo
* fix
2022-09-11 22:31:53 -07:00
chrislu
b834027c5a
refactor
2022-09-11 21:53:15 -07:00
chrislu
d8ca7d34fe
3.27
2022-09-11 19:47:53 -07:00
chrislu
22064c3425
mount: ensure ordered file handle lock and unlock
2022-09-11 19:44:34 -07:00
