Guo Lei
b63d1e9c6b
fix index ( #4158 )
2023-01-28 06:55:27 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6
use one readerCache for the whole file
2023-01-16 22:43:02 -08:00
chrislu
4614e85efa
adjust help message
2023-01-15 21:28:36 -08:00
chrislu
c24678153c
use memory chunks only for sequential writes
2023-01-15 21:10:08 -08:00
chrislu
2885ba0e50
3.40
2023-01-15 20:12:05 -08:00
chrislu
12f8251d32
avoid index out of slice range
2023-01-14 13:31:22 -08:00
chrislu
313fd17ae4
allow random writes for mem chunk
2023-01-13 20:32:18 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload ( #4130 )
...
* merge chunks during upload
* fix test
2023-01-12 23:07:36 -08:00
chrislu
2452f3b2f7
sort chunks by offset
2023-01-12 20:29:07 -08:00
chrislu
361d97941b
mysql: Missed escaping column name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
efcffe4f77
no need to set chunks during flush
2023-01-10 10:01:52 -08:00
chrislu
85f1b70207
reduce group size from 256MB to 64MB
2023-01-10 01:35:44 -08:00
chrislu
86657ed3a8
avoid repeatedly adding manifest chunks
2023-01-10 01:35:10 -08:00
chrislu
28fe578944
avoid possible nil entry
2023-01-10 01:07:34 -08:00
chrislu
47fa5ef979
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-10 00:52:50 -08:00
chrislu
9e0d526163
fix compilation
2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e
chunk group remove manifestChunks and reset sections in SetChunks()
2023-01-10 00:49:31 -08:00
chrislu
e650c8397d
ensure section.chunks is always garbage free
2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1
refactor
2023-01-10 00:46:46 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. ( #4119 )
2023-01-09 06:47:08 -08:00
chrislu
51d4a4b28d
3.39
2023-01-08 18:20:22 -08:00
chrislu
0b52f08226
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-08 18:03:25 -08:00
chrislu
469c959e6a
add missing options for webdav
2023-01-08 18:03:22 -08:00
monchickey
3e2c9ea73d
Add image cropping. ( #4117 )
2023-01-06 09:28:07 -08:00
chrislu
296fdc296c
mount: faster add chunks
2023-01-06 01:03:29 -08:00
chrislu
2abf817580
fix for stream reader
...
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586
add notes for full version
2023-01-04 09:52:25 -08:00
chrislu
5423790b2c
add back previous chunk upload selection algo
2023-01-03 22:50:39 -08:00
chrislu
7bdae5172e
batch delete EC needles
...
fix https://github.com/seaweedfs/seaweedfs/issues/4107
2023-01-03 22:05:26 -08:00
zemul
6b4c033431
add mount log ( #4101 )
...
* filer.backup use replication.source.filer
* add mount log
* Revert "filer.backup use replication.source.filer"
This reverts commit 07bf6f956c
2023-01-03 00:00:45 -08:00
chrislu
48f2edc065
3.38
2023-01-02 23:23:02 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 ( #4094 )
2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax ( #4096 )
2023-01-01 05:06:57 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query ( #4095 )
2023-01-01 05:06:41 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding ( #4092 )
2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
Guo Lei
5f993a5f20
Fix ide warning ( #4082 )
...
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
2022-12-26 00:03:47 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file ( #4060 )
...
* fs.verify needles status of file
* fix parse args
* refactor logging
* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
chrislu
77b7c9f7cc
lock and unlock chunks for read
2022-12-21 14:15:35 -08:00
chrislu
d1797deccb
remove duplicated logic of removing from writableChunks
2022-12-21 14:15:11 -08:00
chrislu
6ede19e825
add a simple file replication progress bar
2022-12-20 19:47:21 -08:00
chrislu
f7beba8515
unlock before submitting the uploading jobs
2022-12-20 12:52:30 -08:00
chrislu
28d479e5c0
mount: adjust locking for upload pipeline
2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305
filer sink retries reading file chunks, skipping missing chunks
...
if the file chunk is not available during replication time, the file is skipped
2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c
metadata skip reading un-available logs
...
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command ( #4061 )
2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993
more detailed logs
2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed ( #4057 )
2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key ( #4047 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-12-11 22:49:57 -08:00
chrislu
438146249f
3.37
2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9
rotate log files
2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c
volume server logs add url info
...
debug https://github.com/seaweedfs/seaweedfs/issues/3964
2022-12-11 13:15:38 -08:00
chrislu
38c63c429b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb
add chunk etag when downloading from remote storage
...
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting ( #4042 )
...
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming ( #4039 )
2022-12-07 07:47:40 -08:00
chrislu
659f28b855
simple refactoring
2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8
duplicated logic
2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" ( #4035 )
...
* Revert "refactor: moved to locked entry"
This reverts commit 94bc9afd9d
2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb
3.36
2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d
refactor: moved to locked entry
2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory ( #4027 )
2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer ( #4025 )
2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn ( #4024 )
2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories ( #4016 )
2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip ( #4012 )
2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata ( #4010 )
2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
...
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57
3.35
2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
...
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f
filer store: skip disabled location specific filer store initialization
...
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f
file name escape uri
...
fix https://github.com/seaweedfs/seaweedfs/issues/3969
2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part ( #3968 )
2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid ( #3963 )
2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers ( #3965 )
2022-11-10 07:09:53 -08:00
chrislu
47db75a695
3.34
2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType ( #3949 )
...
* fix filer.remote.sync to azure with ContentType
* fix pass X-Amz-Meta to X-Ms-Meta
2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration ( #3941 )
2022-11-04 00:34:40 -07:00
chrislu
4193dafce1
azure metadata: skip metadata prefixed with "X-"
...
fix https://github.com/seaweedfs/seaweedfs/issues/3875
2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete ( #3939 )
2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
...
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
...
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted ( #3932 )
2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
...
check needles status from volume server
2022-10-31 11:33:04 -07:00
chrislu
0f360862bf
3.33
2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies ( #3924 )
...
avoid vacuum if not enough replica copies
2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
...
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
8b9957d461
add back "/" prefix if it is missing in object
...
fix https://github.com/seaweedfs/seaweedfs/issues/3737
2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-28 12:53:23 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
...
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error ( #3918 )
2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048
master: delete partially created replicated volumes during volume growth
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit ( #3905 )
2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
...
* fix: set user metadata key to lowercase
* feat: simplify a bit
2022-10-24 17:29:52 -07:00
chrislu
091f999e8c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-24 16:50:41 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo ( #3893 )
...
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2022-10-24 09:25:48 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3
fix merge
2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
...
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
...
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40
skip reading needle if size is invalid
...
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d
remove volume server tcp client
2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a
fix size variable
2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
...
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0
3.32
2022-10-16 20:43:37 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9
Revert "mount: remove mount directory from generating cache directory id"
...
This reverts commit 109dc7fdf6
2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
...
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2022-10-13 23:15:00 -07:00
chrislu
e05637c42c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp ( #3855 )
...
* add acl helper functionalities
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* add tests
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* remove 0 when create map
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* delete when empty at `AssembleEntryWithAcp`
`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb
default hasSlowRead to true
2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true ( #3710 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities ( #3831 )
2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c ( #3834 )
2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata ( #3833 )
2022-10-12 07:14:49 -07:00
chrislu
e55076c46f
cloud tier: add retry when copying data file
...
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-11 21:58:19 -07:00
chrislu
0d817bc347
fix invalid memory address or nil pointer dereference on filer.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
...
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… ( #3824 )
...
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency ( #3813 )
2022-10-10 06:44:29 -07:00
chrislu
0711870f43
3.31
2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6
mount: remove mount directory from generating cache directory id
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
chrislu
f9383aa726
refactor to change capacity data type
2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
...
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options ( #3807 )
...
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not ( #3799 )
...
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452
* fix metric MasterAdminLock
2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics ( #3798 )
2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop ( #3797 )
2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
chrislu
ea271600ec
fix parameters
2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c
filer.sync: limit concurrency when fetching file chunks
...
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels ( #3785 )
2022-10-04 01:20:01 -07:00
chrislu
29632d5a34
3.30
2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis ( #3765 )
2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir ( #3770 )
2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7
better fix for option format
2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb
mount: fix extra option format
2022-09-30 08:44:41 -07:00
chrislu
8e81619d02
mount: accept all extra mount options
...
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
...
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer ( #3759 )
2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account ( #3753 )
...
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
...
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1
skip debug message
2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server ( #3708 )
...
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer ( #3750 )
...
Adjusted filer response headers
2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" ( #3747 )
...
* fs.meta.load load any dirs with prefix "important"
* replace dirPattern to dirPrefix
* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda
skip truncation on error
...
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e
3.29
2022-09-25 19:20:45 -07:00
chrislu
640c53cda8
add future list
2022-09-25 17:43:42 -07:00
chrislu
cc570a4477
assert results are sorted
2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1
ordered execution async wait
2022-09-25 13:45:55 -07:00
chrislu
973b8ff067
rename
2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b
rename file
2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f
filer replication: compare content changes directly
...
Fix https://github.com/seaweedfs/seaweedfs/issues/3714
The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.
For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-20 08:35:10 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf
adjust logs
2022-09-18 18:14:24 -07:00
chrislu
901e8157de
3.28
2022-09-18 17:46:04 -07:00
chrislu
9d230521a4
report write error
2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b
volume server: rename readBufferSize to readBufferSizeMB
2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-16 23:54:39 -07:00
chrislu
06748f2771
remove unused code
2022-09-16 23:54:37 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store ( #3707 )
2022-09-16 09:05:38 -07:00
Ryan Russell
824f7ad9e1
refactor(shell): readability improvements ( #3704 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:49 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements ( #3703 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization ( #3702 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35
mark "hasSlowRead" as experimental
2022-09-15 23:33:46 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements ( #3696 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… ( #3695 )
...
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:48 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep ( #3694 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet ( #3693 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4
volume: add "hasSlowRead" option to customize read optimization
2022-09-15 03:11:32 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var ( #3692 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
Ryan Russell
d65bdeef08
docs(meta_cache): readability fixes ( #3691 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:57 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… ( #3690 )
...
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:29 -07:00
Ryan Russell
a635973847
docs(topology): readability improvements ( #3688 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:03:10 -07:00
Ryan Russell
0fc242b084
docs: panicing -> panicking ( #3687 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:38:04 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes ( #3686 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:33:04 -07:00
Ryan Russell
150ce9f3f6
docs(wdclient): readability fix ( #3685 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:29:26 -07:00
ImFantuan
b64411bda8
fix 0 size ( #3683 )
2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default ( #3682 )
...
* simplify a bit
* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
2022-09-15 00:27:15 -07:00
chrislu
21c0587900
go fmt
2022-09-14 23:06:44 -07:00
chrislu
c8645fd232
master: implement grpc VolumeMarkWritable
...
fix https://github.com/seaweedfs/seaweedfs/issues/3657
2022-09-14 23:05:30 -07:00
Ryan Russell
c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr ( #3679 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:10:27 -07:00
Ryan Russell
12914af4d8
Character readability ( #3678 )
...
* refactor(pb): `quote_charactoer` -> `quote_character`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:09:53 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime ( #3676 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported ( #3677 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:16:39 -07:00
Ryan Russell
bd2dc6d641
refactor(shell): Decending -> Descending ( #3675 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:06:48 -07:00
Ryan Russell
dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… ( #3674 )
...
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:02:33 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive ( #3673 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements ( #3672 )
...
* refactor(net_timeout): `listner` -> `listener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(filer): `localPublicListner` -> `localPublicListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(command): `masterLocalListner` -> `masterLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(net_timeout): `ipListner` -> `ipListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:59:55 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction ( #3671 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:45 -07:00
Ryan Russell
c4363c4b5e
refactor(command_fs_rm): entiries -> entries ( #3670 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:27 -07:00
Ryan Russell
a36d7b77be
refactor(cipher_test): plantext -> plaintext ( #3669 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:09 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates ( #3668 )
...
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `SafeRenewInteval` -> `SafeRenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `InitLockInteval` -> `InitLockInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `RenewInteval` -> `RenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell
e22335ba78
refactor(queue_unbounded): inbountLen -> inboundLen ( #3666 )
2022-09-14 10:12:45 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability ( #3665 )
2022-09-14 10:11:31 -07:00
Ryan Russell
d734fff322
docs: replicte -> replicate ( #3664 )
2022-09-14 10:01:18 -07:00
Ryan Russell
dfaa602661
refactor(notification_kafka): parition -> partition ( #3663 )
2022-09-14 09:15:21 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… ( #3662 )
2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original ( #3661 )
2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer ( #3654 )
2022-09-14 06:33:00 -07:00
Ryan Russell
c9f573eed8
docs: indicates readability comment fix ( #3659 )
2022-09-14 06:31:07 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename ( #3658 )
2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix ( #3655 )
2022-09-14 04:53:04 -07:00
chrislu
58d18b68d8
rename variable
2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas ( #3591 )
...
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578
* put with in the util.Retry
* just purge on any errors
2022-09-13 22:33:35 -07:00
chrislu
0baf3d38c2
adjust logs
2022-09-13 10:33:28 -07:00
chrislu
ce55922884
fixing
2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero ( #3630 )
...
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629
* fix GrpcPort
* skip url :0
* skip empty DataCenter or Rack
* skip empty heartbeat Ip
* dell msg add DataCenter
* comment todo
* fix
2022-09-11 22:31:53 -07:00
chrislu
b834027c5a
refactor
2022-09-11 21:53:15 -07:00
chrislu
d8ca7d34fe
3.27
2022-09-11 19:47:53 -07:00
chrislu
22064c3425
mount: ensure ordered file handle lock and unlock
2022-09-11 19:44:34 -07:00
chrislu
b9112747b5
volume server: synchronously report volume readonly status to master
...
fix https://github.com/seaweedfs/seaweedfs/issues/3628
2022-09-11 19:29:10 -07:00
chrislu
6690236754
rename variables
2022-09-11 19:18:42 -07:00
chrislu
4af8c427eb
refactor
2022-09-10 16:56:03 -07:00
chrislu
4957d8eec6
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-10 15:36:02 -07:00
chrislu
b1501f61dd
Revert "a new test case"
...
This reverts commit eb7cf3de81
2022-09-10 15:35:42 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param ( #3626 )
...
* ADHOC: cut off volumn fsck
* more
* fix typo
* add test
* modify name
* fix comment
* fix comments
* nit
* fix typo
* Update weed/shell/command_volume_fsck.go
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-10 15:29:17 -07:00
chrislu
2c6b68b40e
simplify a bit
2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a
[sync] override amz storage class, None to delete ( #3639 )
...
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636
* use empty string to delete
* without nil check
2022-09-10 14:15:42 -07:00
chrislu
3cb914f7e1
avoid dead lock
2022-09-10 11:26:19 -07:00
chrislu
ebef579ebb
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-10 11:22:18 -07:00
chrislu
576c113c59
replace PR https://github.com/seaweedfs/seaweedfs/pull/3621
...
replace https://github.com/seaweedfs/seaweedfs/pull/3621
2022-09-10 11:22:16 -07:00
Brian
4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes ( #3640 )
2022-09-10 08:05:28 -07:00
chrislu
205ecb5d03
skip directories under ".uploads" directory
...
related to https://github.com/seaweedfs/seaweedfs/issues/3636
skipping all directories under ".uploads" directory.
2022-09-09 22:57:38 -07:00
chrislu
fc4208d128
volume.balance: default to balance ALL_COLLECTIONS
2022-09-09 16:06:05 -07:00
gfx
48db56ddad
arangodb s3 bucket name compatibility ( #3588 )
...
* Update arangodb_store.go
* update readme, properly escape queries, add name patching
* use underscore
* use underscore
* better comment
* fix readme
Co-authored-by: a <a@a.a>
2022-09-09 09:43:42 -07:00
VictorSFF
10d545060f
mount: fix new inode allocation ( #3633 )
2022-09-09 06:32:19 -07:00
famosss
41af5a6ee5
simplify a bit ( #3631 )
2022-09-08 21:01:21 -07:00
chrislu
64e75a286e
simplify a bit
2022-09-08 19:07:57 -07:00
chrislu
d14d029c73
reduce v.dataFileAccessLock lock scope
2022-09-08 19:05:31 -07:00
chrislu
406a80da4b
adjust variable names
2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2
refactor: move ReadNeedleDataInto into volume_read.go
2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88
purge tcp implementation
2022-09-08 18:03:43 -07:00
chrislu
eb7cf3de81
a new test case
2022-09-08 14:04:42 -07:00
chrislu
4260804613
volume.balance: avoid moving out volume with max=1
2022-09-08 14:04:21 -07:00
chrislu
d653c5f811
unused
2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43
rename
2022-09-08 12:06:03 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 ( #3625 )
2022-09-07 23:23:33 -07:00
chrislu
7de112943b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195
return err for processRangeRequest()
2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown ( #3618 )
2022-09-07 18:57:13 -07:00
Konstantin Lebedev
695e3a909c
avoid data race on currentDiskUsage.maxVolumeCount ( #3592 )
...
https://github.com/seaweedfs/seaweedfs/issues/3590
2022-09-07 09:51:15 -07:00
Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount ( #3594 )
...
https://github.com/seaweedfs/seaweedfs/issues/3593
2022-09-07 09:50:17 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error ( #3616 )
2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown ( #3613 )
...
* fix: volume Deadlock when exception is thrown
* adjust log level
2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc ( #3581 )
...
* ADHOC: add read needle meta grpc
* add test
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-06 23:51:27 -07:00
chrislu
c07ab9c060
3.26
2022-09-06 08:26:20 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket ( #3609 )
...
https://github.com/seaweedfs/seaweedfs/issues/3607
2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting ( #3577 )
2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level ( #3610 )
2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap ( #3606 )
2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic ( #3596 )
2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log ( #3598 )
2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF ( #3597 )
2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e
3.25
2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level ( #3589 )
2022-09-04 22:37:32 -07:00
chrislu
f4abefc109
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 22:23:31 -07:00
chrislu
39340f7e42
cloud tier: s3 consume all read response body
...
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2022-09-04 22:23:28 -07:00
chrislu
7c6324b114
adjust log level
2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket ( #3587 )
...
* Update helpers.go
* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context ( #3586 )
...
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d
minor optimization
2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca
filer.backup and filer.sync: include headers during backup and sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3532
2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level ( #3585 )
2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move ( #3583 )
2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync ( #3579 )
...
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed ( #3575 )
2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c
filer: rocksdb store report not found correctly
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e
Revert "avoid data race on grace.hooks ( #3572 )"
...
This reverts commit c37d6fc01a
2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks ( #3572 )
...
https://github.com/seaweedfs/seaweedfs/issues/3564
2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket ( #3571 )
...
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552
* copy localSocket
2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize ( #3570 )
...
https://github.com/seaweedfs/seaweedfs/issues/3557
2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log ( #3568 )
...
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567
* format message
2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab
align memory
2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch ( #3566 )
...
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1
add more logs
2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 ( #3541 )
...
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0
Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud ) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases )
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0 )
---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update code
* more code fix
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount ( #3562 )
...
* Add an e2e workflow to test FUSE mount
* Fix deadlocks during concurrent r/w
2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c
upload_content: upload close response.Body
2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced
refactor
2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c
just a bit safer
2022-08-31 00:10:17 -07:00
chrislu
9c944377dc
minor
2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6
s3: close response body with copying object
2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae
print process before do the work
2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt ( #3554 )
2022-08-30 10:12:02 -07:00
chrislu
de5263364b
master client: use GetMaster() for latest master address
2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a
more detailed error
2022-08-30 09:43:17 -07:00
chrislu
d81db3c703
s3: fix configuring IAM for the same user
...
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]
s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version ( #3539 )
...
* avoid race conditions access to SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515
* superBlockAccessLock replace to sync.Mutex
2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0
rename variable
2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster ( #3538 )
...
https://github.com/seaweedfs/seaweedfs/issues/3510
2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr ( #3550 )
...
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay ( #3548 )
2022-08-30 00:02:09 -07:00
chrislu
570b488572
ensure memory alignment on different CPUs
2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount ( #3537 )
...
https://github.com/seaweedfs/seaweedfs/issues/3511
2022-08-29 13:23:02 -07:00
Patrick Schmidt
00041f6e08
Fix corrupted mount reads after changes on fh ( #3536 )
2022-08-29 08:27:13 -07:00
chrislu
b7a887fea1
3.24
2022-08-28 21:53:13 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug ( #3534 )
2022-08-28 07:51:47 -07:00
chrislu
cc0c8c5f81
simplify
2022-08-27 00:21:57 -07:00
chrislu
3741779144
minor
2022-08-27 00:20:58 -07:00
chrislu
580a7fa685
adjust for data race
...
fix https://github.com/seaweedfs/seaweedfs/issues/3512
2022-08-27 00:19:13 -07:00
chrislu
87b70a6809
clean up
2022-08-27 00:09:04 -07:00
chrislu
27c7611ebf
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 23:47:39 -07:00
chrislu
9920d65bc0
gateway to remote object store: adjust upload concurrency
2022-08-26 23:47:37 -07:00
chrislu
c839ce1b19
s3 sink use s3 upload manager
...
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-26 23:47:12 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children ( #3530 )
2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8
make CodeQL happy
2022-08-26 17:09:11 -07:00
chrislu
301b49b63f
atomic operation
2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0
nano level precision
2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c
more detailed error
...
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount ( #3527 )
2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8
rename varaible
2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate ( #3525 )
...
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount ( #3526 )
2022-08-26 08:41:42 -07:00
chrislu
6f483a4d36
atomic add
...
fix https://github.com/seaweedfs/seaweedfs/issues/3514
2022-08-25 22:20:34 -07:00
chrislu
7394f7feee
minor: adjust ordering
2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1
minor clean up
2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks ( #3505 )
2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races ( #3503 )
2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created ( #3498 )
...
* replace mkdir to mkFile
* ContentLength must be zero
* revert mkDir
* Seaweedfs-Is-Directory-Key return metadata
2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
...
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu
10414fd81c
ping timeout at 15 seconds
...
this 72 minute timeout setting seems unreasonably long
15 seconds is around the time when a new raft leader should be elected.
2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long ( #3398 )
...
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug ( #3499 )
2022-08-23 08:53:24 -07:00
chrislu
21a933c932
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-23 01:52:32 -07:00
chrislu
f0b4a7659a
fix test
2022-08-23 01:52:29 -07:00
Patrick Schmidt
cda8cc22bc
Implement lseek syscall in FUSE ( #3491 )
...
See the man page of lseek:
https://man7.org/linux/man-pages/man2/lseek.2.html
2022-08-23 01:47:18 -07:00
chrislu
bf88006037
format
2022-08-23 01:20:45 -07:00
chrislu
9fce75607d
s3: report http.StatusOK if the directory is explicitly created
...
fix https://github.com/seaweedfs/seaweedfs/issues/3457
2022-08-23 01:16:46 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil ( #3497 )
2022-08-22 23:54:51 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests ( #3496 )
2022-08-22 23:39:34 -07:00
chrislu
676e27c589
shell: stop long running jobs if lock is lost
2022-08-22 14:12:23 -07:00
chrislu
601ba5fb68
master: disable locking if not leader
2022-08-22 14:11:48 -07:00
chrislu
57e7582c36
refactoring
2022-08-22 14:11:13 -07:00
chrislu
7ea2c24c5d
metadata processing use retry instead of retryForEver
2022-08-22 11:25:40 -07:00
Patrick Schmidt
f875031f06
Reuse readDataByFileHandle in Read call ( #3482 )
2022-08-22 08:24:06 -07:00
famosss
bdba3da2e4
Handle preflight cors requests ( #3481 )
2022-08-22 08:21:38 -07:00
chrislu
f4b52d4c52
fix format
2022-08-22 00:20:21 -07:00
chrislu
b4131a3a72
filer.remote.sync: parallelize uploading to remove object storage
2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561
refactor
2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-21 23:26:09 -07:00
chrislu
8bac903d29
mount: add locking to prevent nil pointer
...
fix https://github.com/seaweedfs/seaweedfs/issues/3479
2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files ( #3478 )
...
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
7807f6641a
minor
2022-08-21 22:54:56 -07:00
chrislu
c4e862e908
3.23
2022-08-21 19:13:26 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
...
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
2022-08-21 19:11:18 -07:00
chrislu
63fbf281c7
remove same file copying rage limitation
2022-08-21 17:16:21 -07:00
Patrick Schmidt
3a75d7f7aa
Implement copy_file_range syscall in FUSE ( #3475 )
...
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
2022-08-21 17:11:43 -07:00
chrislu
fef9c6a520
filer: do not always save files in "/etc" folder to filer store
...
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
chrislu
2762154130
fix compilation
2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3
EC: with multiple volume locations, the ec rebuilding may fail
2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f
EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations
...
related to https://github.com/seaweedfs/seaweedfs/issues/3459
2022-08-21 14:52:37 -07:00
chrislu
8b3429858d
align memory for atomic read/write
2022-08-21 12:20:27 -07:00
Patrick Schmidt
2ef6ab998c
Avoid race conditions with current filer address ( #3474 )
...
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
2022-08-21 12:18:13 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher ( #3473 )
...
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu
388f82f322
minor
2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e
refactoring
2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps ( #3472 )
...
* Fix FUSE server buffer leaks in file gaps
This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.
* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu
c7892bc7c4
volume: file counter should be all files
...
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2022-08-20 23:35:31 -07:00
chrislu
f7e0a65e75
retry for all errors
2022-08-20 23:34:45 -07:00
chrislu
649555b223
adjust log
2022-08-20 23:34:07 -07:00
chrislu
973f6dd162
refactoring
2022-08-20 22:38:15 -07:00
chrislu
65ff7198fe
adjust type
2022-08-20 22:37:24 -07:00
chrislu
28b862f45f
use util.Retry to retry
2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8
webdav: retryable data chunk upload
2022-08-20 19:18:12 -07:00
chrislu
f8fa430257
filer: retryable data chunk upload
2022-08-20 19:15:44 -07:00
chrislu
4081d50607
filer sink: retryable data chunk uploading
2022-08-20 19:09:15 -07:00
chrislu
409f39390d
mount: retryable data chunk uploading
2022-08-20 19:04:36 -07:00
chrislu
6c8822f269
filer.copy: retryable file part upload
2022-08-20 18:59:57 -07:00
chrislu
a3553da7f7
add fileId to UploadWithRetry return result
2022-08-20 18:54:59 -07:00
chrislu
d49d0a9fc2
filer.copy: retryable upload
2022-08-20 18:50:57 -07:00
chrislu
689b4ecdcc
remove unused collection and replication from upload result
2022-08-20 18:14:57 -07:00
chrislu
e3f40d538d
cleaner code
2022-08-20 17:51:30 -07:00
chrislu
9f50b310a0
remove unused code
2022-08-20 17:51:12 -07:00
chrislu
11f99836c3
filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
...
fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-19 23:00:56 -07:00
chrislu
fdd8c5d5e0
Filer if-modified-since check doesn't fire on "exact" #3467
...
fix https://github.com/seaweedfs/seaweedfs/issues/3467
2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89
filer: retryable when error is not found
2022-08-19 03:36:15 -07:00
chrislu
58dd880322
minor
2022-08-18 23:51:08 -07:00
Eric Yang
b6f99c4d83
fix naming typo ( #3465 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-18 23:25:32 -07:00
chrislu
5702727db5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-18 02:39:27 -07:00
chrislu
4d72a1ce6e
s3: remove deleted collection during metrics reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2022-08-18 02:39:07 -07:00
Guo Lei
3172c33fa6
fix redundant type from array ( #3462 )
2022-08-18 02:25:25 -07:00
chrislu
42c6e52513
s3: fix regression on HEAD directory operation
2022-08-18 02:13:58 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
e259052a70
volume: fix metrics for volume request time
2022-08-17 04:29:31 -07:00
qzh
7fcfaf7bc9
fix(filer.sync): offset may be set to 0 ( #3451 )
...
* fix(filer.sync): initializing the offset is related to the path
* fix(filer.sync): the offset maybe to be set to 0.
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 23:43:52 -07:00
qzh
400f0c3e5d
fix(filer.sync): initializing the offset is related to the path ( #3450 )
...
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 21:56:47 -07:00
chrislu
fa4d0093e1
3.22
2022-08-15 16:48:23 -07:00
chrislu
93261f5199
Revert "Refactor for Sync method ( #3426 )"
...
This reverts commit 670cb759f8
2022-08-15 15:35:31 -07:00
chrislu
9c588d4010
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-15 10:03:56 -07:00
chrislu
96caf21d09
less verbose log
2022-08-15 10:03:52 -07:00
Andrey Triumfov
31faa6d43d
Remove duplicate slashes in object path to prevent 500 errors ( #3442 )
2022-08-15 08:19:28 -07:00
chrislu
cb476a53ff
remove logs
2022-08-15 01:05:35 -07:00
chrislu
7c029b2183
3.21
2022-08-15 00:32:15 -07:00
Chris Lu
e88392b50f
Fix s3 pagination ( #3436 )
...
* Revert previous changes
* s3: use cursor to track tree traversal
fix https://github.com/seaweedfs/seaweedfs/issues/3166
* special cases for empty prefix and empty directory
* use constants
* address empty folder
* undo local changes
* fix IsTruncated
* adjust counting directories
* fix cases when prefix is a directory
* s3: handle directory object
works for
aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
2022-08-15 00:30:19 -07:00
chrislu
7457c746f0
s3: fix aws s3api head-object
2022-08-14 23:52:35 -07:00
chrislu
789d9dbfe1
webdav: fix nil
...
fix https://github.com/seaweedfs/seaweedfs/issues/3440
2022-08-14 23:33:30 -07:00
Konstantin Lebedev
6199d9b923
rand start garbageThreshold and reset metric ReplicaPlacementMismatch ( #3396 )
...
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch
* vacuum through sleep
2022-08-12 13:11:09 -07:00
Konstantin Lebedev
4d4cd0948d
avoid infinite loop WaitUntilConnected() ( #3431 )
...
https://github.com/seaweedfs/seaweedfs/issues/3421
2022-08-11 15:03:26 -07:00
Eric Yang
986daecbfe
add upload response ( #3428 )
...
* add upload response
* fixing comments
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-10 22:43:09 -07:00
chrislu
ae93c966d9
ensure memory is aligned
...
fix https://github.com/seaweedfs/seaweedfs/issues/3427
2022-08-10 22:27:13 -07:00
Rain Li
670cb759f8
Refactor for Sync method ( #3426 )
2022-08-10 08:21:57 -07:00
chrislu
662ec97602
mount: adjust on du stats reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3424
2022-08-09 20:21:01 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code ( #3423 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-09 01:24:43 -07:00
chrislu
0854171d22
3.20
2022-08-07 14:42:28 -07:00
chrislu
d8e69df295
adjust comments
2022-08-07 11:35:33 -07:00
chrislu
0aeec04c31
quicker to adapt to pattern change
2022-08-07 10:14:01 -07:00
chrislu
928d29af9e
fix wrong logic about reader isRandomMode()
2022-08-07 10:13:04 -07:00
chrislu
d3d52b1818
shell: fs.meta.load add quieter mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3414
2022-08-07 01:42:15 -07:00
chrislu
358ccb5e0e
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-07 01:34:36 -07:00
chrislu
67814a5c79
refactor and fix strings.Split
2022-08-07 01:34:32 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection ( #3413 )
2022-08-07 00:58:53 -07:00
chrislu
1a4bf0dcb5
filer.sync: parallelize the filer.sync
2022-08-07 00:56:15 -07:00
chrislu
02b38f894b
debug
2022-08-07 00:55:49 -07:00
chrislu
0e9478488d
filer.sync: fix when excluded paths is empty
2022-08-07 00:55:34 -07:00
chrislu
8a880a139d
mount: fix truncate operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/2609
2022-08-06 13:58:45 -07:00
LHHDZ
84ec68e11a
Add download speed limit support ( #3408 )
2022-08-05 01:16:42 -07:00
chrislu
b278bb24d3
mount: adjust df stats reporting when close to the limit
...
fix https://github.com/seaweedfs/seaweedfs/issues/3407
2022-08-05 00:31:41 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics ( #3402 )
...
* refactor FilerRequest metrics
* avoid double count proxy
* defer to
2022-08-04 01:44:54 -07:00
chrislu
bd13a7968f
remove unused symlink resolving
2022-08-04 01:35:18 -07:00
chrislu
88945d9954
mount: do not follow soft link with xattr
2022-08-04 01:21:15 -07:00
chrislu
90db4b5a44
minor
2022-08-03 00:10:47 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite
...
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 13:31:01 -07:00
Konstantin Lebedev
3ffa4ba91d
clean
2022-08-02 01:30:38 +05:00
Konstantin Lebedev
2a5629dd28
align
2022-08-02 01:29:44 +05:00
Konstantin Lebedev
df9cc31a0a
refactor
2022-08-02 01:18:26 +05:00
Konstantin Lebedev
0a5d3c3cf6
rename to localWrite and replicatedWrite
2022-08-01 23:09:09 +05:00
chrislu
bd5c9904be
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-01 09:40:05 -07:00
chrislu
e26d1cf4ef
fix test
2022-08-01 09:40:01 -07:00
Konstantin Lebedev
fc65122766
rename to LoadAvg_1M
2022-08-01 21:32:21 +05:00
Konstantin Lebedev
ab50f21ad9
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 21:16:57 +05:00
Konstantin Lebedev
2f85e31f5c
Merge branch 'master' into gentle_vacuum
2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef
remove percent
2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083
...
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 08:23:10 -07:00
Konstantin Lebedev
78cbd8002f
revert Sleep
2022-08-01 20:21:23 +05:00
Konstantin Lebedev
cd5c7ad052
move to github.com/seaweedfs/seaweedfs
2022-08-01 16:36:32 +05:00
Konstantin Lebedev
634bc97840
fix conflicts
2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
Konstantin Lebedev
c0d92f61a1
comment
2022-08-01 14:40:42 +05:00
Konstantin Lebedev
1d29f67c02
revert disk stats
2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3
rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 12:51:41 +05:00
chrislu
036566629a
filer.sync: fix synchronization logic in active-active mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3328
2022-08-01 00:06:18 -07:00
chrislu
303bd067b5
Revert "rename: delete source entry metadata only, skipping hard links"
...
This reverts commit 03466f955ehttps://github.com/seaweedfs/seaweedfs/issues/3386
2022-07-31 22:51:41 -07:00
chrislu
03c6f978f4
3.19
2022-07-31 17:16:51 -07:00
chrislu
afef014832
scaffold
2022-07-31 13:23:44 -07:00
Chris Lu
0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy
...
Add healthy indicator for raft status
2022-07-30 11:55:30 -07:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status
2022-07-30 19:34:26 +02:00
chrislu
cb096d1f53
minor
2022-07-30 09:52:52 -07:00
chrislu
0423d50106
wip note
2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a
moving
2022-07-29 23:39:11 -07:00
chrislu
13b9a52f80
fix deprecated functions
2022-07-29 01:34:39 -07:00
chrislu
be0ec7bc2f
fix test
2022-07-29 01:15:26 -07:00
chrislu
1f161f53dc
add back rack
2022-07-29 01:15:19 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
chrislu
de3afd4199
fix rebasing
2022-07-28 23:25:46 -07:00
chrislu
06cd491abc
add message batch
2022-07-28 23:24:38 -07:00
chrislu
7576c244c4
add flatbuffer serde for message
2022-07-28 23:24:38 -07:00
chrislu
a3fae7cd16
add an empty placeholder for mq.topic.list
2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47
segment serde
2022-07-28 23:24:38 -07:00
chrislu
74f60f246f
dynamically connect to a filer
2022-07-28 23:24:38 -07:00
chrislu
9f479aab98
allocate brokers to serve segments
2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0
refactor
2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c
clean up
2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5
remove old code
2022-07-28 23:24:38 -07:00
chrislu
f25e273e32
display data center and rack in cluster.ps
2022-07-28 23:22:52 -07:00
chrislu
68065128b8
add dc and rack
2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd
mq broker adds data center and rack
2022-07-28 23:22:07 -07:00
chrislu
87ccef72db
cluster.ps can list brokers
2022-07-28 23:22:07 -07:00
chrislu
ac66227798
renaming
2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8
renaming
2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17
rename to SeaweedMQ
2022-07-28 23:22:06 -07:00
Chris Lu
3e7e922d16
Merge pull request #3378 from vanhtuan0409/change-put-policy
...
PutUserAction should completely replace identity actions
2022-07-28 19:43:25 -07:00
chrislu
89e276de64
fix loop
2022-07-28 18:06:49 -07:00
chrislu
a458bee9f2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-28 16:32:02 -07:00
chrislu
e0eda52c54
mount: ensure symlink parent directory is tracked
...
fix https://github.com/chrislusf/seaweedfs/issues/3373
2022-07-28 16:32:00 -07:00
Tuan Vuong
1d9a888be9
PutUserAction should completely replace identity actions
2022-07-28 17:02:56 +07:00
zzq09494
01b7aa79fa
fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted
2022-07-28 12:04:37 +08:00
chrislu
ca836568ac
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87
"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"
2022-07-27 12:12:40 -07:00
Chris Lu
b7e4e26778
Merge pull request #3371 from askeipx/issues/3341
...
delete disk_size metrics when collection deleted
2022-07-27 11:25:42 -07:00
Konstantin Lebedev
7e09a548a6
exclude directories to sync on filer
2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev
ac5ce31278
leave notion to refactor after prometheus upgrade
2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev
47c72e6f35
remove all (currently existing) collection volume metrics
2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c
delete disk_size metrics when collection deleted
2022-07-27 15:02:41 +05:00
chrislu
f401b996eb
mount: fix symlink size reporting
2022-07-26 23:12:25 -07:00
Konstantin Lebedev
c88ea31f62
fix RUnlock of unlocked RWMutex
2022-07-26 12:57:07 +05:00
Konstantin Lebedev
da9d3e8f6c
refactor
2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4
fix logic else brake
2022-07-26 11:47:11 +05:00
chrislu
475185fb72
3.18
2022-07-25 22:41:38 -07:00
chrislu
ec0edb1ac4
filer: fix wrong logic during read
2022-07-25 22:40:00 -07:00
guosj
cc7a4b0a6e
correct comment
2022-07-25 11:46:41 +08:00
chrislu
33cff61992
fix glog test
2022-07-24 19:13:54 -07:00
chrislu
dbe4849ffd
3.17
2022-07-24 18:46:50 -07:00
chrislu
b98c0b4bf4
disable symlink translation for now
2022-07-24 18:46:32 -07:00
chrislu
2dc5d4adac
disable symlink translation for now
2022-07-24 18:24:34 -07:00
chrislu
3d2bbf96d2
adjust glog format
2022-07-24 17:29:08 -07:00
chrislu
2807fe08fc
adjust glog format
2022-07-24 16:44:58 -07:00
chrislu
2b64916d3e
set status to fuse.OK
2022-07-24 15:30:55 -07:00
chrislu
7386cf6a06
mount: resolve symlink
2022-07-24 15:11:24 -07:00
chrislu
40089b6753
maintain node lookup counter
2022-07-24 13:43:11 -07:00
chrislu
d24db396cc
fix removing path from inode2path
2022-07-23 18:20:29 -07:00
chrislu
41eeb4deef
do not add new inode during link
2022-07-23 13:37:07 -07:00
chrislu
02c2d81cde
fix removePathFromInode2Path
2022-07-23 13:21:42 -07:00
chrislu
4d5dc55766
mount: better support hard links
2022-07-23 12:57:00 -07:00
chrislu
ebda0689ab
use linux default max name length
2022-07-23 10:51:06 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
2c8818351f
mount: comment out stuck fuse inode notify during renaming
2022-07-22 18:17:34 -07:00
chrislu
a5c94acd4f
mount: check possible nil entry case
2022-07-22 15:33:37 -07:00
Chris Lu
0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap
...
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
2022-07-22 02:57:50 -07:00
LHHDZ
994a2dec78
fix vid_map_test tests
2022-07-22 17:54:57 +08:00
LHHDZ
58c02d6429
Solve the problem that LookupFileId lookup urls is empty due to leader switching
...
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
ningfd
58f2dd6740
change topology mapping definition, improve spelling
2022-07-22 17:18:56 +08:00
ningfd
6f882eb354
optimiz: master ui will render data in order
2022-07-22 14:37:49 +08:00
chrislu
7a6c559ab4
fix Change replication via volume.configure.replication by collection
...
fix https://github.com/chrislusf/seaweedfs/issues/3346
2022-07-21 22:01:05 -07:00
chrislu
fc8241fb5e
leveldb3: add instant dropping bucket
2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44
explicit bucket aware declaration
2022-07-21 18:23:53 -07:00
Konstantin Lebedev
3c42814b58
avoid deadlock
2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb
use safe onPeerUpdateDoneCns
2022-07-21 15:51:14 +05:00
chrislu
6147b61b0a
mount: fix concurrent map read and map write
...
fix https://github.com/chrislusf/seaweedfs/issues/3344
2022-07-21 01:38:26 -07:00
Chris Lu
9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume
2022-07-20 09:06:06 -07:00
Konstantin Lebedev
7875470e74
onPeerUpdateGoroutineCount use int32
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
c5189c343b
remove ticker
...
update the topology before each file
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
e2d991d8d0
ticker.Stop
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
de4fcc0e2c
sync update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
fa88dff7ce
update otherNodes
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
72dca31cfa
fix update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
884ffbafee
clouse background update
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
39eaf426f8
fix TestVolumeServerEvacuate
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b5e5f6f55a
update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
867269cdcf
help rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6f764e1014
volume server evacuate from rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
ba0e3ce5fa
volume server evacuate to target server
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b6471ecd75
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7
fix design
2022-07-20 18:08:12 +05:00
Tuan Vuong
312e13416b
filter duplicated action
2022-07-20 16:43:50 +07:00
guol-fnst
ac694f0c8f
rename parameter and reuse functions
...
rename milestone to watermark
2022-07-20 17:00:40 +08:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer
...
volume server evacuate to target server
2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03
remove ticker
...
update the topology before each file
2022-07-20 00:54:23 +05:00
Konstantin Lebedev
11e393dbe7
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev
f6a966b4fc
add waiting log message
2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849
check for ping before deleting raft server
...
https://github.com/chrislusf/seaweedfs/issues/3083
2022-07-20 00:04:12 +05:00
chrislu
2ae3f812f8
minor
2022-07-19 11:43:31 -07:00
Chris Lu
9667588af0
Merge pull request #3318 from kmlebedev/issues/3310
...
Use fallback if urls are not found
2022-07-19 11:38:43 -07:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove
...
raftServerRemoval typo and logs
2022-07-19 11:32:17 -07:00
guol-fnst
91285bb51d
remove uncessary code
2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34
optimiz
2022-07-19 16:58:53 +08:00
Konstantin Lebedev
f419d5643a
fix typo
...
add remove logs
2022-07-19 11:50:52 +05:00
guol-fnst
bec9f79659
use snapshot
2022-07-19 11:14:41 +08:00
Konstantin Lebedev
d422e7769c
ticker.Stop
2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b
sync update topologyInfo
2022-07-18 16:27:02 +05:00
chrislu
56ec89625a
3.16
2022-07-18 00:53:05 -07:00
Konstantin Lebedev
2b4112e462
update otherNodes
2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d
fix update topologyInfo
2022-07-18 01:46:31 +05:00
Konstantin Lebedev
7b1497ee63
Use BackoffSchedule for getLookupFileId
2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8
Use fallback if urls are not found
2022-07-15 15:29:15 +05:00
Konstantin Lebedev
4d5144e50d
clouse background update
2022-07-15 13:51:08 +05:00
chrislu
1db012485f
mount: fix bug during busy writes
...
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4
more logging related to filer metadata subscription
2022-07-15 00:42:13 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag
...
File upload should succeed should return `ETag` instead of `Etag`
2022-07-14 15:35:41 -07:00
chrislu
fa61074513
add clientId logging
2022-07-14 12:27:34 -07:00
chrislu
fbd8f868a1
filer may have trouble to re-connect
...
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
2022-07-14 12:15:31 -07:00
LHHDZ
c08f6d0ff7
File upload should succeed should return ETag instead of Etag
...
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
Chris Lu
957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume
...
optimiz concurrency
2022-07-13 23:12:24 -07:00
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2
optimiz concurrency
...
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak
2022-07-13 09:23:37 -07:00
guosj
354b7bdff0
replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()
2022-07-13 22:49:03 +08:00
chrislu
289402a741
writer pattern: similar changes to reader pattern
2022-07-13 02:30:44 -07:00
guosj
ab1b9697e6
supplement check duplicate accesskey
2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0
detect sequential mode with a counter
2022-07-13 02:20:03 -07:00
chrislu
578c8aea0e
typo
2022-07-13 02:18:31 -07:00
chrislu
1451b389a4
Avoid fatal error: sync: Unlock of unlocked RWMutex
...
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
石昌林
392da4e038
Make the prompt information clearer
2022-07-13 11:39:45 +08:00
石昌林
ad5e8f68ec
Check whether there is a duplicate accessKey when modifying iam
2022-07-13 11:36:11 +08:00
Chris Lu
fba5219dab
Merge pull request #3303 from guol-fnst/loading_volume
...
speeding up loading volumes
2022-07-12 03:01:24 -07:00
Konstantin Lebedev
6622240df7
fix TestVolumeServerEvacuate
2022-07-12 14:56:34 +05:00
chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
Konstantin Lebedev
8372721a62
update topologyInfo
2022-07-12 13:47:21 +05:00
guol-fnst
26313060a3
speeding up loading volumes
2022-07-12 16:19:33 +08:00
Konstantin Lebedev
ee95d23a22
help rack
2022-07-12 11:56:58 +05:00
Konstantin Lebedev
087fa1347f
volume server evacuate from rack
2022-07-12 11:33:08 +05:00
Chris Lu
de97ba94c3
Merge pull request #3302 from guo-sj/fix_put_user_policy
2022-07-11 21:15:37 -07:00
guosj
b3076909d1
refactor code
2022-07-12 11:25:36 +08:00
Konstantin Lebedev
4236c36599
volume server evacuate to target server
2022-07-11 16:58:15 +05:00
Chris Lu
7929a50327
Merge pull request #3296 from guo-sj/fix_put_user_policy
...
refine PutUserPolicy
2022-07-11 01:49:25 -07:00
guosj
7dbf19d09a
refine PutUserPolicy
2022-07-11 16:38:41 +08:00
chrislu
37578929d4
fix compilation
2022-07-11 01:30:00 -07:00
chrislu
93ccc6e05f
3.15
2022-07-11 00:50:07 -07:00
chrislu
f87952a411
remove locks
2022-07-11 00:25:20 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
3d0defa663
fix compilation
2022-07-08 19:16:58 -07:00
chrislu
6f00c7f061
comments
2022-07-08 19:07:31 -07:00
chrislu
d685b94102
mount: rename also invalidate source inode
2022-07-08 19:07:03 -07:00
chrislu
48382676d2
fix filtering by volume id
2022-07-08 10:29:24 -07:00
liubaojiang
4e83e92156
add subscriber clientId if it is the first time
2022-07-08 20:27:25 +08:00
guosj
5b3192344d
fix iam CreatePolicy error
2022-07-08 17:42:21 +08:00
chrislu
e2ecf137a2
fix resetting fh.reader
2022-07-08 01:04:15 -07:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
b7de5c6c43
shell: remove unused filer parameter from shell.toml
2022-07-07 15:01:23 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
chrislu
05c53820b9
mount: file handler release memory
2022-07-07 11:49:40 -07:00
chrislu
69ef6459c3
S3 ListObjectsV2 fix pagination under bucket root
...
fix https://github.com/chrislusf/seaweedfs/issues/3166
2022-07-06 02:28:34 -07:00
Konstantin Lebedev
785223e587
rabbitpubsub enable durable
2022-07-06 10:05:29 +05:00
chrislu
57975b93ff
adjust used size reporting
2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5
adjust used size reporting
2022-07-04 13:46:32 -07:00
chrislu
3c79c77056
3.14
2022-07-04 13:15:49 -07:00
guol-fnst
5bc895673e
add getting bucket quota
2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610
fix usage
2022-07-04 11:31:39 +08:00
chrislu
141f662734
edge case checking when volume server does not have capacity to balance
...
fix https://github.com/chrislusf/seaweedfs/issues/3257
2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798
fix tests
2022-07-01 12:09:42 -07:00
Chris Lu
ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255
...
volumeServer.evacuate fix select empty node
2022-07-01 02:36:30 -07:00
Konstantin Lebedev
3d5f1cbc5e
fix pagination
2022-07-01 14:17:49 +05:00
chrislu
06a8b174b5
also remove Sync() for idx file
2022-06-30 13:50:53 -07:00
chrislu
52580743b9
stuck with file.Sync()
...
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
2022-06-30 12:32:55 -07:00
Konstantin Lebedev
5ed8165161
fix logic
...
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
2022-06-30 17:58:24 +05:00
石昌林
200a862701
fixed volume xx not found caused by missing VolumeLocation events
...
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
Chris Lu
4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta
...
[Critical] Avoid synchronizing from scratch due to err
2022-06-29 12:36:28 -07:00
Chris Lu
b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder
...
use const multipart uploads folder
2022-06-29 12:09:03 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
Konstantin Lebedev
bcbdc4cb37
use const multipart uploads folder
...
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu
a8abab2412
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-29 02:44:14 -07:00
chrislu
449ecc8890
remove max connection age
...
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy
...
validate tags on copy object and add regex for validating tags
2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535
remove println
2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2
fix return
2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c
fix return
2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913
fix parseTagging function and add test for validate tags
2022-06-28 14:48:55 +03:00
Chris Lu
a79bf9d92a
Merge pull request #3252 from guol-fnst/update_tbk
...
remove hdfs, add tikv for full version update
2022-06-27 22:37:41 -07:00
guol-fnst
74c9f39d48
remove hdfs, add tikv for full version update
2022-06-28 12:35:07 +08:00
石昌林
e1b94eb6b9
fix error: concurrent map writes when add or remove cluster node
2022-06-28 10:34:59 +08:00
chrislu
4b1f48a399
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-27 12:40:50 -07:00
chrislu
4a65159250
fix reading time
2022-06-27 12:40:47 -07:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files
...
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
2022-06-27 20:19:21 +02:00
Chris Lu
4be406a095
Merge pull request #3249 from kmlebedev/issues_2589
...
avoid set currentMaster svc.local k8s discovery service domains
2022-06-27 11:05:07 -07:00
Konstantin Lebedev
9bff097b01
currentMaster redirected
2022-06-27 21:55:16 +05:00
Konstantin Lebedev
6c20a3b622
avoid set currentMaster k8s svc.local discoveruy service domains
...
https://github.com/chrislusf/seaweedfs/issues/2589
2022-06-27 21:47:05 +05:00
Lapshin Vitaliy
1a5981d583
fix test
2022-06-27 18:46:21 +03:00
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
2022-06-27 22:04:50 +08:00
Lapshin Vitaliy
d7c3493d15
Merge branch 'master' into bufix/validate-tags-on-copy
2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a
adjust log message
2022-06-26 23:12:16 -07:00
Chris Lu
fdd880b111
Merge pull request #3237 from shichanglin5/remove_query_collection
...
Remove the collection query param of s3api and let the collection be …
2022-06-26 23:02:11 -07:00
石昌林
15bfc60a88
Remove the collection query param of s3api and let the collection be determined by the filer
2022-06-27 13:47:53 +08:00
yulai.li
af23e63e3f
Improve filer command help, add supported filer store list
2022-06-27 12:09:16 +08:00
chrislu
40a9634c3d
3.13
2022-06-26 20:13:52 -07:00
Chris Lu
cc01f71752
Merge pull request #3233 from blacktear23/filerstore-tikv
...
Add TiKV Filerstore back
2022-06-26 20:12:18 -07:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
chrislu
9dfea44aeb
fix unit
2022-06-26 12:21:38 -07:00
chrislu
3142e77eca
mount: stats report physical size instead of logical size
2022-06-26 12:14:34 -07:00
yulai.li
13159ad0d8
Merge branch 'master' into filerstore-tikv
2022-06-26 22:44:01 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
chrislu
f77eda4b4f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-25 09:22:52 -07:00
chrislu
3ce46c3566
fix entry full url lookup
2022-06-25 09:22:49 -07:00
Chris Lu
dc59ccd110
Merge pull request #3228 from shichanglin5/fix_volumeNotFound
...
When the connection with the leader is disconnected, the vidMap shoul…
2022-06-24 23:06:13 -07:00
石昌林
f1ea906c13
Code comment optimization
2022-06-25 11:56:09 +08:00
石昌林
0c8e033e68
When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader
2022-06-24 17:41:46 +08:00
chrislu
9c517d2b35
masterclient: fallback to directly querying master in case of missing volume id location
2022-06-24 02:08:57 -07:00
Chris Lu
280e33092c
Merge pull request #3226 from kmlebedev/advancedtls
...
Advancedtls
2022-06-24 01:34:58 -07:00
chrislu
509a9047db
test compact map with snowflake sequencer
2022-06-23 21:45:51 -07:00
chrislu
217d7dc377
verify seq is always unique
2022-06-23 21:22:05 -07:00
Konstantin Lebedev
2e782dfdcb
rm defer comments
2022-06-24 01:50:13 +05:00
Konstantin Lebedev
7efaafb0ee
fix Authenticate
2022-06-24 01:42:04 +05:00
Konstantin Lebedev
b0aa51d7ef
enable require client cert
2022-06-24 00:29:23 +05:00
Konstantin Lebedev
ea7cdb8b0e
seperate option
2022-06-24 00:26:56 +05:00
Konstantin Lebedev
5b388ed6c1
initial advancedtls
2022-06-23 23:32:15 +05:00
chrislu
0c13fc873c
fix compilation
2022-06-23 00:45:11 -07:00
chrislu
96496d5286
master: broadcast new volume locations to clients to avoid possible racing condition
...
fix https://github.com/chrislusf/seaweedfs/issues/3220
2022-06-23 00:41:33 -07:00
chrislu
52c44d646e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-22 00:34:27 -07:00
chrislu
530da0cc89
remove unused code
2022-06-22 00:34:25 -07:00
famosss
8a6e7a7df8
Merge branch 'chrislusf:master' into master
2022-06-22 09:58:56 +08:00
zzq09494
9df5ad5309
fix: vacuum create a lot of connections quickly
2022-06-22 09:57:22 +08:00
Chris Lu
d652ce1a36
Merge pull request #3216 from kmlebedev/mk_raft_basedir
...
avoid no such raft date directory
2022-06-21 12:21:30 -07:00
chrislu
6b3beb709d
fix format
2022-06-21 12:21:02 -07:00
Konstantin Lebedev
c1c8dad677
avoid no such raft date directory
...
https://github.com/chrislusf/seaweedfs/issues/3214
2022-06-21 13:47:51 +05:00
chrislu
4dc27e1ed5
3.12
2022-06-20 19:07:00 -07:00
chrislu
492da3dbce
master: put metadata under instance specific folder
2022-06-20 19:04:49 -07:00
chrislu
41dfe27102
Revert "remove max connection age"
...
This reverts commit b9b684194f
2022-06-20 14:49:39 -07:00
chrislu
2295766e2c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56
remote storage: stop supporting hdfs as a remote storage
2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page
2022-06-20 22:36:07 +02:00
chrislu
bf5a5abfb1
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 13:18:11 -07:00
chrislu
69a2e503ab
non-recursive directory deletion
...
fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.
2022-06-20 13:18:08 -07:00
chrislu
76f26c4b99
mount: delete target entry first during move
2022-06-20 13:15:02 -07:00
Chris Lu
460c7cf68e
Merge pull request #3203 from shichanglin5/volume_copy_preallocate
2022-06-20 07:41:45 -07:00
石昌林
81f7f08708
Determine whether to preallocate according to the master configuration before executing copy volume
2022-06-20 21:12:44 +08:00
石昌林
be5c901bea
Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems
2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48
validate tags on copy object and add regex for validating tags
2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode
...
AbortMultipartUploadHandler should return 204 instead of 200
2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0
AbortMultipartUploadHandler should return 204 instead of 200
...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
2022-06-20 13:26:22 +08:00
石昌林
9e036df356
remove go.uber.org/atomic
2022-06-20 12:35:29 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
...
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
...
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4
fix path bug on windows
...
fix usage
2022-06-16 17:24:56 +08:00
chrislu
aaa9938353
Revert "adjust conditions"
...
This reverts commit e024586ff1
2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97
remove -output, add -dir,-name for more flexibility
2022-06-16 16:07:32 +08:00
chrislu
e024586ff1
adjust conditions
2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
...
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e
add condition when inFlightUploadDataLimitCond signal
2022-06-16 14:07:11 +08:00
zzq09494
62f74f5e3c
add bucket label to s3 prometheus metrics
2022-06-16 13:21:25 +08:00
guol-fnst
4bc8792643
support specific version of weed
2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29
remove unused function
...
remove log when weed is up to date
add check whether weed is full version
2022-06-16 11:40:20 +08:00
liubaojiang
3076ac101e
move vs.concurrentUploadLimit != 0 out of the lock
2022-06-16 09:58:44 +08:00
Chris Lu
097a12dfb4
Merge pull request #3187 from ningfdx/remote
2022-06-15 06:34:32 -07:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req
2022-06-15 06:33:33 -07:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
ningfd
338705f375
fix(wdclient): GetLocations return
2022-06-15 19:20:13 +08:00
Konstantin Lebedev
31a929f7f2
logging processRangeRequest errors
2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149
2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37
Fix: provide filer.ui.deleteDir options for server command
2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update
2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555
fix minor bug
2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6
fix(filer.sync): modify clientName format : from -> to
2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87
Change ui.deleteDir default to true
2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b
feat(filer.sync): add metricsServer in filer.sync.
...
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
2022-06-15 11:33:18 +08:00
chrislu
d4ef06cdcf
3.11
2022-06-14 12:14:52 -07:00
chrislu
b347b2fb54
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-14 11:16:51 -07:00
chrislu
36d2756978
volume: avoid writing too much data for range requests in large files
...
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
yulai.li
28cda854eb
Add filer command line parameter to let Filer UI show/hide directory delete button
2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea
feat(filer.sync): add offset to path.
2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0
add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate "
2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372
add update feature
2022-06-14 19:01:57 +08:00
Chris Lu
b22ca85fbb
Merge pull request #3169 from shichanglin5/fix_auth
...
fix: When there is no access permission configured before startup, th…
2022-06-12 23:40:42 -07:00
shichanglin5
f01dd27752
fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup
2022-06-13 13:23:26 +08:00
chrislu
e437ec5a68
3.10
2022-06-12 19:30:27 -07:00
chrislu
d12f431d98
collect volume server status
2022-06-12 11:56:23 -07:00
chrislu
444ac21050
go fmt
2022-06-11 09:51:11 -07:00
chrislu
a74e926063
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-11 09:51:02 -07:00
chrislu
61b8c9c361
remote object store gateway: disable tagging for backblaze
2022-06-11 09:50:59 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix
2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb
Bug Fix: force reload filer page when upload/delete/rename/create directory
2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64
fix typo
2022-06-10 22:03:11 +08:00
Chris Lu
36757d28a8
Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch
...
stats master_replica_placement_mismatch
2022-06-10 06:44:16 -07:00
chrislu
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
...
This reverts commit 37da68931900d53c34c4
2022-06-10 06:38:17 -07:00
Konstantin Lebedev
fb57aa431c
stats master_replica_placement_mismatch
2022-06-10 15:30:40 +05:00
shichanglin5
a63559ee6c
fix: fix command prompt information
2022-06-10 12:46:10 +08:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID
...
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
...
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
2022-06-09 20:41:16 +08:00
zhihao.qu
cd5cca36a4
feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges
2022-06-09 10:53:19 +08:00
Chris Lu
4a5135961f
Merge pull request #3154 from kmlebedev/s3_test_invalid_range
2022-06-08 01:45:40 -07:00
Konstantin Lebedev
49f058da85
fix s3 tests:
...
ranged_request_invalid_range
ranged_request_empty_object
2022-06-08 13:25:13 +05:00
chrislu
8a1166c021
auto remove previous leftover socket
2022-06-08 00:32:09 -07:00
chrislu
633a4aed18
filer: fix customizable local socket file name
...
fix https://github.com/chrislusf/seaweedfs/issues/3147
2022-06-08 00:15:28 -07:00
chrislu
acbc0286e2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-07 22:33:38 -07:00
chrislu
ebd6097e6f
fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074
...
related to https://github.com/chrislusf/seaweedfs/pull/3074
2022-06-07 22:33:35 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f
fix s3 tests
...
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
chrislu
78be3b0f6c
fix compilation
2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
chrislu
cbf46de5f4
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-06 00:15:40 -07:00
chrislu
8402af0882
filer: detect ttl based on fs configure setting
2022-06-06 00:15:37 -07:00
ningfd
f32142f6f5
add disableXAttr in mount option
2022-06-06 14:09:01 +08:00
chrislu
4a046e4de7
3.09
2022-06-05 19:00:24 -07:00
chrislu
11f1e7996b
fix test
2022-06-05 18:50:41 -07:00
chrislu
d65bb2c6df
mount: file handle locks entry better
...
related to https://github.com/chrislusf/seaweedfs/issues/2952
2022-06-05 18:15:06 -07:00
chrislu
746092a60b
add a comment
2022-06-05 16:44:07 -07:00
chrislu
056c480eb0
volume: checksum remove the hashing step in Value()
2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc
stream read large files
2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f
remove unused logic
2022-06-04 19:11:06 -07:00
chrislu
685643d506
needle read into a writer
2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23
add needle reading in chunks
2022-06-04 18:15:39 -07:00
chrislu
18b29f709c
refactoring, split file
2022-06-03 23:31:14 -07:00
chrislu
39c3cb04af
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-03 08:45:32 -07:00
chrislu
f97acdd489
volume.fix.replication fix retry logic
...
fix https://github.com/chrislusf/seaweedfs/issues/3136
2022-06-03 08:45:29 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d
fix show empty dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-06-01 20:10:52 +05:00
chrislu
6793bc853c
help message when in simulation mode
2022-05-31 14:48:46 -07:00
chrislu
89948a373b
fix error reporting for "Need to a hdd disk type"
...
related to https://github.com/chrislusf/seaweedfs/issues/3128
2022-05-31 12:43:55 -07:00
chrislu
8a49240d64
3.08
2022-05-31 11:57:41 -07:00
chrislu
376b005ad3
fix wrong logic
2022-05-31 11:55:58 -07:00
chrislu
fbd99d53c1
3.07
2022-05-31 00:36:24 -07:00
chrislu
27732ecfa4
move s3 related constants from package http to s3_constants
2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 22:47:29 -07:00
chrislu
596c3860ca
use final destination to resolve fs configuration
...
related to https://github.com/chrislusf/seaweedfs/issues/3075
2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers
...
Filer bootstrap from peers
2022-05-30 21:38:31 -07:00
chrislu
6adc42147f
fresh filer store bootstrap from the oldest peer
2022-05-30 21:27:48 -07:00
chrislu
490e0850bf
filer.meta.tail add one example of using untilTimeAgo
2022-05-30 20:24:59 -07:00
liubaojiang
f0ee3e6f21
reduce the scope of inFlightUploadDataLimitCond lock
2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3
refactor
2022-05-30 16:28:36 -07:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
968ca95b49
filer.meta.tail: support untilTimeAgo for a range
2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5
stop when in memory log is done
2022-05-30 15:25:21 -07:00
chrislu
aece35a64f
stop when on disk log is done
2022-05-30 15:20:51 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
8902fa6ff6
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 13:28:32 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud
...
Create folders in s3 via nextcloud
2022-05-30 13:10:10 -07:00
chrislu
5b9347c938
typo
2022-05-30 12:37:41 -07:00
Konstantin Lebedev
af562e1a20
rm log info
2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c
list self dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961
show fake dirs
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46
s3: keep auth enabled in case identities are set to empty
...
fix https://github.com/chrislusf/seaweedfs/issues/3084
2022-05-29 19:00:03 -07:00
chrislu
81d6159290
volume: report error if a volume has nil data backend
...
fix https://github.com/chrislusf/seaweedfs/issues/3105
2022-05-29 16:59:30 -07:00
chrislu
ca16fbf0ef
shell: cluster.ps display filer group
2022-05-29 16:37:14 -07:00
chrislu
9219664505
reset sync offset if peer filer resets
...
fix https://github.com/chrislusf/seaweedfs/issues/3089
2022-05-29 13:11:24 -07:00
Konstantin Lebedev
54356211c2
list empty folder
2022-05-27 13:17:18 +05:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator.
2022-05-25 01:11:02 -07:00
chrislu
5b8b022985
remove unused parameter
2022-05-24 18:52:04 -07:00
Robert Coelho
0e6e72d462
mount: ReadDir return EIO on EnsureVisited err
2022-05-24 13:02:59 -04:00
Robert Coelho
1fabbe8a25
mount: cleanup ReadDir rewrite's branches to not assume offsets
2022-05-24 12:48:32 -04:00
Robert Coelho
cb422d96f7
mount: rewrite ReadDir to respect input.Offset to fix partial results
2022-05-24 12:16:58 -04:00
Robert Coelho
05602167f4
mount: do not mark meta_cache when there is an err
2022-05-24 12:13:15 -04:00
chrislu
7e25a2d416
reduce busy waiting when reading metadata logs
2022-05-24 00:23:53 -07:00
chrislu
2f846777bb
3.06
2022-05-23 01:21:48 -07:00
chrislu
a5cdcc7e63
fix reading files larger than 20GB due to manifest resolving bug
2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd
fix reading files larger than 20GB due to manifest resolving bug
...
this is introduced since Mar 08 2022
2022-05-23 01:14:56 -07:00
chrislu
acc318e12b
3.05
2022-05-22 21:43:59 -07:00
liubaojiang
076e48a676
add inflight upload data wait timeout
2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e
add inFlightUploadDataLimitCond signal
2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c
exclude replication from the concurrentUploadLimitMB
2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync
2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9
hotfix_fsync fix fsync
2022-05-19 23:17:17 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
...
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879
revert changes of pb file
...
orignal protoc-gen-go-grpc verion is 1.1.0
2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd
just exit in case of duplicated volume directories were loaded
2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775
rename UUID file
...
fix typo
move locationUUID into DiskLocation
2022-05-17 15:41:47 +08:00
Chris Lu
72e7dcde51
Merge pull request #3060 from natmaka/master
2022-05-16 07:42:58 -07:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around
2022-05-16 22:11:33 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
Konstantin Lebedev
44f53ceda6
fix collectionIsMismatch charset
2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2
fix skip loop
2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c
avoid delete volume replica if collection mismatch
2022-05-16 13:07:05 +05:00
chrislu
9ff0d99002
3.04
2022-05-15 21:32:21 -07:00
chrislu
866fed1eab
avoid grpc name conflict
...
fix https://github.com/chrislusf/seaweedfs/issues/3055
2022-05-15 21:31:29 -07:00
chrislu
cd2d10118c
3.03
2022-05-15 18:17:07 -07:00
chrislu
271b5aed96
shell: volume.fsck add a note for -reallyDeleteFromVolume option
2022-05-15 11:07:04 -07:00
chrislu
4076d74813
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-15 00:43:39 -07:00
chrislu
29198720f2
s3: add grpc server to accept configuration changes
2022-05-15 00:43:37 -07:00
Chris Lu
ea4918394c
Merge pull request #3047 from shichanglin5/master
...
fix the problem of metadata and tagging loss when files are copied
2022-05-13 22:52:48 -07:00
shichanglin5
688d55488c
test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go
2022-05-14 10:40:29 +08:00
chrislu
735038b2c1
backup do not need to use preallocation
...
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
shichanglin5
1166dead00
fix the problem of metadata and tagging loss when files are copied
...
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
2022-05-13 19:46:20 +08:00
Konstantin Lebedev
d12a423aa0
refactor
...
add some tests
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9
fixes from comments
...
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-11 21:50:09 +05:00
chrislu
bff1ccc1de
fix compilation
2022-05-11 00:52:15 -07:00
chrislu
76da688176
adjust log level
2022-05-11 00:45:19 -07:00
chrislu
3e20336bde
fix atomic add int64
...
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
2022-05-11 00:44:35 -07:00
chrislu
a85720f0f6
remove deprecated function
2022-05-10 00:02:23 -07:00
chrislu
25559eeb92
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-10 00:01:46 -07:00
chrislu
791c29ae93
remove deprecated function
...
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
2022-05-10 00:01:43 -07:00
qiang.zeng
06cb337b82
fix typo
2022-05-09 14:21:16 +08:00
chrislu
a4ca3ed1f0
3.02
2022-05-08 22:28:32 -07:00
chrislu
8bc16f1149
rename
2022-05-06 23:38:22 -07:00
chrislu
139e039c44
filer.sync: pass attributes for mount
...
fix https://github.com/chrislusf/seaweedfs/issues/3012
2022-05-06 03:54:12 -07:00
chrislu
10207a48a4
how to setup local test env
2022-05-06 02:25:56 -07:00
chrislu
db33b03735
add comments
2022-05-06 02:19:47 -07:00
chrislu
e350ba8896
larger test range
2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5
a bit optimization
2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7
filer: etcd fix pagination
...
fix https://github.com/chrislusf/seaweedfs/issues/3016
2022-05-06 01:51:19 -07:00
Chris Lu
a69bf53405
Merge pull request #2996 from kmlebedev/ydb
...
ydb filer store
2022-05-05 15:34:37 -07:00
Konstantin Lebedev
e6d2cb5911
ydb add empty doc file
2022-05-05 11:12:14 +05:00
chrislu
ef6c6c450e
avoid fatal error if port is already in use
2022-05-04 14:55:14 -07:00
chrislu
f7366a9668
skip unix socket mode for windows
...
fix https://github.com/chrislusf/seaweedfs/issues/3013
2022-05-04 10:14:34 -07:00
Konstantin Lebedev
f58adaab25
build tag ydb since the growth of the binary is 5 mb
2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf
add options to scaffold
2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af
refactor
2022-05-03 20:16:00 +05:00
chrislu
f17cd0d5cd
return false if not found
...
fix https://github.com/chrislusf/seaweedfs/issues/3011
2022-05-03 07:18:34 -07:00
chrislu
b201edb9df
fix wrong assignment
2022-05-03 07:12:33 -07:00
Konstantin Lebedev
04d5dff6bb
add support native ttl
2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb
enable query cache policy instead of prepare
2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3
fix scanNamed
2022-05-03 15:18:28 +05:00
chrislu
9271866d1e
fix segmentation violation
...
fix https://github.com/chrislusf/seaweedfs/issues/3000
2022-05-02 23:56:11 -07:00
Konstantin Lebedev
2d27afb515
Merge branch 'new_master' into ydb
2022-05-03 10:47:39 +05:00
chrislu
76148ce0f7
use storage_class for backend configuration
...
fix https://github.com/chrislusf/seaweedfs/issues/3008
2022-05-02 12:20:01 -07:00
Konstantin Lebedev
d8925b4e83
Merge branch 'new_master' into ydb
...
# Conflicts:
# go.mod
# go.sum
2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5
fix queries
2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48
fix createDB
2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63
ydb do Tx or DB
2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3
ydb BucketAware interface
2022-05-02 12:42:20 +05:00
chrislu
73961e24d8
3.01
2022-05-01 23:33:45 -07:00
chrislu
00c1dfec4f
go fmt
2022-05-01 23:16:29 -07:00
chrislu
ce1e8e58ae
fix test
2022-05-01 23:15:58 -07:00
chrislu
2c0fdf3693
fix tests
2022-05-01 23:14:04 -07:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4
ydb-go-sdk move to v3
2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3
refactor use const CountEntryChunksForGzip
2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37
ydb kv interface
2022-05-01 21:20:37 +05:00
chrislu
4bd6bea429
fix test
2022-05-01 09:02:01 -07:00
Konstantin Lebedev
6a052f6ff2
ydb Sql interface
2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c
ydb initial
...
https://github.com/chrislusf/seaweedfs/issues/2942
2022-05-01 17:28:58 +05:00
chrislu
5d081d5c3c
skip if already copied to remote
...
fix https://github.com/chrislusf/seaweedfs/issues/2993
2022-04-30 20:03:04 -07:00
chrislu
1aae7a3f1b
volume.tier.upload progress starts negative #2992
...
fix https://github.com/chrislusf/seaweedfs/issues/2992
2022-04-30 18:10:01 -07:00
chrislu
ea8a7a5584
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-30 17:36:45 -07:00
chrislu
192983b464
s3 backend support customizing storage class
2022-04-30 17:36:40 -07:00
Konstantin Lebedev
03aec9ce7e
avoid panic if hashicorp raft not initialized
2022-04-30 13:56:35 +05:00
guosj
1a5de36257
fix return value
2022-04-29 11:19:36 +08:00
Chris Lu
4e97b6146a
Merge pull request #2982 from kmlebedev/issues/2981
...
avoid empty listMultipartUploads response
2022-04-27 17:21:44 -07:00
chrislu
b2a6111090
skip ipv6 all interfaces and localhost
...
fix https://github.com/chrislusf/seaweedfs/issues/2983
2022-04-27 17:18:09 -07:00
Konstantin Lebedev
306cf70c4a
avoid empty listMultipartUploads response
2022-04-27 19:27:44 +05:00
chrislu
70e5a1b632
volume close should wait for committing compaction
2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression
...
waite volume being closed during compression idx
2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0
use two flags: v.isCompacting and v.isCommitCompacting
2022-04-26 23:28:34 -07:00
Chris Lu
b557faf175
Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid
...
volume.check.disk add param volumeId
2022-04-26 22:09:06 -07:00
Chris Lu
eb49bd5638
Merge pull request #2970 from kmlebedev/vlm_fsck_forcePurging
...
delete missing data from volumes in one replica
2022-04-26 22:08:34 -07:00
chrislu
94f824e1ce
volume: sync to disk before copying volume files
...
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039
waite volume being closed during compression idx
2022-04-26 13:40:42 +05:00
Konstantin Lebedev
d4343ab7da
forcePurging desc
2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f
change forcePurging to a pointer
2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2
delete missing data from volumes in one replica
2022-04-25 22:59:46 +05:00
Konstantin Lebedev
67476d830a
volume.check.disk add param volumeId
2022-04-25 18:27:14 +05:00
chrislu
2aef5b8b32
3.00
2022-04-24 21:46:03 -07:00
Chris Lu
184e982898
Merge pull request #2957 from guo-sj/handle_implicit_username
...
Handle implicit username
2022-04-24 09:07:48 -07:00
guosj
8fd7b24b80
remove redundant logs & add unit test
2022-04-24 16:09:10 +08:00
guol-fnst
8b2552f944
fix http response error code
2022-04-24 11:07:27 +08:00
guosj
09befe871f
Merge branch 'master' into handle_implicit_username
2022-04-23 09:48:25 +08:00
guosj
893a0eea57
add more checks and comments
2022-04-22 22:47:11 +08:00
chrislu
ad01c63b84
conditionally skip hdfs related code
2022-04-21 01:43:01 -07:00
chrislu
4aa39ef33f
conditionally compile sqlite
2022-04-21 01:22:38 -07:00
chrislu
3885374edf
conditionally build elastic, gocdk to reduce binary size
2022-04-21 01:10:46 -07:00
chrislu
6994e88192
fix typo
2022-04-21 00:02:18 -07:00
chrislu
cb1e7aa27d
fix build
2022-04-20 16:21:56 -07:00
a
1d6a9e66b6
Merge branch 'master' into a
2022-04-20 14:01:42 -07:00
chrislu
bdc4d67de8
Revert "Merge pull request #2944 from guo-sj/handle_implicit_username"
...
This reverts commit bc96e73b8d88a669dd19
2022-04-20 09:40:41 -07:00
chrislu
3aa4dc6ad5
Revert "fix index out of range"
...
This reverts commit 4792e07228
2022-04-20 09:40:24 -07:00
chrislu
4792e07228
fix index out of range
2022-04-20 09:32:17 -07:00
chrislu
f3fdfb07b8
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-19 22:58:59 -07:00
chrislu
8e9ad7db5a
fix bug deleting volume or unmount volume if a volume server has multiple directories
2022-04-19 22:58:57 -07:00
Chris Lu
bc96e73b8d
Merge pull request #2944 from guo-sj/handle_implicit_username
...
handle implicit username
2022-04-19 21:42:21 -07:00
guosj
a23629b2fe
handle implicit username
2022-04-20 10:12:33 +08:00
Chris Lu
88a669dd19
Merge pull request #2858 from kmlebedev/fsck_replicas
...
fsck replicas
2022-04-19 00:17:43 -07:00
Chris Lu
f76c318a1f
Merge pull request #2940 from kmlebedev/volume.list.grep
...
volume.list show only readonly, collectionPattern and volumeId
2022-04-19 00:17:32 -07:00
Konstantin Lebedev
2364fab927
volume.list show only readonly, collectionPattern and volumeId
2022-04-19 11:44:41 +05:00
guosj
94ad557d97
add missing return
2022-04-19 14:16:32 +08:00
Konstantin Lebedev
36c5a59ed8
add help
2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
Chris Lu
97c6e0efb1
Merge pull request #2931 from kmlebedev/fix_nil_dataBackend
...
avoid invalid memory address or nil pointer dereference
2022-04-18 00:13:26 -07:00
Konstantin Lebedev
9438738693
avoid invalid memory address or nil pointer dereference
2022-04-18 12:10:22 +05:00
chrislu
a69f0b62c2
start filer local socket only on non windows
...
fix https://github.com/chrislusf/seaweedfs/issues/2930
2022-04-18 00:08:31 -07:00
chrislu
9ae66f55fc
2.99
2022-04-17 23:01:42 -07:00
chrislu
4524ceae6b
less verbose logs
...
fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085
2022-04-17 22:55:26 -07:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc
2022-04-18 10:39:29 +08:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
chrislu
b597baf488
mount: set file write time correct to work well with rsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2867
2022-04-16 22:56:45 -07:00
chrislu
460d56d283
shell: cluster.check prints out clock delta and network latency
2022-04-16 13:24:17 -07:00
chrislu
b4be56bb3b
add timing info during ping operation
2022-04-16 12:45:49 -07:00
chrislu
1384529eb7
Fix filer.backup deletes files in backup folder in incremental mode
...
fix https://github.com/chrislusf/seaweedfs/issues/2919
2022-04-14 13:35:01 -07:00
a
846858fb43
merge master
2022-04-12 14:42:29 -07:00
chrislu
1f03fcccb1
fix nil in cluster_check shell command
...
fix https://github.com/chrislusf/seaweedfs/issues/2905
2022-04-12 08:47:27 -07:00
Chris Lu
dd13764a5c
Merge pull request #2900 from kmlebedev/fix_cleanupUploads
...
avoid breaking loop in cleanupUploads if error is empty
2022-04-12 02:03:31 -07:00
Konstantin Lebedev
76b1c5ce67
cleanupUploads use jwt token
2022-04-12 13:46:11 +05:00
Chris Lu
72db181d68
Merge pull request #2901 from guol-fnst/fix_multiupload
...
check object name and uploadID when processing multiupload
2022-04-11 23:38:12 -07:00
guol-fnst
44d810d163
rename functions and remove uncessary check
2022-04-12 13:40:53 +08:00
guol-fnst
180aa88a92
check uploadid using object name hash string
2022-04-12 11:04:38 +08:00
guol-fnst
2232cfb5b7
Check object name and uploadID when processing multipart uploading
2022-04-11 19:53:44 +08:00
Konstantin Lebedev
2dfbe210e0
avoid breaking loop in cleanupUploads if error is empty
2022-04-11 11:16:20 +05:00
Chris Lu
a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft
...
hashicorp raft
2022-04-10 23:00:05 -07:00
Konstantin Lebedev
931cb9e581
use "cluster.raft.{ps,add,remove}"
2022-04-11 10:50:01 +05:00
Konstantin Lebedev
d2fe0fae33
use the full ServerAddress as the identifier
2022-04-11 10:44:44 +05:00
chrislu
c6ec5269f4
2.98
2022-04-10 18:56:40 -07:00
chrislu
261ba0208e
lock-protected setting entry
2022-04-09 22:52:59 -07:00
Chris Lu
9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style
...
Change filer UI icon from picture to icon font.
2022-04-08 00:50:32 -07:00
yulai.li
ceb058ddd9
Change filer UI icon from picture to icon font.
2022-04-08 15:07:56 +08:00
chrislu
7f26676f57
adjust td width
2022-04-07 22:34:32 -07:00
yulai.li
dff63e9024
Make operation buttons show when table row hovered
2022-04-08 13:10:12 +08:00
chrislu
4dba102137
use icon instead of button text
2022-04-07 11:15:01 -07:00
Chris Lu
bbc95dac47
Merge pull request #2885 from binbinshi/master
...
fix: master lose some volumes
2022-04-07 10:16:19 -07:00
Konstantin Lebedev
104ea7b029
master ui add raft stats and Max Volume Id
2022-04-07 20:52:01 +05:00
a
41d396edc4
Merge branch 'master' into a
2022-04-07 08:22:59 -07:00
Konstantin Lebedev
a2fdb3e277
fix master ui
2022-04-07 19:37:40 +05:00
Konstantin Lebedev
35bc67f030
fix conflicts
2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
yulai.li
11a9f993a8
Make filer-ui bootstrap style
2022-04-07 20:17:00 +08:00
Konstantin Lebedev
7ff248d5cd
refactor OnPeerUpdate
2022-04-07 16:23:22 +05:00
Konstantin Lebedev
85d80fd36d
fix removing old raft server
2022-04-07 15:31:37 +05:00
chrislu
abe3cc6df2
filer UI touch up
2022-04-07 01:25:55 -07:00
Chris Lu
869d23b099
Merge pull request #2884 from blacktear23/filer-ui-new-features
...
Filer UI new features
2022-04-07 00:58:56 -07:00
chrislu
86ed27f602
mount: remove leftover socket file
2022-04-07 00:33:13 -07:00
yulai.li
2347c21cdd
Tune filer UI add rename feature
2022-04-07 15:19:41 +08:00
yulai.li
2454020a92
Add upload progress support
2022-04-07 15:19:37 +08:00
yulai.li
4f87ee7755
Add create directory and delete web UI features for filer
2022-04-07 15:19:33 +08:00
shibinbin
c20e1edd99
fix: master lose some volumes
2022-04-07 15:18:28 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a
2022-04-06 18:45:55 -05:00
chrislu
3ab2c0e5c0
avoid possible nil attributes
2022-04-06 10:11:11 -07:00
chrislu
9d44b89c21
Revert "avoid possible nil attributes"
...
This reverts commit 0a6703c7f7
2022-04-06 10:01:45 -07:00
chrislu
0a6703c7f7
avoid possible nil attributes
2022-04-06 09:58:48 -07:00
Konstantin Lebedev
14a2cc83bf
raft update peers via OnPeerUpdate
2022-04-06 21:17:04 +05:00
Konstantin Lebedev
357aa818fe
add raft shell cmds
2022-04-06 15:23:53 +05:00
chrislu
3d229fe45c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-06 00:28:51 -07:00
chrislu
79b8e6a8c3
add a place holder for later merge consecutive chunks
2022-04-06 00:28:48 -07:00
Konstantin Lebedev
c1aeded2dd
force raft bootstrap
...
avoid err bootstrap only works on new clusters
2022-04-06 12:18:43 +05:00
Chris Lu
49ecb8d11c
Merge pull request #2879 from leyou240/master
...
remove Redundant type conversion and use strings.TrimSuffix to enhance readability
2022-04-06 00:02:19 -07:00
justin
d02f13c2d1
remove Redundant type conversion and use strings.TrimSuffix to enhance readability
2022-04-06 14:58:09 +08:00
root
f15a737a0a
Add "Location:" in response when creating bucket
...
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html "
2022-04-06 13:56:27 +08:00
chrislu
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
chrislu
8b3d76b24d
prevent concurrent access during shutdown
2022-04-05 10:53:36 -07:00
chrislu
bb4beebce3
prevent nil
2022-04-05 10:49:17 -07:00
Konstantin Lebedev
0e796a5582
rm set NoSnapshotRestoreOnStart
2022-04-05 18:28:42 +05:00
Konstantin Lebedev
68f11b9687
sleep bootstraping
2022-04-05 15:42:19 +05:00
Konstantin Lebedev
1ae7f509f5
LocalID from raw string
2022-04-05 13:50:39 +05:00
Konstantin Lebedev
89d32a0670
fix glog Info
2022-04-05 13:43:34 +05:00
Konstantin Lebedev
17c6e8e39f
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# go.mod
# go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae
auto bootstraping and update peers
2022-04-05 13:17:53 +05:00
chrislu
4aae87f405
check missing hdd disk type
2022-04-04 14:48:00 -07:00
Konstantin Lebedev
622297f1a7
add stats raft handler
2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3
fix cluster status
2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
d310711de0
avoid possible deadlock: move metadata log flush channel out of lock scope
2022-04-03 21:53:30 -07:00
chrislu
77a7d7253f
2.97
2022-04-03 19:08:01 -07:00
chrislu
e8d7bb42e2
avoid concurrent access to map object
...
fix https://github.com/chrislusf/seaweedfs/issues/2866
2022-04-03 01:08:25 -07:00
chrislu
6a2bcd03aa
configure mount quota
2022-04-02 21:34:26 -07:00
chrislu
958f880b70
mount: add grpc method to adjust quota
2022-04-02 15:14:37 -07:00
chrislu
a8eefda452
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-01 20:25:37 -07:00
chrislu
105578a2f2
skip pinging self for master and volume server
2022-04-01 20:25:35 -07:00
root
b7e34016d4
fix error response in case of wrong expire time format
2022-04-02 11:01:33 +08:00
chrislu
4ecba915f3
add check between peers
2022-04-01 17:40:25 -07:00
chrislu
daca2d22a5
use original server address string as map key
2022-04-01 17:34:42 -07:00
chrislu
4b5c0e3fa9
check cluster connectivities
2022-04-01 17:27:49 -07:00
chrislu
6d55716fc3
use constants
2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4
master supports grpc ping
2022-04-01 16:50:58 -07:00
chrislu
743ad690b6
filer supports grpc ping
2022-04-01 16:44:58 -07:00
chrislu
2305508b65
refactor: separate into two files
2022-04-01 16:40:49 -07:00
chrislu
800cbc004c
volume server adds ping function
2022-04-01 16:37:06 -07:00
a
7e92517571
change user and pass to username and password
2022-04-01 14:09:25 -05:00
Konstantin Lebedev
7f1383a41e
findExtraChunksInVolumeServers in consideration of replication
2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0
fix collect filer files
2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7
skip new entities
2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0
fsck replicas
2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02
collecting only bucket volumes
2022-03-31 13:35:58 +05:00
Konstantin Lebedev
691ed50957
use constant for ErrBucketNotEmpty
2022-03-31 08:44:48 +05:00
Konstantin Lebedev
b6a2b43efe
allowDeleteBucketNotEmpty
2022-03-30 22:46:13 +05:00
Konstantin Lebedev
a8b15c676c
s3 test bucket delete nonempty
2022-03-30 21:06:26 +05:00
a
549e341c6f
resolv
2022-03-30 06:31:19 -05:00
Chris Lu
ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit
...
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
2022-03-30 01:27:13 -07:00
Konstantin Lebedev
6317f6a8a9
fix Description
2022-03-30 13:04:15 +05:00
Chris Lu
8732cc24c8
Merge pull request #2849 from guo-sj/update_user
...
Add AWS IAM update user API and its test case
2022-03-29 22:36:26 -07:00
Konstantin Lebedev
993283bb1d
revert sort.Search
2022-03-29 19:53:12 +05:00
Konstantin Lebedev
5b90a39954
fix s3 tests:
...
multipart_upload_incorrect_etag
multipart_resend_first_finishes_last
2022-03-29 19:21:09 +05:00
Konstantin Lebedev
9231971c88
fix test_s3.test_multi_object_delete_key_limit and
...
test_multi_objectv2_delete_key_limit
2022-03-29 16:55:09 +05:00
guosj
ed07e76f0e
add test function
2022-03-29 17:21:59 +08:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517
fix permission mkdir snapshot
...
avoid open file operation not permitted
2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error
...
Fix delete all "Seaweed-" prefixed tagging error
2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc
update code according to PR's comment
2022-03-28 15:23:31 +08:00
guosj
02076f01a6
fix tagging error
2022-03-28 14:50:48 +08:00
justin
d51a724101
fix: encode small chunk return error maybe have some bug.
2022-03-28 13:11:24 +08:00
chrislu
0490ee87ef
2.96
2022-03-27 16:11:17 -07:00
chrislu
86d4c52d6e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-26 13:33:20 -07:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1
refactor
2022-03-26 13:13:19 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545
...
avoid connect to the old filler address
2022-03-26 13:02:49 -07:00
chrislu
cae998eda1
Revert "Revert "remove duplicated metadata subscription in filer""
...
This reverts commit 34b743c481
2022-03-26 13:00:16 -07:00
chrislu
34b743c481
Revert "remove duplicated metadata subscription in filer"
...
This reverts commit 34742be029https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-26 12:33:45 -07:00
chrislu
fba1cfc2d6
simplify a bit
2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da
refactor
2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers
...
Fix remove deleted peers
2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26
fix remove deleted peers of raft server
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe
always clear previous log to avoid server is promotable
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request
...
Need to exit waiting if request is was canceled
2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf
exclude the replication from the concurrentUploadLimitMB
2022-03-24 13:54:42 +05:00
chrislu
9b04f17555
remove dead code
2022-03-23 23:18:32 -07:00
chrislu
0b5faef612
fix
2022-03-23 23:16:54 -07:00
chrislu
0562fceb99
volume: fail fast if too many concurrent requests, to avoid dead lock due to replication.
...
fix https://github.com/chrislusf/seaweedfs/issues/2755
2022-03-23 22:53:58 -07:00
Konstantin Lebedev
849a2c1f68
fix cmdFiler
2022-03-23 14:30:46 +05:00
Konstantin Lebedev
105bcfdbd4
set iam ip from filer
2022-03-23 13:45:51 +05:00
chrislu
e48764be75
s3: multipart upload verifies uploaded parts
2022-03-23 01:05:14 -07:00
chrislu
18ae8943ef
better clean up upload directory
2022-03-23 01:03:51 -07:00
chrislu
a0bad1c997
remove any go mod changes
...
This reverts commit 6c7f7d6887
2022-03-21 23:04:00 -07:00
chrislu
6c7f7d6887
Revert "Merge pull request #2782 from SadmiB/upstream"
...
This reverts commit a644b7236a349257f822
2022-03-21 23:00:50 -07:00
Chris Lu
0d692a6254
Merge branch 'master' into upstream
2022-03-21 11:02:04 -07:00
SadmiB
d12540c9f2
Add contabo api client
2022-03-21 17:16:49 +01:00
chrislu
4e2388e1b5
mount: set file size if it is only on remote gateway
2022-03-21 02:09:54 -07:00
chrislu
8f0410af2c
2.95
2022-03-21 01:47:03 -07:00
chrislu
ae558fa073
log reasons volumes became unwritable
2022-03-21 00:41:44 -07:00
chrislu
57c6eddd22
avoid possible deadlock if volume layout is used in some logs
2022-03-21 00:04:01 -07:00
chrislu
581c93d970
change to config_dir, to be consistent with "weed scaffold -config"
...
fix https://github.com/chrislusf/seaweedfs/issues/2753
2022-03-19 00:24:52 -07:00
chrislu
b75c6dcff5
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-19 00:22:52 -07:00
chrislu
3da2b83b38
Added a "-conf_dir" option to customize *.toml configuration file directory.
...
fix https://github.com/chrislusf/seaweedfs/issues/2753
2022-03-19 00:22:47 -07:00
elee
beb406bbbb
fix ls, onBucketDelete still not triggering
2022-03-18 22:21:57 -05:00
elee
411c0df3fe
switch to multi collection, change readme
2022-03-18 21:51:16 -05:00
zzq09494
9f16df736e
fix-filer: calculation error of the method skipCheckParentDirEntry
2022-03-19 09:15:42 +08:00
elee
1cea6c73d3
update readme
2022-03-18 00:34:19 -05:00
elee
25be96832a
update readme with index info
2022-03-18 00:33:08 -05:00
elee
06f23aa675
put in delete folder children query
2022-03-18 00:29:48 -05:00
elee
0701feeb17
added more to readme
2022-03-18 00:26:11 -05:00
elee
171c27ddf1
zz
2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb
update readme
2022-03-17 21:26:16 -05:00
elee
423ce57cde
prefix search, bucket implemented
2022-03-17 21:12:25 -05:00
chrislu
2824940ecf
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-17 16:54:30 -07:00
chrislu
f247cab5cd
skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1
2022-03-17 16:54:29 -07:00
Berck Nash
7ee38fa3a4
The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems:
...
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified. Instead, it bound to localhost which broke the default configuration. This change implements the documented behavior instead.
(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP. I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
2022-03-17 15:30:23 -06:00
elee
bf745bdccb
revise comment
2022-03-17 14:09:34 -05:00
elee
a0df993cef
cleanup, add a few comments
2022-03-17 14:08:36 -05:00
elee
94883e2fad
ok then
2022-03-17 05:23:12 -05:00
elee
f3ab6769e9
make start from indexing work
2022-03-17 05:09:13 -05:00
elee
31571dd96e
fix file listing
2022-03-17 05:07:40 -05:00
elee
b25d03340b
Merge branch 'a' of github.com:gfxlabs/seaweedfs into a
2022-03-17 04:51:59 -05:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
chrislu
6886167d5f
minor optimization
2022-03-17 00:03:28 -07:00
chrislu
7542fd7f4d
mount: optimize a bit when writing files
2022-03-17 00:02:38 -07:00
chrislu
4042fdf3bb
rename to skipCheckParentDir
...
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7
go fmt
2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3
fix: ut test error
2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3
filer: support uploading file without needEnsureParentDir
2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255
Revert "filer: support uploading file without needEnsureParentDir"
...
This reverts commit a93c4947ba
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba
filer: support uploading file without needEnsureParentDir
2022-03-17 10:18:23 +08:00
Berck Nash
9b14f0c81a
Add mTLS support for both master and volume http server.
2022-03-16 09:52:17 -06:00
chrislu
ed818c90a9
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-15 22:28:21 -07:00
chrislu
3639cad69c
master, filer, s3: also listen to "localhost" in addition to specific ip address
...
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
Konstantin Lebedev
f43c6daeda
Need to exit waiting if request is was canceled
2022-03-15 19:55:22 +05:00
Konstantin Lebedev
c2e7d663bc
reduced the timeout for creating a connection from 30 to 10 seconds
2022-03-15 13:32:05 +05:00
chrislu
fbc9f0eb64
minor
2022-03-14 03:19:16 -07:00
chrislu
2eda3a686f
2.94
2022-03-14 00:55:01 -07:00
chrislu
bd5c5586b5
generate inode via path and time
2022-03-14 00:03:29 -07:00
chrislu
5cba8e51c5
refactor
2022-03-13 18:34:57 -07:00
chrislu
f2f68f675e
write to disk during random writes, limiting total disk spaces used
2022-03-13 18:17:35 -07:00
chrislu
2f4cd20f90
tests skip reader pattern monitoring
2022-03-13 18:15:53 -07:00
chrislu
53513475bf
mount: add back random read support
...
avoid too much memory used also
2022-03-13 01:38:52 -08:00
chrislu
f70c1e449b
add useful doc link
2022-03-13 00:14:50 -08:00
chrislu
b20ddc57a7
mount: return open status
2022-03-12 22:38:14 -08:00
chrislu
b40d252761
mount: chmod for root
2022-03-12 12:10:56 -08:00
chrislu
3a6eb8ca5f
default bind to one ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-11 14:02:39 -08:00
Konstantin Lebedev
f53cff045f
checks disk file exist
2022-03-10 18:58:56 +05:00
Konstantin Lebedev
834210a9dc
avoid connect to the old filler address
...
https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-10 15:24:45 +05:00
chrislu
197ade6aeb
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-09 22:26:55 -08:00
chrislu
011a41b561
add back writes to swap file when too many in memory chunks are used.
2022-03-09 22:26:51 -08:00
zzq09494
2ea18fca48
fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command'
2022-03-10 13:37:34 +08:00
banjiaojuhao
f28dbbe5c5
[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.
2022-03-10 11:40:39 +08:00
chrislu
e2b07737da
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-08 23:02:33 -08:00
chrislu
dc204dd137
fix nil entry
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d340b4]
goroutine 130523 [running]:
github.com/chrislusf/seaweedfs/weed/filer.FileSize(...)
/code/seaweedfs/weed/filer/filechunks.go:26
github.com/chrislusf/seaweedfs/weed/mount.(*WFS).Lookup(0xc000866d80, 0x1, 0xc002897f40, {0xc004b00980, 0x39}, 0x1ec19e0)
/code/seaweedfs/weed/mount/weedfs_dir_lookup.go:59 +0x654
github.com/hanwen/go-fuse/v2/fuse.doLookup(0xc00033c000, 0xc00033c000)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/opcode.go:333 +0x6b
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000ab2420, 0xc00033c000)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:483 +0x1f3
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000ab2420, 0x0)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:456 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:323 +0x534
2022-03-08 23:02:30 -08:00
banjiaojuhao
f7f2a597dd
minor
2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038
[bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized).
2022-03-08 16:22:55 +08:00
chrislu
3aeee3d748
ensure releasing file handle
2022-03-07 14:01:24 -08:00
chrislu
8136384473
remove debug message
2022-03-07 11:22:26 -08:00
banjiaojuhao
b9ff7723dd
[bugfix] filer: nil pointer dereference
2022-03-07 23:26:25 +08:00
chrislu
da3d330616
s3 and filer transport using unix domain socket instead of tcp
2022-03-07 02:00:14 -08:00
chrislu
0cb17b45b1
refactoring
2022-03-07 01:59:01 -08:00
chrislu
6d3db4445b
buffer for all range requests
2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a
a little optimization
2022-03-07 00:24:59 -08:00
chrislu
f7ee60996c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-07 00:08:09 -08:00
chrislu
6f3ec989a7
fix manifest batch size
2022-03-07 00:07:53 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest
...
filer: support get metadata with resolved manifest chunk
2022-03-07 00:05:46 -08:00
chrislu
bb0b784544
minor
2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808
filer: support metadata with resolved manifest chunk
2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841
filer: add back isAppend function
2022-03-07 15:41:07 +08:00
chrislu
0ba4e4cd23
2.93
2022-03-06 18:54:12 -08:00
chrislu
46a28b8819
mount: adjust disk space based on quota
2022-03-06 17:22:49 -08:00
chrislu
ede6ce44c6
fix test
2022-03-06 17:09:55 -08:00
chrislu
21ef152423
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-06 17:04:24 -08:00
chrislu
f3442e36e6
mount: quota adjust error type to be syscall.ENOSPC
2022-03-06 17:04:21 -08:00
Chris Lu
89cd0c27a8
Merge pull request #2728 from kmlebedev/set_default_leveldb2
2022-03-06 07:26:30 -08:00
Konstantin Lebedev
cf444ebd07
Set default leveldb2 enabled
...
avoid Filer store is enabled for both leveldb2 and mysql
2022-03-06 18:27:25 +05:00
banjiaojuhao
a07c93f4dd
filer: reset bytesBuffer before use
2022-03-06 21:07:36 +08:00
chrislu
c7e8ac18f0
mount: quota for one mounted collection
...
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
2022-03-06 02:44:40 -08:00
chrislu
b7c992f410
add flag to enforce quota
2022-03-05 22:10:43 -08:00
chrislu
f1713c96ae
avoid possible runtime error: index out of range [0] with length 0
2022-03-05 21:14:31 -08:00
chrislu
da76af187f
mount: avoid possible index out of bounds error
2022-03-04 22:36:01 -08:00
chrislu
6e49e75a5b
use logical number of files and sizes for statistics and quota
2022-03-04 18:47:44 -08:00
chrislu
f51e20028a
mount: avoid comma in mount options
...
fix https://github.com/chrislusf/seaweedfs/issues/2719
2022-03-03 03:42:29 -08:00
chrislu
dc0f48682c
unused
2022-03-03 03:41:35 -08:00
chrislu
28b8974a3a
mount: fix directory pagination when using midnight commander
2022-03-03 02:59:31 -08:00
chrislu
a96d4254e9
filer, s3, volume server: a bit memory optimization
2022-03-02 20:15:28 -08:00
chrislu
6fbbc78574
stream reading a whole chunk
2022-03-02 13:50:46 -08:00
chrislu
784583afc6
avoid pool memory allocation if too large
2022-03-02 13:50:28 -08:00
chrislu
ba14307319
2.92
2022-02-28 15:22:19 -08:00
chrislu
fcf3714443
mount: add back support for filer.path
2022-02-28 12:16:53 -08:00
chrislu
80c017907b
filer.backup: fix backing up encrypted chunks
...
I have done filer.backup test:
replication.toml:
[sink.local]
enabled = true
directory = "/srv/test"
___
system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy
I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local
I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET
I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec
system@dat1:/srv/test$ ls -l
total 16
drwxr-xr-x 2 system system 4096 Feb 28 12:39 a
-rw-r--r-- 1 system system 48 Feb 28 12:39 fu.txt
-rw-r--r-- 1 system system 32 Feb 28 12:39 _index.html
-rw-r--r-- 1 system system 68 Feb 28 12:39 index.php
system@dat1:/srv/test$ cat fu.txt
? ?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~??
system@dat1:/srv/test$
On the active mount on the target server it's:
system@app1:/srv/app$ ls -l
total 2
drwxrwxr-x 1 system system 0 Feb 28 12:04 a
-rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt
-rw-r--r-- 1 system system 4 Feb 28 12:04 _index.html
-rw-r--r-- 1 system system 40 Feb 28 12:04 index.php
system@app1:/srv/app$ cat fu.txt
This is static boy!
Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData
It seems like it's still encrypted?
2022-02-28 10:07:06 -08:00
chrislu
aad62ee148
mount: mark directory uncached if forgotten
2022-02-28 02:08:24 -08:00
chrislu
554e239097
release file handle
2022-02-28 01:23:14 -08:00
chrislu
3639fedd01
mount: fix fsync opened and renamed files
2022-02-28 00:34:17 -08:00
chrislu
63a9d8f01d
ensure inodes are not duplicating unless hardlinked
2022-02-27 23:13:49 -08:00
chrislu
de77d00c81
correctly clean up for a file
2022-02-27 23:12:28 -08:00
chrislu
18543c6e8b
minor
2022-02-27 23:11:09 -08:00
chrislu
09cd00f356
2.91
2022-02-27 04:03:39 -08:00
chrislu
941ced60a4
download 2 chunks if at the beginning of a file
2022-02-27 03:57:24 -08:00
chrislu
f9d9eed0c9
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-27 03:41:35 -08:00
chrislu
d602d68fd1
remove dead code
2022-02-27 03:41:32 -08:00
Chris Lu
6c07af6014
Merge pull request #2715 from gfx-labs/acl-pr
2022-02-27 03:27:32 -08:00
chrislu
c3792c8352
remove dead code
2022-02-27 03:03:19 -08:00
chrislu
aa9eef81e6
retire mount v1
2022-02-27 02:57:27 -08:00
eddy-gfx
fb940dd807
Merge branch 'chrislusf:master' into acl-pr
2022-02-27 04:52:08 -06:00
elee
881a0fe806
ensure compatibility
2022-02-27 04:50:59 -06:00
elee
954ad98e0d
set canned acl on replication create
2022-02-27 04:49:31 -06:00
chrislu
6e1ab97988
use debug option to see operations
2022-02-27 02:02:30 -08:00
chrislu
2112d99140
mount2: add back readonly mode
2022-02-27 01:13:32 -08:00
chrislu
9ef5bb20f6
mount2: invalidate fuse cache for replaced inode
2022-02-27 00:00:23 -08:00
chrislu
4ddcbaab57
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-26 23:20:47 -08:00
chrislu
551d00d51a
prefetch other chunks when stream reading
2022-02-26 23:20:45 -08:00
Chris Lu
e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments
2022-02-26 09:16:12 -08:00
guosj
82cad5e330
correct comments
2022-02-26 22:44:26 +08:00
chrislu
7b1a713d2a
remove dead code
2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6
a little safer
2022-02-26 03:23:06 -08:00
chrislu
708e14fcfa
avoid possible too big memory allocation
2022-02-26 03:22:41 -08:00
chrislu
3345a50d9b
prefetch 2 chunks
2022-02-26 03:06:17 -08:00
chrislu
86ce69f709
remove logs
2022-02-26 03:00:20 -08:00
chrislu
b2a148cb4c
use file size as max range
2022-02-26 03:00:08 -08:00
chrislu
2ab0ad24a3
use memory pool
2022-02-26 02:59:19 -08:00
chrislu
28b395bef4
better control for reader caching
2022-02-26 02:16:47 -08:00
chrislu
3ad5fa6f6f
chunk cache adds function ReadChunkAt
2022-02-25 21:55:04 -08:00
chrislu
fc7a4957ea
fix mount2 options
2022-02-25 21:22:44 -08:00
chrislu
72c0233938
less logs
2022-02-25 15:34:24 -08:00
chrislu
101e6d80b0
mount2: listXattr return ok if xattr is empty
2022-02-25 14:38:56 -08:00
chrislu
03466f955e
rename: delete source entry metadata only, skipping hard links
2022-02-25 02:57:54 -08:00
