chrislu
50dc2fe96b
cleaning variables
2023-03-19 18:48:40 -07:00
Konstantin Lebedev
29d1312625
Fs.verify.concurrency ( #4293 )
...
* init
* refactor
* refactor print msg
* param just concurrency
* fix race
* next fix race
* reset variables before each run
* clean
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-19 18:38:57 -07:00
chrislu
784daea1fa
fix volume not found if marked as read only
...
fix https://github.com/seaweedfs/seaweedfs/issues/4088
2023-03-19 18:30:13 -07:00
Stewart Miles
264be0d2d4
Retry until a leader is selected. ( #4318 )
...
Fixes regression introduced in
https://github.com/seaweedfs/seaweedfs/pull/4313
Related to #4307
2023-03-16 20:50:38 -07:00
LHHDZ
d695119073
The filer does not set defaultReplication to the defaultReplication o… ( #4315 )
...
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-15 22:10:39 -07:00
Stewart Miles
57ab1f8516
Use exponential backoff to query leader. ( #4313 )
...
`topology.Leader()` was using a backoff that typically
resulted in at least a 5s delay when initially starting
a master and raft server. This changes the backoff
algorithm to use exponential backoff starting with 100ms
and waiting up to 20s for leader selection.
Related to #4307
2023-03-15 17:49:46 -07:00
Stewart Miles
dd71f54c6b
Fix -raftHashicorp and -raftBootstrap flag propagation. ( #4309 )
...
`weed server` was not correctly propagating
`-master.raftHashicorp` and `-master.raftBootstrap` flags when
starting the master server.
Related to #4307
2023-03-15 13:03:20 -07:00
Patrick Schmidt
71b33faef0
Skip parent directory creation in mount ( #4310 )
...
A POSIX filesystem does not implicitly create parent directories when
they do not exist. Directories must be explicitly created and
permissions be set.
This also fixes a bug where asynchronous operations would create a
file in the filer before the parent directory was created. If the
file was a symlink or another special type of file the directory
would inherit that type and become unusable in the mounted FS.
2023-03-15 12:37:25 -07:00
chrislu
f5854d13df
fix test
2023-03-14 22:05:16 -07:00
chrislu
0454bb2d88
fix test
...
when size is zero, the "LastModified" is not written
2023-03-14 22:02:49 -07:00
chrislu
2268d38263
fix naming
...
fix https://github.com/seaweedfs/seaweedfs/issues/4305
2023-03-14 08:38:02 -07:00
chrislu
98bc93cf31
fix tests
2023-03-13 10:00:27 -07:00
Konstantin Lebedev
e17429223e
shell script unclean variables ( #4298 )
2023-03-13 07:16:31 -07:00
Thomas Anderson
bd70683871
fix(weed/topology/node.go): typo in error message ( #4292 )
2023-03-08 23:58:03 -08:00
Rohit Chormale
59706c89fb
#4270 set http status code to 409 if dir already exists ( #4287 )
2023-03-08 07:45:08 -08:00
chrislu
7111c08289
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-03-08 01:48:30 -08:00
chrislu
c589e9837c
fix tests
2023-03-08 01:48:28 -08:00
Kevin Liu
244385bf0d
Fix binding metrics to ipv6 ( #4286 )
...
* Fix binding metrics to ipv6
* Update weed/stats/metrics.go
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-08 01:04:02 -08:00
zemul
bd6f2716d3
fix_read_tomb_needle_meta ( #4263 )
...
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:26:55 -08:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs ( #4264 )
...
* fix
* fix mtime s > ns
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:24:36 -08:00
chrislu
8241c9a829
adjust ui pagination size
...
fix https://github.com/seaweedfs/seaweedfs/issues/4258
2023-02-28 08:31:26 -08:00
lfhy
1976ca9160
add -disk to filer command ( #4247 )
...
* add -disk to filer command
* add diskType to filer.grpc
* use filer.disk when filerWebDavOptions.disk is empty
* add filer.disk to weed server command.
---------
Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
2023-02-25 09:48:59 -08:00
chrislu
214b7cd286
volume.fix.replication: adjust the retry checking times
2023-02-22 10:47:52 -08:00
LHHDZ
db5515eada
Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists ( #4240 )
...
Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-02-21 23:08:52 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency ( #4239 )
2023-02-21 19:47:33 -08:00
chrislu
3227e4175e
3.43
2023-02-20 16:56:05 -08:00
chrislu
9bb0a9e306
clean comments
2023-02-17 01:31:44 -08:00
chrislu
545d5d0cc3
fix for io.ReaderAt used in filer.remote.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/4194
2023-02-17 01:30:55 -08:00
wusong
79af6b41d4
[mount] Fix read file-data in entry.content ( #4207 )
2023-02-15 07:18:56 -08:00
chrislu
5869945f16
avoid infinite loop
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904
2023-02-10 13:05:04 -08:00
chrislu
e037c71ec3
adjust text
2023-02-10 13:04:29 -08:00
chrislu
67b8c2853a
add line return
2023-02-10 12:53:43 -08:00
chrislu
302155a6fa
fix compilation
2023-02-10 12:32:59 -08:00
chrislu
5520b81551
add some error messages for args
...
related to https://github.com/seaweedfs/seaweedfs/issues/4115
2023-02-10 12:30:53 -08:00
chrislu
dc4ed2cd9b
do not move cloud tier volumes
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195
2023-02-09 22:17:54 -08:00
chrislu
31bb91583f
fix bug when vid not found
...
fix https://github.com/seaweedfs/seaweedfs/issues/4193
2023-02-09 17:30:44 -08:00
chrislu
79d68a81fd
text change
2023-02-07 15:00:41 -08:00
chrislu
999a126a3f
typo
2023-02-07 14:59:47 -08:00
chrislu
1c2364936a
detect invalid bucket name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4143
2023-02-07 14:57:29 -08:00
chrislu
5de93fe442
refactoring
2023-02-07 14:45:20 -08:00
Eric Yang
5083429704
ADHOC: s3 bucket name verification ( #4189 )
...
* ADHOC: s3 bucket name verification
* add test
2023-02-07 14:37:29 -08:00
Neo
d5f77706a8
volume:fix return if Replicate write error ( #4188 )
2023-02-06 22:34:56 -08:00
chrislu
8821d6b161
3.42
2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region ( #4177 )
...
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu
84e9934bf9
fix filer.remote.sync on a S3 cloud mount
...
fix https://github.com/seaweedfs/seaweedfs/issues/4175
2023-02-01 20:44:00 -08:00
chrislu
0932437a1c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-02-01 02:38:09 -08:00
chrislu
c47f75eb5b
fix null content type caused exception in hdfs client
...
fix https://github.com/seaweedfs/seaweedfs/issues/4170
2023-02-01 02:38:05 -08:00
wusong
fe573f26a8
memchunk save content twice ( #4174 )
2023-01-31 19:36:59 -08:00
Neo
b9b613a78e
filter system log dir does not make subscribe event ( #4172 )
2023-01-31 18:56:11 -08:00
wusong
665917fdf1
[mount]: delete redundant code ( #4173 )
2023-01-31 18:54:46 -08:00
a
0d085ec175
make directory lister trigger if mime type is httpd/unix-directory ( #4169 )
2023-01-31 06:54:40 -08:00
chrislu
4d71af87f3
3.41
2023-01-29 22:56:01 -08:00
chrislu
ef70415c70
skipping manifest chunk volume id change
2023-01-29 22:55:06 -08:00
chrislu
6e8dbf8868
add fs.meta.changeVolumeId
2023-01-29 22:51:58 -08:00
wusong
de081c0d64
[mount] fix metacache update ( #4161 )
2023-01-29 07:55:27 -08:00
Guo Lei
b63d1e9c6b
fix index ( #4158 )
2023-01-28 06:55:27 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6
use one readerCache for the whole file
2023-01-16 22:43:02 -08:00
chrislu
4614e85efa
adjust help message
2023-01-15 21:28:36 -08:00
chrislu
c24678153c
use memory chunks only for sequential writes
2023-01-15 21:10:08 -08:00
chrislu
2885ba0e50
3.40
2023-01-15 20:12:05 -08:00
chrislu
12f8251d32
avoid index out of slice range
2023-01-14 13:31:22 -08:00
chrislu
313fd17ae4
allow random writes for mem chunk
2023-01-13 20:32:18 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload ( #4130 )
...
* merge chunks during upload
* fix test
2023-01-12 23:07:36 -08:00
chrislu
2452f3b2f7
sort chunks by offset
2023-01-12 20:29:07 -08:00
chrislu
361d97941b
mysql: Missed escaping column name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
efcffe4f77
no need to set chunks during flush
2023-01-10 10:01:52 -08:00
chrislu
85f1b70207
reduce group size from 256MB to 64MB
2023-01-10 01:35:44 -08:00
chrislu
86657ed3a8
avoid repeatedly adding manifest chunks
2023-01-10 01:35:10 -08:00
chrislu
28fe578944
avoid possible nil entry
2023-01-10 01:07:34 -08:00
chrislu
47fa5ef979
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-10 00:52:50 -08:00
chrislu
9e0d526163
fix compilation
2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e
chunk group remove manifestChunks and reset sections in SetChunks()
2023-01-10 00:49:31 -08:00
chrislu
e650c8397d
ensure section.chunks is always garbage free
2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1
refactor
2023-01-10 00:46:46 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. ( #4119 )
2023-01-09 06:47:08 -08:00
chrislu
51d4a4b28d
3.39
2023-01-08 18:20:22 -08:00
chrislu
0b52f08226
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-08 18:03:25 -08:00
chrislu
469c959e6a
add missing options for webdav
2023-01-08 18:03:22 -08:00
monchickey
3e2c9ea73d
Add image cropping. ( #4117 )
2023-01-06 09:28:07 -08:00
chrislu
296fdc296c
mount: faster add chunks
2023-01-06 01:03:29 -08:00
chrislu
2abf817580
fix for stream reader
...
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586
add notes for full version
2023-01-04 09:52:25 -08:00
chrislu
5423790b2c
add back previous chunk upload selection algo
2023-01-03 22:50:39 -08:00
chrislu
7bdae5172e
batch delete EC needles
...
fix https://github.com/seaweedfs/seaweedfs/issues/4107
2023-01-03 22:05:26 -08:00
zemul
6b4c033431
add mount log ( #4101 )
...
* filer.backup use replication.source.filer
* add mount log
* Revert "filer.backup use replication.source.filer"
This reverts commit 07bf6f956c
2023-01-03 00:00:45 -08:00
chrislu
48f2edc065
3.38
2023-01-02 23:23:02 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 ( #4094 )
2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax ( #4096 )
2023-01-01 05:06:57 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query ( #4095 )
2023-01-01 05:06:41 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding ( #4092 )
2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
Guo Lei
5f993a5f20
Fix ide warning ( #4082 )
...
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
2022-12-26 00:03:47 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file ( #4060 )
...
* fs.verify needles status of file
* fix parse args
* refactor logging
* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
chrislu
77b7c9f7cc
lock and unlock chunks for read
2022-12-21 14:15:35 -08:00
chrislu
d1797deccb
remove duplicated logic of removing from writableChunks
2022-12-21 14:15:11 -08:00
chrislu
6ede19e825
add a simple file replication progress bar
2022-12-20 19:47:21 -08:00
chrislu
f7beba8515
unlock before submitting the uploading jobs
2022-12-20 12:52:30 -08:00
chrislu
28d479e5c0
mount: adjust locking for upload pipeline
2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305
filer sink retries reading file chunks, skipping missing chunks
...
if the file chunk is not available during replication time, the file is skipped
2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c
metadata skip reading un-available logs
...
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command ( #4061 )
2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993
more detailed logs
2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed ( #4057 )
2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key ( #4047 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-12-11 22:49:57 -08:00
chrislu
438146249f
3.37
2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9
rotate log files
2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c
volume server logs add url info
...
debug https://github.com/seaweedfs/seaweedfs/issues/3964
2022-12-11 13:15:38 -08:00
chrislu
38c63c429b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb
add chunk etag when downloading from remote storage
...
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting ( #4042 )
...
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming ( #4039 )
2022-12-07 07:47:40 -08:00
chrislu
659f28b855
simple refactoring
2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8
duplicated logic
2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" ( #4035 )
...
* Revert "refactor: moved to locked entry"
This reverts commit 94bc9afd9d
2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb
3.36
2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d
refactor: moved to locked entry
2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory ( #4027 )
2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer ( #4025 )
2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn ( #4024 )
2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories ( #4016 )
2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip ( #4012 )
2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata ( #4010 )
2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
...
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57
3.35
2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
...
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f
filer store: skip disabled location specific filer store initialization
...
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f
file name escape uri
...
fix https://github.com/seaweedfs/seaweedfs/issues/3969
2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part ( #3968 )
2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid ( #3963 )
2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers ( #3965 )
2022-11-10 07:09:53 -08:00
chrislu
47db75a695
3.34
2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType ( #3949 )
...
* fix filer.remote.sync to azure with ContentType
* fix pass X-Amz-Meta to X-Ms-Meta
2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration ( #3941 )
2022-11-04 00:34:40 -07:00
chrislu
4193dafce1
azure metadata: skip metadata prefixed with "X-"
...
fix https://github.com/seaweedfs/seaweedfs/issues/3875
2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete ( #3939 )
2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
...
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
...
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted ( #3932 )
2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
...
check needles status from volume server
2022-10-31 11:33:04 -07:00
chrislu
0f360862bf
3.33
2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies ( #3924 )
...
avoid vacuum if not enough replica copies
2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
...
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
8b9957d461
add back "/" prefix if it is missing in object
...
fix https://github.com/seaweedfs/seaweedfs/issues/3737
2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-28 12:53:23 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
...
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error ( #3918 )
2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048
master: delete partially created replicated volumes during volume growth
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit ( #3905 )
2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
...
* fix: set user metadata key to lowercase
* feat: simplify a bit
2022-10-24 17:29:52 -07:00
chrislu
091f999e8c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-24 16:50:41 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo ( #3893 )
...
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2022-10-24 09:25:48 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3
fix merge
2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
...
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
...
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40
skip reading needle if size is invalid
...
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d
remove volume server tcp client
2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a
fix size variable
2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
...
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0
3.32
2022-10-16 20:43:37 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9
Revert "mount: remove mount directory from generating cache directory id"
...
This reverts commit 109dc7fdf6
2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
...
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2022-10-13 23:15:00 -07:00
chrislu
e05637c42c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp ( #3855 )
...
* add acl helper functionalities
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* add tests
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* remove 0 when create map
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* delete when empty at `AssembleEntryWithAcp`
`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb
default hasSlowRead to true
2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true ( #3710 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities ( #3831 )
2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c ( #3834 )
2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata ( #3833 )
2022-10-12 07:14:49 -07:00
chrislu
e55076c46f
cloud tier: add retry when copying data file
...
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-11 21:58:19 -07:00
chrislu
0d817bc347
fix invalid memory address or nil pointer dereference on filer.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
...
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… ( #3824 )
...
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency ( #3813 )
2022-10-10 06:44:29 -07:00
chrislu
0711870f43
3.31
2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6
mount: remove mount directory from generating cache directory id
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
chrislu
f9383aa726
refactor to change capacity data type
2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
...
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options ( #3807 )
...
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not ( #3799 )
...
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452
* fix metric MasterAdminLock
2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics ( #3798 )
2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop ( #3797 )
2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
chrislu
ea271600ec
fix parameters
2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c
filer.sync: limit concurrency when fetching file chunks
...
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels ( #3785 )
2022-10-04 01:20:01 -07:00
chrislu
29632d5a34
3.30
2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis ( #3765 )
2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir ( #3770 )
2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7
better fix for option format
2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb
mount: fix extra option format
2022-09-30 08:44:41 -07:00
chrislu
8e81619d02
mount: accept all extra mount options
...
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
...
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer ( #3759 )
2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account ( #3753 )
...
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
...
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1
skip debug message
2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server ( #3708 )
...
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer ( #3750 )
...
Adjusted filer response headers
2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" ( #3747 )
...
* fs.meta.load load any dirs with prefix "important"
* replace dirPattern to dirPrefix
* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda
skip truncation on error
...
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e
3.29
2022-09-25 19:20:45 -07:00
chrislu
640c53cda8
add future list
2022-09-25 17:43:42 -07:00
chrislu
cc570a4477
assert results are sorted
2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1
ordered execution async wait
2022-09-25 13:45:55 -07:00
chrislu
973b8ff067
rename
2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b
rename file
2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f
filer replication: compare content changes directly
...
Fix https://github.com/seaweedfs/seaweedfs/issues/3714
The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.
For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-20 08:35:10 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf
adjust logs
2022-09-18 18:14:24 -07:00
