chrislu
b4131a3a72
filer.remote.sync: parallelize uploading to remove object storage
2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561
refactor
2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-21 23:26:09 -07:00
chrislu
8bac903d29
mount: add locking to prevent nil pointer
...
fix https://github.com/seaweedfs/seaweedfs/issues/3479
2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files ( #3478 )
...
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
7807f6641a
minor
2022-08-21 22:54:56 -07:00
chrislu
c4e862e908
3.23
2022-08-21 19:13:26 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
...
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
2022-08-21 19:11:18 -07:00
chrislu
63fbf281c7
remove same file copying rage limitation
2022-08-21 17:16:21 -07:00
Patrick Schmidt
3a75d7f7aa
Implement copy_file_range syscall in FUSE ( #3475 )
...
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
2022-08-21 17:11:43 -07:00
chrislu
fef9c6a520
filer: do not always save files in "/etc" folder to filer store
...
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
chrislu
2762154130
fix compilation
2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3
EC: with multiple volume locations, the ec rebuilding may fail
2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f
EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations
...
related to https://github.com/seaweedfs/seaweedfs/issues/3459
2022-08-21 14:52:37 -07:00
chrislu
8b3429858d
align memory for atomic read/write
2022-08-21 12:20:27 -07:00
Patrick Schmidt
2ef6ab998c
Avoid race conditions with current filer address ( #3474 )
...
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
2022-08-21 12:18:13 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher ( #3473 )
...
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu
388f82f322
minor
2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e
refactoring
2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps ( #3472 )
...
* Fix FUSE server buffer leaks in file gaps
This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.
* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu
c7892bc7c4
volume: file counter should be all files
...
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2022-08-20 23:35:31 -07:00
chrislu
f7e0a65e75
retry for all errors
2022-08-20 23:34:45 -07:00
chrislu
649555b223
adjust log
2022-08-20 23:34:07 -07:00
chrislu
973f6dd162
refactoring
2022-08-20 22:38:15 -07:00
chrislu
65ff7198fe
adjust type
2022-08-20 22:37:24 -07:00
chrislu
28b862f45f
use util.Retry to retry
2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8
webdav: retryable data chunk upload
2022-08-20 19:18:12 -07:00
chrislu
f8fa430257
filer: retryable data chunk upload
2022-08-20 19:15:44 -07:00
chrislu
4081d50607
filer sink: retryable data chunk uploading
2022-08-20 19:09:15 -07:00
chrislu
409f39390d
mount: retryable data chunk uploading
2022-08-20 19:04:36 -07:00
chrislu
6c8822f269
filer.copy: retryable file part upload
2022-08-20 18:59:57 -07:00
chrislu
a3553da7f7
add fileId to UploadWithRetry return result
2022-08-20 18:54:59 -07:00
chrislu
d49d0a9fc2
filer.copy: retryable upload
2022-08-20 18:50:57 -07:00
chrislu
689b4ecdcc
remove unused collection and replication from upload result
2022-08-20 18:14:57 -07:00
chrislu
e3f40d538d
cleaner code
2022-08-20 17:51:30 -07:00
chrislu
9f50b310a0
remove unused code
2022-08-20 17:51:12 -07:00
chrislu
11f99836c3
filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
...
fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-19 23:00:56 -07:00
chrislu
fdd8c5d5e0
Filer if-modified-since check doesn't fire on "exact" #3467
...
fix https://github.com/seaweedfs/seaweedfs/issues/3467
2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89
filer: retryable when error is not found
2022-08-19 03:36:15 -07:00
chrislu
58dd880322
minor
2022-08-18 23:51:08 -07:00
Eric Yang
b6f99c4d83
fix naming typo ( #3465 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-18 23:25:32 -07:00
chrislu
5702727db5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-18 02:39:27 -07:00
chrislu
4d72a1ce6e
s3: remove deleted collection during metrics reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2022-08-18 02:39:07 -07:00
Guo Lei
3172c33fa6
fix redundant type from array ( #3462 )
2022-08-18 02:25:25 -07:00
chrislu
42c6e52513
s3: fix regression on HEAD directory operation
2022-08-18 02:13:58 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
4ffbda1c43
fix building for unmaintained
2022-08-17 14:45:38 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
e259052a70
volume: fix metrics for volume request time
2022-08-17 04:29:31 -07:00
