Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket ( #3609 )
...
https://github.com/seaweedfs/seaweedfs/issues/3607
2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting ( #3577 )
2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level ( #3610 )
2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap ( #3606 )
2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic ( #3596 )
2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log ( #3598 )
2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF ( #3597 )
2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e
3.25
2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level ( #3589 )
2022-09-04 22:37:32 -07:00
chrislu
f4abefc109
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 22:23:31 -07:00
chrislu
39340f7e42
cloud tier: s3 consume all read response body
...
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2022-09-04 22:23:28 -07:00
chrislu
7c6324b114
adjust log level
2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket ( #3587 )
...
* Update helpers.go
* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context ( #3586 )
...
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d
minor optimization
2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca
filer.backup and filer.sync: include headers during backup and sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3532
2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level ( #3585 )
2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move ( #3583 )
2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync ( #3579 )
...
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed ( #3575 )
2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c
filer: rocksdb store report not found correctly
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e
Revert "avoid data race on grace.hooks ( #3572 )"
...
This reverts commit c37d6fc01a
2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks ( #3572 )
...
https://github.com/seaweedfs/seaweedfs/issues/3564
2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket ( #3571 )
...
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552
* copy localSocket
2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize ( #3570 )
...
https://github.com/seaweedfs/seaweedfs/issues/3557
2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log ( #3568 )
...
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567
* format message
2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab
align memory
2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch ( #3566 )
...
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1
add more logs
2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 ( #3541 )
...
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0
Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud ) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases )
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0 )
---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update code
* more code fix
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount ( #3562 )
...
* Add an e2e workflow to test FUSE mount
* Fix deadlocks during concurrent r/w
2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c
upload_content: upload close response.Body
2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced
refactor
2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c
just a bit safer
2022-08-31 00:10:17 -07:00
chrislu
9c944377dc
minor
2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6
s3: close response body with copying object
2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae
print process before do the work
2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt ( #3554 )
2022-08-30 10:12:02 -07:00
chrislu
de5263364b
master client: use GetMaster() for latest master address
2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a
more detailed error
2022-08-30 09:43:17 -07:00
chrislu
d81db3c703
s3: fix configuring IAM for the same user
...
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]
s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version ( #3539 )
...
* avoid race conditions access to SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515
* superBlockAccessLock replace to sync.Mutex
2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0
rename variable
2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster ( #3538 )
...
https://github.com/seaweedfs/seaweedfs/issues/3510
2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr ( #3550 )
...
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay ( #3548 )
2022-08-30 00:02:09 -07:00
chrislu
570b488572
ensure memory alignment on different CPUs
2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount ( #3537 )
...
https://github.com/seaweedfs/seaweedfs/issues/3511
2022-08-29 13:23:02 -07:00
Patrick Schmidt
00041f6e08
Fix corrupted mount reads after changes on fh ( #3536 )
2022-08-29 08:27:13 -07:00
chrislu
b7a887fea1
3.24
2022-08-28 21:53:13 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug ( #3534 )
2022-08-28 07:51:47 -07:00
chrislu
cc0c8c5f81
simplify
2022-08-27 00:21:57 -07:00
chrislu
3741779144
minor
2022-08-27 00:20:58 -07:00
chrislu
580a7fa685
adjust for data race
...
fix https://github.com/seaweedfs/seaweedfs/issues/3512
2022-08-27 00:19:13 -07:00
chrislu
87b70a6809
clean up
2022-08-27 00:09:04 -07:00
chrislu
27c7611ebf
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 23:47:39 -07:00
chrislu
9920d65bc0
gateway to remote object store: adjust upload concurrency
2022-08-26 23:47:37 -07:00
chrislu
c839ce1b19
s3 sink use s3 upload manager
...
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-26 23:47:12 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children ( #3530 )
2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8
make CodeQL happy
2022-08-26 17:09:11 -07:00
chrislu
301b49b63f
atomic operation
2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0
nano level precision
2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c
more detailed error
...
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount ( #3527 )
2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8
rename varaible
2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate ( #3525 )
...
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount ( #3526 )
2022-08-26 08:41:42 -07:00
chrislu
6f483a4d36
atomic add
...
fix https://github.com/seaweedfs/seaweedfs/issues/3514
2022-08-25 22:20:34 -07:00
chrislu
7394f7feee
minor: adjust ordering
2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1
minor clean up
2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks ( #3505 )
2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races ( #3503 )
2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created ( #3498 )
...
* replace mkdir to mkFile
* ContentLength must be zero
* revert mkDir
* Seaweedfs-Is-Directory-Key return metadata
2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
...
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu
10414fd81c
ping timeout at 15 seconds
...
this 72 minute timeout setting seems unreasonably long
15 seconds is around the time when a new raft leader should be elected.
2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long ( #3398 )
...
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug ( #3499 )
2022-08-23 08:53:24 -07:00
chrislu
21a933c932
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-23 01:52:32 -07:00
chrislu
f0b4a7659a
fix test
2022-08-23 01:52:29 -07:00
Patrick Schmidt
cda8cc22bc
Implement lseek syscall in FUSE ( #3491 )
...
See the man page of lseek:
https://man7.org/linux/man-pages/man2/lseek.2.html
2022-08-23 01:47:18 -07:00
chrislu
bf88006037
format
2022-08-23 01:20:45 -07:00
chrislu
9fce75607d
s3: report http.StatusOK if the directory is explicitly created
...
fix https://github.com/seaweedfs/seaweedfs/issues/3457
2022-08-23 01:16:46 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil ( #3497 )
2022-08-22 23:54:51 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests ( #3496 )
2022-08-22 23:39:34 -07:00
chrislu
676e27c589
shell: stop long running jobs if lock is lost
2022-08-22 14:12:23 -07:00
chrislu
601ba5fb68
master: disable locking if not leader
2022-08-22 14:11:48 -07:00
chrislu
57e7582c36
refactoring
2022-08-22 14:11:13 -07:00
chrislu
7ea2c24c5d
metadata processing use retry instead of retryForEver
2022-08-22 11:25:40 -07:00
Patrick Schmidt
f875031f06
Reuse readDataByFileHandle in Read call ( #3482 )
2022-08-22 08:24:06 -07:00
famosss
bdba3da2e4
Handle preflight cors requests ( #3481 )
2022-08-22 08:21:38 -07:00
chrislu
f4b52d4c52
fix format
2022-08-22 00:20:21 -07:00
chrislu
b4131a3a72
filer.remote.sync: parallelize uploading to remove object storage
2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561
refactor
2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-21 23:26:09 -07:00
chrislu
8bac903d29
mount: add locking to prevent nil pointer
...
fix https://github.com/seaweedfs/seaweedfs/issues/3479
2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files ( #3478 )
...
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
7807f6641a
minor
2022-08-21 22:54:56 -07:00
chrislu
c4e862e908
3.23
2022-08-21 19:13:26 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
...
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
2022-08-21 19:11:18 -07:00
chrislu
63fbf281c7
remove same file copying rage limitation
2022-08-21 17:16:21 -07:00
Patrick Schmidt
3a75d7f7aa
Implement copy_file_range syscall in FUSE ( #3475 )
...
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
2022-08-21 17:11:43 -07:00
chrislu
fef9c6a520
filer: do not always save files in "/etc" folder to filer store
...
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
chrislu
2762154130
fix compilation
2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3
EC: with multiple volume locations, the ec rebuilding may fail
2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f
EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations
...
related to https://github.com/seaweedfs/seaweedfs/issues/3459
2022-08-21 14:52:37 -07:00
chrislu
8b3429858d
align memory for atomic read/write
2022-08-21 12:20:27 -07:00
Patrick Schmidt
2ef6ab998c
Avoid race conditions with current filer address ( #3474 )
...
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
2022-08-21 12:18:13 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher ( #3473 )
...
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu
388f82f322
minor
2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e
refactoring
2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps ( #3472 )
...
* Fix FUSE server buffer leaks in file gaps
This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.
* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu
c7892bc7c4
volume: file counter should be all files
...
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2022-08-20 23:35:31 -07:00
chrislu
f7e0a65e75
retry for all errors
2022-08-20 23:34:45 -07:00
chrislu
649555b223
adjust log
2022-08-20 23:34:07 -07:00
chrislu
973f6dd162
refactoring
2022-08-20 22:38:15 -07:00
chrislu
65ff7198fe
adjust type
2022-08-20 22:37:24 -07:00
chrislu
28b862f45f
use util.Retry to retry
2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8
webdav: retryable data chunk upload
2022-08-20 19:18:12 -07:00
chrislu
f8fa430257
filer: retryable data chunk upload
2022-08-20 19:15:44 -07:00
chrislu
4081d50607
filer sink: retryable data chunk uploading
2022-08-20 19:09:15 -07:00
chrislu
409f39390d
mount: retryable data chunk uploading
2022-08-20 19:04:36 -07:00
chrislu
6c8822f269
filer.copy: retryable file part upload
2022-08-20 18:59:57 -07:00
chrislu
a3553da7f7
add fileId to UploadWithRetry return result
2022-08-20 18:54:59 -07:00
chrislu
d49d0a9fc2
filer.copy: retryable upload
2022-08-20 18:50:57 -07:00
chrislu
689b4ecdcc
remove unused collection and replication from upload result
2022-08-20 18:14:57 -07:00
chrislu
e3f40d538d
cleaner code
2022-08-20 17:51:30 -07:00
chrislu
9f50b310a0
remove unused code
2022-08-20 17:51:12 -07:00
chrislu
11f99836c3
filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
...
fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-19 23:00:56 -07:00
chrislu
fdd8c5d5e0
Filer if-modified-since check doesn't fire on "exact" #3467
...
fix https://github.com/seaweedfs/seaweedfs/issues/3467
2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89
filer: retryable when error is not found
2022-08-19 03:36:15 -07:00
chrislu
58dd880322
minor
2022-08-18 23:51:08 -07:00
Eric Yang
b6f99c4d83
fix naming typo ( #3465 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-18 23:25:32 -07:00
chrislu
5702727db5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-18 02:39:27 -07:00
chrislu
4d72a1ce6e
s3: remove deleted collection during metrics reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2022-08-18 02:39:07 -07:00
Guo Lei
3172c33fa6
fix redundant type from array ( #3462 )
2022-08-18 02:25:25 -07:00
chrislu
42c6e52513
s3: fix regression on HEAD directory operation
2022-08-18 02:13:58 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
e259052a70
volume: fix metrics for volume request time
2022-08-17 04:29:31 -07:00
qzh
7fcfaf7bc9
fix(filer.sync): offset may be set to 0 ( #3451 )
...
* fix(filer.sync): initializing the offset is related to the path
* fix(filer.sync): the offset maybe to be set to 0.
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 23:43:52 -07:00
qzh
400f0c3e5d
fix(filer.sync): initializing the offset is related to the path ( #3450 )
...
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 21:56:47 -07:00
chrislu
fa4d0093e1
3.22
2022-08-15 16:48:23 -07:00
chrislu
93261f5199
Revert "Refactor for Sync method ( #3426 )"
...
This reverts commit 670cb759f8
2022-08-15 15:35:31 -07:00
chrislu
9c588d4010
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-15 10:03:56 -07:00
chrislu
96caf21d09
less verbose log
2022-08-15 10:03:52 -07:00
Andrey Triumfov
31faa6d43d
Remove duplicate slashes in object path to prevent 500 errors ( #3442 )
2022-08-15 08:19:28 -07:00
chrislu
cb476a53ff
remove logs
2022-08-15 01:05:35 -07:00
chrislu
7c029b2183
3.21
2022-08-15 00:32:15 -07:00
Chris Lu
e88392b50f
Fix s3 pagination ( #3436 )
...
* Revert previous changes
* s3: use cursor to track tree traversal
fix https://github.com/seaweedfs/seaweedfs/issues/3166
* special cases for empty prefix and empty directory
* use constants
* address empty folder
* undo local changes
* fix IsTruncated
* adjust counting directories
* fix cases when prefix is a directory
* s3: handle directory object
works for
aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
2022-08-15 00:30:19 -07:00
chrislu
7457c746f0
s3: fix aws s3api head-object
2022-08-14 23:52:35 -07:00
chrislu
789d9dbfe1
webdav: fix nil
...
fix https://github.com/seaweedfs/seaweedfs/issues/3440
2022-08-14 23:33:30 -07:00
Konstantin Lebedev
6199d9b923
rand start garbageThreshold and reset metric ReplicaPlacementMismatch ( #3396 )
...
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch
* vacuum through sleep
2022-08-12 13:11:09 -07:00
Konstantin Lebedev
4d4cd0948d
avoid infinite loop WaitUntilConnected() ( #3431 )
...
https://github.com/seaweedfs/seaweedfs/issues/3421
2022-08-11 15:03:26 -07:00
Eric Yang
986daecbfe
add upload response ( #3428 )
...
* add upload response
* fixing comments
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-10 22:43:09 -07:00
chrislu
ae93c966d9
ensure memory is aligned
...
fix https://github.com/seaweedfs/seaweedfs/issues/3427
2022-08-10 22:27:13 -07:00
Rain Li
670cb759f8
Refactor for Sync method ( #3426 )
2022-08-10 08:21:57 -07:00
chrislu
662ec97602
mount: adjust on du stats reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3424
2022-08-09 20:21:01 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code ( #3423 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-09 01:24:43 -07:00
chrislu
0854171d22
3.20
2022-08-07 14:42:28 -07:00
chrislu
d8e69df295
adjust comments
2022-08-07 11:35:33 -07:00
chrislu
0aeec04c31
quicker to adapt to pattern change
2022-08-07 10:14:01 -07:00
chrislu
928d29af9e
fix wrong logic about reader isRandomMode()
2022-08-07 10:13:04 -07:00
chrislu
d3d52b1818
shell: fs.meta.load add quieter mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3414
2022-08-07 01:42:15 -07:00
chrislu
358ccb5e0e
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-07 01:34:36 -07:00
chrislu
67814a5c79
refactor and fix strings.Split
2022-08-07 01:34:32 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection ( #3413 )
2022-08-07 00:58:53 -07:00
chrislu
1a4bf0dcb5
filer.sync: parallelize the filer.sync
2022-08-07 00:56:15 -07:00
chrislu
02b38f894b
debug
2022-08-07 00:55:49 -07:00
chrislu
0e9478488d
filer.sync: fix when excluded paths is empty
2022-08-07 00:55:34 -07:00
chrislu
8a880a139d
mount: fix truncate operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/2609
2022-08-06 13:58:45 -07:00
LHHDZ
84ec68e11a
Add download speed limit support ( #3408 )
2022-08-05 01:16:42 -07:00
chrislu
b278bb24d3
mount: adjust df stats reporting when close to the limit
...
fix https://github.com/seaweedfs/seaweedfs/issues/3407
2022-08-05 00:31:41 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics ( #3402 )
...
* refactor FilerRequest metrics
* avoid double count proxy
* defer to
2022-08-04 01:44:54 -07:00
chrislu
bd13a7968f
remove unused symlink resolving
2022-08-04 01:35:18 -07:00
chrislu
88945d9954
mount: do not follow soft link with xattr
2022-08-04 01:21:15 -07:00
chrislu
90db4b5a44
minor
2022-08-03 00:10:47 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite
...
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 13:31:01 -07:00
Konstantin Lebedev
3ffa4ba91d
clean
2022-08-02 01:30:38 +05:00
Konstantin Lebedev
2a5629dd28
align
2022-08-02 01:29:44 +05:00
Konstantin Lebedev
df9cc31a0a
refactor
2022-08-02 01:18:26 +05:00
Konstantin Lebedev
0a5d3c3cf6
rename to localWrite and replicatedWrite
2022-08-01 23:09:09 +05:00
chrislu
bd5c9904be
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-01 09:40:05 -07:00
chrislu
e26d1cf4ef
fix test
2022-08-01 09:40:01 -07:00
Konstantin Lebedev
fc65122766
rename to LoadAvg_1M
2022-08-01 21:32:21 +05:00
Konstantin Lebedev
ab50f21ad9
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 21:16:57 +05:00
Konstantin Lebedev
2f85e31f5c
Merge branch 'master' into gentle_vacuum
2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef
remove percent
2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083
...
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 08:23:10 -07:00
Konstantin Lebedev
78cbd8002f
revert Sleep
2022-08-01 20:21:23 +05:00
Konstantin Lebedev
cd5c7ad052
move to github.com/seaweedfs/seaweedfs
2022-08-01 16:36:32 +05:00
Konstantin Lebedev
634bc97840
fix conflicts
2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
Konstantin Lebedev
c0d92f61a1
comment
2022-08-01 14:40:42 +05:00
Konstantin Lebedev
1d29f67c02
revert disk stats
2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3
rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 12:51:41 +05:00
chrislu
036566629a
filer.sync: fix synchronization logic in active-active mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3328
2022-08-01 00:06:18 -07:00
chrislu
303bd067b5
Revert "rename: delete source entry metadata only, skipping hard links"
...
This reverts commit 03466f955ehttps://github.com/seaweedfs/seaweedfs/issues/3386
2022-07-31 22:51:41 -07:00
chrislu
03c6f978f4
3.19
2022-07-31 17:16:51 -07:00
chrislu
afef014832
scaffold
2022-07-31 13:23:44 -07:00
Chris Lu
0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy
...
Add healthy indicator for raft status
2022-07-30 11:55:30 -07:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status
2022-07-30 19:34:26 +02:00
chrislu
cb096d1f53
minor
2022-07-30 09:52:52 -07:00
chrislu
0423d50106
wip note
2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a
moving
2022-07-29 23:39:11 -07:00
chrislu
13b9a52f80
fix deprecated functions
2022-07-29 01:34:39 -07:00
chrislu
be0ec7bc2f
fix test
2022-07-29 01:15:26 -07:00
chrislu
1f161f53dc
add back rack
2022-07-29 01:15:19 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
chrislu
de3afd4199
fix rebasing
2022-07-28 23:25:46 -07:00
chrislu
06cd491abc
add message batch
2022-07-28 23:24:38 -07:00
chrislu
7576c244c4
add flatbuffer serde for message
2022-07-28 23:24:38 -07:00
chrislu
a3fae7cd16
add an empty placeholder for mq.topic.list
2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47
segment serde
2022-07-28 23:24:38 -07:00
chrislu
74f60f246f
dynamically connect to a filer
2022-07-28 23:24:38 -07:00
chrislu
9f479aab98
allocate brokers to serve segments
2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0
refactor
2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c
clean up
2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5
remove old code
2022-07-28 23:24:38 -07:00
chrislu
f25e273e32
display data center and rack in cluster.ps
2022-07-28 23:22:52 -07:00
chrislu
68065128b8
add dc and rack
2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd
mq broker adds data center and rack
2022-07-28 23:22:07 -07:00
chrislu
87ccef72db
cluster.ps can list brokers
2022-07-28 23:22:07 -07:00
chrislu
ac66227798
renaming
2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8
renaming
2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17
rename to SeaweedMQ
2022-07-28 23:22:06 -07:00
Chris Lu
3e7e922d16
Merge pull request #3378 from vanhtuan0409/change-put-policy
...
PutUserAction should completely replace identity actions
2022-07-28 19:43:25 -07:00
chrislu
89e276de64
fix loop
2022-07-28 18:06:49 -07:00
chrislu
a458bee9f2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-28 16:32:02 -07:00
chrislu
e0eda52c54
mount: ensure symlink parent directory is tracked
...
fix https://github.com/chrislusf/seaweedfs/issues/3373
2022-07-28 16:32:00 -07:00
Tuan Vuong
1d9a888be9
PutUserAction should completely replace identity actions
2022-07-28 17:02:56 +07:00
zzq09494
01b7aa79fa
fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted
2022-07-28 12:04:37 +08:00
chrislu
ca836568ac
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87
"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"
2022-07-27 12:12:40 -07:00
Chris Lu
b7e4e26778
Merge pull request #3371 from askeipx/issues/3341
...
delete disk_size metrics when collection deleted
2022-07-27 11:25:42 -07:00
Konstantin Lebedev
7e09a548a6
exclude directories to sync on filer
2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev
ac5ce31278
leave notion to refactor after prometheus upgrade
2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev
47c72e6f35
remove all (currently existing) collection volume metrics
2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c
delete disk_size metrics when collection deleted
2022-07-27 15:02:41 +05:00
chrislu
f401b996eb
mount: fix symlink size reporting
2022-07-26 23:12:25 -07:00
Konstantin Lebedev
c88ea31f62
fix RUnlock of unlocked RWMutex
2022-07-26 12:57:07 +05:00
Konstantin Lebedev
da9d3e8f6c
refactor
2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4
fix logic else brake
2022-07-26 11:47:11 +05:00
chrislu
475185fb72
3.18
2022-07-25 22:41:38 -07:00
chrislu
ec0edb1ac4
filer: fix wrong logic during read
2022-07-25 22:40:00 -07:00
guosj
cc7a4b0a6e
correct comment
2022-07-25 11:46:41 +08:00
chrislu
33cff61992
fix glog test
2022-07-24 19:13:54 -07:00
chrislu
dbe4849ffd
3.17
2022-07-24 18:46:50 -07:00
chrislu
b98c0b4bf4
disable symlink translation for now
2022-07-24 18:46:32 -07:00
chrislu
2dc5d4adac
disable symlink translation for now
2022-07-24 18:24:34 -07:00
chrislu
3d2bbf96d2
adjust glog format
2022-07-24 17:29:08 -07:00
chrislu
2807fe08fc
adjust glog format
2022-07-24 16:44:58 -07:00
chrislu
2b64916d3e
set status to fuse.OK
2022-07-24 15:30:55 -07:00
chrislu
7386cf6a06
mount: resolve symlink
2022-07-24 15:11:24 -07:00
chrislu
40089b6753
maintain node lookup counter
2022-07-24 13:43:11 -07:00
chrislu
d24db396cc
fix removing path from inode2path
2022-07-23 18:20:29 -07:00
chrislu
41eeb4deef
do not add new inode during link
2022-07-23 13:37:07 -07:00
chrislu
02c2d81cde
fix removePathFromInode2Path
2022-07-23 13:21:42 -07:00
chrislu
4d5dc55766
mount: better support hard links
2022-07-23 12:57:00 -07:00
chrislu
ebda0689ab
use linux default max name length
2022-07-23 10:51:06 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
2c8818351f
mount: comment out stuck fuse inode notify during renaming
2022-07-22 18:17:34 -07:00
chrislu
a5c94acd4f
mount: check possible nil entry case
2022-07-22 15:33:37 -07:00
Chris Lu
0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap
...
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
2022-07-22 02:57:50 -07:00
LHHDZ
994a2dec78
fix vid_map_test tests
2022-07-22 17:54:57 +08:00
LHHDZ
58c02d6429
Solve the problem that LookupFileId lookup urls is empty due to leader switching
...
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
ningfd
58f2dd6740
change topology mapping definition, improve spelling
2022-07-22 17:18:56 +08:00
ningfd
6f882eb354
optimiz: master ui will render data in order
2022-07-22 14:37:49 +08:00
chrislu
7a6c559ab4
fix Change replication via volume.configure.replication by collection
...
fix https://github.com/chrislusf/seaweedfs/issues/3346
2022-07-21 22:01:05 -07:00
chrislu
fc8241fb5e
leveldb3: add instant dropping bucket
2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44
explicit bucket aware declaration
2022-07-21 18:23:53 -07:00
Konstantin Lebedev
3c42814b58
avoid deadlock
2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb
use safe onPeerUpdateDoneCns
2022-07-21 15:51:14 +05:00
chrislu
6147b61b0a
mount: fix concurrent map read and map write
...
fix https://github.com/chrislusf/seaweedfs/issues/3344
2022-07-21 01:38:26 -07:00
Chris Lu
9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume
2022-07-20 09:06:06 -07:00
Konstantin Lebedev
7875470e74
onPeerUpdateGoroutineCount use int32
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
c5189c343b
remove ticker
...
update the topology before each file
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
e2d991d8d0
ticker.Stop
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
de4fcc0e2c
sync update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
fa88dff7ce
update otherNodes
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
72dca31cfa
fix update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
884ffbafee
clouse background update
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
39eaf426f8
fix TestVolumeServerEvacuate
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b5e5f6f55a
update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
867269cdcf
help rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6f764e1014
volume server evacuate from rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
ba0e3ce5fa
volume server evacuate to target server
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b6471ecd75
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7
fix design
2022-07-20 18:08:12 +05:00
Tuan Vuong
312e13416b
filter duplicated action
2022-07-20 16:43:50 +07:00
guol-fnst
ac694f0c8f
rename parameter and reuse functions
...
rename milestone to watermark
2022-07-20 17:00:40 +08:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer
...
volume server evacuate to target server
2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03
remove ticker
...
update the topology before each file
2022-07-20 00:54:23 +05:00
Konstantin Lebedev
11e393dbe7
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev
f6a966b4fc
add waiting log message
2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849
check for ping before deleting raft server
...
https://github.com/chrislusf/seaweedfs/issues/3083
2022-07-20 00:04:12 +05:00
chrislu
2ae3f812f8
minor
2022-07-19 11:43:31 -07:00
Chris Lu
9667588af0
Merge pull request #3318 from kmlebedev/issues/3310
...
Use fallback if urls are not found
2022-07-19 11:38:43 -07:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove
...
raftServerRemoval typo and logs
2022-07-19 11:32:17 -07:00
guol-fnst
91285bb51d
remove uncessary code
2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34
optimiz
2022-07-19 16:58:53 +08:00
Konstantin Lebedev
f419d5643a
fix typo
...
add remove logs
2022-07-19 11:50:52 +05:00
guol-fnst
bec9f79659
use snapshot
2022-07-19 11:14:41 +08:00
Konstantin Lebedev
d422e7769c
ticker.Stop
2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b
sync update topologyInfo
2022-07-18 16:27:02 +05:00
chrislu
56ec89625a
3.16
2022-07-18 00:53:05 -07:00
Konstantin Lebedev
2b4112e462
update otherNodes
2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d
fix update topologyInfo
2022-07-18 01:46:31 +05:00
Konstantin Lebedev
7b1497ee63
Use BackoffSchedule for getLookupFileId
2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8
Use fallback if urls are not found
2022-07-15 15:29:15 +05:00
Konstantin Lebedev
4d5144e50d
clouse background update
2022-07-15 13:51:08 +05:00
chrislu
1db012485f
mount: fix bug during busy writes
...
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4
more logging related to filer metadata subscription
2022-07-15 00:42:13 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag
...
File upload should succeed should return `ETag` instead of `Etag`
2022-07-14 15:35:41 -07:00
chrislu
fa61074513
add clientId logging
2022-07-14 12:27:34 -07:00
chrislu
fbd8f868a1
filer may have trouble to re-connect
...
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
2022-07-14 12:15:31 -07:00
LHHDZ
c08f6d0ff7
File upload should succeed should return ETag instead of Etag
...
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
Chris Lu
957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume
...
optimiz concurrency
2022-07-13 23:12:24 -07:00
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2
optimiz concurrency
...
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak
2022-07-13 09:23:37 -07:00
guosj
354b7bdff0
replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()
2022-07-13 22:49:03 +08:00
chrislu
289402a741
writer pattern: similar changes to reader pattern
2022-07-13 02:30:44 -07:00
guosj
ab1b9697e6
supplement check duplicate accesskey
2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0
detect sequential mode with a counter
2022-07-13 02:20:03 -07:00
chrislu
578c8aea0e
typo
2022-07-13 02:18:31 -07:00
chrislu
1451b389a4
Avoid fatal error: sync: Unlock of unlocked RWMutex
...
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
石昌林
392da4e038
Make the prompt information clearer
2022-07-13 11:39:45 +08:00
石昌林
ad5e8f68ec
Check whether there is a duplicate accessKey when modifying iam
2022-07-13 11:36:11 +08:00
Chris Lu
fba5219dab
Merge pull request #3303 from guol-fnst/loading_volume
...
speeding up loading volumes
2022-07-12 03:01:24 -07:00
Konstantin Lebedev
6622240df7
fix TestVolumeServerEvacuate
2022-07-12 14:56:34 +05:00
chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
Konstantin Lebedev
8372721a62
update topologyInfo
2022-07-12 13:47:21 +05:00
guol-fnst
26313060a3
speeding up loading volumes
2022-07-12 16:19:33 +08:00
Konstantin Lebedev
ee95d23a22
help rack
2022-07-12 11:56:58 +05:00
Konstantin Lebedev
087fa1347f
volume server evacuate from rack
2022-07-12 11:33:08 +05:00
Chris Lu
de97ba94c3
Merge pull request #3302 from guo-sj/fix_put_user_policy
2022-07-11 21:15:37 -07:00
guosj
b3076909d1
refactor code
2022-07-12 11:25:36 +08:00
Konstantin Lebedev
4236c36599
volume server evacuate to target server
2022-07-11 16:58:15 +05:00
Chris Lu
7929a50327
Merge pull request #3296 from guo-sj/fix_put_user_policy
...
refine PutUserPolicy
2022-07-11 01:49:25 -07:00
guosj
7dbf19d09a
refine PutUserPolicy
2022-07-11 16:38:41 +08:00
chrislu
37578929d4
fix compilation
2022-07-11 01:30:00 -07:00
chrislu
93ccc6e05f
3.15
2022-07-11 00:50:07 -07:00
chrislu
f87952a411
remove locks
2022-07-11 00:25:20 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
3d0defa663
fix compilation
2022-07-08 19:16:58 -07:00
chrislu
6f00c7f061
comments
2022-07-08 19:07:31 -07:00
chrislu
d685b94102
mount: rename also invalidate source inode
2022-07-08 19:07:03 -07:00
chrislu
48382676d2
fix filtering by volume id
2022-07-08 10:29:24 -07:00
liubaojiang
4e83e92156
add subscriber clientId if it is the first time
2022-07-08 20:27:25 +08:00
guosj
5b3192344d
fix iam CreatePolicy error
2022-07-08 17:42:21 +08:00
chrislu
e2ecf137a2
fix resetting fh.reader
2022-07-08 01:04:15 -07:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
b7de5c6c43
shell: remove unused filer parameter from shell.toml
2022-07-07 15:01:23 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
chrislu
05c53820b9
mount: file handler release memory
2022-07-07 11:49:40 -07:00
chrislu
69ef6459c3
S3 ListObjectsV2 fix pagination under bucket root
...
fix https://github.com/chrislusf/seaweedfs/issues/3166
2022-07-06 02:28:34 -07:00
Konstantin Lebedev
785223e587
rabbitpubsub enable durable
2022-07-06 10:05:29 +05:00
chrislu
57975b93ff
adjust used size reporting
2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5
adjust used size reporting
2022-07-04 13:46:32 -07:00
chrislu
3c79c77056
3.14
2022-07-04 13:15:49 -07:00
guol-fnst
5bc895673e
add getting bucket quota
2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610
fix usage
2022-07-04 11:31:39 +08:00
chrislu
141f662734
edge case checking when volume server does not have capacity to balance
...
fix https://github.com/chrislusf/seaweedfs/issues/3257
2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798
fix tests
2022-07-01 12:09:42 -07:00
Chris Lu
ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255
...
volumeServer.evacuate fix select empty node
2022-07-01 02:36:30 -07:00
Konstantin Lebedev
3d5f1cbc5e
fix pagination
2022-07-01 14:17:49 +05:00
chrislu
06a8b174b5
also remove Sync() for idx file
2022-06-30 13:50:53 -07:00
chrislu
52580743b9
stuck with file.Sync()
...
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
2022-06-30 12:32:55 -07:00
Konstantin Lebedev
5ed8165161
fix logic
...
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
2022-06-30 17:58:24 +05:00
石昌林
200a862701
fixed volume xx not found caused by missing VolumeLocation events
...
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
Chris Lu
4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta
...
[Critical] Avoid synchronizing from scratch due to err
2022-06-29 12:36:28 -07:00
Chris Lu
b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder
...
use const multipart uploads folder
2022-06-29 12:09:03 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
Konstantin Lebedev
bcbdc4cb37
use const multipart uploads folder
...
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu
a8abab2412
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-29 02:44:14 -07:00
chrislu
449ecc8890
remove max connection age
...
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy
...
validate tags on copy object and add regex for validating tags
2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535
remove println
2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2
fix return
2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c
fix return
2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913
fix parseTagging function and add test for validate tags
2022-06-28 14:48:55 +03:00
Chris Lu
a79bf9d92a
Merge pull request #3252 from guol-fnst/update_tbk
...
remove hdfs, add tikv for full version update
2022-06-27 22:37:41 -07:00
guol-fnst
74c9f39d48
remove hdfs, add tikv for full version update
2022-06-28 12:35:07 +08:00
石昌林
e1b94eb6b9
fix error: concurrent map writes when add or remove cluster node
2022-06-28 10:34:59 +08:00
chrislu
4b1f48a399
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-27 12:40:50 -07:00
chrislu
4a65159250
fix reading time
2022-06-27 12:40:47 -07:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files
...
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
2022-06-27 20:19:21 +02:00
Chris Lu
4be406a095
Merge pull request #3249 from kmlebedev/issues_2589
...
avoid set currentMaster svc.local k8s discovery service domains
2022-06-27 11:05:07 -07:00
Konstantin Lebedev
9bff097b01
currentMaster redirected
2022-06-27 21:55:16 +05:00
Konstantin Lebedev
6c20a3b622
avoid set currentMaster k8s svc.local discoveruy service domains
...
https://github.com/chrislusf/seaweedfs/issues/2589
2022-06-27 21:47:05 +05:00
Lapshin Vitaliy
1a5981d583
fix test
2022-06-27 18:46:21 +03:00
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
2022-06-27 22:04:50 +08:00
Lapshin Vitaliy
d7c3493d15
Merge branch 'master' into bufix/validate-tags-on-copy
2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a
adjust log message
2022-06-26 23:12:16 -07:00
Chris Lu
fdd880b111
Merge pull request #3237 from shichanglin5/remove_query_collection
...
Remove the collection query param of s3api and let the collection be …
2022-06-26 23:02:11 -07:00
石昌林
15bfc60a88
Remove the collection query param of s3api and let the collection be determined by the filer
2022-06-27 13:47:53 +08:00
yulai.li
af23e63e3f
Improve filer command help, add supported filer store list
2022-06-27 12:09:16 +08:00
chrislu
40a9634c3d
3.13
2022-06-26 20:13:52 -07:00
Chris Lu
cc01f71752
Merge pull request #3233 from blacktear23/filerstore-tikv
...
Add TiKV Filerstore back
2022-06-26 20:12:18 -07:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
chrislu
9dfea44aeb
fix unit
2022-06-26 12:21:38 -07:00
chrislu
3142e77eca
mount: stats report physical size instead of logical size
2022-06-26 12:14:34 -07:00
yulai.li
13159ad0d8
Merge branch 'master' into filerstore-tikv
2022-06-26 22:44:01 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
chrislu
f77eda4b4f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-25 09:22:52 -07:00
chrislu
3ce46c3566
fix entry full url lookup
2022-06-25 09:22:49 -07:00
Chris Lu
dc59ccd110
Merge pull request #3228 from shichanglin5/fix_volumeNotFound
...
When the connection with the leader is disconnected, the vidMap shoul…
2022-06-24 23:06:13 -07:00
石昌林
f1ea906c13
Code comment optimization
2022-06-25 11:56:09 +08:00
石昌林
0c8e033e68
When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader
2022-06-24 17:41:46 +08:00
chrislu
9c517d2b35
masterclient: fallback to directly querying master in case of missing volume id location
2022-06-24 02:08:57 -07:00
Chris Lu
280e33092c
Merge pull request #3226 from kmlebedev/advancedtls
...
Advancedtls
2022-06-24 01:34:58 -07:00
chrislu
509a9047db
test compact map with snowflake sequencer
2022-06-23 21:45:51 -07:00
chrislu
217d7dc377
verify seq is always unique
2022-06-23 21:22:05 -07:00
Konstantin Lebedev
2e782dfdcb
rm defer comments
2022-06-24 01:50:13 +05:00
Konstantin Lebedev
7efaafb0ee
fix Authenticate
2022-06-24 01:42:04 +05:00
Konstantin Lebedev
b0aa51d7ef
enable require client cert
2022-06-24 00:29:23 +05:00
Konstantin Lebedev
ea7cdb8b0e
seperate option
2022-06-24 00:26:56 +05:00
Konstantin Lebedev
5b388ed6c1
initial advancedtls
2022-06-23 23:32:15 +05:00
chrislu
0c13fc873c
fix compilation
2022-06-23 00:45:11 -07:00
chrislu
96496d5286
master: broadcast new volume locations to clients to avoid possible racing condition
...
fix https://github.com/chrislusf/seaweedfs/issues/3220
2022-06-23 00:41:33 -07:00
chrislu
52c44d646e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-22 00:34:27 -07:00
chrislu
530da0cc89
remove unused code
2022-06-22 00:34:25 -07:00
famosss
8a6e7a7df8
Merge branch 'chrislusf:master' into master
2022-06-22 09:58:56 +08:00
zzq09494
9df5ad5309
fix: vacuum create a lot of connections quickly
2022-06-22 09:57:22 +08:00
Chris Lu
d652ce1a36
Merge pull request #3216 from kmlebedev/mk_raft_basedir
...
avoid no such raft date directory
2022-06-21 12:21:30 -07:00
chrislu
6b3beb709d
fix format
2022-06-21 12:21:02 -07:00
Konstantin Lebedev
c1c8dad677
avoid no such raft date directory
...
https://github.com/chrislusf/seaweedfs/issues/3214
2022-06-21 13:47:51 +05:00
chrislu
4dc27e1ed5
3.12
2022-06-20 19:07:00 -07:00
chrislu
492da3dbce
master: put metadata under instance specific folder
2022-06-20 19:04:49 -07:00
chrislu
41dfe27102
Revert "remove max connection age"
...
This reverts commit b9b684194f
2022-06-20 14:49:39 -07:00
chrislu
2295766e2c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56
remote storage: stop supporting hdfs as a remote storage
2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page
2022-06-20 22:36:07 +02:00
chrislu
bf5a5abfb1
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 13:18:11 -07:00
chrislu
69a2e503ab
non-recursive directory deletion
...
fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.
2022-06-20 13:18:08 -07:00
chrislu
76f26c4b99
mount: delete target entry first during move
2022-06-20 13:15:02 -07:00
Chris Lu
460c7cf68e
Merge pull request #3203 from shichanglin5/volume_copy_preallocate
2022-06-20 07:41:45 -07:00
石昌林
81f7f08708
Determine whether to preallocate according to the master configuration before executing copy volume
2022-06-20 21:12:44 +08:00
石昌林
be5c901bea
Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems
2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48
validate tags on copy object and add regex for validating tags
2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode
...
AbortMultipartUploadHandler should return 204 instead of 200
2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0
AbortMultipartUploadHandler should return 204 instead of 200
...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
2022-06-20 13:26:22 +08:00
石昌林
9e036df356
remove go.uber.org/atomic
2022-06-20 12:35:29 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
...
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
...
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4
fix path bug on windows
...
fix usage
2022-06-16 17:24:56 +08:00
chrislu
aaa9938353
Revert "adjust conditions"
...
This reverts commit e024586ff1
2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97
remove -output, add -dir,-name for more flexibility
2022-06-16 16:07:32 +08:00
chrislu
e024586ff1
adjust conditions
2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
...
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e
add condition when inFlightUploadDataLimitCond signal
2022-06-16 14:07:11 +08:00
zzq09494
62f74f5e3c
add bucket label to s3 prometheus metrics
2022-06-16 13:21:25 +08:00
guol-fnst
4bc8792643
support specific version of weed
2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29
remove unused function
...
remove log when weed is up to date
add check whether weed is full version
2022-06-16 11:40:20 +08:00
liubaojiang
3076ac101e
move vs.concurrentUploadLimit != 0 out of the lock
2022-06-16 09:58:44 +08:00
Chris Lu
097a12dfb4
Merge pull request #3187 from ningfdx/remote
2022-06-15 06:34:32 -07:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req
2022-06-15 06:33:33 -07:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
ningfd
338705f375
fix(wdclient): GetLocations return
2022-06-15 19:20:13 +08:00
Konstantin Lebedev
31a929f7f2
logging processRangeRequest errors
2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149
2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37
Fix: provide filer.ui.deleteDir options for server command
2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update
2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555
fix minor bug
2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6
fix(filer.sync): modify clientName format : from -> to
2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87
Change ui.deleteDir default to true
2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b
feat(filer.sync): add metricsServer in filer.sync.
...
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
2022-06-15 11:33:18 +08:00
chrislu
d4ef06cdcf
3.11
2022-06-14 12:14:52 -07:00
chrislu
b347b2fb54
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-14 11:16:51 -07:00
chrislu
36d2756978
volume: avoid writing too much data for range requests in large files
...
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
yulai.li
28cda854eb
Add filer command line parameter to let Filer UI show/hide directory delete button
2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea
feat(filer.sync): add offset to path.
2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0
add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate "
2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372
add update feature
2022-06-14 19:01:57 +08:00
Chris Lu
b22ca85fbb
Merge pull request #3169 from shichanglin5/fix_auth
...
fix: When there is no access permission configured before startup, th…
2022-06-12 23:40:42 -07:00
shichanglin5
f01dd27752
fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup
2022-06-13 13:23:26 +08:00
chrislu
e437ec5a68
3.10
2022-06-12 19:30:27 -07:00
chrislu
d12f431d98
collect volume server status
2022-06-12 11:56:23 -07:00
chrislu
444ac21050
go fmt
2022-06-11 09:51:11 -07:00
chrislu
a74e926063
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-11 09:51:02 -07:00
chrislu
61b8c9c361
remote object store gateway: disable tagging for backblaze
2022-06-11 09:50:59 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix
2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb
Bug Fix: force reload filer page when upload/delete/rename/create directory
2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64
fix typo
2022-06-10 22:03:11 +08:00
Chris Lu
36757d28a8
Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch
...
stats master_replica_placement_mismatch
2022-06-10 06:44:16 -07:00
chrislu
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
...
This reverts commit 37da68931900d53c34c4
2022-06-10 06:38:17 -07:00
Konstantin Lebedev
fb57aa431c
stats master_replica_placement_mismatch
2022-06-10 15:30:40 +05:00
shichanglin5
a63559ee6c
fix: fix command prompt information
2022-06-10 12:46:10 +08:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID
...
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
...
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
2022-06-09 20:41:16 +08:00
zhihao.qu
cd5cca36a4
feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges
2022-06-09 10:53:19 +08:00
Chris Lu
4a5135961f
Merge pull request #3154 from kmlebedev/s3_test_invalid_range
2022-06-08 01:45:40 -07:00
Konstantin Lebedev
49f058da85
fix s3 tests:
...
ranged_request_invalid_range
ranged_request_empty_object
2022-06-08 13:25:13 +05:00
chrislu
8a1166c021
auto remove previous leftover socket
2022-06-08 00:32:09 -07:00
chrislu
633a4aed18
filer: fix customizable local socket file name
...
fix https://github.com/chrislusf/seaweedfs/issues/3147
2022-06-08 00:15:28 -07:00
chrislu
acbc0286e2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-07 22:33:38 -07:00
chrislu
ebd6097e6f
fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074
...
related to https://github.com/chrislusf/seaweedfs/pull/3074
2022-06-07 22:33:35 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f
fix s3 tests
...
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
chrislu
78be3b0f6c
fix compilation
2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
chrislu
cbf46de5f4
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-06 00:15:40 -07:00
chrislu
8402af0882
filer: detect ttl based on fs configure setting
2022-06-06 00:15:37 -07:00
ningfd
f32142f6f5
add disableXAttr in mount option
2022-06-06 14:09:01 +08:00
chrislu
4a046e4de7
3.09
2022-06-05 19:00:24 -07:00
chrislu
11f1e7996b
fix test
2022-06-05 18:50:41 -07:00
chrislu
d65bb2c6df
mount: file handle locks entry better
...
related to https://github.com/chrislusf/seaweedfs/issues/2952
2022-06-05 18:15:06 -07:00
chrislu
746092a60b
add a comment
2022-06-05 16:44:07 -07:00
chrislu
056c480eb0
volume: checksum remove the hashing step in Value()
2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc
stream read large files
2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f
remove unused logic
2022-06-04 19:11:06 -07:00
chrislu
685643d506
needle read into a writer
2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23
add needle reading in chunks
2022-06-04 18:15:39 -07:00
chrislu
18b29f709c
refactoring, split file
2022-06-03 23:31:14 -07:00
chrislu
39c3cb04af
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-03 08:45:32 -07:00
chrislu
f97acdd489
volume.fix.replication fix retry logic
...
fix https://github.com/chrislusf/seaweedfs/issues/3136
2022-06-03 08:45:29 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d
fix show empty dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-06-01 20:10:52 +05:00
chrislu
6793bc853c
help message when in simulation mode
2022-05-31 14:48:46 -07:00
chrislu
89948a373b
fix error reporting for "Need to a hdd disk type"
...
related to https://github.com/chrislusf/seaweedfs/issues/3128
2022-05-31 12:43:55 -07:00
chrislu
8a49240d64
3.08
2022-05-31 11:57:41 -07:00
chrislu
376b005ad3
fix wrong logic
2022-05-31 11:55:58 -07:00
chrislu
fbd99d53c1
3.07
2022-05-31 00:36:24 -07:00
chrislu
27732ecfa4
move s3 related constants from package http to s3_constants
2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 22:47:29 -07:00
chrislu
596c3860ca
use final destination to resolve fs configuration
...
related to https://github.com/chrislusf/seaweedfs/issues/3075
2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers
...
Filer bootstrap from peers
2022-05-30 21:38:31 -07:00
chrislu
6adc42147f
fresh filer store bootstrap from the oldest peer
2022-05-30 21:27:48 -07:00
chrislu
490e0850bf
filer.meta.tail add one example of using untilTimeAgo
2022-05-30 20:24:59 -07:00
liubaojiang
f0ee3e6f21
reduce the scope of inFlightUploadDataLimitCond lock
2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3
refactor
2022-05-30 16:28:36 -07:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
968ca95b49
filer.meta.tail: support untilTimeAgo for a range
2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5
stop when in memory log is done
2022-05-30 15:25:21 -07:00
chrislu
aece35a64f
stop when on disk log is done
2022-05-30 15:20:51 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
8902fa6ff6
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 13:28:32 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud
...
Create folders in s3 via nextcloud
2022-05-30 13:10:10 -07:00
chrislu
5b9347c938
typo
2022-05-30 12:37:41 -07:00
Konstantin Lebedev
af562e1a20
rm log info
2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c
list self dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961
show fake dirs
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46
s3: keep auth enabled in case identities are set to empty
...
fix https://github.com/chrislusf/seaweedfs/issues/3084
2022-05-29 19:00:03 -07:00
chrislu
81d6159290
volume: report error if a volume has nil data backend
...
fix https://github.com/chrislusf/seaweedfs/issues/3105
2022-05-29 16:59:30 -07:00
chrislu
ca16fbf0ef
shell: cluster.ps display filer group
2022-05-29 16:37:14 -07:00
chrislu
9219664505
reset sync offset if peer filer resets
...
fix https://github.com/chrislusf/seaweedfs/issues/3089
2022-05-29 13:11:24 -07:00
Konstantin Lebedev
54356211c2
list empty folder
2022-05-27 13:17:18 +05:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator.
2022-05-25 01:11:02 -07:00
chrislu
5b8b022985
remove unused parameter
2022-05-24 18:52:04 -07:00
Robert Coelho
0e6e72d462
mount: ReadDir return EIO on EnsureVisited err
2022-05-24 13:02:59 -04:00
Robert Coelho
1fabbe8a25
mount: cleanup ReadDir rewrite's branches to not assume offsets
2022-05-24 12:48:32 -04:00
Robert Coelho
cb422d96f7
mount: rewrite ReadDir to respect input.Offset to fix partial results
2022-05-24 12:16:58 -04:00
Robert Coelho
05602167f4
mount: do not mark meta_cache when there is an err
2022-05-24 12:13:15 -04:00
chrislu
7e25a2d416
reduce busy waiting when reading metadata logs
2022-05-24 00:23:53 -07:00
chrislu
2f846777bb
3.06
2022-05-23 01:21:48 -07:00
chrislu
a5cdcc7e63
fix reading files larger than 20GB due to manifest resolving bug
2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd
fix reading files larger than 20GB due to manifest resolving bug
...
this is introduced since Mar 08 2022
2022-05-23 01:14:56 -07:00
chrislu
acc318e12b
3.05
2022-05-22 21:43:59 -07:00
liubaojiang
076e48a676
add inflight upload data wait timeout
2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e
add inFlightUploadDataLimitCond signal
2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c
exclude replication from the concurrentUploadLimitMB
2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync
2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9
hotfix_fsync fix fsync
2022-05-19 23:17:17 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
...
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879
revert changes of pb file
...
orignal protoc-gen-go-grpc verion is 1.1.0
2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd
just exit in case of duplicated volume directories were loaded
2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775
rename UUID file
...
fix typo
move locationUUID into DiskLocation
2022-05-17 15:41:47 +08:00
Chris Lu
72e7dcde51
Merge pull request #3060 from natmaka/master
2022-05-16 07:42:58 -07:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around
2022-05-16 22:11:33 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
Konstantin Lebedev
44f53ceda6
fix collectionIsMismatch charset
2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2
fix skip loop
2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c
avoid delete volume replica if collection mismatch
2022-05-16 13:07:05 +05:00
chrislu
9ff0d99002
3.04
2022-05-15 21:32:21 -07:00
chrislu
866fed1eab
avoid grpc name conflict
...
fix https://github.com/chrislusf/seaweedfs/issues/3055
2022-05-15 21:31:29 -07:00
chrislu
cd2d10118c
3.03
2022-05-15 18:17:07 -07:00
chrislu
271b5aed96
shell: volume.fsck add a note for -reallyDeleteFromVolume option
2022-05-15 11:07:04 -07:00
chrislu
4076d74813
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-15 00:43:39 -07:00
chrislu
29198720f2
s3: add grpc server to accept configuration changes
2022-05-15 00:43:37 -07:00
Chris Lu
ea4918394c
Merge pull request #3047 from shichanglin5/master
...
fix the problem of metadata and tagging loss when files are copied
2022-05-13 22:52:48 -07:00
shichanglin5
688d55488c
test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go
2022-05-14 10:40:29 +08:00
chrislu
735038b2c1
backup do not need to use preallocation
...
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
shichanglin5
1166dead00
fix the problem of metadata and tagging loss when files are copied
...
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
2022-05-13 19:46:20 +08:00
