Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic ( #5238 )
2024-01-25 00:17:32 -08:00
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
...
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency ( #4529 )
...
* fix: increase speed cmd fs meta load
* fix: add wg
2023-12-28 11:54:37 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck ( #5133 )
...
* Add modifyTimeAgo to volume.fsck
* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler ( #5132 )
2023-12-23 09:18:16 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
zehweh
d818ca9631
fix TestMisplacedChecking() and add test-case
2023-11-21 08:24:34 -08:00
Amir H. Yeganemehr
6f559fe838
fs.mergeVolumes: change dir to a named parameter
2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be
Fix typo (dist => dest)
2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3
fs.mergeVolumes: Make a plan based on volumes size
2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
b05d682c48
make help command in shell case insensitive
2023-11-13 06:06:45 -08:00
jsh
47112917ff
ec.decode: mount the collected ec shards
2023-11-10 00:04:42 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go ( #4988 )
2023-11-07 07:58:19 -08:00
chrislu
2bc05f70e7
log full percentage
2023-10-22 12:59:34 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file ( #4881 )
...
* fix: disallow file name too long when writing a file
* bool LongerName to MaxFilenameLength
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero ( #4909 )
...
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 08:15:20 -07:00
Amir H. Yeganemehr
3e8604a4c5
add fs.mergeVolumes command into shell ( #4907 )
...
* add fs.compact command into shell
* rename fs.compact to fs.mergeVolumes
2023-10-10 12:47:18 -07:00
Konstantin Lebedev
2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment ( #4893 )
...
* fix: skipping checking active volumes with the same number of files at the moment
https://github.com/seaweedfs/seaweedfs/issues/4140
* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140
* add TestShouldSkipVolume
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-09 09:57:26 -07:00
chrislu
0fd7222d65
default to skip if less than 4 nodes
2023-10-05 11:13:48 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg ( #4880 )
2023-10-02 08:29:09 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync ( #4875 )
...
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647
* Update weed/shell/command_volume_fix_replication.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix: revert this existing do option to positive
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-27 23:12:10 -07:00
chrislu
9d589b48e6
rename function
2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica ( #4873 )
...
https://github.com/seaweedfs/seaweedfs/issues/4647
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4
describe a topic
2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance ( #4864 )
...
* fix(ec): ignore 0 byte data files
refers to parts of #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
* fix(ec): ignore volumes not from the current collection during balance
fixes #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
---------
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b
Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs ""
...
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu
3cf9b8d621
create topic and report topic
2023-09-24 22:00:43 -07:00
chrislu
b3f94feede
do create topic
2023-09-24 22:00:43 -07:00
chrislu
0361c321b4
add CreateTopic API
2023-09-24 22:00:43 -07:00
chrislu
8cb42c39ad
Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs "
...
This reverts commit 2e5aa060264d414f54a2
2023-09-18 16:12:50 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 ( #4850 )
...
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone ) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases )
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md )
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0 )
---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* API changes
* go mod
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2023-09-18 14:43:05 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer ( #4767 )
2023-08-24 07:08:56 -07:00
SmsS4
1e72579ac7
Fix s3.clean.uploads ( #4766 )
2023-08-19 06:30:18 -07:00
chrislu
3365468d0d
added an error message
2023-08-08 20:35:21 -07:00
wusong
3ec977cd6c
minor ( #4723 )
2023-08-01 02:34:59 -07:00
chrislu
31b2751aff
clone volume locations in case they are changed
...
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-07-06 00:32:58 -07:00
chrislu
1ce725728d
minor
2023-06-19 18:43:41 -07:00
chrislu
8ec1bc2c99
remove unused cluster node leader
2023-06-19 18:19:13 -07:00
柏杰
0b0fb9b9e4
avoid data race read volume.IsEmpty ( #4574 )
...
* avoid data race read volume.IsEmpty
- avoid phantom read isEmpty for onlyEmpty
- use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope
* add Destroy(onlyEmpty: true) test
* add Destroy(onlyEmpty: false) test
* remove unused `IsEmpty()`
* change literal `8` to `SuperBlockSize`
2023-06-14 14:39:58 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty ( #4561 )
...
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
Konstantin Lebedev
5caff2c0f0
fix: deletes newly created volumes ( #4551 )
...
https://github.com/seaweedfs/seaweedfs/issues/4531
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-06 09:08:27 -07:00
mervynzhang
1ebb549f77
support swift ( #4480 )
2023-05-19 06:39:25 -07:00
SmsS4
17e91d2917
Use filerGroup for s3 buckets collection prefix ( #4465 )
...
* Use filerGroup for s3 buckets collection prefix
* Fix templates
* Remove flags
* Remove s3CollectionPrefix
2023-05-16 09:39:43 -07:00
themarkchen
7592d013fe
fix shell volume.balance bug ( #4447 )
2023-05-04 07:35:37 -07:00
chrislu
e1b92bc701
reset local file content if remote file is updated
...
fix https://github.com/seaweedfs/seaweedfs/issues/4289
2023-04-22 22:32:21 -07:00
Matt
da1d3b5a33
adjusted regex to be from 1 to 255 for the value ( #4377 )
...
* compatibility patch for csi driver
* added namespace to all component parameters
* added namespace to all component parameters
* dereference in range
* added namespace to values.yml defaults
* added namespace to s3 component
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* changed helm directory structure
* update charts location
* fixed dereference
* updated permissions
* updated permissions
* match current action schema
* added helm chart liniting CI
* modified chart list changed
* fixed nodejs warning
* standardized a few defaults
* added chart dirs
* lowerd period seconds for volume startup to test chart lint changes
* lowerd period seconds for volume startup to test chart lint changes
* test
* changed back
* adjustment
* debug ls statment
* removed change detection
* remvoed change detection
* always lint the charts
* added missing )
* fixed typo
* added spaces in front of all comments
* pdated values.yaml
* pdated values.yaml
* don't validate maintainers for now
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* - adds helm chart lint on changes
- adds test helm chart install on k8s
* updated helm chart readme.md
* added artifact hub
* added artifact hub
* added ttl validation
* adjusted regex to be from 1 to 255 for the value
* added better error message
* fixed regex
2023-04-10 12:24:38 -07:00
Matt
2d213d87e1
File Path Configuration TTL Validation ( #4376 )
...
* compatibility patch for csi driver
* added namespace to all component parameters
* added namespace to all component parameters
* dereference in range
* added namespace to values.yml defaults
* added namespace to s3 component
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* changed helm directory structure
* update charts location
* fixed dereference
* updated permissions
* updated permissions
* match current action schema
* added helm chart liniting CI
* modified chart list changed
* fixed nodejs warning
* standardized a few defaults
* added chart dirs
* lowerd period seconds for volume startup to test chart lint changes
* lowerd period seconds for volume startup to test chart lint changes
* test
* changed back
* adjustment
* debug ls statment
* removed change detection
* remvoed change detection
* always lint the charts
* added missing )
* fixed typo
* added spaces in front of all comments
* pdated values.yaml
* pdated values.yaml
* don't validate maintainers for now
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* - adds helm chart lint on changes
- adds test helm chart install on k8s
* updated helm chart readme.md
* added artifact hub
* added artifact hub
* added ttl validation
2023-04-04 22:07:17 -07:00
chrislu
f1f14e28bc
adjust name
2023-03-26 12:17:23 -07:00
chrislu
50dc2fe96b
cleaning variables
2023-03-19 18:48:40 -07:00
Konstantin Lebedev
29d1312625
Fs.verify.concurrency ( #4293 )
...
* init
* refactor
* refactor print msg
* param just concurrency
* fix race
* next fix race
* reset variables before each run
* clean
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-19 18:38:57 -07:00
chrislu
2268d38263
fix naming
...
fix https://github.com/seaweedfs/seaweedfs/issues/4305
2023-03-14 08:38:02 -07:00
chrislu
98bc93cf31
fix tests
2023-03-13 10:00:27 -07:00
Konstantin Lebedev
e17429223e
shell script unclean variables ( #4298 )
2023-03-13 07:16:31 -07:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs ( #4264 )
...
* fix
* fix mtime s > ns
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:24:36 -08:00
chrislu
214b7cd286
volume.fix.replication: adjust the retry checking times
2023-02-22 10:47:52 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency ( #4239 )
2023-02-21 19:47:33 -08:00
chrislu
5869945f16
avoid infinite loop
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904
2023-02-10 13:05:04 -08:00
chrislu
e037c71ec3
adjust text
2023-02-10 13:04:29 -08:00
chrislu
67b8c2853a
add line return
2023-02-10 12:53:43 -08:00
chrislu
302155a6fa
fix compilation
2023-02-10 12:32:59 -08:00
chrislu
5520b81551
add some error messages for args
...
related to https://github.com/seaweedfs/seaweedfs/issues/4115
2023-02-10 12:30:53 -08:00
chrislu
dc4ed2cd9b
do not move cloud tier volumes
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195
2023-02-09 22:17:54 -08:00
chrislu
31bb91583f
fix bug when vid not found
...
fix https://github.com/seaweedfs/seaweedfs/issues/4193
2023-02-09 17:30:44 -08:00
chrislu
5de93fe442
refactoring
2023-02-07 14:45:20 -08:00
Eric Yang
5083429704
ADHOC: s3 bucket name verification ( #4189 )
...
* ADHOC: s3 bucket name verification
* add test
2023-02-07 14:37:29 -08:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region ( #4177 )
...
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu
ef70415c70
skipping manifest chunk volume id change
2023-01-29 22:55:06 -08:00
chrislu
6e8dbf8868
add fs.meta.changeVolumeId
2023-01-29 22:51:58 -08:00
Guo Lei
b63d1e9c6b
fix index ( #4158 )
2023-01-28 06:55:27 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
chrislu
2452f3b2f7
sort chunks by offset
2023-01-12 20:29:07 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding ( #4092 )
2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file ( #4060 )
...
* fs.verify needles status of file
* fix parse args
* refactor logging
* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn ( #4024 )
2022-11-30 06:30:55 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
...
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
...
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
...
check needles status from volume server
2022-10-31 11:33:04 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
...
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
0d9f2f9e7a
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-28 12:53:23 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
...
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
