Commit Graph

803 Commits

Author SHA1 Message Date
chrislu
aed2a18223 get, set and add path conf 2024-07-20 11:27:11 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
Dan
794a5aa67d
Add warning with hints to why a EC shard cant be moved (#5790) 2024-07-15 11:17:22 -07:00
chrislu
182f6d00a9 always copy ecj files 2024-07-10 01:25:57 -07:00
rhysm
7ee1f520a4
volume.tier.upload: Fix deleting replicated volumes (#5743)
Co-authored-by: Rhys M <rhysm@protonmail.com>
2024-07-03 15:51:25 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Konstantin Lebedev
34165013c8
chore: hot fix panic fs.verify (#5696) 2024-06-20 15:24:31 -07:00
Konstantin Lebedev
7988ee0805
[fs.verify] skip failed files if entry not found on filerStore (#5693) 2024-06-19 06:50:55 -07:00
Konstantin Lebedev
ece018b1a9
[fs.verify] getmeta data from filer log (#5678)
* initial

* fix: print entry path
2024-06-18 22:56:47 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628)
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
Taehyung Lim
4744889973
fix issue: sometimes volume.fsck report 'volume not found' (#5537)
* fix issue: sometimes volume.fsck report 'volume not found' when a volume server has multiple disk types

* rename variable

* adjust counters

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
2024-06-11 22:22:57 -07:00
NyaMisty
579ebbdf60
Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed (#5636) 2024-06-02 14:25:42 -07:00
NyaMisty
0c62d591e2
Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) (#5635) 2024-06-02 14:16:05 -07:00
NyaMisty
e1c671c028
Fix volume.tier.upload nil pointer panic (#5634) 2024-06-02 14:15:39 -07:00
Vlad
72f7b4e9b1
Replaces volume.unmount in the help of the volumeServer.leave (#5619) 2024-05-27 01:13:06 -07:00
Chris Lu
cfcf383ee7
fix wrong prefix example 2024-05-21 21:45:29 -07:00
Vlad
a9b057389f
chore: add missing concurrency help in fs.meta.load command (#5609) 2024-05-21 21:42:48 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
skycope
6e4b9181f5
fix "volume.fix.replication" move many replications only to one volumeServer (#5522) 2024-04-23 06:33:50 -07:00
chrislu
7a72677f98 add test 2024-04-12 01:31:23 -07:00
steve.wei
67ead9b18f
fix(volume.fix.replication): adjust volume count, not free volume count (#5479) 2024-04-08 07:30:04 -07:00
chrislu
ca042bd067 simplify 2024-03-17 01:01:45 -07:00
chrislu
4dce1e9d29 Revert "remove github.com/golang/protobuf/proto"
This reverts commit 1516e0249f.
2024-03-17 01:00:43 -07:00
chrislu
1516e0249f remove github.com/golang/protobuf/proto 2024-03-16 23:18:11 -07:00
chrislu
8f79bb3987 Merge branch 'master' into mq-subscribe 2024-03-07 10:42:29 -08:00
chrislu
b544a69550 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-07 10:42:15 -08:00
chrislu
5c0f01c0eb purge cluster id 2024-03-07 10:42:12 -08:00
Konstantin Lebedev
872b5161e6
[shell] fs log purge (#5344)
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
2024-03-05 09:09:08 -08:00
chrislu
a52ce076fc Merge branch 'master' into mq-subscribe 2024-02-25 08:00:37 -08:00
Konstantin Lebedev
6181aa7594
fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Chris Lu
fa72761bb8 fix on finding lock owner 2024-02-05 17:30:14 -08:00
Chris Lu
53612b770c Merge branch 'master' into mq-subscribe 2024-02-04 10:44:01 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types (#5245)
* week/shell: Cluster check other disk types

The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.

This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Replace loop that copies slice

Use the following construct instead of a `for` loop:

```golang
x = append(x, y...)
```

See https://staticcheck.dev/docs/checks#S1011.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Check disk types when filer is in use

Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

---------

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2024-01-29 10:36:37 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic (#5241) 2024-01-25 06:19:58 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic (#5238) 2024-01-25 00:17:32 -08:00
chrislu
34a78ffad0 remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency (#4529)
* fix: increase speed cmd fs meta load

* fix: add wg
2023-12-28 11:54:37 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck (#5133)
* Add modifyTimeAgo to volume.fsck

* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler (#5132) 2023-12-23 09:18:16 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00
zehweh
d818ca9631 fix TestMisplacedChecking() and add test-case 2023-11-21 08:24:34 -08:00
Amir H. Yeganemehr
6f559fe838 fs.mergeVolumes: change dir to a named parameter 2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be Fix typo (dist => dest) 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3 fs.mergeVolumes: Make a plan based on volumes size 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
b05d682c48 make help command in shell case insensitive 2023-11-13 06:06:45 -08:00
jsh
47112917ff ec.decode: mount the collected ec shards 2023-11-10 00:04:42 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go (#4988) 2023-11-07 07:58:19 -08:00
chrislu
2bc05f70e7 log full percentage 2023-10-22 12:59:34 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file (#4881)
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero (#4909)
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 08:15:20 -07:00