mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-19 05:27:50 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,194 Commits 32 Branches 296 Tags 152 MiB
c531e2eb0e
Commit Graph

11194 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
coffeecloudgit
c531e2eb0e
Update disk_location_ec.go (#6359) 
Add lock for l.ecVolumes
 2024-12-15 22:05:47 -08:00
coffeecloudgit
7eb8257126
Update store.go (#6358) 
Add lock for location.ecVolumes And single delete
 2024-12-15 22:05:22 -08:00
Lisandro Pin
9b48ce0613
Parallelize EC shards balancing within racks (#6354) 
Parallelize EC shards balancing within racks.
 2024-12-15 13:36:23 -08:00
Lisandro Pin
926cfea3dc
Parallelize EC shards balancing across racks. (#6352) 2024-12-13 06:05:32 -08:00
Lisandro Pin
b81def5e5c
Parallelize EC balancing for racks. (#6351) 2024-12-13 05:33:53 -08:00
zouyixiong
d6f3e1970d
fix: filer may crash by bucketLastActiveTsNs concurrency access. (#6350) 2024-12-13 05:30:21 -08:00
Konstantin Lebedev
0a4b1909a2
[shell] only apply the balancing for writable volumes (#6346) 2024-12-13 01:10:00 -08:00
zouyixiong
9987a65e8a
fix: record and delete bucket metrics after inactive (#6349) 2024-12-12 20:34:02 -08:00
Lisandro Pin
b0210df081
Begin implementing EC balancing parallelization support. (#6342) 
* Begin implementing EC balancing parallelization support.

Impacts both `ec.encode` and `ec.balance`,

* Nit: improve type naming.

* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.

* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.

* Fix concurrency bug.

* Fix whitespace after Gitlab automerge.

* Delete stray TODO.
 2024-12-12 09:14:44 -08:00
Lisandro Pin
23ffbb083c
Limit EC re-balancing for ec.encode to relevant collections when a volume ID argument is provided. (#6347) 
Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
 2024-12-12 08:41:33 -08:00
Lisandro Pin
6320036c56
Delete legacy balancing code for ec.encode. (#6344) 2024-12-12 07:42:03 -08:00
Konstantin Lebedev
700b95304b
[shell] volume.list show only writable volumes (#6338) 
* show only writable volumes

* fix import
 2024-12-11 09:06:15 -08:00
Konstantin Lebedev
c37281735e
volume.list avoid output empty data center and rack and disk info (#6341) 2024-12-11 09:03:16 -08:00
Lisandro Pin
8c82c037b9
Unify the re-balancing logic for ec.encode with ec.balance. (#6339) 
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
 2024-12-10 13:30:13 -08:00
Konstantin Lebedev
ff1392f7f4
[shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
dependabot[bot]
4f6c989309
chore(deps): bump golang.org/x/net from 0.31.0 to 0.32.0 (#6330) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.31.0 to 0.32.0.
- [Commits](https://github.com/golang/net/compare/v0.31.0...v0.32.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 09:59:05 -08:00
dependabot[bot]
af4c51aee6
chore(deps): bump cloud.google.com/go/pubsub from 1.45.1 to 1.45.3 (#6328) 
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.45.1 to 1.45.3.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.45.1...pubsub/v1.45.3)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/pubsub
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 09:04:45 -08:00
dependabot[bot]
3cd2f06279
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.69.0 to 1.71.0 (#6329) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.69.0 to 1.71.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.69.0...service/s3/v1.71.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 08:46:16 -08:00
dependabot[bot]
e438fdc16c
chore(deps): bump github.com/hanwen/go-fuse/v2 from 2.7.1 to 2.7.2 (#6331) 
Bumps [github.com/hanwen/go-fuse/v2](https://github.com/hanwen/go-fuse) from 2.7.1 to 2.7.2.
- [Commits](https://github.com/hanwen/go-fuse/compare/v2.7.1...v2.7.2)

---
updated-dependencies:
- dependency-name: github.com/hanwen/go-fuse/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 08:46:00 -08:00
dependabot[bot]
3680bbac56
chore(deps): bump golang.org/x/sys from 0.27.0 to 0.28.0 (#6332) 
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.27.0 to 0.28.0.
- [Commits](https://github.com/golang/sys/compare/v0.27.0...v0.28.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 08:45:26 -08:00
dependabot[bot]
67461ccd27
chore(deps): bump docker/build-push-action from 6.9.0 to 6.10.0 (#6333) 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.9.0 to 6.10.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](4f58ea7922...48aba3b46d)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-12-09 08:45:13 -08:00
Lisandro Pin
522a25790a
Remove average constraints when selecting nodes/racks to balance EC shards into. (#6325) 2024-12-06 09:00:06 -08:00
vadimartynov
914ff0e718
Replaced https.client.enabled with false in scaffold/security (#6320) 2024-12-05 10:39:28 -08:00
chrislu
f3dde99796 adjust error message 2024-12-05 09:33:50 -08:00
Lisandro Pin
34cdbdd279
Share common parameters for EC re-balancing functions under a single struct. (#6319) 
TODO cleanup for https://github.com/seaweedfs/seaweedfs/discussions/6179.
 2024-12-05 09:00:46 -08:00
Lisandro Pin
edef485333
Account for replication placement settings when balancing EC shards within the same rack. (#6317) 
* Account for replication placement settings when balancing EC shards within racks.

* Update help contents for `ec.balance`.

* Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
 2024-12-04 10:47:51 -08:00
Lisandro Pin
351efa134d
Account for replication placement settings when balancing EC shards across racks. (#6316) 2024-12-04 09:00:55 -08:00
Konstantin Lebedev
b65eb2ec45
[security] reload whiteList on http seerver (#6302) 
* reload whiteList

* white_list add to scaffold
 2024-12-02 10:38:10 -08:00
Lisandro Pin
b2ba7d7408
Resolve replica placement for EC volumes from master server defaults. (#6303) 2024-12-02 08:44:07 -08:00
Lisandro Pin
9a741a61b1
Display details upon failures to re-balance EC shards racks. (#6299) 2024-11-28 08:42:41 -08:00
Lisandro Pin
559a1fd0f4
Improve EC shards rebalancing logic across nodes (#6297) 
* Improve EC shards rebalancing logic across nodes.

- Favor target nodes with less preexisting shards, to ensure a fair distribution.
- Randomize selection when multiple possible target nodes are available.
- Add logic to account for replication settings when selecting target nodes (currently disabled).

* Fix minor test typo.

* Clarify internal error messages for `pickEcNodeToBalanceShardsInto()`.
 2024-11-27 11:51:57 -08:00
steve.wei
88fa64a01a
change comment for expect header (#6289) 
* remove case of expect

* Set the default value of the Expect header for compatibility.
 2024-11-26 08:35:41 -08:00
Konstantin Lebedev
e2e97db917
[master] avoid timeout when assigning for main request with filter by DC or rack (#6291) 
* avoid timeout when assigning for main request with filter by DC or rack

https://github.com/seaweedfs/seaweedfs/issues/6290

* use constant NoWritableVolumes
 2024-11-26 08:33:31 -08:00
Konstantin Lebedev
fec88e64eb
[master] update LastLeaderChangeTime for hashicorp raft (#6292) 2024-11-26 08:02:45 -08:00
Trim21
d43fa07f06
use readable bytes size string in shell output (#6288) 2024-11-25 17:25:17 -08:00
dependabot[bot]
f965af51b7
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.66.3 to 1.69.0 (#6287) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.66.3 to 1.69.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.66.3...service/s3/v1.69.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:35:21 -08:00
dependabot[bot]
11042ddb71
chore(deps): bump docker/metadata-action from 5.5.1 to 5.6.1 (#6281) 
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5.5.1 to 5.6.1.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](8e5442c4ef...369eb591f4)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:27:46 -08:00
dependabot[bot]
e4abb6cbea
chore(deps): bump github.com/hanwen/go-fuse/v2 from 2.6.1 to 2.7.1 (#6283) 
Bumps [github.com/hanwen/go-fuse/v2](https://github.com/hanwen/go-fuse) from 2.6.1 to 2.7.1.
- [Commits](https://github.com/hanwen/go-fuse/compare/v2.6.1...v2.7.1)

---
updated-dependencies:
- dependency-name: github.com/hanwen/go-fuse/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:26:51 -08:00
dependabot[bot]
19d01c02c6
chore(deps): bump github.com/stretchr/testify from 1.9.0 to 1.10.0 (#6284) 
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:26:44 -08:00
dependabot[bot]
d54fbca481
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.45 to 1.17.46 (#6285) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.45 to 1.17.46.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.45...credentials/v1.17.46)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:26:37 -08:00
dependabot[bot]
29aaf3e68e
chore(deps): bump google.golang.org/protobuf from 1.35.1 to 1.35.2 (#6286) 
Bumps google.golang.org/protobuf from 1.35.1 to 1.35.2.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-25 14:26:29 -08:00
Konstantin Lebedev
8836fa19b6
use ShouldGrowVolumesByDcAndRack (#6280) 2024-11-25 09:30:37 -08:00
Konstantin Lebedev
167b50be88
fix missing register master metric MasterPickForWriteErrorCounter (#6277) 2024-11-25 08:59:11 -08:00
Trim21
fd6c787c02
fix EcVolumes sorting in volume UI (#6275) 
* Update store_ec.go

* fix: EcVolume sorting not work

* use stdlib

* revert
 2024-11-23 22:43:08 -08:00
chrislu
ccf1795e6f wait a bit before getting the next volume id if the leader is recently elected 2024-11-23 19:58:45 -08:00
chrislu
04081128a9 use math rand v2 2024-11-21 08:54:03 -08:00
chrislu
54204bc89b format error 2024-11-21 08:52:39 -08:00
Lisandro Pin
ca499de1cb
Improve EC shards rebalancing logic across racks (#6270) 
Improve EC shards rebalancing logic across racks.

  - Favor target shards with less preexisting shards, to ensure a fair distribution.
  - Randomize selection when multiple possible target shards are available.
  - Add logic to account for replication settings when selecting target shards (currently disabled).
 2024-11-21 08:46:24 -08:00
Numblgw
e56327e3b0
ExclusiveLocker only create one renew goroutine (#6269) 
Co-authored-by: liguowei <liguowei@xinye.com>
 2024-11-21 08:27:12 -08:00
Numblgw
aebf3952b7
filer sync: source path and exclude path support dir suffix (#6268) 
filer sync: source path and exclude paht support dir suffix

Co-authored-by: liguowei <liguowei@xinye.com>
 2024-11-21 08:25:12 -08:00