mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-24 02:59:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,147 Commits 32 Branches 296 Tags 147 MiB
ca499de1cb
Commit Graph

11147 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Lisandro Pin
ca499de1cb
Improve EC shards rebalancing logic across racks (#6270) 
Improve EC shards rebalancing logic across racks.

  - Favor target shards with less preexisting shards, to ensure a fair distribution.
  - Randomize selection when multiple possible target shards are available.
  - Add logic to account for replication settings when selecting target shards (currently disabled).
 2024-11-21 08:46:24 -08:00
Numblgw
e56327e3b0
ExclusiveLocker only create one renew goroutine (#6269) 
Co-authored-by: liguowei <liguowei@xinye.com>
 2024-11-21 08:27:12 -08:00
Numblgw
aebf3952b7
filer sync: source path and exclude path support dir suffix (#6268) 
filer sync: source path and exclude paht support dir suffix

Co-authored-by: liguowei <liguowei@xinye.com>
 2024-11-21 08:25:12 -08:00
Konstantin Lebedev
7fa4e5c2a1
[master] Revert back automatic volume grow in case of compensate for crowded volumes (#6267) 
brought back the Automatic Volume Grow to compensate for crowded volumes
 2024-11-21 08:19:19 -08:00
Konstantin Lebedev
a143c888e5
[shell] don't require lock when there are no changes for volume.fix.replication (#6266) 
* don't require lock when there are no changes

* revert takeAction
 2024-11-21 08:17:25 -08:00
Konstantin Lebedev
a49d9e020c
[master] avoid crowded more writable for auto grow (#6214) 
avoid crowded more writable
https://github.com/seaweedfs/seaweedfs/issues/6121
 2024-11-21 00:57:42 -08:00
chrislu
75f5afa571 fix compilation 2024-11-21 00:41:22 -08:00
chrislu
1c143cfcea display actual offset value 2024-11-21 00:38:03 -08:00
chrislu
c9f3448692 ReadAt may return io.EOF t end of file 
related to https://github.com/seaweedfs/seaweedfs/issues/6219
 2024-11-21 00:37:38 -08:00
chrislu
be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
chrislu
6b2612690b consistent error message if auth is not setup 
fix https://github.com/seaweedfs/seaweedfs/issues/6262
 2024-11-20 23:11:18 -08:00
Konstantin Lebedev
254ed8897e
[shell] add noLock param for volume.move (#6261) 2024-11-20 08:35:24 -08:00
chrislu
96f1a2e01a adjust error message 2024-11-19 14:16:06 -08:00
dependabot[bot]
bb37886f6a
chore(deps): bump github.com/rclone/rclone from 1.68.1 to 1.68.2 (#6260) 
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.68.1 to 1.68.2.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.68.1...v1.68.2)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-19 13:44:32 -08:00
chrislu
e28f55eb08 typo 2024-11-19 08:32:31 -08:00
chrislu
98f03862aa rename 2024-11-19 08:31:54 -08:00
chrislu
07cf8cf22d minor 2024-11-19 08:31:33 -08:00
Lisandro Pin
0d5393641e
Unify usage of shell.EcNode.dc as DataCenterId. (#6258) 2024-11-19 06:33:18 -08:00
chrislu
2caa0e3741 java 3.80 2024-11-18 22:41:59 -08:00
chrislu
bc7650bd61 adds more info on skipped volumes 2024-11-18 18:25:44 -08:00
Lisandro Pin
f2db746690
Introduce logic to resolve volume replica placement within EC rebalancing. (#6254) 
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.

All tests defined in this file are now for `command_ec_common.go`.

* Minor code cleanups.

- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.

* Introduce logic to resolve volume replica placement within EC rebalancing.

This will be used to make rebalancing logic topology-aware.

* Give shell.EcNode.dc a dedicated DataCenterId type.
 2024-11-18 18:05:06 -08:00
chrislu
7b3c0e937f 3.80 2024-11-18 09:48:33 -08:00
dependabot[bot]
180f267867
chore(deps): bump github.com/linxGnu/grocksdb from 1.9.5 to 1.9.7 (#6249) 
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.9.5 to 1.9.7.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.9.5...v1.9.7)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-18 09:37:06 -08:00
dependabot[bot]
b21f599cde
chore(deps): bump modernc.org/sqlite from 1.33.1 to 1.34.1 (#6251) 
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.33.1 to 1.34.1.
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.33.1...v1.34.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-18 09:36:52 -08:00
dependabot[bot]
3fba0adc9d
chore(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.28.0 to 1.28.4 (#6252) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/config

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.28.0 to 1.28.4.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.28.0...config/v1.28.4)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-18 09:36:43 -08:00
dependabot[bot]
f8f162aa72
chore(deps): bump go.etcd.io/etcd/client/v3 from 3.5.16 to 3.5.17 (#6253) 
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.5.16 to 3.5.17.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.5.16...v3.5.17)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-18 09:36:31 -08:00
zemul
6d3e38f439
Adjust threshold config (#6246) 
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2024-11-17 20:51:26 -08:00
Kevin Bulteel
4a0dc883cf
Fix/parse upload filename (#6241) 
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
 2024-11-15 11:25:18 -08:00
Omar Hamadeh
8ebbf37e8e
Add a loop to wait for master and filer services in post-install-bucket-hook (#6240) 
Add a loop to wait for master and filer services before creating buckets
 2024-11-15 11:15:33 -08:00
fo40225
008ac38ebc
fix java.lang.IllegalArgumentException: Comparison method violates its general contract! (#6239) 2024-11-14 17:26:50 -08:00
Max Denushev
a5fe6e21bc
feat(filer.backup): add ignore errors option (#6235) 
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2024-11-14 08:40:55 -08:00
Eugeniy E. Mikhailov
3003c9e17e
added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
dependabot[bot]
31d8907e77
chore(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (#6231) 
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.0...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-11 11:54:07 -08:00
dependabot[bot]
7b0a37312d
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.66.2 to 1.66.3 (#6225) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.66.2 to 1.66.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.66.2...service/s3/v1.66.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-11 11:15:24 -08:00
dependabot[bot]
623fc3a4fc
chore(deps): bump github.com/aws/aws-sdk-go-v2 from 1.32.3 to 1.32.4 (#6227) 
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.32.3 to 1.32.4.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.32.3...v1.32.4)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-11 11:15:09 -08:00
dependabot[bot]
aa31c5422b
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.89.2 to 3.91.0 (#6228) 
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.89.2 to 3.91.0.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.89.2...v3.91.0)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-11 11:15:00 -08:00
dependabot[bot]
6232041c64
chore(deps): bump golang.org/x/tools from 0.26.0 to 0.27.0 (#6229) 
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: golang.org/x/tools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-11-11 11:14:48 -08:00
Chris Lu
72b14a451e
delete aborted ec shards from both source and target servers (#6221) 
fix https://github.com/seaweedfs/seaweedfs/issues/6205#issuecomment-2465004586
 2024-11-09 11:32:08 -08:00
Nathan Dorsey
3400962877
[helm] fix s3 servicemonitor label matching (#6218)
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
helm: lint and test charts / lint-test (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
2024-11-08 12:04:40 -08:00
chrislu
113c9ce6a8 remove the direct_io flag, as it is not well-supported on macOS 2024-11-07 14:18:59 -08:00
chrislu
88a1eee31c adjust logs 2024-11-07 13:00:12 -08:00
chrislu
62c4980754 add debug info 2024-11-06 11:59:03 -08:00
Konstantin Lebedev
9a5d3e7b31
[shell] add admin noLock for balance (#6209) 
add admin noLock for balance
 2024-11-05 19:09:22 -08:00
Bruce
15fc58bfd9
remove dup code for section.reader access (#6169) 2024-11-05 19:08:59 -08:00
chrislu
706d1a8e22 fix tests 2024-11-05 08:52:07 -08:00
chrislu
90acfd9394 remove case when read request is out side of the file size 2024-11-05 08:42:44 -08:00
chrislu
6161d0b05d a bit optimization 2024-11-05 08:41:14 -08:00
chrislu
98b519b113 fix FUSE mount on mac 2024-11-05 08:28:54 -08:00
Lisandro Pin
efdebf712e
Refactor ec.balance logic into a weeed/shell/command_ec_common.go… (#6195) 
* Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go` standalone function.

This is a prerequisite to unify the balance logic for `ec.balance` and `ec.encode'.

* s/Balance()/EcBalance()/g
 2024-11-04 17:56:20 -08:00
Chris Lu
dc784bf217
merge current message queue code changes (#6201) 
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
 2024-11-04 12:08:25 -08:00