Commit Graph

7185 Commits

Author SHA1 Message Date
Numblgw
aebf3952b7
filer sync: source path and exclude path support dir suffix (#6268)
filer sync: source path and exclude paht support dir suffix

Co-authored-by: liguowei <liguowei@xinye.com>
2024-11-21 08:25:12 -08:00
Konstantin Lebedev
7fa4e5c2a1
[master] Revert back automatic volume grow in case of compensate for crowded volumes (#6267)
brought back the Automatic Volume Grow to compensate for crowded volumes
2024-11-21 08:19:19 -08:00
Konstantin Lebedev
a143c888e5
[shell] don't require lock when there are no changes for volume.fix.replication (#6266)
* don't require lock when there are no changes

* revert takeAction
2024-11-21 08:17:25 -08:00
Konstantin Lebedev
a49d9e020c
[master] avoid crowded more writable for auto grow (#6214)
avoid crowded more writable
https://github.com/seaweedfs/seaweedfs/issues/6121
2024-11-21 00:57:42 -08:00
chrislu
75f5afa571 fix compilation 2024-11-21 00:41:22 -08:00
chrislu
1c143cfcea display actual offset value 2024-11-21 00:38:03 -08:00
chrislu
c9f3448692 ReadAt may return io.EOF t end of file
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-11-21 00:37:38 -08:00
chrislu
be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
chrislu
6b2612690b consistent error message if auth is not setup
fix https://github.com/seaweedfs/seaweedfs/issues/6262
2024-11-20 23:11:18 -08:00
Konstantin Lebedev
254ed8897e
[shell] add noLock param for volume.move (#6261) 2024-11-20 08:35:24 -08:00
chrislu
96f1a2e01a adjust error message 2024-11-19 14:16:06 -08:00
chrislu
e28f55eb08 typo 2024-11-19 08:32:31 -08:00
chrislu
98f03862aa rename 2024-11-19 08:31:54 -08:00
chrislu
07cf8cf22d minor 2024-11-19 08:31:33 -08:00
Lisandro Pin
0d5393641e
Unify usage of shell.EcNode.dc as DataCenterId. (#6258) 2024-11-19 06:33:18 -08:00
chrislu
bc7650bd61 adds more info on skipped volumes 2024-11-18 18:25:44 -08:00
Lisandro Pin
f2db746690
Introduce logic to resolve volume replica placement within EC rebalancing. (#6254)
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.

All tests defined in this file are now for `command_ec_common.go`.

* Minor code cleanups.

- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.

* Introduce logic to resolve volume replica placement within EC rebalancing.

This will be used to make rebalancing logic topology-aware.

* Give shell.EcNode.dc a dedicated DataCenterId type.
2024-11-18 18:05:06 -08:00
chrislu
7b3c0e937f 3.80 2024-11-18 09:48:33 -08:00
zemul
6d3e38f439
Adjust threshold config (#6246)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2024-11-17 20:51:26 -08:00
Kevin Bulteel
4a0dc883cf
Fix/parse upload filename (#6241)
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
2024-11-15 11:25:18 -08:00
Max Denushev
a5fe6e21bc
feat(filer.backup): add ignore errors option (#6235)
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-11-14 08:40:55 -08:00
Eugeniy E. Mikhailov
3003c9e17e
added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
Chris Lu
72b14a451e
delete aborted ec shards from both source and target servers (#6221)
fix https://github.com/seaweedfs/seaweedfs/issues/6205#issuecomment-2465004586
2024-11-09 11:32:08 -08:00
chrislu
113c9ce6a8 remove the direct_io flag, as it is not well-supported on macOS 2024-11-07 14:18:59 -08:00
chrislu
88a1eee31c adjust logs 2024-11-07 13:00:12 -08:00
chrislu
62c4980754 add debug info 2024-11-06 11:59:03 -08:00
Konstantin Lebedev
9a5d3e7b31
[shell] add admin noLock for balance (#6209)
add admin noLock for balance
2024-11-05 19:09:22 -08:00
Bruce
15fc58bfd9
remove dup code for section.reader access (#6169) 2024-11-05 19:08:59 -08:00
chrislu
706d1a8e22 fix tests 2024-11-05 08:52:07 -08:00
chrislu
90acfd9394 remove case when read request is out side of the file size 2024-11-05 08:42:44 -08:00
chrislu
6161d0b05d a bit optimization 2024-11-05 08:41:14 -08:00
chrislu
98b519b113 fix FUSE mount on mac 2024-11-05 08:28:54 -08:00
Lisandro Pin
efdebf712e
Refactor ec.balance logic into a weeed/shell/command_ec_common.go… (#6195)
* Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go` standalone function.

This is a prerequisite to unify the balance logic for `ec.balance` and `ec.encode'.

* s/Balance()/EcBalance()/g
2024-11-04 17:56:20 -08:00
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
wyang
a7973ed7d1
fix deadlock hang when broadcast to clients (#6184)
fix deadlock when broadcast to clients

when master thransfer leader, the old master will disconnect with all
filers and volumeServers, if the cluster is a big , the broadcast
messages may be more big than the max of the channel len 100, then if the
KeepConnect was not listen on the channel in disconnect, it will
deadlock. and the whole cluster will not serve!
2024-11-03 23:20:48 -08:00
Konstantin Lebedev
5bddf0c085
[shell] volume.balance collect volume servers by dc rack node (#6191)
* chore: balance by rack

* fix: rm check lock

* fix: selected racks

* fix: selected nodes

* fix: containts

* fix: one collectVolumeServersByDcRackNode

* fix: revert lock and add lock

* fix: panic test

* revert noLock
2024-11-03 11:08:45 -08:00
han
65fb8fad99
Update filer_multipart.go (#6188)
fix the #6177 bug
2024-11-01 00:38:35 -07:00
Bruce
0060a2cf9c
Fix 6181/6182 (#6183)
* set larger buf size for LogBuffer

* jump to next day when no more entry found

* Update weed/filer/filer_notify_read.go

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-10-31 08:40:05 -07:00
wyang
c29c912bdc
fix format (#6185)
unitest weed/shell fail
2024-10-31 08:30:35 -07:00
chrislu
ad6240f066 fix parameter 2024-10-30 20:30:51 -07:00
chrislu
8802843511 pass in bytes buffer to avoid possible race condition
related to https://github.com/seaweedfs/seaweedfs/issues/6180
2024-10-30 09:31:22 -07:00
chrislu
3feb66d0a1 adding more debug message
related to https://github.com/seaweedfs/seaweedfs/issues/6180
2024-10-30 09:16:40 -07:00
chrislu
228946369c 3.79 2024-10-28 19:44:39 -07:00
chrislu
9873b033d1 backward compatible vif loading 2024-10-28 19:44:30 -07:00
chrislu
9105c6bdd1 fix format 2024-10-28 11:29:08 -07:00
chrislu
ebbb35d533 3.78 2024-10-27 17:44:22 -07:00
chrislu
089d4316ef ensure 2 volume space since actual need 1.4x volume size empty space 2024-10-24 22:44:53 -07:00
chrislu
6e388e29c9 correcting free volume count, factor it during ec encoding to ensure enough disk space available
fix https://github.com/seaweedfs/seaweedfs/issues/6163
2024-10-24 22:42:38 -07:00
chrislu
8d6189bcc5 adjust output format 2024-10-24 21:41:39 -07:00
chrislu
2f3d820f52 rename proto field
This should not have any impact.
2024-10-24 21:36:56 -07:00