chrislu
4093115ca9
Merge branch 'master' into mq
2024-06-06 19:27:07 -07:00
Gaspare Iengo
dc6b750424
Fix panic ( #5654 )
2024-06-06 18:59:50 -07:00
Gaspare Iengo
fce8fc1e16
Fix Issue #5649 ( #5652 )
2024-06-06 12:49:33 -07:00
steve.wei
d8da4bbaa7
Set the capacity of clientChan to 10000 ( #5647 )
2024-06-05 05:41:46 -07:00
chrislu
9f02bf4ede
Merge branch 'master' into mq
2024-06-04 21:12:30 -07:00
Lê Hoàng Nam
1f2dc54647
fix(s3api): make post api upload object default return 204 ( #5648 )
2024-06-04 06:18:47 -07:00
LHHDZ
36b5b713ba
fix deadlock caused by message chan blocked ( #5639 )
2024-06-03 07:42:40 -07:00
Dominic Pearson
d8bde9b96e
Solaris disk support ( #5638 )
2024-06-02 22:10:28 -07:00
NyaMisty
579ebbdf60
Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed ( #5636 )
2024-06-02 14:25:42 -07:00
NyaMisty
0c62d591e2
Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) ( #5635 )
2024-06-02 14:16:05 -07:00
NyaMisty
e1c671c028
Fix volume.tier.upload nil pointer panic ( #5634 )
2024-06-02 14:15:39 -07:00
NyaMisty
f2a37933b1
Fix sorted needle file's write mode for (*SortedFileNeedleMap).Delete() ( #5633 )
2024-06-02 14:12:25 -07:00
chrislu
41cc825ddb
Merge branch 'master' into mq
2024-05-30 12:03:22 -07:00
chrislu
d1f0c40402
remove per-message debug logs
2024-05-30 09:49:08 -07:00
chrislu
df9d889489
fix size calculation
2024-05-30 09:41:04 -07:00
chrislu
ff14dfa0ee
debug
2024-05-30 09:15:50 -07:00
chrislu
b485b9695c
fix passing sub follower offset
2024-05-30 09:15:23 -07:00
chrislu
f7b818483a
fix cases when buffer wraps around
2024-05-30 09:10:30 -07:00
chrislu
9a5cc065b0
fix
2024-05-30 08:29:06 -07:00
chrislu
23a72db1df
stop partitionOffsetChan if closed
2024-05-30 00:27:44 -07:00
chrislu
af19256dbf
skip control messages
2024-05-29 23:34:39 -07:00
chrislu
ad2b878130
fix inflight message tracker
2024-05-29 23:33:37 -07:00
chrislu
f8a22038c5
passing along publisher name
2024-05-29 23:31:48 -07:00
chrislu
f0b9e163a0
avoid nil stopChan
2024-05-29 16:18:02 -07:00
chrislu
a5874954dc
less logs
2024-05-29 16:17:37 -07:00
chrislu
3ddfd15ec1
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-05-29 10:41:38 -07:00
chrislu
3e7a92061b
pass along volume server grpc port
...
fix https://github.com/seaweedfs/seaweedfs/issues/5617
2024-05-29 10:41:33 -07:00
chrislu
edc87c16f9
balance subscribers
...
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
dependabot[bot]
722a14ef57
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.2.0 to 0.4.2 ( #5621 )
...
* chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ
Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ ) from 0.2.0 to 0.4.2.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.2.0...v0.4.2 )
---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-05-27 12:36:00 -07:00
Vlad
72f7b4e9b1
Replaces volume.unmount in the help of the volumeServer.leave ( #5619 )
2024-05-27 01:13:06 -07:00
chrislu
6366898af2
add market
2024-05-26 14:35:22 -07:00
chrislu
96b326a304
use type ConsumerGroupInstanceId
2024-05-26 14:09:51 -07:00
chrislu
ba778539d9
Merge branch 'master' into mq
2024-05-24 17:15:27 -07:00
chrislu
364bb6c7b4
avoid ticker leak
2024-05-24 17:15:12 -07:00
chrislu
37d1ee562d
refactor
2024-05-23 09:21:48 -07:00
chrislu
cdeaaf95b4
go fmt
2024-05-23 08:25:53 -07:00
chrislu
77a7c5c2a0
Merge branch 'master' into mq
2024-05-23 08:25:27 -07:00
chrislu
ba98f02d02
go fmt
2024-05-23 08:25:16 -07:00
chrislu
d40b350f54
coordinator receives unassignment ack
2024-05-23 08:23:35 -07:00
chrislu
73d008fafa
Merge branch 'master' into mq
2024-05-22 20:39:31 -07:00
Chris Lu
cfcf383ee7
fix wrong prefix example
2024-05-21 21:45:29 -07:00
Vlad
a9b057389f
chore: add missing concurrency help in fs.meta.load command ( #5609 )
2024-05-21 21:42:48 -07:00
chrislu
554ae09f82
rename
2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa
rename file
2024-05-21 09:57:59 -07:00
chrislu
67e6051585
rename Coordinator to SubCoordinator
2024-05-21 09:57:45 -07:00
chrislu
b5099263a4
rename file
2024-05-21 09:56:46 -07:00
chrislu
d5abffa42c
rename Balancer to PubBalancer
2024-05-21 09:56:30 -07:00
chrislu
fa98ecf71e
client side stop partition subscribing if unassigned
2024-05-21 08:42:04 -07:00
chrislu
6634b42981
fix compilation
2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3
refactor
...
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6
avoid concurrent processing for the same key
2024-05-20 13:29:37 -07:00
chrislu
a7d30d0705
refactor
2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4
remove ProcessorConfiguration
2024-05-20 12:28:01 -07:00
chrislu
65dd5ac6fb
fix
2024-05-20 12:27:45 -07:00
chrislu
3b7f24b1fc
unused
2024-05-20 12:26:54 -07:00
chrislu
f13475ac2e
sending keyed offset
2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d
minor
2024-05-20 11:07:54 -07:00
chrislu
3622196881
go fmt
2024-05-20 11:05:18 -07:00
chrislu
5038577f7e
Merge branch 'master' into mq
2024-05-20 11:04:53 -07:00
chrislu
aba212dedd
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-05-20 11:03:58 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
7a4ace5dfb
adjust logs
2024-05-20 09:34:07 -07:00
chrislu
d8ab22012a
track offset
2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5
add InflightMessageTracker
2024-05-20 09:19:39 -07:00
chrislu
a2885512e1
fix
2024-05-19 15:06:33 -07:00
chrislu
8d02712c63
ConcurrentPartitionLimit
2024-05-19 14:55:03 -07:00
chrislu
8ce2cced47
less logs
2024-05-19 14:54:46 -07:00
chrislu
c6db3f31a1
start consuming ASAP
2024-05-19 14:52:38 -07:00
chrislu
c21b32dd20
logs
2024-05-19 11:02:17 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
sb
a3a00d9499
Feature/mongodb security ( #5602 )
2024-05-18 12:19:10 -07:00
sb
3fae87632f
add filer configuration options for username/password and tls ( #5601 )
2024-05-18 09:02:03 -07:00
chrislu
8d5bb7420d
rename
2024-05-17 18:21:21 -07:00
chrislu
94742b8ace
add subscribe follower
2024-05-17 18:17:23 -07:00
Riccardo Bertossa
f6e8a9bf9c
added s3 iam DeleteBucket permission management ( #5599 )
2024-05-17 04:54:09 -07:00
chrislu
279cb2b85e
consumer acks received messages
2024-05-16 20:28:19 -07:00
chrislu
faffb2973c
adjust errors
2024-05-16 11:02:48 -07:00
chrislu
759a8e50cf
remove logs
2024-05-14 23:22:51 -07:00
chrislu
1f20178ded
subscriber receives partitions and dispatch to processors
2024-05-14 23:22:43 -07:00
chrislu
972e9faaa2
move initial assignment to rebalance logic
2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e
move read write topic config into filer client accessor
2024-05-14 08:50:17 -07:00
Martin Stiborský
54f3913bed
[s3] Fixed s3 replication by sending content-md as base64 ( #5596 )
2024-05-14 06:48:24 -07:00
chrislu
ac63f2b5a1
refactor
2024-05-13 22:03:57 -07:00
chrislu
844a1a0192
rename
2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d
consumer instance passing MaxPartitionCount to coordinator
2024-05-13 18:30:10 -07:00
chrislu
2142842f82
refactor
2024-05-13 18:25:38 -07:00
Konstantin Lebedev
d389c5b27e
fix: recreate index include deleted files ( #5579 )
...
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508
* fix: counting the number of files
* fix: log
2024-05-12 11:31:34 -07:00
Dave St.Germain
731b3aadbe
Add support for OpenBSD ( #5578 )
...
Co-authored-by: Dave St.Germain <dcs@adullmoment.com>
2024-05-10 14:35:41 -07:00
chrislu
8cc5298a08
clean up unused variables
2024-05-10 13:10:45 -07:00
chrislu
d3032d1e80
3.67
2024-05-07 08:47:30 -07:00
chrislu
1917f70f78
also delete parent folder if empty
...
fix https://github.com/seaweedfs/seaweedfs/issues/5567
2024-05-07 08:46:19 -07:00
Patrick Schmidt
96c48bd0da
Fix deadlock in lock table locks ( #5566 )
2024-05-07 05:44:53 -07:00
chrislu
3682eb929f
3.66
2024-05-05 22:23:18 -07:00
chrislu
fcedf97e94
fix test
2024-05-05 22:19:59 -07:00
chrislu
664b704e30
add delay
2024-05-05 14:17:41 -07:00
chrislu
4e3701df77
reset last error
2024-05-05 14:16:54 -07:00
chrislu
658d302aa4
init candidate servers when setting a starting snapshot
2024-05-05 14:16:26 -07:00
chrislu
a5645d50a7
ensure head index is within range
2024-05-05 12:20:08 -07:00
chrislu
253ec73a17
log
2024-05-05 12:19:31 -07:00
chrislu
30f8600a24
working
2024-05-03 21:27:06 -07:00
chrislu
7a9b115cc2
rename
2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb
subscriber can unmarshal the record
2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7
rename
2024-05-02 09:47:29 -07:00
chrislu
f2f4630029
rename field types
2024-05-02 08:59:22 -07:00
chrislu
b8af997e41
consistent with protobuf data types
2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea
add example to publish a record
2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff
rename
2024-05-02 08:35:06 -07:00
chrislu
5bc1b70a4c
rename to WithField()
2024-05-02 08:32:15 -07:00
chrislu
a568d128fb
Merge branch 'master' into mq-subscribe
2024-05-02 08:13:53 -07:00
chrislu
f80fb1c02e
Merge branch 'master' into mq-subscribe
2024-04-29 21:57:31 -07:00
shenxingwuying
ee25ada732
reduce ambiguity about use memory_sequencer ( #5555 )
2024-04-29 21:51:00 -07:00
dependabot[bot]
36a1cf0361
chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 ( #5552 )
...
* chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1
Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie ) from 0.3.1 to 1.0.1.
- [Release notes](https://github.com/viant/ptrie/releases )
- [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md )
- [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1 )
---
updated-dependencies:
- dependency-name: github.com/viant/ptrie
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-04-29 18:42:31 -07:00
chrislu
5e8d788311
Merge branch 'master' into mq-subscribe
2024-04-29 06:44:03 -07:00
chrislu
e6bfd061c7
AllowEmptyFolder checks during object deletion
2024-04-29 06:25:06 -07:00
chrislu
d521466a37
split file
2024-04-29 06:23:42 -07:00
chrislu
a1b59948cc
rename files
2024-04-29 05:33:56 -07:00
chrislu
b570e015ce
Update write_parquet_test.go
2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01
record value API
2024-04-28 13:42:45 -07:00
chrislu
b166597c83
fix test
2024-04-28 13:03:01 -07:00
chrislu
189a7fc90e
record type begin and end
2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9
improve api
2024-04-28 12:42:58 -07:00
chrislu
1a2332c257
rename api
2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546
Delete schema_builder_test.go
2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54
struct to schema correctly
2024-04-28 12:23:54 -07:00
chrislu
f37862bffe
compile test
2024-04-28 12:23:33 -07:00
chrislu
7310d022ad
recursive struct to schema
2024-04-28 11:14:13 -07:00
Konstantin Lebedev
6f9267ebdf
fix: skip file size checking for volume with TTL ( #5541 )
...
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
2024-04-28 00:21:57 -07:00
chrislu
a2a872ca03
rename
2024-04-27 23:44:11 -07:00
chrislu
60103171b6
rename
2024-04-27 23:30:15 -07:00
chrislu
111a4e1126
add field index
2024-04-27 23:30:06 -07:00
Konstantin Lebedev
3d3ee04eb9
[s3] Put bucket lifecycle configuration ( #5510 )
2024-04-27 07:39:22 -07:00
Kevin Bulteel
a3dab1fcd0
feat: upload files directly without multipart/form-data ( #5539 )
2024-04-26 16:03:40 -07:00
Nikita Borzykh
0d04264494
Move file deletion logging to v2 verbisity ( #5538 )
2024-04-26 06:41:06 -07:00
chrislu
72b50980f4
a little bit more efficient
2024-04-25 23:59:30 -07:00
chrislu
8948fb51e3
Merge branch 'master' into mq-subscribe
2024-04-25 23:42:19 -07:00
chrislu
63036133ff
minor
2024-04-25 22:02:03 -07:00
chrislu
2a9b0a9332
use parquet levels to construct row
2024-04-25 21:29:45 -07:00
M@
abf01a0eb7
Fixes unlocked read from logBuffer.LastTsNs that is racey. ( #5536 )
2024-04-25 15:46:12 -07:00
chrislu
977e7988e6
toRow conversion with levels info
2024-04-25 09:14:37 -07:00
chrislu
9cb9d27b5b
file rename
2024-04-25 08:56:31 -07:00
chrislu
313097aca0
printout test values
2024-04-25 08:54:57 -07:00
chrislu
827e293010
add visitor
2024-04-25 08:54:41 -07:00
chrislu
138a027a2e
add field types but without value
2024-04-24 23:33:11 -07:00
chrislu
30b5aef62e
record type is also optional
2024-04-24 23:32:35 -07:00
chrislu
ce4597c623
delete the generated file
2024-04-24 23:29:55 -07:00
chrislu
f91484ca07
add some notes
2024-04-24 23:28:20 -07:00
chrislu
f93c4cf48c
adjust test
2024-04-24 23:24:53 -07:00
M@
855607c536
Fixes race with unlocked setter. ( #5532 )
2024-04-24 23:18:28 -07:00
M@
607927da60
Migrates from disintegration/imaging c2019 to cognusion/imaging c2024. ( #5533 )
2024-04-24 23:16:04 -07:00
chrislu
d88c1872ac
works with single level of list
2024-04-24 23:04:47 -07:00
Konstantin Lebedev
2f3fee9bb9
filer read chunk retry if status code 499 ( #5528 )
2024-04-24 06:51:05 -07:00
chrislu
d7e5f6b2a5
fix ordering
2024-04-24 01:32:02 -07:00
skycope
6e4b9181f5
fix "volume.fix.replication" move many replications only to one volumeServer ( #5522 )
2024-04-23 06:33:50 -07:00
chrislu
9ef238b610
has a bug
2024-04-22 00:42:18 -07:00
chrislu
cc1ce3de87
all record fields are optional by default
2024-04-18 23:17:52 -07:00
chrislu
384ed7a381
add test
2024-04-18 22:41:20 -07:00
chrislu
57949f9959
support list type
2024-04-18 22:41:12 -07:00
chrislu
05d18130b6
add schema builder
2024-04-18 22:27:18 -07:00
chrislu
20666bdf81
Merge branch 'master' into mq-subscribe
2024-04-18 21:33:06 -07:00
chrislu
55976ae04a
avoid repeated calls to heavy-weighted viper
2024-04-18 09:09:45 -07:00
chrislu
31f1f96038
improve perf a bit
2024-04-18 08:47:55 -07:00
chrislu
d9490c5e1f
rename
2024-04-18 08:47:45 -07:00
chrislu
0847a17484
convert to parquet schema and value
2024-04-17 23:49:21 -07:00
steve.wei
0bdf121e51
rename VolumeServerVolumeGauge ( #5504 )
2024-04-17 04:49:50 -07:00
chrislu
ce2b2fa9b2
Merge branch 'master' into mq-subscribe
2024-04-15 20:09:14 -07:00
chrislu
0edc5aad9a
3.65
2024-04-14 20:15:46 -07:00
Konstantin Lebedev
33537ae29f
[s3] fix s3 test_multipart_get_part ( #5476 )
...
* try fix s3 test_multipart_get_part
* add passed s3 tests
* fix SeaweedFSUploadId
* rm spaces
* convert part request to range
* add passed s3 tests of multipart
2024-04-14 10:41:32 -07:00
yudrywet
8833745fbc
chore: fix some typos in comments ( #5497 )
2024-04-14 07:42:40 -07:00
chrislu
68ade92d4f
publisher send schema
2024-04-12 23:36:15 -07:00
chrislu
8415476c2d
fix test
2024-04-12 23:35:54 -07:00
chrislu
5324e4288a
rename parameter
2024-04-12 22:33:00 -07:00
chrislu
6f80dbdfe5
Create schema.go
2024-04-12 22:31:28 -07:00
chrislu
53d1d2b78a
save schema when configuring topic
2024-04-12 22:29:53 -07:00
chrislu
04fb4c34e3
add proto for value
2024-04-12 22:27:16 -07:00
chrislu
48fe50df81
minor
2024-04-12 13:31:54 -07:00
chrislu
0cbf364a3d
all fields are optional
2024-04-12 13:29:35 -07:00
chrislu
7a72677f98
add test
2024-04-12 01:31:23 -07:00
chrislu
d45a372615
add schema.proto
2024-04-12 01:30:29 -07:00
chrislu
d5b2cee34a
log possible racing condition
...
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:54 -07:00
chrislu
2a4a9c6343
log possible racing condition
...
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:18 -07:00
chrislu
e53a972483
Merge branch 'master' into mq-subscribe
2024-04-11 19:48:00 -07:00
Konstantin Lebedev
5189a09de0
[volume] Reduce the number of buffers for uploading one chunk ( #5458 )
2024-04-11 04:47:21 -07:00
skycope
6dae685f9c
fix append disorder ( #5492 )
2024-04-10 19:21:44 -07:00
chrislu
cc1c69f312
Merge branch 'master' into mq-subscribe
2024-04-08 11:03:14 -07:00
steve.wei
67ead9b18f
fix(volume.fix.replication): adjust volume count, not free volume count ( #5479 )
2024-04-08 07:30:04 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server ( #5480 )
2024-04-08 07:27:00 -07:00
clonefetch
9e07a87fcb
chore: fix function names in comment ( #5478 )
2024-04-08 07:19:02 -07:00
Konstantin Lebedev
3e25ed1b11
[s3] add s3 pass test_multipart_upload_size_too_small ( #5475 )
...
* add s3 pass test_multipart_upload_size_too_small
* refactor metric names
* return ErrNoSuchUpload if empty parts
* fix test
2024-04-07 11:52:35 -07:00
Konstantin Lebedev
35cba720a5
[s3] add s3 pass test_multipart_upload ( #5474 )
...
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev
d42a04cceb
[s3] fix s3 test_multipart_resend_first_finishes_last ( #5471 )
...
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466
* add error handler metrics
* refactor
* refactor multipartExt
* delete bad entry parts
2024-04-06 10:56:39 -07:00
Nikita Borzykh
7aa25c1137
fix: correctly form partNumber from new entry.Name format in "listObjectParts" ( #5470 )
2024-04-04 09:16:16 -07:00
skycope
b19c9847c6
fix completed multiupload lost data ( #5460 )
...
If there are putObjectPart requests with the same uploadId during
completeMultiPart, it can result in data loss. putObjectPart requests
might be due to timeout retries.
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-04-02 20:18:42 -07:00
chrislu
ccdd9cd8de
refactor
2024-04-02 16:25:43 -07:00
chrislu
f37c0d0d7a
comment out println
2024-04-02 15:31:31 -07:00
Konstantin Lebedev
d5d8b8e2ae
fix panic at isAllWritable ( #5457 )
...
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
2024-04-02 09:06:19 -07:00
Henco Appel
2a88da4de7
fix: always close volume file ( #4530 ) ( #5459 )
...
If sync fails then close is never called. We should always be calling
close on the file.
Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2024-04-02 08:31:40 -07:00
chrislu
d935f70e3c
shutdown follower when leader stops
2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1
send flush message to follower before shutting down logBuffer
2024-04-01 16:01:26 -07:00
Nikita Borzykh
df9031e19a
fix: ensure complete gRPC stream consumption in doListFilerEntries to prevent context canceled errors. ( #5441 )
2024-04-01 15:55:16 -07:00
chrislu
e568e742c9
renaming
2024-04-01 15:02:52 -07:00
chrislu
47975e75bf
purge old code
2024-03-31 17:49:34 -07:00
chrislu
237e68059d
shutdown follower if no clients, not only publishers
...
to avoid overhead when a publisher repeatedly send one message and close
2024-03-31 17:45:26 -07:00
chrislu
66a878af39
add control message to data message
2024-03-31 16:35:46 -07:00
chrislu
546ae87c39
send is_close message to broker
2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b
add publisher name for debugging
2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6
simplify
2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75
logs
2024-03-30 20:37:46 -07:00
Nikita Borzykh
829c0da381
fix: adjust condition in prefixFilterEntries prevent infinite loop ( #5440 )
2024-03-30 14:47:00 -07:00
Konstantin Lebedev
9c1e0f5811
[master] grow volumes if no writable volumes in current dataCenter ( #5434 )
...
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886
* fix tests with volume grow
* automatic volume grow one volume
* add ErrorChunkAssign metrics
2024-03-29 00:38:27 -07:00
Konstantin Lebedev
df40908e57
fix panic 5435 ( #5436 )
2024-03-28 16:17:59 -07:00
chrislu
13b7c7b366
Merge branch 'master' into mq-subscribe
2024-03-27 23:57:00 -07:00
chrislu
e3c8be7f89
LocalPartition shutdown
2024-03-27 23:55:06 -07:00
chrislu
ab9fc0ad70
simplify
2024-03-27 23:42:19 -07:00
chrislu
7463058299
change visibility
2024-03-27 23:11:35 -07:00
chrislu
dab5454332
publish and send to follower
2024-03-27 23:10:24 -07:00
chrislu
f249f831b1
MaybeShutdownLocalPartition refactor
2024-03-27 23:02:51 -07:00
chrislu
2a0ab7e90d
fix
2024-03-27 22:59:50 -07:00
chrislu
059a120708
refactor
2024-03-27 22:58:20 -07:00
chrislu
7d87c1d2bb
refactor
2024-03-27 22:48:37 -07:00
chrislu
6aa804b368
lock instead of RLock, to prevent racing condition
...
https://github.com/seaweedfs/seaweedfs/discussions/5432#discussioncomment-8933608
2024-03-27 16:05:11 -07:00
chrislu
5cc94a05b9
separate goroutine to send ack to publisher
2024-03-27 10:27:08 -07:00
chrislu
17806cde2a
separate context
...
If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.
2024-03-27 10:24:57 -07:00
chrislu
f67c8309d8
rename
2024-03-26 21:52:12 -07:00
chrislu
7d69552974
log server list
2024-03-26 21:51:28 -07:00
chrislu
36fec34c47
print only adapted url
...
fix https://github.com/seaweedfs/seaweedfs/issues/5424
2024-03-25 12:50:43 -07:00
chrislu
707ff536a3
Merge branch 'master' into mq-subscribe
2024-03-25 11:21:19 -07:00
chrislu
aed6fdffd2
fix nil volume data backend
...
fix https://github.com/seaweedfs/seaweedfs/issues/5417
2024-03-25 08:19:31 -07:00
chrislu
d6a066495b
adjust logs
2024-03-25 08:15:47 -07:00
chrislu
1b1b96ea1e
close local partition if no more publishers and subscribers
2024-03-24 21:10:02 -07:00
Nikita Borzykh
9ca871af53
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries in etcd meta storage backend ( #5416 )
...
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries
2024-03-24 14:08:54 -07:00
chrislu
4f5c4c3388
refactor
2024-03-24 13:07:51 -07:00
chrislu
d558a58946
refactor
2024-03-24 13:04:59 -07:00
chrislu
205d2285f3
setup follower by publisher
...
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
Nikita Korolev
2dd2bb3e16
fix filer address parsing ( #5415 )
2024-03-24 11:20:33 -07:00
Nikita Borzykh
1f3742850d
Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage ( #5403 )
2024-03-24 08:36:21 -07:00
Konstantin Lebedev
dc9568fc0d
[master] add test for PickForWrite add metrics for volume layout ( #5413 )
2024-03-22 07:39:11 -07:00
Nikita Korolev
953f571349
add basic support for sentry error reporting ( #5410 )
2024-03-22 07:10:57 -07:00
adasauce
61f4e40ad9
fix s3api: delimeter properly takes prefixes into account ( #5411 )
2024-03-22 07:03:34 -07:00
wusong
a5582cc921
[mount] fix GetAttr blocks count ( #5409 )
2024-03-22 06:11:16 -07:00
chrislu
ac13644fc9
Merge branch 'master' into mq-subscribe
2024-03-21 23:31:49 -07:00
chrislu
03f18e43df
clean up logs
2024-03-21 23:30:49 -07:00
chrislu
b5a6f10612
Merge branch 'master' into mq-subscribe
2024-03-21 23:27:41 -07:00
chrislu
26e24db721
adjust log level
2024-03-21 23:27:19 -07:00
chrislu
36a3301e0c
Merge branch 'master' into mq-subscribe
2024-03-21 23:24:28 -07:00
chrislu
5c0fcae039
clean up logs
2024-03-21 23:24:18 -07:00
chrislu
407a52fc93
adjust log level
2024-03-21 23:23:26 -07:00
chrislu
da31e9b939
correctly wait for the publisher to finish
2024-03-21 09:08:53 -07:00
zuzuviewer
211171e669
Fix iam service response 500 ( #5405 )
2024-03-21 08:43:54 -07:00
chrislu
e641d49f9f
Merge branch 'master' into mq-subscribe
2024-03-21 08:22:14 -07:00
Konstantin Lebedev
25643cfbc6
fix: panic: assignment to entry in nil map on S3Sink.CreateEntry ( #5406 )
2024-03-21 08:12:31 -07:00
chrislu
8b18dd940f
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-03-21 08:07:58 -07:00
chrislu
50023fe1cf
adjust log level
2024-03-21 08:07:56 -07:00
Konstantin Lebedev
d821cb3b18
fix: sync without dir /buckets/some/.uploads/hash_hash ( #5402 )
2024-03-20 12:54:29 -07:00
chrislu
7003ce7425
publisher is able to fully send data and receive acks
...
still needs to close the pipes cleanly
2024-03-20 12:25:40 -07:00
chrislu
96149cbf7f
Merge branch 'master' into mq-subscribe
2024-03-19 09:14:27 -07:00
chrislu
a1816d21e9
fix compilation
2024-03-19 09:09:14 -07:00
chrislu
ad408a7d38
Merge branch 'master' into mq-subscribe
2024-03-19 08:58:09 -07:00
dependabot[bot]
359b00596f
chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 ( #5387 )
...
* chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone ) from 1.65.2 to 1.66.0.
- [Release notes](https://github.com/rclone/rclone/releases )
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md )
- [Commits](https://github.com/rclone/rclone/compare/v1.65.2...v1.66.0 )
---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix compilation error with rclone update (#5395 )
Fix compilation error
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Damiano Albani <damiano.albani@gmail.com>
2024-03-18 17:09:59 -07:00
jerebear12
7091269e65
fix get file metadata bug ( #5394 )
...
* Removed problematic if statement
This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.
* Update filer_server_handers_read.go
* Updated filer responses to invalid requests
2024-03-18 11:08:16 -07:00
chrislu
516cba7053
remove FollowInMemoryMessages
2024-03-17 22:27:13 -07:00
chrislu
6ef3e6bcad
logging
2024-03-17 22:24:20 -07:00
chrislu
a375b2815e
Merge branch 'master' into mq-subscribe
2024-03-17 11:46:55 -07:00
chrislu
ca042bd067
simplify
2024-03-17 01:01:45 -07:00
chrislu
4dce1e9d29
Revert "remove github.com/golang/protobuf/proto"
...
This reverts commit 1516e0249f
2024-03-17 01:00:43 -07:00
chrislu
a8b73296d7
fix test3
2024-03-16 23:53:46 -07:00
chrislu
3c559dc294
fix test2
2024-03-16 23:48:31 -07:00
chrislu
ea2601799c
fix test
2024-03-16 23:22:17 -07:00
chrislu
1516e0249f
remove github.com/golang/protobuf/proto
2024-03-16 23:18:11 -07:00
chrislu
8e5068fd2f
notify
2024-03-16 23:16:33 -07:00
chrislu
aba934f0b5
adjust log level
2024-03-16 23:15:06 -07:00
chrislu
44ab42d03c
unnecessary
2024-03-16 17:11:42 -07:00
chrislu
3f2dd8cc3e
add follower id info
2024-03-16 17:11:18 -07:00
chrislu
b74e8082ba
3.64
2024-03-16 11:46:40 -07:00
chrislu
6a61b54f29
Merge branch 'mq-subscribe'
2024-03-16 11:42:23 -07:00
chrislu
27bb38228b
only broad cast when there are waiting threads
2024-03-16 11:38:27 -07:00
chrislu
054d0b3487
wait for data
2024-03-16 11:14:14 -07:00
chrislu
f8957b84c1
rename to LastTsNs
2024-03-16 10:52:47 -07:00
chrislu
529fa8c4c8
go fmt
2024-03-16 10:51:47 -07:00
chrislu
059df3957d
rename to lastFlushDataTime
2024-03-16 10:51:16 -07:00
chrislu
1b5c547cfe
logs
2024-03-15 21:08:57 -07:00
chrislu
55714f54ab
local partition is generated or not
2024-03-15 21:08:42 -07:00
chrislu
56ce040ab6
log errors
2024-03-15 21:05:40 -07:00
chrislu
549455c30d
streaming mode
2024-03-15 21:05:24 -07:00
chrislu
f653838f34
follower receive a hello message before actual data
...
the leader can ensure the follower has connected
2024-03-15 20:36:01 -07:00
chrislu
64a32730a2
add size()
2024-03-15 20:34:39 -07:00
Michał Frąckiewicz
205829fa22
Document default method used by compact ( #5379 )
2024-03-15 12:16:44 -07:00
Konstantin Lebedev
54ee73287d
[filer] replace an invalid methods in a metric with an invalid one ( #5378 )
...
fix: replace an invalid methods in a metric with an invalid one for filer
2024-03-14 08:19:09 -07:00
chrislu
6f615c62ae
Merge branch 'master' into mq-subscribe
2024-03-12 09:19:06 -07:00
chrislu
604091a480
use stopChan to close previous filer peer meta subscription instances
2024-03-12 09:18:54 -07:00
chrislu
1efa502dbb
go fmt
2024-03-12 09:17:20 -07:00
chrislu
6f75a0af55
ask follower to follow
2024-03-10 14:34:28 -07:00
chrislu
c1a5952009
rename
2024-03-09 12:57:01 -08:00
chrislu
e7823ee967
retry connecting to broker leader
2024-03-09 12:56:49 -08:00
rustrover
ab70aa92da
remove repetitive words ( #5364 )
2024-03-09 07:35:47 -08:00
chrislu
3b28433cb1
change LogBuffer visibility in LocalPartition
2024-03-07 10:57:04 -08:00
chrislu
28f8f33d6a
include key in LogEntry
2024-03-07 10:53:30 -08:00
chrislu
34f2b600ac
each log function adds a "done" return parameter
2024-03-07 10:50:09 -08:00
chrislu
62397f2371
adjust error
2024-03-07 10:47:50 -08:00
chrislu
832b60c13f
pass errors
2024-03-07 10:47:06 -08:00
chrislu
49869eec83
edge cases for broker assignment
2024-03-07 10:45:38 -08:00
chrislu
33ab6cfcf7
adjust error
2024-03-07 10:44:09 -08:00
chrislu
8f79bb3987
Merge branch 'master' into mq-subscribe
2024-03-07 10:42:29 -08:00
chrislu
b544a69550
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-03-07 10:42:15 -08:00
chrislu
5c0f01c0eb
purge cluster id
2024-03-07 10:42:12 -08:00
Konstantin Lebedev
170b63d6f8
[filer.backup] add param uploader_part_size for S3sink ( #5352 )
...
* fix: install cronie
* chore: refactor configure S3Sink
* chore: refactor cinfig
* add filer-backup compose file
* fix: X-Amz-Meta-Mtime and resolve with comments
* fix: attr mtime
* fix: MaxUploadPartst is reduced to the maximum allowable
* fix: env and force set max MaxUploadParts
* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri
ecc154fa9e
Add deleted bytes to total_disk_size ( #5161 )
2024-03-06 07:28:46 -08:00
adasauce
3e3e0fa240
feat: add support for S3 ListObjects and ListObjectsV2 ( #5350 )
2024-03-06 07:24:44 -08:00
Konstantin Lebedev
872b5161e6
[shell] fs log purge ( #5344 )
...
* fix: install cronie
* chore: shell delete old logs
* Revert "fix: install cronie"
This reverts commit 920832b04b
2024-03-05 09:09:08 -08:00
chrislu
fe03b1b522
handle single broker case
2024-03-01 05:59:40 -08:00
chrislu
50c5dd7313
test low active brokers
2024-02-29 20:50:57 -08:00
chrislu
7598922b41
assign followers
2024-02-29 14:51:06 -08:00
chrislu
7e443ef0a1
latest protoc-gen-go
2024-02-29 10:06:23 -08:00
chrislu
ead27a4f59
publish init add followers
2024-02-29 09:39:12 -08:00
chrislu
1b4484bf0a
go fmt
2024-02-29 09:38:52 -08:00
chrislu
2a7028373d
Merge branch 'master' into mq-subscribe
2024-02-29 06:22:39 -08:00
Sean Ross
0833057503
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… ( #5351 )
...
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.
2024-02-29 02:16:28 -08:00
jerebear12
85d6d5371b
Disable filer UI in configuration ( #5297 )
...
* Add filer.ui.enabled configuration property
* Add filer.expose_directory_metadata to config
* Ammend commit
* Remove ShowUI reference
* Update all routes that allow directory metadata
* Add cmd flag to server.go
2024-02-27 08:38:55 -08:00
chrislu
76ae39a363
Merge branch 'master' into mq-subscribe
2024-02-26 07:03:09 -08:00
chrislu
54d7748a4a
3.63
2024-02-25 22:17:28 -08:00
chrislu
a52ce076fc
Merge branch 'master' into mq-subscribe
2024-02-25 08:00:37 -08:00
Konstantin Lebedev
7187346cc1
avoid unexpected compact size ( #5272 )
...
https://github.com/seaweedfs/seaweedfs/issues/5215
2024-02-24 05:27:35 -08:00
chrislu
7c45992c79
when verifyNeedleIntegrity, do not truncate files
...
related to https://github.com/seaweedfs/seaweedfs/pull/5259
2024-02-24 02:37:03 -08:00
Chris Lu
32cb9b3276
fix unaligned 64-bit atomic operation in weed mount on arm when readi… ( #5308 )
...
fix unaligned 64-bit atomic operation in weed mount on arm when reading a file
fix #5306
2024-02-19 18:51:45 -08:00
7y-9
41ec9d7759
refactor: put the auth outside ( #5313 )
2024-02-19 03:07:48 -08:00
7y-9
f037c09c11
fix: only admin auth can delete S3 bucket ( #5312 )
2024-02-19 00:38:05 -08:00
Konstantin Lebedev
6181aa7594
fix: fs verify error counter ( #5261 )
2024-02-15 01:31:51 -08:00
Sébastien
c694cb4e7d
filer: reduce delete entry ErrNotFound logs ( #5301 )
2024-02-14 14:22:08 -08:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) ( #5304 )
2024-02-14 14:18:56 -08:00
Justin Dhillon
a48e2ec45b
Fix Broken Links ( #5287 )
...
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize
* remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css
* https://github.com/AShiou/hof
2024-02-14 08:26:38 -08:00
Chris Lu
16b4f9d9ab
subscriber find broker leader first
2024-02-05 23:14:25 -08:00
Chris Lu
e27deed4bc
upgrade protoc
2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af
proxy to broker leader
2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26
report EOF error
2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864
minor
2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8
fix on finding lock owner
2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833
proxy requests to lock owner
2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header ( #5280 )
...
* Add s3 postpolicy support header
* optimized code
2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224
fix divide by zero
2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f
fix lock TTL
2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99
less verbose
2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c
Merge branch 'master' into mq-subscribe
2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f
fix when two consecutive chunks with the same modified_ts_ns
...
fix https://github.com/seaweedfs/seaweedfs/issues/5276
2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d
avoid too large expiration time
2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType ( #5275 )
2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c
avoid timestamp overflow
2024-02-03 13:45:59 -08:00
chrislu
d41792461c
lock returns host and owner
2024-02-02 15:54:57 -08:00
chrislu
d30150dde1
use a short-lived lock
2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709
fix nil error
2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822
re-lock if the lock owner is not found
2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types ( #5245 )
...
* week/shell: Cluster check other disk types
The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.
This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
* weed/shell: Replace loop that copies slice
Use the following construct instead of a `for` loop:
```golang
x = append(x, y...)
```
See https://staticcheck.dev/docs/checks#S1011 .
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
* weed/shell: Check disk types when filer is in use
Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
---------
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2024-01-29 10:36:37 -08:00
