1
0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-29 04:27:54 +08:00
Commit Graph

42 Commits

Author SHA1 Message Date
chrislu
25b2850e7d refactor out FilerClientAccessor 2024-06-06 19:44:19 -07:00
chrislu
d1f0c40402 remove per-message debug logs 2024-05-30 09:49:08 -07:00
chrislu
df9d889489 fix size calculation 2024-05-30 09:41:04 -07:00
chrislu
f7b818483a fix cases when buffer wraps around 2024-05-30 09:10:30 -07:00
chrislu
ad2b878130 fix inflight message tracker 2024-05-29 23:33:37 -07:00
chrislu
f0b9e163a0 avoid nil stopChan 2024-05-29 16:18:02 -07:00
chrislu
a5874954dc less logs 2024-05-29 16:17:37 -07:00
chrislu
edc87c16f9 balance subscribers
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
6366898af2 add market 2024-05-26 14:35:22 -07:00
chrislu
96b326a304 use type ConsumerGroupInstanceId 2024-05-26 14:09:51 -07:00
chrislu
37d1ee562d refactor 2024-05-23 09:21:48 -07:00
chrislu
d40b350f54 coordinator receives unassignment ack 2024-05-23 08:23:35 -07:00
chrislu
554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa rename file 2024-05-21 09:57:59 -07:00
chrislu
67e6051585 rename Coordinator to SubCoordinator 2024-05-21 09:57:45 -07:00
chrislu
d5abffa42c rename Balancer to PubBalancer 2024-05-21 09:56:30 -07:00
chrislu
2190d59bc3 refactor
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6 avoid concurrent processing for the same key 2024-05-20 13:29:37 -07:00
chrislu
3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu
d8ab22012a track offset 2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5 add InflightMessageTracker 2024-05-20 09:19:39 -07:00
chrislu
c6db3f31a1 start consuming ASAP 2024-05-19 14:52:38 -07:00
chrislu
b1871427c3 persist consumer group offset
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
759a8e50cf remove logs 2024-05-14 23:22:51 -07:00
chrislu
972e9faaa2 move initial assignment to rebalance logic 2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
chrislu
ac63f2b5a1 refactor 2024-05-13 22:03:57 -07:00
chrislu
844a1a0192 rename 2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d consumer instance passing MaxPartitionCount to coordinator 2024-05-13 18:30:10 -07:00
chrislu
2142842f82 refactor 2024-05-13 18:25:38 -07:00
chrislu
205d2285f3 setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
1b4484bf0a go fmt 2024-02-29 09:38:52 -08:00
chrislu
34839237ab refactor 2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30 refactor 2024-01-20 11:41:11 -08:00
chrislu
f750a5e03b passing timestamp 2024-01-11 23:03:55 -08:00
chrislu
6a7a679137 passing partition generation timestamp 2024-01-10 22:36:17 -08:00
chrislu
35869b5c80 subscriber can be notified of the assignment change when topic is just configured
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
chrislu
2845230329 passing broker into the assignments 2023-12-28 21:13:49 -08:00
chrislu
093fdc1621 subscriber can get assignments 2023-12-28 20:35:15 -08:00
chrislu
c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
chrislu
e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ()
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00