chrislu
5038577f7e
Merge branch 'master' into mq
2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
8d5bb7420d
rename
2024-05-17 18:21:21 -07:00
chrislu
1f20178ded
subscriber receives partitions and dispatch to processors
2024-05-14 23:22:43 -07:00
chrislu
ccdd9cd8de
refactor
2024-04-02 16:25:43 -07:00
chrislu
f37c0d0d7a
comment out println
2024-04-02 15:31:31 -07:00
chrislu
f07875e8e1
send flush message to follower before shutting down logBuffer
2024-04-01 16:01:26 -07:00
chrislu
e568e742c9
renaming
2024-04-01 15:02:52 -07:00
chrislu
47975e75bf
purge old code
2024-03-31 17:49:34 -07:00
chrislu
237e68059d
shutdown follower if no clients, not only publishers
...
to avoid overhead when a publisher repeatedly send one message and close
2024-03-31 17:45:26 -07:00
chrislu
c9df613b6b
add publisher name for debugging
2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6
simplify
2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75
logs
2024-03-30 20:37:46 -07:00
chrislu
e3c8be7f89
LocalPartition shutdown
2024-03-27 23:55:06 -07:00
chrislu
ab9fc0ad70
simplify
2024-03-27 23:42:19 -07:00
chrislu
7463058299
change visibility
2024-03-27 23:11:35 -07:00
chrislu
dab5454332
publish and send to follower
2024-03-27 23:10:24 -07:00
chrislu
f249f831b1
MaybeShutdownLocalPartition refactor
2024-03-27 23:02:51 -07:00
chrislu
2a0ab7e90d
fix
2024-03-27 22:59:50 -07:00
chrislu
059a120708
refactor
2024-03-27 22:58:20 -07:00
chrislu
7d87c1d2bb
refactor
2024-03-27 22:48:37 -07:00
chrislu
5cc94a05b9
separate goroutine to send ack to publisher
2024-03-27 10:27:08 -07:00
chrislu
f67c8309d8
rename
2024-03-26 21:52:12 -07:00
chrislu
205d2285f3
setup follower by publisher
...
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
6ef3e6bcad
logging
2024-03-17 22:24:20 -07:00
chrislu
8e5068fd2f
notify
2024-03-16 23:16:33 -07:00
chrislu
529fa8c4c8
go fmt
2024-03-16 10:51:47 -07:00
chrislu
64a32730a2
add size()
2024-03-15 20:34:39 -07:00
chrislu
6f75a0af55
ask follower to follow
2024-03-10 14:34:28 -07:00
chrislu
3b28433cb1
change LogBuffer visibility in LocalPartition
2024-03-07 10:57:04 -08:00
chrislu
1b4484bf0a
go fmt
2024-02-29 09:38:52 -08:00
chrislu
fdf0ea8e11
passing partition info
2024-01-16 09:30:57 -08:00
chrislu
3795d8dca8
release local topic partition if no publisher and subscribers
2024-01-16 08:43:07 -08:00
chrislu
ba73199174
reuse local partition
2024-01-15 21:22:41 -08:00
chrislu
fa59a5d67e
read from disk if not in memory
2024-01-15 00:20:12 -08:00
chrislu
45994641e9
lint
2024-01-11 23:03:35 -08:00
chrislu
6a7a679137
passing partition generation timestamp
2024-01-10 22:36:17 -08:00
chrislu
d51efddf5c
flush to disk
...
Need to do: read from disk
2024-01-08 23:27:02 -08:00
chrislu
49428a303b
add batch index for each memory buffer
2024-01-08 00:03:08 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
chrislu
bf13f3ced7
add subscriber coordinator
2023-10-02 01:02:27 -07:00
chrislu
734178093e
refactor TopicPartition struct
2023-10-02 01:01:45 -07:00
chrislu
23f334dd13
comment
2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71
ensure latest stats are reported
2023-09-24 23:34:31 -07:00
chrislu
1492bf7552
fix listing topics
2023-09-24 23:05:41 -07:00
chrislu
6c6655de9d
wip
2023-09-19 14:03:42 -07:00
chrislu
4827425146
balancer works
2023-09-16 15:05:38 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
