Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
...
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
ff66155302
fix tests
2024-08-12 09:30:42 -07:00
chrislu
edc87c16f9
balance subscribers
...
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
554ae09f82
rename
2024-05-21 10:02:07 -07:00
chrislu
b5099263a4
rename file
2024-05-21 09:56:46 -07:00
chrislu
d5abffa42c
rename Balancer to PubBalancer
2024-05-21 09:56:30 -07:00
chrislu
5038577f7e
Merge branch 'master' into mq
2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
b166597c83
fix test
2024-04-28 13:03:01 -07:00
chrislu
ab9fc0ad70
simplify
2024-03-27 23:42:19 -07:00
chrislu
ea2601799c
fix test
2024-03-16 23:22:17 -07:00
chrislu
529fa8c4c8
go fmt
2024-03-16 10:51:47 -07:00
chrislu
49869eec83
edge cases for broker assignment
2024-03-07 10:45:38 -08:00
chrislu
fe03b1b522
handle single broker case
2024-03-01 05:59:40 -08:00
chrislu
50c5dd7313
test low active brokers
2024-02-29 20:50:57 -08:00
chrislu
7598922b41
assign followers
2024-02-29 14:51:06 -08:00
chrislu
1b4484bf0a
go fmt
2024-02-29 09:38:52 -08:00
Chris Lu
0d74ac9224
fix divide by zero
2024-02-04 13:38:00 -08:00
chrislu
428fea45f3
updated and added assignments
2024-01-22 11:04:24 -08:00
chrislu
e8b05ecc91
add/remove assigned partitions
2024-01-22 10:47:39 -08:00
chrislu
b0a2e9aea3
fix assignments if brokers changed
2024-01-21 01:27:22 -08:00
chrislu
9ce7c482b3
refactor
2024-01-21 01:24:12 -08:00
chrislu
34839237ab
refactor
2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30
refactor
2024-01-20 11:41:11 -08:00
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5
simplify api
2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
...
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
chrislu
f782165638
rename
2024-01-16 08:40:45 -08:00
chrislu
dd1ec70e62
rename
2024-01-16 08:40:23 -08:00
chrislu
6a7a679137
passing partition generation timestamp
2024-01-10 22:36:17 -08:00
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
...
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
chrislu
efb695fd93
lookup existing topic partitions
2024-01-02 17:29:35 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
chrislu
e0727071c8
go fmt
2023-12-22 11:33:50 -08:00
chrislu
61b05fad95
fix test
2023-12-11 20:57:29 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
