Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
...
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
480fcb1d03
fix build
2024-08-12 09:45:51 -07:00
chrislu
9a5cc065b0
fix
2024-05-30 08:29:06 -07:00
chrislu
23a72db1df
stop partitionOffsetChan if closed
2024-05-30 00:27:44 -07:00
chrislu
af19256dbf
skip control messages
2024-05-29 23:34:39 -07:00
chrislu
edc87c16f9
balance subscribers
...
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
cdeaaf95b4
go fmt
2024-05-23 08:25:53 -07:00
chrislu
d40b350f54
coordinator receives unassignment ack
2024-05-23 08:23:35 -07:00
chrislu
fa98ecf71e
client side stop partition subscribing if unassigned
2024-05-21 08:42:04 -07:00
chrislu
6634b42981
fix compilation
2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3
refactor
...
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
a7d30d0705
refactor
2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4
remove ProcessorConfiguration
2024-05-20 12:28:01 -07:00
chrislu
f13475ac2e
sending keyed offset
2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d
minor
2024-05-20 11:07:54 -07:00
chrislu
3622196881
go fmt
2024-05-20 11:05:18 -07:00
chrislu
d8ab22012a
track offset
2024-05-20 09:33:37 -07:00
chrislu
a2885512e1
fix
2024-05-19 15:06:33 -07:00
chrislu
8d02712c63
ConcurrentPartitionLimit
2024-05-19 14:55:03 -07:00
chrislu
c6db3f31a1
start consuming ASAP
2024-05-19 14:52:38 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
279cb2b85e
consumer acks received messages
2024-05-16 20:28:19 -07:00
chrislu
1f20178ded
subscriber receives partitions and dispatch to processors
2024-05-14 23:22:43 -07:00
chrislu
372bd8d71d
consumer instance passing MaxPartitionCount to coordinator
2024-05-13 18:30:10 -07:00
chrislu
8cc5298a08
clean up unused variables
2024-05-10 13:10:45 -07:00
chrislu
205d2285f3
setup follower by publisher
...
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
Chris Lu
16b4f9d9ab
subscriber find broker leader first
2024-02-05 23:14:25 -08:00
chrislu
3aa3991f0f
adjust client side logs
2024-01-11 23:08:53 -08:00
chrislu
49428a303b
add batch index for each memory buffer
2024-01-08 00:03:08 -08:00
chrislu
aa5b6a8187
adjust logs
2024-01-05 15:35:32 -08:00
chrislu
496fc8fbbf
refactor
2024-01-05 15:35:19 -08:00
chrislu
531f854af2
rename functions
2024-01-05 15:16:53 -08:00
chrislu
f8787a9761
rename functions
2024-01-05 15:16:41 -08:00
chrislu
ddd0fde094
rename functions
2024-01-05 15:16:01 -08:00
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
...
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
chrislu
8b4cd50aac
log errors
2024-01-01 19:09:17 -08:00
chrislu
32bc8d6a38
adjust wait time
2023-12-31 17:42:44 -08:00
chrislu
458ddbf919
clean up dead code
2023-12-31 13:25:33 -08:00
chrislu
2845230329
passing broker into the assignments
2023-12-28 21:13:49 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
chrislu
c950a40aad
subscriber keep connected to the balancer
2023-12-28 11:56:37 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
chrislu
bf13f3ced7
add subscriber coordinator
2023-10-02 01:02:27 -07:00
chrislu
2a578b9033
refactor
2023-10-01 11:59:19 -07:00
chrislu
ebd6f96d35
adjust mq.proto
2023-09-30 21:47:57 -07:00
chrislu
ba67e6ca29
api for sub
2023-09-04 21:43:50 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5
connect and publish
2023-08-28 09:02:12 -07:00
chrislu
4d6c18d86f
pub sub initial tests
2023-08-27 17:50:59 -07:00
