Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
...
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
25b2850e7d
refactor out FilerClientAccessor
2024-06-06 19:44:19 -07:00
chrislu
d1f0c40402
remove per-message debug logs
2024-05-30 09:49:08 -07:00
chrislu
df9d889489
fix size calculation
2024-05-30 09:41:04 -07:00
chrislu
f7b818483a
fix cases when buffer wraps around
2024-05-30 09:10:30 -07:00
chrislu
ad2b878130
fix inflight message tracker
2024-05-29 23:33:37 -07:00
chrislu
f0b9e163a0
avoid nil stopChan
2024-05-29 16:18:02 -07:00
chrislu
a5874954dc
less logs
2024-05-29 16:17:37 -07:00
chrislu
edc87c16f9
balance subscribers
...
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
6366898af2
add market
2024-05-26 14:35:22 -07:00
chrislu
96b326a304
use type ConsumerGroupInstanceId
2024-05-26 14:09:51 -07:00
chrislu
37d1ee562d
refactor
2024-05-23 09:21:48 -07:00
chrislu
d40b350f54
coordinator receives unassignment ack
2024-05-23 08:23:35 -07:00
chrislu
554ae09f82
rename
2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa
rename file
2024-05-21 09:57:59 -07:00
chrislu
67e6051585
rename Coordinator to SubCoordinator
2024-05-21 09:57:45 -07:00
chrislu
d5abffa42c
rename Balancer to PubBalancer
2024-05-21 09:56:30 -07:00
chrislu
2190d59bc3
refactor
...
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6
avoid concurrent processing for the same key
2024-05-20 13:29:37 -07:00
chrislu
3622196881
go fmt
2024-05-20 11:05:18 -07:00
chrislu
d8ab22012a
track offset
2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5
add InflightMessageTracker
2024-05-20 09:19:39 -07:00
chrislu
c6db3f31a1
start consuming ASAP
2024-05-19 14:52:38 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
759a8e50cf
remove logs
2024-05-14 23:22:51 -07:00
chrislu
972e9faaa2
move initial assignment to rebalance logic
2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e
move read write topic config into filer client accessor
2024-05-14 08:50:17 -07:00
chrislu
ac63f2b5a1
refactor
2024-05-13 22:03:57 -07:00
chrislu
844a1a0192
rename
2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d
consumer instance passing MaxPartitionCount to coordinator
2024-05-13 18:30:10 -07:00
chrislu
2142842f82
refactor
2024-05-13 18:25:38 -07:00
chrislu
205d2285f3
setup follower by publisher
...
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
1b4484bf0a
go fmt
2024-02-29 09:38:52 -08:00
chrislu
34839237ab
refactor
2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30
refactor
2024-01-20 11:41:11 -08:00
chrislu
f750a5e03b
passing timestamp
2024-01-11 23:03:55 -08:00
chrislu
6a7a679137
passing partition generation timestamp
2024-01-10 22:36:17 -08:00
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
...
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
chrislu
2845230329
passing broker into the assignments
2023-12-28 21:13:49 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
chrislu
c950a40aad
subscriber keep connected to the balancer
2023-12-28 11:56:37 -08:00
chrislu
e0727071c8
go fmt
2023-12-22 11:33:50 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
