Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
...
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
554ae09f82
rename
2024-05-21 10:02:07 -07:00
chrislu
5038577f7e
Merge branch 'master' into mq
2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
6e5075e14e
move read write topic config into filer client accessor
2024-05-14 08:50:17 -07:00
chrislu
53d1d2b78a
save schema when configuring topic
2024-04-12 22:29:53 -07:00
Chris Lu
656b78d1af
proxy to broker leader
2024-02-05 18:00:50 -08:00
Chris Lu
4dc6681833
proxy requests to lock owner
2024-02-05 16:46:40 -08:00
chrislu
9e6ea80dff
read existing topic conf
2024-01-28 14:27:52 -08:00
chrislu
d268fbe18a
when configure, cancel existing assignments
2024-01-22 11:57:17 -08:00
chrislu
c3f8530f97
refactor
2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0
no need to assign partitions explicitly
2024-01-22 11:04:47 -08:00
chrislu
e8b05ecc91
add/remove assigned partitions
2024-01-22 10:47:39 -08:00
chrislu
861ad732ca
refactor
2024-01-22 09:00:22 -08:00
chrislu
47924afa1c
refactor
2024-01-22 08:52:44 -08:00
chrislu
7121ae9617
separate out read topic conf and ensure topic active assignments
2024-01-22 00:51:31 -08:00
chrislu
a5b906c0dd
refactor
2024-01-21 01:23:43 -08:00
chrislu
bc8d2a01cc
create local topic partition from config on filer
2024-01-17 00:15:52 -08:00
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5
simplify api
2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
...
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
chrislu
ba73199174
reuse local partition
2024-01-15 21:22:41 -08:00
chrislu
fa59a5d67e
read from disk if not in memory
2024-01-15 00:20:12 -08:00
chrislu
6a7a679137
passing partition generation timestamp
2024-01-10 22:36:17 -08:00
chrislu
8af61dec91
avoid empty spaces in file name
2024-01-09 09:00:54 -08:00
chrislu
aed54eda61
refactor
2024-01-09 08:01:01 -08:00
chrislu
d51efddf5c
flush to disk
...
Need to do: read from disk
2024-01-08 23:27:02 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
