Commit Graph

17 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu
6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
chrislu
e568e742c9 renaming 2024-04-01 15:02:52 -07:00
chrislu
ab9fc0ad70 simplify 2024-03-27 23:42:19 -07:00
chrislu
4f5c4c3388 refactor 2024-03-24 13:07:51 -07:00
chrislu
d558a58946 refactor 2024-03-24 13:04:59 -07:00
chrislu
205d2285f3 setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
55714f54ab local partition is generated or not 2024-03-15 21:08:42 -07:00
chrislu
7598922b41 assign followers 2024-02-29 14:51:06 -08:00
chrislu
a507069d19 lock for creating local partition 2024-01-28 15:55:26 -08:00
chrislu
841fafd0a8 publish to input buffer
currently, the input buffer may not exist when start to publish
2024-01-27 23:43:22 -08:00
chrislu
2beaa2d0b3 pub/sub brokers check filer for assigned partitions 2024-01-22 11:44:21 -08:00
chrislu
428fea45f3 updated and added assignments 2024-01-22 11:04:24 -08:00
chrislu
7121ae9617 separate out read topic conf and ensure topic active assignments 2024-01-22 00:51:31 -08:00
chrislu
b0a2e9aea3 fix assignments if brokers changed 2024-01-21 01:27:22 -08:00
chrislu
a5b906c0dd refactor 2024-01-21 01:23:43 -08:00