Commit Graph

15 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
7a9b115cc2 rename 2024-05-02 11:14:58 -07:00
chrislu
f2f4630029 rename field types 2024-05-02 08:59:22 -07:00
chrislu
b8af997e41 consistent with protobuf data types 2024-05-02 08:52:10 -07:00
chrislu
5bc1b70a4c rename to WithField() 2024-05-02 08:32:15 -07:00
chrislu
189a7fc90e record type begin and end 2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9 improve api 2024-04-28 12:42:58 -07:00
chrislu
1a2332c257 rename api 2024-04-28 12:40:00 -07:00
chrislu
8a0e5b6d54 struct to schema correctly 2024-04-28 12:23:54 -07:00
chrislu
7310d022ad recursive struct to schema 2024-04-28 11:14:13 -07:00
chrislu
a2a872ca03 rename 2024-04-27 23:44:11 -07:00
chrislu
d7e5f6b2a5 fix ordering 2024-04-24 01:32:02 -07:00
chrislu
9ef238b610 has a bug 2024-04-22 00:42:18 -07:00
chrislu
05d18130b6 add schema builder 2024-04-18 22:27:18 -07:00