Commit Graph

101 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
480fcb1d03 fix build 2024-08-12 09:45:51 -07:00
chrislu
7438648d1c Merge branch 'master' into mq 2024-08-10 10:01:57 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
9a5cc065b0 fix 2024-05-30 08:29:06 -07:00
chrislu
23a72db1df stop partitionOffsetChan if closed 2024-05-30 00:27:44 -07:00
chrislu
af19256dbf skip control messages 2024-05-29 23:34:39 -07:00
chrislu
f8a22038c5 passing along publisher name 2024-05-29 23:31:48 -07:00
chrislu
edc87c16f9 balance subscribers
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
cdeaaf95b4 go fmt 2024-05-23 08:25:53 -07:00
chrislu
d40b350f54 coordinator receives unassignment ack 2024-05-23 08:23:35 -07:00
chrislu
fa98ecf71e client side stop partition subscribing if unassigned 2024-05-21 08:42:04 -07:00
chrislu
6634b42981 fix compilation 2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3 refactor
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
a7d30d0705 refactor 2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4 remove ProcessorConfiguration 2024-05-20 12:28:01 -07:00
chrislu
f13475ac2e sending keyed offset 2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d minor 2024-05-20 11:07:54 -07:00
chrislu
3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu
5038577f7e Merge branch 'master' into mq 2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
d8ab22012a track offset 2024-05-20 09:33:37 -07:00
chrislu
a2885512e1 fix 2024-05-19 15:06:33 -07:00
chrislu
8d02712c63 ConcurrentPartitionLimit 2024-05-19 14:55:03 -07:00
chrislu
c6db3f31a1 start consuming ASAP 2024-05-19 14:52:38 -07:00
chrislu
b1871427c3 persist consumer group offset
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
279cb2b85e consumer acks received messages 2024-05-16 20:28:19 -07:00
chrislu
1f20178ded subscriber receives partitions and dispatch to processors 2024-05-14 23:22:43 -07:00
chrislu
372bd8d71d consumer instance passing MaxPartitionCount to coordinator 2024-05-13 18:30:10 -07:00
chrislu
8cc5298a08 clean up unused variables 2024-05-10 13:10:45 -07:00
chrislu
664b704e30 add delay 2024-05-05 14:17:41 -07:00
chrislu
4e3701df77 reset last error 2024-05-05 14:16:54 -07:00
chrislu
7a9b115cc2 rename 2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb subscriber can unmarshal the record 2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7 rename 2024-05-02 09:47:29 -07:00
chrislu
f2f4630029 rename field types 2024-05-02 08:59:22 -07:00
chrislu
b8af997e41 consistent with protobuf data types 2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea add example to publish a record 2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff rename 2024-05-02 08:35:06 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
yudrywet
8833745fbc
chore: fix some typos in comments (#5497) 2024-04-14 07:42:40 -07:00
chrislu
68ade92d4f publisher send schema 2024-04-12 23:36:15 -07:00
chrislu
5324e4288a rename parameter 2024-04-12 22:33:00 -07:00
chrislu
66a878af39 add control message to data message 2024-03-31 16:35:46 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
205d2285f3 setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
da31e9b939 correctly wait for the publisher to finish 2024-03-21 09:08:53 -07:00
chrislu
7003ce7425 publisher is able to fully send data and receive acks
still needs to close the pipes cleanly
2024-03-20 12:25:40 -07:00
chrislu
33ab6cfcf7 adjust error 2024-03-07 10:44:09 -08:00