Commit Graph

339 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
ff47956530 add helper functions 2024-10-03 00:13:46 -07:00
chrislu
480fcb1d03 fix build 2024-08-12 09:45:51 -07:00
chrislu
ff66155302 fix tests 2024-08-12 09:30:42 -07:00
chrislu
270e91b0be adds locking 2024-08-11 13:06:01 -07:00
chrislu
b6fd1ff4ce refactor 2024-08-10 12:39:16 -07:00
chrislu
7438648d1c Merge branch 'master' into mq 2024-08-10 10:01:57 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
cb1ba0275a sort imports 2024-06-14 11:44:42 -07:00
chrislu
32dbb6a2ce Merge branch 'master' into mq 2024-06-14 11:42:05 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628)
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
chrislu
25b2850e7d refactor out FilerClientAccessor 2024-06-06 19:44:19 -07:00
chrislu
d1f0c40402 remove per-message debug logs 2024-05-30 09:49:08 -07:00
chrislu
df9d889489 fix size calculation 2024-05-30 09:41:04 -07:00
chrislu
ff14dfa0ee debug 2024-05-30 09:15:50 -07:00
chrislu
b485b9695c fix passing sub follower offset 2024-05-30 09:15:23 -07:00
chrislu
f7b818483a fix cases when buffer wraps around 2024-05-30 09:10:30 -07:00
chrislu
9a5cc065b0 fix 2024-05-30 08:29:06 -07:00
chrislu
23a72db1df stop partitionOffsetChan if closed 2024-05-30 00:27:44 -07:00
chrislu
af19256dbf skip control messages 2024-05-29 23:34:39 -07:00
chrislu
ad2b878130 fix inflight message tracker 2024-05-29 23:33:37 -07:00
chrislu
f8a22038c5 passing along publisher name 2024-05-29 23:31:48 -07:00
chrislu
f0b9e163a0 avoid nil stopChan 2024-05-29 16:18:02 -07:00
chrislu
a5874954dc less logs 2024-05-29 16:17:37 -07:00
chrislu
edc87c16f9 balance subscribers
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
chrislu
6366898af2 add market 2024-05-26 14:35:22 -07:00
chrislu
96b326a304 use type ConsumerGroupInstanceId 2024-05-26 14:09:51 -07:00
chrislu
37d1ee562d refactor 2024-05-23 09:21:48 -07:00
chrislu
cdeaaf95b4 go fmt 2024-05-23 08:25:53 -07:00
chrislu
d40b350f54 coordinator receives unassignment ack 2024-05-23 08:23:35 -07:00
chrislu
554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa rename file 2024-05-21 09:57:59 -07:00
chrislu
67e6051585 rename Coordinator to SubCoordinator 2024-05-21 09:57:45 -07:00
chrislu
b5099263a4 rename file 2024-05-21 09:56:46 -07:00
chrislu
d5abffa42c rename Balancer to PubBalancer 2024-05-21 09:56:30 -07:00
chrislu
fa98ecf71e client side stop partition subscribing if unassigned 2024-05-21 08:42:04 -07:00
chrislu
6634b42981 fix compilation 2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3 refactor
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6 avoid concurrent processing for the same key 2024-05-20 13:29:37 -07:00
chrislu
a7d30d0705 refactor 2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4 remove ProcessorConfiguration 2024-05-20 12:28:01 -07:00
chrislu
65dd5ac6fb fix 2024-05-20 12:27:45 -07:00
chrislu
f13475ac2e sending keyed offset 2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d minor 2024-05-20 11:07:54 -07:00
chrislu
3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu
5038577f7e Merge branch 'master' into mq 2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
7a4ace5dfb adjust logs 2024-05-20 09:34:07 -07:00
chrislu
d8ab22012a track offset 2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5 add InflightMessageTracker 2024-05-20 09:19:39 -07:00