Commit Graph

45 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
7438648d1c Merge branch 'master' into mq 2024-08-10 10:01:57 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
cdeaaf95b4 go fmt 2024-05-23 08:25:53 -07:00
chrislu
33aecb7bb4 remove ProcessorConfiguration 2024-05-20 12:28:01 -07:00
chrislu
3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu
5038577f7e Merge branch 'master' into mq 2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
d8ab22012a track offset 2024-05-20 09:33:37 -07:00
chrislu
8d02712c63 ConcurrentPartitionLimit 2024-05-19 14:55:03 -07:00
chrislu
8cc5298a08 clean up unused variables 2024-05-10 13:10:45 -07:00
chrislu
664b704e30 add delay 2024-05-05 14:17:41 -07:00
chrislu
7a9b115cc2 rename 2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb subscriber can unmarshal the record 2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7 rename 2024-05-02 09:47:29 -07:00
chrislu
f2f4630029 rename field types 2024-05-02 08:59:22 -07:00
chrislu
b8af997e41 consistent with protobuf data types 2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea add example to publish a record 2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff rename 2024-05-02 08:35:06 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
yudrywet
8833745fbc
chore: fix some typos in comments (#5497) 2024-04-14 07:42:40 -07:00
chrislu
5324e4288a rename parameter 2024-04-12 22:33:00 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
1b4484bf0a go fmt 2024-02-29 09:38:52 -08:00
Chris Lu
16b4f9d9ab subscriber find broker leader first 2024-02-05 23:14:25 -08:00
chrislu
dedfd31dfb refactor 2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f refactor 2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff wait for publishing clients 2024-01-28 12:06:58 -08:00
chrislu
841fafd0a8 publish to input buffer
currently, the input buffer may not exist when start to publish
2024-01-27 23:43:22 -08:00
chrislu
7e6497cc1c adjust publisher subscriber 2024-01-15 20:42:46 -08:00
chrislu
026c54a9bb fix publisher 2024-01-15 00:20:21 -08:00
chrislu
49428a303b add batch index for each memory buffer 2024-01-08 00:03:08 -08:00
chrislu
ddd0fde094 rename functions 2024-01-05 15:16:01 -08:00
chrislu
47a4963d7a subscription start from specified timestamp 2024-01-03 15:57:36 -08:00
chrislu
35869b5c80 subscriber can be notified of the assignment change when topic is just configured
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
chrislu
458ddbf919 clean up dead code 2023-12-31 13:25:33 -08:00
chrislu
c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00
chrislu
2a578b9033 refactor 2023-10-01 11:59:19 -07:00
chrislu
39941edc0b add publisher shutdown 2023-09-07 23:55:19 -07:00
chrislu
ba67e6ca29 api for sub 2023-09-04 21:43:50 -07:00
chrislu
9e4f985698 publish, benchmark 2023-09-04 21:43:30 -07:00
chrislu
cb470d44df can pub and sub 2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5 connect and publish 2023-08-28 09:02:12 -07:00