Commit Graph

338 Commits

Author SHA1 Message Date
chrislu
8d02712c63 ConcurrentPartitionLimit 2024-05-19 14:55:03 -07:00
chrislu
8ce2cced47 less logs 2024-05-19 14:54:46 -07:00
chrislu
c6db3f31a1 start consuming ASAP 2024-05-19 14:52:38 -07:00
chrislu
c21b32dd20 logs 2024-05-19 11:02:17 -07:00
chrislu
b1871427c3 persist consumer group offset
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
chrislu
8d5bb7420d rename 2024-05-17 18:21:21 -07:00
chrislu
94742b8ace add subscribe follower 2024-05-17 18:17:23 -07:00
chrislu
279cb2b85e consumer acks received messages 2024-05-16 20:28:19 -07:00
chrislu
faffb2973c adjust errors 2024-05-16 11:02:48 -07:00
chrislu
759a8e50cf remove logs 2024-05-14 23:22:51 -07:00
chrislu
1f20178ded subscriber receives partitions and dispatch to processors 2024-05-14 23:22:43 -07:00
chrislu
972e9faaa2 move initial assignment to rebalance logic 2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
chrislu
ac63f2b5a1 refactor 2024-05-13 22:03:57 -07:00
chrislu
844a1a0192 rename 2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d consumer instance passing MaxPartitionCount to coordinator 2024-05-13 18:30:10 -07:00
chrislu
2142842f82 refactor 2024-05-13 18:25:38 -07:00
chrislu
8cc5298a08 clean up unused variables 2024-05-10 13:10:45 -07:00
chrislu
fcedf97e94 fix test 2024-05-05 22:19:59 -07:00
chrislu
664b704e30 add delay 2024-05-05 14:17:41 -07:00
chrislu
4e3701df77 reset last error 2024-05-05 14:16:54 -07:00
chrislu
30f8600a24 working 2024-05-03 21:27:06 -07:00
chrislu
7a9b115cc2 rename 2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb subscriber can unmarshal the record 2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7 rename 2024-05-02 09:47:29 -07:00
chrislu
f2f4630029 rename field types 2024-05-02 08:59:22 -07:00
chrislu
b8af997e41 consistent with protobuf data types 2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea add example to publish a record 2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff rename 2024-05-02 08:35:06 -07:00
chrislu
5bc1b70a4c rename to WithField() 2024-05-02 08:32:15 -07:00
chrislu
b570e015ce Update write_parquet_test.go 2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01 record value API 2024-04-28 13:42:45 -07:00
chrislu
b166597c83 fix test 2024-04-28 13:03:01 -07:00
chrislu
189a7fc90e record type begin and end 2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9 improve api 2024-04-28 12:42:58 -07:00
chrislu
1a2332c257 rename api 2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546 Delete schema_builder_test.go 2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54 struct to schema correctly 2024-04-28 12:23:54 -07:00
chrislu
f37862bffe compile test 2024-04-28 12:23:33 -07:00
chrislu
7310d022ad recursive struct to schema 2024-04-28 11:14:13 -07:00
chrislu
a2a872ca03 rename 2024-04-27 23:44:11 -07:00
chrislu
60103171b6 rename 2024-04-27 23:30:15 -07:00
chrislu
72b50980f4 a little bit more efficient 2024-04-25 23:59:30 -07:00
chrislu
63036133ff minor 2024-04-25 22:02:03 -07:00
chrislu
2a9b0a9332 use parquet levels to construct row 2024-04-25 21:29:45 -07:00
chrislu
977e7988e6 toRow conversion with levels info 2024-04-25 09:14:37 -07:00
chrislu
9cb9d27b5b file rename 2024-04-25 08:56:31 -07:00
chrislu
313097aca0 printout test values 2024-04-25 08:54:57 -07:00
chrislu
827e293010 add visitor 2024-04-25 08:54:41 -07:00
chrislu
138a027a2e add field types but without value 2024-04-24 23:33:11 -07:00
chrislu
30b5aef62e record type is also optional 2024-04-24 23:32:35 -07:00
chrislu
ce4597c623 delete the generated file 2024-04-24 23:29:55 -07:00
chrislu
f91484ca07 add some notes 2024-04-24 23:28:20 -07:00
chrislu
f93c4cf48c adjust test 2024-04-24 23:24:53 -07:00
chrislu
d88c1872ac works with single level of list 2024-04-24 23:04:47 -07:00
chrislu
d7e5f6b2a5 fix ordering 2024-04-24 01:32:02 -07:00
chrislu
9ef238b610 has a bug 2024-04-22 00:42:18 -07:00
chrislu
cc1ce3de87 all record fields are optional by default 2024-04-18 23:17:52 -07:00
chrislu
384ed7a381 add test 2024-04-18 22:41:20 -07:00
chrislu
57949f9959 support list type 2024-04-18 22:41:12 -07:00
chrislu
05d18130b6 add schema builder 2024-04-18 22:27:18 -07:00
chrislu
0847a17484 convert to parquet schema and value 2024-04-17 23:49:21 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
yudrywet
8833745fbc
chore: fix some typos in comments (#5497) 2024-04-14 07:42:40 -07:00
chrislu
68ade92d4f publisher send schema 2024-04-12 23:36:15 -07:00
chrislu
8415476c2d fix test 2024-04-12 23:35:54 -07:00
chrislu
5324e4288a rename parameter 2024-04-12 22:33:00 -07:00
chrislu
6f80dbdfe5 Create schema.go 2024-04-12 22:31:28 -07:00
chrislu
53d1d2b78a save schema when configuring topic 2024-04-12 22:29:53 -07:00
chrislu
d45a372615 add schema.proto 2024-04-12 01:30:29 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
chrislu
f37c0d0d7a comment out println 2024-04-02 15:31:31 -07:00
chrislu
d935f70e3c shutdown follower when leader stops 2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1 send flush message to follower before shutting down logBuffer 2024-04-01 16:01:26 -07:00
chrislu
e568e742c9 renaming 2024-04-01 15:02:52 -07:00
chrislu
47975e75bf purge old code 2024-03-31 17:49:34 -07:00
chrislu
237e68059d shutdown follower if no clients, not only publishers
to avoid overhead when a publisher repeatedly send one message and close
2024-03-31 17:45:26 -07:00
chrislu
66a878af39 add control message to data message 2024-03-31 16:35:46 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6 simplify 2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75 logs 2024-03-30 20:37:46 -07:00
chrislu
e3c8be7f89 LocalPartition shutdown 2024-03-27 23:55:06 -07:00
chrislu
ab9fc0ad70 simplify 2024-03-27 23:42:19 -07:00
chrislu
7463058299 change visibility 2024-03-27 23:11:35 -07:00
chrislu
dab5454332 publish and send to follower 2024-03-27 23:10:24 -07:00
chrislu
f249f831b1 MaybeShutdownLocalPartition refactor 2024-03-27 23:02:51 -07:00
chrislu
2a0ab7e90d fix 2024-03-27 22:59:50 -07:00
chrislu
059a120708 refactor 2024-03-27 22:58:20 -07:00
chrislu
7d87c1d2bb refactor 2024-03-27 22:48:37 -07:00
chrislu
5cc94a05b9 separate goroutine to send ack to publisher 2024-03-27 10:27:08 -07:00
chrislu
17806cde2a separate context
If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.
2024-03-27 10:24:57 -07:00
chrislu
f67c8309d8 rename 2024-03-26 21:52:12 -07:00
chrislu
d6a066495b adjust logs 2024-03-25 08:15:47 -07:00
chrislu
1b1b96ea1e close local partition if no more publishers and subscribers 2024-03-24 21:10:02 -07:00
chrislu
4f5c4c3388 refactor 2024-03-24 13:07:51 -07:00
chrislu
d558a58946 refactor 2024-03-24 13:04:59 -07:00
chrislu
205d2285f3 setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
chrislu
da31e9b939 correctly wait for the publisher to finish 2024-03-21 09:08:53 -07:00
chrislu
7003ce7425 publisher is able to fully send data and receive acks
still needs to close the pipes cleanly
2024-03-20 12:25:40 -07:00