337 Commits

Author	SHA1	Message	Date
chrislu	8ce2cced47	less logs	2024-05-19 14:54:46 -07:00
chrislu	c6db3f31a1	start consuming ASAP	2024-05-19 14:52:38 -07:00
chrislu	c21b32dd20	logs	2024-05-19 11:02:17 -07:00
chrislu	b1871427c3	persist consumer group offset ... 1. use one follower 2. read write consumer group offset	2024-05-19 00:46:12 -07:00
chrislu	8d5bb7420d	rename	2024-05-17 18:21:21 -07:00
chrislu	94742b8ace	add subscribe follower	2024-05-17 18:17:23 -07:00
chrislu	279cb2b85e	consumer acks received messages	2024-05-16 20:28:19 -07:00
chrislu	faffb2973c	adjust errors	2024-05-16 11:02:48 -07:00
chrislu	759a8e50cf	remove logs	2024-05-14 23:22:51 -07:00
chrislu	1f20178ded	subscriber receives partitions and dispatch to processors	2024-05-14 23:22:43 -07:00
chrislu	972e9faaa2	move initial assignment to rebalance logic	2024-05-14 09:09:36 -07:00
chrislu	6e5075e14e	move read write topic config into filer client accessor	2024-05-14 08:50:17 -07:00
chrislu	ac63f2b5a1	refactor	2024-05-13 22:03:57 -07:00
chrislu	844a1a0192	rename	2024-05-13 19:24:47 -07:00
chrislu	372bd8d71d	consumer instance passing MaxPartitionCount to coordinator	2024-05-13 18:30:10 -07:00
chrislu	2142842f82	refactor	2024-05-13 18:25:38 -07:00
chrislu	8cc5298a08	clean up unused variables	2024-05-10 13:10:45 -07:00
chrislu	fcedf97e94	fix test	2024-05-05 22:19:59 -07:00
chrislu	664b704e30	add delay	2024-05-05 14:17:41 -07:00
chrislu	4e3701df77	reset last error	2024-05-05 14:16:54 -07:00
chrislu	30f8600a24	working	2024-05-03 21:27:06 -07:00
chrislu	7a9b115cc2	rename	2024-05-02 11:14:58 -07:00
chrislu	4a3e8869bb	subscriber can unmarshal the record	2024-05-02 09:56:11 -07:00
chrislu	1c5043baf7	rename	2024-05-02 09:47:29 -07:00
chrislu	f2f4630029	rename field types	2024-05-02 08:59:22 -07:00
chrislu	b8af997e41	consistent with protobuf data types	2024-05-02 08:52:10 -07:00
chrislu	0f35b3a4ea	add example to publish a record	2024-05-02 08:48:51 -07:00
chrislu	928a4e8dff	rename	2024-05-02 08:35:06 -07:00
chrislu	5bc1b70a4c	rename to WithField()	2024-05-02 08:32:15 -07:00
chrislu	b570e015ce	Update write_parquet_test.go	2024-04-28 13:56:55 -07:00
chrislu	5dc186ef01	record value API	2024-04-28 13:42:45 -07:00
chrislu	b166597c83	fix test	2024-04-28 13:03:01 -07:00
chrislu	189a7fc90e	record type begin and end	2024-04-28 13:00:52 -07:00
chrislu	0da1b7b0f9	improve api	2024-04-28 12:42:58 -07:00
chrislu	1a2332c257	rename api	2024-04-28 12:40:00 -07:00
chrislu	5ce8f50546	Delete schema_builder_test.go	2024-04-28 12:39:27 -07:00
chrislu	8a0e5b6d54	struct to schema correctly	2024-04-28 12:23:54 -07:00
chrislu	f37862bffe	compile test	2024-04-28 12:23:33 -07:00
chrislu	7310d022ad	recursive struct to schema	2024-04-28 11:14:13 -07:00
chrislu	a2a872ca03	rename	2024-04-27 23:44:11 -07:00
chrislu	60103171b6	rename	2024-04-27 23:30:15 -07:00
chrislu	72b50980f4	a little bit more efficient	2024-04-25 23:59:30 -07:00
chrislu	63036133ff	minor	2024-04-25 22:02:03 -07:00
chrislu	2a9b0a9332	use parquet levels to construct row	2024-04-25 21:29:45 -07:00
chrislu	977e7988e6	toRow conversion with levels info	2024-04-25 09:14:37 -07:00
chrislu	9cb9d27b5b	file rename	2024-04-25 08:56:31 -07:00
chrislu	313097aca0	printout test values	2024-04-25 08:54:57 -07:00
chrislu	827e293010	add visitor	2024-04-25 08:54:41 -07:00
chrislu	138a027a2e	add field types but without value	2024-04-24 23:33:11 -07:00
chrislu	30b5aef62e	record type is also optional	2024-04-24 23:32:35 -07:00
chrislu	ce4597c623	delete the generated file	2024-04-24 23:29:55 -07:00
chrislu	f91484ca07	add some notes	2024-04-24 23:28:20 -07:00
chrislu	f93c4cf48c	adjust test	2024-04-24 23:24:53 -07:00
chrislu	d88c1872ac	works with single level of list	2024-04-24 23:04:47 -07:00
chrislu	d7e5f6b2a5	fix ordering	2024-04-24 01:32:02 -07:00
chrislu	9ef238b610	has a bug	2024-04-22 00:42:18 -07:00
chrislu	cc1ce3de87	all record fields are optional by default	2024-04-18 23:17:52 -07:00
chrislu	384ed7a381	add test	2024-04-18 22:41:20 -07:00
chrislu	57949f9959	support list type	2024-04-18 22:41:12 -07:00
chrislu	05d18130b6	add schema builder	2024-04-18 22:27:18 -07:00
chrislu	0847a17484	convert to parquet schema and value	2024-04-17 23:49:21 -07:00
chrislu	ce2b2fa9b2	Merge branch 'master' into mq-subscribe	2024-04-15 20:09:14 -07:00
yudrywet	8833745fbc	chore: fix some typos in comments (#5497)	2024-04-14 07:42:40 -07:00
chrislu	68ade92d4f	publisher send schema	2024-04-12 23:36:15 -07:00
chrislu	8415476c2d	fix test	2024-04-12 23:35:54 -07:00
chrislu	5324e4288a	rename parameter	2024-04-12 22:33:00 -07:00
chrislu	6f80dbdfe5	Create schema.go	2024-04-12 22:31:28 -07:00
chrislu	53d1d2b78a	save schema when configuring topic	2024-04-12 22:29:53 -07:00
chrislu	d45a372615	add schema.proto	2024-04-12 01:30:29 -07:00
chrislu	ccdd9cd8de	refactor	2024-04-02 16:25:43 -07:00
chrislu	f37c0d0d7a	comment out println	2024-04-02 15:31:31 -07:00
chrislu	d935f70e3c	shutdown follower when leader stops	2024-04-01 22:32:30 -07:00
chrislu	f07875e8e1	send flush message to follower before shutting down logBuffer	2024-04-01 16:01:26 -07:00
chrislu	e568e742c9	renaming	2024-04-01 15:02:52 -07:00
chrislu	47975e75bf	purge old code	2024-03-31 17:49:34 -07:00
chrislu	237e68059d	shutdown follower if no clients, not only publishers ... to avoid overhead when a publisher repeatedly send one message and close	2024-03-31 17:45:26 -07:00
chrislu	66a878af39	add control message to data message	2024-03-31 16:35:46 -07:00
chrislu	546ae87c39	send is_close message to broker	2024-03-31 01:28:40 -07:00
chrislu	c9df613b6b	add publisher name for debugging	2024-03-31 00:19:16 -07:00
chrislu	ca4f89a6f6	simplify	2024-03-30 20:37:57 -07:00
chrislu	0d55bddc75	logs	2024-03-30 20:37:46 -07:00
chrislu	e3c8be7f89	LocalPartition shutdown	2024-03-27 23:55:06 -07:00
chrislu	ab9fc0ad70	simplify	2024-03-27 23:42:19 -07:00
chrislu	7463058299	change visibility	2024-03-27 23:11:35 -07:00
chrislu	dab5454332	publish and send to follower	2024-03-27 23:10:24 -07:00
chrislu	f249f831b1	MaybeShutdownLocalPartition refactor	2024-03-27 23:02:51 -07:00
chrislu	2a0ab7e90d	fix	2024-03-27 22:59:50 -07:00
chrislu	059a120708	refactor	2024-03-27 22:58:20 -07:00
chrislu	7d87c1d2bb	refactor	2024-03-27 22:48:37 -07:00
chrislu	5cc94a05b9	separate goroutine to send ack to publisher	2024-03-27 10:27:08 -07:00
chrislu	17806cde2a	separate context ... If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.	2024-03-27 10:24:57 -07:00
chrislu	f67c8309d8	rename	2024-03-26 21:52:12 -07:00
chrislu	d6a066495b	adjust logs	2024-03-25 08:15:47 -07:00
chrislu	1b1b96ea1e	close local partition if no more publishers and subscribers	2024-03-24 21:10:02 -07:00
chrislu	4f5c4c3388	refactor	2024-03-24 13:07:51 -07:00
chrislu	d558a58946	refactor	2024-03-24 13:04:59 -07:00
chrislu	205d2285f3	setup follower by publisher ... * the subscriber would getOrGen a local partition and wait * the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.	2024-03-24 12:57:09 -07:00
chrislu	da31e9b939	correctly wait for the publisher to finish	2024-03-21 09:08:53 -07:00
chrislu	7003ce7425	publisher is able to fully send data and receive acks ... still needs to close the pipes cleanly	2024-03-20 12:25:40 -07:00
chrislu	516cba7053	remove FollowInMemoryMessages	2024-03-17 22:27:13 -07:00