273 Commits

Author	SHA1	Message	Date
chrislu	8415476c2d	fix test	2024-04-12 23:35:54 -07:00
chrislu	5324e4288a	rename parameter	2024-04-12 22:33:00 -07:00
chrislu	6f80dbdfe5	Create schema.go	2024-04-12 22:31:28 -07:00
chrislu	53d1d2b78a	save schema when configuring topic	2024-04-12 22:29:53 -07:00
chrislu	d45a372615	add schema.proto	2024-04-12 01:30:29 -07:00
chrislu	ccdd9cd8de	refactor	2024-04-02 16:25:43 -07:00
chrislu	f37c0d0d7a	comment out println	2024-04-02 15:31:31 -07:00
chrislu	d935f70e3c	shutdown follower when leader stops	2024-04-01 22:32:30 -07:00
chrislu	f07875e8e1	send flush message to follower before shutting down logBuffer	2024-04-01 16:01:26 -07:00
chrislu	e568e742c9	renaming	2024-04-01 15:02:52 -07:00
chrislu	47975e75bf	purge old code	2024-03-31 17:49:34 -07:00
chrislu	237e68059d	shutdown follower if no clients, not only publishers ... to avoid overhead when a publisher repeatedly send one message and close	2024-03-31 17:45:26 -07:00
chrislu	66a878af39	add control message to data message	2024-03-31 16:35:46 -07:00
chrislu	546ae87c39	send is_close message to broker	2024-03-31 01:28:40 -07:00
chrislu	c9df613b6b	add publisher name for debugging	2024-03-31 00:19:16 -07:00
chrislu	ca4f89a6f6	simplify	2024-03-30 20:37:57 -07:00
chrislu	0d55bddc75	logs	2024-03-30 20:37:46 -07:00
chrislu	e3c8be7f89	LocalPartition shutdown	2024-03-27 23:55:06 -07:00
chrislu	ab9fc0ad70	simplify	2024-03-27 23:42:19 -07:00
chrislu	7463058299	change visibility	2024-03-27 23:11:35 -07:00
chrislu	dab5454332	publish and send to follower	2024-03-27 23:10:24 -07:00
chrislu	f249f831b1	MaybeShutdownLocalPartition refactor	2024-03-27 23:02:51 -07:00
chrislu	2a0ab7e90d	fix	2024-03-27 22:59:50 -07:00
chrislu	059a120708	refactor	2024-03-27 22:58:20 -07:00
chrislu	7d87c1d2bb	refactor	2024-03-27 22:48:37 -07:00
chrislu	5cc94a05b9	separate goroutine to send ack to publisher	2024-03-27 10:27:08 -07:00
chrislu	17806cde2a	separate context ... If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.	2024-03-27 10:24:57 -07:00
chrislu	f67c8309d8	rename	2024-03-26 21:52:12 -07:00
chrislu	d6a066495b	adjust logs	2024-03-25 08:15:47 -07:00
chrislu	1b1b96ea1e	close local partition if no more publishers and subscribers	2024-03-24 21:10:02 -07:00
chrislu	4f5c4c3388	refactor	2024-03-24 13:07:51 -07:00
chrislu	d558a58946	refactor	2024-03-24 13:04:59 -07:00
chrislu	205d2285f3	setup follower by publisher ... * the subscriber would getOrGen a local partition and wait * the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.	2024-03-24 12:57:09 -07:00
chrislu	da31e9b939	correctly wait for the publisher to finish	2024-03-21 09:08:53 -07:00
chrislu	7003ce7425	publisher is able to fully send data and receive acks ... still needs to close the pipes cleanly	2024-03-20 12:25:40 -07:00
chrislu	516cba7053	remove FollowInMemoryMessages	2024-03-17 22:27:13 -07:00
chrislu	6ef3e6bcad	logging	2024-03-17 22:24:20 -07:00
chrislu	a375b2815e	Merge branch 'master' into mq-subscribe	2024-03-17 11:46:55 -07:00
chrislu	a8b73296d7	fix test3	2024-03-16 23:53:46 -07:00
chrislu	ea2601799c	fix test	2024-03-16 23:22:17 -07:00
chrislu	8e5068fd2f	notify	2024-03-16 23:16:33 -07:00
chrislu	44ab42d03c	unnecessary	2024-03-16 17:11:42 -07:00
chrislu	3f2dd8cc3e	add follower id info	2024-03-16 17:11:18 -07:00
chrislu	529fa8c4c8	go fmt	2024-03-16 10:51:47 -07:00
chrislu	1b5c547cfe	logs	2024-03-15 21:08:57 -07:00
chrislu	55714f54ab	local partition is generated or not	2024-03-15 21:08:42 -07:00
chrislu	549455c30d	streaming mode	2024-03-15 21:05:24 -07:00
chrislu	f653838f34	follower receive a hello message before actual data ... the leader can ensure the follower has connected	2024-03-15 20:36:01 -07:00
chrislu	64a32730a2	add size()	2024-03-15 20:34:39 -07:00
chrislu	6f75a0af55	ask follower to follow	2024-03-10 14:34:28 -07:00
chrislu	e7823ee967	retry connecting to broker leader	2024-03-09 12:56:49 -08:00
chrislu	3b28433cb1	change LogBuffer visibility in LocalPartition	2024-03-07 10:57:04 -08:00
chrislu	28f8f33d6a	include key in LogEntry	2024-03-07 10:53:30 -08:00
chrislu	34f2b600ac	each log function adds a "done" return parameter	2024-03-07 10:50:09 -08:00
chrislu	62397f2371	adjust error	2024-03-07 10:47:50 -08:00
chrislu	49869eec83	edge cases for broker assignment	2024-03-07 10:45:38 -08:00
chrislu	33ab6cfcf7	adjust error	2024-03-07 10:44:09 -08:00
chrislu	fe03b1b522	handle single broker case	2024-03-01 05:59:40 -08:00
chrislu	50c5dd7313	test low active brokers	2024-02-29 20:50:57 -08:00
chrislu	7598922b41	assign followers	2024-02-29 14:51:06 -08:00
chrislu	ead27a4f59	publish init add followers	2024-02-29 09:39:12 -08:00
chrislu	1b4484bf0a	go fmt	2024-02-29 09:38:52 -08:00
Chris Lu	16b4f9d9ab	subscriber find broker leader first	2024-02-05 23:14:25 -08:00
Chris Lu	656b78d1af	proxy to broker leader	2024-02-05 18:00:50 -08:00
Chris Lu	90fcde0e26	report EOF error	2024-02-05 17:47:34 -08:00
Chris Lu	07028a4864	minor	2024-02-05 17:30:20 -08:00
Chris Lu	4dc6681833	proxy requests to lock owner	2024-02-05 16:46:40 -08:00
Chris Lu	0d74ac9224	fix divide by zero	2024-02-04 13:38:00 -08:00
Chris Lu	f9131de43f	fix lock TTL	2024-02-04 12:47:21 -08:00
Chris Lu	0a12301b3d	avoid too large expiration time	2024-02-04 09:20:21 -08:00
chrislu	d41792461c	lock returns host and owner	2024-02-02 15:54:57 -08:00
chrislu	d30150dde1	use a short-lived lock	2024-02-01 23:01:44 -08:00
chrislu	ac50d8a822	re-lock if the lock owner is not found	2024-01-29 22:46:23 -08:00
chrislu	0b2e5ddc7c	wait 3 seconds before shutting down publish client, to wait for all messages to be received	2024-01-28 22:04:42 -08:00
chrislu	545d5fbdf6	unused code	2024-01-28 22:03:13 -08:00
chrislu	a507069d19	lock for creating local partition	2024-01-28 15:55:26 -08:00
chrislu	9e6ea80dff	read existing topic conf	2024-01-28 14:27:52 -08:00
chrislu	a297247137	wait for each publish clients	2024-01-28 14:27:32 -08:00
chrislu	b6c5e57c30	read will block if no items	2024-01-28 13:10:34 -08:00
chrislu	0bf5424a2e	adjust visibility	2024-01-28 13:09:30 -08:00
chrislu	dedfd31dfb	refactor	2024-01-28 12:30:08 -08:00
chrislu	cbf750a31f	refactor	2024-01-28 12:23:20 -08:00
chrislu	b51dfe2bff	wait for publishing clients	2024-01-28 12:06:58 -08:00
chrislu	f8c55f101e	clean up	2024-01-28 12:06:30 -08:00
chrislu	d40225ecd2	return the lookup results	2024-01-28 12:05:44 -08:00
chrislu	841fafd0a8	publish to input buffer ... currently, the input buffer may not exist when start to publish	2024-01-27 23:43:22 -08:00
chrislu	91af1f3069	schedule jobs	2024-01-26 14:09:57 -08:00
chrislu	d268fbe18a	when configure, cancel existing assignments	2024-01-22 11:57:17 -08:00
chrislu	3ff6b31d94	sub broker wait for partition	2024-01-22 11:56:55 -08:00
chrislu	2beaa2d0b3	pub/sub brokers check filer for assigned partitions	2024-01-22 11:44:21 -08:00
chrislu	c3f8530f97	refactor	2024-01-22 11:06:53 -08:00
chrislu	5f35857cd0	no need to assign partitions explicitly	2024-01-22 11:04:47 -08:00
chrislu	428fea45f3	updated and added assignments	2024-01-22 11:04:24 -08:00
chrislu	e8b05ecc91	add/remove assigned partitions	2024-01-22 10:47:39 -08:00
chrislu	861ad732ca	refactor	2024-01-22 09:00:22 -08:00
chrislu	47924afa1c	refactor	2024-01-22 08:52:44 -08:00
chrislu	581cf6644c	lookup reads from filer instead of local memory	2024-01-22 00:52:55 -08:00
chrislu	7121ae9617	separate out read topic conf and ensure topic active assignments	2024-01-22 00:51:31 -08:00
chrislu	c77d35313e	pub/sub broker only check local assigned partitions	2024-01-22 00:49:57 -08:00
chrislu	b0a2e9aea3	fix assignments if brokers changed	2024-01-21 01:27:22 -08:00