116 Commits

Author	SHA1	Message	Date
Chris Lu	16b4f9d9ab	subscriber find broker leader first	2024-02-05 23:14:25 -08:00
Chris Lu	656b78d1af	proxy to broker leader	2024-02-05 18:00:50 -08:00
Chris Lu	4dc6681833	proxy requests to lock owner	2024-02-05 16:46:40 -08:00
Chris Lu	f9131de43f	fix lock TTL	2024-02-04 12:47:21 -08:00
Chris Lu	0a12301b3d	avoid too large expiration time	2024-02-04 09:20:21 -08:00
chrislu	d41792461c	lock returns host and owner	2024-02-02 15:54:57 -08:00
chrislu	d30150dde1	use a short-lived lock	2024-02-01 23:01:44 -08:00
chrislu	ac50d8a822	re-lock if the lock owner is not found	2024-01-29 22:46:23 -08:00
chrislu	0b2e5ddc7c	wait 3 seconds before shutting down publish client, to wait for all messages to be received	2024-01-28 22:04:42 -08:00
chrislu	a507069d19	lock for creating local partition	2024-01-28 15:55:26 -08:00
chrislu	9e6ea80dff	read existing topic conf	2024-01-28 14:27:52 -08:00
chrislu	d40225ecd2	return the lookup results	2024-01-28 12:05:44 -08:00
chrislu	841fafd0a8	publish to input buffer ... currently, the input buffer may not exist when start to publish	2024-01-27 23:43:22 -08:00
chrislu	d268fbe18a	when configure, cancel existing assignments	2024-01-22 11:57:17 -08:00
chrislu	3ff6b31d94	sub broker wait for partition	2024-01-22 11:56:55 -08:00
chrislu	2beaa2d0b3	pub/sub brokers check filer for assigned partitions	2024-01-22 11:44:21 -08:00
chrislu	c3f8530f97	refactor	2024-01-22 11:06:53 -08:00
chrislu	5f35857cd0	no need to assign partitions explicitly	2024-01-22 11:04:47 -08:00
chrislu	428fea45f3	updated and added assignments	2024-01-22 11:04:24 -08:00
chrislu	e8b05ecc91	add/remove assigned partitions	2024-01-22 10:47:39 -08:00
chrislu	861ad732ca	refactor	2024-01-22 09:00:22 -08:00
chrislu	47924afa1c	refactor	2024-01-22 08:52:44 -08:00
chrislu	581cf6644c	lookup reads from filer instead of local memory	2024-01-22 00:52:55 -08:00
chrislu	7121ae9617	separate out read topic conf and ensure topic active assignments	2024-01-22 00:51:31 -08:00
chrislu	c77d35313e	pub/sub broker only check local assigned partitions	2024-01-22 00:49:57 -08:00
chrislu	b0a2e9aea3	fix assignments if brokers changed	2024-01-21 01:27:22 -08:00
chrislu	a5b906c0dd	refactor	2024-01-21 01:23:43 -08:00
chrislu	34839237ab	refactor	2024-01-20 12:16:40 -08:00
chrislu	2828ccbb30	refactor	2024-01-20 11:41:11 -08:00
chrislu	541140f735	rename	2024-01-19 11:43:40 -08:00
chrislu	c8090b1f99	send assignments to newly connected subscribers	2024-01-17 22:44:39 -08:00
chrislu	c616081ab9	refactor	2024-01-17 22:29:59 -08:00
chrislu	bc8d2a01cc	create local topic partition from config on filer	2024-01-17 00:15:52 -08:00
chrislu	be0c426dc7	simplify to LookupTopicPartitions(topic)	2024-01-16 09:30:46 -08:00
chrislu	db3670a3a5	simplify api	2024-01-16 08:55:47 -08:00
chrislu	34a78ffad0	remove isForPublish from LookupTopicBrokers ... also adds a return parameter: whether the topic exists or not	2024-01-16 08:52:42 -08:00
chrislu	3795d8dca8	release local topic partition if no publisher and subscribers	2024-01-16 08:43:07 -08:00
chrislu	ba73199174	reuse local partition	2024-01-15 21:22:41 -08:00
chrislu	fa59a5d67e	read from disk if not in memory	2024-01-15 00:20:12 -08:00
chrislu	7afaad31a4	wait on local topic partition creation	2024-01-11 23:08:02 -08:00
chrislu	61dbdd0ff6	log ctrl messages	2024-01-11 23:07:32 -08:00
chrislu	6a7a679137	passing partition generation timestamp	2024-01-10 22:36:17 -08:00
chrislu	8af61dec91	avoid empty spaces in file name	2024-01-09 09:00:54 -08:00
chrislu	aed54eda61	refactor	2024-01-09 08:01:01 -08:00
chrislu	d51efddf5c	flush to disk ... Need to do: read from disk	2024-01-08 23:27:02 -08:00
chrislu	49428a303b	add batch index for each memory buffer	2024-01-08 00:03:08 -08:00
chrislu	e8611ed85d	subscribe with partition offset	2024-01-05 15:24:14 -08:00
chrislu	ee41dbb7fc	rename functions	2024-01-05 15:14:25 -08:00
chrislu	47a4963d7a	subscription start from specified timestamp	2024-01-03 15:57:36 -08:00
chrislu	35869b5c80	subscriber can be notified of the assignment change when topic is just configured ... Next: Subscriber needs to read by the timestamp offset.	2024-01-03 13:30:30 -08:00