chrislu
|
28f8f33d6a
|
include key in LogEntry
|
2024-03-07 10:53:30 -08:00 |
|
chrislu
|
34f2b600ac
|
each log function adds a "done" return parameter
|
2024-03-07 10:50:09 -08:00 |
|
chrislu
|
62397f2371
|
adjust error
|
2024-03-07 10:47:50 -08:00 |
|
chrislu
|
7598922b41
|
assign followers
|
2024-02-29 14:51:06 -08:00 |
|
chrislu
|
1b4484bf0a
|
go fmt
|
2024-02-29 09:38:52 -08:00 |
|
Chris Lu
|
16b4f9d9ab
|
subscriber find broker leader first
|
2024-02-05 23:14:25 -08:00 |
|
Chris Lu
|
656b78d1af
|
proxy to broker leader
|
2024-02-05 18:00:50 -08:00 |
|
Chris Lu
|
4dc6681833
|
proxy requests to lock owner
|
2024-02-05 16:46:40 -08:00 |
|
Chris Lu
|
f9131de43f
|
fix lock TTL
|
2024-02-04 12:47:21 -08:00 |
|
Chris Lu
|
0a12301b3d
|
avoid too large expiration time
|
2024-02-04 09:20:21 -08:00 |
|
chrislu
|
d41792461c
|
lock returns host and owner
|
2024-02-02 15:54:57 -08:00 |
|
chrislu
|
d30150dde1
|
use a short-lived lock
|
2024-02-01 23:01:44 -08:00 |
|
chrislu
|
ac50d8a822
|
re-lock if the lock owner is not found
|
2024-01-29 22:46:23 -08:00 |
|
chrislu
|
0b2e5ddc7c
|
wait 3 seconds before shutting down publish client, to wait for all messages to be received
|
2024-01-28 22:04:42 -08:00 |
|
chrislu
|
a507069d19
|
lock for creating local partition
|
2024-01-28 15:55:26 -08:00 |
|
chrislu
|
9e6ea80dff
|
read existing topic conf
|
2024-01-28 14:27:52 -08:00 |
|
chrislu
|
d40225ecd2
|
return the lookup results
|
2024-01-28 12:05:44 -08:00 |
|
chrislu
|
841fafd0a8
|
publish to input buffer
currently, the input buffer may not exist when start to publish
|
2024-01-27 23:43:22 -08:00 |
|
chrislu
|
d268fbe18a
|
when configure, cancel existing assignments
|
2024-01-22 11:57:17 -08:00 |
|
chrislu
|
3ff6b31d94
|
sub broker wait for partition
|
2024-01-22 11:56:55 -08:00 |
|
chrislu
|
2beaa2d0b3
|
pub/sub brokers check filer for assigned partitions
|
2024-01-22 11:44:21 -08:00 |
|
chrislu
|
c3f8530f97
|
refactor
|
2024-01-22 11:06:53 -08:00 |
|
chrislu
|
5f35857cd0
|
no need to assign partitions explicitly
|
2024-01-22 11:04:47 -08:00 |
|
chrislu
|
428fea45f3
|
updated and added assignments
|
2024-01-22 11:04:24 -08:00 |
|
chrislu
|
e8b05ecc91
|
add/remove assigned partitions
|
2024-01-22 10:47:39 -08:00 |
|
chrislu
|
861ad732ca
|
refactor
|
2024-01-22 09:00:22 -08:00 |
|
chrislu
|
47924afa1c
|
refactor
|
2024-01-22 08:52:44 -08:00 |
|
chrislu
|
581cf6644c
|
lookup reads from filer instead of local memory
|
2024-01-22 00:52:55 -08:00 |
|
chrislu
|
7121ae9617
|
separate out read topic conf and ensure topic active assignments
|
2024-01-22 00:51:31 -08:00 |
|
chrislu
|
c77d35313e
|
pub/sub broker only check local assigned partitions
|
2024-01-22 00:49:57 -08:00 |
|
chrislu
|
b0a2e9aea3
|
fix assignments if brokers changed
|
2024-01-21 01:27:22 -08:00 |
|
chrislu
|
a5b906c0dd
|
refactor
|
2024-01-21 01:23:43 -08:00 |
|
chrislu
|
34839237ab
|
refactor
|
2024-01-20 12:16:40 -08:00 |
|
chrislu
|
2828ccbb30
|
refactor
|
2024-01-20 11:41:11 -08:00 |
|
chrislu
|
541140f735
|
rename
|
2024-01-19 11:43:40 -08:00 |
|
chrislu
|
c8090b1f99
|
send assignments to newly connected subscribers
|
2024-01-17 22:44:39 -08:00 |
|
chrislu
|
c616081ab9
|
refactor
|
2024-01-17 22:29:59 -08:00 |
|
chrislu
|
bc8d2a01cc
|
create local topic partition from config on filer
|
2024-01-17 00:15:52 -08:00 |
|
chrislu
|
be0c426dc7
|
simplify to LookupTopicPartitions(topic)
|
2024-01-16 09:30:46 -08:00 |
|
chrislu
|
db3670a3a5
|
simplify api
|
2024-01-16 08:55:47 -08:00 |
|
chrislu
|
34a78ffad0
|
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
|
2024-01-16 08:52:42 -08:00 |
|
chrislu
|
3795d8dca8
|
release local topic partition if no publisher and subscribers
|
2024-01-16 08:43:07 -08:00 |
|
chrislu
|
ba73199174
|
reuse local partition
|
2024-01-15 21:22:41 -08:00 |
|
chrislu
|
fa59a5d67e
|
read from disk if not in memory
|
2024-01-15 00:20:12 -08:00 |
|
chrislu
|
7afaad31a4
|
wait on local topic partition creation
|
2024-01-11 23:08:02 -08:00 |
|
chrislu
|
61dbdd0ff6
|
log ctrl messages
|
2024-01-11 23:07:32 -08:00 |
|
chrislu
|
6a7a679137
|
passing partition generation timestamp
|
2024-01-10 22:36:17 -08:00 |
|
chrislu
|
8af61dec91
|
avoid empty spaces in file name
|
2024-01-09 09:00:54 -08:00 |
|
chrislu
|
aed54eda61
|
refactor
|
2024-01-09 08:01:01 -08:00 |
|
chrislu
|
d51efddf5c
|
flush to disk
Need to do: read from disk
|
2024-01-08 23:27:02 -08:00 |
|
chrislu
|
49428a303b
|
add batch index for each memory buffer
|
2024-01-08 00:03:08 -08:00 |
|
chrislu
|
e8611ed85d
|
subscribe with partition offset
|
2024-01-05 15:24:14 -08:00 |
|
chrislu
|
ee41dbb7fc
|
rename functions
|
2024-01-05 15:14:25 -08:00 |
|
chrislu
|
47a4963d7a
|
subscription start from specified timestamp
|
2024-01-03 15:57:36 -08:00 |
|
chrislu
|
35869b5c80
|
subscriber can be notified of the assignment change when topic is just configured
Next: Subscriber needs to read by the timestamp offset.
|
2024-01-03 13:30:30 -08:00 |
|
chrislu
|
efb695fd93
|
lookup existing topic partitions
|
2024-01-02 17:29:35 -08:00 |
|
chrislu
|
093fdc1621
|
subscriber can get assignments
|
2023-12-28 20:35:15 -08:00 |
|
chrislu
|
c950a40aad
|
subscriber keep connected to the balancer
|
2023-12-28 11:56:37 -08:00 |
|
chrislu
|
e0727071c8
|
go fmt
|
2023-12-22 11:33:50 -08:00 |
|
chrislu
|
034db049a0
|
comment fix
|
2023-12-22 11:33:00 -08:00 |
|
Chris Lu
|
580940bf82
|
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
|
2023-12-11 12:05:54 -08:00 |
|
chrislu
|
17710e1ecb
|
adjust logs
|
2023-11-15 18:11:35 -08:00 |
|
chrislu
|
de0b969b36
|
Revert "rename"
This reverts commit 35b5264ab7 .
|
2023-10-12 20:28:11 -07:00 |
|
chrislu
|
35b5264ab7
|
rename
|
2023-10-11 21:44:56 -07:00 |
|
chrislu
|
734178093e
|
refactor TopicPartition struct
|
2023-10-02 01:01:45 -07:00 |
|
chrislu
|
ebd6f96d35
|
adjust mq.proto
|
2023-09-30 21:47:57 -07:00 |
|
chrislu
|
02cb799481
|
add a test framework for grpc
|
2023-09-30 21:36:58 -07:00 |
|
chrislu
|
c4e6545197
|
fix compilation
|
2023-09-30 21:26:09 -07:00 |
|
chrislu
|
9d589b48e6
|
rename function
|
2023-09-26 15:17:33 -07:00 |
|
chrislu
|
23f334dd13
|
comment
|
2023-09-24 23:36:43 -07:00 |
|
chrislu
|
c7e05e4e71
|
ensure latest stats are reported
|
2023-09-24 23:34:31 -07:00 |
|
chrislu
|
dff2ce5d2f
|
read existing broker stats
|
2023-09-24 23:10:33 -07:00 |
|
chrislu
|
1492bf7552
|
fix listing topics
|
2023-09-24 23:05:41 -07:00 |
|
chrislu
|
3cf9b8d621
|
create topic and report topic
|
2023-09-24 22:00:43 -07:00 |
|
chrislu
|
b3f94feede
|
do create topic
|
2023-09-24 22:00:43 -07:00 |
|
chrislu
|
d74348048a
|
implement create topic
|
2023-09-24 22:00:43 -07:00 |
|
chrislu
|
0361c321b4
|
add CreateTopic API
|
2023-09-24 22:00:43 -07:00 |
|
chrislu
|
b18112ae10
|
fix compilation bugs during merge
|
2023-09-19 14:08:17 -07:00 |
|
chrislu
|
6c6655de9d
|
wip
|
2023-09-19 14:03:42 -07:00 |
|
chrislu
|
27af11f1e8
|
Revert "Revert "Merge branch 'master' into sub""
This reverts commit 0bb97709d4 .
|
2023-09-18 18:47:34 -07:00 |
|
chrislu
|
0bb97709d4
|
Revert "Merge branch 'master' into sub"
This reverts commit 4d414f54a2 , reversing
changes made to 4827425146 .
|
2023-09-18 16:13:20 -07:00 |
|
chrislu
|
4d414f54a2
|
Merge branch 'master' into sub
|
2023-09-18 16:04:59 -07:00 |
|
chrislu
|
a65088d739
|
broker report stats to balancer
|
2023-09-16 23:00:51 -07:00 |
|
chrislu
|
89a1fd1751
|
Squashed commit of the following:
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date: Sat Sep 16 15:05:38 2023 -0700
balancer works
commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:22:32 2023 -0700
comments
commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:20:05 2023 -0700
adjust APIs
commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 14 23:49:05 2023 -0700
receive broker stats
commit b771fefa37
Merge: 0a851ec00 890881037
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 13 00:03:47 2023 -0700
Merge branch 'master' into sub
commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date: Sun Sep 10 22:01:25 2023 -0700
Create balancer.go
commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 7 23:55:19 2023 -0700
add publisher shutdown
commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:16:41 2023 -0700
server side send response at least once per second
commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:15:29 2023 -0700
ack interval 128
commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 22:39:46 2023 -0700
ack interval
commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:50 2023 -0700
api for sub
commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:30 2023 -0700
publish, benchmark
commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 1 00:36:51 2023 -0700
can pub and sub
commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:02:12 2023 -0700
connect and publish
commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:01:25 2023 -0700
protoc version
commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:59:04 2023 -0700
rename to lookup
commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:33:46 2023 -0700
move functions
commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 17:50:59 2023 -0700
pub sub initial tests
commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:14:39 2023 -0700
rename
commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:13:14 2023 -0700
sub
commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date: Sat Aug 26 13:39:21 2023 -0700
adjust proto
|
2023-09-16 15:06:16 -07:00 |
|
chrislu
|
4827425146
|
balancer works
|
2023-09-16 15:05:38 -07:00 |
|
chrislu
|
436d99443b
|
receive broker stats
|
2023-09-14 23:49:05 -07:00 |
|
chrislu
|
b771fefa37
|
Merge branch 'master' into sub
|
2023-09-13 00:03:47 -07:00 |
|
chrislu
|
39941edc0b
|
add publisher shutdown
|
2023-09-07 23:55:19 -07:00 |
|
chrislu
|
875f562779
|
server side send response at least once per second
|
2023-09-06 23:16:41 -07:00 |
|
chrislu
|
2492a45499
|
ack interval
|
2023-09-06 22:39:46 -07:00 |
|
chrislu
|
ba67e6ca29
|
api for sub
|
2023-09-04 21:43:50 -07:00 |
|
chrislu
|
9e4f985698
|
publish, benchmark
|
2023-09-04 21:43:30 -07:00 |
|
chrislu
|
cb470d44df
|
can pub and sub
|
2023-09-01 00:36:51 -07:00 |
|
chrislu
|
1eb2da46d5
|
connect and publish
|
2023-08-28 09:02:12 -07:00 |
|
chrislu
|
dbcba75271
|
rename to lookup
|
2023-08-27 18:59:04 -07:00 |
|
chrislu
|
c9caf33119
|
move functions
|
2023-08-27 18:33:46 -07:00 |
|
chrislu
|
4d6c18d86f
|
pub sub initial tests
|
2023-08-27 17:50:59 -07:00 |
|
chrislu
|
4eb8e8624d
|
rename
|
2023-08-27 13:14:39 -07:00 |
|
chrislu
|
1990456670
|
sub
|
2023-08-27 13:13:14 -07:00 |
|
chrislu
|
905911853d
|
adjust proto
|
2023-08-26 13:39:21 -07:00 |
|