Skip to content

Single Active Consumer for streams #3753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
acogoluegnes opened this issue Nov 17, 2021 · 0 comments
Closed

Single Active Consumer for streams #3753

acogoluegnes opened this issue Nov 17, 2021 · 0 comments
Assignees
@acogoluegnes
Labels
enhancement

Comments

@acogoluegnes
Copy link
Contributor

No description provided.

@acogoluegnes acogoluegnes self-assigned this Nov 17, 2021
acogoluegnes added a commit that referenced this issue Nov 17, 2021
@acogoluegnes
Handle single active consumer registration
8c19092 
WIP. Uses a simple in-memory coordinator for now.
No failover yet.

References #3753
@acogoluegnes acogoluegnes mentioned this issue Nov 17, 2021
Merged
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 17, 2021
@acogoluegnes
Support consumer update command
b11786a 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 18, 2021
@acogoluegnes
Add SAC failover test
e7af78c 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit that referenced this issue Nov 18, 2021
@acogoluegnes
Handle SAC (simple) failover
796268b 
When consumers unsubscribe normally.

References #3753
acogoluegnes added a commit that referenced this issue Nov 18, 2021
@acogoluegnes
Unregister stream SAC consumers on connection closing
a0a93da 
References #3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 18, 2021
@acogoluegnes
Add SAC connection closing test
9734045 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 18, 2021
@acogoluegnes
Use server-side offset tracking in SAC test
6ed5964 
Instead of client-side in-memory tracking, because it's more realistic.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 19, 2021
@acogoluegnes
Add single active consumer API to Consumer
23a96b7 
WIP.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 19, 2021
@acogoluegnes
Add SAC test with custom consumer update listener
a911afe 
To mimick custom offset tracking.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit that referenced this issue Nov 22, 2021
@acogoluegnes
Compute stream partition index
787db9b 
References #3753
acogoluegnes added a commit that referenced this issue Nov 23, 2021
@acogoluegnes
Notify stream consumer connection of SAC changes
88c6347 
References #3753
acogoluegnes added a commit that referenced this issue Nov 24, 2021
@acogoluegnes
Refactor SAC coordinator for SAC in partition context
fe41c15 
References #3753
acogoluegnes added a commit that referenced this issue Nov 24, 2021
@acogoluegnes
Remove unused function
3f453c4 
References #3753
acogoluegnes added a commit that referenced this issue Nov 24, 2021
@acogoluegnes
Implement basic SAC rollover on stream partition
e68f77e 
References #3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 24, 2021
@acogoluegnes
Add test for SAC on super stream partition
9605cd9 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit that referenced this issue Nov 25, 2021
@acogoluegnes
Fix stream SAC computation and notification
9758753 
Start index from the end because of lists:foldr.
Make sure to notify only the passive newcomer when there's
no change in SAC.

References #3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add SAC test for consumer balance on super stream
90c55c7 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Support consumer update command
8688fe9 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add SAC failover test
d12b400 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add SAC connection closing test
eada615 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Use server-side offset tracking in SAC test
ac13ef0 
Instead of client-side in-memory tracking, because it's more realistic.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add single active consumer API to Consumer
7649045 
WIP.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add SAC test with custom consumer update listener
3c0426d 
To mimick custom offset tracking.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add test for SAC on super stream partition
2117f64 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add SAC test for consumer balance on super stream
80c28cf 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Nov 25, 2021
@acogoluegnes
Add super stream SAC balancing test
b86f3f3 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit that referenced this issue Nov 26, 2021
@acogoluegnes
Fix stream SAC computation and notification
550541d 
Make sure to notify former active when unsubscription
triggers rebalancing.

References #3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Handle no offset response in default consumer update handler
eb36d6b 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Add SAC super stream test with manual offset tracking
f490e76 
Using server-side offset tracking.

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Add SAC super stream test with custom offset tracking
e5838ee 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Put SAC-related tests in their own classes
490d077 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Adapt SAC tests to SAC coordinator refactoring
effeab5 
The (de)activation protocol changed a bit internally. New consumers
assume they are passive by default, until they are activated
(no passive update notification if the consumer was not active
previously).

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Add test for SAC + connection killing
19f35e6 
The stream SAC coordinator monitors connection processes
and should send consumer update accordingly when connections
die.

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Add SAC test with different groups on same super stream
69fbf1a 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Add test to check SAC must have a name
c110d96 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Jul 13, 2022
@acogoluegnes
Increase publishing rate in test
4329ef6 
Otherwise the expected response code may not be received
because there's no enough concurrency. This way an error
path is exercised on the broker side.

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Support consumer update command
59e6a0c 
References rabbitmq/rabbitmq-server#3753

Conflicts:
	src/main/java/com/rabbitmq/stream/Constants.java
	src/main/java/com/rabbitmq/stream/impl/Client.java
	src/main/java/com/rabbitmq/stream/impl/ServerFrameHandler.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC failover test
909ceac 
References rabbitmq/rabbitmq-server#3753

Conflicts:
	src/test/java/com/rabbitmq/stream/impl/SingleActiveConsumerTest.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC connection closing test
ba54cf3 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Use server-side offset tracking in SAC test
00c0112 
Instead of client-side in-memory tracking, because it's more realistic.

References rabbitmq/rabbitmq-server#3753

Conflicts:
	src/test/java/com/rabbitmq/stream/impl/SingleActiveConsumerTest.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add single active consumer API to Consumer
c63e872 
WIP.

References rabbitmq/rabbitmq-server#3753

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/OffsetTrackingCoordinator.java
	src/main/java/com/rabbitmq/stream/impl/StreamConsumer.java

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/StreamConsumer.java

Conflicts:
	src/test/java/com/rabbitmq/stream/impl/SingleActiveConsumerTest.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC test with custom consumer update listener
8249b65 
To mimick custom offset tracking.

References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add test for SAC on super stream partition
7301ca9 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC test for consumer balance on super stream
0ffa0de 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add super stream SAC balancing test
8543515 
References rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Support clean SAC change in super stream consumer
09f3918 
References rabbitmq/rabbitmq-server#3753

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/OffsetTrackingCoordinator.java
	src/main/java/com/rabbitmq/stream/impl/StreamConsumer.java
	src/main/java/com/rabbitmq/stream/impl/Utils.java

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/ConsumersCoordinator.java
	src/main/java/com/rabbitmq/stream/impl/StreamConsumer.java

Conflicts:
	src/main/java/com/rabbitmq/stream/MessageHandler.java
	src/main/java/com/rabbitmq/stream/impl/ConsumersCoordinator.java
	src/main/java/com/rabbitmq/stream/impl/Utils.java
	src/test/java/com/rabbitmq/stream/impl/OffsetTrackingCoordinatorTest.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC branch to changes from main
a6ba730 
Mainly queryOffset returning QueryOffsetResponse instead of
just a long.

References #46, rabbitmq/rabbitmq-server#3753

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/OffsetTrackingCoordinator.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC test with manual offset tracking
003eea8 
Requires to add Consumer#storedOffset() to let the user
make sure the stored offset request made it to the
broker before closing the consumer.

References #46, rabbitmq/rabbitmq-server#3753

Conflicts:
	src/main/java/com/rabbitmq/stream/impl/OffsetTrackingCoordinator.java
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Handle no offset response in default consumer update handler
78999b4 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC super stream test with manual offset tracking
30bc15d 
Using server-side offset tracking.

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC super stream test with custom offset tracking
d07b3d3 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Put SAC-related tests in their own classes
7a487d8 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Adapt SAC tests to SAC coordinator refactoring
e553017 
The (de)activation protocol changed a bit internally. New consumers
assume they are passive by default, until they are activated
(no passive update notification if the consumer was not active
previously).

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add test for SAC + connection killing
3b61ad7 
The stream SAC coordinator monitors connection processes
and should send consumer update accordingly when connections
die.

References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add SAC test with different groups on same super stream
a385665 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Add test to check SAC must have a name
befd3b9 
References #46, rabbitmq/rabbitmq-server#3753
acogoluegnes added a commit to rabbitmq/rabbitmq-stream-java-client that referenced this issue Aug 10, 2022
@acogoluegnes
Increase publishing rate in test
2210495 
Otherwise the expected response code may not be received
because there's no enough concurrency. This way an error
path is exercised on the broker side.

References #46, rabbitmq/rabbitmq-server#3753
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@acogoluegnes acogoluegnes

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@acogoluegnes