Replies: 7 comments 19 replies
-
|
You haven't provided a lot of details on what exactly you do in your test. Assuming that you restart a node hosting a stream leader replica, there are several factors at play:
Overall, it can take 20 to 60s with all defaults. For a lot of environments, that would hardly be "very long". With heartbeat timeouts lowered to 5s I can see how it can get to the 10-15s range. Even if some tuning would give you a sub-5s recovery time, there is a very real cost to this: the false positives when networks or peers slow down. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
In case of Kubernetes or similar container orchestration systems there is one more factor at play: if you use a This entirely depends on the type and configuration of the K8S service. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
The default is 10s. Lowering this to 5s could help. I don't know if Stream PerfTest exposes this setting (likely not), and would not recommend very low values for the aforementioned real (not hypothetical) risk of false positives around timeouts. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
I've reproduced a similar setup to yours, but running locally inside a VM, namely Rancher Desktop with Kubernetes. In this setup, I get recovery times of 7-10 seconds when I forcibly stop the stream leader. I'm running with the default net tick time and heart beats. My setup is a lot more optimistic than yours, since I have every container running inside a single VM. If you have multiple VMs and your RabbitMQ Pods distributed accross different Kubernetes Nodes (a.k.a. VMs), your recovery time can increase to what you are describing, due to the added network round trip latency. I also want to echo Michael's words regarding recovery times and false positives. Recovery times sub-5 seconds sound a bit too optimistic to me.
This is probably related to the Channels are a concept from AMQP. RabbitMQ Streams do not use channels. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @Zerpet are there any other tuning parameters which can be set either in PerfTest client or in the Server config, in order to achieve better (shorter) downtimes, or the above-mentioned parameters are all we can set? |
Beta Was this translation helpful? Give feedback.
-
|
@albionb96 you are asking us to guess as to what may be taking the time in your test. We do not guess in this community. Use logs from nodes, Kubernetes, and Stream PerfTest to at least initially narrow down the problem. In any case, the approach of asking for settings to tweak without collecting data and narrowing down the biggest contributor is a really expensive way to troubleshoot distributed systems. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin the reason why I asked is because in your first comment you said that the time that my failovertest "is taking" is something expected by default. For me that statement means that there is no problem on my test-setup for which I need to look into logs or similar, because that behavior is normal. That's why I asked once again which parameters does the PerfTest or RabbitmqCluster offer to reduce these downtimes, because everything that I could find in the documentation, are the above-mentioned parameters. I see, there are no other parameters PerfTest offers to reduce the heartbeat or similar. That feature may be introduced in the future. Thank you |
Beta Was this translation helpful? Give feedback.
-
|
@Zerpet I wanted to ask you how did you forcibly stop the stream leader, because killing the pod forcefully with: |
Beta Was this translation helpful? Give feedback.
-
|
Hi again, during the last days I was focused on testing the RabbitMQ cluster a little bit more, and also was trying to gather logs from different places. I created 3 extra VMs (which are in the same time Kubernetes Nodes) and I placed in each of them one of the RabbitMQ cluster pods. In order to be sure that the failover delay is not coming because of the load-balancer, I started the PerfTool as a pod inside the Kubernetes cluster like this: PerfTool as pod
What happened: Note: Errors happen sometimes after the second iteration of the failovertest. That means:
PerfTool error messagesIt states that: java.net.UnknownHostException: message-broker-server-1.message-broker-nodes.message-broker: Name or service not known, but actually that hostname is valid. I run nslookup on the same pod and I could resolve this hostname, this hostname is automatically generated from the RMQ cluster operator. The "message-broker-nodes" in this case is the headless service created from the RMQ cluster operator. I don't know if this is a bug, or not. If I use the --load-balancer flag during the PerfTest then I don't get these errors, but I don't know if it makes sense. I would expect that the Cluster Service which serves the cluster and comes from the RMQ cluster operator, is not to be seen as an load-balancer, by default this service is a ClusterIP service, and its' endpoints are to be accessed from everywhere in the cluster with the help of the headless service which also comes automatically from the RMQ Cluster Operator. If you are interested to pursue the issue of the errors coming from the PerfTool, I can provide you with some more information below: In order to better understand what is happening in the background I installed the hubble from cilium in order to catch the traffic and here are the results: Traffic between PerfTool (Client) and RMQClusterAt the beginning: After the failure: And here also the logs from the rabbitmq cluster nodes which lived during the failover: Logs from RabbitMQ cluster nodesIf I read these logs correctly it means that it took in this case around 15 sec to elect the new leader, right? p.s.
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks for investigating this and providing us with more details. I have limited access to my computer this week and am looking into this issue from my phone, but I'll be able to investigate more next week. The performance tool uses the load balancer and may retry creating connections with the In the meantime, you may want to have a look at the "Connecting to Streams" blog post if you haven't already. It covers the stream topology discovery mechanism. |
Beta Was this translation helpful? Give feedback.
-
|
@albionb96 I fixed a bug in the stream Java client that was likely the cause of long recovery times and unnecessary consumer connections when the |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Begin too eager to detect an unresponsive peer has very well known downsides. Our Raft's library failure detection parameters can be adjusted via [
{aten, [
{detection_threshold, 0.95}
]}
].Only trial and error can tell what value will be optimal for your needs. The default detection threshold (with what probability will Aten consider the peer as down based on earlier communication latencies) is 0.99. |
Beta Was this translation helpful? Give feedback.
-
|
@albionb96 Thanks for trying with the stream-perf-test snapshot and providing additional info, I'll investigate more on my side. Can you confirm you used the |
Beta Was this translation helpful? Give feedback.
-
|
IIUC you provide the logs from step 4.
This is not normal: the producer should be connected to the leader, the consumer should be connected to a replica. As suggested in my comment below, can you enable logging for a couple of classes: |
Beta Was this translation helpful? Give feedback.
-
If you look to the outputs that I posted, there are I think two moments, where the most of the time is being lost.
I don't know what the client does during this time, because there is not too much output, but it seems like it tries to establish a new connection on the switched-off leader, and then waits for a very long time, and only after that it starts the creation of a connection on the new (correct) leader. @michaelklishin It is not that I myself want to be eager to reduce the downtime during a failure, but as you also know there are some real-time applications which require to reduce this downtime to a minimum, and because I really like all the features that the RabbitMQ offers, and I also like the throughput and latency performance during the normal operation, I want to explore all possibilities until the end, in order to find out if this down-time can be reduced, before deciding if I can use the RabbitMQ as message broker for my application at all or not. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the extra info. I managed to reproduce the issue on a K8S environment. As Karl mentioned, the Erlang distribution takes time to detect the node failure, but we'll try to improve this. There are still a couple of issues on the client side. It seems the producer takes too long to recover, even after the stream leader is up again. I need to investigate more on this. The consumer sometimes connects to the stream leader, even though a replica is available. The stream is "restarted" to elect a new leader and it's likely the consumer tries to resume when there's only the leader available (it's started first, before the replicas), so it uses it. That's just a bad timing. We'd like to introduce a new flag in the library to force the usage of a replica, at least a few times before falling back to the leader. This would not be the default, because it is not always the most appropriate solution (if one wants to recover as fast as possible or if there's only one node and thus only the leader, at least in development, etc). |
Beta Was this translation helpful? Give feedback.
-
|
@albionb96 About the You can enable logging for a couple of classes: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@acogoluegnes I started another test with logging enabled. PerfTool as container in pod- args: - --uris - rabbitmq-stream://user:pass@message-broker - --delete-streams - --prometheus - -z - "3000" - -r - "1000" - -s - "1024" - --batch-size - "1" - --max-length-bytes - "2000000000" - -sac - --rpc-timeout - "5" image: pivotalrabbitmq/stream-perf-test:dev imagePullPolicy: Always name: rabbitmq-stream-perf-test env: - name: RABBITMQ_STREAM_PERF_TEST_LOGGERS value: "com.rabbitmq.stream.impl.ConsumersCoordinator=debug,com.rabbitmq.stream.impl.AsyncRetry=debug,com.rabbitmq.stream.impl.Utils=debug" resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: Filehere I didn't use the --load-balancer flag, since I expect that the nodes are reachable through the services that come by default with the Kubernetes RMQ Cluster Operator, right? At the beginning: I shut-down the VM where the message-broker-server-0 was running, then I got the same error messages (but this time with some more details) and the PerfTool couldn't re-establish the connection anymore. After shut-down: PerfTool output first timeI restarted the pod where the PerfTool was running, and then the PerfTool could establish the connections to the nodes, which it couldn't reach before. Output below: p.s. I let the state of the cluster as it is, that means, I didn't turn-on the VM where the message-broker-server-0, was running, the cluster was running only with two nodes: message-broker-server-1 leader, message-broker-server-2 follower. PerfTool output second time |
Beta Was this translation helpful? Give feedback.
-
|
As already been mentioned, sreams rely on erlang monitors for failure detection. Erlang monitors of processes on a remote node in the case of network partitions rely on the erlang distribution connection to time out before monitors are sent. This could, by default, take around a minute . This is what we see in this case. As the nodes are force stopped they leave dangling erlang distribution TCP connections on the other nodes. The nodes don't know a node is gone as they rely on the TCP connection for this and it looks like it is still there. The only viable option here is lower the Of course a more orderly shutdown procedure where the server being shut down has enough time to send FIN or RST packets (as would be used for upgrades / maintenance etc) so that the remotes can detect the connection is lost will not have this undue delay and detection will be near instantaneous. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
@kjnilsson following your hint regarding the quorum queues, I replaced the streams with quorum queues for a moment and tested my setup quite intensively in the last 2-3 days. I used perf-tool version 2.19.0 both as jar running on a host outside Kubernetes and as image inside Kubernetes. During these tests I noticed some things which I will mention below:
First example: Running the perf-tool-2.19.0, outside Kubernetes as jar file.
Perftool outputRMQ Logs from surviving nodesTraffic outputIn this test-case all three connection were on the leader node, we see that it took overall approx. 24seconds, and the RMQ logs tell that the new leader was elected very fast, but when we see the traffic and the perftool metrics, we see that it took 8-10 seconds to notice that the leader is gone and to elect the new one. We also notice that it took the longest for the production connection to be re-established at the end. Second example Running perf-tool-2.18.0 inside Kubernetes. I started same tests with perf-tool-2.19.0 and the output was the same. Perftool as podRMQ Logs from surviving nodesTraffic outputWe see here that the traffic between the client and rabbitmq cluster is very chaotic, and it takes too long for the client to establish the right connections. |
Beta Was this translation helpful? Give feedback.
-
|
There's no way to find the topology of a queue with AMQP (contrary to the stream protocol), so PerfTest cannot connect a publisher to the node that hosts the leader of a queue (ditto for a consumer and followers). The default connection recovery time is 5 seconds, you can use |
Beta Was this translation helpful? Give feedback.
-
|
@acogoluegnes I already tried setting the But my main question in case of the quorum queues is: We've seen that in regard of electing the new leader the quorum queues are much faster, but in regard of noticing that the leader node is down, they are not much faster than the streams, we have seen that there is a time (8-10 seconds), which the cluster needs to notice that the leader is down. Is this time based on the net_tick_time since I already set the net_tick_time to 8, or is this based on some other parameters, and can this time further be reduced for the quorum queues? |
Beta Was this translation helpful? Give feedback.
-
|
Streams can take around a minute to react to a stream leader being on a partitioned node (and is affected by net_ticktime settings), for quorum queues this takes 5-10s I would say that is substantially faster (and is not affected by net_ticktime). It could be even faster but as a partitioned node is indistinguishable from one that is simply running slowly we favour stability and thus wait a bit to see if the node starts responding before proceeding with an election. I hope this clarifies. Also remember you are testing a scenario where a server is force terminated in a way that doesn't close the open TCP connections which I'd hope would be a relatively rare event in your infrastructure. Availability isn't a property of a single component such as a message broker - it is an aggregate of your entire system. |
Beta Was this translation helpful? Give feedback.
-
|
@albionb96 Just following up on this. I did some testing and improved the stream Java client to make it more responsive after a failure. You can use the Not related to faster recovery, there's also now the You need to use the latest stream-perf-test snapshot for all this. We are still planning on faster stream recovery on the server side. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Describe the bug
Hi,
I tried some failovertest with both stream-perf-test-0.9.0 and stream-perf-test-0.12.0, and although I looked at all possible configuration parameters of the tool and also config parameters of the RabbitMQ Cluster, I was not able to remove or reduce the downtimes shown from the client metrics when a node is killed (leader node or a node with connections on it).
The downtimes vary from 20-60 seconds. I tried to run also the docker image format of the client inside my kubernetes cluster and there I had even longer downtimes. I don't know why it takes the client so long to automatically recover the connections.
Reproduction steps
Expected behavior
I would expect to not see any significant downtimes during these failovertests, but the metrics of the perf tool like: latency, rabbitmq_stream_published_total etc. show downtimes from 20-60seconds, which are too big I think when somebody wants to use rabbitMQ in HA fashion for a HA application which can accept let's say max 1-5 seconds of downtimes in case of failures.
Additional context
I don't know the internals of the java client in this case, but although it says that by default it will create only one consumer and producer I see in the connections overview more than one consumer connection:
Beta Was this translation helpful? Give feedback.
All reactions