A throughput experiment demonstrates significant variation with specific maxUnconfirmedMessages values #750
-
|
I posted this question on the Google group but did not get any feedback, so let's try it here too. While testing the streams with 4.1.0 and Java client 1.0.0 I found some strange behavior. I have a performance test where I publish 10,000 messages to a stream and measure the throughput (messages/second). The stream, the environment and the producer is recreated before each run. The producer is all default, except with different maxUnconfirmedMessages setting at each run. I run the same test series, once with 64 bytes per message, once with 2048 bytes per message. The broker is a single node, running on Kubernetes in Docker Desktop, I run my JUnit test natively on Windows from my IDE. I expect the throughput to increase as I use bigger values for maxUnconfirmedMessages, and that is mostly the trend, however, there are some very strange outliers. maxUnconfirmedMessages=10 seems always producing almost exactly 100 messages per second, while 9 or 11 is two orders of magnitudes higher, with both 64 and 2048 byte messages. The 64 byte message throughput also shows a dip at maxUnconfirmedMessages=100 (but not at 99 or 101) and there are some weird variations around 1000, see below. Can someone explain me what is going on here?
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 12 replies
-
|
@pvaiko not with a single spreadsheet screenshot for problem definition. I don't know what this test does but the numbers are much lower than what I get on a several year old machine with a fast SSD, 8 cores and not containers or VMs between Stream PerfTest and your numbers. java -jar stream-perf-test.jarproduces numbers well over a million of messages per second: Therefore I conclude that your environment is resources constrained for what even a single stream can do. Throughput variation for streams can be directly linked to any I/O throughput variation in the container and not depend on the maximum number of unconfirmed messages alone. That variable is probably not isolated nearly as much as you expect it to be. In fact, I would not expect the maximum number of unconfirmed messages to matter much, there are likely significantly more important factors such as the I/O throughput, since streams are relatively CPU light but very I/O heavy. Finally, |
Beta Was this translation helpful? Give feedback.
-
Sorry, not sure what you mean by that, I have quite a long description here above the spreadsheet. Also not sure why is this closed? You have not answered my question.
I run the tests directly from my IDE (IntelliJ Idea) as I described in the text above, and know it is much slower that way. The question was not about the absolute throughput numbers. We are planning to use RabbitMQ streams for a mission critical application and obviously we'll run real performance test in an environment that matches the production environment. So let me try this again. The question is, why are throughput numbers with maxUnconfirmedMessages=10 are two orders of magnitude lower than maxUnconfirmedMessages=9 or 11? Regarding 10 being a conservative value, yes, we are aware of that, we have system running with prefetch 2, simply because throughput isn't our primary concern for our use cases, even these numbers I got (except the ones for 10) are well over what we need. |
Beta Was this translation helpful? Give feedback.
-
I did not either. But it consistently is two orders of magnitude lower at 10 than at 9 or 11. Every single time I run the test. |
Beta Was this translation helpful? Give feedback.
-
|
@pvaiko it was closed because you haven't provided any reproduction details and I could not reproduce this behavior with Stream PerfTest. See below. I have tried again, with different settings. The numbers are two orders of magnitude different from yours and the 999, 1000, 1001 values do not demonstrate virtually any variability. Something is specific to your home grown benchmark and/or RabbitMQ setup. We won't investigate that for you. It's on you to provide evidence and reproduction steps, not on us to prove that there is no issue (even though I did try several tests in two different environments). |
Beta Was this translation helpful? Give feedback.
-
@pvaiko my point was that the standard benchmarking tool in this community demonstrates numbers that are close to two orders of magnitude higher than what your home grown, known only to you, test does. It's reasonable to assume that your test is the culprit, not this library. You use values such as 10 or 100 while the Stream PerfTest default is 20K. That default hints at what kind of values are expected to be used, something in thousands and not 10 or 100. Team RabbitMQ won't troubleshoot your custom benchmark running in an environment we know nothing about. I did share the basics of my environment: 8 and 10 cores, no virtualization or containers, the tool I used for these benchmarks and the exact command line flags. I did not share the JVM version but that's about the only relevant factor. Why do you believe it's reasonable to ask the core team to analyze a benchmark no one but you can run? We do not guess in this community. Look at every discussion around any of our repos, we always ask for details, logs, an executable way to reproduce, and those who can't are told to figure it on their own. |
Beta Was this translation helpful? Give feedback.
-
I did not ask the core team to analyze anything. I asked a question. You could have simply said, "sorry, we don't know, haven't seen this before, could you please try if you see the same thing with the standard benchmark?" |
Beta Was this translation helpful? Give feedback.
-
|
So to make this a fairer comparison, I have tried a few more runs on a 10 core machine (from early 2022). I don't know what @pvaiko's test does but let's assume that the Stream PerfTest's The default of that setting is 10K, which already tells you what our team believes to be a reasonable number given that Stream PerfTest can approach 1.5-2M messages a second. --confirms 999java -jar stream-perf-test.jar --confirms=999produces --confirms 1000java -jar stream-perf-test.jar --confirms=1000produces --confirms 1001java -jar stream-perf-test.jar --confirms=1001produces --confirms 20000java -jar stream-perf-test.jar --confirms=20000produces ConclusionStream PerfTest does not reproduce this kind of variability in an environment where a reasonably fast 3 year old SSD drive is used and there isn't much disk I/O variability. The problem can be in the benchmark, they are difficult to get right, as we still sometimes discover with Stream PerfTest and PerfTest even after a decade of experience. Or it can be in the disk or network I/O variability in the @pvaiko's environment. The more virtualization or containerization layers ther are, the greater and less predictable it can be. |
Beta Was this translation helpful? Give feedback.
-
|
Have you tried with 10? And then with 9: |
Beta Was this translation helpful? Give feedback.
-
|
@pvaiko 9, 10, 11, 12 all produces very similar values, ranging from 80 to 100 messages a second. With numbers that low you can say that that is some low double digit % of variability but it's not two orders of magnitude and the absolute differences seem irrelevant, I will explain below why. Given that streams in this environment can go up to 1.8M, it clearly suggests that those values are not optimal. The Stream PerfTest default of 20K further suggests that the expected values should be in thousands, not tens. Increasing values to 20, 30, 40, 50 roughly proportionally increases the reported throughput. It will be very difficult for our small team to justify investigating what is the limiting factor with the values as low as 9, 10, 11 but you can dig in, you have the source of this library and Stream PerfTest alike. Profiling tools for Java are mature and likely will help you narrow the problem fairly quickly. Why is that a problem that would need fixing, I don't personally understand. Perhaps @acogoluegnes would have something to add (next week). |
Beta Was this translation helpful? Give feedback.
-
I'm sorry, where? On your environment? On my, as you see above, 10 produces around 100 messages per second, 9 over 5000. |
Beta Was this translation helpful? Give feedback.
-
|
@pvaiko I do observe a significant jump (about two orders of magnitude) in my environments when the 100 happens to correspond to the default So, the client will intentionally throttle itself before that many messages are accumulated, I cannot find any defaults or constants that would match |
Beta Was this translation helpful? Give feedback.
-
|
There's also /**
* Adapt batch size depending on ingress rate.
*
* <p>A dynamic-batch approach improves latency for low ingress rates.
*
* <p>Set this flag to <code>true</code> if you want as little delay as possible between calling
* {@link Producer#send(Message, ConfirmationHandler)} and the message being sent to the broker.
* Consumers should provide enough initial credits (between 5 and 10, depending on the workload),
* see {@link ConsumerBuilder#flow()} and {@link
* ConsumerBuilder.FlowConfiguration#initialCredits(int)}.
*
* <p>Set this flag to <code>false</code> if latency is not critical for your use case and you
* want the highest throughput possible for both publishing and consuming. Consumers can provide 1
* initial credit (depending on the workload), see {@link ConsumerBuilder#flow()} and {@link
* ConsumerBuilder.FlowConfiguration#initialCredits(int)}.
*
* <p>Dynamic batch is activated by default (<code>dynamicBatch = true</code>).
*
* @param dynamicBatch
* @return this builder instance
* @since 0.20.0
* @see ConsumerBuilder#flow()
* @see com.rabbitmq.stream.ConsumerBuilder.FlowConfiguration#initialCredits(int)
*/
ProducerBuilder dynamicBatch(boolean dynamicBatch);In Stream PerfTest, this setting is exposed via @pvaiko now you know what our team uses for benchmarking and that this client heavily relies on credit-based flow control and batching (in more than place), that the default In fact, when comparing older versions of Stream PerfTest I could notice more variability with those values of 10 and 100. The most likely reason is the introduction of dynamic batching in November 2024. It then was enabled by default in December: In #671 you see a mention of a default value of 10: it's the initial credit value. I do not see anything "broken" in the client and the behavior has been narrowed down significantly. Using values of 1K or a few thousands seems perfectly acceptable to me Please take it from here. |
Beta Was this translation helpful? Give feedback.
-
|
Do me a favor, run it with --confirms 9, --confirms 10 and --confirms 11. And then with 99, 100 and 101. |
Beta Was this translation helpful? Give feedback.
-
|
@pvaiko I do see a drop for 10 with the latest Stream PerfTest, and a drop around 100 with both the latest and an older version I had from September 2024. The most likely reason is the introduction of dynamic batching in November 2024. It then was enabled by default in December: In #671 you see a mention of a default value of 10: it's the initial credit. I don't see how it changes anything in practical terms. Use a value of 1K (or higher, like Stream PerfTest does by default) and move on. |
Beta Was this translation helpful? Give feedback.
-
|
I have filed #751, we'll see what @acogoluegnes thinks. In any case, using values higher than 100 is a perfectly acceptable, if not recommended, thing to do, so I'm moving on. |
Beta Was this translation helpful? Give feedback.
-
|
I have filed #751, we'll see what @acogoluegnes thinks. In any case, using values higher than 100 is a perfectly acceptable, if not recommended, thing to do, so I'm moving on. |
Beta Was this translation helpful? Give feedback.
There's also
ProducerBuilder#dynamicBatchthat is enabled by default but can be used if you care more about throughput than latency: