Document compression

Fixes #90

Author: acogoluegnes

src/docs/asciidoc/api.adoc

@@ -362,12 +362,20 @@ The following table sums up the main settings to create a `Producer`:
 |100
 
 |`subEntrySize`
-|[[producer-sub-entry-size-configuration-entry]]The number of messages to put in a sub-entry. A sub-entry is one "slot" in a publishing
+|[[producer-sub-entry-size-configuration-entry]]The number of messages to put in a sub-entry.
+A sub-entry is one "slot" in a publishing
 frame, meaning outbound messages are not only batched in publishing frames, but in sub-entries
-as well. Use this feature to increase throughput at the cost of increased latency and
+as well.
+Use this feature to increase throughput at the cost of increased latency and
 potential duplicated messages even when deduplication is enabled.
+See the <<sub-entry-batching-and-compression, dedicated section>> for more information.
 |1 (meaning no use of sub-entry batching)
 
+|`compression`
+|Compression algorithm to use when sub-entry batching is in use.
+See the <<sub-entry-batching-and-compression, dedicated section>> for more information.
+|Compression.NONE
+
 |`maxUnconfirmedMessages`
 |The maximum number of unconfirmed outbound messages. `Producer#send` will start
 blocking when the limit is reached.
@@ -501,7 +509,7 @@ on 2 client-side elements: the _producer name_ and the _message publishing ID_.
 .Deduplication is not guaranteed when using sub-entries batching
 ====
 It is not possible to guarantee deduplication when
-<<producer-sub-entry-size-configuration-entry, sub-entry batching>> is in use.
+<<sub-entry-batching-and-compression, sub-entry batching>> is in use.
 Sub-entry batching is disabled by default and it does not prevent from
 batching messages in a single publish frame, which can already provide
 very high throughput.
@@ -623,6 +631,81 @@ include::{test-examples}/ProducerUsage.java[tag=producer-queries-last-publishing
 <4> Scroll to the content for the next publishing ID
 <5> Set the message publishing
 
+[[sub-entry-batching-and-compression]]
+===== Sub-Entry Batching and Compression
+
+RabbitMQ Stream provides a special mode to publish, store, and dispatch messages: sub-entry batching.
+This mode increases throughput at the cost of increased latency and potential duplicated messages even when deduplication is enabled.
+It also allows using compression to reduce bandwidth and storage if messages are reasonably similar, at the cost of increasing CPU usage on the client side.
+
+Sub-entry batching consists in squeezing several messages – a batch – in the slot that is usually used for one message.
+This means outbound messages are not only batched in publishing frames, but in sub-entries as well.
+
+You can enable sub-entry batching by setting the `ProducerBuilder#subEntrySize` parameter to a value greater than 1, like in the following snippet:
+
+.Enabling sub-entry batching
+[source,java,indent=0]
+--------
+include::{test-examples}/ProducerUsage.java[tag=producer-sub-entry-batching]
+--------
+<1> Set batch size to 100 (the default)
+<2> Set sub-entry size to 10
+
+Reasonable values for the sub-entry size usually go from 10 to a few dozens.
+
+A sub-entry batch will go directly to disc after it reached the broker, so the publishing client has complete control over it.
+This is the occasion to take advantage of the similarity of messages and compress them.
+There is no compression by default but you can choose among several algorithms with the `ProducerBuilder#compression(Compression)` method:
+
+.Enabling compression of sub-entry messages
+[source,java,indent=0]
+--------
+include::{test-examples}/ProducerUsage.java[tag=producer-sub-entry-batching-and-compression]
+--------
+<1> Set batch size to 100 (the default)
+<2> Set sub-entry size to 10
+<3> Use the Zstandard compression algorithm
+
+Note the messages in a sub-entry are compressed altogether to benefit from their potential similarity, not one by one.
+
+The following table lists the supported algorithms, general information about them, and the respective implementations used by default.
+
+[%header,cols=3*]
+|===
+|Algorithm
+|Overview
+|Implementation used
+
+|https://en.wikipedia.org/wiki/Gzip[gzip]
+|Has a high compression ratio but is slow compared to other algorithms.
+|JDK implementation
+
+|https://en.wikipedia.org/wiki/Snappy_(compression)[Snappy]
+|Aims for reasonable compression ratio and very high speeds.
+|https://github.com/xerial/snappy-java[Xerial Snappy] (framed)
+
+|https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)[LZ4]
+|Aims for good trade-off between speed and compression ratio.
+|https://github.com/lz4/lz4-java[LZ4 Java] (framed)
+
+|https://en.wikipedia.org/wiki/Zstd[zstd] (Zstandard)
+|Aims for high compression ratio and high speed, especially for decompression.
+|https://github.com/luben/zstd-jni[zstd-jni]
+|===
+
+You are encouraged to test and evaluate the compression algorithms depending on your needs.
+
+The compression libraries are pluggable thanks to the `EnvironmentBuilder#compressionCodecFactory(CompressionCodecFactory)` method.
+
+
+[NOTE]
+.Consumers, sub-entry batching, and compression
+====
+There is no configuration required for consumers with regard to sub-entry batching and compression.
+The broker dispatches messages to client libraries: they are supposed to figure out the format of messages, extract them from their sub-entry, and decompress them if necessary.
+So when you set up sub-entry batching and compression in your publishers, the consuming applications must use client libraries that support this mode, which is the case for the stream Java client.
+====
+
 ==== Consumer
 
 `Consumer` is the API to consume messages from a stream.

src/main/java/com/rabbitmq/stream/EnvironmentBuilder.java

@@ -1,4 +1,4 @@
-// Copyright (c) 2020-2021 VMware, Inc. or its affiliates.  All rights reserved.
+// Copyright (c) 2020-2022 VMware, Inc. or its affiliates.  All rights reserved.
 //
 // This software, the RabbitMQ Stream Java client library, is dual-licensed under the
 // Mozilla Public License 2.0 ("MPL"), and the Apache License version 2 ("ASL").
@@ -13,6 +13,7 @@
 // [email protected].
 package com.rabbitmq.stream;
 
+import com.rabbitmq.stream.compression.Compression;
 import com.rabbitmq.stream.compression.CompressionCodecFactory;
 import com.rabbitmq.stream.metrics.MetricsCollector;
 import com.rabbitmq.stream.sasl.CredentialsProvider;
@@ -101,6 +102,14 @@ public interface EnvironmentBuilder {
 
   EnvironmentBuilder byteBufAllocator(ByteBufAllocator byteBufAllocator);
 
+  /**
+   * Compression codec factory to use for compression in sub-entry batching.
+   *
+   * @param compressionCodecFactory
+   * @return this builder instance
+   * @see ProducerBuilder#subEntrySize(int)
+   * @see ProducerBuilder#compression(Compression)
+   */
   EnvironmentBuilder compressionCodecFactory(CompressionCodecFactory compressionCodecFactory);
 
   /**

src/main/java/com/rabbitmq/stream/ProducerBuilder.java

@@ -1,4 +1,4 @@
-// Copyright (c) 2020-2021 VMware, Inc. or its affiliates.  All rights reserved.
+// Copyright (c) 2020-2022 VMware, Inc. or its affiliates.  All rights reserved.
 //
 // This software, the RabbitMQ Stream Java client library, is dual-licensed under the
 // Mozilla Public License 2.0 ("MPL"), and the Apache License version 2 ("ASL").
@@ -49,6 +49,20 @@ public interface ProducerBuilder {
    */
   ProducerBuilder subEntrySize(int subEntrySize);
 
+  /**
+   * Compression algorithm to use to compress a batch of sub-entries.
+   *
+   * <p>Compression can take advantage of similarity in messages to significantly reduce the size of
+   * the sub-entry batch. This translates to less bandwidth and storage used, at the cost of more
+   * CPU usage to compress and decompress on the client side. Note the server is not involved in the
+   * compression/decompression process.
+   *
+   * <p>Default is no compression.
+   *
+   * @param compression
+   * @return this builder instance
+   * @see Compression
+   */
   ProducerBuilder compression(Compression compression);
 
   /**

src/test/java/com/rabbitmq/stream/docs/ProducerUsage.java

@@ -1,4 +1,4 @@
-// Copyright (c) 2020-2021 VMware, Inc. or its affiliates.  All rights reserved.
+// Copyright (c) 2020-2022 VMware, Inc. or its affiliates.  All rights reserved.
 //
 // This software, the RabbitMQ Stream Java client library, is dual-licensed under the
 // Mozilla Public License 2.0 ("MPL"), and the Apache License version 2 ("ASL").
@@ -17,6 +17,7 @@
 import com.rabbitmq.stream.Environment;
 import com.rabbitmq.stream.Message;
 import com.rabbitmq.stream.Producer;
+import com.rabbitmq.stream.compression.Compression;
 import java.nio.charset.StandardCharsets;
 import java.time.Duration;
 import java.util.UUID;
@@ -110,6 +111,29 @@ void producerWithNameQueryLastPublishingId() {
         // end::producer-queries-last-publishing-id[]
     }
 
+    void producerSubEntryBatching() {
+        Environment environment = Environment.builder().build();
+        // tag::producer-sub-entry-batching[]
+        Producer producer = environment.producerBuilder()
+            .stream("my-stream")
+            .batchSize(100) // <1>
+            .subEntrySize(10) // <2>
+            .build();
+        // end::producer-sub-entry-batching[]
+    }
+
+    void producerSubEntryBatchingCompression() {
+        Environment environment = Environment.builder().build();
+        // tag::producer-sub-entry-batching-and-compression[]
+        Producer producer = environment.producerBuilder()
+            .stream("my-stream")
+            .batchSize(100) // <1>
+            .subEntrySize(10) // <2>
+            .compression(Compression.ZSTD) // <3>
+            .build();
+        // end::producer-sub-entry-batching-and-compression[]
+    }
+
     boolean moreContent(long publishingId) {
         return true;
     }