Skip to content

Commit 03a2124

Browse files
authored
GH-2164: Remove EOSMode.V1
Resolves #2164 * Remove dead DKPF code.
1 parent a8f98bd commit 03a2124

12 files changed

+135
-295
lines changed

spring-kafka-docs/src/main/asciidoc/changes-since-1.0.adoc

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,104 @@
11
[[migration]]
2+
=== What's New in 2.8 Since 2.7
3+
4+
This section covers the changes made from version 2.7 to version 2.8.
5+
For changes in earlier version, see <<history>>.
6+
7+
[[x28-kafka-client]]
8+
==== Kafka Client Version
9+
10+
This version requires the 3.0.0 `kafka-clients`
11+
12+
[[x28-packages]]
13+
==== Package Changes
14+
15+
Classes and interfaces related to type mapping have been moved from `...support.converter` to `...support.mapping`.
16+
17+
* `AbstractJavaTypeMapper`
18+
* `ClassMapper`
19+
* `DefaultJackson2JavaTypeMapper`
20+
* `Jackson2JavaTypeMapper`
21+
22+
[[x28-ooo-commits]]
23+
==== Out of Order Manual Commits
24+
25+
The listener container can now be configured to accept manual offset commits out of order (usually asynchronously).
26+
The container will defer the commit until the missing offset is acknowledged.
27+
See <<ooo-commits>> for more information.
28+
29+
[[x28-batch-overrude]]
30+
==== `@KafkaListener` Changes
31+
32+
It is now possible to specify whether the listener method is a batch listener on the method itself.
33+
This allows the same container factory to be used for both record and batch listeners.
34+
35+
See <<batch-listeners>> for more information.
36+
37+
Batch listeners can now handle conversion exceptions.
38+
39+
See <<batch-listener-conv-errors>> for more information.
40+
41+
`RecordFilterStrategy`, when used with batch listeners, can now filter the entire batch in one call.
42+
See the note at the end of <<batch-listeners>> for more information.
43+
44+
The `@KafkaListener` annotation now has the `filter` attribute, to override the container factory's `RecordFilterStrategy` for just this listener.
45+
46+
The `@KafkaListener` annotation now has the `info` attribute; this is used to populate the new listener container property `listenerInfo`.
47+
This is then used to populate a `KafkaHeaders.LISTENER_INFO` header in each record which can be used in `RecordInterceptor`, `RecordFilterStrategy`, or the listener itself.
48+
See <<li-header>> and <<alc-props,Abstract Listener Container Properties>> for more information.
49+
50+
[[x28-template]]
51+
==== `KafkaTemplate` Changes
52+
53+
You can now receive a single record, given the topic, partition and offset.
54+
See <<kafka-template-receive>> for more information.
55+
56+
[[x28-eh]]
57+
==== `CommonErrorHandler` Added
58+
59+
The legacy `GenericErrorHandler` and its sub-interface hierarchies for record an batch listeners have been replaced by a new single interface `CommonErrorHandler` with implementations corresponding to most legacy implementations of `GenericErrorHandler`.
60+
See <<error-handlers>> and <<migrating-legacy-eh>> for more information.
61+
62+
[[x28-lcc]]
63+
==== Listener Container Changes
64+
65+
The `interceptBeforeTx` container property is now `true` by default.
66+
67+
The `authorizationExceptionRetryInterval` property has been renamed to `authExceptionRetryInterval` and now applies to `AuthenticationException` s in addition to `AuthorizationException` s previously.
68+
Both exceptions are considered fatal and the container will stop by default, unless this property is set.
69+
70+
See <<kafka-container>> and <<container-props>> for more information.
71+
72+
[[x28-serializers]]
73+
==== Serializer/Deserializer Changes
74+
75+
The `DelegatingByTopicSerializer` and `DelegatingByTopicDeserializer` are now provided.
76+
See <<delegating-serialization>> for more information.
77+
78+
[[x28-dlpr]]
79+
==== `DeadLetterPublishingRecover` Changes
80+
81+
The property `stripPreviousExceptionHeaders` is now `true` by default.
82+
83+
There are now several techniques to customize which headers are added to the output record.
84+
85+
See <<dlpr-headers>> for more information.
86+
87+
[[x28-retryable-topics-changes]]
88+
==== Retryable Topics Changes
89+
90+
Now you can use the same factory for retryable and non-retryable topics.
91+
See <<retry-topic-lcf>> for more information.
92+
93+
There's now a manageable global list of fatal exceptions that will make the failed record go straight to the DLT.
94+
Refer to <<retry-topic-ex-classifier>> to see how to manage it.
95+
96+
You can now use blocking and non-blocking retries in conjunction.
97+
See <<retry-topic-combine-blocking>> for more information.
98+
99+
The KafkaBackOffException thrown when using the retryable topics feature is now logged at DEBUG level.
100+
See <<change-kboe-logging-level>> if you need to change the logging level back to WARN or set it to any other level.
101+
2102
=== Changes between 2.6 and 2.7
3103

4104
[[x27-kafka-client]]

spring-kafka-docs/src/main/asciidoc/kafka.adoc

Lines changed: 5 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -2585,7 +2585,7 @@ See <<after-rollback>> for more information.
25852585
|[[subBatchPerPartition]]<<subBatchPerPartition,`subBatchPerPartition`>>
25862586
|See desc.
25872587
|When using a batch listener, if this is `true`, the listener is called with the results of the poll split into sub batches, one per partition.
2588-
Default `false` except when using transactions with `EOSMode.ALPHA` - see <<exactly-once>>.
2588+
Default `false`.
25892589

25902590
|[[syncCommitTimeout]]<<syncCommitTimeout,`syncCommitTimeout`>>
25912591
|`null`
@@ -3381,10 +3381,6 @@ This is to properly support fencing zombies, https://www.confluent.io/blog/trans
33813381
This new behavior was added in versions 1.3.7, 2.0.6, 2.1.10, and 2.2.0.
33823382
If you wish to revert to the previous behavior, you can set the `producerPerConsumerPartition` property on the `DefaultKafkaProducerFactory` to `false`.
33833383

3384-
NOTE: While transactions are supported with batch listeners, by default, zombie fencing is not supported because a batch may contain records from multiple topics or partitions.
3385-
However, starting with version 2.3.2, zombie fencing is supported if you set the container property `subBatchPerPartition` to true.
3386-
In that case, the batch listener is invoked once per partition received from the last poll, as if each poll only returned records for a single partition.
3387-
This is `true` by default since version 2.5 when transactions are enabled with `EOSMode.ALPHA`; set it to `false` if you are using transactions but are not concerned about zombie fencing.
33883384
Also see <<exactly-once>>.
33893385

33903386
Also see <<transaction-id-prefix>>.
@@ -3469,22 +3465,9 @@ Instead, a new "nested" transaction is used.
34693465
[[transaction-id-prefix]]
34703466
===== `transactionIdPrefix`
34713467

3472-
As mentioned in <<transactions, the overview>>, the producer factory is configured with this property to build the producer `transactional.id` property.
3473-
There is a dichotomy when specifying this property in that, when running multiple instances of the application with `EOSMode.ALPHA`, it must be the same on all instances to satisfy fencing zombies (also mentioned in the overview) when producing records on a listener container thread.
3474-
However, when producing records using transactions that are **not** started by a listener container, the prefix has to be different on each instance.
3475-
Version 2.3, makes this simpler to configure, especially in a Spring Boot application.
3476-
In previous versions, you had to create two producer factories and `KafkaTemplate` s - one for producing records on a listener container thread and one for stand-alone transactions started by `kafkaTemplate.executeInTransaction()` or by a transaction interceptor on a `@Transactional` method.
3477-
3478-
Now, you can override the factory's `transactionalIdPrefix` on the `KafkaTemplate` and the `KafkaTransactionManager`.
3479-
3480-
When using a transaction manager and template for a listener container, you would normally leave this to default to the producer factory's property.
3481-
This value should be the same for all application instances when using `EOSMode.ALPHA`.
3482-
With `EOSMode.BETA` it is no longer necessary to use the same `transactional.id`, even for consumer-initiated transactions; in fact, it must be unique on each instance the same as producer-initiated transactions.
3483-
For transactions started by the template (or the transaction manager for `@Transaction`) you should set the property on the template and transaction manager respectively.
3468+
With `EOSMode.V2` (aka `BETA`), the only supported mode, it is no longer necessary to use the same `transactional.id`, even for consumer-initiated transactions; in fact, it must be unique on each instance the same as for producer-initiated transactions.
34843469
This property must have a different value on each application instance.
34853470

3486-
This problem (different rules for `transactional.id`) has been eliminated when `EOSMode.BETA` is being used (with broker versions >= 2.5); see <<exactly-once>>.
3487-
34883471
[[tx-template-mixed]]
34893472
===== `KafkaTemplate` Transactional and non-Transactional Publishing
34903473

@@ -3572,37 +3555,17 @@ Using transactions enables Exactly Once Semantics (EOS).
35723555
This means that, for a `read->process-write` sequence, it is guaranteed that the **sequence** is completed exactly once.
35733556
(The read and process are have at least once semantics).
35743557

3575-
Spring for Apache Kafka version 2.5 and later supports two EOS modes:
3558+
Spring for Apache Kafka version 3.0 and later only supports `EOSMode.V2`:
35763559

3577-
* `ALPHA` - alias for `V1` (deprecated)
3578-
* `BETA` - alias for `V2` (deprecated)
3579-
* `V1` - aka `transactional.id` fencing (since version 0.11.0.0)
35803560
* `V2` - aka fetch-offset-request fencing (since version 2.5)
35813561

3582-
With mode `V1`, the producer is "fenced" if another instance with the same `transactional.id` is started.
3583-
Spring manages this by using a `Producer` for each `group.id/topic/partition`; when a rebalance occurs a new instance will use the same `transactional.id` and the old producer is fenced.
3562+
IMPORTANT: This requires the brokers to be version 2.5 or later.
35843563

35853564
With mode `V2`, it is not necessary to have a producer for each `group.id/topic/partition` because consumer metadata is sent along with the offsets to the transaction and the broker can determine if the producer is fenced using that information instead.
35863565

3587-
Starting with version 2.6, the default `EOSMode` is `V2`.
3588-
3589-
To configure the container to use mode `ALPHA`, set the container property `EOSMode` to `ALPHA`, to revert to the previous behavior.
3590-
3591-
IMPORTANT: With `V2` (default), your brokers must be version 2.5 or later; `kafka-clients` version 3.0, the producer will no longer fall back to `V1`; if the broker does not support `V2`, an exception is thrown.
3592-
If your brokers are earlier than 2.5, you must set the `EOSMode` to `V1`, leave the `DefaultKafkaProducerFactory` `producerPerConsumerPartition` set to `true` and, if you are using a batch listener, you should set `subBatchPerPartition` to `true`.
3593-
3594-
When your brokers are upgraded to 2.5 or later, you should switch the mode to `V2`, but the number of producers will remain as before.
3595-
You can then do a rolling upgrade of your application with `producerPerConsumerPartition` set to `false` to reduce the number of producers; you should also no longer set the `subBatchPerPartition` container property.
3596-
3597-
If your brokers are already 2.5 or newer, you should set the `DefaultKafkaProducerFactory` `producerPerConsumerPartition` property to `false`, to reduce the number of producers needed.
3598-
3599-
IMPORTANT: When using `EOSMode.V2` with `producerPerConsumerPartition=false` the `transactional.id` must be unique across all application instances.
3600-
3601-
When using `V2` mode, it is no longer necessary to set the `subBatchPerPartition` to `true`; it will default to `false` when the `EOSMode` is `V2`.
3602-
36033566
Refer to https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.
36043567

3605-
`V1` and `V2` were previously `ALPHA` and `BETA`; they have been changed to align the framework with https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2[KIP-732].
3568+
`V2` was previously `BETA`; the `EOSMode` has been changed to align the framework with https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2[KIP-732].
36063569

36073570
[[interceptors]]
36083571
==== Wiring Spring Beans into Producer/Consumer Interceptors

spring-kafka-docs/src/main/asciidoc/whats-new.adoc

Lines changed: 6 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -3,101 +3,16 @@
33
This section covers the changes made from version 2.7 to version 2.8.
44
For changes in earlier version, see <<history>>.
55

6-
[[x28-kafka-client]]
6+
[[x30-kafka-client]]
77
==== Kafka Client Version
88

99
This version requires the 3.1.0 `kafka-clients`
1010

11-
IMPORTANT: When using transactions, the minimum broker version is 2.5.
12-
13-
See <<exactly-once>> and https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.
14-
15-
[[x28-packages]]
16-
==== Package Changes
17-
18-
Classes and interfaces related to type mapping have been moved from `...support.converter` to `...support.mapping`.
19-
20-
* `AbstractJavaTypeMapper`
21-
* `ClassMapper`
22-
* `DefaultJackson2JavaTypeMapper`
23-
* `Jackson2JavaTypeMapper`
24-
25-
[[x28-ooo-commits]]
26-
==== Out of Order Manual Commits
27-
28-
The listener container can now be configured to accept manual offset commits out of order (usually asynchronously).
29-
The container will defer the commit until the missing offset is acknowledged.
30-
See <<ooo-commits>> for more information.
31-
32-
[[x28-batch-overrude]]
33-
==== `@KafkaListener` Changes
34-
35-
It is now possible to specify whether the listener method is a batch listener on the method itself.
36-
This allows the same container factory to be used for both record and batch listeners.
37-
38-
See <<batch-listeners>> for more information.
39-
40-
Batch listeners can now handle conversion exceptions.
41-
42-
See <<batch-listener-conv-errors>> for more information.
43-
44-
`RecordFilterStrategy`, when used with batch listeners, can now filter the entire batch in one call.
45-
See the note at the end of <<batch-listeners>> for more information.
46-
47-
The `@KafkaListener` annotation now has the `filter` attribute, to override the container factory's `RecordFilterStrategy` for just this listener.
48-
49-
The `@KafkaListener` annotation now has the `info` attribute; this is used to populate the new listener container property `listenerInfo`.
50-
This is then used to populate a `KafkaHeaders.LISTENER_INFO` header in each record which can be used in `RecordInterceptor`, `RecordFilterStrategy`, or the listener itself.
51-
See <<li-header>> and <<alc-props,Abstract Listener Container Properties>> for more information.
52-
53-
[[x28-template]]
54-
==== `KafkaTemplate` Changes
11+
[[x30-eos]]
12+
==== Exactly Once Semantics
5513

56-
You can now receive a single record, given the topic, partition and offset.
57-
See <<kafka-template-receive>> for more information.
14+
`EOSMode.V1` (aka `ALPHA`) is no longer supported.
5815

59-
[[x28-eh]]
60-
==== `CommonErrorHandler` Added
61-
62-
The legacy `GenericErrorHandler` and its sub-interface hierarchies for record an batch listeners have been replaced by a new single interface `CommonErrorHandler` with implementations corresponding to most legacy implementations of `GenericErrorHandler`.
63-
See <<error-handlers>> and <<migrating-legacy-eh>> for more information.
64-
65-
[[x28-lcc]]
66-
==== Listener Container Changes
67-
68-
The `interceptBeforeTx` container property is now `true` by default.
69-
70-
The `authorizationExceptionRetryInterval` property has been renamed to `authExceptionRetryInterval` and now applies to `AuthenticationException` s in addition to `AuthorizationException` s previously.
71-
Both exceptions are considered fatal and the container will stop by default, unless this property is set.
72-
73-
See <<kafka-container>> and <<container-props>> for more information.
74-
75-
[[x28-serializers]]
76-
==== Serializer/Deserializer Changes
77-
78-
The `DelegatingByTopicSerializer` and `DelegatingByTopicDeserializer` are now provided.
79-
See <<delegating-serialization>> for more information.
80-
81-
[[x28-dlpr]]
82-
==== `DeadLetterPublishingRecover` Changes
83-
84-
The property `stripPreviousExceptionHeaders` is now `true` by default.
85-
86-
There are now several techniques to customize which headers are added to the output record.
87-
88-
See <<dlpr-headers>> for more information.
89-
90-
[[x28-retryable-topics-changes]]
91-
==== Retryable Topics Changes
92-
93-
Now you can use the same factory for retryable and non-retryable topics.
94-
See <<retry-topic-lcf>> for more information.
95-
96-
There's now a manageable global list of fatal exceptions that will make the failed record go straight to the DLT.
97-
Refer to <<retry-topic-ex-classifier>> to see how to manage it.
98-
99-
You can now use blocking and non-blocking retries in conjunction.
100-
See <<retry-topic-combine-blocking>> for more information.
16+
IMPORTANT: When using transactions, the minimum broker version is 2.5.
10117

102-
The KafkaBackOffException thrown when using the retryable topics feature is now logged at DEBUG level.
103-
See <<change-kboe-logging-level>> if you need to change the logging level back to WARN or set it to any other level.
18+
See <<exactly-once>> and https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.

0 commit comments

Comments
 (0)