You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: spring-kafka-docs/src/main/asciidoc/changes-since-1.0.adoc
+100Lines changed: 100 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,104 @@
1
1
[[migration]]
2
+
=== What's New in 2.8 Since 2.7
3
+
4
+
This section covers the changes made from version 2.7 to version 2.8.
5
+
For changes in earlier version, see <<history>>.
6
+
7
+
[[x28-kafka-client]]
8
+
==== Kafka Client Version
9
+
10
+
This version requires the 3.0.0 `kafka-clients`
11
+
12
+
[[x28-packages]]
13
+
==== Package Changes
14
+
15
+
Classes and interfaces related to type mapping have been moved from `...support.converter` to `...support.mapping`.
16
+
17
+
* `AbstractJavaTypeMapper`
18
+
* `ClassMapper`
19
+
* `DefaultJackson2JavaTypeMapper`
20
+
* `Jackson2JavaTypeMapper`
21
+
22
+
[[x28-ooo-commits]]
23
+
==== Out of Order Manual Commits
24
+
25
+
The listener container can now be configured to accept manual offset commits out of order (usually asynchronously).
26
+
The container will defer the commit until the missing offset is acknowledged.
27
+
See <<ooo-commits>> for more information.
28
+
29
+
[[x28-batch-overrude]]
30
+
==== `@KafkaListener` Changes
31
+
32
+
It is now possible to specify whether the listener method is a batch listener on the method itself.
33
+
This allows the same container factory to be used for both record and batch listeners.
34
+
35
+
See <<batch-listeners>> for more information.
36
+
37
+
Batch listeners can now handle conversion exceptions.
38
+
39
+
See <<batch-listener-conv-errors>> for more information.
40
+
41
+
`RecordFilterStrategy`, when used with batch listeners, can now filter the entire batch in one call.
42
+
See the note at the end of <<batch-listeners>> for more information.
43
+
44
+
The `@KafkaListener` annotation now has the `filter` attribute, to override the container factory's `RecordFilterStrategy` for just this listener.
45
+
46
+
The `@KafkaListener` annotation now has the `info` attribute; this is used to populate the new listener container property `listenerInfo`.
47
+
This is then used to populate a `KafkaHeaders.LISTENER_INFO` header in each record which can be used in `RecordInterceptor`, `RecordFilterStrategy`, or the listener itself.
48
+
See <<li-header>> and <<alc-props,Abstract Listener Container Properties>> for more information.
49
+
50
+
[[x28-template]]
51
+
==== `KafkaTemplate` Changes
52
+
53
+
You can now receive a single record, given the topic, partition and offset.
54
+
See <<kafka-template-receive>> for more information.
55
+
56
+
[[x28-eh]]
57
+
==== `CommonErrorHandler` Added
58
+
59
+
The legacy `GenericErrorHandler` and its sub-interface hierarchies for record an batch listeners have been replaced by a new single interface `CommonErrorHandler` with implementations corresponding to most legacy implementations of `GenericErrorHandler`.
60
+
See <<error-handlers>> and <<migrating-legacy-eh>> for more information.
61
+
62
+
[[x28-lcc]]
63
+
==== Listener Container Changes
64
+
65
+
The `interceptBeforeTx` container property is now `true` by default.
66
+
67
+
The `authorizationExceptionRetryInterval` property has been renamed to `authExceptionRetryInterval` and now applies to `AuthenticationException` s in addition to `AuthorizationException` s previously.
68
+
Both exceptions are considered fatal and the container will stop by default, unless this property is set.
69
+
70
+
See <<kafka-container>> and <<container-props>> for more information.
71
+
72
+
[[x28-serializers]]
73
+
==== Serializer/Deserializer Changes
74
+
75
+
The `DelegatingByTopicSerializer` and `DelegatingByTopicDeserializer` are now provided.
76
+
See <<delegating-serialization>> for more information.
77
+
78
+
[[x28-dlpr]]
79
+
==== `DeadLetterPublishingRecover` Changes
80
+
81
+
The property `stripPreviousExceptionHeaders` is now `true` by default.
82
+
83
+
There are now several techniques to customize which headers are added to the output record.
84
+
85
+
See <<dlpr-headers>> for more information.
86
+
87
+
[[x28-retryable-topics-changes]]
88
+
==== Retryable Topics Changes
89
+
90
+
Now you can use the same factory for retryable and non-retryable topics.
91
+
See <<retry-topic-lcf>> for more information.
92
+
93
+
There's now a manageable global list of fatal exceptions that will make the failed record go straight to the DLT.
94
+
Refer to <<retry-topic-ex-classifier>> to see how to manage it.
95
+
96
+
You can now use blocking and non-blocking retries in conjunction.
97
+
See <<retry-topic-combine-blocking>> for more information.
98
+
99
+
The KafkaBackOffException thrown when using the retryable topics feature is now logged at DEBUG level.
100
+
See <<change-kboe-logging-level>> if you need to change the logging level back to WARN or set it to any other level.
@@ -3381,10 +3381,6 @@ This is to properly support fencing zombies, https://www.confluent.io/blog/trans
3381
3381
This new behavior was added in versions 1.3.7, 2.0.6, 2.1.10, and 2.2.0.
3382
3382
If you wish to revert to the previous behavior, you can set the `producerPerConsumerPartition` property on the `DefaultKafkaProducerFactory` to `false`.
3383
3383
3384
-
NOTE: While transactions are supported with batch listeners, by default, zombie fencing is not supported because a batch may contain records from multiple topics or partitions.
3385
-
However, starting with version 2.3.2, zombie fencing is supported if you set the container property `subBatchPerPartition` to true.
3386
-
In that case, the batch listener is invoked once per partition received from the last poll, as if each poll only returned records for a single partition.
3387
-
This is `true` by default since version 2.5 when transactions are enabled with `EOSMode.ALPHA`; set it to `false` if you are using transactions but are not concerned about zombie fencing.
3388
3384
Also see <<exactly-once>>.
3389
3385
3390
3386
Also see <<transaction-id-prefix>>.
@@ -3469,22 +3465,9 @@ Instead, a new "nested" transaction is used.
3469
3465
[[transaction-id-prefix]]
3470
3466
===== `transactionIdPrefix`
3471
3467
3472
-
As mentioned in <<transactions, the overview>>, the producer factory is configured with this property to build the producer `transactional.id` property.
3473
-
There is a dichotomy when specifying this property in that, when running multiple instances of the application with `EOSMode.ALPHA`, it must be the same on all instances to satisfy fencing zombies (also mentioned in the overview) when producing records on a listener container thread.
3474
-
However, when producing records using transactions that are **not** started by a listener container, the prefix has to be different on each instance.
3475
-
Version 2.3, makes this simpler to configure, especially in a Spring Boot application.
3476
-
In previous versions, you had to create two producer factories and `KafkaTemplate` s - one for producing records on a listener container thread and one for stand-alone transactions started by `kafkaTemplate.executeInTransaction()` or by a transaction interceptor on a `@Transactional` method.
3477
-
3478
-
Now, you can override the factory's `transactionalIdPrefix` on the `KafkaTemplate` and the `KafkaTransactionManager`.
3479
-
3480
-
When using a transaction manager and template for a listener container, you would normally leave this to default to the producer factory's property.
3481
-
This value should be the same for all application instances when using `EOSMode.ALPHA`.
3482
-
With `EOSMode.BETA` it is no longer necessary to use the same `transactional.id`, even for consumer-initiated transactions; in fact, it must be unique on each instance the same as producer-initiated transactions.
3483
-
For transactions started by the template (or the transaction manager for `@Transaction`) you should set the property on the template and transaction manager respectively.
3468
+
With `EOSMode.V2` (aka `BETA`), the only supported mode, it is no longer necessary to use the same `transactional.id`, even for consumer-initiated transactions; in fact, it must be unique on each instance the same as for producer-initiated transactions.
3484
3469
This property must have a different value on each application instance.
3485
3470
3486
-
This problem (different rules for `transactional.id`) has been eliminated when `EOSMode.BETA` is being used (with broker versions >= 2.5); see <<exactly-once>>.
3487
-
3488
3471
[[tx-template-mixed]]
3489
3472
===== `KafkaTemplate` Transactional and non-Transactional Publishing
3490
3473
@@ -3572,37 +3555,17 @@ Using transactions enables Exactly Once Semantics (EOS).
3572
3555
This means that, for a `read->process-write` sequence, it is guaranteed that the **sequence** is completed exactly once.
3573
3556
(The read and process are have at least once semantics).
3574
3557
3575
-
Spring for Apache Kafka version 2.5 and later supports two EOS modes:
3558
+
Spring for Apache Kafka version 3.0 and later only supports `EOSMode.V2`:
3576
3559
3577
-
* `ALPHA` - alias for `V1` (deprecated)
3578
-
* `BETA` - alias for `V2` (deprecated)
3579
-
* `V1` - aka `transactional.id` fencing (since version 0.11.0.0)
3580
3560
* `V2` - aka fetch-offset-request fencing (since version 2.5)
3581
3561
3582
-
With mode `V1`, the producer is "fenced" if another instance with the same `transactional.id` is started.
3583
-
Spring manages this by using a `Producer` for each `group.id/topic/partition`; when a rebalance occurs a new instance will use the same `transactional.id` and the old producer is fenced.
3562
+
IMPORTANT: This requires the brokers to be version 2.5 or later.
3584
3563
3585
3564
With mode `V2`, it is not necessary to have a producer for each `group.id/topic/partition` because consumer metadata is sent along with the offsets to the transaction and the broker can determine if the producer is fenced using that information instead.
3586
3565
3587
-
Starting with version 2.6, the default `EOSMode` is `V2`.
3588
-
3589
-
To configure the container to use mode `ALPHA`, set the container property `EOSMode` to `ALPHA`, to revert to the previous behavior.
3590
-
3591
-
IMPORTANT: With `V2` (default), your brokers must be version 2.5 or later; `kafka-clients` version 3.0, the producer will no longer fall back to `V1`; if the broker does not support `V2`, an exception is thrown.
3592
-
If your brokers are earlier than 2.5, you must set the `EOSMode` to `V1`, leave the `DefaultKafkaProducerFactory` `producerPerConsumerPartition` set to `true` and, if you are using a batch listener, you should set `subBatchPerPartition` to `true`.
3593
-
3594
-
When your brokers are upgraded to 2.5 or later, you should switch the mode to `V2`, but the number of producers will remain as before.
3595
-
You can then do a rolling upgrade of your application with `producerPerConsumerPartition` set to `false` to reduce the number of producers; you should also no longer set the `subBatchPerPartition` container property.
3596
-
3597
-
If your brokers are already 2.5 or newer, you should set the `DefaultKafkaProducerFactory` `producerPerConsumerPartition` property to `false`, to reduce the number of producers needed.
3598
-
3599
-
IMPORTANT: When using `EOSMode.V2` with `producerPerConsumerPartition=false` the `transactional.id` must be unique across all application instances.
3600
-
3601
-
When using `V2` mode, it is no longer necessary to set the `subBatchPerPartition` to `true`; it will default to `false` when the `EOSMode` is `V2`.
3602
-
3603
3566
Refer to https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.
3604
3567
3605
-
`V1` and `V2` were previously `ALPHA` and `BETA`; they have been changed to align the framework with https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2[KIP-732].
3568
+
`V2` was previously `BETA`; the `EOSMode` has been changed to align the framework with https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2[KIP-732].
3606
3569
3607
3570
[[interceptors]]
3608
3571
==== Wiring Spring Beans into Producer/Consumer Interceptors
Copy file name to clipboardExpand all lines: spring-kafka-docs/src/main/asciidoc/whats-new.adoc
+6-91Lines changed: 6 additions & 91 deletions
Original file line number
Diff line number
Diff line change
@@ -3,101 +3,16 @@
3
3
This section covers the changes made from version 2.7 to version 2.8.
4
4
For changes in earlier version, see <<history>>.
5
5
6
-
[[x28-kafka-client]]
6
+
[[x30-kafka-client]]
7
7
==== Kafka Client Version
8
8
9
9
This version requires the 3.1.0 `kafka-clients`
10
10
11
-
IMPORTANT: When using transactions, the minimum broker version is 2.5.
12
-
13
-
See <<exactly-once>> and https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.
14
-
15
-
[[x28-packages]]
16
-
==== Package Changes
17
-
18
-
Classes and interfaces related to type mapping have been moved from `...support.converter` to `...support.mapping`.
19
-
20
-
* `AbstractJavaTypeMapper`
21
-
* `ClassMapper`
22
-
* `DefaultJackson2JavaTypeMapper`
23
-
* `Jackson2JavaTypeMapper`
24
-
25
-
[[x28-ooo-commits]]
26
-
==== Out of Order Manual Commits
27
-
28
-
The listener container can now be configured to accept manual offset commits out of order (usually asynchronously).
29
-
The container will defer the commit until the missing offset is acknowledged.
30
-
See <<ooo-commits>> for more information.
31
-
32
-
[[x28-batch-overrude]]
33
-
==== `@KafkaListener` Changes
34
-
35
-
It is now possible to specify whether the listener method is a batch listener on the method itself.
36
-
This allows the same container factory to be used for both record and batch listeners.
37
-
38
-
See <<batch-listeners>> for more information.
39
-
40
-
Batch listeners can now handle conversion exceptions.
41
-
42
-
See <<batch-listener-conv-errors>> for more information.
43
-
44
-
`RecordFilterStrategy`, when used with batch listeners, can now filter the entire batch in one call.
45
-
See the note at the end of <<batch-listeners>> for more information.
46
-
47
-
The `@KafkaListener` annotation now has the `filter` attribute, to override the container factory's `RecordFilterStrategy` for just this listener.
48
-
49
-
The `@KafkaListener` annotation now has the `info` attribute; this is used to populate the new listener container property `listenerInfo`.
50
-
This is then used to populate a `KafkaHeaders.LISTENER_INFO` header in each record which can be used in `RecordInterceptor`, `RecordFilterStrategy`, or the listener itself.
51
-
See <<li-header>> and <<alc-props,Abstract Listener Container Properties>> for more information.
52
-
53
-
[[x28-template]]
54
-
==== `KafkaTemplate` Changes
11
+
[[x30-eos]]
12
+
==== Exactly Once Semantics
55
13
56
-
You can now receive a single record, given the topic, partition and offset.
57
-
See <<kafka-template-receive>> for more information.
14
+
`EOSMode.V1` (aka `ALPHA`) is no longer supported.
58
15
59
-
[[x28-eh]]
60
-
==== `CommonErrorHandler` Added
61
-
62
-
The legacy `GenericErrorHandler` and its sub-interface hierarchies for record an batch listeners have been replaced by a new single interface `CommonErrorHandler` with implementations corresponding to most legacy implementations of `GenericErrorHandler`.
63
-
See <<error-handlers>> and <<migrating-legacy-eh>> for more information.
64
-
65
-
[[x28-lcc]]
66
-
==== Listener Container Changes
67
-
68
-
The `interceptBeforeTx` container property is now `true` by default.
69
-
70
-
The `authorizationExceptionRetryInterval` property has been renamed to `authExceptionRetryInterval` and now applies to `AuthenticationException` s in addition to `AuthorizationException` s previously.
71
-
Both exceptions are considered fatal and the container will stop by default, unless this property is set.
72
-
73
-
See <<kafka-container>> and <<container-props>> for more information.
74
-
75
-
[[x28-serializers]]
76
-
==== Serializer/Deserializer Changes
77
-
78
-
The `DelegatingByTopicSerializer` and `DelegatingByTopicDeserializer` are now provided.
79
-
See <<delegating-serialization>> for more information.
80
-
81
-
[[x28-dlpr]]
82
-
==== `DeadLetterPublishingRecover` Changes
83
-
84
-
The property `stripPreviousExceptionHeaders` is now `true` by default.
85
-
86
-
There are now several techniques to customize which headers are added to the output record.
87
-
88
-
See <<dlpr-headers>> for more information.
89
-
90
-
[[x28-retryable-topics-changes]]
91
-
==== Retryable Topics Changes
92
-
93
-
Now you can use the same factory for retryable and non-retryable topics.
94
-
See <<retry-topic-lcf>> for more information.
95
-
96
-
There's now a manageable global list of fatal exceptions that will make the failed record go straight to the DLT.
97
-
Refer to <<retry-topic-ex-classifier>> to see how to manage it.
98
-
99
-
You can now use blocking and non-blocking retries in conjunction.
100
-
See <<retry-topic-combine-blocking>> for more information.
16
+
IMPORTANT: When using transactions, the minimum broker version is 2.5.
101
17
102
-
The KafkaBackOffException thrown when using the retryable topics feature is now logged at DEBUG level.
103
-
See <<change-kboe-logging-level>> if you need to change the logging level back to WARN or set it to any other level.
18
+
See <<exactly-once>> and https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics[KIP-447] for more information.
0 commit comments