Non-Blocking Retries cannot combine with container transactions.

[Wzy19930507]

Retry topic will break container txns atomicity

1. container txns commit success.
2. Message send to retryable topic retry again.

• Add adoc to notice.

See #2934

[sobychacko]

@ Wzy19930507 We need some more due diligence analysis on this issue. What you are saying is that retryable topics and nested transactions are not supported. Maybe the minimum we need to do is add a note in the doc saying that nested txns are not supported with retryable topics. I don't think we need to add the tests and the detailed docs you have in this PR.

We can probably take your test as a POC for analyzing the issue (at a later time). Could you re-work the PR to only include the note in the docs?

[Wzy19930507]

@sobychacko Thanks for your quick reply. I will re-work this PR.
Sorry for my an ambiguous description, let me see if I can describe the problem more precisely.

[artembilan]

Isn't that what expected from asynchronous, non-blocking retries?
To be able to send to the retry topic we indeed has to commit the current transaction.
I'm not fully sure how this non-blocking retires work internally, but apparently we don't start a new transaction, when we consume from the next retry topic.

It might work some way with a @Transactional on the @KafkaListener, but pay attention that @EnableTransactionManagement has to be activated.

Either way: I would not say in the docs break.
It sounds like a bug which is not supposed to be there.

It is just a feature which cannot be combined with transactions due to async nature.

Do you see things differently?

Thanks

[Wzy19930507]

Totally agree with you, i'll re-test the @Transactional case.

[Wzy19930507]

Test result is non-blocking retries cannot be combined with transactions.

---

@Transactional case as follows:
When PROPAGATION_REQUIRED will infinite retry because of container transactions rollback. (un-expected).
When PROPAGATION_REQUIRED_NEW

1. If listener code fail and container code success, listener code rollback, container code commit success. (expected)
2. If listener code fail and container code fail, listener code rollback, container code rollback. (expected)
3. If listener code success and container code fail, listener code commit success, container code rollback then infinite retry. (un-expected)
   Current KafkaTransactionManager only support PROPAGATION_REQUIRES_NEW and PROPAGATION_REQUIRED, the above case tests only these two cases.

PROPAGATION_REQUIRED_NEW un-expected Report see https://github.com/Wzy19930507/spring-kafka/tree/report_pr_3072

Please see if there are any omissions, thanks.

---

I'll modify docs to Non-blocking retries cannot be combined with container transactions first and see if there's a way to implement it.

[sobychacko]

@Wzy19930507 We might want to use present tense here: When the listener code throws an exception, container transaction commit succeeds, and the record is sent to the retryable topic.

[Wzy19930507]

Thanks for the correction, fix it.

[sobychacko]

@Wzy19930507 We might want to use present tense here: When the listener code throws an exception, container transaction commit succeeds, and the record is sent to the retryable topic.