Docuement id is not generated as expected when upgrade to Spring Boot 3.2.x

[hantsy]

I tried to update my example projects to the latest Spring 6.1.x, Spring Boot 3.2 and Java 21.

The Spring Data ElasticSearch example failed, the example source codes is here, https://github.com/hantsy/spring6-sandbox/tree/master/boot-data-elasticsearch

@Document(indexName = "products")
public record Product(@Id String id, String name, BigDecimal price) {
}

And test is like :

@DataElasticsearchTest
@Testcontainers
@Slf4j
public class ElasticsearchIntegrationTests {

    @Container
    @ServiceConnection
    public static ElasticsearchContainer ES_CONTAINER = new ElasticsearchContainer("docker.elastic.co/elasticsearch/elasticsearch:7.17.9");

    @Autowired
    private ProductRepository productRepository;

    @Test
    void testDatabaseIsRunning() {
        assertThat(ES_CONTAINER.isRunning()).isTrue();
    }

    @Test
    public void testProductRepository() {
        var product = productRepository.save(new Product(null, "test", BigDecimal.ONE));
        assertThat(product).isNotNull();
        assertThat(product.id()).isNotNull();

        productRepository.findById(product.id()).ifPresent(
            p -> {
                log.debug("found product by id: {}", p);
                assertThat(p.name()).isEqualTo("test");
            }
        );
    }
}

And got the following error.

2023-09-24T20:38:24.405+08:00 DEBUG 17172 --- [           main] .c.s.DirtiesContextTestExecutionListener : After test method: class [ElasticsearchIntegrationTests], method [testProductRepository], class annotated with @DirtiesContext [false] with mode [null], method annotated with @DirtiesContext [false] with mode [null]

java.lang.AssertionError: 
Expecting actual not to be null

	at com.example.demo.ElasticsearchIntegrationTests.testProductRepository(ElasticsearchIntegrationTests.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

It means the entity id is not filled as expected.

This example worked with Spring Boot 3.1.x and Java 17.

[sothawo]

Please provide a complete runnable example showing this behaviour.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[sothawo]

I just had another look at the repository you linked. If you want us to look into this, provide a standalone runnable example, not a GitHub repository containing lots of different spring data subdirectories which are irrelevant for this problem.

The I don't find any code configuring the Spring Data Elasticsearch client. The test is starting an Elasticsearch TestContainer in version 7.17.9, Spring Data Elasticsearch 5.2 is built against Elasticsearch 8, so this might only work with Elasticsearch compatibility headers set, but I do not see that they are set anywhere.

When changing to new versions you should adapt your code/configurations to changes in the used libraries.

[hantsy]

boot-data-elasticsearch.zip

Updated to ES8, and run the test will get the failed message.

[sothawo]

I tried to run the test from the zip file

1. The mvnw file has Windows linebreaks and cannot be run on Linux/macOs without fixing it.
2. I had to uncomment the commented out code in the setup exposing the @DynamicPropertySource
3. The whole Spring application startup fails on the first attempt of accessing the Elasticsearch server, it doesn't even get to a point where a test is executed.
4. The reason for this is that the Elasticsearch server running in testcontainers is starting a HTTPS endpoint, which on your side is not configured.

There's a great article by Alex Reelsen about Elasticsearch 8 and TestContainers (https://spinscale.de/posts/2022-02-17-running-elasticsearch-8-with-testcontainers.html)

This whole setup is no Spring Data Elasticsearch problem anyway, it is a test setup / Spring Boot configuration problem.

The official way to configure Spring Data Elasticsearch is found in the documentation (https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients). If you rely on Spring Boot autoconfiguration then that is a Spring Boot problem and this is the wrong place to address it, check https://github.com/spring-projects/spring-boot for that.

[hantsy]

Maybe it is an autoconfiguration issue of Spring boot testcontainers, it was working in Spring Boot 3.1.

[hantsy]

The purpose of this sample project is testing spring-boot-testcontainers to replace the manual testcontainers config which requires @DynamicPropertySource.

[hantsy]

@sothawo Check this comment on Spring Boot spring-projects/spring-boot#37672

I have tried to update my another example project to Spring Boot 3.1.4, it also encountered failure, check the testing codes, https://github.com/hantsy/spring-reactive-sample/blob/master/boot-data-elasticsearch/src/test/java/com/example/demo/PostRepositoryWithTestContainersTest.java,

@Test
void testLoadUsers() {
    this.posts.findAll(Sort.by(Sort.Direction.ASC, "title"))
            .log()
            .as(StepVerifier::create)
            .consumeNextWith(user -> assertThat(user.getTitle()).isEqualTo("Post one"))
            .consumeNextWith(user -> assertThat(user.getTitle()).isEqualTo("Post two"))
            //.expectNextCount(2)
            .verifyComplete();
}

The consumeNextWith encountered a onComplete event instead, which means the data I tried to save in the @BeforeEach is not found.

[sothawo]

Like commented in spring-projects/spring-boot#37672 (comment)

If you want to get further, I would suggest refining the sample to configure Spring Data Elasticsearch without Boot auto-configuration #2709 (comment) and create a new issue in Spring Data Elasticsearch.

Provide a reproducible runnable example that configures Spring Data Elasticsearch independently from Spring Boot and that shows that error.

[hantsy]

@sothawo OK, created another None SpringBoot project sample to reproduce the issue, based on Spring Boot 3.1.4 and Spring WebFlux Stack.

data-elasticsearch.zip

[sothawo]

Ok, I had a look at your test setup and the test

The first thing I changed for my tests is to add an intercepting proxy on port 8080 (you can use ZAP or Burp for this); this needs adapting the configuration

@Override
public ClientConfiguration clientConfiguration() {
		return ClientConfiguration.builder()
						.connectedTo(elasticsearchUri)
						.withProxy("localhost:8080")
						.build();
}

You have a timing problem in your test setup / test code. When looking at the proxy output you can see that the search starts before the refreshes from the previous saves are done. Spring Data Elasticsearch repositories by default do a refresh request after an save if nothing else is configured. Your test setup code doesn't wait before returning, the logs show

13:26:19.795 [main] DEBUG com.example.demo.PostRepositoryTest - sample data is ready for tests
13:26:20.346 [I/O dispatcher 2] DEBUG com.example.demo.PostRepositoryTest - saved post: Post(id=ubjjCYsBjxIb6bh6ZaZh, title=Post two, content=content of Post two)
13:26:20.351 [I/O dispatcher 2] DEBUG com.example.demo.PostRepositoryTest - saved post: Post(id=uLjjCYsBjxIb6bh6ZaZN, title=Post one, content=content of Post one)

The setup runs on the main thread, the saves on a different one.
The corresponding proxy entries are:

• First we have the two saves (180, 181)
• the search creates a point in time snapshot before the index is refreshed, so the new saves are missed (182)
• the two refresh call go to the index but do not affect the pit (183, 184)
• the search runs on the pit which does not contain the saved data (185)
• pit is closed (186)

Why this worked before? I don't know there might be a change anywhere in the reactive Spring code or in reactor, but there was no change concerning this in Spring Data Elasticsearch.

modified test setup method

With this modified method, the save, refresh and search calls are in order:

@BeforeEach
void setUp() {
    this.posts.deleteAll()
            .thenMany(
                    Flux.just("Post one", "Post two")
                            .flatMap(
                                    title ->
                                            posts.save(Post.builder().title(title).content("content of " + title).build())
                            )
            )
            .map(p -> {
                log.debug("saved post: {}", p);
                return p;
            })
            .blockLast();
    log.debug("sample data is ready for tests");
}

change default refresh policy

You can get rid of the extra refresh calls and have the refresh parameter set on the save call directly by adding this method to the ElasticsearchConfig class:

@Override
protected RefreshPolicy refreshPolicy() {
		return RefreshPolicy.IMMEDIATE;
}

This affects then every repository and ElasticsearchOperations call and might not be what is wanted.

adjust the flaky test

Your test method expects the result to be in the order "Post one", "Post two". A call to findAll() without specifying an order does not guarantee an order of the returned data. It might be in the order that data was inserted, it might not. This btw is the case for every datastore, Elastic, Mongo, or a relational database. I adjusted the test method:

@Test
public void testGetAllPosts() {
    this.posts.findAll()
            .map(Post::getTitle)
            .collectList()
            .as(StepVerifier::create)
            .consumeNextWith(titleList ->
                    assertThat(titleList)
                            .containsExactlyInAnyOrder("Post one", "Post two"))
            .verifyComplete();
}

With these changes the tests are running fine.

[hantsy]

1. The blockLast in the before setup method, I also used this approach when came to the WebFlux world, but in some cases in before projects, the block will throw timeout exceptions. So now I would like to use a latch counter instead.
2. I am not sure RefreshPolicy will affect the saving process, it cause the saved items are not shown immediately?
3. If changing the save via ElasticSearchTemplate, it will resolve the issues?

[hantsy]

A little confused about the test method. Your modified test block is working well.

But I tried to use the following instead.

this.posts.findAll(Sort.by(Sort.Order.asc("title"))) // add order in query.
      .as(StepVerifier::create)
      .consumeNextWith(p -> assertThat(p.getTitle()).contains("one"))
      .consumeNextWith(p -> assertThat(p.getTitle()).contains("two"))
      .expectComplete()
      .verify();

And got the exception:

12:05:30.497 [I/O dispatcher 1] ERROR org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchTemplate -- Error during pit/search_after
org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
	at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:102)
	at org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchTemplate.translateException(ReactiveElasticsearchTemplate.java:673)
	at reactor.core.publisher.Flux.lambda$onErrorMap$27(Flux.java:7201)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:122)
	at reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:71)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at co.elastic.clients.transport.rest_client.RestClientTransport$1.onSuccess(RestClientTransport.java:185)
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:676)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:399)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:393)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.client.InternalRequestExecutor.inputReady(InternalRequestExecutor.java:83)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:334)
	at co.elastic.clients.transport.rest_client.RestClientTransport.access$200(RestClientTransport.java:70)
	at co.elastic.clients.transport.rest_client.RestClientTransport$1.onSuccess(RestClientTransport.java:181)
	... 19 common frames omitted

[hantsy]

Almost same codes with Mongo, it is working well, https://github.com/hantsy/spring-reactive-sample/blob/master/data-mongo/src/test/java/com/example/demo/PostRepositoryTest.java

[hantsy]

It seems all ElasticSearch Repository save issues is related to changing the default refresh policy to null.

[sothawo]

It seems all ElasticSearch Repository save issues is related to changing the default refresh policy to null.

That change is from 07.03.2022, 1 and a half years ago, introduced in 4.4-RC1 and is documented (https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch-migration-guide-4.3-4.4.breaking-changes.3)

If that change is relevant for your test it means, you did not test the changes in 4.4, 5.0 or 5.1.

[hantsy]

In the newer Spring 6 sandbox, the Spring Data Elasticsearch example was working well with Spring Boot 3.1, https://github.com/hantsy/spring6-sandbox/commits/master/boot-data-elasticsearch, and the test example is similar to the above comments, https://github.com/hantsy/spring6-sandbox/blob/05454334713980fcbbed61cedb4aca50a2d41d31/boot-data-elasticsearch/src/test/java/com/example/demo/ElasticsearchIntegrationTests.java, the main difference in the new Spring 6 sandbox projects, I tried to use the new Java features, so it used record as document class instead of traditional POJO class. The test is broken when updated to Spring Boot 3.2.0-M3, this is really confused me.

Thanks for your help @sothawo .

[hantsy]

I realized a big difference between my two example projects, it seems RefreshPoplicy affects the Reactive APIs for a couple of months, even the stable Spring Boot 3.1(as mentioned in above comments, still failed with Spring Boot autoconfiguration).

And the blocking API only failed when upgraded to Spring Boot 3.2.0-M3 which worked with Spring Boot 3.1, as the first post here.

[hantsy]

@sothawo Please reopen this issue, for the blocking API(None reactive), it does not work with the latest Spring 6.1.0-RC2/Spring Boot 3.2.0-RC2.

Check the example project: data-elasticsearch and data-elasticsearch(Spring Boot)

[hantsy]

In the above preparing the example project, I used the Reactive API, the encountered refresh policy is another issue. The original issue is not resolved.

[hantsy]

Compare to the example's codes provided by @eddumelendez, spring-projects/spring-boot#35926 (comment) I just used record as document definition instead of the POJO.

And ref: spring-projects/spring-boot#37672 (comment)

[sothawo]

I cloned your project and ran the test application in the data_elasticsearch directory. This gives me the following output:

2023-11-07 12:55:56,428 DEBUG [main] c.e.d.DataInitializer: start data initialization...
2023-11-07 12:55:57,083 DEBUG [main] c.e.d.DataInitializer: saved product: Product[id=f3qjqYsBB4iipurLprHO, name=Apple, price=10.0]
2023-11-07 12:55:57,091 DEBUG [main] c.e.d.DataInitializer: saved product: Product[id=gHqjqYsBB4iipurLprHQ, name=Orange, price=1.0]
2023-11-07 12:55:57,092 DEBUG [main] c.e.d.DataInitializer: done data initialization...
2023-11-07 12:55:57,160 DEBUG [main] c.e.d.Application: get the initial products: Product[id=f3qjqYsBB4iipurLprHO, name=Apple, price=10.0]
2023-11-07 12:55:57,160 DEBUG [main] c.e.d.Application: get the initial products: Product[id=gHqjqYsBB4iipurLprHQ, name=Orange, price=1.0]
... the end...

The ids generated by Elasticsearch are there, so there is no issue here.

[hantsy]

But the tests failed at id assertion when upgrading to Spring Boot 3.2.x

[hantsy]

https://github.com/hantsy/spring6-sandbox/blob/master/data-elasticsearch/src/test/java/com/example/demo/ProductRepositoryTest.java#L69

Check the exception stack in Github actions, https://github.com/hantsy/spring6-sandbox/actions/runs/6766720093/job/18388213394#step:4:201.

Error:  com.example.demo.ProductRepositoryTest.testProductRepository -- Time elapsed: 0.307 s <<< FAILURE!
java.lang.AssertionError: 

Expecting actual not to be null
	at com.example.demo.ProductRepositoryTest.testProductRepository(ProductRepositoryTest.java:69)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[hantsy]

Similarly, check the exception stack for the Spring Boot version, https://github.com/hantsy/spring6-sandbox/actions/runs/6708664577/job/18229964179 @sothawo

[hantsy]

As mentioned, the boot example projects was working with Spring Boot 3.1.x and Java 17. This example was created for experiencing Spring 6 and Spring Boot 3.0, I just did a simple upgrade to test new versions.

[sothawo]

The only relevant information that would have been needed to find what is wrong was

I just used record as document definition instead of the POJO.

And that's why I asked to provide a minimal reproducing example. And minimal does not mean with the least effort by your side. By providing a repository with lots of subprojects, switching from reactive to imperative and mixing versions of boot and Elasticsearch (your code uses SDE 5 but in your tests you start an ES 7 instance), this just provides too many things that might go wrong.

The minimal example would have been about 5 lines of configuration and a 2 line test saving and asserting an entity.

The bug was introduced by a change in spring-data-commons and is fixed in #2757.

[hantsy]

And that's why I asked to provide a minimal reproducing example.

I was always providing a min project with a couple of lines.

By providing a repository with lots of subprojects

You can pick up to download. I just use a repository to categorize my sample projects, if flatten into standalone repos, there are thousands of repos, for me, it is terrible to maintain.

And I never thought Spring Data Elasticseach was so different in blocking and reactive.