Refactor code using Elasticsearch libs.

Original Pull Request spring-projects#2196
Closes spring-projects#2157

Author: sothawo

src/main/asciidoc/preface.adoc

@@ -37,6 +37,7 @@ built and tested.
 [cols="^,^,^,^,^",options="header"]
 |===
 | Spring Data Release Train | Spring Data Elasticsearch | Elasticsearch | Spring Framework | Spring Boot
+| 2022.0 (Turing) | 5.0.x | 8.2.2 | 6.0.x | 3.0.x?
 | 2021.2 (Raj) | 4.4.x | 7.17.3 | 5.3.x | 2.7.x
 | 2021.1 (Q) | 4.3.x | 7.15.2 | 5.3.x | 2.6.x
 | 2021.0 (Pascal) | 4.2.xfootnote:oom[Out of maintenance] | 7.12.0 | 5.3.x | 2.5.x

src/main/asciidoc/reference/elasticsearch-migration-guide-4.4-5.0.adoc

@@ -0,0 +1,144 @@
+[[elasticsearch-migration-guide-4.4-5.0]]
+= Upgrading from 4.4.x to 5.0.x
+
+This section describes breaking changes from version 4.4.x to 5.0.x and how removed features can be replaced by new 
+introduced features.
+
+[[elasticsearch-migration-guide-4.4-4.5.deprecations]]
+== Deprecations
+
+=== `org.springframework.data.elasticsearch.client.erhlc` package
+
+See <<elasticsearch-migration-guide-4.4-5.0.breaking-changes-packages>>, all classes in this package have been 
+deprecated, as the default client implementations to use are the ones based on the new Java Client from 
+Elasticsearch, se <<elasticsearch-migration-guide-4.4-5.0.new-clients>>
+
+[[elasticsearch-migration-guide-4.4-5.0.breaking-changes]]
+== Breaking Changes
+
+=== Removal of deprecated calls
+
+==== suggest calls in operations interfaces have been removed
+
+Both `SearchOperations` and `ReactiveSearchOperations` had deprecated calls that were using Elasticsearch classes as 
+parameters. These now have been removed and so the dependency on Elasticsearch classes in these APIs has been cleaned.
+
+[[elasticsearch-migration-guide-4.4-5.0.breaking-changes-packages]]
+=== Package changes
+
+All the classes that are using or depend on the deprecated Elasticsearch `RestHighLevelClient` have been moved to the 
+package `org.springframework.data.elasticsearch.client.erhlc`. By this change we now have a clear separation of code 
+using the old deprecated Elasticsearch libraries, code using the new Elasticsearch client and code that is 
+independent of the client implementation. Also the reactive implementation that was provided up to now has been moved
+here, as this implementation contains code that was copied and adapted from Elasticsearch libraries.
+
+
+[[elasticsearch-migration-guide-4.4-5.0.new-clients]]
+== New Elasticsearch client
+
+Spring Data Elasticsearch  now uses the new `ElasticsearchClient` and has 
+deprecated the use of the previous `RestHighLevelClient`.
+
+=== How to use the new client
+
+In order to use the new client the following steps are necessary:
+
+==== Add dependencies
+
+The dependencies for the new Elasticsearch client are still optional in Spring Data Elasticsearch so they need to be added explicitly:
+
+====
+[source,xml]
+----
+<dependencies>
+    <dependency>
+        <groupId>co.elastic.clients</groupId>
+        <artifactId>elasticsearch-java</artifactId>
+        <version>7.17.3</version>
+        <exclusions>
+            <exclusion>
+                <groupId>commons-logging</groupId>
+                <artifactId>commons-logging</artifactId>
+            </exclusion>
+        </exclusions>
+    </dependency>
+    <dependency>
+        <groupId>org.elasticsearch.client</groupId>
+        <artifactId>elasticsearch-rest-client</artifactId> <!-- is Apache 2-->
+        <version>7.17.3</version>
+        <exclusions>
+            <exclusion>
+                <groupId>commons-logging</groupId>
+                <artifactId>commons-logging</artifactId>
+            </exclusion>
+        </exclusions>
+    </dependency>
+</dependencies>
+----
+====
+
+When using Spring Boot, it is necessary to set the following property in the _pom.xml_.
+
+====
+[source,xml]
+----
+<properties>
+    <jakarta-json.version>2.0.1</jakarta-json.version>
+</properties>
+----
+====
+
+==== New configuration classes
+
+===== Imperative style
+
+In order configure Spring Data Elasticsearch to use the new client, it is necessary to create a configuration bean that derives from `org.springframework.data.elasticsearch.client.elc.ElasticsearchConfiguration`:
+
+====
+[source,java]
+----
+@Configuration
+public class NewRestClientConfig extends ElasticsearchConfiguration {
+
+	@Override
+	public ClientConfiguration clientConfiguration() {
+		return ClientConfiguration.builder() //
+			.connectedTo("localhost:9200") //
+			.build();
+	}
+}
+----
+====
+
+The configuration is done in the same way as with the old client, but it is not necessary anymore to create more than the configuration bean.
+With this configuration, the following beans will be available in the Spring application context:
+
+* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
+* an `ElasticsearchClient` bean, this is the new client that uses the `RestClient`
+* an `ElasticsearchOperations` bean, available with the bean names _elasticsearchOperations_ and _elasticsearchTemplate_, this uses the `ElasticsearchClient`
+
+===== Reactive style
+
+To use the new client in a reactive environment the only difference is the class from which to derive the configuration:
+
+====
+[source,java]
+----
+@Configuration
+public class NewRestClientConfig extends ReactiveElasticsearchConfiguration {
+
+	@Override
+	public ClientConfiguration clientConfiguration() {
+		return ClientConfiguration.builder() //
+			.connectedTo("localhost:9200") //
+			.build();
+	}
+}
+----
+====
+
+With this configuration, the following beans will be available in the Spring application context:
+
+* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
+* an `ReactiveElasticsearchClient` bean, this is the new reactive client that uses the `RestClient`
+* an `ReactiveElasticsearchOperations` bean, available with the bean names _reactiveElasticsearchOperations_ and _reactiveElasticsearchTemplate_, this uses the `ReactiveElasticsearchClient`

src/main/asciidoc/reference/elasticsearch-new.adoc

@@ -1,6 +1,12 @@
 [[new-features]]
 = What's new
 
+[[new-features.5-0-0]]
+== New in Spring Data Elasticsearch 5.0
+
+* Upgrade to Java 17 baseline
+* Upgrade to Spring Framework 6
+
 [[new-features.4-4-0]]
 == New in Spring Data Elasticsearch 4.4
 

src/main/asciidoc/reference/migration-guides.adoc

@@ -12,4 +12,7 @@ include::elasticsearch-migration-guide-4.1-4.2.adoc[]
 include::elasticsearch-migration-guide-4.2-4.3.adoc[]
 
 include::elasticsearch-migration-guide-4.3-4.4.adoc[]
+
+include::elasticsearch-migration-guide-4.4-5.0.adoc[]
+
 :leveloffset: -1

src/main/java/org/springframework/data/elasticsearch/client/ClientConfiguration.java

@@ -27,7 +27,8 @@
 import javax.net.ssl.SSLContext;
 
 import org.elasticsearch.client.RestClientBuilder.HttpClientConfigCallback;
-import org.springframework.data.elasticsearch.client.reactive.ReactiveRestClients;
+import org.springframework.data.elasticsearch.client.erhlc.ReactiveRestClients;
+import org.springframework.data.elasticsearch.client.erhlc.RestClients;
 import org.springframework.http.HttpHeaders;
 import org.springframework.lang.Nullable;
 import org.springframework.web.reactive.function.client.WebClient;
@@ -68,8 +69,8 @@ static ClientConfiguration localhost() {
 	}
 
 	/**
-	 * Creates a new {@link ClientConfiguration} instance configured to a single host given {@code hostAndPort}.
-	 * For example given the endpoint http://localhost:9200
+	 * Creates a new {@link ClientConfiguration} instance configured to a single host given {@code hostAndPort}. For
+	 * example given the endpoint http://localhost:9200
 	 *
 	 * <pre class="code">
 	 * ClientConfiguration configuration = ClientConfiguration.create("localhost:9200");
@@ -82,8 +83,8 @@ static ClientConfiguration create(String hostAndPort) {
 	}
 
 	/**
-	 * Creates a new {@link ClientConfiguration} instance configured to a single host given {@link InetSocketAddress}.
-	 * For example given the endpoint http://localhost:9200
+	 * Creates a new {@link ClientConfiguration} instance configured to a single host given {@link InetSocketAddress}. For
+	 * example given the endpoint http://localhost:9200
 	 *
 	 * <pre class="code">
 	 * ClientConfiguration configuration = ClientConfiguration

src/main/java/org/springframework/data/elasticsearch/client/ClientConfigurationBuilder.java

@@ -31,7 +31,8 @@
 import org.springframework.data.elasticsearch.client.ClientConfiguration.ClientConfigurationBuilderWithRequiredEndpoint;
 import org.springframework.data.elasticsearch.client.ClientConfiguration.MaybeSecureClientConfigurationBuilder;
 import org.springframework.data.elasticsearch.client.ClientConfiguration.TerminalClientConfigurationBuilder;
-import org.springframework.data.elasticsearch.client.reactive.ReactiveRestClients;
+import org.springframework.data.elasticsearch.client.erhlc.ReactiveRestClients;
+import org.springframework.data.elasticsearch.client.erhlc.RestClients;
 import org.springframework.http.HttpHeaders;
 import org.springframework.lang.Nullable;
 import org.springframework.util.Assert;

src/main/java/org/springframework/data/elasticsearch/client/elc/JsonUtils.java

@@ -20,9 +20,11 @@
 
 import java.io.ByteArrayOutputStream;
 import java.io.UnsupportedEncodingException;
+import java.nio.charset.StandardCharsets;
 
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
+import org.springframework.lang.Nullable;
 
 /**
  * @author Peter-Josef Meisch
@@ -41,13 +43,28 @@ public static String toJson(Object object, JsonpMapper mapper) {
 		JsonGenerator generator = mapper.jsonProvider().createGenerator(baos);
 		mapper.serialize(object, generator);
 		generator.close();
-		String jsonMapping = "{}";
+		String json = "{}";
 		try {
-			jsonMapping = baos.toString("UTF-8");
+			json = baos.toString("UTF-8");
 		} catch (UnsupportedEncodingException e) {
 			LOGGER.warn("could not read json", e);
 		}
 
-		return jsonMapping;
+		return json;
 	}
+
+	@Nullable
+	public static String queryToJson(@Nullable co.elastic.clients.elasticsearch._types.query_dsl.Query query, JsonpMapper mapper) {
+
+		if (query == null) {
+			return null;
+		}
+
+		var baos = new ByteArrayOutputStream();
+		var generator = mapper.jsonProvider().createGenerator(baos);
+		query.serialize(generator, mapper);
+		generator.close();
+		return baos.toString(StandardCharsets.UTF_8);
+	}
+
 }

src/main/java/org/springframework/data/elasticsearch/client/elc/ReactiveClusterTemplate.java

@@ -19,8 +19,8 @@
 import co.elastic.clients.elasticsearch.cluster.HealthResponse;
 import reactor.core.publisher.Mono;
 
+import org.springframework.data.elasticsearch.client.erhlc.ReactiveClusterOperations;
 import org.springframework.data.elasticsearch.core.cluster.ClusterHealth;
-import org.springframework.data.elasticsearch.core.cluster.ReactiveClusterOperations;
 import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
 
 /**

src/main/java/org/springframework/data/elasticsearch/client/elc/ReactiveElasticsearchTemplate.java

@@ -39,14 +39,14 @@
 import org.springframework.data.elasticsearch.NoSuchIndexException;
 import org.springframework.data.elasticsearch.UncategorizedElasticsearchException;
 import org.springframework.data.elasticsearch.client.UnsupportedBackendOperation;
+import org.springframework.data.elasticsearch.client.erhlc.ReactiveClusterOperations;
 import org.springframework.data.elasticsearch.client.util.ScrollState;
 import org.springframework.data.elasticsearch.core.AbstractReactiveElasticsearchTemplate;
 import org.springframework.data.elasticsearch.core.AggregationContainer;
 import org.springframework.data.elasticsearch.core.IndexedObjectInformation;
 import org.springframework.data.elasticsearch.core.MultiGetItem;
 import org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations;
 import org.springframework.data.elasticsearch.core.ReactiveIndexOperations;
-import org.springframework.data.elasticsearch.core.cluster.ReactiveClusterOperations;
 import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
 import org.springframework.data.elasticsearch.core.document.Document;
 import org.springframework.data.elasticsearch.core.document.SearchDocument;

src/main/java/org/springframework/data/elasticsearch/client/elc/SearchDocumentResponseBuilder.java

@@ -29,7 +29,6 @@
 
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
-import org.elasticsearch.search.SearchHits;
 import org.springframework.data.elasticsearch.core.TotalHitsRelation;
 import org.springframework.data.elasticsearch.core.document.SearchDocument;
 import org.springframework.data.elasticsearch.core.document.SearchDocumentResponse;
@@ -76,11 +75,11 @@ public static <T> SearchDocumentResponse from(ResponseBody<EntityAsMap> response
 	}
 
 	/**
-	 * creates a {@link SearchDocumentResponseBuilder} from {@link SearchHits} with the given scrollId aggregations and
+	 * creates a {@link SearchDocumentResponseBuilder} from {@link HitsMetadata} with the given scrollId aggregations and
 	 * suggestES
 	 *
 	 * @param <T> entity type
-	 * @param hitsMetadata the {@link SearchHits} to process
+	 * @param hitsMetadata the {@link HitsMetadata} to process
 	 * @param scrollId scrollId
 	 * @param aggregations aggregations
 	 * @param suggestES the suggestion response from Elasticsearch

src/main/java/org/springframework/data/elasticsearch/config/AbstractElasticsearchConfiguration.java renamed to src/main/java/org/springframework/data/elasticsearch/client/erhlc/AbstractElasticsearchConfiguration.java

@@ -13,20 +13,22 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.springframework.data.elasticsearch.config;
+package org.springframework.data.elasticsearch.client.erhlc;
 
 import org.elasticsearch.client.RestHighLevelClient;
 import org.springframework.context.annotation.Bean;
+import org.springframework.data.elasticsearch.config.ElasticsearchConfigurationSupport;
 import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
-import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
 import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
 
 /**
  * @author Christoph Strobl
  * @author Peter-Josef Meisch
  * @since 3.2
  * @see ElasticsearchConfigurationSupport
+ * @deprecated since 5.0
  */
+@Deprecated
 public abstract class AbstractElasticsearchConfiguration extends ElasticsearchConfigurationSupport {
 
 	/**

src/main/java/org/springframework/data/elasticsearch/config/AbstractReactiveElasticsearchConfiguration.java renamed to src/main/java/org/springframework/data/elasticsearch/client/erhlc/AbstractReactiveElasticsearchConfiguration.java

@@ -13,13 +13,12 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.springframework.data.elasticsearch.config;
+package org.springframework.data.elasticsearch.client.erhlc;
 
 import org.elasticsearch.action.support.IndicesOptions;
 import org.springframework.context.annotation.Bean;
-import org.springframework.data.elasticsearch.client.reactive.ReactiveElasticsearchClient;
+import org.springframework.data.elasticsearch.config.ElasticsearchConfigurationSupport;
 import org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations;
-import org.springframework.data.elasticsearch.core.ReactiveElasticsearchTemplate;
 import org.springframework.data.elasticsearch.core.RefreshPolicy;
 import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
 import org.springframework.lang.Nullable;
@@ -29,7 +28,9 @@
  * @author Peter-Josef Meisch
  * @since 3.2
  * @see ElasticsearchConfigurationSupport
+ * @deprecated since 5.0
  */
+@Deprecated
 public abstract class AbstractReactiveElasticsearchConfiguration extends ElasticsearchConfigurationSupport {
 
 	/**

src/main/java/org/springframework/data/elasticsearch/core/CriteriaFilterProcessor.java renamed to src/main/java/org/springframework/data/elasticsearch/client/erhlc/CriteriaFilterProcessor.java

@@ -13,7 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.springframework.data.elasticsearch.core;
+package org.springframework.data.elasticsearch.client.erhlc;
 
 import static org.springframework.data.elasticsearch.core.query.Criteria.*;
 
@@ -47,7 +47,9 @@
  * @author Mohsin Husen
  * @author Artur Konczak
  * @author Peter-Josef Meisch
+ * @deprecated since 5.0
  */
+@Deprecated
 class CriteriaFilterProcessor {
 
 	@Nullable

src/main/java/org/springframework/data/elasticsearch/core/CriteriaQueryProcessor.java renamed to src/main/java/org/springframework/data/elasticsearch/client/erhlc/CriteriaQueryProcessor.java

@@ -13,7 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.springframework.data.elasticsearch.core;
+package org.springframework.data.elasticsearch.client.erhlc;
 
 import static org.elasticsearch.index.query.Operator.*;
 import static org.elasticsearch.index.query.QueryBuilders.*;
@@ -44,7 +44,9 @@
  * @author Rasmus Faber-Espensen
  * @author James Bodkin
  * @author Peter-Josef Meisch
+ * @deprecated since 5.0
  */
+@Deprecated
 class CriteriaQueryProcessor {
 
 	@Nullable