Skip to content

Commit 1fb034a

Browse files
sothawosothawo
authored
Scripted and runtime fields improvements.
Original Pull Request #2663 Closes #2035
1 parent 82ae118 commit 1fb034a

33 files changed

+1404
-299
lines changed

‎src/main/asciidoc/reference/elasticsearch-migration-guide-5.1-5.2.adoc

+7
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ This section describes breaking changes from version 5.1.x to 5.2.x and how remo
66
[[elasticsearch-migration-guide-5.1-5.2.breaking-changes]]
77
== Breaking Changes
88

9+
=== Bulk failures
910
In the `org.springframework.data.elasticsearch.BulkFailureException` class, the return type of the `getFailedDocuments` is changed from `Map<String, String>`
1011
to `Map<String, FailureDetails>`, which allows to get additional details about failure reasons.
1112

@@ -14,6 +15,12 @@ The definition of the `FailureDetails` class (inner to `BulkFailureException`):
1415
public record FailureDetails(Integer status, String errorMessage) {
1516
}
1617

18+
=== scripted and runtime fields
19+
20+
The classes `org.springframework.data.elasticsearch.core.RuntimeField` and `org.springframework.data.elasticsearch.core.query.ScriptType` have been moved to the subpackage `org.springframework.data.elasticsearch.core.query`.
21+
22+
The `type` parameter of the `ScriptData` constructir is not nullable any longer.
23+
1724
[[elasticsearch-migration-guide-5.1-5.2.deprecations]]
1825
== Deprecations
1926

‎src/main/asciidoc/reference/elasticsearch-misc-scripted-and-runtime-fields.adoc

+223
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
[[elasticsearch.misc.scripted-and-runtime-fields]]
2+
= Scripted and runtime fields
3+
4+
Spring Data Elasticsearch supports scripted fields and runtime fields.
5+
Please refer to the Elasticsearch documentation about scripting (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) and runtime fields (https://www.elastic.co/guide/en/elasticsearch/reference/8.9/runtime.html) for detailed information about this.
6+
In the context of Spring Data Elasticsearch you can use
7+
8+
* scripted fields that are used to return fields that are calculated on the result documents and added to the returned document.
9+
* runtime fields that are calculated on the stored documents and can be used in a query and/or be returned in the search result.
10+
11+
The following code snippets will show what you can do (this show imperative code, but the reactive implementation works similar).
12+
13+
== The person entity
14+
15+
The enity that is used in these examples is a `Person` entity.
16+
This entity has a `birthDate` and an `age` property.
17+
Whereas the birthdate is fix, the age depends on the time when a query is issued and needs to be calculated dynamically.
18+
19+
====
20+
[source,java]
21+
----
22+
import org.springframework.data.annotation.Id;
23+
import org.springframework.data.elasticsearch.annotations.DateFormat;
24+
import org.springframework.data.elasticsearch.annotations.Document;
25+
import org.springframework.data.elasticsearch.annotations.Field;
26+
import org.springframework.data.elasticsearch.annotations.ScriptedField;
27+
import org.springframework.lang.Nullable;
28+
29+
import java.time.LocalDate;
30+
import java.time.format.DateTimeFormatter;
31+
32+
import static org.springframework.data.elasticsearch.annotations.FieldType.*;
33+
34+
import java.lang.Integer;
35+
36+
@Document(indexName = "persons")
37+
public record Person(
38+
@Id
39+
@Nullable
40+
String id,
41+
@Field(type = Text)
42+
String lastName,
43+
@Field(type = Text)
44+
String firstName,
45+
@Field(type = Keyword)
46+
String gender,
47+
@Field(type = Date, format = DateFormat.basic_date)
48+
LocalDate birthDate,
49+
@Nullable
50+
@ScriptedField Integer age <.>
51+
) {
52+
public Person(String id,String lastName, String firstName, String gender, String birthDate) {
53+
this(id, <.>
54+
lastName,
55+
firstName,
56+
LocalDate.parse(birthDate, DateTimeFormatter.ISO_LOCAL_DATE),
57+
gender,
58+
null);
59+
}
60+
}
61+
62+
----
63+
64+
<.> the `age` property will be calculated and filled in search results.
65+
<.> a convenience constructor to set up the test data
66+
====
67+
68+
Note that the `age` property is annotated with `@ScriptedField`.
69+
This inhibits the writing of a corresponding entry in the index mapping and marks the property as a target to put a calculated field from a search response.
70+
71+
== The repository interface
72+
73+
The repository used in this example:
74+
75+
====
76+
[source,java]
77+
----
78+
public interface PersonRepository extends ElasticsearchRepository<Person, String> {
79+
80+
SearchHits<Person> findAllBy(ScriptedField scriptedField);
81+
82+
SearchHits<Person> findByGenderAndAgeLessThanEqual(String gender, Integer age, RuntimeField runtimeField);
83+
}
84+
85+
----
86+
====
87+
88+
== The service class
89+
90+
The service class has a repository injected and an `ElasticsearchOperations` instance to show several ways of poplauting and using the `age` property.
91+
We show the code split up in different pieces to put the explanations in
92+
93+
====
94+
[source,java]
95+
----
96+
import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
97+
import org.springframework.data.elasticsearch.core.SearchHits;
98+
import org.springframework.data.elasticsearch.core.query.Criteria;
99+
import org.springframework.data.elasticsearch.core.query.CriteriaQuery;
100+
import org.springframework.data.elasticsearch.core.query.FetchSourceFilter;
101+
import org.springframework.data.elasticsearch.core.query.RuntimeField;
102+
import org.springframework.data.elasticsearch.core.query.ScriptData;
103+
import org.springframework.data.elasticsearch.core.query.ScriptType;
104+
import org.springframework.data.elasticsearch.core.query.ScriptedField;
105+
import org.springframework.data.elasticsearch.core.query.StringQuery;
106+
import org.springframework.stereotype.Service;
107+
108+
import java.util.List;
109+
110+
@Service
111+
public class PersonService {
112+
private final ElasticsearchOperations operations;
113+
private final PersonRepository repository;
114+
115+
public PersonService(ElasticsearchOperations operations, SaRPersonRepository repository) {
116+
this.operations = operations;
117+
this.repository = repository;
118+
}
119+
120+
public void save() { <.>
121+
List<Person> persons = List.of(
122+
new Person("1", "Smith", "Mary", "f", "1987-05-03"),
123+
new Person("2", "Smith", "Joshua", "m", "1982-11-17"),
124+
new Person("3", "Smith", "Joanna", "f", "2018-03-27"),
125+
new Person("4", "Smith", "Alex", "m", "2020-08-01"),
126+
new Person("5", "McNeill", "Fiona", "f", "1989-04-07"),
127+
new Person("6", "McNeill", "Michael", "m", "1984-10-20"),
128+
new Person("7", "McNeill", "Geraldine", "f", "2020-03-02"),
129+
new Person("8", "McNeill", "Patrick", "m", "2022-07-04"));
130+
131+
repository.saveAll(persons);
132+
}
133+
----
134+
135+
<.> a utility method to store some data in Elasticsearch.
136+
====
137+
138+
=== Scripted fields
139+
140+
The next piece show how to use a scripted field to calculate and return the age of the persons.
141+
Scripted fields can only add something to the returned data, the age cannot be used in the query (see runtime fields for that).)
142+
143+
====
144+
[source,java]
145+
----
146+
public SearchHits<Person> findAllWithAge() {
147+
148+
var scriptedField = ScriptedField.of("age", <.>
149+
ScriptData.of(b -> b
150+
.withType(ScriptType.INLINE)
151+
.withScript("""
152+
Instant currentDate = Instant.ofEpochMilli(new Date().getTime());
153+
Instant startDate = doc['birth-date'].value.toInstant();
154+
return (ChronoUnit.DAYS.between(startDate, currentDate) / 365);
155+
""")));
156+
157+
// version 1: use a direct query
158+
var query = new StringQuery("""
159+
{ "match_all": {} }
160+
""");
161+
query.addScriptedField(scriptedField); <.>
162+
query.addSourceFilter(FetchSourceFilter.of(b -> b.withIncludes("*"))); <.>
163+
164+
var result1 = operations.search(query, Person.class); <.>
165+
166+
// version 2: use the repository
167+
var result2 = repository.findAllBy(scriptedField); <.>
168+
169+
return result1;
170+
}
171+
----
172+
173+
<.> define the `ScriptedField` that calculates the age of a person.
174+
<.> when using a `Query`, add the scripted field to the query.
175+
<.> when adding a scripted field to a `Query`, an additional source filter is needed to also retrieve the _normal_ fields from the document source.
176+
<.> get the data where the `Person` entities now have the values set in their `age` property.
177+
<.> when using the repository, all that needs to be done is adding the scripted field as method parameter.
178+
====
179+
180+
=== Runtime fields
181+
182+
When using runtime fields, the calculated value can be used in the query itself.
183+
In the following code this is used to run a query for a given gender and maximum age of persons:
184+
185+
====
186+
[source,java]
187+
----
188+
public SearchHits<Person> findWithGenderAndMaxAge(String gender, Integer maxAge) {
189+
190+
var runtimeField = new RuntimeField("age", "long", """ <.>
191+
Instant currentDate = Instant.ofEpochMilli(new Date().getTime());
192+
Instant startDate = doc['birth-date'].value.toInstant();
193+
emit (ChronoUnit.DAYS.between(startDate, currentDate) / 365);
194+
""");
195+
196+
// variant 1 : use a direct query
197+
var query = CriteriaQuery.builder(Criteria
198+
.where("gender").is(gender)
199+
.and("age").lessThanEqual(maxAge))
200+
.withRuntimeFields(List.of(runtimeField)) <.>
201+
.withFields("age") <.>
202+
.withSourceFilter(FetchSourceFilter.of(b -> b.withIncludes("*"))) <.>
203+
.build();
204+
205+
var result1 = operations.search(query, Person.class); <.>
206+
207+
// variant 2: use the repository <.>
208+
var result2 = repository.findByGenderAndAgeLessThanEqual(gender, maxAge, runtimeField);
209+
210+
return result1;
211+
}
212+
}
213+
----
214+
215+
<.> define the runtime field that caclulates the // see https://asciidoctor.org/docs/user-manual/#builtin-attributes for builtin attributes.
216+
<.> when using `Query`, add the runtime field.
217+
<.> when adding a scripted field to a `Query`, an additional field parameter is needed to have the calculated value returned.
218+
<.> when adding a scripted field to a `Query`, an additional source filter is needed to also retrieve the _normal_ fields from the document source.
219+
<.> get the data filtered with the query and where the returned entites have the age property set.
220+
<.> when using the repository, all that needs to be done is adding the runtime field as method parameter.
221+
====
222+
223+
In addition to define a runtime fields on a query, they can also be defined in the index by setting the `runtimeFIeldPath` property of the `@Mapping` annotation to point to a JSON file that contains the runtime field definitions.

‎src/main/asciidoc/reference/elasticsearch-misc.adoc

+4-2
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,7 @@ operations.putScript( <.>
365365

366366
To use a search template in a search query, Spring Data Elasticsearch provides the `SearchTemplateQuery`, an implementation of the `org.springframework.data.elasticsearch.core.query.Query` interface.
367367

368-
In the following code, we will add a call using a search template query to a custom repository implementation (see
368+
In the following code, we will add a call using a search template query to a custom repository implementation (see
369369
<<repositories.custom-implementations>>) as
370370
an example how this can be integrated into a repository call.
371371

@@ -399,7 +399,7 @@ public class PersonCustomRepositoryImpl implements PersonCustomRepository {
399399
var query = SearchTemplateQuery.builder() <.>
400400
.withId("person-firstname") <.>
401401
.withParams(
402-
Map.of( <.>
402+
Map.of( <.>
403403
"firstName", firstName,
404404
"from", pageable.getOffset(),
405405
"size", pageable.getPageSize()
@@ -450,3 +450,5 @@ var query = Query.findAll().addSort(Sort.by(order));
450450
About the filter query: It is not possible to use a `CriteriaQuery` here, as this query would be converted into a Elasticsearch nested query which does not work in the filter context. So only `StringQuery` or `NativeQuery` can be used here. When using one of these, like the term query above, the Elasticsearch field names must be used, so take care, when these are redefined with the `@Field(name="...")` definition.
451451

452452
For the definition of the order path and the nested paths, the Java entity property names should be used.
453+
454+
include::elasticsearch-misc-scripted-and-runtime-fields.adoc[leveloffset=+1]

‎src/main/java/org/springframework/data/elasticsearch/annotations/ScriptedField.java

+1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import java.lang.annotation.*;
44

55
/**
6+
* Marks a property to be populated with the result of a scripted field retrieved from an Elasticsearch response.
67
* @author Ryan Murfitt
78
*/
89
@Retention(RetentionPolicy.RUNTIME)

‎src/main/java/org/springframework/data/elasticsearch/client/elc/RequestConverter.java

+3-5
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
import org.springframework.dao.InvalidDataAccessApiUsageException;
6868
import org.springframework.data.domain.Sort;
6969
import org.springframework.data.elasticsearch.core.RefreshPolicy;
70-
import org.springframework.data.elasticsearch.core.ScriptType;
70+
import org.springframework.data.elasticsearch.core.query.ScriptType;
7171
import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
7272
import org.springframework.data.elasticsearch.core.document.Document;
7373
import org.springframework.data.elasticsearch.core.index.*;
@@ -1333,10 +1333,8 @@ private <T> void prepareSearchRequest(Query query, @Nullable String routing, @Nu
13331333
}
13341334

13351335
if (!isEmpty(query.getFields())) {
1336-
builder.fields(fb -> {
1337-
query.getFields().forEach(fb::field);
1338-
return fb;
1339-
});
1336+
var fieldAndFormats = query.getFields().stream().map(field -> FieldAndFormat.of(b -> b.field(field))).toList();
1337+
builder.fields(fieldAndFormats);
13401338
}
13411339

13421340
if (!isEmpty(query.getStoredFields())) {

‎src/main/java/org/springframework/data/elasticsearch/core/query/BaseQuery.java

+10-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@
2929

3030
import org.springframework.data.domain.Pageable;
3131
import org.springframework.data.domain.Sort;
32-
import org.springframework.data.elasticsearch.core.RuntimeField;
3332
import org.springframework.lang.Nullable;
3433
import org.springframework.util.Assert;
3534

@@ -548,6 +547,16 @@ public void setDocValueFields(List<DocValueField> docValueFields) {
548547
this.docValueFields = docValueFields;
549548
}
550549

550+
/**
551+
* @since 5.2
552+
*/
553+
public void addScriptedField(ScriptedField scriptedField) {
554+
555+
Assert.notNull(scriptedField, "scriptedField must not be null");
556+
557+
this.scriptedFields.add(scriptedField);
558+
}
559+
551560
@Override
552561
public List<ScriptedField> getScriptedFields() {
553562
return scriptedFields;

‎src/main/java/org/springframework/data/elasticsearch/core/query/BaseQueryBuilder.java

-1
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525

2626
import org.springframework.data.domain.Pageable;
2727
import org.springframework.data.domain.Sort;
28-
import org.springframework.data.elasticsearch.core.RuntimeField;
2928
import org.springframework.lang.Nullable;
3029
import org.springframework.util.Assert;
3130

‎src/main/java/org/springframework/data/elasticsearch/core/query/FetchSourceFilter.java

+20
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
*/
1616
package org.springframework.data.elasticsearch.core.query;
1717

18+
import java.util.function.Function;
19+
1820
import org.springframework.lang.Nullable;
21+
import org.springframework.util.Assert;
1922

2023
/**
2124
* SourceFilter implementation for providing includes and excludes.
@@ -28,6 +31,23 @@ public class FetchSourceFilter implements SourceFilter {
2831
@Nullable private final String[] includes;
2932
@Nullable private final String[] excludes;
3033

34+
/**
35+
* @since 5.2
36+
*/
37+
public static SourceFilter of(@Nullable final String[] includes, @Nullable final String[] excludes) {
38+
return new FetchSourceFilter(includes, excludes);
39+
}
40+
41+
/**
42+
* @since 5.2
43+
*/
44+
public static SourceFilter of(Function<FetchSourceFilterBuilder, FetchSourceFilterBuilder> builderFunction) {
45+
46+
Assert.notNull(builderFunction, "builderFunction must not be null");
47+
48+
return builderFunction.apply(new FetchSourceFilterBuilder()).build();
49+
}
50+
3151
public FetchSourceFilter(@Nullable final String[] includes, @Nullable final String[] excludes) {
3252
this.includes = includes;
3353
this.excludes = excludes;

‎src/main/java/org/springframework/data/elasticsearch/core/query/Query.java

-1
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525
import org.springframework.data.domain.PageRequest;
2626
import org.springframework.data.domain.Pageable;
2727
import org.springframework.data.domain.Sort;
28-
import org.springframework.data.elasticsearch.core.RuntimeField;
2928
import org.springframework.data.elasticsearch.core.SearchHit;
3029
import org.springframework.lang.Nullable;
3130
import org.springframework.util.Assert;

‎src/main/java/org/springframework/data/elasticsearch/core/RuntimeField.java renamed to ‎src/main/java/org/springframework/data/elasticsearch/core/query/RuntimeField.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
1515
*/
16-
package org.springframework.data.elasticsearch.core;
16+
package org.springframework.data.elasticsearch.core.query;
1717

1818
import java.util.HashMap;
1919
import java.util.Map;

0 commit comments

Comments
 (0)