The documentation code does not compile and I'm not sure if it's a bug in the code or a miss in the documentation.
When you want to configure a client, the doc says to use:
```java
ClientConfiguration.builder().withClientConfigurer( // ...
```
But `withClientConfigurer(ClientConfiguration.ClientConfigurationCallback<?> clientConfigurer)` is only available in `TerminalClientConfigurationBuilder` interface.
And `ClientConfiguration.builder()` returns a `ClientConfigurationBuilderWithRequiredEndpoint` interface.
* support highlight_query
* implement highlight query with spring data elasticsearch query
* highight query by StringQuery
* split highligh fields assertion into different parts
This change publishes a build scan to ge.spring.io for every local build from an authenticated Spring committer and for CI where appropriate access tokens are available. The build will not fail if publishing fails.
This change also allows the build to benefit from local and remote build caching, providing faster builds for all contributors.
Additionally, the project will have access to all features of Gradle Enterprise such as:
- Dashboards to view all historical build scans, along with performance trends over time
- Build failure analytics for enhanced investigation and diagnosis of build failures
- Test failure analytics to better understand trends and causes around slow, failing, and flaky tests
See #2718
When reading from Elasticsearch into a property of type Collection<T> (List<T> or Set<T>) the MappingElasticsearchConverter now can read both from the returned JSON:
an array of T objects - will put the objects in a corresponding collection
a single T object will put the single object into a corrsponding colletcion
This is implemented and tested for both: entities where the properties have setters and immutable classes that only provide an all-args constructor.
Original Pull Request #2282Closes#2280
Fix PersistenceConstructor of Completion to accept the property type String[].
See #1675.
(cherry picked from commit bb3c006b9218a16f3d4497f52de7fba1d5f67880)
* Add support for SpEL expressions via @Value.
* Simplify readCollectionOrArray to consider properly nested lists and maps
* Simplify readMap to allow reading generic maps and entities in maps.
* Report a fallback TypeInformation in DefaultElasticsearchTypeMapper to properly convert nested maps.
We now no longer rely on isSimpleType when writing Maps. This is the preparation to consider Map as simple type.
Resolves#1676.
See #1675.
(cherry picked from commit 877de9c51ce94e5efe294944cf752b20ac2f2cd6)
Original PR: #561
Add a withMaxResults method to NativeSearchQueryBuilder for specifying the maxResults
on the built Query. This is common for aggregation queries where search hits are not
needed.
Having the builder method for setting maxResults is a minor ergonomic improvement, e.g.
NativeSearchQuery searchQuery = NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.addAggregation(terms("examples").field("example"))
.withMaxResults(0)
.build();
versus what was required before:
NativeSearchQuery searchQuery = NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.addAggregation(terms("examples").field("example"))
.build();
searchQuery.setMaxResults(0);
We now use Flux.expand(…) to recursively fetch search results (SearchRequest followed by multiple SearchScrollRequests) until consuming all search hits.
Previously we used inbound/outbound sinks to mimic a continuations by sending a request once the previous request finished. The recursive operator allows now for a simplified operator chain along with improved readability.
We now ensure that response bodies from ClientResponse get released as part of our result handling. This is to prevent cancel signals issuing the connection release so that the connection release can be synchronized (awaited) before any subsequent requests get issued.
Connection release should be part of the Framework but the fallback interferes with Reactor Netty's HttpClient therefore we're ensuring proper resource disposal.
We now use ElasticsearchStatusException instead of HttpClientErrorException to simplify exception translation so that ElasticsearchExceptionTranslator does no longer depend on spring-web.
Added information about which Boot version uses which Spring Data Elasticsearch version as well as hints about which versions are still under active maintenance.
Tweaked the readme to rather point to the reference documentation. This needs to be updated to point to the "current" alias once we've reached GA.
Original pull request: #317.
Un-implement ClientResponse from RawActionResponse to not require us updating the code whenever ClientResponse introduces new methods. Retain only methods from ClientResponse that are actually used.
Consistent versions actross netty dependencies.
To resolve XSD files properly from the classpath, their HTTPS reference must be present in the spring.schemas to avoid internet interaction for resolving an XSD file.
Add nullable annotations to nullable properties in BulkOptions. Add assertions to non-nullable method arguments. Reformat code. Make methods static where possible.
Original pull request: #296.
Ensure copying HttpHeaders before using these as default to prevent modifications of the original object. Adapt tests. Reuse header constants.
Original pull request: #293.
Use Lombok's Data where possible. Extract duplicate code into ResourceUtil and StreamQueries. Generalize multiGet to return List instead of LinkedList. Reorder methods.
Reduce properties usage by removing unused properties from test entities.
Remove final keyword usage in methods. Formatting.
Original pull request: #283.
Add package-info and nullability annotations to org.springframework.data.elasticsearch.core.mapping.
Extract method to avoid excessive nesting.
Add ticket references/convert old references to test methods. Move test models to inner classes. Use static imports for JSON Assert. Formatting.
Original pull request: #281.
Make sure required resources are cleaned before running tests. Some leftovers from other tests may still be lingering around and we need to get rid of those.
We now register ElasticsearchOperations through AbstractElasticsearchConfiguration with an additional bean alias to retain compatibility with existing code and to not require additional configuration when using @EnableElasticsearchRepositories.
EnableElasticsearchRepositories defaults to a bean named elasticsearchTemplate while AbstractElasticsearchConfiguration exposed the template bean named elasticsearchOperations.
Original pull request: #271.
Add test and directly use SearchHit to pass on the index name.
Fix minor flaw in Exception translation for non existing indices along the way.
Original Pull Request: #257
Fix generics in ElasticsearchEntityMapper. Extract methods. Move enum conversion handling to simple type handling for symmetric implementation. Swap comparison of nullable types to avoid potential null dereference.
Rename ElasticsearchDefaultTypeMapper to DefaultElasticsearchTypeMapper. Move MapTypeAliasAccessor to DefaultElasticsearchTypeMapper.
Introduce SearchResultMapperAdapter to avoid empty method implementations in anonymous classes. Javadoc, reference docs, formatting.
Original Pull Request: #237
The root cause of the deletion problem was the doScroll which did not apply the given query and therefor returned all entries from the index.
The doScroll implementation has been fixed to apply the query and multiple unit tests have been added to ensure that the delete methods delete the desired documents only and leave the rest untouched.
Also added unit tests for the scrolling to test it against real queries (not only matchAll).
Original pull request: #240
The getMapping methods in the ElasticsearchRestTemplate now behave like the methods in the ElasticsearchTemplate and return the mapping for the specified index and type.
Original pull request: #239
Reactive Elasticsearch repository support builds on the core repository support utilizing
operations provided via ReactiveElasticsearchOperations executed by a ReactiveElasticsearchClient.
Spring Data Elasticsearchs reactive repository support uses Project Reactor as its reactive
composition library of choice.
There are 3 main interfaces to be used:
* ReactiveRepository
* ReactiveCrudRepository
* ReactiveSortingRepository
For Java configuration, use the @EnableReactiveElasticsearchRepositories annotation.
The following listing shows how to use Java configuration for a repository:
@Configuration
@EnableReactiveElasticsearchRepositories
public class Config extends AbstractReactiveElasticsearchConfiguration {
@Override
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
return ReactiveRestClients.create(ClientConfiguration.localhost());
}
}
Using a repository that extends ReactiveSortingRepository makes all CRUD operations available
as well as methods for sorted access to the entities. Working with the repository instance is a matter of dependency
injecting it into a client.
The repository itself allows defining additional methods backed by the inferred proxy.
public interface ReactivePersonRepository extends ReactiveSortingRepository<Person, String> {
Flux<Person> findByFirstname(String firstname);
Flux<Person> findByFirstname(Publisher<String> firstname);
Flux<Person> findByFirstnameOrderByLastname(String firstname);
Flux<Person> findByFirstname(String firstname, Sort sort);
Flux<Person> findByFirstname(String firstname, Pageable page);
Mono<Person> findByFirstnameAndLastname(String firstname, String lastname);
Mono<Person> findFirstByLastname(String lastname);
@Query("{ \"bool\" : { \"must\" : { \"term\" : { \"lastname\" : \"?0\" } } } }")
Flux<Person> findByLastname(String lastname);
Mono<Long> countByFirstname(String firstname)
Mono<Boolean> existsByFirstname(String firstname)
Mono<Long> deleteByFirstname(String firstname)
}
Original Pull Request: #235
Wrap Scroll execution with usingWhen and run cleanup through usingWhen callback to clean up scrolls state on success/on error/on cancellation.
Extract isEmpty(SearchHits) check into own method. Improve synchronization in ScrollState to prevent concurrent modification exceptions during read.
Original Pull Request: #231
The ReactiveElasticsearchClient now support scrolling through large result sets issuing subsequent _search/scroll requests while emitting data on the outbound channel. Resources bound via their scrollId get freed on completion of the flux.
Original Pull Request: #231
Update documentation to cover newly added configuration options for the ReactiveElasticsearchClient.
Make sure to apply postFilter correctly and set a default limit for unpaged search requests.
Also fix some code format issues.
We now log HTTP requests and responses with the org.springframework.data.elasticsearch.client.WIRE logger for both, the HighLevelRestClient and our reactive client and associate a logging Id for improved traceability.
Configuration of connection/socket timeouts is now available via the ClientConfiguration.
Along the lines we also aligned entity handling to EntityOperations already common in other modules. EntityOperations centralizes how aspects of entities (versioning, retrieval of index name/index type) are handled.
Original Pull Request: #229
ReactiveElasticsearchOperations is the gateway to executing high level commands against an Elasticsearch cluster using the ReactiveElasticsearchClient.
The ReactiveElasticsearchTemplate is the default implementation of ReactiveElasticsearchOperations and offers the following set of features.
* Read/Write mapping support for domain types.
* A rich query and criteria api.
* Resource management and Exception translation.
To get started the ReactiveElasticsearchTemplate needs to know about the actual client to work with.
The easiest way of setting up the ReactiveElasticsearchTemplate is via AbstractReactiveElasticsearchConfiguration providing
dedicated configuration method hooks for base package, the initial entity set etc.
@Configuration
public class Config extends AbstractReactiveElasticsearchConfiguration {
@Bean
@Override
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
// ...
}
}
NOTE: If applicable set default HttpHeaders via the ClientConfiguration of the ReactiveElasticsearchClient.
TIP: If needed the ReactiveElasticsearchTemplate can be configured with default RefreshPolicy and IndicesOptions that get applied to the related requests by overriding the defaults of refreshPolicy() and indicesOptions().
The ReactiveElasticsearchTemplate lets you save, find and delete your domain objects and map those objects to documents stored in Elasticsearch.
@Document(indexName = "marvel", type = "characters")
public class Person {
private @Id String id;
private String name;
private int age;
// Getter/Setter omitted...
}
template.save(new Person("Bruce Banner", 42)) // save a new document
.doOnNext(System.out::println)
.flatMap(person -> template.findById(person.id, Person.class)) // then go find it
.doOnNext(System.out::println)
.flatMap(person -> template.delete(person)) // just to remove remove it again
.doOnNext(System.out::println)
.flatMap(id -> template.count(Person.class)) // so we've got nothing at the end
.doOnNext(System.out::println)
.subscribe(); // yeah :)
The above outputs the following sequence on the console.
> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> QjWCWWcBXiLAnp77ksfR
> 0
Original Pull Request: #229
Convert spaces to tabs for pom.xml. Switch reactive dependencies to optional. Remove unused commonscollections property. Use managed versions for reactor and Spring dependencies.
Introduce WebClientProvider to avoid reinstantiation of WebClient instances. Introduce ClientConfiguration to encapsulate common Elasticsearch client configuration properties. Split ElasticsearchClients into RestClients and ReactiveRestClients to avoid mandatory dependency on WebFlux/Project Reactor. Adapt tests and code referring to WebClient creation.
Extract response body as byte array instead of Flux of DataBuffer to avoid chunking and to parse an entire response.
Encapsulate hostAndPort string used across configuration/HostProvider with InetSocketAddress. Add parser for InetSocketAddress.
Original Pull Request: #226
Initial implementation of a ReactiveElasticsearchClient using WebClient to connect to cluster nodes.
ReactiveElasticsearchClient client = ElasticsearchClients.createClient()
.connectedTo("http://localhost:9200", "http://localhost:9201")
.reactive();
A HostProvider selects active nodes and routes requests.
client.index(request ->
request.index("spring-data")
.type("elasticsearch")
.id(randomUUID().toString())
.source(singletonMap("feature", "reactive-client"))
.setRefreshPolicy(IMMEDIATE);
);
This implementation provides the first building block for reactive Template and Repository support to be added subsequently.
Along the lines we upgraded to Elasticsearch 6.5.
Original Pull Request: #226
* Move @Parent property recognition to ElasticsearchPersistentProperty
* Move type contraints for @Version and @Parent fields to ElasticsearchPersistentProperty to fail faster
* remove unused constant SUPPORTED_ID_TYPES from SimpleElasticsearchPersistentProperty
Original pull request: #208
Extracted ClusterNodes value object to capture the parsing logic and actually properly test it. Added unit tests to verify the proper rejection and the two cases outlined in the ticket.
Related tickets: DATAES-283.
Changes made for Spring Data Commons result in final fields being rejected for manipulation unless there's a wither method available on the object at hand. Unfortunately adapting Spring Data Elasticsearch to support that doesn't work easily as it requires breaking changes to ElasticsearchTemplate as most of the methods assume parameters being handed to be mutable, see the implementation of SimpleElasticsearchRepository.save(…) for instance.
We now mitigate the problem, by enforcing the BeanWrapperPropertyAccessor being used and treating all properties as mutable.
Related tickets: DATACMNS-1322.
Replace all StringUtils and ArrayUtils usages with Springframework's StringUtils and ObjectUtils. Left the commons-lang as test-scope dependency as I believe it brings some values in the tests.
Original pull request: #211.
SimpleElasticsearchPersistentProperty now already checks for the correct type of score properties. Added unit tests for that. Also added unit tests for SimpleElasticsearchPersistentEntity rejecting more than one score property being present.
Additional non-null assertions on components that are required so that we can remove superfluous null checks.
A bit o formatting, Javadoc, missing @since tags and license headers.
Original pull request: #207.
* drop superfluous class FieldIndex
* change FieldType text and keyword starting with capital letters
Original commit: 089d7746be2f2fc4a395bd5c814f664729121f21
We now export composable repositories through our CDI extension. Repositories can now be customized either by a single custom implementation (as it was before) and by providing fragment interfaces along their fragment implementation.
This change aligns CDI support with the existing RepositoryFactory support we provide within a Spring application context.
The configuration file for Log4j2 had been added to src/main/resources accidentally which caused it to be picked up by user applications, e.g. in a Spring Boot application effectively disabling Boot's auto-configuration. This file has now been moved to the test resources.
Original pull request: #191.
Related tickets: spring-projects/spring-boot#10634
We now follow a more consistent naming scheme for the methods in repository that are driven by the following guidelines:
* Methods referring to an identifier now all end on …ById(…).
* Methods taking or returning a collection are named …All(…)
Please see DATACMNS-944 for details.
When an id is set in the IndexQuery object, use it without requiring an
@Document annotation. When it is not, try to resolve it from the object
when it has @Document and @Id annotations.
Added unit test for change in DefaultResultMapperTests. Tweaked and simplified integration tests.
Make use of IdentifierAccessor in ElasticsearcTemplate.getPersistentEntityId(…). Added missing generics in DefaultResultMapper.
Removed unused imports, fixed copyright ranges, authors.
Original pull request: #156.
DefaultResultMapper and ElasticsearchTemplate now use the PersistentPropertyAccessor API to set identifier values on objects instead of using reflection.
Original pull request: #156.
* need to specify Locale.ENGLISH in String.format()
to make tests succeed on non-english locales
* that's because the String.format involves floating
point numbers which are formatted differently in
some locales
* ElasticsearchTemplate.prepareUpdate() properly copies routing
from given UpdateRequest
* add unit tests to ElasticsearchTemplateParentChildTests
for update of a child document
MappingElasticsearchEntityInformation now extends PersistentEntityInformation over AbstractInformation to be able to use the default implementations of getId(…) and getIdType().
Related ticket: DATACMNS-738.
Tweaked method signatures in ElasticsearchRepositoryFactory after some signature changes in Spring Data Commons. Use newly introduced getTragetRepositoryViaReflection(…) to obtain the repository instance via the super class.
Added repositoryBaseClass() attribute to @EnableMongoRepositories.
Related tickets: DATACMNS-542
Updated copyright years. Fall back to locally available Spring Data Commons reference docs as the remote variant doesn't seem to work currently. Included reference to the return type listing appendix from Spring Data Commons.
Thank you for proposing a pull request. This template will guide you through the essential steps necessary for a pull request.
When contributing, please make sure an issue exists in issue tracker and comment on this issue with how you want to address it. By this we not only know that someone is working on an issue, we can also align architectural questions and possible solutions before work is invested . We so can prevent that much work is put into Pull Requests that have little or no chances of being merged.
Make sure that:
-->
- [ ] You have read the [Spring Data contribution guidelines](https://github.com/spring-projects/spring-data-build/blob/master/CONTRIBUTING.adoc).
- [ ] **There is a ticket in the bug tracker for the project in our [issue tracker](https://github.com/spring-projects/spring-data-elasticsearch/issues)**. Add the issue number to the _Closes #issue-number_ line below
- [ ] You use the code formatters provided [here](https://github.com/spring-projects/spring-data-build/tree/master/etc/ide) and have them applied to your changes. Don’t submit any formatting related changes.
- [ ] You submit test cases (unit or integration tests) that back your changes.
- [ ] You added yourself as author in the headers of the classes you touched. Amend the date range in the Apache license header if needed. For new types, add the license header (copy from another file and set the current year only).
You find the contribution guidelines for Spring Data projects https://github.com/spring-projects/spring-data-build/blob/main/CONTRIBUTING.adoc[here].
**Please read these carefully!**
Do not submit a Pull Request before having created an issue and having discussed it. This prevents you from doing work that might be rejected.
== Running the test locally
In order to run the tests locally with `./mvnw test` you need to have docker running because Spring Data Elasticsearch uses https://www.testcontainers.org/[Testcontainers] to start a local running Elasticsearch instance.
= Spring Data for Elasticsearch image:https://jenkins.spring.io/buildStatus/icon?job=spring-data-elasticsearch%2Fmain&subject=Build[link=https://jenkins.spring.io/view/SpringData/job/spring-data-elasticsearch/] https://gitter.im/spring-projects/spring-data[image:https://badges.gitter.im/spring-projects/spring-data.svg[Gitter]] image:https://img.shields.io/badge/Revved%20up%20by-Develocity-06A0CE?logo=Gradle&labelColor=02303A["Revved up by Develocity", link="https://ge.spring.io/scans?search.rootProjectNames=Spring Data Elasticsearch"]
The primary goal of the https://projects.spring.io/spring-data[Spring Data] project is to make it easier to build Spring-powered applications that use new data access technologies such as non-relational databases, map-reduce frameworks, and cloud based data services.
The Spring Data Elasticsearch project provides integration with the https://www.elastic.co/[Elasticsearch] search engine.
Key functional areas of Spring Data Elasticsearch are a POJO centric model for interacting with Elasticsearch Documents and easily writing a Repository style data access layer.
This project is lead and maintained by the community.
== Features
* Spring configuration support using Java based `@Configuration` classes or an XML namespace for an ES client instances.
* `ElasticsearchOperations` class and implementations that increases productivity performing common ES operations.
Includes integrated object mapping between documents and POJOs.
* Feature Rich Object Mapping integrated with Spring’s Conversion Service
* Annotation based mapping metadata
* Automatic implementation of `Repository` interfaces including support for custom search methods.
* CDI support for repositories
== Code of Conduct
This project is governed by the https://github.com/spring-projects/.github/blob/e3cc2ff230d8f1dca06535aa6b5a4a23815861d4/CODE_OF_CONDUCT.md[Spring Code of Conduct].
By participating, you are expected to uphold this code of conduct.
Please report unacceptable behavior to spring-code-of-conduct@pivotal.io.
== Getting Started
Here is a quick teaser of an application using Spring Data Repositories in Java:
[source,java]
----
public interface PersonRepository extends CrudRepository<Person, Long> {
Please check the https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients.configuration[official documentation].
The compatibility between Spring Data Elasticsearch, Elasticsearch client drivers and Spring Boot versions can be found in the https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#preface.versions[reference documentation].
To use the Release candidate versions of the upcoming major version, use our Maven milestone repository and declare the appropriate dependency version:
<version>${version}.RCx</version> <!-- x being 1, 2, ... -->
</dependency>
<repository>
<id>spring-snapshot</id>
<name>Spring Snapshot Repository</name>
<url>https://repo.spring.io/milestone</url>
</repository>
----
If you'd rather like the latest snapshots of the upcoming major version, use our Maven snapshot repository and declare the appropriate dependency version:
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/[reference documentation], and https://docs.spring.io/spring-data/elasticsearch/docs/current/api/[Javadocs].
* Learn the Spring basics – Spring Data builds on Spring Framework, check the https://spring.io[spring.io] web-site for a wealth of reference documentation.
If you are just starting out with Spring, try one of the https://spring.io/guides[guides].
* Ask a question or chat with the community on https://app.gitter.im/#/room/#spring-projects_spring-data:gitter.im[Gitter].
* Report bugs with Spring Data for Elasticsearch at https://github.com/spring-projects/spring-data-elasticsearch/issues[https://github.com/spring-projects/spring-data-elasticsearch/issues].
== Reporting Issues
Spring Data uses GitHub as issue tracking system to record bugs and feature requests.
If you want to raise an issue, please follow the recommendations below:
* Before you log a bug, please search the
https://github.com/spring-projects/spring-data-elasticsearch/issues[issue tracker] to see if someone has already reported the problem.
* If the issue doesn't already exist, https://github.com/spring-projects/spring-data-elasticsearch/issues/new[create a new issue].
* Please provide as much information as possible with the issue report, we like to know the version of Spring Data Elasticsearch that you are using and JVM version.
* If you need to paste code, or include a stack trace use Markdown +++```+++ escapes before and after your text.
* If possible try to create a test-case or project that replicates the issue.
Attach a link to your code or a compressed file containing your code.
== Building from Source
You don’t need to build from source to use Spring Data (binaries in https://repo.spring.io[repo.spring.io]), but if you want to try out the latest and greatest, Spring Data can be easily built with the https://github.com/takari/maven-wrapper[maven wrapper].
You need JDK 17 or above to build the _main_ branch.
For the branches up to and including release 4.4, JDK 8 is required.
[source,bash]
----
$ ./mvnw clean install
----
If you want to build with the regular `mvn` command, you will need https://maven.apache.org/run-maven/index.html[Maven v3.5.0 or above].
_Also see link:CONTRIBUTING.adoc[CONTRIBUTING.adoc] if you wish to submit pull requests, and in particular please sign the https://cla.pivotal.io/sign/spring[Contributor’s Agreement] before submitting your first pull request._
IMPORTANT: When contributing, please make sure an issue exists in https://github.com/spring-projects/spring-data-elasticsearch/issues[issue tracker] and comment on this issue with how you want to address it.
By this we not only know that someone is working on an issue, we can also align architectural questions and possible solutions before work is invested . We so can prevent that much work is put into Pull Requests that have little or no chances of being merged.
=== Building reference documentation
Building the documentation builds also the project without running tests.
[source,bash]
----
$ ./mvnw clean install -Pantora
----
The generated documentation is available from `target/site/index.html`.
== Examples
For examples on using the Spring Data for Elasticsearch, see the https://github.com/spring-projects/spring-data-examples/tree/main/elasticsearch/example[spring-data-examples] project.
== License
Spring Data for Elasticsearch Open Source software released under the https://www.apache.org/licenses/LICENSE-2.0.html[Apache 2.0 license].
Spring Data makes it easier to build Spring-powered applications that use new data access technologies such as non-relational databases, map-reduce frameworks, and cloud based data services as well as provide improved support for relational database technologies.
The Spring Data Elasticsearch project provides integration with the [elasticsearch](http://www.elasticsearch.org/) search engine.
Spring data elaticsearch 1.0.0.RELEASE version is on elasticsearch 1.1.1 library which uses java 1.6 or later version.
As latest version of elasticsearch 1.2.x is now supporting java 1.7, future releases of spring data elasticsearch library can be used with Java 7 or later version.
In order to use with java 6 it has to be with elasticsearch 1.1.x.
### ElasticsearchRepository
A default implementation of ElasticsearchRepository, aligning to the generic Repository Interfaces, is provided. Spring can do the Repository implementation for you depending on method names in the interface definition.
The ElasticsearchCrudRepository extends PagingAndSortingRepository
```java
public interface ElasticsearchCrudRepository<T,IDextendsSerializable> extends ElasticsearchRepository<T,ID>, PagingAndSortingRepository<T,ID> {
}
```
Extending ElasticsearchRepository for custom methods
```java
public interface BookRepository extends Repository<Book,String> {
Here are some ways for you to get involved in the community:
* Get involved with the Spring community on the Spring Community Forums. Please help out on the [forum](http://forum.springsource.org/forumdisplay.php?f=80) by responding to questions and joining the debate.
* Create [JIRA](https://jira.springframework.org/browse/DATAES) tickets for bugs and new features and comment and vote on the ones that you are interested in.
* Github is for social coding: if you want to write code, we encourage contributions through pull requests from [forks of this repository](http://help.github.com/forking/). If you want to contribute code this way, please reference a JIRA ticket as well covering the specific issue you are addressing.
* Watch for upcoming articles on Spring by [subscribing](http://www.springsource.org/node/feed) to springframework.org
Before we accept a non-trivial patch or pull request we will need you to sign the [contributor's agreement](https://support.springsource.com/spring_committer_signup). Signing the contributor's agreement does not grant anyone commit rights to the main repository, but it does mean that we can accept your contributions, and you will get an author credit if we do. Active contributors might be asked to join the core team, and given the ability to merge pull requests.
Code formatting for [Eclipse and Intellij](https://github.com/spring-projects/spring-data-build/tree/master/etc/ide)
[More information about contributing to Spring Data](https://github.com/spring-projects/spring-data-build/blob/master/CONTRIBUTING.md)
is run. There must be _docker_ running, as the integration tests use docker to start an Elasticsearch server.
Integration tests are tests that have the Junit5 Tag `@Tag("integration-test")` on the test class. Normally this should not be set explicitly, but the annotation `@SpringIntegrationTest` should be used. This not only marks the test as integration test, but integrates an automatic setup of an Elasticsearch Testcontainer and integrate this with Spring, so
that the required Beans can be automatically injected. Check _src/test/java/org/springframework/data/elasticsearch/JUnit5SampleRestClientBasedTests.java_ as a reference setup
Spring Data support for Elasticsearch contains a wide range of features:
* Spring configuration support for various xref:elasticsearch/clients.adoc[Elasticsearch clients].
* The xref:elasticsearch/template.adoc[`ElasticsearchTemplate` and `ReactiveElasticsearchTemplate`] helper classes that provide object mapping between ES index operations and POJOs.
* xref:elasticsearch/template.adoc#exception-translation[Exception translation] into Spring's portable {springDocsUrl}data-access.html#dao-exceptions[Data Access Exception Hierarchy].
* Feature rich xref:elasticsearch/object-mapping.adoc[object mapping] integrated with _Spring's_ {springDocsUrl}core.html#core-convert[Conversion Service].
* xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations[Annotation-based mapping] metadata that is extensible to support other metadata formats.
* Java-based xref:elasticsearch/template.adoc#cassandra.template.query[query, criteria, and update DSLs].
* Automatic implementation of xref:repositories.adoc[imperative and reactive `Repository` interfaces] including support for xref:repositories/custom-implementations.adoc[custom query methods].
For most data-oriented tasks, you can use the `[Reactive]ElasticsearchTemplate` or the `Repository` support, both of which use the rich object-mapping functionality.
Spring Data Elasticsearch uses consistent naming conventions on objects in various APIs to those found in the DataStax Java Driver so that they are familiar and so that you can map your existing knowledge onto the Spring APIs.
In order for the auditing code to be able to decide whether an entity instance is new, the entity must implement the `Persistable<ID>` interface which is defined as follows:
[source,java]
----
package org.springframework.data.domain;
import org.jspecify.annotations.Nullable;
public interface Persistable<ID> {
@Nullable
ID getId();
boolean isNew();
}
----
As the existence of an Id is not a sufficient criterion to determine if an enitity is new in Elasticsearch, additional information is necessary. One way is to use the creation-relevant auditing fields for this decision:
A `Person` entity might look as follows - omitting getter and setter methods for brevity:
[source,java]
----
@Document(indexName = "person")
public class Person implements Persistable<Long> {
@Id private Long id;
private String lastName;
private String firstName;
@CreatedDate
@Field(type = FieldType.Date, format = DateFormat.basic_date_time)
private Instant createdDate;
@CreatedBy
private String createdBy
@Field(type = FieldType.Date, format = DateFormat.basic_date_time)
<.> the getter is the required implementation from the interface
<.> an object is new if it either has no `id` or none of fields containing creation attributes are set.
[[elasticsearch.auditing.activating]]
== Activating auditing
After the entities have been set up and providing the `AuditorAware` - or `ReactiveAuditorAware` - the Auditing must be activated by setting the `@EnableElasticsearchAuditing` on a configuration class:
[source,java]
----
@Configuration
@EnableElasticsearchRepositories
@EnableElasticsearchAuditing
class MyConfiguration {
// configuration code
}
----
When using the reactive stack this must be:
[source,java]
----
@Configuration
@EnableReactiveElasticsearchRepositories
@EnableReactiveElasticsearchAuditing
class MyConfiguration {
// configuration code
}
----
If your code contains more than one `AuditorAware` bean for different types, you must provide the name of the bean to use as an argument to the `auditorAwareRef` parameter of the
This chapter illustrates configuration and usage of supported Elasticsearch client implementations.
Spring Data Elasticsearch operates upon an Elasticsearch client (provided by Elasticsearch client libraries) that is connected to a single Elasticsearch node or a cluster.
Although the Elasticsearch Client can be used directly to work with the cluster, applications using Spring Data Elasticsearch normally use the higher level abstractions of xref:elasticsearch/template.adoc[Elasticsearch Operations] and xref:elasticsearch/repositories/elasticsearch-repositories.adoc[Elasticsearch Repositories].
[[elasticsearch.clients.rest5client]]
== Imperative Rest5Client
To use the imperative (non-reactive) Rest5Client, a configuration bean must be configured like this:
public class MyClientConfig extends ElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() <.>
.connectedTo("localhost:9200")
.build();
}
}
----
<.> for a detailed description of the builder methods see xref:elasticsearch/clients.adoc#elasticsearch.clients.configuration[Client Configuration]
====
The javadoc:org.springframework.data.elasticsearch.client.elc.ElasticsearchConfiguration[] class allows further configuration by overriding for example the `jsonpMapper()` or `transportOptions()` methods.
The following beans can then be injected in other Spring components:
<.> an implementation of javadoc:org.springframework.data.elasticsearch.core.ElasticsearchOperations[]
<.> the `co.elastic.clients.elasticsearch.ElasticsearchClient` that is used.
<.> the low level `Rest5Client` from the Elasticsearch libraries
<.> the `JsonpMapper` user by the Elasticsearch `Transport`
====
Basically one should just use the javadoc:org.springframework.data.elasticsearch.core.ElasticsearchOperations[] to interact with the Elasticsearch cluster.
When using repositories, this instance is used under the hood as well.
[[elasticsearch.clients.restclient]]
== Deprecated Imperative RestClient
To use the imperative (non-reactive) RestClient - deprecated since version 6 - , the following dependency needs to be added, adapt the correct version. The exclusion is needed in a Spring Boot application:
public class MyClientConfig extends ElasticsearchLegacyRestClientConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() <.>
.connectedTo("localhost:9200")
.build();
}
}
----
<.> for a detailed description of the builder methods see xref:elasticsearch/clients.adoc#elasticsearch.clients.configuration[Client Configuration]
====
The javadoc:org.springframework.data.elasticsearch.client.elc.ElasticsearchConfiguration[] class allows further configuration by overriding for example the `jsonpMapper()` or `transportOptions()` methods.
The following beans can then be injected in other Spring components:
<.> an implementation of javadoc:org.springframework.data.elasticsearch.core.ElasticsearchOperations[]
<.> the `co.elastic.clients.elasticsearch.ElasticsearchClient` that is used.
<.> the low level `RestClient` from the Elasticsearch libraries
<.> the `JsonpMapper` user by the Elasticsearch `Transport`
====
Basically one should just use the javadoc:org.springframework.data.elasticsearch.core.ElasticsearchOperations[] to interact with the Elasticsearch cluster.
When using repositories, this instance is used under the hood as well.
[[elasticsearch.clients.reactiverest5client]]
== Reactive Rest5Client
When working with the reactive stack, the configuration must be derived from a different class:
public class MyClientConfig extends ReactiveElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() <.>
.connectedTo("localhost:9200")
.build();
}
}
----
<.> for a detailed description of the builder methods see xref:elasticsearch/clients.adoc#elasticsearch.clients.configuration[Client Configuration]
====
The javadoc:org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchConfiguration[] class allows further configuration by overriding for example the `jsonpMapper()` or `transportOptions()` methods.
The following beans can then be injected in other Spring components:
<.> an implementation of javadoc:org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations[]
<.> the `org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient` that is used.
This is a reactive implementation based on the Elasticsearch client implementation.
<.> the low level `RestClient` from the Elasticsearch libraries
<.> the `JsonpMapper` user by the Elasticsearch `Transport`
====
Basically one should just use the javadoc:org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations[] to interact with the Elasticsearch cluster.
When using repositories, this instance is used under the hood as well.
[[elasticsearch.clients.reactiverestclient]]
== Deprecated Reactive RestClient
See the section above for the imperative code to use the deprecated RestClient for the necessary dependencies to include.
When working with the reactive stack, the configuration must be derived from a different class:
public class MyClientConfig extends ReactiveElasticsearchLegacyRestClientConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() <.>
.connectedTo("localhost:9200")
.build();
}
}
----
<.> for a detailed description of the builder methods see xref:elasticsearch/clients.adoc#elasticsearch.clients.configuration[Client Configuration]
====
The javadoc:org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchConfiguration[] class allows further configuration by overriding for example the `jsonpMapper()` or `transportOptions()` methods.
The following beans can then be injected in other Spring components:
<.> an implementation of javadoc:org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations[]
<.> the `org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient` that is used.
This is a reactive implementation based on the Elasticsearch client implementation.
<.> the low level `RestClient` from the Elasticsearch libraries
<.> the `JsonpMapper` user by the Elasticsearch `Transport`
====
Basically one should just use the javadoc:org.springframework.data.elasticsearch.core.ReactiveElasticsearchOperations[] to interact with the Elasticsearch cluster.
When using repositories, this instance is used under the hood as well.
[[elasticsearch.clients.configuration]]
== Client Configuration
Client behaviour can be changed via the javadoc:org.springframework.data.elasticsearch.client.ClientConfiguration[] that allows to set options for SSL, connect and socket timeouts, headers and other parameters.
<.> Define default headers, if they need to be customized
<.> Use the builder to provide cluster addresses, set default `HttpHeaders` or enable SSL.
<.> Optionally enable SSL.There exist overloads of this function that can take a `SSLContext` or as an alternative the fingerprint of the certificate as it is output by Elasticsearch 8 on startup.
<.> Optionally set a proxy.
<.> Optionally set a path prefix, mostly used when different clusters a behind some reverse proxy.
<.> Set the connection timeout.
<.> Set the socket timeout.
<.> Optionally set headers.
<.> Add basic authentication.
<.> A `Supplier<HttpHeaders>` function can be specified which is called every time before a request is sent to Elasticsearch - here, as an example, the current time is written in a header.
<.> a function to configure the created client (see xref:elasticsearch/clients.adoc#elasticsearch.clients.configuration.callbacks[Client configuration callbacks]), can be added multiple times.
====
IMPORTANT: Adding a Header supplier as shown in above example allows to inject headers that may change over the time, like authentication JWT tokens.
If this is used in the reactive setup, the supplier function *must not* block!
[[elasticsearch.clients.configuration.callbacks]]
=== Client configuration callbacks
The javadoc:org.springframework.data.elasticsearch.client.ClientConfiguration[] class offers the most common parameters to configure the client.
In the case this is not enough, the user can add callback functions by using the `withClientConfigurer(ClientConfigurationCallback<?>)` method.
To see what is actually sent to and received from the server `Request` / `Response` logging on the transport level needs to be turned on as outlined in the snippet below.
This can be enabled in the Elasticsearch client by setting the level of the `tracer` package to "trace" (see
* Use the new Elasticsearch Rest5Client as default
[[new-features.5-5-0]]
== New in Spring Data Elasticsearch 5.5
* Upgrade to Elasticsearch 8.18.1.
* Add support for the `@SearchTemplateQuery` annotation on repository methods.
* Scripted field properties of type collection can be populated from scripts returning arrays.
[[new-features.5-4-0]]
== New in Spring Data Elasticsearch 5.4
* Upgrade to Elasticsearch 8.15.3.
* Allow to customize the mapped type name for `@InnerField` and `@Field` annotations.
* Support for Elasticsearch SQL.
* Add support for retrieving request executionDuration.
[[new-features.5-3-0]]
== New in Spring Data Elasticsearch 5.3
* Upgrade to Elasticsearch 8.13.2.
* Add support for highlight queries in highlighting.
* Add shard statistics to the `SearchHit` class.
* Add support for multi search template API.
* Add support for SpEL in @Query.
* Add support for field aliases in the index mapping.
* Add support for has_child and has_parent queries.
[[new-features.5-2-0]]
== New in Spring Data Elasticsearch 5.2
* Upgrade to Elasticsearch 8.11.1
* The `JsonpMapper` for Elasticsearch is now configurable and provided as bean.
* Improved AOT runtime hints for Elasticsearch client library classes.
* Add Kotlin extensions and repository coroutine support.
* Introducing `VersionConflictException` class thrown in case thatElasticsearch reports an 409 error with a version conflict.
* Enable MultiField annotation on property getter
* Support nested sort option
* Improved scripted und runtime field support
* Improved refresh policy support
[[new-features.5-1-0]]
== New in Spring Data Elasticsearch 5.1
* Upgrade to Elasticsearch 8.7.1
* Allow specification of the TLS certificate when connecting to an Elasticsearch 8 cluster
[[new-features.5-0-0]]
== New in Spring Data Elasticsearch 5.0
* Upgrade to Java 17 baseline
* Upgrade to Spring Framework 6
* Upgrade to Elasticsearch 8.5.0
* Use the new Elasticsearch client library
[[new-features.4-4-0]]
== New in Spring Data Elasticsearch 4.4
* Introduction of new imperative and reactive clients using the classes from the new Elasticsearch Java client
* Upgrade to Elasticsearch 7.17.3.
[[new-features.4-3-0]]
== New in Spring Data Elasticsearch 4.3
* Upgrade to Elasticsearch 7.15.2.
* Allow runtime_fields to be defined in the index mapping.
* Add native support for range field types by using a range object.
* Add repository search for nullable or empty properties.
* Enable custom converters for single fields.
* Supply a custom `Sort.Order` providing Elasticsearch specific parameters.
[[new-features.4-2-0]]
== New in Spring Data Elasticsearch 4.2
* Upgrade to Elasticsearch 7.10.0.
* Support for custom routing values
[[new-features.4-1-0]]
== New in Spring Data Elasticsearch 4.1
* Uses Spring 5.3.
* Upgrade to Elasticsearch 7.9.3.
* Improved API for alias management.
* Introduction of `ReactiveIndexOperations` for index management.
* Index templates support.
* Support for Geo-shape data with GeoJson.
[[new-features.4-0-0]]
== New in Spring Data Elasticsearch 4.0
* Uses Spring 5.2.
* Upgrade to Elasticsearch 7.6.2.
* Deprecation of `TransportClient` usage.
* Implements most of the mapping-types available for the index mappings.
* Removal of the Jackson `ObjectMapper`, now using the xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model[MappingElasticsearchConverter]
* Cleanup of the API in the `*Operations` interfaces, grouping and renaming methods so that they match the Elasticsearch API, deprecating the old methods, aligning with other Spring Data modules.
* Introduction of `SearchHit<T>` class to represent a found document together with the relevant result metadata for this document (i.e. _sortValues_).
* Introduction of the `SearchHits<T>` class to represent a whole search result together with the metadata for the complete search result (i.e. _max_score_).
* Introduction of `SearchPage<T>` class to represent a paged result containing a `SearchHits<T>` instance.
* Introduction of the `GeoDistanceOrder` class to be able to create sorting by geographical distance
* Implementation of Auditing Support
* Implementation of lifecycle entity callbacks
[[new-features.3-2-0]]
== New in Spring Data Elasticsearch 3.2
* Secured Elasticsearch cluster support with Basic Authentication and SSL transport.
* Upgrade to Elasticsearch 6.8.1.
* Reactive programming support with xref:elasticsearch/repositories/reactive-elasticsearch-repositories.adoc[Reactive Elasticsearch Repositories] and xref:.
* Introduction of the xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model[ElasticsearchEntityMapper] as an alternative to the Jackson `ObjectMapper`.
| Invoked after a domain object is converted from `org.springframework.data.elasticsearch.core.document.Document` on reading result data from Elasticsearch.
Spring Data Elasticsearch supports the https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html[Join data type] for creating the corresponding index mappings and for storing the relevant information.
[[elasticsearch.jointype.setting-up]]
== Setting up the data
For an entity to be used in a parent child join relationship, it must have a property of type `JoinField` which must be annotated.
Let's assume a `Statement` entity where a statement may be a _question_, an _answer_, a _comment_ or a _vote_ (a _Builder_ is also shown in this example, it's not necessary, but later used in the sample code):
====
[source,java]
----
@Document(indexName = "statements")
@Routing("routing") <.>
public class Statement {
@Id
private String id;
@Field(type = FieldType.Text)
private String text;
@Field(type = FieldType.Keyword)
private String routing;
@JoinTypeRelations(
relations =
{
@JoinTypeRelation(parent = "question", children = {"answer", "comment"}), <.>
@JoinTypeRelation(parent = "answer", children = "vote") <.>
}
)
private JoinField<String> relation; <.>
private Statement() {
}
public static StatementBuilder builder() {
return new StatementBuilder();
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getRouting() {
return routing;
}
public void setRouting(String routing) {
this.routing = routing;
}
public String getText() {
return text;
}
public void setText(String text) {
this.text = text;
}
public JoinField<String> getRelation() {
return relation;
}
public void setRelation(JoinField<String> relation) {
this.relation = relation;
}
public static final class StatementBuilder {
private String id;
private String text;
private String routing;
private JoinField<String> relation;
private StatementBuilder() {
}
public StatementBuilder withId(String id) {
this.id = id;
return this;
}
public StatementBuilder withRouting(String routing) {
this.routing = routing;
return this;
}
public StatementBuilder withText(String text) {
this.text = text;
return this;
}
public StatementBuilder withRelation(JoinField<String> relation) {
this.relation = relation;
return this;
}
public Statement build() {
Statement statement = new Statement();
statement.setId(id);
statement.setRouting(routing);
statement.setText(text);
statement.setRelation(relation);
return statement;
}
}
}
----
<.> for routing related info see xref:elasticsearch/routing.adoc[Routing values]
<.> a question can have answers and comments
<.> an answer can have votes
<.> the `JoinField` property is used to combine the name (_question_, _answer_, _comment_ or _vote_) of the relation with the parent id.
The generic type must be the same as the `@Id` annotated property.
====
Spring Data Elasticsearch will build the following mapping for this class:
====
[source,json]
----
{
"statements": {
"mappings": {
"properties": {
"_class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"routing": {
"type": "keyword"
},
"relation": {
"type": "join",
"eager_global_ordinals": true,
"relations": {
"question": [
"answer",
"comment"
],
"answer": "vote"
}
},
"text": {
"type": "text"
}
}
}
}
}
----
====
[[elasticsearch.jointype.storing]]
== Storing data
Given a repository for this class the following code inserts a question, two answers, a comment and a vote:
<5> a vote for the first answer, this needs to have the routing set to the weather document, see xref:elasticsearch/routing.adoc[Routing values].
====
[[elasticsearch.jointype.retrieving]]
== Retrieving data
Currently native queries must be used to query the data, so there is no support from standard repository methods. xref:repositories/custom-implementations.adoc[] can be used instead.
The following code shows as an example how to retrieve all entries that have a _vote_ (which must be _answers_, because only answers can have a vote) using an `ElasticsearchOperations` instance:
This chapter covers additional support for Elasticsearch operations that cannot be directly accessed via the repository interface.
It is recommended to add those operations as custom implementation as described in xref:repositories/custom-implementations.adoc[] .
[[elasticsearc.misc.index.settings]]
== Index settings
When creating Elasticsearch indices with Spring Data Elasticsearch different index settings can be defined by using the `@Setting` annotation.
The following arguments are available:
* `useServerConfiguration` does not send any settings parameters, so the Elasticsearch server configuration determines them.
* `settingPath` refers to a JSON file defining the settings that must be resolvable in the classpath
* `shards` the number of shards to use, defaults to _1_
* `replicas` the number of replicas, defaults to _1_
* `refreshIntervall`, defaults to _"1s"_
* `indexStoreType`, defaults to _"fs"_
It is as well possible to define https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-modules-index-sorting.html[index sorting] (check the linked Elasticsearch documentation for the possible field types and values):
@Field(name = "first_field", type = FieldType.Keyword)
private String firstField;
@Nullable @Field(name = "second_field", type = FieldType.Keyword)
private String secondField;
// getter and setter...
}
----
<.> when defining sort fields, use the name of the Java property (_firstField_), not the name that might be defined for Elasticsearch (_first_field_)
<.> `sortModes`, `sortOrders` and `sortMissingValues` are optional, but if they are set, the number of entries must match the number of `sortFields` elements
====
[[elasticsearch.misc.mappings]]
== Index Mapping
When Spring Data Elasticsearch creates the index mapping with the `IndexOperations.createMapping()` methods, it uses the annotations described in xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations[Mapping Annotation Overview], especially the `@Field` annotation.
In addition to that it is possible to add the `@Mapping` annotation to a class.
This annotation has the following properties:
* `mappingPath` a classpath resource in JSON format; if this is not empty it is used as the mapping, no other mapping processing is done.
* `enabled` when set to false, this flag is written to the mapping and no further processing is done.
* `dateDetection` and `numericDetection` set the corresponding properties in the mapping when not set to `DEFAULT`.
* `dynamicDateFormats` when this String array is not empty, it defines the date formats used for automatic date detection.
* `runtimeFieldsPath` a classpath resource in JSON format containing the definition of runtime fields which is written to the index mappings, for example:
Elasticsearch has a scroll API for getting big result set in chunks.
This is internally used by Spring Data Elasticsearch to provide the implementations of the `<T> SearchHitsIterator<T> SearchOperations.searchForStream(Query query, Class<T> clazz, IndexCoordinates index)` method.
====
[source,java]
----
IndexCoordinates index = IndexCoordinates.of("sample-index");
List<SampleEntity> sampleEntities = new ArrayList<>();
while (stream.hasNext()) {
sampleEntities.add(stream.next());
}
stream.close();
----
====
There are no methods in the `SearchOperations` API to access the scroll id, if it should be necessary to access this, the following methods of the `AbstractElasticsearchTemplate` can be used (this is the base implementation for the different `ElasticsearchOperations` implementations):
In addition to the default sort options described in xref:repositories/query-methods-details.adoc#repositories.paging-and-sorting[Paging and Sorting], Spring Data Elasticsearch provides the class `org.springframework.data.elasticsearch.core.query.Order` which derives from `org.springframework.data.domain.Sort.Order`.
It offers additional parameters that can be sent to Elasticsearch when specifying the sorting of the result (see https://www.elastic.co/guide/en/elasticsearch/reference/7.15/sort-search-results.html).
There also is the `org.springframework.data.elasticsearch.core.query.GeoDistanceOrder` class which can be used to have the result of a search operation ordered by geographical distance.
If the class to be retrieved has a `GeoPoint` property named _location_, the following `Sort` would sort the results by distance to the given point:
====
[source,java]
----
Sort.by(new GeoDistanceOrder("location", new GeoPoint(48.137154, 11.5761247)))
----
====
[[elasticsearch.misc.runtime-fields]]
== Runtime Fields
From version 7.12 on Elasticsearch has added the feature of runtime fields (https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime.html).
Spring Data Elasticsearch supports this in two ways:
=== Runtime field definitions in the index mappings
The first way to define runtime fields is by adding the definitions to the index mappings (see https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime-mapping-fields.html).
To use this approach in Spring Data Elasticsearch the user must provide a JSON file that contains the corresponding definition, for example:
The second way to define runtime fields is by adding the definitions to a search query (see https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime-search-request.html).
The following code example shows how to do this with Spring Data Elasticsearch :
The entity used is a simple object that has a `price` property:
The following query uses a runtime field that calculates a `priceWithTax` value by adding 19% to the price and uses this value in the search query to find all entities where `priceWithTax` is higher or equal than a given value:
====
[source,java]
----
RuntimeField runtimeField = new RuntimeField("priceWithTax", "double", "emit(doc['price'].value * 1.19)");
Query query = new CriteriaQuery(new Criteria("priceWithTax").greaterThanEqual(16.5));
This works with every implementation of the `Query` interface.
[[elasticsearch.misc.point-in-time]]
== Point In Time (PIT) API
`ElasticsearchOperations` supports the point in time API of Elasticsearch (see https://www.elastic.co/guide/en/elasticsearch/reference/8.3/point-in-time-api.html).
The following code snippet shows how to use this feature with a fictional `Person` class:
<.> Use the `putScript()` method to store a search template script
<.> The name / id of the script
<.> Scripts that are used in search templates must be in the _mustache_ language.
<.> The script source
<.> The search parameter in the script
<.> Paging request offset
<.> Paging request size
====
To use a search template in a search query, Spring Data Elasticsearch provides the `SearchTemplateQuery`, an implementation of the `org.springframework.data.elasticsearch.core.query.Query` interface.
NOTE: Although `SearchTemplateQuery` is an implementation of the `Query` interface, not all of the functionality provided by the base class is available for a `SearchTemplateQuery` like setting a `Pageable` or a `Sort`. Values for this functionality must be added to the stored script like shown in the following example for paging parameters. If these values are set on the `Query` object, they will be ignored.
In the following code, we will add a call using a search template query to a custom repository implementation (see
xref:repositories/custom-implementations.adoc[]) as an example how this can be integrated into a repository call.
We first define the custom repository fragment interface:
<.> The parameters are passed in a `Map<String,Object>`
<.> Do the search in the same way as with the other query types.
====
[[elasticsearch.misc.nested-sort]]
== Nested sort
Spring Data Elasticsearch supports sorting within nested objects (https://www.elastic.co/guide/en/elasticsearch/reference/8.9/sort-search-results.html#nested-sorting)
The following example, taken from the `org.springframework.data.elasticsearch.core.query.sort.NestedSortIntegrationTests` class, shows how to define the nested sort.
====
[source,java]
----
var filter = StringQuery.builder("""
{ "term": {"movies.actors.sex": "m"} }
""").build();
var order = new org.springframework.data.elasticsearch.core.query.Order(Sort.Direction.DESC,
"movies.actors.yearOfBirth")
.withNested(
Nested.builder("movies")
.withNested(
Nested.builder("movies.actors")
.withFilter(filter)
.build())
.build());
var query = Query.findAll().addSort(Sort.by(order));
----
====
About the filter query: It is not possible to use a `CriteriaQuery` here, as this query would be converted into a Elasticsearch nested query which does not work in the filter context. So only `StringQuery` or `NativeQuery` can be used here. When using one of these, like the term query above, the Elasticsearch field names must be used, so take care, when these are redefined with the `@Field(name="...")` definition.
For the definition of the order path and the nested paths, the Java entity property names should be used.
Spring Data Elasticsearch Object Mapping is the process that maps a Java object - the domain entity - into the JSON representation that is stored in Elasticsearch and back.
The class that is internally used for this mapping is the
`MappingElasticsearchConverter`.
[[elasticsearch.mapping.meta-model]]
== Meta Model Object Mapping
The Metamodel based approach uses domain type information for reading/writing from/to Elasticsearch.
This allows to register `Converter` instances for specific domain type mapping.
[[elasticsearch.mapping.meta-model.annotations]]
=== Mapping Annotation Overview
The `MappingElasticsearchConverter` uses metadata to drive the mapping of objects to documents.
The metadata is taken from the entity's properties which can be annotated.
The following annotations are available:
* `@Document`: Applied at the class level to indicate this class is a candidate for mapping to the database.
The most important attributes are (check the API documentation for the complete list of attributes):
** `indexName`: the name of the index to store this entity in.
This can contain a SpEL template expression like `"log-#{T(java.time.LocalDate).now().toString()}"`
** `createIndex`: flag whether to create an index on repository bootstrapping.
Default value is _true_.
See xref:elasticsearch/repositories/elasticsearch-repositories.adoc#elasticsearch.repositories.autocreation[Automatic creation of indices with the corresponding mapping]
* `@Id`: Applied at the field level to mark the field used for identity purpose.
* `@Transient`, `@ReadOnlyProperty`, `@WriteOnlyProperty`: see the following section xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations.read-write[Controlling which properties are written to and read from Elasticsearch] for detailed information.
* `@PersistenceConstructor`: Marks a given constructor - even a package protected one - to use when instantiating the object from the database.
Constructor arguments are mapped by name to the key values in the retrieved Document.
* `@Field`: Applied at the field level and defines properties of the field, most of the attributes map to the respective https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html[Elasticsearch Mapping] definitions (the following list is not complete, check the annotation Javadoc for a complete reference):
** `name`: The name of the field as it will be represented in the Elasticsearch document, if not set, the Java field name is used.
** `type`: The field type, can be one of _Text, Keyword, Long, Integer, Short, Byte, Double, Float, Half_Float, Scaled_Float, Date, Date_Nanos, Boolean, Binary, Integer_Range, Float_Range, Long_Range, Double_Range, Date_Range, Ip_Range, Object, Nested, Ip, TokenCount, Percolator, Flattened, Search_As_You_Type_.
See https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html[Elasticsearch Mapping Types].
If the field type is not specified, it defaults to `FieldType.Auto`.
This means, that no mapping entry is written for the property and that Elasticsearch will add a mapping entry dynamically when the first data for this property is stored (check the Elasticsearch documentation for dynamic mapping rules).
** `format`: One or more built-in date formats, see the next section xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations.date-formats[Date format mapping].
** `pattern`: One or more custom date formats, see the next section xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations.date-formats[Date format mapping].
** `store`: Flag whether the original field value should be store in Elasticsearch, default value is _false_.
** `analyzer`, `searchAnalyzer`, `normalizer` for specifying custom analyzers and normalizer.
* `@GeoPoint`: Marks a field as _geo_point_ datatype.
Can be omitted if the field is an instance of the `GeoPoint` class.
* `@ValueConverter` defines a class to be used to convert the given property.
In difference to a registered Spring `Converter` this only converts the annotated property and not every property of the given type.
The mapping metadata infrastructure is defined in a separate spring-data-commons project that is technology agnostic.
==== Controlling which properties are written to and read from Elasticsearch
This section details the annotations that define if the value of a property is written to or read from Elasticsearch.
`@Transient`: A property annotated with this annotation will not be written to the mapping, it's value will not be sent to Elasticsearch and when documents are returned from Elasticsearch, this property will not be set in the resulting entity.
`@ReadOnlyProperty`: A property with this annotation will not have its value written to Elasticsearch, but when returning data, the property will be filled with the value returned in the document from Elasticsearch.
One use case for this are runtime fields defined in the index mapping.
`@WriteOnlyProperty`: A property with this annotation will have its value stored in Elasticsearch but will not be set with any value when reading document.
This can be used for example for synthesized fields which should go into the Elasticsearch index but are not used elsewhere.
Properties that derive from `TemporalAccessor` or are of type `java.util.Date` must either have a `@Field` annotation of type `FieldType.Date` or a custom converter must be registered for this type.
This paragraph describes the use of
`FieldType.Date`.
There are two attributes of the `@Field` annotation that define which date format information is written to the mapping (also see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#built-in-date-formats[Elasticsearch Built In Formats] and https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#custom-date-formats[Elasticsearch Custom Date Formats])
The `format` attribute is used to define at least one of the predefined formats.
If it is not defined, then a default value of __date_optional_time_ and _epoch_millis_ is used.
The `pattern` attribute can be used to add additional custom format strings.
If you want to use only custom date formats, you must set the `format` property to empty `{}`.
The following table shows the different attributes and the mapping created from their values:
NOTE: If you are using a custom date format, you need to use _uuuu_ for the year instead of _yyyy_.
This is due to a https://www.elastic.co/guide/en/elasticsearch/reference/current/migrate-to-java-time.html#java-time-migration-incompatible-date-formats[change in Elasticsearch 7].
Check the code of the `org.springframework.data.elasticsearch.annotations.DateFormat` enum for a complete list of predefined values and their patterns.
When a field is annotated with a type of one of _Integer_Range, Float_Range, Long_Range, Double_Range, Date_Range,_ or _Ip_Range_ the field must be an instance of a class that will be mapped to an Elasticsearch range, for example:
====
[source,java]
----
class SomePersonData {
@Field(type = FieldType.Integer_Range)
private ValidAge validAge;
// getter and setter
}
class ValidAge {
@Field(name="gte")
private Integer from;
@Field(name="lte")
private Integer to;
// getter and setter
}
----
====
As an alternative Spring Data Elasticsearch provides a `Range<T>` class so that the previous example can be written as:
====
[source,java]
----
class SomePersonData {
@Field(type = FieldType.Integer_Range)
private Range<Integer> validAge;
// getter and setter
}
----
====
Supported classes for the type `<T>` are `Integer`, `Long`, `Float`, `Double`, `Date` and classes that implement the
Without further configuration, Spring Data Elasticsearch will use the property name of an object as field name in Elasticsearch.
This can be changed for individual field by using the `@Field` annotation on that property.
It is also possible to define a `FieldNamingStrategy` in the configuration of the client (xref:elasticsearch/clients.adoc[Elasticsearch Clients]).
If for example a `SnakeCaseFieldNamingStrategy` is configured, the property _sampleProperty_ of the object would be mapped to _sample_property_ in Elasticsearch.
A `FieldNamingStrategy` applies to all entities; it can be overwritten by setting a specific name with `@Field` on a property.
Normally the properties used in an entity are fields of the entity class.
There might be cases, when a property value is calculated in the entity and should be stored in Elasticsearch.
In this case, the getter method (`getProperty()`) can be annotated with the `@Field` annotation, in addition to that the method must be annotated with `@AccessType(AccessType.Type
.PROPERTY)`.
The third annotation that is needed in such a case is `@WriteOnlyProperty`, as such a value is only written to Elasticsearch.
This annotation can be set on a String property of an entity.
This property will not be written to the mapping, it will not be stored in Elasticsearch and its value will not be read from an Elasticsearch document.
After an entity is persisted, for example with a call to `ElasticsearchOperations.save(T entity)`, the entity returned from that call will contain the name of the index that an entity was saved to in that property.
This is useful when the index name is dynamically set by a bean, or when writing to a write alias.
Putting some value into such a property does not set the index into which an entity is stored!
Mapping uses _type hints_ embedded in the document sent to the server to allow generic type mapping.
Those type hints are represented as `_class` attributes within the document and are written for each aggregate root.
.Type Hints
====
[source,java]
----
public class Person { <1>
@Id String id;
String firstname;
String lastname;
}
----
[source,json]
----
{
"_class" : "com.example.Person", <1>
"id" : "cb7bef",
"firstname" : "Sarah",
"lastname" : "Connor"
}
----
<1> By default the domain types class name is used for the type hint.
====
Type hints can be configured to hold custom information.
Use the `@TypeAlias` annotation to do so.
NOTE: Make sure to add types with `@TypeAlias` to the initial entity set (`AbstractElasticsearchConfiguration#getInitialEntitySet`) to already have entity information available when first reading data from the store.
.Type Hints with Alias
====
[source,java]
----
@TypeAlias("human") <1>
public class Person {
@Id String id;
// ...
}
----
[source,json]
----
{
"_class" : "human", <1>
"id" : ...
}
----
<1> The configured alias is used when writing the entity.
====
NOTE: Type hints will not be written for nested Objects unless the properties type is `Object`, an interface or the actual value type does not match the properties declaration.
[[disabling-type-hints]]
===== Disabling Type Hints
It may be necessary to disable writing of type hints when the index that should be used already exists without having the type hints defined in its mapping and with the mapping mode set to strict.
In this case, writing the type hint will produce an error, as the field cannot be added automatically.
Type hints can be disabled for the whole application by overriding the method `writeTypeHints()` in a configuration class derived from `AbstractElasticsearchConfiguration` (see xref:elasticsearch/clients.adoc[Elasticsearch Clients]).
As an alternative they can be disabled for a single index with the `@Document` annotation:
WARNING: We strongly advise against disabling Type Hints.
Only do this if you are forced to.
Disabling type hints can lead to documents not being retrieved correctly from Elasticsearch in case of polymorphic data or document retrieval may fail completely.
Spring Data Elasticsearch supports the GeoJson types by providing an interface `GeoJson` and implementations for the different geometries.
They are mapped to Elasticsearch documents according to the GeoJson specification.
The corresponding properties of the entity are specified in the index mappings as `geo_shape` when the index mappings is written. (check the https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html[Elasticsearch documentation] as well)
For values inside Collections apply the same mapping rules as for aggregate roots when it comes to _type hints_ and xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.conversions[Custom Conversions].
For values inside Maps apply the same mapping rules as for aggregate roots when it comes to _type hints_ and xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.conversions[Custom Conversions].
However the Map key needs to a String to be processed by Elasticsearch.
Looking at the `Configuration` from the xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model[previous section] `ElasticsearchCustomConversions` allows registering specific rules for mapping domain and simple types.
.Meta Model Object Mapping Configuration
====
[source,java]
----
@Configuration
public class Config extends ElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.build();
}
@Bean
@Override
public ElasticsearchCustomConversions elasticsearchCustomConversions() {
return new ElasticsearchCustomConversions(
Arrays.asList(new AddressToMap(), new MapToAddress())); <1>
}
@WritingConverter <2>
static class AddressToMap implements Converter<Address, Map<String, Object>> {
@Override
public Map<String, Object> convert(Address source) {
LinkedHashMap<String, Object> target = new LinkedHashMap<>();
target.put("ciudad", source.getCity());
// ...
return target;
}
}
@ReadingConverter <3>
static class MapToAddress implements Converter<Map<String, Object>, Address> {
@Override
public Address convert(Map<String, Object> source) {
`ReactiveElasticsearchOperations` is the gateway to executing high level commands against an Elasticsearch cluster using the `ReactiveElasticsearchClient`.
The `ReactiveElasticsearchTemplate` is the default implementation of `ReactiveElasticsearchOperations`.
To get started the `ReactiveElasticsearchOperations` needs to know about the actual client to work with.
Please see xref:elasticsearch/clients.adoc#elasticsearch.clients.reactiverestclient[Reactive Rest Client] for details on the client and how to configure it.
[[elasticsearch.reactive.operations.usage]]
== Reactive Operations Usage
`ReactiveElasticsearchOperations` lets you save, find and delete your domain objects and map those objects to documents stored in Elasticsearch.
This chapter includes details of the Elasticsearch repository implementation.
.The sample `Book` entity
====
[source,java]
----
@Document(indexName="books")
class Book {
@Id
private String id;
@Field(type = FieldType.Text)
private String name;
@Field(type = FieldType.Text)
private String summary;
@Field(type = FieldType.Integer)
private Integer price;
// getter/setter ...
}
----
====
[[elasticsearch.repositories.autocreation]]
== Automatic creation of indices with the corresponding mapping
The `@Document` annotation has an argument `createIndex`.
If this argument is set to true - which is the default value - Spring Data Elasticsearch will during bootstrapping the repository support on application startup check if the index defined by the `@Document` annotation exists.
If it does not exist, the index will be created and the mappings derived from the entity's annotations (see xref:elasticsearch/object-mapping.adoc[Elasticsearch Object Mapping]) will be written to the newly created index.
Details of the index that will be created can be set by using the `@Setting` annotation, refer to xref:elasticsearch/misc.adoc#elasticsearc.misc.index.settings[Index settings] for further information.
The `@Highlight` annotation on a repository method defines for which fields of the returned entity highlighting should be included.To search for some text in a `Book` 's name or summary and have the found data highlighted, the following repository method can be used:
It is possible to define multiple fields to be highlighted like above, and both the `@Highlight` and the `@HighlightField` annotation can further be customized with a `@HighlightParameters` annotation. Check the Javadocs for the possible configuration options.
In the search results the highlight data can be retrieved from the `SearchHit` class.
<1> The `EnableElasticsearchRepositories` annotation activates the Repository support.
If no base package is configured, it will use the one of the configuration class it is put on.
<2> Provide a Bean named `elasticsearchTemplate` of type `ElasticsearchOperations` by using one of the configurations shown in the xref:elasticsearch/template.adoc[Elasticsearch Operations] chapter.
<3> Let Spring inject the Repository bean into your class.
====
[[elasticsearch.namespace]]
== Spring Namespace
The Spring Data Elasticsearch module contains a custom namespace allowing definition of repository beans as well as elements for instantiating a `ElasticsearchServer` .
Using the `repositories` element looks up Spring Data repositories as described in xref:repositories/create-instances.adoc[].
.Setting up Elasticsearch repositories using Namespace
The Elasticsearch module supports all basic query building feature as string queries, native search queries, criteria based queries or have it being derived from the method name.
[[elasticsearch.query-methods.finders.declared]]
=== Declared queries
Deriving the query from the method name is not always sufficient and/or may result in unreadable method names.
In this case one might make use of the `@Query` annotation (see xref:elasticsearch/repositories/elasticsearch-repository-queries.adoc#elasticsearch.query-methods.at-query[Using the @Query Annotation] ).
Another possibility is the use of a search-template, (see xref:elasticsearch/repositories/elasticsearch-repository-queries.adoc#elasticsearch.query-methods.at-searchtemplate-query[Using the @SearchTemplateQuery Annotation] ).
[[elasticsearch.query-methods.criterions]]
== Query creation
Generally the query creation mechanism for Elasticsearch works as described in xref:repositories/query-methods-details.adoc[].
Here's a short example of what a Elasticsearch query method translates into:
The String that is set as the annotation argument must be a valid Elasticsearch JSON query.
It will be sent to Easticsearch as value of the query element; if for example the function is called with the parameter _John_, it would produce the following query body:
[source,json]
----
{
"query": {
"match": {
"name": {
"query": "John"
}
}
}
}
----
====
.`@Query` annotation on a method taking a Collection argument
would make an https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html[IDs query] to return all the matching documents.
So calling the method with a `List` of `["id1", "id2", "id3"]` would produce the query body
[source,json]
----
{
"query": {
"ids": {
"values": ["id1", "id2", "id3"]
}
}
}
----
====
[[elasticsearch.query-methods.at-query.spel]]
=== Using SpEL Expressions
.Declare query on the method using the `@Query` annotation with SpEL expression.
====
https://docs.spring.io/spring-framework/reference/core/expressions.html[SpEL expression] is also supported when defining query in `@Query`.
We can pass `new QueryParameter("John")` as the parameter now, and it will produce the same query string as above.
====
.accessing bean property.
====
https://docs.spring.io/spring-framework/reference/core/expressions/language-ref/bean-references.html[Bean property] is also supported to access.
Given that there is a bean named `queryParameter` of type `QueryParameter`, we can access the bean with symbol `@` rather than `#`, and there is no need to declare a parameter of type `QueryParameter` in the query method:
NOTE: collection values should not be quoted when declaring the elasticsearch json query.
A collection of `names` like `List.of("name1", "name2")` will produce the following terms query:
[source,json]
----
{
"bool":{
"must":[
{
"terms":{
"name": ["name1", "name2"]
}
}
]
}
}
----
====
.access property in the `Collection` param.
====
https://docs.spring.io/spring-framework/reference/core/expressions/language-ref/collection-projection.html[SpEL Collection Projection] is convenient to use when values in the `Collection` parameter is not plain `String`:
When using Elasticsearch search templates - (see xref:elasticsearch/misc.adoc#elasticsearch.misc.searchtemplates [Search Template support]) it is possible to specify that a repository method should use a template by adding the `@SearchTemplateQuery` annotation to that method.
Let's assume that there is a search template stored with the name "book-by-title" and this template need a parameter named "title", then a repository method using that search template can be defined like this:
The parameters of the repository method are sent to the seacrh template as key/value pairs where the key is the parameter name and the value is taken from the actual value when the method is invoked.
Reactive Elasticsearch repository support builds on the core repository support explained in xref:repositories.adoc[] utilizing operations provided via xref:elasticsearch/reactive-template.adoc[] executed by a xref:elasticsearch/clients.adoc#elasticsearch.clients.reactiverestclient[Reactive REST Client].
Spring Data Elasticsearch reactive repository support uses https://projectreactor.io/[Project Reactor] as its reactive composition library of choice.
There are 3 main interfaces to be used:
* `ReactiveRepository`
* `ReactiveCrudRepository`
* `ReactiveSortingRepository`
[[elasticsearch.reactive.repositories.usage]]
== Usage
To access domain objects stored in a Elasticsearch using a `Repository`, just create an interface for it.
Before you can actually go on and do that you will need an entity.
.Sample `Person` entity
====
[source,java]
----
public class Person {
@Id
private String id;
private String firstname;
private String lastname;
private Address address;
// … getters and setters omitted
}
----
====
NOTE: Please note that the `id` property needs to be of type `String`.
.Basic repository interface to persist Person entities
When Elasticsearch stores a document in an index that has multiple shards, it determines the shard to you use based on the _id_ of the document.
Sometimes it is necessary to predefine that multiple documents should be indexed on the same shard (join-types, faster search for related data).
For this Elasticsearch offers the possibility to define a routing, which is the value that should be used to calculate the shard from instead of the _id_.
Spring Data Elasticsearch supports routing definitions on storing and retrieving data in the following ways:
[[elasticsearch.routing.join-types]]
== Routing on join-types
When using join-types (see xref:elasticsearch/join-types.adoc[Join-Type implementation]), Spring Data Elasticsearch will automatically use the `parent` property of the entity's `JoinField` property as the value for the routing.
This is correct for all the use-cases where the parent-child relationship has just one level.
If it is deeper, like a child-parent-grandparent relationship - like in the above example from _vote_ -> _answer_ -> _question_ - then the routing needs to explicitly specified by using the techniques described in the next section (the _vote_ needs the _question.id_ as routing value).
[[elasticsearch.routing.custom]]
== Custom routing values
To define a custom routing for an entity, Spring Data Elasticsearch provides a `@Routing` annotation (reusing the `Statement` class from above):
====
[source,java]
----
@Document(indexName = "statements")
@Routing("routing") <.>
public class Statement {
@Id
private String id;
@Field(type = FieldType.Text)
private String text;
@JoinTypeRelations(
relations =
{
@JoinTypeRelation(parent = "question", children = {"answer", "comment"}),
@JoinTypeRelation(parent = "answer", children = "vote")
}
)
private JoinField<String> relation;
@Nullable
@Field(type = FieldType.Keyword)
private String routing; <.>
// getter/setter...
}
----
<.> This defines _"routing"_ as routing specification
<.> a property with the name _routing_
====
If the `routing` specification of the annotation is a plain string and not a SpEL expression, it is interpreted as the name of a property of the entity, in the example it's the _routing_ property.
The value of this property will then be used as the routing value for all requests that use the entity.
It is also possible to us a SpEL expression in the `@Document` annotation like this:
====
[source,java]
----
@Document(indexName = "statements")
@Routing("@myBean.getRouting(#entity)")
public class Statement{
// all the needed stuff
}
----
====
In this case the user needs to provide a bean with the name _myBean_ that has a method `String getRouting(Object)`. To reference the entity _"#entity"_ must be used in the SpEL expression, and the return value must be `null` or the routing value as a String.
If plain property's names and SpEL expressions are not enough to customize the routing definitions, it is possible to define provide an implementation of the `RoutingResolver` interface. This can then be set on the `ElasticOperations` instance:
The `withRouting()` functions return a copy of the original `ElasticsearchOperations` instance with the customized routing set.
When a routing has been defined on an entity when it is stored in Elasticsearch, the same value must be provided when doing a _get_ or _delete_ operation. For methods that do not use an entity - like `get(ID)` or `delete(ID)` - the `ElasticsearchOperations.withRouting(RoutingResolver)` method can be used like this:
Spring Data Elasticsearch supports scripted fields and runtime fields.
Please refer to the Elasticsearch documentation about scripting (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) and runtime fields (https://www.elastic.co/guide/en/elasticsearch/reference/8.9/runtime.html) for detailed information about this.
In the context of Spring Data Elasticsearch you can use
* scripted fields that are used to return fields that are calculated on the result documents and added to the returned document.
* runtime fields that are calculated on the stored documents and can be used in a query and/or be returned in the search result.
The following code snippets will show what you can do (these show imperative code, but the reactive implementation works similar).
[[the-person-entity]]
== The person entity
The enity that is used in these examples is a `Person` entity.
This entity has a `birthDate` and an `age` property.
Whereas the birthdate is fix, the age depends on the time when a query is issued and needs to be calculated dynamically.
<.> the `age` property will be calculated and filled in search results.
<.> a convenience constructor to set up the test data.
====
Note that the `age` property is annotated with `@ScriptedField`.
This inhibits the writing of a corresponding entry in the index mapping and marks the property as a target to put a calculated field from a search response.
[[the-repository-interface]]
== The repository interface
The repository used in this example:
====
[source,java]
----
public interface PersonRepository extends ElasticsearchRepository<Person, String> {
var result1 = operations.search(query, Person.class); <.>
// variant 2: use the repository <.>
var result2 = repository.findByGenderAndAgeLessThanEqual(gender, maxAge, runtimeField);
return result1;
}
}
----
<.> define the runtime field that calculates the age of a person. // see https://asciidoctor.org/docs/user-manual/#builtin-attributes for builtin attributes.
<.> when using `Query`, add the runtime field.
<.> when adding a scripted field to a `Query`, an additional field parameter is needed to have the calculated value returned.
<.> when adding a scripted field to a `Query`, an additional source filter is needed to also retrieve the _normal_ fields from the document source.
<.> get the data filtered with the query and where the returned entites have the age property set.
<.> when using the repository, all that needs to be done is adding the runtime field as method parameter.
====
In addition to define a runtime fields on a query, they can also be defined in the index by setting the `runtimeFieldsPath` property of the `@Mapping` annotation to point to a JSON file that contains the runtime field definitions.
Spring Data Elasticsearch uses several interfaces to define the operations that can be called against an Elasticsearch index (for a description of the reactive interfaces see xref:elasticsearch/reactive-template.adoc[]).
* javadoc:org.springframework.data.elasticsearch.core.IndexOperations[] defines actions on index level like creating or deleting an index.
* javadoc:org.springframework.data.elasticsearch.core.DocumentOperations[] defines actions to store, update and retrieve entities based on their id.
* javadoc:org.springframework.data.elasticsearch.core.SearchOperations[] define the actions to search for multiple entities using queries
* javadoc:org.springframework.data.elasticsearch.core.ElasticsearchOperations[] combines the `DocumentOperations` and `SearchOperations` interfaces.
These interfaces correspond to the structuring of the https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html[Elasticsearch API].
The default implementations of the interfaces offer:
* index management functionality.
* Read/Write mapping support for domain types.
* A rich query and criteria api.
* Resource management and Exception translation.
[NOTE]
====
.Index management and automatic creation of indices and mappings.
The `IndexOperations` interface and the provided implementation which can be obtained from an `ElasticsearchOperations` instance - for example with a call to `operations.indexOps(clazz)`- give the user the ability to create indices, put mappings or store template and alias information in the Elasticsearch cluster.
Details of the index that will be created can be set by using the `@Setting` annotation, refer to xref:elasticsearch/misc.adoc#elasticsearc.misc.index.settings[Index settings] for further information.
**None of these operations are done automatically** by the implementations of `IndexOperations` or `ElasticsearchOperations`.
It is the user's responsibility to call the methods.
There is support for automatic creation of indices and writing the mappings when using Spring Data Elasticsearch repositories, see xref:elasticsearch/repositories/elasticsearch-repositories.adoc#elasticsearch.repositories.autocreation[Automatic creation of indices with the corresponding mapping]
====
[[elasticsearch.operations.usage]]
== Usage examples
The example shows how to use an injected `ElasticsearchOperations` instance in a Spring REST controller.
The example assumes that `Person` is a class that is annotated with `@Document`, `@Id` etc (see xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations[Mapping Annotation Overview]).
public String save(@RequestBody Person person) { <.>
Person savedEntity = elasticsearchOperations.save(person);
return savedEntity.getId();
}
@GetMapping("/person/{id}")
public Person findById(@PathVariable("id") Long id) { <.>
Person person = elasticsearchOperations.get(id.toString(), Person.class);
return person;
}
}
----
<.> Let Spring inject the provided `ElasticsearchOperations` bean in the constructor.
<.> Store some entity in the Elasticsearch cluster.
The id is read from the returned entity, as it might have been null in the `person` object and been created by Elasticsearch.
<.> Retrieve the entity with a get by id.
====
To see the full possibilities of `ElasticsearchOperations` please refer to the API documentation.
[[elasticsearch.operations.searchresulttypes]]
== Search Result Types
When a document is retrieved with the methods of the `DocumentOperations` interface, just the found entity will be returned.
When searching with the methods of the `SearchOperations` interface, additional information is available for each entity, for example the _score_ or the _sortValues_ of the found entity.
In order to return this information, each entity is wrapped in a `SearchHit` object that contains this entity-specific additional information.
These `SearchHit` objects themselves are returned within a `SearchHits` object which additionally contains informations about the whole search like the _maxScore_ or requested aggregations or the execution duration it took to complete the request.
The following classes and interfaces are now available:
.SearchHit<T>
Contains the following information:
* Id
* Score
* Sort Values
* Highlight fields
* Inner hits (this is an embedded `SearchHits` object containing eventually returned inner hits)
* The retrieved entity of type <T>
.SearchHits<T>
Contains the following information:
* Number of total hits
* Total hits relation
* Maximum score
* A list of `SearchHit<T>` objects
* Returned aggregations
* Returned suggest results
.SearchPage<T>
Defines a Spring Data `Page` that contains a `SearchHits<T>` element and can be used for paging access using repository methods.
.SearchScrollHits<T>
Returned by the low level scroll API functions in `ElasticsearchRestTemplate`, it enriches a `SearchHits<T>` with the Elasticsearch scroll id.
.SearchHitsIterator<T>
An Iterator returned by the streaming functions of the `SearchOperations` interface.
.ReactiveSearchHits
`ReactiveSearchOperations` has methods returning a `Mono<ReactiveSearchHits<T>>`, this contains the same information as a `SearchHits<T>` object, but will provide the contained `SearchHit<T>` objects as a `Flux<SearchHit<T>>` and not as a list.
[[elasticsearch.operations.queries]]
== Queries
Almost all of the methods defined in the `SearchOperations` and `ReactiveSearchOperations` interface take a `Query` parameter that defines the query to execute for searching. `Query` is an interface and Spring Data Elasticsearch provides three implementations: `CriteriaQuery`, `StringQuery` and `NativeQuery`.
[[elasticsearch.operations.criteriaquery]]
=== CriteriaQuery
`CriteriaQuery` based queries allow the creation of queries to search for data without knowing the syntax or basics of Elasticsearch queries.
They allow the user to build queries by simply chaining and combining `Criteria` objects that specify the criteria the searched documents must fulfill.
NOTE: when talking about AND or OR when combining criteria keep in mind, that in Elasticsearch AND are converted to a **must** condition and OR to a **should**
`Criteria` and their usage are best explained by example (let's assume we have a `Book` entity with a `price` property):
.Get books with a given price
====
[source,java]
----
Criteria criteria = new Criteria("price").is(42.0);
Query query = new CriteriaQuery(criteria);
----
====
Conditions for the same field can be chained, they will be combined with a logical AND:
.Get books with a given price
====
[source,java]
----
Criteria criteria = new Criteria("price").greaterThan(42.0).lessThan(34.0);
Query query = new CriteriaQuery(criteria);
----
====
When chaining `Criteria`, by default a AND logic is used:
.Get all persons with first name _James_ and last name _Miller_:
====
[source,java]
----
Criteria criteria = new Criteria("lastname").is("Miller") <1>
.and("firstname").is("James") <2>
Query query = new CriteriaQuery(criteria);
----
<1> the first `Criteria`
<2> the and() creates a new `Criteria` and chaines it to the first one.
====
If you want to create nested queries, you need to use subqueries for this.
Let's assume we want to find all persons with a last name of _Miller_ and a first name of either _Jack_ or _John_:
.Nested subqueries
====
[source,java]
----
Criteria miller = new Criteria("lastName").is("Miller") <.>
.subCriteria( <.>
new Criteria().or("firstName").is("John") <.>
.or("firstName").is("Jack") <.>
);
Query query = new CriteriaQuery(criteria);
----
<.> create a first `Criteria` for the last name
<.> this is combined with AND to a subCriteria
<.> This sub Criteria is an OR combination for the first name _John_
<.> and the first name Jack
====
Please refer to the API documentation of the `Criteria` class for a complete overview of the different available operations.
[[elasticsearch.operations.stringquery]]
=== StringQuery
This class takes an Elasticsearch query as JSON String.
The following code shows a query that searches for persons having the first name "Jack":
Using `StringQuery` may be appropriate if you already have an Elasticsearch query to use.
[[elasticsearch.operations.nativequery]]
=== NativeQuery
`NativeQuery` is the class to use when you have a complex query, or a query that cannot be expressed by using the `Criteria` API, for example when building queries and using aggregates.
It allows to use all the different `co.elastic.clients.elasticsearch._types.query_dsl.Query` implementations from the Elasticsearch library therefore named "native".
The following code shows how to search for persons with a given `firstName` and for the found documents have a terms aggregation that counts the number of occurrences of the `lastName` for these persons:
====
[source,java]
----
Query query = NativeQuery.builder()
.withAggregation("lastNames", Aggregation.of(a -> a
The following table shows the Elasticsearch and Spring versions that are used by Spring Data release trains and the version of Spring Data Elasticsearch included in that.
[cols="^,^,^,^",options="header"]
|===
| Spring Data Release Train | Spring Data Elasticsearch | Elasticsearch | Spring Framework
| 2025.1 (in development) | 6.0.x | 9.1.0 | 7.0.x
| 2025.0 | 5.5.x | 8.18.1 | 6.2.x
| 2024.1 | 5.4.x | 8.15.5 | 6.1.x
| 2024.0 | 5.3.xfootnote:oom[Out of maintenance] | 8.13.4 | 6.1.x
Support for upcoming versions of Elasticsearch is being tracked and general compatibility should be given assuming the usage of the xref:elasticsearch/template.adoc[ElasticsearchOperations interface].
BioMed Central Development Team; Oliver Drotbohm; Greg Turnquist; Christoph Strobl; Peter-Josef Meisch
(C) 2008-{copyright-year} VMware, Inc.
Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.
One of the changes in version 4.0.x is that Spring Data Elasticsearch does not use the Jackson Mapper anymore to map an entity to the JSON representation needed for Elasticsearch (see xref:elasticsearch/object-mapping.adoc[Elasticsearch Object Mapping]).
In version 3.2.x the Jackson Mapper was the default that was used.
It was possible to switch to the meta-model based converter (named `ElasticsearchEntityMapper`) by explicitly configuring it (xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model[Meta Model Object Mapping]).
In version 4.0.x the meta-model based converter is the only one that is available and does not need to be configured explicitly.
If you had a custom configuration to enable the meta-model converter by providing a bean like this:
[source,java]
----
@Bean
@Override
public EntityMapper entityMapper() {
ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(
elasticsearchMappingContext(), new DefaultConversionService()
You now have to remove this bean, the `ElasticsearchEntityMapper` interface has been removed.
.Entity configuration
Some users had custom Jackson annotations on the entity class, for example in order to define a custom name for the mapped document in Elasticsearch or to configure date conversions.
These are not taken into account anymore.
The needed functionality is now provided with Spring Data Elasticsearch's `@Field` annotation.
Please see xref:elasticsearch/object-mapping.adoc#elasticsearch.mapping.meta-model.annotations[Mapping Annotation Overview] for detailed information.
== Removal of implicit index name from query objects
In 3.2.x the different query classes like `IndexQuery` or `SearchQuery` had properties that were taking the index name or index names that they were operating upon.If these were not set, the passed in entity was inspected to retrieve the index name that was set in the `@Document` annotation. +
In 4.0.x the index name(s) must now be provided in an additional parameter of type `IndexCoordinates`.By separating this, it now is possible to use one query object against different indices.
To make it easier to work with entities and use the index name that is contained in the entitie's `@Document` annotation, new methods have been added like `DocumentOperations.save(T entity)`;
In version 3.2 there was the `ElasticsearchOperations` interface that defined all the methods for the `ElasticsearchTemplate` class. In version 4 the functions have been split into different interfaces, aligning these interfaces with the Elasticsearch API:
* `DocumentOperations` are the functions related documents like saving, or deleting
* `SearchOperations` contains the functions to search in Elasticsearch
* `IndexOperations` define the functions to operate on indexes, like index creation or mappings creation.
`ElasticsearchOperations` now extends `DocumentOperations` and `SearchOperations` and has methods get access to an `IndexOperations` instance.
NOTE: All the functions from the `ElasticsearchOperations` interface in version 3.2 that are now moved to the `IndexOperations` interface are still available, they are marked as deprecated and have default implementations that delegate to the new implementation:
[source,java]
----
/**
* Create an index for given indexName.
*
* @param indexName the name of the index
* @return {@literal true} if the index was created
* @deprecated since 4.0, use {@link IndexOperations#create()}
Since version 7 the Elasticsearch `TransportClient` is deprecated, it will be removed with Elasticsearch version 8. Spring Data Elasticsearch deprecates the `ElasticsearchTemplate` class which uses the `TransportClient` in version 4.0.
Mapping types were removed from Elasticsearch 7, they still exist as deprecated values in the Spring Data `@Document` annotation and the `IndexCoordinates` class but they are not used anymore internally.
[[elasticsearch-migration-guide-3.2-4.0.removal]]
== Removals
* As already described, the `ElasticsearchEntityMapper` interface has been removed.
* The `SearchQuery` interface has been merged into it's base interface `Query`, so it's occurrences can just be replaced with `Query`.
* The method `org.springframework.data.elasticsearch.core.ElasticsearchOperations.query(SearchQuery query, ResultsExtractor<T> resultsExtractor);` and the `org.springframework.data.elasticsearch.core.ResultsExtractor` interface have been removed.
These could be used to parse the result from Elasticsearch for cases in which the response mapping done with the Jackson based mapper was not enough.
Since version 4.0, there are the new xref:elasticsearch/template.adoc#elasticsearch.operations.searchresulttypes[Search Result Types] to return the information from an Elasticsearch response, so there is no need to expose this low level functionality.
* The low level methods `startScroll`, `continueScroll` and `clearScroll` have been removed from the `ElasticsearchOperations` interface.
For low level scroll API access, there now are `searchScrollStart`, `searchScrollContinue` and `searchScrollClear` methods on the `ElasticsearchRestTemplate` class.
The parameters of the `@Document` annotation that are relevant for the index settings (`useServerConfiguration`, `shards`. `replicas`, `refreshIntervall` and `indexStoretype`) have been moved to the `@Setting` annotation. Use in `@Document` is still possible but deprecated.
[[elasticsearch-migration-guide-4.1-4.2.removal]]
== Removals
The `@Score` annotation that was used to set the score return value in an entity was deprecated in version 4.0 and has been removed.
Score values are returned in the `SearchHit` instances that encapsulate the returned entities.
The `org.springframework.data.elasticsearch.ElasticsearchException` class has been removed.
The remaining usages have been replaced with `org.springframework.data.mapping.MappingException` and `org.springframework.dao.InvalidDataAccessApiUsageException`.
The deprecated `ScoredPage`, `ScrolledPage` `@AggregatedPage` and implementations has been removed.
The deprecated `GetQuery` and `DeleteQuery` have been removed.
The deprecated `find` methods from `ReactiveSearchOperations` and `ReactiveDocumentOperations` have been removed.
It was possible in 4.1 to configure the refresh policy for the `ReactiveElasticsearchTemplate` by overriding the method `AbstractReactiveElasticsearchConfiguration.refreshPolicy()` in a custom configuration class.
The return value of this method was an instance of the class `org.elasticsearch.action.support.WriteRequest.RefreshPolicy`.
Now the configuration must return `org.springframework.data.elasticsearch.core.RefreshPolicy`.
This enum has the same values and triggers the same behaviour as before, so only the `import` statement has to be adjusted.
`ElasticsearchOperations` and `ReactiveElasticsearchOperations` now explicitly use the `RefreshPolicy` set on the template for write requests if not null.
If the refresh policy is null, then nothing special is done, so the cluster defaults are used. `ElasticsearchOperations` was always using the cluster default before this version.
The provided implementations for `ElasticsearchRepository` and `ReactiveElasticsearchRepository` will do an explicit refresh when the refresh policy is null.
This is the same behaviour as in previous versions.
If a refresh policy is set, then it will be used by the repositories as well.
When configuring Spring Data Elasticsearch like described in xref:elasticsearch/clients.adoc[Elasticsearch Clients] by using `ElasticsearchConfigurationSupport`, `AbstractElasticsearchConfiguration` or `AbstractReactiveElasticsearchConfiguration` the refresh policy will be initialized to `null`.
Previously the reactive code initialized this to `IMMEDIATE`, now reactive and non-reactive code show the same behaviour.
The reactive methods previously returned a `Mono<Long>` with the number of deleted documents, the non reactive versions were void. They now return a `Mono<ByQueryResponse>` which contains much more detailed information about the deleted documents and errors that might have occurred.
The implementations of _multiget_ previousl only returned the found entities in a `List<T>` for non-reactive implementations and in a `Flux<T>` for reactive implementations. If the request contained ids that were not found, the information that these are missing was not available. The user needed to compare the returned ids to the requested ones to find
which ones were missing.
Now the `multiget` methods return a `MultiGetItem` for every requested id. This contains information about failures (like non existing indices) and the information if the item existed (then it is contained in the `MultiGetItem) or not.
This section describes breaking changes from version 4.2.x to 4.3.x and how removed features can be replaced by new introduced features.
[NOTE]
====
Elasticsearch is working on a new Client that will replace the `RestHighLevelClient` because the `RestHighLevelClient` uses code from Elasticsearch core libraries which are not Apache 2 licensed anymore.
Spring Data Elasticsearch is preparing for this change as well.
This means that internally the implementations for the `*Operations` interfaces need to change - which should be no problem if users program against the interfaces like `ElasticsearchOperations` or `ReactiveElasticsearchOperations`.
If you are using the implementation classes like `ElasticsearchRestTemplate` directly, you will need to adapt to these changes.
Spring Data Elasticsearch also removes or replaces the use of classes from the `org.elasticsearch` packages in it's API classes and methods, only using them in the implementation where the access to Elasticsearch is implemented.
For the user that means, that some enum classes that were used are replaced by enums that live in `org.springframework.data.elasticsearch` with the same values, these are internally mapped onto the Elasticsearch ones.
Places where classes are used that cannot easily be replaced, this usage is marked as deprecated, we are working on replacements.
Check the sections on xref:migration-guides/migration-guide-4.2-4.3.adoc#elasticsearch-migration-guide-4.2-4.3.deprecations[Deprecations] and xref:migration-guides/migration-guide-4.2-4.3.adoc#elasticsearch-migration-guide-4.2-4.3.breaking-changes[Breaking Changes] for further details.
In `SearchOperations`, and so in `ElasticsearchOperations` as well, the `suggest` methods taking a `org.elasticsearch.search.suggest.SuggestBuilder` as argument and returning a `org.elasticsearch.action.search.SearchResponse` have been deprecated.
Use `SearchHits<T> search(Query query, Class<T> clazz)` instead, passing in a `NativeSearchQuery` which can contain a `SuggestBuilder` and read the suggest results from the returned `SearchHit<T>`.
In `ReactiveSearchOperations` the new `suggest` methods return a `Mono<org.springframework.data.elasticsearch.core.suggest.response.Suggest>` now.
=== Removal of `org.elasticsearch` classes from the API.
* In the `org.springframework.data.elasticsearch.annotations.CompletionContext` annotation the property `type()` has changed from `org.elasticsearch.search.suggest.completion.context.ContextMapping.Type` to `org.springframework.data.elasticsearch.annotations.CompletionContext.ContextMappingType`, the available enum values are the same.
* In the `org.springframework.data.elasticsearch.annotations.Document` annotation the `versionType()` property has changed to `org.springframework.data.elasticsearch.annotations.Document.VersionType`, the available enum values are the same.
* In the `org.springframework.data.elasticsearch.core.query.Query` interface the `searchType()` property has changed to `org.springframework.data.elasticsearch.core.query.Query.SearchType`, the available enum values are the same.
* In the `org.springframework.data.elasticsearch.core.query.Query` interface the return value of `timeout()` was changed to `java.time.Duration`.
* The `SearchHits<T>`class does not contain the `org.elasticsearch.search.aggregations.Aggregations` anymore.
Instead it now contains an instance of the `org.springframework.data.elasticsearch.core.AggregationsContainer<T>` class where `T` is the concrete aggregations type from the underlying client that is used.
Currently this will be a `org
.springframework.data.elasticsearch.core.clients.elasticsearch7.ElasticsearchAggregations` object; later different implementations will be available.
The same change has been done to the `ReactiveSearchOperations.aggregate()` functions, the now return a `Flux<AggregationContainer<?>>`.
Programs using the aggregations need to be changed to cast the returned value to the appropriate class to further proces it.
* methods that might have thrown a `org.elasticsearch.ElasticsearchStatusException` now will throw `org.springframework.data.elasticsearch.RestStatusException` instead.
=== Handling of field and sourceFilter properties of Query
Up to version 4.2 the `fields` property of a `Query` was interpreted and added to the include list of the `sourceFilter`.
This was not correct, as these are different things for Elasticsearch.
This has been corrected.
As a consequence code might not work anymore that relies on using `fields` to specify which fields should be returned from the document's `_source' and should be changed to use the `sourceFilter`.
Spring Data Elasticsearch now uses `org.springframework.data.elasticsearch.core.query.IndicesOptions` instead of `org.elasticsearch.action.support.IndicesOptions`.
The classes from the package `org.springframework.data.elasticsearch.core.completion` have been moved to `org.springframework.data.elasticsearch.core.suggest`.
The `org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentPropertyConverter` interface has been renamed to `org.springframework.data.elasticsearch.core.mapping.PropertyValueConverter`.
Likewise the implementations classes named _XXPersistentPropertyConverter_ have been renamed to _XXPropertyValueConverter_.
The method `<T> Publisher<T> execute(ClientCallback<Publisher<T>> callback)` has been deprecated.
As there now are multiple implementations using different client libraries the `execute` method is still available in the different implementations, but there is no more method in the interface, because there is no common callback interface for the different clients.
==== `org.springframework.data.elasticsearch.core.ElasticsearchTemplate` has been removed
As of version 4.4 Spring Data Elasticsearch does not use the `TransportClient` from Elasticsearch anymore (which itself is deprecated since Elasticsearch 7.0).
This means that the `org.springframework.data.elasticsearch.core.ElasticsearchTemplate` class which was deprecated since Spring Data Elasticsearch 4.0 has been removed.
This was the implementation of the `ElasticsearchOperations` interface that was using the `TransportClient`.
Connections to Elasticsearch must be made using either the imperative `ElasticsearchRestTemplate` or the reactive `ReactiveElasticsearchTemplate`.
In 4.3 two classes (`ElasticsearchAggregations` and `ElasticsearchAggregation`) had been moved to the `org.springframework.data.elasticsearch.core.clients.elasticsearch7` package in preparation for the integration of the new Elasticsearch client.
The were moved back to the `org.springframework.data.elasticsearch.core` package as we keep the classes use the old Elasticsearch client where they were.
The `ReactiveElasticsearchTemplate`, when created directly or by Spring Boot configuration had a default refresh policy of IMMEDIATE.
This could cause performance issues on heavy indexing and was different than the default behaviour of Elasticsearch.
This has been changed to that now the default refresh policy is NONE.
When the
`ReactiveElasticsearchTemplate` was provided by using the configuration like described in xref:elasticsearch/clients.adoc#elasticsearch.clients.reactiverestclient[Reactive REST Client] the default refresh policy already was set to NONE.
Elasticsearch has introduced it's new `ElasticsearchClient` and has deprecated the previous `RestHighLevelClient`.
Spring Data Elasticsearch 4.4 still uses the old client as the default client for the following reasons:
* The new client forces applications to use the `jakarta.json.spi.JsonProvider` package whereas Spring Boot will stick to `javax.json.spi.JsonProvider` until version 3. So switching the default implementaiton in Spring Data Elasticsearch can only come with Spring Data Elasticsearch 5 (Spring Data 3, Spring 6).
* There are still some bugs in the Elasticsearch client which need to be resolved
* The implementation using the new client in Spring Data Elasticsearch is not yet complete, due to limited resources working on that - remember Spring Data Elasticsearch is a community driven project that lives from public contributions.
CAUTION: The implementation using the new client is not complete, some operations will throw a `java.lang.UnsupportedOperationException` or might throw NPE (for example when the Elasticsearch cannot parse a response from the server, this still happens sometimes) +
Use the new client to test the implementations but do not use it in productive code yet!
In order to try and use the new client the following steps are necessary:
Remove Spring Data Elasticsearch related properties from your application configuration.
If Spring Data Elasticsearch was configured using a programmatic configuration (see xref:elasticsearch/clients.adoc[Elasticsearch Clients]), remove these beans from the Spring application context.
In order configure Spring Data Elasticsearch to use the new client, it is necessary to create a configuration bean that derives from `org.springframework.data.elasticsearch.client.elc.ElasticsearchConfiguration`:
====
[source,java]
----
@Configuration
public class NewRestClientConfig extends ElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.build();
}
}
----
====
The configuration is done in the same way as with the old client, but it is not necessary anymore to create more than the configuration bean.
With this configuration, the following beans will be available in the Spring application context:
* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
* an `ElasticsearchClient` bean, this is the new client that uses the `RestClient`
* an `ElasticsearchOperations` bean, available with the bean names _elasticsearchOperations_ and _elasticsearchTemplate_, this uses the `ElasticsearchClient`
To use the new client in a reactive environment the only difference is the class from which to derive the configuration:
====
[source,java]
----
@Configuration
public class NewRestClientConfig extends ReactiveElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.build();
}
}
----
====
With this configuration, the following beans will be available in the Spring application context:
* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
* an `ReactiveElasticsearchClient` bean, this is the new reactive client that uses the `RestClient`
* an `ReactiveElasticsearchOperations` bean, available with the bean names _reactiveElasticsearchOperations_ and _reactiveElasticsearchTemplate_, this uses the `ReactiveElasticsearchClient`
Logging by setting the property `logging.level.org.springframework.data.elasticsearch.client.WIRE=trace` is deprecated now, the Elasticsearch `RestClient` provides a better solution that can be activated by setting the logging level of the `tracer` package to "trace".
See xref:migration-guides/migration-guide-4.4-5.0.adoc#elasticsearch-migration-guide-4.4-5.0.breaking-changes-packages[Package changes], all classes in this package have been deprecated, as the default client implementations to use are the ones based on the new Java Client from Elasticsearch, see xref:migration-guides/migration-guide-4.4-5.0.adoc#elasticsearch-migration-guide-4.4-5.0.new-clients[New Elasticsearch client]
All the classes that are using or depend on the deprecated Elasticsearch `RestHighLevelClient` have been moved to the package `org.springframework.data.elasticsearch.client.erhlc`.
By this change we now have a clear separation of code using the old deprecated Elasticsearch libraries, code using the new Elasticsearch client and code that is independent of the client implementation.
Also the reactive implementation that was provided up to now has been moved here, as this implementation contains code that was copied and adapted from Elasticsearch libraries.
If you are using `ElasticsearchRestTemplate` directly and not the `ElasticsearchOperations` interface you'll need to adjust your imports as well.
When working with the `NativeSearchQuery` class, you'll need to switch to the `NativeQuery` class, which can take a
`Query` instance coming from the new Elasticsearch client libraries.
To configure Spring Data Elasticsearch to use the new client, it is necessary to create a configuration bean that derives from `org.springframework.data.elasticsearch.client.elc.ElasticsearchConfiguration`:
====
[source,java]
----
@Configuration
public class NewRestClientConfig extends ElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.build();
}
}
----
====
The configuration is done in the same way as with the old client, but it is not necessary anymore to create more than the configuration bean.
With this configuration, the following beans will be available in the Spring application context:
* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
* an `ElasticsearchClient` bean, this is the new client that uses the `RestClient`
* an `ElasticsearchOperations` bean, available with the bean names _elasticsearchOperations_ and _elasticsearchTemplate_, this uses the `ElasticsearchClient`
To use the new client in a reactive environment the only difference is the class from which to derive the configuration:
====
[source,java]
----
@Configuration
public class NewRestClientConfig extends ReactiveElasticsearchConfiguration {
@Override
public ClientConfiguration clientConfiguration() {
return ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.build();
}
}
----
====
With this configuration, the following beans will be available in the Spring application context:
* a `RestClient` bean, that is the configured low level `RestClient` that is used by the Elasticsearch client
* an `ReactiveElasticsearchClient` bean, this is the new reactive client that uses the `RestClient`
* an `ReactiveElasticsearchOperations` bean, available with the bean names _reactiveElasticsearchOperations_ and _reactiveElasticsearchTemplate_, this uses the `ReactiveElasticsearchClient`
The old deprecated `RestHighLevelClient` can still be used, but you will need to add the dependency explicitly to your application as Spring Data Elasticsearch does not pull it in automatically anymore:
====
[source,xml]
----
<!-- include the RHLC, specify version explicitly -->
In the `org.springframework.data.elasticsearch.core.index.AliasData` class, which is used for alias information returned from Elasticsearch, the property `filter` (of type `Document`) is replaced by `filterQuery` which is of type
`org.springframework.data.elasticsearch.annotations.Similarity` was an enum class until 5.1. This enum was used in the `@Field` annotation to specify a similarity value.
But besides the values defined by the enum, it is possible to have similarities with custom names in Elasticsearch.
Therefore, the annotation property was changed from the type of the enum to a simple `String`.
The previous enum values like `Similarity.Default` do still exist as String constants, so existing code will compile unmodified.
Adaptions are necessary when this enum was used at other places than as a property of the `@Field` annotation.
In the `org.springframework.data.elasticsearch.BulkFailureException` class, the return type of the `getFailedDocuments` is changed from `Map<String, String>`
to `Map<String, FailureDetails>`, which allows to get additional details about failure reasons.
The definition of the `FailureDetails` class (inner to `BulkFailureException`):
[source,java]
public record FailureDetails(Integer status, String errorMessage) {
}
[[scripted-and-runtime-fields]]
=== scripted and runtime fields
The classes `org.springframework.data.elasticsearch.core.RuntimeField` and `org.springframework.data.elasticsearch.core.query.ScriptType` have been moved to the subpackage `org.springframework.data.elasticsearch.core.query`.
The `type` parameter of the `ScriptData` constructor is not nullable any longer.
* All the code using the old deprecated `RestHighLevelClient` has been removed.
The default Elasticsearch client used since version 5.0 is the (not so) new Elasticsearch Java client.
* The `org.springframework.data.elasticsearch.client.ClientLogger` class has been removed.
This logger was configured with the `org.springframework.data.elasticsearch.client.WIRE` setting, but was not working with all clients.
From version 5 on, use the trace logger available in the Elasticsearch Java client, see xref:elasticsearch/clients.adoc#elasticsearch.clients.logging[Client Logging].
* The method `org.springframework.data.elasticsearch.core.ElasticsearchOperations.stringIdRepresentation(Object)` has been removed, use the `convertId(Object)` method defined in the same interface instead.
* The class `org.springframework.data.elasticsearch.core.Range` has been removed, use `org.springframework.data.domain.Range` instead.
* The methods `org.springframework.data.elasticsearch.core.query.IndexQuery.getParentId() and `setParentId(String)` have been removed, they weren't used anymore and were no-ops.
It has been removed from the `org.springframework.data.elasticsearch.core.query.IndexQuery` class as well.
During the parameter replacement in `@Query` annotated repository methods previous versions wrote the String `"null"` into the query that was sent to Elasticsearch when the actual parameter value was `null`.
As Elasticsearch does not store `null` values, this behaviour could lead to problems, for example whent the fields to be searched contains the string `"null"`.
In Version 5.3 a `null` value in a parameter will cause a `ConversionException` to be thrown.
If you are using `"null"` as the
`null_value` defined in a field mapping, then pass that string into the query instead of a Java `null`.
The deprecated classes `org.springframework.data.elasticsearch.ELCQueries`
and `org.springframework.data.elasticsearch.client.elc.QueryBuilders` have been removed, use `org.springframework.data.elasticsearch.client.elc.Queries` instead.
The `withKnnQuery` method in `NativeQueryBuilder` has been replaced with `withKnnSearches` to build a `NativeQuery` with knn search.
`KnnQuery` and `KnnSearch` are two different classes in elasticsearch java client and are used for different queries, with different parameters supported:
- `KnnSearch`: is https://www.elastic.co/guide/en/elasticsearch/reference/8.13/search-search.html#search-api-knn[the top level `knn` query] in the elasticsearch request;
- `KnnQuery`: is https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-knn-query.html[the `knn` query inside `query` clause];
If `KnnQuery` is still preferable, please be sure to construct it inside `query` clause manually, by means of `withQuery(co.elastic.clients.elasticsearch._types.query_dsl.Query query)` clause in `NativeQueryBuilder`.
From version 6.0 on, Spring Data Elasticsearch uses the Elasticsearch 9 libraries and as default the new `Rest5Client` provided by these libraries. It is still possible to use the old `RestClient`, check xref:elasticsearch/clients.adoc[Elasticsearch clients] for information. The configuration callbacks for this `RestClient` have been moved from `org.springframework.data.elasticsearch.client.elc.ElasticsearchClients` to the `org.springframework.data.elasticsearch.client.elc.rest_client.RestClients` class.
In the `org.springframework.data.elasticsearch.core.query.UpdateQuery` class the type of the two fields `ifSeqNo` and `ifPrimaryTerm` has changed from `Integer` to `Long` to align with the normal query and the underlying Elasticsearch client.
All the code using the old `RestClient` has been moved to the `org.springframework.data.elasticsearch.client.elc.rest_client` package and has been deprecated. Users should switch to the classes from the `org.springframework.data.elasticsearch.client.elc.rest5_client` package.
=== Removals
The `org.springframework.data.elasticsearch.core.query.ScriptType` enum has been removed. To distinguish between an inline and a stored script set the appropriate values in the `org.springframework.data.elasticsearch.core.query.ScriptData` record.
These methods have been removed because the Elasticsearch Client 9 does not support them anymore:
This chapter explains the basic foundations of Spring Data repositories and Elasticsearch specifics.
Before continuing to the Elasticsearch specifics, make sure you have a sound understanding of the basic concepts.
The goal of the Spring Data repository abstraction is to significantly reduce the amount of boilerplate code required to implement data access layers for various persistence stores.
NOTE: Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.
The Spring Data Elasticsearch project applies core Spring concepts to the development of solutions using the Elasticsearch Search Engine. We have povided a "template" as a high-level abstraction for storing,querying,sorting and faceting documents. You will notice similarities to the Spring data solr and mongodb support in the Spring Framework.
[[project]]
[preface]
== Project Metadata
* Version Control - https://github.com/spring-projects/spring-data-elasticsearch
Requires http://www.elasticsearch.org/download/[Elasticsearch] 0.20.2 and above or optional dependency or not even that if you are using Embedded Node Client
This chapter includes details of the Elasticsearch repository implementation.
[[elasticsearch.introduction]]
== Introduction
[[elasticsearch.namespace]]
=== Spring Namespace
The Spring Data Elasticsearch module contains a custom namespace allowing definition of repository beans as well as elements for instantiating a `ElasticsearchServer` .
Using the `repositories` element looks up Spring Data repositories as described in <<repositories.create-instances>> .
.Setting up Elasticsearch repositories using Namespace
public ElasticsearchOperations elasticsearchTemplate() {
return new ElasticsearchTemplate(nodeBuilder().local(true).node().client());
}
}
----
====
The configuration above sets up an `Embedded Elasticsearch Server` which is used by the `ElasticsearchTemplate` . Spring Data Elasticsearch Repositories are activated using the `@EnableElasticsearchRepositories` annotation, which essentially carries the same attributes as the XML namespace does. If no base package is configured, it will use the one the configuration class resides in.
[[elasticsearch.cdi]]
=== Elasticsearch Repositores using CDI
The Spring Data Elasticsearch repositories can also be set up using CDI functionality.
.Spring Data Elasticsearch repositories using JavaConfig
====
[source,java]
----
class ElasticsearchTemplateProducer {
@Produces
@ApplicationScoped
public ElasticsearchOperations createElasticsearchTemplate() {
return new ElasticsearchTemplate(nodeBuilder().local(true).node().client());
}
}
class ProductService {
private ProductRepository repository;
public Page<Product> findAvailableBookByName(String name, Pageable pageable) {
public void setRepository(ProductRepository repository) {
this.repository = repository;
}
}
----
====
[[elasticsearch.query-methods]]
== Query methods
[[elasticsearch.query-methods.finders]]
=== Query lookup strategies
The Elasticsearch module supports all basic query building feature as String,Abstract,Criteria or have it being derived from the method name.
==== Declared queries
Deriving the query from the method name is not always sufficient and/or may result in unreadable method names. In this case one might make either use of `@Query` annotation (see <<elasticsearch.query-methods.at-query>> ).
[[elasticsearch.query-methods.criterions]]
=== Query creation
Generally the query creation mechanism for Elasticsearch works as described in <<repositories.query-methods>> . Here's a short example of what a Elasticsearch query method translates into:
.Query creation from method names
====
[source,java]
----
public interface BookRepository extends Repository<Book, String>
This chapter covers additional support for Elasticsearch operations that cannot be directly accessed via the repository interface. It is recommended to add those operations as custom implementation as described in <<repositories.custom-implementations>> .
Elasticsearch has scan and scroll feature for getting big result set in chunks. `ElasticsearchTemplate` has scan and scroll methods that can be used as below.
.Using Scan and Scroll
====
[source,java]
----
SearchQuery searchQuery = new NativeSearchQueryBuilder()
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.