This PR enables Ingest plugins to leverage processor-scoped REST
endpoints. First of which being the Grok endpoint that retrieves
Grok Patterns for users to retrieve all the built-in patterns.
Example usage: Kibana Grok Autocomplete!
When we disabled `_all` by default for indices created in 6.0, we missed adding
a layer that would handle the situation where `_all` was not enabled in 5.x and
then the cluster was updated to 6.0, this means that when the cluster was
updated the `_all` field would be disabled for 5.x indices and field values
would not be added to the `_all` field.
This adds a compatibility layer for 5.x indices where we treat the default
enabled value for the `_all` field to be `true` if unset on 5.x indices.
Resolves#25068
The Log4j dependency is separated into two artifacts, the API and the
core implementation. This is to enable replacing Log4j on the backend
through the SLF4J bridge with another logging implementation. For this
reason, the dependencies are marked as optional. This causes confusion
amongst users as to use the bridge, the API should be non-optional since
it is needed for the bridge to function correctly. While they could pull
it into their application directly, it would be clearer if we simply
marked this depdendency as non-optional. Note that this does not mean
that users have to use Log4j for logging in their application, so we are
not marking core as required, it only clarifies what they need to be
able to plug in a different logging implementation.
Relates #25136
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Previously this would output:
```
GET /test-1/_mappings
{ }
```
And after this change:
```
GET /test-1/_mappings
{
"test-1": {
"mappings": {}
}
}
```
To bring parity back to the REST output after #24723.
Relates to #25090
Previously in #24723 we changed the `_alias` API to not go through the
`RestGetIndicesAction` endpoint, instead creating a `RestGetAliasesAction` that
did the same thing.
This changes the formatting so that it matches the old formatting of the
endpoint, before:
```
GET /test-1/_alias
{ }
```
And after this change:
```
GET /test-1/_alias
{
"test-1": {
"aliases": {}
}
}
```
This is related to #25090
GeoUtils#isValidLongitude is inconsistent with GeoUtils#isValidLatitude.
Neither technically need the isInfinite() check because they then compare
against min and max values.
When parsing resonses we should be ignoring any new unknown fields or inner
objects in most cases to be forward compatible with changes in core on the
client side. This change adds test for this for QueryProfileShardResult and
nested substructures and changes the parsing code where necessary to be able to
ignore new fields and objects in the xContent.
Test: randomVersionBetween works with unreleased
Modifies randomVersionBetween so that it works with unreleased
versions. This should make switching a version from unreleased
to released much simpler.
Added common base class for ScriptDocValues.Strings and ScriptDocValues.BytesRefs now that these classes are very similar.
Also cleaned up the BinaryDVFieldDataTests:
* Use junit assertions instead of hamcrest
* Use BytesRef directly instead of byte[]
Closes#24785
The FVH fails with an NPE when a match phrase prefix is rewritten in an empty phrase query.
This change makes sure that the multi match query rewrites to a MatchNoDocsQuery (instead of an empty phrase query) when there is
a single term and that term does not expand to any term in the index.
Fixes#25088
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.
Relates #6720
This commit modifies query_string, simple_query_string and multi_match queries to always use a DisjunctionMaxQuery when a disjunction over multiple fields is built. The tiebreaker is set to 1 in order to behave like the boolean query in terms of scoring.
The removal of the coord factor in Lucene 7 made this change mandatory to correctly handle minimum_should_match.
Closes#23966
This change extracts the main logic from `TransportClearScrollAction`
into a new class `ClearScrollController` and adds a corresponding unit test.
Relates to #25094
The `scorerSupplier` API allows to give a hint to queries in order to let them
know that they will be consumed in a random-access fashion. We should use this
for aggregations, function_score and matched queries.
This commit moves the assumeFalse() calls that implement test skipping
and blacklisting out of the @Before method of ESClientYamlSuiteTestCase.
The problem with having them in the @Before method is that if an
assumption triggers then the @Before methods of classes that extend
ESClientYamlSuiteTestCase will not run, but their @After methods will.
This can lead to inconsistencies that cause assertions in the @After
methods and fail the test even though it was skipped/blacklisted.
Instead the assumeFalse() calls are now at the beginning of the test()
method, which runs after all @Before methods (including those in classes
that extend ESClientYamlSuiteTestCase) have completed. The only side
effect is that overridden test() methods in classes that extend
ESClientYamlSuiteTestCase which call super.test() and also do other things
must now be designed not to consume any InternalAssumptionViolatedException
that may be thrown by the super.test() call.
Relates elastic/x-pack-elasticsearch#1650
When we open a translog, we rely on the `translog.ckp` file to tell us what the maximum generation file should be and on the information stored in the last lucene commit to know the first file we need to recover. This requires coordination and is currently subject to a race condition: if a node dies after a lucene commit is made but before we remove the translog generations that were unneeded by it, the next time we open the translog we will ignore those files and never delete them (I have added tests for this).
This PR changes the approach to have the translog store both of those numbers in the `translog.ckp`. This means it's more self contained and easier to control.
This change also decouples the translog recovery logic from the specific commit we're opening. This prepares the ground to fully utilize the deletion policy introduced in #24950 and store more translog data that's needed for Lucene, keep multiple lucene commits around and be free to recover from any of them.
For the response parsing we want to be lenient when it comes to parsing
new xContent fields. In order to ensure this in our testing, this change
adds a utility method to XContentTestUtils that takes xContent bytes
representation as input and recursively a random field on each object
level.
Sometimes we also want to exclude a whole subtree from this treatment
(e.g. skipping "_source"), other times an element (e.g. "fields", "highlight"
in SearchHit) can have arbitraryly named objects. Those cases can be
specified as exceptions.
This commit removes wrapper methods on QueryShardContext used to compile
scripts. Instead, the script service is made accessible in the context,
and calls to compile can be made directly. This will ease transition to
each of those location becoming their own context, since they would no
longer be able to expect the same script class type.
Splits TranslogRecoveryPerformer into three parts:
- the translog operation to engine operation converter
- the operation perfomer (that indexes the operation into the engine)
- the translog statistics (for which there is already RecoveryState.Translog)
This makes it possible for peer recovery to use the same IndexShard interface as bulk shard requests (i.e. Engine operations instead of Translog operations). It also pushes the "fail on bad mapping" logic outside of IndexShard. Future pull requests could unify the BulkShard and peer recovery path even more.
The unified highlighter rewrites MultiPhrasePrefixQuery to SpanNearQuer even when there is a single term in the phrase.
Though SpanNearQuery throws an exception when the number of clauses is less than 2.
This change returns a simple PrefixQuery when there is a single term and builds the SpanNearQuery otherwise.
Relates #25088
Pins the random testing seed at build start rather than letting
it vary with every randomized testing invocation. This is useful
for projects where random decisions in one randomized testing run
can effect the outcome of a second randomized testing run such as
the full cluster restart tests.
The goal isn't for tests to be able to assume that random decision
will be the same in both tests. It is more to make sure that the
seed printed when a test fails reproduces the appropriate random
decisions. And pinning the seed at startup should do just that.
This works by taking the key passed as a system property if one
is passed, otherwise picking a random long and getting it into
appropriate key format. The build just calls
`new Random().nextLong()` to get the seed while randomized testing
uses a Murmur3 hash of `System.nanoTime`.
This change skips rest tests that use mutlitple types if the cluster
is a pure 6.x cluster. This allows all indics to be created with a version
less than 6.0 and that means we can safely use the `mapping.single_type` setting.
Relates to #24961
The PR takes a different approach to solve #24806 than currently implemented via #25052. The `refreshMetric` that IndexShard maintains is updated using the refresh listeners infrastructure in lucene. This means that we truly count all refreshes that lucene makes and not have to worry about each individual caller (like `IndexShard@refresh` and `Engine#get()`)
parent/child: Allow updating mapping without specifying `_parent` field on each update.
Prior to this change when a mapping has a `_parent` field then any update (also updates that didn't modify the `_parent` field) to the mapping involved specifying the `_parent` field again. With this change specifying the `_parent` field on each mapping update is no longer required.
Closes#23381
We have a callback interface that is not needed because it is
effectively the same as java.util.function.Consumer. This commit removes
it.
Relates #25089
We use a callback in recovery land during primary relocation to ensure
the relocation target is on at least the same version as the relocation
source. This callback is typed as a Callback<Long> which is an
unnecessary custom type (we can use Consumer<T> or the appropriate
primitive callbacks). Here, we can use LongConsumer.
Relates #25081
Previous work modified the status code on the get aliases API when an
alias is missing so that these requests 404 now. This change was also
backported to 5.5 so we can adjust the skips to skip everything before
5.5.0.
Previously the HEAD and GET aliases endpoints were misaigned in
behavior. The HEAD verb would 404 if any aliases are missing while the
GET verb would not if any aliases existed. When HEAD was aligned with
GET, this broke the previous usage of HEAD to serve as an existence
check for aliases. It is the behavior of GET that is problematic here
though, if any alias is missing the request should 404. This commit
addresses this by modifying the behavior of GET to behave in this
way. This fixes the behavior for HEAD to also 404 when aliases are
missing.
Relates #25043
This change moves the parent_id query to the parent-join module and handles the case when only the parent-join field can be declared on an index (index with single type on).
If single type is off it uses the legacy parent join field mapper and switch to the new one otherwise (default in 6).
Relates #20257
This commit fixes the group methdos of Settings to properly include
grouped secure settings. Previously the secure settings were included
but without the group prefix being removed.
closes#25069
Deb Adair
Tal Levy
Lee Hinman
Guillaume Le Floch
Jason Tedor
jaymode
Lee Hinman
Eli Skeggs
Christoph Büscher
Nik Everett
Yannick Welsch
Martijn van Groningen
Jim Ferenczi
Simon Willnauer
Adrien Grand
David Roberts
Boaz Leskes
Jack Conradson
Ryan Ernst
Tim Brooks
Yu