Allows plugins to register pre-configured tokenizers. Much
of the decisions are the same as those in #24223, #24572,
and #24223. This only migrates the lowercase tokenizer but
I figure that is a good start because it proves out the features.
This change removes the field data specialization needed for the parent field and replaces it with
a simple DocValuesIndexFieldData. The underlying global ordinals are retrieved via a new function called
IndexOrdinalsFieldData#getOrdinalMap.
The children aggregation is also modified to use a simple WithOrdinals value source rather than the deleted WithOrdinals.Parent.
Relates #20257
Today when we get a metadata snapshot from the index shard we ensure
that if there is no engine started on the shard that we lock the index
writer before we go and fetch the store metadata. Yet, if we concurrently
recover that shard, recovery finalization might fail since it can't acquire
the IW lock on the directory. This is mainly due to the wrong order of aquiring
the IW lock and the metadata lock. Fetching store metadata without a started engine
should block on the metadata lock in Store.java but since IndexShard locks the writer
first we get into a failed recovery dance especially in test. In production
this is less of an issue since we rarely get into this siutation if at all.
Closes#24481
This commit moves some functionality from PublishClusterStateAction to ZenDiscovery, which allows each class to focus on it's core competencies:
- PendingStatesQueue is now solely managed by ZenDiscovery (no shared access by both PublishClusterStateAction and ZenDiscovery)
- Validation logic is handled exclusively by ZenDiscovery
This commit is a simple refactoring of the update shard logic for
primaries. Namely, there was some duplicated code here that was annoying
to have to read twice so it is now collapsed with this commit.
We have decided not to force a future version upgrade to deal with this todo. Rather, we'll keep the code until its in our way / the opportunity arises to deal with it.
Now that we generate the versions list from Versions.java we can
drop the list of versions maintained for vagrant testing. One nice
thing that the vagrant testing did was to check if the list of
versions was out of date. This moves that test to the core
project.
This PR revolves around places in the code where introducing a StringBuilder might make the construction
of a String easier to follow and also, maybe avoid a case where the compiler's very safe way of introducing
StringBuilder instead of String might not always be optimal for performance.
This fixes a bug in the 'date_histogram' aggregation that can happen when using 'extended_bounds'
together with some 'offset' parameter. Offsets should be applied after rounding the extended bounds
and also be applied when adding empty buckets during the reduce phase in InternalDateHistogram.
Closes#23776
Native scripts have been replaced in documentation by implementing
a ScriptEngine and they were deprecated in 5.5.0. This commit
removes the native script infrastructure for 6.0.
closes#19966
Shared settings were added intially to allow the few common settings
names across aws plugins. However, in 6.0 these settings have been
removed. The last use was in netty, but since 6.0 also has the netty 3
modules removed, there is no longer a need for the shared property. This
commit removes the shared setting property.
This commit adds a new method to the SearchOperationListener that allows implementers to validate
the SearchContext immediately after it is retrieved from the active contexts. The listener may
throw a runtime exception if it deems the SearchContext is not valid and that the use of the context
should be terminated.
* [TEST] Fix TransportReplicationActionTests.testRetryOnReplica for replica request
We were improperly testing that it was a `ConcreteShardRequest` instead of a
`ConcreteReplicaRequest`. This adds that change and also ensures that the
checkpoint is retrievable from the request.
* Fix line-length
Approaching the release of 6.0 we need to sort out the usage of
`Version#minimumCompatibilityVersion` which was still set to 5.0.0.
Now this change moves it to the latest released version of 5.x (5.4 at this point)
to ensure we are compatible with the latest minor of the previous major. This change
also removes all the `_UNRELEASED` from the versions that where released and drops versions
that were never released and are not expected to be released (bugfixes in minors that are not
the latest in the previous major).
Today the `_field_caps` API doesn't implement its request serialization
correctly since indices and indices options are not serialized at all.
This will likely break with all transport clients etc. and if this request
must be send across the network. This commit fixes this and adds correct
handling if we have only remote indices to prevent the inclusion of
all local indices.
* Fix ArrayIndexOutOfBoundsException in Range Aggregation when no ranges are specified in the query
* Revert "Fix ArrayIndexOutOfBoundsException in Range Aggregation when no ranges are specified in the query"
This reverts commit ad57d8feb3577a64b37de28c6f3df96a3a49fe93.
* Fix range aggregation out of bounds exception when there are no ranges in a range or date_range query
* Fix range aggregation out of bounds exception when there are no ranges in the query
This fix is applied to range queries, date range queries, ip range queries and geo distance aggregation queries
Native scripts are no longer documented and instead using a ScriptEngine
is recommended. This change adds a deprecation warning for removal in
6.0.
relates #19966
This PR adds a new thread pool type: `fixed_auto_queue_size`. This thread pool
behaves like a regular `fixed` threadpool, except that every
`auto_queue_frame_size` operations (default: 10,000) in the thread pool,
[Little's Law](https://en.wikipedia.org/wiki/Little's_law) is calculated and
used to adjust the pool's `queue_size` either up or down by 50. A minimum and
maximum is taken into account also. When the min and max are the same value, a
regular fixed executor is used instead.
The `SEARCH` threadpool is changed to use this new type of thread pool. However,
the min and max are both set to 1000, meaning auto adjustment is opt-in rather
than opt-out.
Resolves#3890
Moves the remaining preconfigured token figured into the analysis-common module. There were a couple of tests in core that depended on the pre-configured token filters so I had to touch them:
* `GetTermVectorsCheckDocFreqIT` depended on `type_as_payload` but didn't do anything important with it. I dropped the dependency. Then I moved the test to a single node test case because we're trying to cut down on the number of `ESIntegTestCase` subclasses.
* `AbstractTermVectorsTestCase` and its subclasses depended on `type_as_payload`. I dropped their usage of the token filter and added an integration test for the termvectors API that uses `type_as_payload` to the `analysis-common` module.
* `AnalysisModuleTests` expected a few pre-configured token filtes be registered by default. They aren't any more so I dropped this assertion. We assert that the `CommonAnalysisPlugin` registers these pre-built token filters in `CommonAnalysisFactoryTests`
* `SearchQueryIT` and `SuggestSearchIT` had tests that depended on the specific behavior of the token filters so I moved the tests to integration tests in `analysis-common`.
In scripts (at least some of the languages), the terms dictionary and
postings can be access with the special _index variable. This is for
very advanced use cases which want to do their own scoring. The problem
is segment level statistics must be recomputed for every document.
Additionally, this is not friendly to the terms index caching as the
order of looking up terms should be controlled by lucene.
This change removes _index from scripts. Anyone using it can and should
instead write a Similarity plugin, which is explicitly designed to allow
doing the calculations needed for a relevance score.
closes#19359
Today when an index is `read-only` the index is also blocked from
being deleted which sometimes is undesired since in-order to make
changes to a cluster indices must be deleted to free up space. This is
a likely scenario in a hosted environment when disk-space is limited to switch
indices read-only but allow deletions to free up space.
This moves the releasing logic to the base test, so that individual test cases don't need
to worry about releasing the aggregators. It's not a big deal for individual aggs,
but once tests start using sub-aggs, it can become tricky to free (without double-freeing)
all the aggregators.
Range queries with now based date ranges were previously not allowed,
but since #23921 these queries were allowed. This change should really
fix range queries with now based date ranges.
Adding a unit test to InternalAdjecencyMatrix that extends the shared InternalAggregationTestCase
that we use for testing aggregations.
Relates to #22278
The test check that the number of outgoing/incoming recoveries of a shard is 0 after recoveries were done. Sadly that is not guaranteed by the current recovery logic as we decrement the counters only when all references to the relevant RecoveryTarget object have been released. This may happen in an async fashion to the recovery completion which causes the test to fail. I looked at options to change the recovery logic to have the recovery counters decrease before the recovery is done *under normally circumstances* but I don't see a clean way to do it. Since it won't give hard guarantees anyway I opted to add assertBusy to the test
Closes#24669
This commit adds a deprecation warning if `_index` is used in scripts.
It is emitted each time a script is invoked, but not per document. There
is no test because constructing a LeafIndexLookup is quite difficult,
but the deprecation warning does show up in IndexLookupIT, there is just
no way to assert warnings in integ tests.
relates #19359
Today we assert hart if failure listeners are invoked more than once. Yet, this
can happen if we cancel the execution since the caller and the handler will get
the exception on the cancelable threads and will notify the listener concurrently
if timinig allows. This commit relaxes the assertion towards handling multiple
invocations with `ExecutionCancelledException`
Closes#24010Closes#24179
Closes vagnerclementino/elasticsearch/#98
Template script engines (mustache, the only one) currently return a
BytesReference that users must know is utf8 encoded. This commit
modifies all callers and mustache to have the template engine return
String. This is much simpler, and does not require decoding in order to
use (for example, in ingest).
When retrieving documents to extract terms from as part of a more like this query, the _routing value can be set, yet it gets lost. That leads to not being able to retrieve the documents, hence more_like_this used to return no matches all the time.
Closes#23699
We generally accept string values when a boolean is expected. We've been doing that in our parsing code, but we missed that bit when moving parsing code to ObjectParser, which throws an error instead. This commit makes ObjectParser parse also string values into booleans. It throws an error in case the value is not `true` or `false`.
Closes#21802
The rendering methods in String and Long Significant String aggregations
and buckets are very similar. They can be factored out in the
InternalSignificantTerms class an InternalMappedSignificantTerms class.
This commit changes SignificantTerms.Bucket so that it is not an
abstract class anymore but an interface. It will be easier for the Java
High Level Rest Client to provide its own implementation of
SignificantTerms and SignificantTerms.Bucket. Also, it is now more
coherent with the others aggregations.
The disruption tests sit in a single test suite which causes these tests
to be single-threaded. We can split this test suite into multiple suites
(logically, of course) enabling them to be run in parallel reducing the
total run time of all integration tests in core. This commit splits the
discovery with service disruptions test suite into three suites
- master disruptions
- discovery disruptions
- cluster disruptions
The last one could probably be better named, it is meant to represent
performing actions in the cluster (indexing, failing a shard, etc.)
while a disruption is taking place.
Relates #24662
Jason Tedor
Nik Everett
Luca Cavanna
Nicholas Knize
Jim Ferenczi
Simon Willnauer
Tanguy Leroux
Yannick Welsch
Jack Conradson
Boaz Leskes
Koen De Groote
Christoph Büscher
Ryan Ernst
Jay Modi
Lee Hinman
George Papadrosou
Lee Hinman
Zachary Tong
Martijn van Groningen
Igor Motov