The PR #21694 was initially planned to go into v6.0.0 and v5.1.0. Due to another PR relying on this one though for backport to v5.0.2, #21694 must go to v5.0.2
as well. As such, the initial backward compatibility rules established by the PR must be changed to include v5.0.2 and above.
As part of #20925 and #21341 we added an "all-fields" mode to the
`query_string` and `simple_query_string`. This would expand the query to
all fields and automatically set `lenient` to true.
However, we should still allow a user to override the `lenient` flag to
whichever value they desire, should they add it in the request. This
commit does that.
When Elasticsearch starts, we go through some initialization before we
install a security manager. Yet, the JVM makes internal policy decisions
on the basis of whether or not a security manager is present. This
commit installs a security manager immediately on startup so that the
JVM always thinks a security manager is present when making such policy
decisions.
Relates #21716
Today we read a vint from the stream to allocate the size of an array up-front
before we start reading the values. This can be dangerous if for instance we read
from a corrupted stream or if some manipulated bytes are send for instance from
an attacker or a fuzzer. In most of the cases we can apply some best effort and
validate the array size to be _sane_ by ensuring we can at read at least N bytes
where N is the expected size of the array.
* Add support for merging custom meta data in tribe node
Currently, when any underlying cluster has custom metadata
(via plugin), tribe node does not store custom meta data in its
cluster state. This is because the tribe node has no idea how to
select the appropriate custom metadata from one or many custom
metadata (corresponding to the number of underlying clusters).
This change adds an interface that custom metadata implementations
can extend to add support for merging mulitple custom metadata of
the same type for storing in the tribe state.
Relates to #20544
Supersedes #20791
* Simplify updating tribe state
* Add tests for merging multiple custom metadata types in tribe node
* cleanup merging custom md logic in tribe service
Today it's not possible to add exceptions to the serialization layer
without breaking BWC. This commit adds the ability to specify the Version
an exception was added that allows to fall back not NotSerializableExceptionWrapper
if the exception is not present in the streams version.
Relates to #21656
TransportSearchAction optimizes the search_type in certain cases, when for instance we are searching against a single shard, or when there is only a suggest section in the request. That optimization is wrapped in a try catch, and when an exception happens we log it and ignore it. This may be a leftover from the past though, as no exception is expected to be thrown in that code block, hence if there is any exception we are probably better off bubbling it up rather than ignoring it.
The `lookupPrototype` method is not used anywhere. Seems like we rather use its `lookupProrotypeSafe` variant (which also throws exception if the prototype is not found) is always. This commit makes the safer variant the default one, by renaming it to "lookupPrototype" and removes the previous "unsafe" variant.
Today we call `writeByte` up to 3x per character in each string written via
`StreamOutput#writeString` this can have quite some overhead when strings
are long or many strings are written. This change adds a local buffer to
convert chars to bytes into the local buffer. Converted bytes are then
written via `writeBytes` instead reducing the overhead of this opertion.
Closes#21660
The overflows were happening in two places, the parsing of the template that
implicitly truncates the `order` when its value does not fall into the `integer`
range, and the comparator that sorts templates in ascending order, since it
returns `order2-order1`, which might overflow.
Closes#21622
* Fix highlighting on a stored keyword field
The highlighter converts stored keyword fields using toString().
Since the keyword fields are stored as utf8 bytes the conversion is broken.
This change uses BytesRef.utf8toString() to convert the field value in a valid string.
Fixes#21636
* Replace BytesRef#utf8ToString with MappedFieldType#valueForDisplay
If the node name is explicitly set it's not derived from the node ID
meaning that it doesn't immediately appear in the logs. While it can be
tracked down in other places, it would be easier for info purposes if it
just showed up explicitly. This commit adds the node ID to the logs,
whether or not the node name is set.
Relates #21673
Plugins are closed if they implement java.io.Closeable but this is not
clear from the plugin interface. This commit clarifies this by declaring
that Plugins implement java.io.Closeable and adding an empty
implementation to the base Plugin class.
Relates #21669
AbstractScopedSettings has the ability to only apply updates/deletes
to dynamic settings. The flag is currently not respected when a setting
is reset/deleted which causes static node settings to be reset if a non-dynamic
key is reset via `null` value.
Closes#21593
This commit moves several allocation decider related inner classes
into their own top-level class, in order to use more easily in
the allocation explain API. This commit also renames some of those
decision related classes to more suitable names.
This is simply a cosmetic change - no functionality changes with this
commit whatsoever.
To summarize the changes:
1. ShardAllocationDecision renamed to AllocateUnassignedDecision
2. RelocationDecision moved to a top-level class
3. MoveDecision moved to a top-level class
4. RebalanceDecision moved to a top-level class
5. ShardAllocationDecisionTests renamed to AllocateUnassignedDecisionTests
6. NodeRebalanceResult moved to a top-level class
7. ShardAllocationDecision#WeightedDecision moved to a top-level class and renamed to NodeAllocationResult.
The "test" task can complete its execution with a timeout exception before the "block-task" actually starts executing. The test thus has to wait for both to be
completed before checking that the updateTasksPerExecutor map has been properly cleaned up.
According to the docs and our own tests we accept an ids query without specified
types and default to all types in the index mapping in this case. This changes
the builder to reflect this by making the types no longer a required constructor
argument and changes the parser to reflect that.
A first step moving away from the current parsing to use the generalized
Objectparser and ConstructingObjectParser. This PR start by making use of it in
MatchAllQueryBuilder and IdsQueryBuilder.
* Fix compilation in Eclipse
I'm not sure what the bug is, but ecj doesn't like this expression
unless the type is set explicitly.
* Add comment explaining why no diamond operator
This commit exposes the executor service interface from thread
pool. This will enable some high-level concurrency primitives that will
make some code cleaner and simpler.
Relates #21608
The default search timeout is not respected because the timeout is
unconditionally set from the query. This commit fixes this issue, and
adds a test that the default search timeout is correctly attached to the
search context.
Relates #21599
* master: (22 commits)
Add proper toString() method to UpdateTask (#21582)
Fix `InternalEngine#isThrottled` to not always return `false`. (#21592)
add `ignore_missing` option to SplitProcessor (#20982)
fix trace_match behavior for when there is only one grok pattern (#21413)
Remove dead code from GetResponse.java
Fixes date range query using epoch with timezone (#21542)
Do not cache term queries. (#21566)
Updated dynamic mapper section
Docs: Clarify date_histogram bucket sizes for DST time zones
Handle release of 5.0.1
Fix skip reason for stats API parameters test
Reduce skip version for stats API parameter tests
Strict level parsing for indices stats
Remove cluster update task when task times out (#21578)
[DOCS] Mention "all-fields" mode doesn't search across nested documents
InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
Fixed bad asciidoc in boolean mapping docs
Fixed bad asciidoc ID in node stats
Be strict when parsing values searching for booleans (#21555)
Fix time zone rounding edge case for DST overlaps
...
This change fixes the rnage query so that an exception is always thrown if the range query uses epoch time together with a time zone. Since epoch time is always UTC it should not be used with a time zone.
Closes#21501
There have been reports that the query cache did not manage to speed up search
requests when the query includes a large number of different sub queries since
a single request may manage to exhaust the whole history (256 queries) while
the query cache only starts caching queries once they appear multiple times in
the history (#16031). On the other hand, increasing the size of the query cache
is a bit controversial (#20116) so this pull request proposes a different
approach that consists of never caching term queries, and not adding them to the
history of queries either. The reasoning is that these queries should be fast
anyway, regardless of caching, so taking them out of the equation should not
cause any slow down. On the other hand, the fact that they are not added to the
cache history anymore means that other queries have greater chances of being
cached.
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.
Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
A previous commit added strict level parsing for the node stats API, but
that commit missed adding the same for the indices stats API. This
commit rectifies this miss.
Relates #21577
Fixes an issue where the cluster service does not remove an update task from its internal data structures that are used for batching cluster state updates.
* review comments
* checkstyle
This changes only the query parsing behavior to be strict when searching on
boolean values. We continue to accept the variety of values during index time,
but searches will only be parsed using `"true"` or `"false"`.
Resolves#21545
When using TimeUnitRounding with a DAY_OF_MONTH unit, failing tests in #20833
uncovered an issue when the DST shift happenes just one hour after midnight
local time and sets back the clock to midnight, leading to an overlap.
Previously this would lead to two different rounding values, depending on
whether a date before or after the transition was rounded. This change detects
this special case and correct for it by using the previous rounding date for
both cases.
Closes#20833
Today when parsing a stats request, Elasticsearch silently ignores
incorrect metrics. This commit removes lenient parsing of stats requests
for the nodes stats and indices stats APIs.
Relates #21417
#20960 removed `LocalDiscovery` and we now use `ZenDiscovery` in all our tests. To keep cluster forming fast, we are using a `MockZenPing` implementation which uses static maps to return instant results making master election fast. Currently, we don't set `minimum_master_nodes` causing the occasional split brain when starting multiple nodes concurrently and their pinging is so fast that it misses the fact that one of the node has elected it self master. To solve this, `InternalTestCluster` is modified to behave like a true cluster and manage and set `minimum_master_nodes` correctly with every change to the number of nodes.
Tests that want to manage the settings themselves can opt out using a new `autoMinMasterNodes` parameter to the `ClusterScope` annotation.
Having `min_master_nodes` set means the started node may need to wait for other nodes to be started as well. To combat this, we set `discovery.initial_state_timeout` to `0` and wait for the cluster to form once all node have been started. Also, because a node may wait and ping while other nodes are started, `MockZenPing` is adapted to wait rather than busy-ping.
Yannick Welsch
Lee Hinman
Areek Zillur
Nik Everett
Jason Tedor
Simon Willnauer
Tanguy Leroux
Luca Cavanna
Adrien Grand
Jim Ferenczi
David Roberts
javanna
Ali Beyad
Christoph Büscher
Dimitris Athanasiou
Tal Levy
Simon Willnauer
Colin Goodheart-Smithe
Lee Hinman
Boaz Leskes