This commit disalbes HTTP for all the suite and test scope tests
since it's an unused / unneeded module which takes time to startup.
This also uses a JVM private port range for HTTP ports to ensure
there are no cross JVM conflicts.
We rely on retry logic when reading a snapshot since it's concurrently
serialized. We should move to a better logic here but the refactoring
of the blobstore change the semantics and this now throws Json
exceptions rather than returning an unexpected Token
When executing a bulk request, with create index operation and auto generate id, if while the primary is relocating the bulk is executed, and the relocation is done while N items from the bulk have executed, the full shard bulk request will be retried on the new primary. This can create duplicates because the request is not makred as potentially holding conflicts.
This change carries over the response for each item on the request level, and if a conflict is detected on the primary shard, and the response is there (indicating that the request was executed once already), use the mentioned response as the actual response for that bulk shard item.
On top of that, when a primary fails and is retried, the change now marks the request as potentially causing duplicates, so the actual impl will do the extra lookup needed.
This change also fixes a bug in our exception handling on the replica, where if a specific item failed, and its not an exception we can ignore, we should actually cause the shard to fail.
closes#7729
When a node is elected as master or receives a join request, we submit a cluster state update task. We should give the node join update task a lower priority than the elect as master to increase the chance it will not be rejected. During master election there is a big chance that these will happen concurrently.
This commit lowers the priority of node joins from IMMEDIATE to URGENT
Closes#7733
Some tests like CorruptedTranslogTests rely on the fact that we
are recovering from translog. In those cases we need to prevent
flushes from happening during indexing. This change adds an optional
flag on the #indexRandom utility to disable flushes.
ActionNamesTests#testIncomingAction rarely uses a random action name to make sure that actions registered via plugins work properly. In some cases the random action would conflict with existing one (e.g. tv) and make the test fail. Fixed also testOutgoingAction although the probability of conflict there is way lower due to longer action names used from 1.4 on.
Most notably the elected_as_master task should run as soon as possible. This is an issue as node join request do use `Priority.IMMEDIATE` and can be unjustly rejected.
Closes#7718
It uses a cluster state update task and it gets rejected if not run on a master node. We should enable running on non-masters if the local flag is set.
Also, report any unexpected error that may happen during this cluster state update task
Closes#7731
Similar to NettyTransport.doStop() all actions which disconnect
from a node (and thus call awaitUnterruptibly) should not be executed
on the I/O thread.
This patch ensures that all disconnects happen in the generic threadpool, trying to avoid unnecessary `disconnectFromNode` calls.
Also added a missing return statement in case the component was not yet
started when catching an exception on the netty layer.
Closes#7726
The Unicast Zen Ping mechanism is configured to ping certain host:port combinations in order to discover other node. Since this is only a ping, we do not setup a full connection but rather do a light connect with one channel. This light connection is closed at the end of the pinging.
During pinging, we may discover disco nodes which are not yet connected (via temporalResponses). UnicastZenPing will setup the same light connection for those node. However, during pinging a cluster state may arrive with those nodes in it. In that case , we will mistakenly believe those nodes are connected and at the end of pinging we will mistakenly disconnect those valid node.
This commit makes sure that all nodes UnicastZenPing connects to have a unique id and can be safely disconnected.
Closes#7719
When cancelling recoveries, we wait for up to 10s for the source node to be notified before continuing. This is not needed in two cases:
1) The source node has been disconnected due to node shutdown (recovery is canceled as a response to cluster state processing)
2) The current thread is the one that will be notifying the source node (happens when one of the calls from the source nodes discoveres the local index is closed)
The first one is especially important as it may delay cluster state update processing with 10s.
Closes#7717
Make the http addresses within the REST client final. It makes no sense to update them before each test if we don't check the version of the nodes again, which would mean adding too much overhead (an additional http call before each test) for no reason. We just reuse the same nodes for the whole suite and check the version once while initializing the client. Would be nice to make the REST client within the execution context final but its initialization still needs to happen after the `ElasticsearchIntegrationTest#beforeInternal` that assigns `GLOBAL_CLUSTER` to `currentCluster`.
Closes#7723
When communicating with 1.3 and earlier nodes, it's possible that the
field data breaker info is not sent at all. When this happens, we should
leave the `breaker` variable as-is (unset) instead of creating an
AllCircuitBreakerStats object with a null fd breaker and fake request &
parent breakers.
The terms aggregation can now support sorting on multiple criteria by replacing the sort object with an array or sort object whose order signifies the priority of the sort. The existing syntax for sorting on a single criteria also still works.
Contributes to #6917
Replaces #7588
Also fix the test
FunctionScoreTests#simpleWeightedFunctionsTestWithRandomWeightsAndRandomCombineMode
which sometimes failed due to rounding issues. Make sure
only floats are returned as scores to assure ratio of
expected and returned score is 1.0f.
The test starts a cluster with random nodes as unicast hosts but *doesn't* use min_master_nodes. If the unicast hosts are started last, nodes may elect themselves as master as they do not have mechanism yet to share information.
ElasticsearchRestTests has now a `restClientSettings` method that can be overriden to provide headers as settings (similarly to what we do with transport client). Those headers will be sent together with every REST requests within the tests.
Closes#7710
Adds a special case to the GeoBoundingBoxFilterParser so that the left of the box is not normalised in the case where left and right are 360 apart. Before this change the left would be normalised to 180 in this case and the filter would only match points with a longitude of 180 (or -180).
Closes#5128
GroupShardsIterator is used in many places like the search execution
to determin which shards to query. This can hold shards of one index
as well as shards of multiple indices. The iteration order is used
to assigne a per-request shard ID for each shard that is used as a
tie-breaker when scores are the same. Today the iteration order is
soely depending on the HashMap iteration order which is undefined or
rather implementation dependent. This causes search requests to return
inconsistent results across requests if for instance different nodes
are coordinating the requests.
Simple queries like `match_all` may return results in arbitrary order
if pagination is used or may even return different results for the same
request even though there hasn't been a refresh call and preferences are
used.
Today if we run into exception like NumberFormatException or IAE
when we try to open a commit point to retrieve checksums and calculate
store metadata we just bubble them up. Yet, those are very likely index
corruptions. In such a case we should really mark the shard as
corrupted.
In the reduce logic of the top_hits aggregation if the first shard result to process contained has no results then the merging of all the shard results can go wrong resulting in an incorrect sorted hits.
This bug can only manifest with a sort other than score.
Closes#7697
If you have previously corrupted files, this method currently builds an
exception like:
```
failed engine [corrupted preexisting index]
failed to start shard
```
Followed by a CorruptIndexException. This commit writes the entire
stacktrace to provide additional information. It also changes the
failure message from `corrupted preexisting index` to `preexisting
corrupted index` to prevent confusion.
Closes#7596
Simon Willnauer
Shay Banon
Boaz Leskes
javanna
Ryan Ernst
Alexander Reelsen
mikemccand
uboness
Lee Hinman
Colin Goodheart-Smithe
Britta Weber
Philipp Jardas
Martijn van Groningen
markharwood
Clinton Gormley