Today any call to the current randomized context modifies the random
sequence such that cluster initialization is context dependent. If due
to an error for instance a static util method is used like `randomLong`
inside the TestCluster instead of the provided Random instance all
reproducibility guarantees are gone. This commit adds a safe mechanism
to initialize these clusters even if a static helper is used. All none
test scope clusters are now initialized in a private randomized context.
Today we simply close the IW when we have to open a new one if ie.
we change merge settings. Yet, the API is deprecated and under the hood
it commits the IW. For safety we should add the current translog ID and
use rollback instead to not wait for merges.
One can set the headers sent with request by the clients by setting the `request.headers` setting. This commit enables overriding any such set headers directly on the requests.
In case the multicast binding fails (due to socket errors), abort zen
pinging code and throw a better error message
Relates #8225
(cherry picked from commit f819cff77a0ef95b340afc2f22e3464283803960)
Break-out common functionality between elasticsearch.bat and service.bat
Relates #8237
(cherry picked from commit 87095e79534bcef97b2b275322c924bd96b34e3b)
Fixes a bug where alias creation would allow `null` for index name, which thereby
applied the alias to _all_ indices. This patch makes the validator throw an
exception if the index is null.
```bash
POST /_aliases
{
"actions": [
{
"add": {
"alias": "empty-alias",
"index": null
}
}
]
}
```
```json
{
"error": "ActionRequestValidationException[Validation Failed: 1: Alias action [add]: [index] may not be null;]",
"status": 400
}
```
The reason this bug wasn't caught by the existing tests is because
the old test for nullness only validated against a cluster which had
zero indices. The null index is translated into "_all", and since
there are no indices, this fails because the index doesn't exist.
So the test passes.
However, as soon as you add an index, "_all" resolves and you get the
situation described in the original bug report: null index is
accepted by the alias, resolves to "_all" and gets applied to everything.
The REST tests, otoh, explicitly tested this bug as a real feature and therefore
passed. The REST tests were modified to change this behavior.
Fixes#7863
If the filter producing the FBS were to be cached then it would flip bits each time the NonNestedDocsFilter was executed.
In our case we cache the NonNestedDocsFilter itself so that didn't happen each time this filter was used.
However the hashcode of NonNestedDocsFilter and NestedDocsFilter was the same, since NonNestedDocsFilter directly used NestedDocsFilter#hashCode()
Closes#8227Closes#8232
The test validates that minimum master node is honored after shutting down nodes. When nodes are restarted we may go through a couple of master election of a master is elected and discovers some of the old nodes left before all new node have joined. Processing that node failure the master de-elects it self, which is fine because we will start a new master election. Test however runs a clusterHealth call with a wait for event. This is implemented using a cluster state update task which fails when the master steps down. Longer term fix requires a rewrite of the cluster health API code but a quick fix is not check for events (not needed for this test).
This commit adds a new field to the response of the terms aggregation called
`sum_other_doc_count` which is equal to the sum of the doc counts of the buckets
that did not make it to the list of top buckets. It is typically useful to have
a sector called eg. `other` when using terms aggregations to build pie charts.
Example query and response:
```json
GET test/_search?search_type=count
{
"aggs": {
"colors": {
"terms": {
"field": "color",
"size": 3
}
}
}
}
```
```json
{
[...],
"aggregations": {
"colors": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 4,
"buckets": [
{
"key": "blue",
"doc_count": 65
},
{
"key": "red",
"doc_count": 14
},
{
"key": "brown",
"doc_count": 3
}
]
}
}
}
```
Close#8213
Add source_node and target_node fields to the recovery cat API. Also fixed and updated the documentation which was not complete concerning fields names.
Closes#8041
- Added parseBooleanExact in booleans which throws exception in case of
parse failure
- Used ParseExact in static's to make it consistent
+ code review fixes
- Added unit test
- Changed Exception Type to ElasticSearchIllegalArg from Parse
- used isExplicit*
Closes#8097
initialize ports.
JUnit uses an instance per test which caused the prot range to be
initialized twice since suite level tests are not configured in a
different context.
The cluster for `Scope.SUITE` tests must be initialize in a static manner
before the first test runs otherwise the random context used to initialize
the cluster is taken from tests randomness rather than the suites randomness.
This means test clusters will have different setups if only a single test is
executed or even the test might have a entirely different random sequence.
This is due to a bug in older version causing refreshes to potentially be missed due to relocations #6545
Also:
- Changed test to keep track of ids and report missing ones.
- Removed total count check from assertSearchHits in order to enable per id checks in cased of a mismatch
- Added a printable unique id part the ids of dummy documents added by indexRandom. The current random unicode id
sometimes prints as ???? to the logs making them hard to trace
The MLT query has a lot of parameters. For example, a set of documents is
specified with either `like_text`, `ids` or `docs`, with at least one
parameter required. This commit groups all the document specification
parameters under one called `like`. The syntax is described below and could
easily be extended to allow for new means of specifying document input. The
`like_text`, `ids` and `docs` parameters are deprecated.
As a single piece text:
{
"query": {
"more_like_this": {
"like": "some text here"
}
}
}
As a single item:
{
"query": {
"more_like_this": {
"like": {
"_index": "imdb",
"_type": "movies",
"_id": "88247"
}
}
}
}
Or as a mixture of all:
{
"query": {
"more_like_this": {
"like": [
"Some random text ...",
{
"_index": "imdb",
"_type": "movies",
"_id": "88247"
},
{
"_index": "imdb",
"_type": "movies",
"doc": {
"title": "Document with an artificial title!"
}
}
]
}
}
}
Closes#8039
When reading the [rolling upgrade process](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades), you can see that we wrote:
* disable allocation
* upgrade node1
* upgrade node2
* upgrade node3
* ...
* enable allocation
That won't work as after a node has been removed and restarted, no shard will be allocated anymore.
So closing node2 and remaining nodes, won't help to serve index and search request anymore.
We should write:
* disable allocation
* upgrade node1
* enable allocation
* wait for shards being recovered on node1
* disable allocation
* upgrade node2
* enable allocation
* wait for shards being recovered on node2
* disable allocation
* upgrade node3
* enable allocation
* wait for shards being recovered on node3
* disable allocation
* ...
* enable allocation
I think this documentation update should go in 1.3, 1.4, 1.x and master branches.
Closes#8218Closes#7973.
If a bulk request contains a mix of indexing requests for an existing index and one that needs to be auto-created but a cluster configuration prevents the auto-create of the new index the ingest process hangs. The exception for the failure to create an index was not caught or reported back properly. Added a Junit test to recreate the issue and the associated fix is in TransportBulkAction.
Closes#8125
When reading metadata we do catch FileNotFound and NoSuchFileExceptions
today, log the even and return an empty metadata object. Yet, in some cases
this might be the wrong thing todo ie. if a commit point is provided these
situations are actually an error and should be rethrown. This commit
pushes the responsiblity to the caller to handle this exception.
Closes#8207
The concurrency level allows to configure the cache internal segments
used to cache data. This can have direct impact on evicition rates since
memory bound caches are equally divided into segments which can cause
early evictions if cache entries are not well balanced.
Relates to #7836
We already have two places duplicating this rather hairy logic, this
commit intorduces a new RefCoutned interace and an abstract implementation
that can be used for delegation. It factors out all the reference counting
and adds single and multithreaded test for it.
Closes#8210
Windows can throw NoSuchFileException when using File.walkFileTree and deleting files concurrently. This commit changes IO exceptions into assertion error so that assertBusy will wait for them as well.
This commit rewrites the state controls in the RecoveryTarget family classes to make it easier to guarantee that:
- recovery resources are only cleared once there are no ongoing requests
- recovery is automatically canceled when the target shard is closed/removed
- canceled recoveries do not leave temp files behind when canceled.
Highlights of the change:
1) All temporary files are cleared upon failure/cancel (see #7315 )
2) All newly created files are always temporary
3) Doesn't list local files on the cluster state update thread (which throw unwanted exception)
4) Recoveries are canceled by a listener to IndicesLifecycle.beforeIndexShardClosed, so we don't need to explicitly call it.
5) Simplifies RecoveryListener to only notify when a recovery is done or failed. Removed subtleties like ignore and retry (they are dealt with internally)
Closes#8092 , Closes#7315
Simon Willnauer
Areek Zillur
Michael McCandless
Martijn Laarman
Jérémy Lecour
uboness
Costin Leau
Clinton Gormley
Uwe Schindler
Dmitriy Khvatov
Spencer
tlrx
Zachary Tong
Alex Ksikes
Martijn van Groningen
Boaz Leskes
Adrien Grand
Nirmal Chidambaram
David Pilato
Marcin Mikosik
markharwood