cat/nodes currently does not report any details related to file descriptors. This adds the current number in use, the maximum number available as well as their ratio (percentage) to cat/nodes as hidden-by-default metrics. In addition, this also adds current heap usage (as a non-percentage of ts max) and ram usage (as a non-percerntage of its max) to allow tools to provide more granularity.
Closes#7652
Dangling indices are indexes found on disk which are not part of the cluster state. By default, we don't delete them but rather import them into the cluster state in order to not accidentally delete data and also allow for the ease of copying index data folders from one cluster to another. Currently, the import logic doesn't check for existing aliases of the same name as the imported dangling index, causing both an index and an alias with the same name.
This commit add a protection against this. Note that the index is still kept as dangling and is only deleted from disk after `gateway.local.dangling_timeout` has passed (2 hours).
We also change the log message indicating deletion of dangling indices to a `WARN` level.
Closes#8059
This commit adds checksumming for cluster and index states. It moves
from a plain XContent based on-disk format to a more structured binary
format including a header and footer as well as a CRC32 checksum for
each of these files. Since previous versions didn't write any format
identifier etc. this commit adds a file extension `.st` for states
that have header/footer and checksum.
This commit also moves over to write temporary files and moves them into
place with an atomic move operation. This commit also serializes and
deserializes the states without materilazing them into opaque memory.
Closes#7586
Though we check that internet connection is fine and that the service is available, it could happen that some services are not available.
```java
assumeTrue(isDownloadServiceWorking("search.maven.org", 80, "/"));
singlePluginInstallAndRemove("org.elasticsearch/elasticsearch-transport-thrift/2.4.0", null);
```
In that case, the first check was successful but downloading thrift plugin from maven central download services was not successful.
This is not an issue with the plugin manager itself but sometimes the test could fail.
Adding a check that actual download service in Maven central works:
```java
assumeTrue(isDownloadServiceWorking("repo1.maven.org", 443, "/maven2/org/elasticsearch/elasticsearch-transport-thrift/2.4.0/elasticsearch-transport-thrift-2.4.0.pom"));
```
If refresh was already running when the refresh_interval is
dynamically disabled (set to a non-positive value like 0, -1, etc.),
then it's possible for the refresh thread to go into while (true)
refresh() loop.
Closes#8085Closes#8087
Today we use the current version which might enable features that are
not supported. We should throw an exception if this setting is not
present for any index.
Closes#8018
Since we don't use the cache, it's okay to clear it entirely if needed,
Elasticsearch maintains its own cache for compiled scripts.
Adds loader.clearCache() into a listener, the listener is called when a
script is removed from the Guava cache.
This also lowers the amount of cached scripts to 100, since 500 is
around the limit some users have run into before hitting an out of
memory error in permgem.
Fixes#7658
Today it's easy to use the wrong version when initializing DiscoveryNode
instances. This commit adds javadocs and a utility constant to
initialize DiscoveryNode instances if the the remotes node version is
unknown.
Closes#8051
Fixes a bug where alias creation would allow `null` for index name, which thereby
applied the alias to _all_ indices. This patch makes the validator throw an
exception if the index is null.
```bash
POST /_aliases
{
"actions": [
{
"add": {
"alias": "empty-alias",
"index": null
}
}
]
}
```
```json
{
"error": "ActionRequestValidationException[Validation Failed: 1: Alias action [add]: [index] may not be null;]",
"status": 400
}
```
The reason this bug wasn't caught by the existing tests is because
the old test for nullness only validated against a cluster which had
zero indices. The null index is translated into "_all", and since
there are no indices, this fails because the index doesn't exist.
So the test passes.
However, as soon as you add an index, "_all" resolves and you get the
situation described in the original bug report: null index is
accepted by the alias, resolves to "_all" and gets applied to everything.
Fixes#7976
Original indices were introduced in 1.4.0.Beta1 as optional, preceded by a boolean flag that told whether they were empty or not. We have to keep serializing as optional only when reading and writing from/to 1.4.0.Beta1, although the original indices are not optional whenever we go and serialize the request they belong to (which is why the boolean got removed in 1.4.0).
The scripted metric aggregation is now a PER_BUCKET aggregation so that parent buckets are evaluated independently. Also the params and reduceParams are copied for each instance of the aggregator (each parent bucket) so modifications to the values are kept only within the scope of its parent bucket
Closes#8036
short[] were mistakenly encoded as a float[]. This is not an issue for the
text-based xcontents that we have (yaml, json) since floats can represent any
short value and are serialized as strings. However, this will make the binary
xcontents serialize shorts as int instead of floats.
Close#7845
By letting the fetch phase understand the nested docs structure we can serve nested docs as hits.
The `top_hits` aggregation can because of this commit be placed in a `nested` or `reverse_nested` aggregation.
Closes#7164
In some cases a shard search request gets created on a node to be only used there and never sent over the transport. This commit clarifies that and creates a new base class called `ShardSearchLocalRequest` that can and will be only used locally. `ShardSearchTransportRequest` on the other hand delegates to the local version but extends `TransportRequest` and is `Streamable`, which means that it is supposed to be sent over the transport.
This way we can make the `OriginalIndices` only required (and mandatory now) in the transport variant.
Took the chance to remove an unused InternalScrollSearchRequest constructor and an empty else branch in `TransportSearchScrollQueryAndFetchAction`.
Closes#7855
This patch adds to `_cat/indices` information about memory usage per index by adding memory used by FieldData, IdCache, Percolate, Segments (memory, index writer, version map).
```
% curl 'localhost:9200/_cat/indices?v&h=i,tm'
i tm
wiki 8.1gb
test 30.5kb
user 1.9mb
```
Closes#7008
* Run flush in beforeIndexShardClosed to prevent an empty shard.
* Only run check index if the shard state before closing was: started, relocated or post_recovery
This commit does the following:
* Add the new API at the rest layer, being backed by the optimize API
with upgrade flag, and segments api to find upgrade status.
* Add `upgrade` flag to optimize API, and deprecate `force` flag (will
remove in master)
* Add test for both synchronous and async upgrade
closes#7884closes#7922
Boaz Leskes
Ryan Ernst
Chris Earle
Lee Hinman
Simon Willnauer
David Pilato
Michael McCandless
Adrien Grand
Igor Motov
Zachary Tong
javanna
Colin Goodheart-Smithe
Alex Ksikes
Martijn van Groningen
Britta Weber