Commit Graph

249 Commits

Author SHA1 Message Date
Sandeep Kanabar 7e0fc8a112 [Docs] Correct spelling in update-settings.asciidoc () 2017-12-14 10:16:50 +01:00
Clinton Gormley e1aa6e2cda Fix cluster usage docs test
 broke the docs tests because $node_name in the URL doesn't ()seem to be replaced.

Changing this to a * to match all nodes seems to fix the test
2017-12-01 16:55:10 +01:00
佛陀.RML 756e170674 [Docs] Fix order of nodes usage example () 2017-12-01 10:42:42 +01:00
Nhat Nguyen 46b508d6c9
Add wait_for_no_initializing_shards to cluster health API ()
This adds a new option to the cluster health request allowing to wait
until there is no initializing shards.

Closes 
2017-11-23 15:09:58 -05:00
Luca Cavanna 29450de7b5
Cross Cluster Search: make remote clusters optional ()
Today Cross Cluster Search requires at least one node in each remote cluster to be up once the cross cluster search is run. Otherwise the whole search request fails despite some of the data (either local and/or remote) is available. This happens when performing the _search/shards calls to find out which remote shards the query has to be executed on. This scenario is different from shard failures that may happen later on when the query is actually executed, in case e.g. remote shards are missing, which is not going to fail the whole request but rather yield partial results, and the _shards section in the response will indicate that.

This commit introduces a boolean setting per cluster called search.remote.$cluster_alias.skip_if_disconnected, set to false by default, which allows to skip certain clusters if they are down when trying to reach them through a cross cluster search requests. By default all clusters are mandatory.

Scroll requests support such setting too when they are first initiated (first search request with scroll parameter), but subsequent scroll rounds (_search/scroll endpoint) will fail if some of the remote clusters went down meanwhile.

The search API response contains now a new _clusters section, similar to the _shards section, that gets returned whenever one or more clusters were disconnected and got skipped:

"_clusters" : {
    "total" : 3,
    "successful" : 2,
    "skipped" : 1
}
Such section won't be part of the response if no clusters have been skipped.

The per cluster skip_unavailable setting value has also been added to the output of the remote/info API.
2017-11-21 11:41:47 +01:00
David Roberts a292740b9e Add cgroup memory usage/limit to OS stats on Linux ()
This change adds cgroup memory usage/limit to the OS stats section of
the node stats on Linux.  This information is useful because in Docker
containers the standard node stats report the host memory limit, not
taking account of extra restrictions that may have been applied to the
container.

The original idea was to store these values as Long, truncating any values
outside the range of long.  However, this meant that in the relatively common
case of no limit being applied, users would not see the same value in the OS
stats as they see by querying Linux directly.  So instead the values are stored
as String.  This change places a burden on consumers of the strings to
convert the strings to numbers and decide what to do about extremely large
values, but there will be very few consumers and they would need to have a
policy for dealing with "no limit" in any case.
2017-10-03 12:08:36 +01:00
Peter Dyson 1f9e0fd0dd [Docs] improved description for fs.total.available_in_bytes () 2017-09-18 16:56:19 +10:00
Tanguy Leroux 643eb286dc [Docs] Convert remaining code snippets in docs ()
This commit converts the last remaining code snippets so that they are
now testable.
2017-08-30 12:11:10 +02:00
Tanguy Leroux db54c4dc7c [Docs] Convert more doc snippets ()
This commit converts some remaining doc snippets so that they are now
testable.
2017-08-30 09:30:36 +02:00
Tanguy Leroux f95dec797d [Docs] Convert more doc snippets ()
This commit converts some remaining doc snippets so that they are now
testable.
2017-08-28 11:23:09 +02:00
Jason Tedor fd18e3239a Remove mention of http_address in nodes info docs
This commit removes an outdated reference to http_address in the nodes
info docs. This information is available in the http object for each
node in the nodes info API response.

Relates 
2017-07-31 22:04:16 +09:00
Daniel Mitterdorfer db90455afd Update plugin-related output in reference docs ()
The example output for node info and cluster stats was outdated w.r.t.
to the information that is shown for plugins. With this commit we
updated the example output and update the explanation of the respective
fields.
2017-07-28 11:27:54 +02:00
Clinton Gormley ff4a2519f2 Update experimental labels in the docs ()
Relates https://github.com/elastic/elasticsearch/issues/19798

Removed experimental label from:
* Painless
* Diversified Sampler Agg
* Sampler Agg
* Significant Terms Agg
* Terms Agg document count error and execution_hint
* Cardinality Agg precision_threshold
* Pipeline Aggregations
* index.shard.check_on_startup
* index.store.type (added warning)
* Preloading data into the file system cache
* foreach ingest processor
* Field caps API
* Profile API

Added experimental label to:
* Moving Average Agg Prediction


Changed experimental to beta for:
* Adjacency matrix agg
* Normalizers
* Tasks API
* Index sorting

Labelled experimental in Lucene:
* ICU plugin custom rules file
* Flatten graph token filter
* Synonym graph token filter
* Word delimiter graph token filter
* Simple pattern tokenizer
* Simple pattern split tokenizer

Replaced experimental label with warning that details may change in the future:
* Analysis explain output format
* Segments verbose output format
* Percentile Agg compression and HDR Histogram
* Percentile Rank Agg HDR Histogram
2017-07-18 14:06:22 +02:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions ()
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Clinton Gormley 1b0c93b07c Documented the level parameter to nodes stats
Closes 
2017-06-01 12:11:21 +02:00
propulkit 25516868fe TCorrecting api name ()
As per REST request signature for reroute, API has no underscore.
2017-05-29 13:58:31 +02:00
Simon Willnauer f22e0dc30b Add cross-cluster search remote cluster info API ()
This commit adds an API to discover information like seed nodes,
http addresses and connection status of a configured remote cluster.

Closes 
2017-04-11 09:24:40 +02:00
Christoph Wurm 2720fc0b43 Clarify task cancellation command ()
Makes it explicit that the node_id has to be included when canceling a task.
2017-03-30 20:21:21 +02:00
Florian Hopf 09753d6a86 Fix typo in allocation explain API docs
This commit addresses a simple typo in the application explain API docs.

Relates 
2017-03-21 08:41:54 -04:00
Ali Beyad 577d2a6a1d Adds cluster state size to /_cluster/state response ()
This commit adds the size of the cluster state to the response for the
get cluster state API call (GET /_cluster/state).  The size that is
returned is the size of the full cluster state in bytes when compressed.
This is the same size of the full cluster state when serialized to
transmit over the network.  Specifying the ?human flag displays the
compressed size in a more human friendly manner.  Note that even if the
cluster state request filters items from the cluster state (so a subset
of the cluster state is returned), the size that is returned is the
compressed size of the entire cluster state.

Closes 
2017-03-02 14:20:29 -05:00
Lee Hinman 6c9b89b882 [TEST] Fix incorrect test cluster name in cluster health doc tests 2017-02-22 17:18:11 -07:00
Lee Hinman 5443f7d625 Console-ify curl statements for allocation explain API docs ()
* Console-ify curl statements for allocation explain API docs

Relates to 

* Fix tests

* Remove exclusion from build.gradle

* Call out index creation in prose

* Add console back and skip test
2017-02-15 17:18:07 -07:00
Ryan Ernst c91848e6a7 Docs: Consoleify cluster and indices settings docs ()
relates 
2017-02-10 14:57:43 -08:00
Nik Everett 0e98c9107a Docs: CONSOLEify some more docs
These need to be CONSOLEified *now* because we're starting to
require Content-Type headers and they didn't have any.

* cluster/reroute: Marked as CONSOLE but skipped because the docs
build runs with a single node.
* docs/bulk: Marked as NOTCONSOLE because the snippets describe
either examples or `curl` commands. Fixed the `curl` command to
include the `Content-Type` header.
* query-dsl/terms-query: Marked as CONSOLE.
* search/request/rescore: Marked as CONSOLE. Fixed deprecated
syntax.

Relates 
Relates 
2017-02-07 16:49:01 -05:00
Clinton Gormley f5a0d18c4c Docs: Cluster allocation explain should be on one page 2017-01-26 11:38:19 +01:00
Ali Beyad 26f92f8482 Cluster allocation explain API documentation ()
This commit updates the cluster allocation explain API documentation to
explain the new request parameters and response formats, and gives
examples of the explain API responses under various scenarios.
2017-01-10 08:55:39 -06:00
Nik Everett 923820c6c9 Document the `detailed` parameter of tasks API ()
Provides an example of using is and an example return description
and explains that we've added descriptions for some tasks but not
even close to all of them. And that we expect to change the
descriptions as we learn more.

Closes 

* Fix example

Getting a single task is always detailed, no need to specify.

* Rewrite like imotov wants it
2017-01-06 10:24:52 -05:00
Jason Tedor 41ffb008ad Fix doc bug for cgroup cpuacct usage metric
This commit fixes a silly doc bug where the field that represents the
total CPU time consumed by all tasks in the same cgroup was mistakenly
reported as "usage" instead of "usage_nanos".

Relates 
2016-12-15 23:22:54 -05:00
Clinton Gormley cfabc95f59 Fixed bad asciidoc ID in node stats 2016-11-15 17:39:15 +00:00
Jason Tedor f5ac0e5076 Remove lenient stats parsing
Today when parsing a stats request, Elasticsearch silently ignores
incorrect metrics. This commit removes lenient parsing of stats requests
for the nodes stats and indices stats APIs.

Relates 
2016-11-15 12:17:26 -05:00
Jason Tedor aec09a76d6 Clarify requesting all stats in node stats docs
This commit clarifies how to explicitly obtain all stats from the node
stats API.
2016-11-08 13:47:15 -05:00
Igor Motov 17ad88d539 Makes search action cancelable by task management API
Long running searches now can be cancelled using standard task cancellation mechanism.
2016-10-25 12:27:34 -10:00
Jason Tedor 900ee0536e Strengthen handling of unavailable cgroup stats
On some systems, cgroups will be available but not configured. And in
some cases, cgroups will be configured, but not for the subsystems that
we are expecting (e.g., cpu and cpuacct). This commit strengthens the
handling of cgroup stats on such systems.

Relates 
2016-10-24 16:36:51 -04:00
Jason Tedor 3d642ab0eb Add basic cgroup CPU metrics
This commit adds basic cgroup CPU metrics to the node stats API.

Relates 
2016-10-24 08:26:56 -04:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates 
2016-10-04 12:45:29 -04:00
Tanguy Leroux 656596c2a9 [DOC] Remove obsolete node names from documentation
Funny node names have been removed in  and replaced by UUID. This commit removes these obsolete node names and replace them by real UUIDs in the documentation.

closes 
2016-09-19 11:56:28 +02:00
Lee Hinman fd3392aef8 [DOCS] Mark cluster allocation explain API as experimental in docs 2016-09-06 11:29:33 -06:00
Nik Everett 5cff2a046d Remove most of the need for `// NOTCONSOLE`
and be much more stingy about what we consider a console candidate.

* Add `// CONSOLE` to check-running
* Fix version in some snippets
* Mark groovy snippets as groovy
* Fix versions in plugins
* Fix language marker errors
* Fix language parsing in snippets

  This adds support for snippets who's language is written like
  `[source, txt]` and `["source","js",subs="attributes,callouts"]`.

  This also makes language required for snippets which is nice because
  then we can be sure we can grep for snippets in a particular language.
2016-09-06 10:32:54 -04:00
javanna 5f299ff46f add mem section back to cluster stats
The mem section was buggy in cluster stats and removed. It is now added back with the same structure as in node stats, containing total memory, available memory, used memory and percentages. All the values are the sum of all the nodes across the cluster (or at least the ones that we were able to get the values from).
2016-09-01 11:26:03 +02:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards ()
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Nik Everett 777ea124c7 Fix health docs test
It failed inconsistently when there were pending tasks.
2016-07-16 07:18:11 -04:00
Nik Everett 9f78f8cc91 Convert snippets in health docs to CONSOLE
This should make them easier to read and adds them to the test suite
I changed the example from a two node cluster to a single node cluster
because that is what we have running in the integration tests. It is also
what a user just starting out is likely to see so I think that is ok.
2016-07-15 16:31:37 -04:00
Lee Hinman 58db63b610 Expose the ClusterInfo object in the allocation explain output
This adds an optional parameter to the cluster allocation explain API
that will return the cluster info object, `include_disk_info`, the
output looks like:

GET /_cluster/allocation/explain?include_disk_info -d'
{"index": "i", "shard": 0, "primary": false}'

{
  ... other info ...

  "cluster_info" : {
    "nodes" : {
      "7Uws-vL7R6WVm3ZwQA1n5A" : {
        "node_name" : "Kraven the Hunter",
        "least_available" : {
          "path" : "/path/to/data1",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        },
        "most_available" : {
          "path" : "/path/to/data2",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        }
      }
    },
    "shard_sizes" : {
      "[i][2][p]_bytes" : 0,
      "[i][4][p]_bytes" : 130,
      "[i][1][p]_bytes" : 0,
      "[i][3][p]_bytes" : 0,
      "[i][0][p]_bytes" : 130
    },
    "shard_paths" : {
      "[i][3], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=LegZLDniTVaw0Y1urv7s3g]" : "/path/to/data1/nodes/0",
      "[i][1], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=lAU_4vf_SKmoRdtg0ACnjQ]" : "/path/to/data1/nodes/0",
      "[i][2], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Aurpeuj7SeGeyPDDpCtRgg]" : "/path/to/data1/nodes/0",
      "[i][0], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Vgg8GlQTQ82C2j6HYBq8DQ]" : "/path/to/data1/nodes/0",
      "[i][4], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=t8hQlVSxQe-58fSeaXcAqg]" : "/path/to/data1/nodes/0"
    }
  }
}

Resolves 
2016-07-12 15:52:20 -06:00
Mike McCandless eecf094ac1 add indices nodes info flag to docs 2016-06-20 14:23:32 -04:00
Mike McCandless 3f221bf7cb Add total_indexing_buffer/_in_bytes to nodes info API 2016-06-16 04:39:34 -04:00
Nik Everett e392e0b1df Create get task API that falls back to the .tasks index
This adds a get task API that supports GET /_tasks/${taskId} and
removes that responsibility from the list tasks API. The get task
API supports wait_for_complation just as the list tasks API does
but doesn't support any of the list task API's filters. In exchange,
it supports falling back to the .results index when the task isn't
running any more. Like any good GET API it 404s when it doesn't
find the task.

Then we change reindex, update-by-query, and delete-by-query to
persist the task result when wait_for_completion=false. The leads
to the neat behavior that, once you start a reindex with
wait_for_completion=false, you can fetch the result of the task by
using the get task API and see the result when it has finished.

Also rename the .results index to .tasks.
2016-06-14 13:37:34 -04:00
Mike McCandless 5c525e6606 Remove index_writer_max_memory stat from segment stats 2016-05-31 06:29:29 -04:00
Lee Hinman bfce901edf Merge remote-tracking branch 'dakrone/explain-add-fetch-in-progress' 2016-05-23 09:43:16 -06:00
Lee Hinman 8040ed0c16 Add whether the shard state fetch is pending to the allocation explain API
If the shard state fetch is still pending, this will now return a
message like:

```json
{
  "shard" : {
    "index" : "i",
    "index_uuid" : "de1W1374T4qgvUP4a9Ieaw",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "shard_state_fetch_pending": true,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-04-26T16:34:53.227Z"
  },
  "allocation_delay_ms" : 0,
  "remaining_delay_ms" : 0,
  "nodes" : {
    "z-CbkiELT-SoWT91HIszLA" : {
      "node_name" : "Brain Cell",
      "node_attributes" : {
        "testattr" : "test"
      },
      "store" : {
        "shard_copy" : "NONE"
      },
      "final_decision" : "NO",
      "final_explanation" : "the shard state fetch is pending",
      "weight" : 5.0,
      "decisions" : [ ]
    }
  }
}
```

Adds the `shard_state_fetch_pending` field and uses the state to
influence the final decision and final explanation.

Relates to 
2016-05-23 09:42:57 -06:00
Simon Willnauer 35e705877b Limit retries of failed allocations per index ()
Today if a shard fails during initialization phase due to misconfiguration, broken disks,
missing analyzers, not installed plugins etc. elasticsaerch keeps on trying to initialize
or rather allocate that shard. Yet, in the worst case scenario this ends in an endless
allocation loop. To prevent this loop and all it's sideeffects like spamming log files over
and over again this commit adds an allocation decider that stops allocating a shard that
failed more than N times in a row to allocate. The number or retries can be configured via
`index.allocation.max_retry` and it's default is set to `5`. Once the setting is updated
shards with less failures than the number set per index will be allowed to allocate again.

Internally we maintain a counter on the UnassignedInfo that is reset to `0` once the shards
has been started.

Relates to 
2016-05-20 20:37:45 +02:00
Jason Tedor ecce53f0df Add I/O statistics on Linux
This commit adds a variety of real disk metrics for the block devices
that back Elasticsearch data paths. A collection of statistics are read
from /proc/diskstats and are used to report the raw metrics for
operations and read/write bytes.

Relates 
2016-05-17 16:16:39 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE () 2016-05-09 15:42:23 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes . That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Lee Hinman 5648253d45 Add documentation for shard store output in allocation explain API
Relates to 
2016-05-03 09:51:15 -06:00
Igor Motov 81c59cae18 Add _cat/tasks
Adds new _cat endpoint that lists all tasks
2016-04-07 09:28:21 -06:00
Igor Motov f599ac5d5a Expose whether a task is cancellable in the _tasks list API
Closes 
2016-04-05 19:16:08 -06:00
Daniel Mitterdorfer 930ce1bfec Add up-to-date example of cluster stats API output 2016-03-31 14:41:37 +02:00
Igor Motov e073b0c75d Add ability to group tasks by common parent
By default, tasks are grouped by node. However, task execution in elasticsearch can be quite complex and an individual task that runs on a coordinating node can have many subtasks running on other nodes in the cluster. This commit makes it possible to list task grouped by common parents instead of by node. When this option is enabled all subtask are grouped under the coordinating node task that started all subtasks in the group. To group tasks by common parents, use the following syntax:

 GET /tasks?group_by=parents
2016-03-30 17:50:27 -04:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Igor Motov c356b30cff Update task management docs to reflect the latest changes in the interface
Brings docs in line with new list task syntax and adds task cancellation API docs.
2016-03-29 12:26:37 -04:00
javanna 8fc9dbbb99 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 14:27:04 +02:00
Clinton Gormley 978b24327e Docs: Included Nodes Task API and tidied reindex/update-by-query 2016-03-29 13:51:11 +02:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Lee Hinman 80ab366de4 Add API to explain why a shard is or isn't assigned
This adds a new `/_cluster/allocation/explain` API that explains why a
shard can or cannot be allocated to nodes in the cluster. Additionally,
it will show where the master *desires* to put the shard, according to
the `ShardsAllocator`.

It looks like this:

```
GET /_cluster/allocation/explain?pretty
{
  "index": "only-foo",
  "shard": 0,
  "primary": false
}
```

Though, you can optionally send an empty body, which means "explain the
allocation for the first unassigned shard you find".

The output when a shard is unassigned looks like this:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-03-22T20:04:23.620Z"
  },
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 0.06666675,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "NO",
      "weight" : -1.3833332,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 2.3166666,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

And when the shard *is* assigned, the output looks like:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : true
  },
  "assigned" : true,
  "assigned_node_id" : "Qc6VL8c5RWaw1qXZ0Rg57g",
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 1.4499999,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "CURRENTLY_ASSIGNED",
      "weight" : 0.0,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 3.6999998,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

Only "NO" decisions are returned by default, but all decisions can be
shown by specifying the `?include_yes_decisions=true` parameter in the
request.

Resolves 
2016-03-28 15:21:02 -06:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Robin Clarke 046212035c Clarification about precedence of settings
Closes 
2016-03-10 14:29:51 +01:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
javanna 9c4a5bbe7e adapt cluster stats api to node.client setting removal
The cluster stats api now returns counts for each node role. The `master_data`, `master_only`, `data_only` and `client` fields have been removed from the response in favour of `master`, `data`, `ingest` and `coordinating_only`. The same node can have multiple roles, hence contribute to multiple roles counts. Every node is implicitly a coordinating node, so whenever a node has no explicit roles, it will be counted as coordinating only.
2016-03-05 10:55:19 +01:00
Clinton Gormley 4e5316591a Update stats.asciidoc
Renamed filter_cache->query_cache and removed id_cache

Closes 
2016-01-26 13:48:46 +01:00
Yannick Welsch d5b691b68e Extend reroute with an option to force assign stale primary shard copies
Closes 
2016-01-19 12:07:01 +01:00
Jason Tedor df598e8129 Modify load average formats
This commit modifies the load_average in the node stats API response
to be an object containing the one-minute, five-minute and
fifteen-minute load averages as fields (if those values are
available). Additionally, this commit modifies the cat nodes API
response to format the one-minute, five-minute and fifteen-minute load
averages as null if any of the respective values are not available.
2016-01-18 11:41:34 -05:00
Jason Tedor 1de2081ed3 Reintroduce five-minute and fifteen-minute load averages on Linux
This commit reintroduces the five-minute and fifteen-minute load stats
on Linux, and changes the format of the load_average field back to an
array.
2016-01-11 23:42:47 -05:00
Simon Willnauer 6ea266a89c Merge branch 'master' into settings_prototype 2015-12-15 16:33:01 +01:00
Felipe Forbeck 708abcc59a Added desc for parameter <local> 2015-12-11 22:26:33 -02:00
Simon Willnauer ce417540c5 apply review from @clintongormley 2015-12-09 12:24:40 +01:00
Simon Willnauer 2e27ee393f add rest API to reset settings 2015-12-08 14:39:16 +01:00
Jason Tedor 6872d545ac Add system CPU percent to OS stats
This commit adds the system CPU percent reflecting the recent CPU usage
for the whole system.
2015-11-17 13:48:46 -05:00
xuzha fb1d8bb149 Add os.allocated_processors
Current processors setting is not reflected in nodes info API
("os.available_processors"). Add os.allocated_processors to shows
actual number of processors that we are using.
2015-11-03 09:50:17 -08:00
xuzha 97ecd7bf5a Expose pending cluster state queue size in node stats
Add 3 stats about the queue: total queue size, number of committed cluster
states, and number of pending cluster states.
2015-10-28 10:59:15 -07:00
Tanguy Leroux db7aecab4d update list of available os stats
os cpu information is no longer exposed through the nodes stats api
2015-08-31 17:03:45 +02:00
Tanguy Leroux 8e052f0da2 Make platform specific assumptions in OS & Process probes tests 2015-08-17 14:47:23 +02:00
Andrey Fadeyev 081fb1a899 Fixes - update "Cluster Stats" documentation with valid example 2015-08-13 12:09:31 +02:00
Tanguy Leroux 03c327ff12 Expose ClassloadingMXBean in Node Stats
Closes 
2015-08-12 14:29:13 +02:00
Clinton Gormley db541d6fbe Docs: Add warning about allow_primary to the cluster reroute docs
Closes 
2015-08-07 12:03:19 +02:00
Tanguy Leroux cf6acbd7c2 Remove obsolete plugins.info_refresh_interval setting
This setting has been removed in  
2015-08-04 21:46:31 +02:00
Tanguy Leroux 19e348a82c Update OS stats 2015-07-08 17:48:10 +02:00
Tanguy Leroux 1c5d8efd47 Process Stats: remove sigar specific stats from APIs and add JMX implementation 2015-07-08 15:12:45 +02:00
Tanguy Leroux 26fd4ba95b Docs: fix wrong title level 2015-07-08 09:29:21 +02:00
Tanguy Leroux fbcf4dbbf7 FS Stats: remove sigar specific stats from APIs:
- fs.*.disk_reads
- fs.*.disk_writes
- fs.*.disk_io_op
- fs.*.disk_read_size_in_bytes
- fs.*.disk_write_size_in_bytes
- fs.*.disk_io_size_in_bytes
- fs.*.disk_queue
- fs.*.disk_service_time
2015-07-07 22:16:39 +02:00
Tanguy Leroux 30892c4129 Remove network stats & info 2015-07-07 21:16:42 +02:00
Clinton Gormley 84acb65ca1 Docs: Documented delayed allocation settings
Relates to: 
2015-06-30 13:53:04 +02:00
Simon Willnauer fcdcce3bba Consolidate shard level abstractions
This commit consolidates several abstractions on the shard level in
ordinary classes not managed by the shard level guice injector.

Several classes have been collapsed into IndexShard and IndexShardGatewayService
was cleaned up to be more lightweight and self-contained. It has also been moved into
the index.shard package and it's operation is renamed from recovery from "gateway" to recovery
from "store" or "shard_store".

Closes 
2015-06-24 15:18:04 +02:00
Clinton Gormley f123a53d72 Docs: Refactored modules and index modules sections 2015-06-22 23:49:45 +02:00
Boaz Leskes 1df2d3015e Add OS name to _nodes and _cluster/nodes
we currently don't expose this.

This adds the following to the OS section of `_nodes`:

```
"os": {
     "name": "Mac OS X",
     ...
 }
 ```

 and the following to the OS section of `_cluster/stats`:

 ```
      "os": {
         ...
          "names": [
             {
                "name": "Mac OS X",
                "count": 1
             }
          ],
          ...
       },
 ```

Closes 
2015-06-22 20:36:29 +02:00
Martijn van Groningen ece18f162e Removed `id_cache` from stats and cat apis.
Also removed the `id_cache` option from the clear cache api.

Closes 
2015-05-15 14:06:18 +02:00
Adrien Grand b72f27a410 Core: Cut over to the Lucene filter cache.
This removes Elasticsearch's filter cache and uses Lucene's instead. It has some
implications:
 - custom cache keys (`_cache_key`) are unsupported
 - decisions are made internally and can't be overridden by users ('_cache`)
 - not only filters can be cached but also all queries that do not need scores
 - parent/child queries can now be cached, however cached entries are only
   valid for the current top-level reader so in practice it will likely only
   be used on read-only indices
 - the cache deduplicates filters, which plays nicer with large keys (eg. `terms`)
 - better stats: we already had ram usage and evictions, but now also hit count,
   miss count, lookup count, number of cached doc id sets and current number of
   doc id sets in the cache
 - dynamically changing the filter cache size is not supported anymore

Internally, an important change is that it removes the NoCacheFilter infrastructure
in favour of making Query.rewrite specializing the query for the current reader so
that it will only be cached on this reader (look for IndexCacheableQuery).

Note that consuming filters with the query API (createWeight/scorer) instead of
the filter API (getDocIdSet) is important for parent/child queries because
otherwise a QueryWrapperFilter(ParentQuery) would run the wrapped query per
segment while relations might be cross segments.
2015-05-04 09:02:15 +02:00
Simon Willnauer d164526d27 Remove `_shutdown` API
Thsi commit removes the `_shutdown` API entirely without any replacement.
Nodes should be managed from the operating system not via REST APIs
2015-04-27 17:19:36 +02:00
Clinton Gormley abc7de96ae Docs: Updated version annotations in master 2015-04-09 14:50:11 +02:00
Lee Hinman eed7c8af6d [DOCS] Document `indices.recovery.concurrent_small_file_streams` 2015-04-06 11:16:50 -06:00
wittyameta 728f834716 [DOCS] add wait_for_active_shards option to health.asciidoc 2015-04-02 09:33:54 +02:00
Boaz Leskes 6953777c3a API: add pending tasks count to cluster health
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.

It also adds rest tests for the cluster health variants.

Closes 
2015-02-25 14:58:44 +01:00
gseng d1deb6bd1e Update update-settings.asciidoc
Updating to the fields as mentioned on http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html

Closes 
2015-02-12 13:09:16 +01:00
Michael McCandless def2d34f80 don't mention fixed throttling in the docs 2015-01-14 10:13:10 -05:00
Michael McCandless 107099affa put back fixed throttling, but off by default 2015-01-14 05:35:09 -05:00
Michael McCandless 1aad275c55 expose current CMS throttle in merge stats; fix tests, docs; also log per-merge stop/throttle/rate 2015-01-11 05:52:43 -05:00
Simon Willnauer 236e2491b4 [ALLOCATION] Remove primary balance factor
The `cluster.routing.allocation.balance.primary` setting has caused
a lot of confusion in the past while it has very little benefit form a
shard allocatioon point of view. Users tend to modify this value to
evently distribute primaries across the nodes which is dangerous since
a prmiary flag on it's own can trigger relocations. The primary flag for a shard
is should not have any impact on cluster performance unless the high level feature
suffereing from primary hotspots is buggy. Yet, this setting was intended to be a
tie-breaker which is not necessary anymore since the algorithm is deterministic.

This commit removes this setting entriely.
2015-01-06 16:43:39 +01:00
Simon Willnauer 4900f52619 [ALLOCATION] Weight deltas must be absolute deltas
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue . This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes 
2015-01-06 15:48:44 +01:00
Mikhail Korobov 707025fb7a [Docs] fix curl examples in Nodes Stats docs
Closes 
2014-12-31 14:01:37 +01:00
Michael McCandless 242e631e95 Core: ignore known idle threads by default in /_nodes/hot_threads
Add a new ignore_idle_threads boolean option (default true) to
/_nodes/hot_threads, to filter out threads in known idle places like
waiting on a socket select or on pulling the next task from an empty
queue.

Closes 

Closes 
2014-12-17 11:59:31 -05:00
spapin ad747ba67f Docs: fix a typo in cluster stats documentation example
Closes 
2014-12-15 14:14:38 +01:00
Clinton Gormley a8b21f2cd5 Update update-settings.asciidoc
Removed deprecated `cluster.routing.allocation.disable` settings
2014-10-22 12:46:33 +02:00
Clinton Gormley cb00d4a542 Docs: Removed all the added/deprecated tags from 1.x 2014-09-26 21:04:42 +02:00
Clinton Gormley 3f9aea883f Docs: Made current version, branch and jdk into asciidoc attributes 2014-07-23 11:55:35 +02:00
mahdeto e78f1edca3 DOC:Added field data circuit breaker settings 2014-06-26 10:29:41 +02:00
Clinton Gormley e52364a95a Docs: Updated cluster health docs 2014-06-20 18:05:46 +02:00
darkwarriors d8765a8f1d [DOCS] fixed urls in nodes-stats docs 2014-06-03 13:48:42 +02:00
Patrik Ragnarsson 9a3368b937 [DOCS] Fix minor error in cluster stats example 2014-06-03 13:38:37 +02:00
Kevin Wang 33d256119d fix field data stats doc 2014-05-06 15:57:00 +02:00
Alexander Reelsen d4fcf23057 Cluster State API: Remove index template filtering
The possibility of filtering for index templates in the cluster state API
had been introduced before there was a dedicated index templates API. This
commit removes this support from the cluster state API, as it was not really
clean, requiring you to specify the metadata and the index templates.

Closes 
2014-05-05 14:54:14 +02:00
Nik Everett af0278b51b [Docs] Allocation setting explanation
Closes 
2014-04-09 12:11:36 -06:00
Costin Leau 960d353dbd Remove plugin isolation feature for a future version
relates 
2014-04-09 17:28:11 +03:00
Nik Everett 1df942b463 [docs] Indices stats groups in nodes api
Closes 
2014-03-31 19:54:48 +02:00
bleskes 5d832374dd Update Documentation Feature Flags [1.1.0] 2014-03-25 17:51:30 +01:00
Costin Leau 9624b215fb Add docs for plugin isolation 2014-03-11 12:32:58 +02:00
Lee Hinman e53a43800e Add `explain` flag support to the reroute API
By specifying the `explain` flag, an explanation for the reason a
command can or cannot be executed is returned. No allocation commands
are actually performed.

Returns a response similar to:

{
  "state": {...cluster state...},
  "acknowledged": true,
  "explanations" : [ {
    "command" : "cancel",
      "parameters" : {
        "index" : "decide",
        "shard" : 0,
        "node" : "IvpoKRdtRiGrQ_WKtt4_4w",
        "allow_primary" : false
      },
      "decisions" : [ {
        "decider" : "cancel_allocation_command",
        "decision" : "YES",
        "explanation" : "..."
        } ]
     }, {
      "command" : "move",
      "parameters" : {
        "index" : "decide",
        "shard" : 0,
        "from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
        "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
       },
       "decisions" : [ {
         "decider" : "same_shard",
         "decision" : "NO",
         "explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
       },
       etc
       ]
  }]
}

also removes AllocationExplanation from cluster state

Closes 
Closes 
2014-02-27 09:48:51 -07:00
Boaz Leskes e379f419e6 [DOCS] Remove clear flag from node-stats as it is not used anymore 2014-02-17 15:20:12 +01:00
Luca Cavanna 1d8d58391f [DOCS] added coming tags for `zen.discovery.publish_timeout` made dynamic 2014-02-12 15:24:38 +01:00
Luca Cavanna 847521b44c [DOCS] added `discovery.zen.publish_timeout` to the dynamic settings list 2014-02-12 10:45:30 +01:00
Alexander Reelsen b02e6dc996 Migrating NodesInfo API to use plugins instead of singular plugin
In order to be consistent (and because in 1.0 we switched from
parameter driven information to specifzing the metrics as part of the URI)
this patch moves from 'plugin' to 'plugins' in the Nodes Info API.
2014-02-11 10:05:10 +01:00
Shay Banon e5f43a1867 add version and master_node flags to cluster state 2014-02-10 02:24:03 +01:00
Sean Gallagher e935a301df Doc fix explaining resynchronization with the Cancel command.
Added line explaining resync process to Reroute/Cancel command.

Closes 
2014-02-07 17:02:36 -05:00
Clinton Gormley 93930d6dc7 Removed 0.90.* deprecation and addition notifications
Closes 
2014-02-07 20:52:49 +01:00
Clinton Gormley 6238d406b5 [DOCS] Removed the experimental label from Tribe, Hot Threads
and Completion Suggester
2014-02-06 14:19:17 +01:00
Luca Cavanna 4126ae2631 [DOCS] updated json responses after and
- Removed "ok": true from response examples
 - Added "created" flag to index response examples
 - Replaced exists flag with found in delete response examples
2014-01-16 12:01:39 +01:00
Boaz Leskes 5ac7bd83ad Expose min/max open file descriptors in Cluster Stats API
Also changes the response format of that section to:

```
 "open_file_descriptors": {
      "min": 200,
      "max": 346,
       "avg": 273
 }
```

Closes 

Note: this is an aggregate of 3 commits in the 0.90 branch
2014-01-10 12:15:56 +01:00
Martijn van Groningen e6f83248a2 Deprecated disable allocation decider which has the following options:
`allocation.disable_new_allocation`, `allocation.disable_allocation`, `allocation.disable_replica_allocation`,
in favour for the enable allocation decider which has a single option `allocation.enable` wich can be set to the following values:
`none`, `new_primaries`, `primaries` and `all` (default).

Closes 
2014-01-09 10:01:46 +01:00
Alexander Reelsen ad50afbec8 Simplify usage of nodes info API
Important: This breaks backwards compatibility with 0.90

* Removed endpoints: /_cluster/nodes, /_cluster/nodes/nodeId1,nodeId2
* Disallow usage of parameters, but make required metrics part of URI
* Changed NodesInfoRequest to return everything by default
* Fixed NPE in NodesInfoResponse

Closes 
2014-01-08 09:46:04 +01:00
Alexander Reelsen 6ef6bb993c Cluster state API: Improved consistency
Instead of specifying what kind of data should be filtered, this commit
streamlines the API to actually specify, what kind of data should be displayed.
This makes its behaviour similar to the other requests, like NodeIndicesStats.

A small feature has been added as well: If you specify an index to select on, not
only the metadata, but also the routing tables are filtered by index in order
to prevent too big cluster states to be returned.

Also the CAT apis have been changed to only return the wanted data in order to keep
network traffic as small as needed.

Tests for the cluster state API filtering have been added as well.

Note: This change breaks backwards compatibility with 0.90!

Closes 
2014-01-08 09:25:20 +01:00
Alexander Reelsen bb275166f1 Simplify nodes stats API
First, this breaks backwards compatibility!

* Removed /_cluster/nodes/stats endpoint
* Excpect the stats types not as parameters, but as part of the URL
* Returning all indices stats by default, returning all nodes stats by default
* Supporting groups & types in nodes stats now as well
* Updated documentation & tests accordingly
* Allow level parameter for "shards" and "indices" (cluster does not make sense here)

Closes 
2014-01-06 08:33:32 +01:00
Lee Hinman a754224751 Add field data memory circuit breaker.
This adds the field data circuit breaker, which is used to estimate
the amount of memory required to load field data before loading it. It
then raises a CircuitBreakingException if the limit is exceeded.

It is configured with two parameters:

`indices.fielddata.cache.breaker.limit` - the maximum number of bytes
of field data to be loaded before circuit breaking. Defaults to
`indices.fielddata.cache.size` if set, unbounded otherwise.

`indices.fielddata.cache.breaker.overhead` - a contast for all field
data estimations to be multiplied with before aggregation. Defaults to
1.03.

Both settings can be configured dynamically using the cluster update
settings API.
2014-01-02 15:04:47 -07:00
Boaz Leskes 3c5106ae98 Added cluster health status to the Cluster Stats API
Relates to 
2013-12-18 12:03:49 +01:00
Boaz Leskes 2b6214cff7 Added Cluster Stats API
Closes 
2013-12-17 13:14:46 +01:00
Clinton Gormley b48344f296 [DOCS] Doc'ed cluster pending tasks 2013-11-29 08:21:26 +01:00
Richard Boulton b62cc7c716 Clean up wording to reduce confusion
The description of the timeout parameter was worded misleadingly; it implied that the API would wait until the cluster reached the desired level and then stayed at that level for the timeout. I've tweaked the sentence to remove the risk of confusion.
2013-10-21 12:37:50 +01:00
Lee Hinman ba40aa374e Uniquify anchor links to fix asciidoc/docbook generation 2013-09-30 15:32:00 -06:00
Lee Hinman 0442b737be Add more anchor links to documentation
Related to 
2013-09-30 13:13:16 -06:00
Clinton Gormley 393c28bee4 [DOCS] Removed outdated new/deprecated version notices 2013-09-03 21:28:31 +02:00
Clinton Gormley 822043347e Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00