Commit Graph

2704 Commits

Author SHA1 Message Date
Adrien Grand 51d9574916 Fix reference guide about the default value of min_should_match. 2016-03-31 10:49:02 +02:00
Adrien Grand 26a0fb37a4 Add examples of useful dynamic templates to the docs. #17413 2016-03-31 09:45:11 +02:00
Igor Motov e073b0c75d Add ability to group tasks by common parent
By default, tasks are grouped by node. However, task execution in elasticsearch can be quite complex and an individual task that runs on a coordinating node can have many subtasks running on other nodes in the cluster. This commit makes it possible to list task grouped by common parents instead of by node. When this option is enabled all subtask are grouped under the coordinating node task that started all subtasks in the group. To group tasks by common parents, use the following syntax:

 GET /tasks?group_by=parents
2016-03-30 17:50:27 -04:00
Clinton Gormley 48c566da73 Documented how tribe node clients inherit their settings
Closes #17309
2016-03-30 20:05:44 +02:00
Adrien Grand d7179cafcc Add a soft limit on the number of shards that can be queried in a single search request. #17396
This commit adds the new `action.search.shard_count.limit` setting which
configures the maximum number of shards that can be queried in a single search
request. It has a default value of 1000.
2016-03-30 16:55:01 +02:00
Martijn van Groningen 7e2696c570 Refactored inner hits parsing and intoduced InnerHitBuilder
Both top level and inline inner hits are now covered by InnerHitBuilder.
Although there are differences between top level and inline inner hits,
they now make use of the same builder logic.

The parsing of top level inner hits slightly changed to be more readable.
Before the nested path or parent/child type had to be specified as encapsuting
json object, now these settings are simple fields. Before this was required
to allow streaming parsing of inner hits without missing contextual information.

Once some issues are fixed with inline inner hits (around multi level hierachy of inner hits),
top level inner hits will be deprecated and removed in the next major version.
2016-03-30 15:15:56 +02:00
Adrien Grand fc47007e17 Add a soft limit on the mapping depth. #17400
This commit adds the new `index.mapping.depth.limit` setting which controls the
maximum mapping depth that is allowed. It has a default value of 20.
2016-03-30 14:37:00 +02:00
Simon Willnauer 2514681f66 updateing filtering.asciidoc to also use 'node.attr' namespace 2016-03-30 14:11:59 +02:00
Simon Willnauer ad24653948 update allocation_awareness.asciidoc to also use 'node.attr' namespace 2016-03-30 13:52:45 +02:00
Simon Willnauer 8b075dbb75 Remove ability to specify arbitrary node attributes with `node.` prefix
Today the basic node settings like `node.data` and `node.master` can't really be fully validated
since we allow to specify custom user attributes on the node level. We have to, in order to
support that, add a wildcard setting for `node.*` to let these setting pass validation.
Instead we should require a more contraint prefix like `node.attr.` that defines a namespace
that is reserved for user attributes.
This commit adds a new namespace for attributes in `node.attr`.

Closes #17280
2016-03-30 13:29:48 +02:00
Isabel Drost-Fromm f27399dc0e Merge pull request #17282 from MaineC/deprecation/sort-option-reverse-removal
Remove deprecated reverse option from sorting
2016-03-30 11:02:19 +02:00
javanna 19eeb68bc4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 21:53:22 +02:00
Spencer f07b35b328 Merge pull request #17368 from spalger/docs/clarifyMinMasterCheck
[docs] clarify where discovery.zen.minimum_master_node is required
2016-03-29 11:44:50 -07:00
javanna ae34c20a62 add node.client breaking changes to migrate guide 2016-03-29 20:33:59 +02:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Yanjun Huang 361adcf387 Add limit to total number of fields in mapping. #17357
This is to prevent mapping explosion when dynamic keys such as UUID are used as field names. index.mapping.total_fields.limit specifies the total number of fields an index can have. An exception will be thrown when the limit is reached. The default limit is 1000. Value 0 means no limit. This setting is runtime adjustable

Closes #11443
2016-03-29 19:39:46 +02:00
Igor Motov c356b30cff Update task management docs to reflect the latest changes in the interface
Brings docs in line with new list task syntax and adds task cancellation API docs.
2016-03-29 12:26:37 -04:00
Tal Levy 9ac3887139 Merge pull request #17263 from talevy/auto-convert
add  type conversion support to ConvertProcessor
2016-03-29 07:57:57 -07:00
Tal Levy 2064fe3985 add type conversion support to ConvertProcessor 2016-03-29 07:56:53 -07:00
Clinton Gormley 798e4281fa Added experimental annotation to the update-by-query and reindex docs 2016-03-29 15:06:27 +02:00
javanna 8fc9dbbb99 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 14:27:04 +02:00
Clinton Gormley 978b24327e Docs: Included Nodes Task API and tidied reindex/update-by-query 2016-03-29 13:51:11 +02:00
Isabel Drost-Fromm 407e2cdcf9 Merge branch 'master' into deprecation/sort-option-reverse-removal
Conflicts:
	core/src/main/java/org/elasticsearch/search/sort/ScoreSortBuilder.java
	core/src/test/java/org/elasticsearch/search/sort/FieldSortBuilderTests.java
2016-03-29 11:04:02 +02:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Lee Hinman 80ab366de4 Add API to explain why a shard is or isn't assigned
This adds a new `/_cluster/allocation/explain` API that explains why a
shard can or cannot be allocated to nodes in the cluster. Additionally,
it will show where the master *desires* to put the shard, according to
the `ShardsAllocator`.

It looks like this:

```
GET /_cluster/allocation/explain?pretty
{
  "index": "only-foo",
  "shard": 0,
  "primary": false
}
```

Though, you can optionally send an empty body, which means "explain the
allocation for the first unassigned shard you find".

The output when a shard is unassigned looks like this:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-03-22T20:04:23.620Z"
  },
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 0.06666675,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "NO",
      "weight" : -1.3833332,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 2.3166666,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

And when the shard *is* assigned, the output looks like:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : true
  },
  "assigned" : true,
  "assigned_node_id" : "Qc6VL8c5RWaw1qXZ0Rg57g",
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 1.4499999,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "CURRENTLY_ASSIGNED",
      "weight" : 0.0,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 3.6999998,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

Only "NO" decisions are returned by default, but all decisions can be
shown by specifying the `?include_yes_decisions=true` parameter in the
request.

Resolves #14593
2016-03-28 15:21:02 -06:00
spalger ce44bbfadf [docs] clarify where discovery.zen.minimum_master_node is required
https://github.com/elastic/elasticsearch/pull/17288 added a check to enforce that the `discovery.zen.minimum_master_nodes` configuration is set when nodes have the `host`, `port`, or `bind_host` set in either `transport` or general `network` configuration sections. This was documented incorrectly as "nodes that are bound to a non-loopback interface", which lead to confusion as I set `network.host: "localhost"` and the check was still failing.

This change updates the docs to detail the actual check. I think it also highlights how complex the check is and the need for a simpler solution.
2016-03-28 12:53:40 -07:00
javanna a9f4982c40 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-25 20:16:40 +01:00
Clinton Gormley 3da7393b00 Tidied up percolator doc annotations 2016-03-25 15:36:51 +01:00
Boaz Leskes b8227a7222 Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip #17288
discovery.zen.minimum_master_nodes is the single most important setting to set on a production cluster. We have no way of supplying a good default so it must be set by the user. Binding a node to a public IP (as opposed to the default local host) is a good enough indication that a node will be part of a production cluster cluster and thus it's a good tradeoff to enforce the settings. Note that nothing prevent users from setting it to 1 in a single node cluster.

Closes #17288
2016-03-25 12:56:20 +01:00
Jason Tedor 7f0134e725 Revert "Merge pull request #16843 from xuzha/s3-encryption"
This reverts commit 37a183d9ed, reversing
changes made to 08903f1ed8.
2016-03-24 17:11:02 -04:00
Xu Zhang 7499e3aa4a Update and rebase the init implementation.
Also removes the MD5 checks from our side, AWS S3 SDK java is doing the
check.
2016-03-24 11:21:40 -07:00
Clinton Gormley 08903f1ed8 Tidied the Painless docs and added the experimental tag 2016-03-24 18:34:40 +01:00
javanna 27d4994aff Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-24 18:10:11 +01:00
Boaz Leskes 6dd164d0bd Include pings from client nodes in master election
We currently have a `discovery.zen.master_election.filter_client` setting that control whether their ping responses are ignored for master election (which is the current default). With the push to treat client nodes as normal nodes (and promote the transport/rest clients for client work), this should be changed. This commit remove this setting and it's companion `discovery.zen.master_election.filter_data` setting (currently defaulting to  false) in favor of singe `discovery.zen.master_election.ignore_non_master_pings` setting with more intuitive name (defaulting to false).

Resolves #17325
Closes #17329
2016-03-24 17:48:05 +01:00
Clinton Gormley 0e7054cf02 Merge pull request #17272 from pengqiuyuan/master
Update template-query.asciidoc
2016-03-24 16:49:49 +01:00
javanna ce86fc5647 Cluster Stats: remove mem section
The available memory metric was always set to `0` since 2.0.beta1 (bug).  was left behind but never set. Turns out the section wasn't that useful, as it would only output the total memory available throughout all nodes in the cluster. We decided to remove the section then.
2016-03-24 15:49:27 +01:00
Isabel Drost-Fromm 08d989d9b6 Merge branch 'master' into deprecation/sort-option-reverse-removal
Conflicts:
	core/src/main/java/org/elasticsearch/search/sort/FieldSortBuilder.java
	core/src/main/java/org/elasticsearch/search/sort/ScoreSortBuilder.java
2016-03-24 12:06:10 +01:00
Isabel Drost-Fromm 801d178ade Remove mention of reverse in docs and add to migration doc 2016-03-24 12:04:31 +01:00
Jim Ferenczi da42f199bd Enforce isolated mode for all plugins
This commit removes the isolated option, each plugin have its own classloader.
2016-03-24 09:17:33 +01:00
Areek Zillur e16e113691 Remove suggest threadpool
In #17198, we removed suggest transport action, which
used the `suggest` threadpool to execute requests. Now
`suggest` threadpool is unused and suggest requests are
executed on the `search` threadpool.
2016-03-23 18:01:45 -04:00
debadair ad28fb9ec0 Docs: Adding Painless to the Scripting documentation. 2016-03-23 13:52:40 -07:00
Areek Zillur 442a6e0009 document suggest stats being merged with search stats 2016-03-23 16:37:57 -04:00
Areek Zillur e7e93f98e3 add migration guide to use search api for suggest 2016-03-23 16:37:57 -04:00
David Pilato 6e80b5f2dd Merge branch 'fix/17244-s3-chunk-buffer-sizes' 2016-03-23 18:51:54 +01:00
Nik Everett bbe0f3af3b Merge pull request #17255 from simonw/patch-2
Link to named queries docs from bool query page
2016-03-23 11:51:40 -04:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
javanna 4bfef1fde1 [DOCS] clarify that tribe node connects to every node in every cluster
Closes #16756
2016-03-23 10:43:58 +01:00
David Pilato e907b7c11e Check that S3 setting `buffer_size` is always lower than `chunk_size`
We can be better at checking `buffer_size` and `chunk_size` for S3 repositories.
For example, we know that:

* `buffer_size` should be more than `5mb`
* `chunk_size` should be no more than `5tb`
* `buffer_size` should be lower than `chunk_size`

Otherwise, setting `buffer_size` is useless.

For the record:

`chunk_size` is a Snapshot setting whatever the implementation is.
`buffer_size` is an S3 implementation setting.

Let say that you are snapshotting a 500mb file. If you set `chunk_size` to `200mb`, then Snapshot service will call S3 repository to snapshot 3 files with the following sizes:

* `200mb`
* `200mb`
* `100mb`

If you set `buffer_size` to `100mb` (AWS maximum size recommendation), the first file of `200mb` will be uploaded on S3 using the multipart feature in 2 chunks and the workflow is basically the following:

* create the multipart request and get back an `id` from AWS S3 platform
* upload part1: `100mb`
* upload part2: `100mb`
* "commit" the full upload using the `id`.

Closes #17244.
2016-03-23 10:39:54 +01:00
pengqiuyuan 80ef18c3b2 Update template-query.asciidoc 2016-03-23 17:14:31 +08:00
Simon Willison fdac0c7c6c Link to named queries docs from bool query page
The named queries feature only makes sense with bool queries, but was not cross-referenced from the bool query documentation page.
2016-03-22 12:07:57 -07:00