Commit Graph

6929 Commits

Author SHA1 Message Date
lgypro be3090138e [Docs] Fix typo in _analyze api docs (#53837) 2020-03-20 12:02:40 +01:00
István Zoltán Szabó 53f7e31462 [DOCS] Fixes typo in start datafeed API docs. (#53811) 2020-03-19 17:56:40 +01:00
István Zoltán Szabó 29583288b3 [DOCS] Adds performance considerations section to transforms overview (#53791)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-03-19 17:51:00 +01:00
Lisa Cawley 68f7036a4c [DOCS] Adds example links to transform tutorial (#53640) 2020-03-19 09:41:57 -07:00
István Zoltán Szabó 00203c35fe [DOCS] Changes seconds to milliseconds since the Epoch in AD docs. (#53797) 2020-03-19 15:42:58 +01:00
Dominic Page b0884baf46
Geo shape query vs geo point backport (#53774)
Backport to 7x

Enable geo_shape query to work on geo_point fields for shapes: circle, polygon, multipolygon, rectangle see: #48928
Co-Authored-By:  @iverase
2020-03-19 13:00:36 +01:00
James Rodewig 8f4a3eb07f [DOCS] Add token graph concept docs (#53339)
Adds conceptual docs for token graphs.
These docs cover:

* How a token graph is constructed from a token stream
* How synonyms and multi-position tokens impact token graphs
* How token graphs are used during search
* Why some token filters produce invalid token graphs

Also makes the following supporting changes:
* Adds anchors to the 'Anatomy of an Analyzer' docs for cross-linking
* Adds several SVGs for token graph diagrams
2020-03-19 07:43:18 -04:00
Leaf-Lin b649b0d273
Update jvm options doc
There's a typo with extra `
2020-03-19 12:00:01 +11:00
Benjamin Trent 415d73c27d
[Transform] renamed _cat/transform to _cat/transforms (#53743) (#53771)
renaming _cat/transform to  _cat/transforms for uniformity with the other _cat apis.
2020-03-18 19:54:03 -04:00
Lisa Cawley 268e512f0b [DOCS] Add transform nodes (#53698) 2020-03-18 15:26:37 -07:00
James Rodewig 0e2e06bd7e [DOCS] Remove incorrect parms from put index template API docs (#53750)
Removes the `flat_settings` and `timeout` query parameters from the JSON
spec and asciidoc docs for the put index template API.

These parameters are not supported by the API.
2020-03-18 14:36:27 -04:00
markharwood 598d4c1bf9
Formatting fix
Bullet points were not rendered correctly
2020-03-18 15:36:52 +00:00
Lisa Cawley c6e37c7662 [DOCS] Adds stub for cat transform API (#53737) 2020-03-18 08:31:10 -07:00
James Rodewig 0732475cb9 [DOCS] Streamline `analyzer` mapping parm def (#51874)
Simplifies the `analyzer` mapping parameter definition to remove
duplicated analysis content and examples.
2020-03-18 09:43:59 -04:00
James Rodewig e4b7af11ab [DOCS] Remove `light_bengali` stemmer (#53697)
Only the `bengali` stemmer is available in Lucene and surfaced through
Elasticsearch. This removes the incorrect `light_bengali` link in our
docs.
2020-03-18 08:34:27 -04:00
Dan Hermann 94ac979c66
Support array for all string ingest processors (#53694) 2020-03-18 07:07:49 -05:00
Ralph Ursprung 474dfbf9c7 [Docs] Fix highlighting in match-query example (#52426) 2020-03-18 11:53:14 +01:00
Tianlun Li e7ae9ae596 Deprecate delaying state recovery for master nodes (#53646)
It is useful to be able to delay state recovery until enough data nodes have
joined the cluster, since this gives the shard allocator a decent opportunity
to re-use as much existing data as possible. However we also have the option to
delay state recovery until a certain number of master-eligible nodes have
joined, and this is unnecessary: we require a majority of master-eligible nodes
for state recovery, and there is no advantage in waiting for more.

This commit deprecates the unnecessary settings in preparation for their
removal.

Relates #51806
2020-03-18 10:04:22 +00:00
Yash Joshi 85cc7a06c4 [Docs] Fix typo in range query (#53656) 2020-03-18 10:18:08 +01:00
Karen Metts 3c5437894e
Remove link to old settings 7.x (#53639) 2020-03-17 14:38:50 -04:00
David Turner 347b7594b7 Fix deprecation in history retention docs (#53655)
This commit adjusts a `deprecation[...]` message in the docs since such
messages must be on a single line. It also moves this message to the start of
the description of the deprecated setting as is the case with other such
messages.
2020-03-17 14:06:37 +00:00
Luca Cavanna c3d2417448
Cumulative backport of async search changes (#53635)
* Submit async search to work only with POST (#53368)

Currently the submit async search API can be called using both GET and POST at REST, but given that it submits a call and creates internal state, POST should be the only allowed method.

* Refine SearchProgressListener internal API (#53373)

The following cumulative improvements have been made:
- rename `onReduce` and `notifyReduce` to `onFinalReduce` and `notifyFinalReduce`
- add unit test for `SearchShard`
- on* methods in `SearchProgressListener` shouldn't need to be public as they should never be called directly, they only need to be overridden hence they can be made protected. They are actually called directly from a test which required some adapting, like making `AsyncSearchTask.Listener` class package private instead of private
- Instead of overriding `getProgressListener` in `AsyncSearchTask`, as it feels weird to override a getter method, added a specific method that allows to retrieve the Listener directly without needing to cast it. Made the getter and setter for the listener final in the base class.
- rename `SearchProgressListener#searchShards` methods to `buildSearchShards` and make it static given that it accesses no instance members
- make `SearchShard` and `SearchShardTask` classes final

* Move async search yaml tests to x-pack yaml test folder (#53537)

The yaml tests for async search currently sit in its qa folder. There is no reason though for them to live in a separate folder as they don't require particular setup. This commit moves them to the main folder together with the other x-pack yaml tests so that they will be run by the client test runners too.

* [DOCS] Add temporary redirect for async-search (#53454)

The following API spec files contain a link to a not-yet-created
async search docs page:

* [async_search.delete.json][0]
* [async_search.get.json][1]
* [async_search.submit.json][2]

The Elaticsearch-js client uses these spec files to create their docs.
This created a broken link in the Elaticsearch-js docs, which has broken
the docs build.

This PR adds a temporary redirect for the docs page. This redirect
should be removed when the actual API docs are added.

[0]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json
[1]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json
[2]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-17 00:08:17 +01:00
Ryan Earle b4e203a735 [DOCS] Remove `force` as valid value for `version_type` (#53428)
The `force` value for the `version_type` parameter was deprecated in 6.8.

This removes the value from the parameter definition.
2020-03-16 16:50:58 -04:00
Tim Brooks 531cd5aaa4
Add documentation for remote cluster proxy mode (#52779)
This is related to #49067.
2020-03-16 14:18:29 -06:00
Lisa Cawley 278e3fce50 [DOCS] Add anchors for scripted metric aggregations (#53618) 2020-03-16 12:20:41 -07:00
Abhilash Bolla 508e5239b4 [DOCS] Remove unneeded anchor from ILM snippet (#53432) 2020-03-16 15:19:50 -04:00
Nik Everett f7482f794a
Improve top_metrics docs (#53521) (#53619)
* Removes experimental.
* Replaces `"v"` (for value) with `"m"` (for metric).
* Move the note about tiebreaking into the list of limitations of the
  sort.
* Explain how you ask for `metrics`.
* Clean up some wording.
* Link to the docs from `top_metrics`.

Closes #51813
2020-03-16 13:47:43 -04:00
Andrei Stefan 91ca9c5c33
QL: constant_keyword support (#53241) (#53602)
(cherry picked from commit d6cd4ce7849ba215407c8c5fa815c9b373fb8480)
2020-03-16 18:06:31 +02:00
James Rodewig e1eebea846
[DOCS] Reformat `remove_duplicates` token filter (#53608)
Makes the following changes to the `remove_duplicates` token filter
docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Adds custom analyzer example
2020-03-16 11:37:06 -04:00
markharwood 2c74f3e22c
Backport of new wildcard field type (#53590)
* New wildcard field optimised for wildcard queries (#49993)

Indexes values using size 3 ngrams and also stores the full original as a binary doc value.
Wildcard queries operate by using a cheap approximation query on the ngram field followed up by a more expensive verification query using an automaton on the binary doc values.  Also supports aggregations and sorting.
2020-03-16 15:07:13 +00:00
Mayya Sharipova a906f8a0e4
Highlighters skip ignored keyword values (#53408) (#53604)
Keyword field values with length more than ignore_above are not
indexed. But highlighters still were retrieving these values
from _source and were trying to highlight them. This sometimes lead to
errors if a field length exceeded  max_analyzed_offset. But also this
is an overall wrong behaviour to attempt to highlight something that was
ignored during indexing.

This PR checks if a keyword value was ignored because of its length,
and if yes, skips highlighting it.

Backport: #53408
Closes #43800
2020-03-16 11:06:25 -04:00
Benjamin Trent 4e43ede735
[ML] renaming inference processor field field_mappings to new name field_map (#53433) (#53502)
This renames the `inference` processor configuration field `field_mappings` to `field_map`.

`field_mappings` is now deprecated.
2020-03-13 15:40:57 -04:00
Tom Veasey 690099553c
[7.x][ML] Adds the class_assignment_objective parameter to classification (#53552)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-13 17:35:51 +00:00
Lisa Cawley b1d589f276 [DOCS] Adds operations_behind to transform stats (#53518) 2020-03-13 09:34:49 -07:00
James Rodewig 1e31031b73 [DOCS] Clarify `max_shingle_size` parm def (#53480)
Rewrites the `search_as_you_type` field datatype's `max_shingle_size`
mapping parameter to improve clarity and better communicate trade-offs
regarding index size.

Relates to [elastic/kibana#55161][0].

Closes #51774.

[0]: https://github.com/elastic/kibana/pull/55161#discussion_r368107177
2020-03-13 04:08:44 -04:00
Nik Everett 9dcd64c110
Preserve metric types in top_metrics (backport of #53288) (#53440)
This changes the `top_metrics` aggregation to return metrics in their
original type. Since it only supports numerics, that means that dates,
longs, and doubles will come back as stored, with their appropriate
formatter applied.
2020-03-12 17:17:09 -04:00
Jim Ferenczi 97621e7f65 Removes old Lucene's experimental flag from analyzer documentations (#53217)
This change removes the Lucene's experimental flag from the documentations of the following
tokenizer/filters:
  * Simple Pattern Split Tokenizer
  * Simple Pattern tokenizer
  * Flatten Graph Token Filter
  * Word Delimiter Graph Token Filter

The flag is still present in Lucene codebase but we're fully supporting these tokenizers/filters
in ES for a long time now so the docs flag is misleading.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-12 21:18:19 +01:00
Jason Tedor 5b08ea84c9
Add deprecation check for listener thread pool (#53438)
This commit adds a deprecation check for the listener thread pool
settings as these will be removed in 8.0.0.
2020-03-12 14:32:41 -04:00
István Zoltán Szabó 969164cc47 [DOCS] Adds a warning about reindexing docs with the same ID to the PUT DFA docs. (#53490) 2020-03-12 18:01:43 +01:00
James Rodewig af987fb2d4
[DOCS] Reduce content reuse in enrich docs (#53460)
Restructures the 'Update an enrich policy' section to:

* Migrate the content to the section. It was previously stored in the
  Put Enrich Policy API docs.
* Remove the warning tag admonition from the section content.
* Replace a reused section earlier in the "Set up an enrich processor"
  page with a link.

No substantive changes were made to the content.
2020-03-12 05:57:23 -04:00
Benjamin Trent 89668c5ea0
[ML][Inference] adds new default_field_map field to trained models (#53294) (#53419)
Adds a new `default_field_map` field to trained model config objects.

This allows the model creator to supply field map if it knows that there should be some map for inference to work directly against the training data.

The use case internally is having analytics jobs supply a field mapping for multi-field fields. This allows us to use the model "out of the box" on data where we trained on `foo.keyword` but the `_source` only references `foo`.
2020-03-11 13:49:39 -04:00
Ignacio Vera 562a9eff33
remove sneaked placeholder from histogram docs (#53391) (#53405) 2020-03-11 14:04:22 +01:00
James Rodewig 933a9c6fca
[DOCS] Reformat `word_delimiter` token filter (#53387)
Makes the following changes to the `word_delimiter` token filter docs:

* Adds a warning admonition recommending the `word_delimiter_graph`
  filter instead. This warning includes a link to the deprecated Lucene
  `WordDelimiterFilter`.
* Updates the description
* Adds detailed analyze snippet
* Adds custom analyzer and custom filter snippets
* Reorganizes and updates parameter documentation
2020-03-11 09:03:57 -04:00
Dimitris Athanasiou 0fd0516d0d
[7.x][ML] Rename data frame analytics maximum_number_trees to max_trees (#53300) (#53390)
Deprecates `maximum_number_trees` parameter of classification and
regression and replaces it with `max_trees`.

Backport of #53300
2020-03-11 12:45:27 +02:00
James Rodewig a9dd7773d2 [DOCS] Use keyword tokenizer in word delimiter graph examples (#53384)
In a tip admonition, we recommend using the `keyword` tokenizer with the
`word_delimiter_graph` token filter. However, we only use the
`whitespace` tokenizer in the example snippets. This updates those
snippets to use the `keyword` tokenizer instead.

Also corrects several spacing issues for arrays in these docs.
2020-03-11 04:46:33 -04:00
James Rodewig 166b5a92f6 [DOCS] Correct anchor in word delimiter graph token filter docs 2020-03-10 10:32:42 -04:00
James Rodewig d503bf9a45 [DOCS] Make token graph diagrams consistent (#53331)
Updates the SVG for a token graph to make the layout consistent with
other graphs. This means moving the text directly above the edge lines.
Previously, the text was above the edge line.
2020-03-10 06:53:28 -04:00
James Rodewig 5e3df18d56 [DOCS] Adds Beats tip to EQL search docs (#53292)
Adds a tip admonition to the basic example in the EQL search docs.

This tip lets users know they can set up a Beat to automatically
index data in ES, rather than manually indexing using the bulk or index
APIs.
2020-03-10 05:16:18 -04:00
James Rodewig 641eb69520 [DOCS] Document `nodes` cluster stats (#52813)
Documents the `nodes` response parameters returned by the
`_cluster/stats` API.

Also adds collapsible attributes for the `indices` and `nodes`
sections.
2020-03-10 05:05:05 -04:00
Jason Tedor 1860c57147
Deprecate the listener thread pool (#53266)
The listener thread pool is being removed from use in the server
codebase. This commit deprecates configuring the listener thread pool.
2020-03-09 16:56:01 -04:00
Julie Tibshirani c33afea9fb Small corrections to stored_fields docs. (#53247)
* Fix a reference to the 'field' option.
* Remove claim about detecting script fields.
* Specify that object fields will just be ignored.
2020-03-09 10:59:17 -07:00
Romain Gonord 00657901ec [DOCS] Fix "Wait for Snapshot" link in ILM actions reference (#53269)
Corrects an anchor for the `Wait for Snapshot` action, which previously linked to the `Delete` action.
2020-03-09 08:40:36 -04:00
Anton Dollmaier 35c8226419 [DOCS] Fix parameter formatting for GeoHash grid agg docs (#53032)
Adds missing colon (`:`) to the parameter definition list.
2020-03-09 08:16:40 -04:00
James Rodewig 28cb4a167d
[DOCS] Reformat `word_delimiter_graph` token filter (#53170) (#53272)
Makes the following changes to the `word_delimiter_graph` token filter
docs:

* Updates the Lucene experimental admonition.
* Updates description
* Adds analyze snippet
* Adds custom analyzer and custom filter snippets
* Reorganizes and updates parameter list
* Expands and updates section re: differences between `word_delimiter`
  and `word_delimiter_graph`
2020-03-09 06:45:44 -04:00
István Zoltán Szabó aafc0409a9 [DOCS] Makes the description clearer on how to use aggregations in an anomaly detection job (#53103)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-09 09:49:59 +01:00
Lisa Cawley 341417613e
[7.x][DOCS] Adds common definitions for security settings (#51017) (#53242)
Co-Authored-By: Tim Vernum <tim@adjective.org>
2020-03-06 16:28:54 -08:00
Gordon Brown ff9b8bda63
Implement hidden aliases (#52547)
This commit introduces hidden aliases. These are similar to hidden
indices, in that they are not visible by default, unless explicitly
specified by name or by indicating that hidden indices/aliases are
desired.

The new alias property, `is_hidden` is implemented similarly to
`is_write_index`, except that it must be consistent across all indices
with a given alias - that is, all indices with a given alias must
specify the alias as either hidden, or all specify it as non-hidden,
either explicitly or by omitting the `is_hidden` property.
2020-03-06 16:02:38 -07:00
Adam Canady a88d0c7ca3 [Docs] Correct examples for * and + in regexp-syntax.asciidoc (#53210) 2020-03-06 17:17:32 +01:00
István Zoltán Szabó bf3dcd4229 [DOCS] Adds deleting flag to the GET job stats API docs (#53223) 2020-03-06 16:04:20 +01:00
James Rodewig 9bb9f63364 [DOCS] Note that `trim` filter doesn't change offsets (#53220)
The [word delimiter graph token filter docs][0] note that the `trim`
filter changes the length of tokens without changing their offsets.

This explicitly mentions that in the `trim` filter docs.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html
2020-03-06 07:32:35 -05:00
James Rodewig 4bc6d2dbec [DOCS] Correct link for Lucene StopFilter 2020-03-05 14:52:25 -05:00
Mayya Sharipova 7e2a9f58ee
script_score query errors on negative scores (#53133)
7.5 and 7.6 had a regression that allowed for
script_score queries to have negative scores.
We have corrected this regression in #52478.
This is an addition to #52478 that adds
a test and release notes.
2020-03-05 14:23:39 -05:00
David Turner c2627aa22f Clarify futher the order for a rolling upgrade (#52964)
Expands the "master-ineligible then master-eligible" sentence into a list and
specifies that within these subsets the order doesn't matter.
2020-03-05 15:28:59 +00:00
István Zoltán Szabó 58ce56f6c8 [DOCS] Makes the naming convention of the DFA response objects coherent (#53172) 2020-03-05 16:26:57 +01:00
István Zoltán Szabó 48707ec55a [DOCS] Expands GET DFA stat API docs with response objects. (#53107) 2020-03-05 15:31:55 +01:00
Nik Everett 28df7ae5ed
Support multiple metrics in `top_metrics` agg (backport of #52965) (#53163)
This adds support for returning multiple metrics to the `top_metrics`
agg. It looks like:
```
POST /test/_search?filter_path=aggregations
{
  "aggs": {
    "tm": {
      "top_metrics": {
        "metrics": [
          {"field": "v"},
          {"field": "m"}
        ],
        "sort": {"s": "desc"}
      }
    }
  }
}
```
2020-03-05 08:12:01 -05:00
James Rodewig e46bb54c7b
[DOCS] Document `any` keyword in EQL syntax (#52821) (#53157)
Adds documentation for the `any` keyword to the EQL syntax docs.

Includes:

* Definition of an event category and its relationship to the event
   category field.
* Example matching all event categories using `any` keyword
* Example using `any` with `where true`
2020-03-05 05:02:47 -05:00
James Rodewig 801e50203e [DOCS] Add missing doc type to EQL search results 2020-03-04 10:26:11 -05:00
Yannick Welsch d1e7951e00 [DOCS] Add 7.6.1. release notes (#52874)
Adds the release notes for 7.6.1.
2020-03-04 15:47:54 +01:00
James Rodewig e3d3c3400c [DOCS] Update EQL default event category and timestamp values (#53102)
Updates the documented default `event_category_field` and `timestamp_field`
values for the EQL search API. Also updates related guidance in the
EQL requirement docs.

Relates to #53073.
2020-03-04 09:17:37 -05:00
James Rodewig 0c4bf64095 [DOCS] Fix several Asciidoctor double arrow replacements (#52827)
Per the [Asciidoctor docs][0], Asciidoctor replaces the following
syntax with double arrows in the rendered HTML:

* => renders as ⇒
* <= renders as ⇐

This escapes several unintended replacements, such as in the Painless
docs.

Where appropriate, it also replaces some double arrow instances with
single arrows for consistency.

[0]: https://asciidoctor.org/docs/user-manual/#replacements
2020-03-04 08:43:19 -05:00
James Rodewig 2b59f8ac34 [DOCS] Correct `hits.total.relation` response parm def (#52847)
Fixes a partially completed definition for the `hits.total.relation`
response parameter in the search API docs.
2020-03-04 08:23:34 -05:00
Aleksandr Maus b47bffba24
EQL: consistent naming for event type vs event category (#53073) (#53090)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Lisa Cawley 892f0d5848 [DOCS] Adds link in datafeed indices_options (#53067) 2020-03-03 10:31:32 -08:00
Lisa Cawley bb4bcbc54c [DOCS] Adds ignore_throttled to index conventions (#53063) 2020-03-03 10:23:56 -08:00
James Rodewig cf87724ff6
[DOCS] Reformat `stop` token filter (#53059)
Makes the following changes to the `stop` token filter docs:

* Updates description
* Adds a link to the related Lucene filter
* Adds detailed analyze snippet
* Updates custom analyzer and custom filter snippets
* Adds a list of predefined stop words by language

Co-authored-by: ScottieL <36999642+ScottieL@users.noreply.github.com>
2020-03-03 13:22:52 -05:00
István Zoltán Szabó 6cece3a709 [DOCS] Adds response body documentation to GET inference API (#53050) 2020-03-03 16:26:40 +01:00
Adrien Grand cb868d2f5e
Introduce a `constant_keyword` field. (#49713) (#53024)
This field is a specialization of the `keyword` field for the case when all
documents have the same value. It typically performs more efficiently than
keywords at query time by figuring out whether all or none of the documents
match at rewrite time, like `term` queries on `_index`.

The name is up for discussion. I liked including `keyword` in it, so that we
still have room for a `singleton_numeric` in the future. However I'm unsure
whether to call it `singleton`, `constant` or something else, any opinions?

For this field there is a choice between
 1. accepting values in `_source` when they are equal to the value configured
    in mappings, but rejecting mapping updates
 2. rejecting values in `_source` but then allowing updates to the value that
    is configured in the mapping
This commit implements option 1, so that it is possible to reindex from/to an
index that has the field mapped as a keyword with no changes to the source.

Backport of #49713
2020-03-03 16:01:47 +01:00
James Rodewig bcb68c860c [DOCS] Reorganize EQL requirements page 2020-03-03 07:02:30 -05:00
James Rodewig 5cffa14f45 [DOCS] Fix typo in EQL docs 2020-03-02 16:09:03 -05:00
Costin Leau 712e0c05cd EQL: Add implicit ordering on timestamp (#53004)
QL: Move Sort base class from SQL to QL
(cherry picked from commit 798015b7bbd565e9c4222724614baeb432c7c2b3)
2020-03-02 22:41:36 +02:00
Orhan Toy ad2f630795 [DOCS] Fix formatting of simulate ingest pipeline API docs (#52754) 2020-03-02 11:46:27 -05:00
István Zoltán Szabó a267849def [DOCS] Adds transform-settings page and its subpage to redirects (#52944)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-02 16:36:40 +01:00
Lisa Cawley 4fbe1b0550
[DOCS] Adds cat anomaly detectors API (#52866) (#52970) 2020-03-02 07:28:55 -08:00
Hendrik Muhs a328a8eaf1
[7.x][Transform] implement node.transform to control where to… (#52998)
implement transform node attributes to disable transform on certain nodes and
test which nodes are allowed to do remote connections

closes #52200
closes #50033
closes #48734

backport #52712
2020-03-02 16:10:57 +01:00
James Rodewig db64029919
[7.x] [DOCS] Add parameter examples to EQL search tutorial (#52953)
Makes the following updates to the EQL search tutorial:

* Adds an API response to the basic tutorial
* Adds an example using the `event_type_field` parm
* Adds an example using the `timestamp_field`parm
* Adds an example using the `query` parm
* Updates example dataset to support more EQL query variety
2020-03-02 10:08:03 -05:00
Aleksandr Maus 89ed857c79
EQL: Change request parameter query to filter and rule to query (#52971) (#53006)
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
James Rodewig d336faa0b0 [DOCS] Reformat trim token filter docs (#51649)
Makes the following changes to the `trim` token filter docs:

* Updates description
* Adds a link to the related Lucene filter
* Adds tip about removing whitespace using tokenizers
* Adds detailed analyze snippets
* Adds custom analyzer snippet
2020-03-02 07:48:23 -05:00
James Rodewig f5bccad847 [DOCS] Correct guidance for `index_options` mapping parm (#52899)
Adds a warning admonition stating that the `index_options` mapping
parameter is intended only for `text` fields.

Removes an outdated statement regarding default values for numeric
and other datatypes.
2020-03-02 07:39:35 -05:00
rhymes 7eb4c07f1f [DOCS] Fix typo in index and search analysis docs (#52988) 2020-03-02 07:25:01 -05:00
Dimitris Athanasiou 85b4e45093
[7.x]ML] Parse and report memory usage for DF Analytics (#52778) (#52980)
Adds reporting of memory usage for data frame analytics jobs.
This commit introduces a new index pattern `.ml-stats-*` whose
first concrete index will be `.ml-stats-000001`. This index serves
to store instrumentation information for those jobs.

Backport of #52778 and #52958
2020-02-29 13:03:40 +02:00
Rory Hunter b1be7dcd2d Document how to change GC logging behaviour (#52879)
Closes #43990. Describe how to change the default GC settings without changing
the default `jvm.options`. Give examples using `jvm.options.d`, and
`ES_JAVA_OPTS` with Docker.
2020-02-28 21:27:45 +00:00
Martijn van Groningen 6aa9aaa2c6
Add validation for dynamic templates (#52890)
Backport of #51233 to the seven dot x branch.

Tries to load a `Mapper` instance for the mapping snippet of a dynamic template.
This should catch things like using an analyzer that is undefined or mapping attributes that are unused.

This is best effort:
* If `{{name}}` placeholder is used in the mapping snippet then validation is skipped.
* If `match_mapping_type` is not specified then validation is performed for all mapping types.
  If parsing succeeds with a single mapping type then this the dynamic mapping is considered valid.

If is detected that a dynamic template mapping snippet is invalid at mapping update time then the mapping update is failed for indices created on 8.0.0-alpha1 and later. For indices created on prior version a deprecation warning is omitted instead. In 7.x clusters the mapping update will never fail in case of an invalid dynamic template mapping snippet and a deprecation warning will always be omitted.

Closes #17411
Closes #24419

Co-authored-by: Adrien Grand <jpountz@gmail.com>
2020-02-28 10:35:04 +01:00
Nik Everett 1d1956ee93
Add size support to `top_metrics` (backport of #52662) (#52914)
This adds support for returning the top "n" metrics instead of just the
very top.

Relates to #51813
2020-02-27 16:12:52 -05:00
Benjamin Trent eac38e9847
[ML] Add indices_options to datafeed config and update (#52793) (#52905)
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index.

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.

closes https://github.com/elastic/elasticsearch/issues/48056
2020-02-27 13:43:25 -05:00
Nattachai Suteerapongpan 14f847cc8f [DOCS] Fix typo in task management API docs (#52881) 2020-02-27 11:31:11 -05:00
Josh Devins 68ba571f70
Adds recall@k metric to rank eval API (#52889)
This change adds the recall@k metric and refactors precision@k to match
the new metric.

Recall@k is an important metric to use for learning to rank (LTR)
use-cases. Candidate generation or first ranking phase ranking functions
are often optimized for high recall, in order to generate as many
relevant candidates in the top-k as possible for a second phase of
ranking. Adding this metric allows tuning that base query for LTR.

See: https://github.com/elastic/elasticsearch/issues/51676
Backports: https://github.com/elastic/elasticsearch/pull/52577
2020-02-27 16:04:24 +01:00
István Zoltán Szabó 8785f57dfe [DOCS] Reformats cat DFA API docs. (#52885) 2020-02-27 14:21:52 +01:00
István Zoltán Szabó 4a33352a94 [DOCS] Adds cat trained model API documentation (#52824) 2020-02-27 12:54:11 +01:00
Costin Leau 40bc06f6ad EQL: Hook engine to Elasticsearch (#52828)
Add query execution and return actual results returned from
Elasticsearch inside the tests

(cherry picked from commit 3e039282bf991af87604a6d4f8eada19d5e33842)
2020-02-27 11:22:22 +02:00
David Turner 69b78f7f8a "Adding nodes" instructions only work on localhost (#52677)
The introductory sections of the reference manual contains some simplified
instructions for adding a node to the cluster. Unfortunately they are a little
too simplified and only really work for clusters running on `localhost`. If you
try and follow these instructions for a distributed cluster then the new node
will, confusingly, auto-bootstrap itself into a distinct one-node cluster.

Multiple nodes running on localhost is a valid config, of course, but we should
spell out that these instructions are really only for experimentation and that
it takes a bit more work to add nodes to a distributed cluster. This commit
does so.

Also, the "important config" instructions for discovery say that you MUST set
`discovery.seed_hosts` whereas in fact it is fine to ignore this setting and
use a dynamic discovery mechanism instead. This commit weakens this statement
and links to the docs for dynamic discovery mechanisms.

Finally, this section is also overloaded with some technical details that are
not important for this context and are adequately covered elsewhere, and
completely fails to note that the default discovery port is 9300. This commit
addresses this.
2020-02-27 09:18:37 +00:00
James Rodewig f5253d20f7 [DOCS] Update term vectors snippet to prevent CI failure (#52819)
Adds the `?refresh=wait_for` query argument to an index API snippet in
the term vectors API docs.

This should ensure the document is indexed and available before a
subsequent term vectors API request executes.

Fixes #52814.
2020-02-26 12:41:40 -05:00
Lisa Cawley b788ec7157 [DOCS] Adds cat datafeeds API (#52738) 2020-02-26 09:28:57 -08:00
Bogdan Pintea 304e1e69b8 remove references to the SQL API from ODBC config (#52765)
Remove reference to an "SQL API" which could suggest that one needs to
treat this in a special way when configuring the ODBC driver.

(cherry picked from commit 451c341e0193b542409e8891ec2a31e62529a5e7)
2020-02-26 13:39:54 +01:00
István Zoltán Szabó f57422bbfd [DOCS] Adds cat data frame analytics API (#52764)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-26 11:10:42 +01:00
Lisa Cawley 05f1cd74a6 [DOCS] Fixes monitoring links (#52790) 2020-02-25 18:08:23 -08:00
Lisa Cawley 924f0bd243 [DOCS] Updates custom rules example (#52731) 2020-02-25 09:32:52 -08:00
Andrei Stefan 51c6aefa55
SQL: Use calendar_interval of 1d for HISTOGRAMs with 1 DAY intervals (#52749) (#52771)
(cherry picked from commit 556f5fa33be88570c4f8550cb8f784323d26a707)
2020-02-25 18:44:02 +02:00
Pius 563f033511 Update ilm-settings.asciidoc (#51577) 2020-02-25 10:18:55 -05:00
bellengao d2db16e046 [DOCS] Correct policy name in ILM docs example (#52354)
Updates an example snippet to use a consistent policy name.
2020-02-25 09:36:22 -05:00
David Pilato 6c6ab8fa47 [DOS] Fix typo in CSV processor docs (#52649)
Corrects an example array in a snippet of the CSV processor docs.
2020-02-25 08:48:50 -05:00
bellengao 49f37989c4 [DOCS] Fix typo in ingest node docs (#52671) 2020-02-25 07:57:52 -05:00
David Roberts cf122d13b8 [ML] Use event.timezone in file_structure_finder ingest pipeline (#52720)
This is because beat.timezone was renamed to event.timezone in
elastic/beats#9458
2020-02-25 12:33:53 +00:00
James Rodewig 9b05f6a668 [DOCS] Add admonition for app using cat APIs (#52727)
Adds an explicit "important" admonition discouraging apps from using
cat APIs.

cat APIs are intended for human consumption via the command line or
Kibana console only. They are not intended for consumption by
applications.
2020-02-25 07:20:33 -05:00
James Rodewig 1a14ae4e1b [DOCS] Document `include_in_*` nested mapping parms (#52648)
Adds documentation for the `include_in_parent` and `include_in_root`
mapping parameters for the `nested` mapping datatype.
2020-02-25 07:13:49 -05:00
Adrien Grand 5f81906fcf Discourage from opting in for the `niofs` store. (#52638)
Indices open with the `niofs` store type load much more data on-heap than
indices open with the `mmapfs` store type. This limitation is now documented
and examples have been updated to show how to update settings to use the
`mmapfs` store type rather than `niofs`.
2020-02-25 08:54:11 +01:00
Adrien Grand 9b0ddc1c03 Clarify the resiliency trade-off of disabling replicas to speed up indexing. (#52714)
We should be more explicit about the downsides of disabling replicas and
explain that users should be ready to re-do the entire load in case of
issues mid-way.
2020-02-25 08:54:10 +01:00
Adrien Grand 5ce66b8b3c Document how CCR may be used to speed up indexing. (#52717)
One architecture that we have recommended to several users to speed up
indexing involved using CCR to prevent searching from stealing resources
from indexing.
2020-02-25 08:54:10 +01:00
Bob Blank 28d4b71947
Clarified http.max_content_length description (#52329)
Adding "greater than" based on discussion with @jasontedor for clarity.
2020-02-24 21:01:14 -05:00
Andrei Stefan ed6b10bc03
SQL: use a calendar interval for histograms over 1 month intervals (#52586) (#52715)
(cherry picked from commit 928b11a34ec92d90d082abdf4fa09f7ce1d7c0c4)
2020-02-25 01:41:51 +02:00
Julie Tibshirani ba0401ecfd Correct the name of the search timeout parameter. (#52733)
The request body parameter is called 'timeout', not 'search_timeout'.
2020-02-24 14:59:06 -08:00
lcawl c6e35b460e [DOCS] Adds anchor for custom rules 2020-02-24 11:39:15 -08:00
Mayya Sharipova 034b1c0ba3
Correct boost calculation in script_score query (#52478) (#52724)
Before boost in script_score query was wrongly applied only to the subquery.
This commit makes sure that the boost is applied to the whole score
that comes out of script.

Closes #48465
2020-02-24 13:48:21 -05:00
James Rodewig 5e48811585 [DOCS] Document CCS-supported APIs (#52708)
Explicitly notes the Elasticsearch API endpoints that support CCS.

This should deter users from attempting to use CCS with other API
endpoints, such as `GET <index>/_doc/<_id>`.
2020-02-24 09:59:08 -05:00
Ignacio Vera ba9d3c6389
Add support for multipoint shape queries (#52564) (#52705) 2020-02-24 13:46:51 +01:00
James Rodewig 98bcf06bae [DOCS] Correct multi search API docs (#52523)
* Adds an example request to the top of the page.
* Relocates several parameters erroneously listed under "Request body"
to the appropriate "Query parameters" section.
* Updates the "Request body" section to better document the NDJSON
  structure of msearch requests.
2020-02-24 07:43:10 -05:00
Marios Trivyzas c03f51f68f
[Docs] Clarify default value for `allow_no_indices` (#52635) (#52697)
Add default value to each one of the usages of `allow_no_indices`
since it differs between different APIs.

Relates to: #52534

(cherry picked from commit 2eb986488ac326d6da6ab8ad0203a94e08684a36)
2020-02-24 11:57:32 +01:00
Benjamin Trent afd90647c9
[ML] Adds feature importance to option to inference processor (#52218) (#52666)
This adds machine learning model feature importance calculations to the inference processor.

The new flag in the configuration matches the analytics parameter name: `num_top_feature_importance_values`
Example:
```
"inference": {
   "field_mappings": {},
   "model_id": "my_model",
   "inference_config": {
      "regression": {
         "num_top_feature_importance_values": 3
      }
   }
}
```

This will write to the document as follows:
```
"inference" : {
   "feature_importance" : {
      "FlightTimeMin" : -76.90955548511226,
      "FlightDelayType" : 114.13514762158526,
      "DistanceMiles" : 13.731580450792187
   },
   "predicted_value" : 108.33165831875137,
   "model_id" : "my_model"
}
```

This is done through calculating the [SHAP values](https://arxiv.org/abs/1802.03888).

It requires that models have populated `number_samples` for each tree node. This is not available to models that were created before 7.7.

Additionally, if the inference config is requesting feature_importance, and not all nodes have been upgraded yet, it will not allow the pipeline to be created. This is to safe-guard in a mixed-version environment where only some ingest nodes have been upgraded.

NOTE: the algorithm is a Java port of the one laid out in ml-cpp: https://github.com/elastic/ml-cpp/blob/master/lib/maths/CTreeShapFeatureImportance.cc

usability blocked by: https://github.com/elastic/ml-cpp/pull/991
2020-02-21 18:42:31 -05:00
Mayya Sharipova 3840a763d8 Correct release notes for 7.5 (#52660)
Remove a mention to a feature that was not merged,
as its corresponding PR was closed.
2020-02-21 14:59:46 -05:00
Nik Richers 101bca86d2 [DOCS] Switch to standard ESS trial links (#52552)
Switches ESS trial sign-up links over to a standard attribute. This provides better metrics for how effective these links are.
2020-02-21 12:07:10 -05:00
Lisa Cawley 4ff78e8a00
[7.x][DOCS] Adds X-Pack usage API (#52592) 2020-02-21 06:57:11 -08:00
James Rodewig 068181b0b6 [DOCS] Add missing `indices` parms returned by `_nodes/stats` (#52055)
Adds several human-readable `indices` parameters returned by the
`_nodes/stats` API.
2020-02-21 08:15:59 -05:00
Andrei Stefan 7fe2843a9e SQL: specify command to run the CLI on a remote machine without Elasticsearch (#52626)
(cherry picked from commit 477b0eda8322c5dcb6861bd262bfeec17ff133fe)
2020-02-21 13:29:58 +02:00
James Rodewig 80b77e92d4
[7.x] [DOCS] Re-add redirects for API relocation (#52628)
Re-adds several redirects removed with #50510.

These redirects were related to the relocation of several API docs to
new pages under the 'REST APIs' chapter.

We've since decided to only remove such redirects with major releases.
2020-02-21 05:32:10 -05:00
István Zoltán Szabó 1d895118dd [DOCS] Links transforms in aggregation docs (#52563)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-21 08:23:34 +01:00
Ignacio Vera 107f00a4ec
Add support for multipoint geoshape queries (#52133) (#52553)
Currently multi-point queries are not supported when indexing your data using BKD-backed geoshape strategy. This commit removes this limitation.
2020-02-21 07:45:53 +01:00
Yannick Welsch d76358c875
Deprecate fixed_auto_queue_size thread pool type (#52399)
Relates #52280
2020-02-20 11:11:06 +01:00
Russ Cam 62da077beb Specify name on enrich.get_policy as list type (#50217)
This commit updates the enrich.get_policy API to specify name
as a list, in line with other URL parts that accept a comma-separated
list of values.

In addition, update the get enrich policy API docs
to align the URL part name in the documentation with
the name used in the REST API specs.

(cherry picked from commit 94f6f946ef283dc93040e052b4676c5bc37f4bde)
2020-02-20 11:39:28 +10:00
Lee Hinman b11dbb2205 Correct SLM retention timezone documentation (#52533)
This erroneously said that retention is run in the master node's timezone, however, it is actually
run in UTC.
2020-02-19 13:46:43 -07:00
Valentin Crettaz a68fafd64b [DOCS] Clarify that "now" cannot be used in `date_range` at index time (#52446)
`date_range` fields do not accept `"now"` as a value of either bounds at indexing time.

This corrects an error in the range data type mapping docs.
2020-02-19 12:40:58 -05:00
Bogdan Pintea db8b306085 SQL: update ODBC docs, cover Cloud ID, latest params (#52291)
* Refresh snapshots with latest look

Add new snapshots with the connection editor to reflect the latest UI.

* Document the effect of the late added params

Add details about the Cloud ID setting, as well as those on the Misc
tab.


(cherry picked from commit afa67625e847e99a22264f5dd6fa0daa37786c6f)
2020-02-19 17:42:28 +01:00
James Rodewig 43376c6e06 [DOCS] Document how CCS handles cluster-level settings (#49941)
Updates the cross-cluster search (CCS) documentation to note how
cluster-level settings are applied.

When `ccs_minimize_roundtrips` is `true`, each cluster applies its own
cluster-level settings to the request.

When `ccs_minimize_roundtrips` is `false`, cluster-level settings for
the local cluster is used. This includes shard limit settings, such as
`action.search.shard_count.limit`, `pre_filter_shard_size`, and
`max_concurrent_shard_requests`. If these limits are set too low, the
request could be rejected.
2020-02-19 09:21:57 -05:00
debadair 969cdfaaa4 [DOCS] Clean up links from SQL client app pages. (#52442)
* [DOCS] Clean up links from SQL client app pages.

* Linked to client apps from prereqs.
2020-02-18 12:42:20 -08:00
Lisa Cawley 123b3c6f55 [DOCS] Clarifies description of num_top_feature_importance_values (#52246)
Co-Authored-By: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
2020-02-18 08:50:21 -08:00
James Rodewig 9128106b4c [DOCS] Remove 'analyzed string' references (#51946)
The `string` field datatype was replaced by the `text` and `keyword`
field datatypes in [5.0][0].

This removes several outdated references to 'analyzed string' fields.

[0]:https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html#_string_fields_replaced_by_textkeyword_fields
2020-02-14 12:34:37 -05:00
Andrei Stefan 4eea9c20ee SQL: document the use of a filter on _routing (#52355)
* Fix "Description"s for various sections in the functions pages.
* Added a TIP for searching using a routing key.
* Other small polishings

(cherry picked from commit 9fad0b1ac4409a42c435ed040f41cbaea18930a3)
2020-02-14 19:00:26 +02:00
Lisa Cawley e77e49e956
[DOCS] Adds machine learning highlights (#52334) 2020-02-14 08:51:55 -08:00
Nik Everett 146def8caa
Implement top_metrics agg (#51155) (#52366)
The `top_metrics` agg is kind of like `top_hits` but it only works on
doc values so it *should* be faster.

At this point it is fairly limited in that it only supports a single,
numeric sort and a single, numeric metric. And it only fetches the "very
topest" document worth of metric. We plan to support returning a
configurable number of top metrics, requesting more than one metric and
more than one sort. And, eventually, non-numeric sorts and metrics. The
trick is doing those things fairly efficiently.

Co-Authored by: Zachary Tong <zach@elastic.co>
2020-02-14 11:19:11 -05:00
bellengao cabc1769e2 [DOC] Remove definition typo in update alias API docs (#52184)
Removes an erroneously duplicated definition heading from the update alias API reference docs.
2020-02-14 08:31:21 -05:00
Igor Motov a66988281f
Add histogram field type support to boxplot aggs (#52265)
Add support for the histogram field type to boxplot aggs.

Closes #52233
Relates to #33112
2020-02-13 18:09:26 -05:00
debadair 291713f284 [DOCS] Fixed typo in jump link. (#52302) 2020-02-12 17:53:00 -08:00
Ryan Ernst 12e378b3ac Fix incorrect date nanos docs example (#52249)
The example of how to access the nano value of a date_nanos field has
been broken since it was created. This commit fixes it to use the
correct scripting methods.

closes #51931
2020-02-12 15:55:41 -08:00
Marios Trivyzas dac720d7a1
Add a cluster setting to disallow expensive queries (#51385) (#52279)
Add a new cluster setting `search.allow_expensive_queries` which by
default is `true`. If set to `false`, certain queries that have
usually slow performance cannot be executed and an error message
is returned.

- Queries that need to do linear scans to identify matches:
  - Script queries
- Queries that have a high up-front cost:
  - Fuzzy queries
  - Regexp queries
  - Prefix queries (without index_prefixes enabled
  - Wildcard queries
  - Range queries on text and keyword fields
- Joining queries
  - HasParent queries
  - HasChild queries
  - ParentId queries
  - Nested queries
- Queries on deprecated 6.x geo shapes (using PrefixTree implementation)
- Queries that may have a high per-document cost:
  - Script score queries
  - Percolate queries

Closes: #29050
(cherry picked from commit a8b39ed842c7770bd9275958c9f747502fd9a3ea)
2020-02-12 22:56:14 +01:00
Lisa Cawley 40b58e612d [DOCS] Fixes, sorts ML tagged regions (#52283) 2020-02-12 13:52:34 -08:00
Marios Trivyzas d9fd6fc90c
SQL: [Docs] Fix typo
Add missing closing "`"

Follows: c2e0552537
2020-02-12 21:50:57 +01:00
James Rodewig ca34817659 [DOCS] Add EQL limitations page (#52001)
Documents limitations for EQL in Elasticsearch.
2020-02-12 08:45:43 -05:00
James Rodewig 20453d3ac8 [DOCS] Add basic EQL search tutorial docs (#51574)
I plan to add additional sections to this page with future PRs:

* Specify timestamp and event type fields
* Specify a join key field
* Filter using query DSL
* Paginate a large response

See #51057.
2020-02-12 08:42:09 -05:00
James Rodewig 3f151d1d75 [DOCS] Add redirects, update JSON spec to fix docs build (#51747)
Docs build [#11556][0] broke due to several outdated or incorrect links
in the JSON REST spec.

This fixes those links where possible and adds redirects.

[0]: https://elasticsearch-ci.elastic.co/job/elastic+docs+master+build/11556/
2020-02-12 08:30:59 -05:00
Marios Trivyzas c2e0552537 SQL: [Docs] Add limitation for sorting on aggs (#52210)
Add a section to point out that when ordering by an aggregate
only plain aggregate functions are allowed, no scalars/operators
can be used on top of them.

Fixes: #52204
(cherry picked from commit 78a1185549ff7f3229fd2d036567eb2a4f2cf230)
2020-02-12 12:56:06 +01:00
James Rodewig 6fe8f1649b [DOCS] Include docs on permanently unreleased branches only (#51743)
Adds the ability to display docs on permanently unreleased branches,
such as `master` and `7.x`.

Also updates how the autoscaling and EQL docs are included.
Currently, these feature-flag docs would display on any unreleased
branches that contain the changes, such as 7.7.
2020-02-11 11:24:13 -05:00
Igor Motov 667e1a5225
Add Boxplot Aggregation (#52174)
Adds a `boxplot` aggregation that calculates min, max, medium and the first
and the third quartiles of the given data set.

Closes #33112
2020-02-11 09:38:17 -05:00
David Turner 00b9098250 Ignore timeouts with single-node discovery (#52159)
Today we use `cluster.join.timeout` to prevent nodes from waiting indefinitely
if joining a faulty master that is too slow to respond, and
`cluster.publish.timeout` to allow a faulty master to detect that it is unable
to publish its cluster state updates in a timely fashion. If these timeouts
occur then the node restarts the discovery process in an attempt to find a
healthier master.

In the special case of `discovery.type: single-node` there is no point in
looking for another healthier master since the single node in the cluster is
all we've got. This commit suppresses these timeouts and instead lets the node
wait for joins and publications to succeed no matter how long this might take.
2020-02-11 14:15:01 +00:00
David Roberts 4c88996cd7 [DOCS] Correct important note for xpack.transform.enabled (#52194)
Because transforms get assigned to an arbitrary data node it
is important that the transforms plugin is enabled on every
data node.
2020-02-11 13:02:10 +00:00
Yang Wang 16ba59e9d1
Expose more authentication info to ingest pipeline (#51305) (#52119)
The changes add more granularity for identiying the data ingestion user.
The ingest pipeline can now be configure to record authentication realm and
type. It can also record API key name and ID when one is in use. 
This improves traceability when data are being ingested from multiple agents
and will become more relevant with the incoming support of required
pipelines (#46847)

Resolves: #49106
2020-02-11 23:05:01 +11:00
Andrei Stefan 2f1631d9d0
Telemetry data initial implementation (#51715) (#52175)
(cherry picked from commit f1d1cceacaacf226fcd2459f34689843b822fe4b)
2020-02-11 09:15:47 +02:00
Lisa Cawley c4525f8cca
[DOCS] Adds ml-cpp PRs to release notes (#52158)
Co-Authored-By: David Roberts <dave.roberts@elastic.co>
2020-02-10 18:06:01 -08:00
Lee Hinman 37a2e9bac6
[7.x] Allow forcemerge in the hot phase for ILM policies (#520… (#52083)
* Allow forcemerge in the hot phase for ILM policies

This commit changes the `forcemerge` action to also be allowed in the `hot` phase for policies. The
forcemerge will occur after a rollover, and allows users to take advantage of higher disk speeds for
performing the force merge (on a separate node type, for example).

On caveat with this is that a `forcemerge` in the `hot` phase *MUST* be accompanied by a `rollover`
action. ILM validates policies to ensure this is the case.

Resolves #43165

* Use anyMatch instead of findAny in validation

* Make randomTimeseriesLifecyclePolicy single-pass
2020-02-10 08:54:49 -07:00
David Roberts 1cefafdd14 [ML] Add new categorization stats to model_size_stats (#52009)
This change adds support for the following new model_size_stats
fields:

- categorized_doc_count
- total_category_count
- frequent_category_count
- rare_category_count
- dead_category_count
- categorization_status

Backport of #51879
2020-02-10 09:10:50 +00:00
Jason Tedor c4c0db6f21
Introduce jvm.options.d for customizing JVM options (#51882)
This commit introduces the ability to override JVM options by adding
custom JVM options files to a jvm.options.d directory. This simplifies
administration of Elasticsearch by not requiring administrators to keep
the root jvm.options file in sync with changes that we make to the root
jvm.options file. Instead, they are not expected to modify this file but
instead supply their own in jvm.options.d. In Docker installations, this
means they can bind mount this directory in. In future versions of
Elasticsearch, we can consider removing the root jvm.options file
(instead, providing all options there as system JVM options).
2020-02-08 18:50:14 -05:00
Zachary Tong c8f0fe135d Update release notes for BC5 2020-02-07 16:41:51 -05:00
debadair 2588022b81 [DOCS] Fixed typo. (#52071) 2020-02-07 11:04:56 -08:00
James Rodewig db22fb6e1c Revert "[DOCS] Include docs on permanently unreleased branches only (#51743)"
This reverts commit 9d09796815.
2020-02-07 12:14:44 -05:00
David Kyle 8f10a7c6ca [ML] Make Ensemble feature names optional (#51996)
The featureNames field is requisite in individual models but is not required by the Ensemble.
2020-02-07 10:08:37 +00:00
Armin Braun 91e938ead8
Add Trace Logging of REST Requests (#51684) (#52015)
Being able to trace log all REST requests to a node would make debugging
a number of issues a lot easier.
2020-02-07 09:03:20 +01:00
Jason Tedor 25daf5f1e1
Add autoscaling API skelton (#51564)
The main purpose of this commit is to add a single autoscaling REST
endpoint skeleton, for the purpose of starting to build out the build
and testing infrastructure that will surround it. For example, rather
than commiting a fully-functioning autoscaling API, we introduce here
the skeleton so that we can start wiring up the build and testing
infrastructure, establish security roles/permissions, an so on. This
way, in a forthcoming PR that introduces actual functionality, that PR
will be smaller and have less distractions around that sort of
infrastructure.
2020-02-06 21:55:01 -05:00
James Rodewig 9d09796815 [DOCS] Include docs on permanently unreleased branches only (#51743)
Adds the ability to display docs on permanently unreleased branches,
such as `master` and `7.x`.

Also updates how the autoscaling and EQL docs are included.
Currently, these feature-flag docs would display on any unreleased
branches that contain the changes, such as 7.7.
2020-02-06 14:47:06 -05:00
Rory Hunter cdb9862495
Clarify use of ES_JAVA_OPTS and Docker (#51984)
Backport of #51867.

Tweak the documentation around configuring the heap size when using
Docker, to state that:

- using `ES_JAVA_OPTS` is the preferred method
- Any `ES_JAVA_OPTS` overrides the defaults in `jvm.options`
- It's possible to bind-mount a custom `jvm.options`
2020-02-06 10:00:14 +00:00
Lisa Cawley 4c2dcf2bde [DOCS] Adds curl explanation to getting started content (#51963) 2020-02-05 19:02:21 -08:00
Przemko Robakowski 6332de40b4
Add empty_value parameter to CSV processor (#51567) (#51966)
* Add empty_value parameter to CSV processor

This change adds `empty_value` parameter to the CSV processor.
This value is used to fill empty fields. Fields will be skipped
if this parameter is ommited. This behavior is the same for both
quoted and unquoted fields.

* docs updated

* Fix compilation problem

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-02-05 23:35:52 +01:00
Lisa Cawley ea1d9e0803 [DOCS] Defines data frame transform stats API objects (#44197) 2020-02-05 12:30:55 -08:00
Lisa Cawley b93ebc29c1 [DOCS] Augments update license API (#51903) 2020-02-05 11:08:11 -08:00
Lee Hinman f8c8a10f05 Add documentation about ILM forcemerge with best_compression (#51893)
This adds the option to the parameter list and a warning about the index being unavailable during
the close and open operations.

Relates to #49974
2020-02-05 09:37:41 -07:00
James Rodewig b70cbc97aa [DOCS] Add EQL syntax page (#51821)
Adds documentation for basic EQL syntax.

Joins, sequences, and other syntax to be added as its supported
in future development.

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-02-05 08:14:07 -05:00
David Kyle 289d4f4f4d [ML] Remove stray field from inference docs (#51870)
model_info_field is not a valid option
2020-02-05 10:50:51 +00:00
Adrien Grand ad9d2f1922
Move analysis/mappings stats to cluster-stats. (#51875)
Closes #51138
2020-02-05 11:02:25 +01:00
debadair c0156cbb5d
Backporting updates to ILM org, overview, & GS (#51898)
* [DOCS] Align with ILM API docs (#48705)

* [DOCS] Reconciled with Snapshot/Restore reorg

* [DOCS] Split off ILM overview to a separate topic. (#51287)

* [DOCS} Split off overview to a separate topic.

* [DOCS] Incorporated feedback from @jrodewig.

* [DOCS] Edit ILM GS tutorial (#51513)

* [DOCS] Edit ILM GS tutorial

* [DOCS] Incorporated review feedback from @andreidan.

* [DOCS] Removed test link & fixed anchor & title.

* Update docs/reference/ilm/getting-started-ilm.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Fixed glossary merge error.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-02-04 16:45:18 -08:00
Florian Kelbert 43a7aadd46 [DOCS] Remove unneeded comma from CSV processor example (#51859) 2020-02-04 09:26:20 -05:00
baifan 60a53f2897 [DOCS] Fix `disk.used_percent` typo in `_cat/nodes` docs (#51854)
Corrects an example for the `disk.used_percent` parameter in `_cat/nodes` API.
2020-02-04 09:15:56 -05:00
Grzegorz Banasiak 87b126bbfc [DOCS] Fix index_prefixes link in 'faster prefix queries' docs (#51833)
Fixes a link in 'faster prefix queries' which incorrectly redirects to index_phrases mapping parameter description instead of index_prefixes.
2020-02-04 08:40:18 -05:00
William Brafford ba2810f23d
Use standard format for reload settings API (#51560) (#51828)
* Use standard format for reload settings API

The reload-secure-settings API page was not reorganized for the standard
API format, so this commit is reorganizing the page and adding some
links to the page in related documentation.

* Fix broken links

* Reorder examples to correctly check API response

* Note that only certain settings are reloadable

* [DOCS] Edits layout

* [DOCS] Removes unnecessary callouts

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-03 18:07:26 -05:00
Dan Hermann 4083eae0b7
[7.x] Secure password for monitoring HTTP exporter (#51775)
Adds a secure and reloadable SECURE_AUTH_PASSWORD setting to allow keystore entries in the form "xpack.monitoring.exporters.*.auth.secure_password" to securely supply passwords for monitoring HTTP exporters. Also deprecates the insecure `AUTH_PASSWORD` setting.
2020-02-03 07:42:30 -06:00
James Rodewig 1545c2ab26 [DOCS] Document node stats response meta (#51263)
Documents several metadata-related parameters returned by the
`GET _nodes/stats` API.
2020-02-03 08:33:57 -05:00
Darren LaCasse 480e9238a4 [DOCS] Remove extra word (#51757) 2020-01-31 10:30:06 -08:00
Zachary Tong 3147453600 Update Release Notes for BC4 2020-01-31 11:41:44 -05:00
Christoph Büscher 86f3b47299
Make `date_range` query rounding consistent with `date` (#50237) (#51741)
Currently the rounding used in range queries can behave differently for `date`
and `date_range` as explained in #50009. The behaviour on `date` fields is
the one we document in https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#range-query-date-math-rounding.
This change adapts the rounding behaviour for RangeType.DATE so it uses the
same logic as the `date` for the `date_range` type.

Backport of #50237
2020-01-31 15:35:05 +01:00
István Zoltán Szabó dfc9f2330c [DOCS] Adds PUT inference API docs (#51231)
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-31 13:13:34 +01:00
István Zoltán Szabó 9600ab4f57 [DOCS] Adds recommendation on dedicated master-eligible nodes (#51674)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-01-31 12:56:58 +01:00
Sven Schliesing dc99acee66 [Docs] Fix typo in node-tool.asciidoc (#51667) 2020-01-31 10:36:21 +01:00
Yang Wang 77b00fc0c0
Add warnings for invalid realm order config (#51195) (#51515)
The changes are to help users prepare for migration to next major
release (v8.0.0) regarding to the break change of realm order config.

Warnings are added for when:
* A realm does not have an order config
* Multiple realms have the same order config

The warning messages are added to both deprecation API and loggings.
The main reasons for doing this are: 1) there is currently no automatic relay
between the two; 2) deprecation API is under basic and we need logging
for OSS.
2020-01-31 12:32:37 +11:00
Lisa Cawley 1a40ebfa67 [DOCS] Adds missing testenv attribute (#51719) 2020-01-30 16:15:17 -08:00