Commit Graph

8308 Commits

Author SHA1 Message Date
James Rodewig 114894dd76 [DOCS] Add redirect for changed anchor ID (#54533)
The anchor ID for the snapshot repository plugins section in the docs
was recently changes from `_repository_plugins` to
`snapshots-repository-plugins`.

This adds a corresponding redirect so no links are broken.
2020-03-31 16:42:16 -04:00
James Rodewig 7401191019
[DOCS] Include 7.7.0 release notes (#54529)
Includes the 7.7.0 release notes so they render in the HTML docs.

Also removes a few legacy `coming[7.6.0]` tags.
2020-03-31 16:23:49 -04:00
Lisa Cawley 922ec8e961
[DOCS] Collapses nested objects in data frame analytics APIs (#54472) (#54526) 2020-03-31 12:51:04 -07:00
Glen Smith 0d4a001ef2 [DOCS] Clarify cluster health status during rolling upgrade (#40757)
Remove mention of the `yellow` and `red` starting
health status from the rolling upgrade docs.

Instead, we should emphasize that users wait 
for the node to recover with a health status of
`green` rather than the starting status.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-31 14:13:05 -04:00
Dimitris Athanasiou 0b25e3b66c
[7.x][ML] Fix DF analytics explain API request in docs (#54510) (#54514)
The explain API expects a data frame analytics config
as its request.

Backport of #54410
2020-03-31 18:56:52 +03:00
István Zoltán Szabó a5497cd9e0 [DOCS] Adds HTTP response count example to Painless examples (#54412) 2020-03-31 15:12:58 +02:00
István Zoltán Szabó eeb23e9e73 [DOCS] Adds description of analysis_stats object and its properties to GET DFA stats API docs (#53881)
Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-31 13:30:06 +02:00
James Rodewig ce7539ce6c
[DOCS] Fix broken deprecated macros (#54466)
Updates formatting for `deprecated` macros in the translog and
synced flush docs.

Previously, these macros were rendering literally.
2020-03-30 17:16:27 -04:00
Lisa Cawley 0fa1060ca4
[DOCS] Collapses content in machine learning APIs (#54234) (#54453) 2020-03-30 11:06:33 -07:00
James Rodewig ed1edb4964 [DOCS] Add missing word to keyword marker token filter docs 2020-03-30 10:52:14 -04:00
Christoph Büscher 67b9b68c66 [Docs] Add HLRC Async Search API documentation (#54353)
Adds documentation and a corresponding test case containing typical API usage
for the Async Search API to the High Level Rest Client.
2020-03-30 15:37:22 +02:00
James Rodewig 21f362a2a8
[DOCS] Add a lowercase email example to keyword tokenizer docs (#53257) 2020-03-30 09:06:04 -04:00
Benjamin Trent 374e76d7cd
[Transform] fixing naming in HLRC and _cat to match API content (#54300) (#54408)
Fixing the naming of the HLRC values to match the ToXContent field names (i.e. the field names returned from an API call).

Also fixes the names in the _cat API as well.

closes #53946
2020-03-30 08:57:02 -04:00
István Zoltán Szabó 00eaa0ebe5 [DOCS] Changes scripted metric to filter aggs in transforms example (#54167) 2020-03-30 09:51:07 +02:00
Jason Tedor f0033783db
Deprecate node local storage setting (#54374)
This setting is not documented and has dubious value since it means
there can be nodes in the cluster (non-data and non-master nodes) that
do not have persistent node IDs. This does not have any use cases so
this commit removes the setting.
2020-03-28 14:36:41 -04:00
Stuart Tettemer 30c56087fd
Docs: Use splitOnToken instead of custom function (#48408) (#54364)
Painless ingest example uses a custom split function but
new splitOnToken function was added in 7.2

Backport of: 0c52a92
2020-03-27 15:04:27 -06:00
Gil Raphaelli 2984a54b7f [DOCS] Fix typos in top metrics agg docs (#54299) 2020-03-27 10:49:21 -04:00
AndyHunt66 2dd8946539 [DOCS] Remove redundant sentence in ingest processor docs (#54329) 2020-03-27 08:25:09 -04:00
Christoph Büscher f7ea794312 [Test] Don't expect specific scores in docs tests (#54297)
The failing suggester documentation test was expecting specific scores in the
test response, which is fragile implementation details that e.g. can change with
different lucene versions and generally shouldn't be done in documentation test.
Instead we usually replace the float values in the output response by the ones
in the actual response.

Closes #54257
2020-03-27 10:27:47 +01:00
Lisa Cawley 27cd5b343c
[DOCS] Augments cat transforms API (#53776) (#54232) 2020-03-26 07:56:46 -07:00
Jason Tedor d8f745736b
Clarify the remove keystore command can handle many (#54244)
The remove keystore command can handle multiple settings. In a few
places, we were not consistent about mentioning this. This commit
addreses this, in the CLI help, and the docs.
2020-03-26 08:49:43 -04:00
Luca Cavanna ff269160af Async search: rename REST parameters (#54198)
This commit renames wait_for_completion to wait_for_completion_timeout in submit async search and get async search.
Also it renames clean_on_completion to keep_on_completion and turns around its behaviour.

Closes #54069
2020-03-26 09:40:50 +01:00
István Zoltán Szabó 487b273286 [DOCS] Adds feature importance mapping subsection to inference processor docs (#54190) 2020-03-26 09:26:50 +01:00
Jason Tedor 6af89e62d1
Allow keystore add-file to handle multiple settings (#54240)
Today the keystore add-file command can only handle adding a single
setting/file pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add-file keystore command to accept adding multiple
settings in a single invocation.
2020-03-26 00:07:05 -04:00
Jason Tedor 4fbc0e9ab8
Complete keystore CLI options documentation (#54242)
The documentation was missing the long option for the force option, and
the short option for the stdin option. This commit addresses this by
adding these to the documentation.
2020-03-25 23:53:34 -04:00
Jason Tedor fe8257d981
Allow keystore add to handle multiple settings (#54229)
Today the keystore add command can only handle adding a single
setting/value pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add keystore command to accept adding multiple
settings in a single invocation.
2020-03-25 22:58:20 -04:00
James Rodewig 30a32040d3
[DOCS] EQL: Document `substring` function (#53867)
Adds documentation for the EQL `substring` function.

Supporting changes:

* Creates a new "EQL function reference" page
* Updates the title of the "EQL syntax reference" page for consistency
* Adds a brief "Functions" section to the EQL syntax docs
* Updates EQL limitations docs to state that only array functions are
  unsupported
2020-03-25 12:23:59 -04:00
Nik Everett 16e4bd50e2 Add breaking change note for #53669 2020-03-25 09:31:14 -04:00
James Rodewig 74051d68a8
[DOCS] Reformat `keyword_marker` token filter (#54076)
Makes the following changes to the `keyword_marker` token filter docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer and filter example
2020-03-25 09:26:06 -04:00
James Rodewig 2fdf6b2f96
[DOCS] Document missing data types for node stats API's response parameters (#53475)
Documents missing data types for several response parameters returned
by the node stats API.

Also adds several missing human-readable parameters returned by the API.
2020-03-25 08:42:58 -04:00
Jason Tedor 381d7586e4
Introduce formal role for remote cluster client (#54138)
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.

Relates #53924
2020-03-24 21:59:43 -04:00
David Roberts 7667004b20
[ML] Add a model memory estimation endpoint for anomaly detection (#54129)
A new endpoint for estimating anomaly detection job
model memory requirements:

POST _ml/anomaly_detectors/estimate_model_memory

Backport of #53507
2020-03-24 22:55:11 +00:00
Jim Ferenczi 55f2e8bff0 [DOCS] Add 7.6.2 release notes (#53720)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-24 22:42:25 +01:00
markharwood 6a60f85bba
Wildcard field - add normalizer support (#53851) (#54109)
Backport support for normalisation to wildcard field

Closes #53603
2020-03-24 17:37:47 +00:00
Lisa Cawley 88b1b2f36f [DOCS] Adds transform security privileges (#53908) 2020-03-24 09:35:45 -07:00
Tim Brooks caefa78513
Align remote info api with new settings (#54102)
Currently the remote info api has added a number of possible fields
(proxy, num_socket_connections, etc) that are available in proxy mode.
These fields are not aligned with what the settings are named. This
commit modifies this API to align with the settings.
2020-03-24 10:27:24 -06:00
Luca Cavanna 6b457abbd3 Async search: prevent users from overriding pre_filter_shard_size (#54088)
Submit async search forces pre_filter_shard_size for the underlying search that it creates.
With this commit we also prevent users from overriding such default as part of request validation.
2020-03-24 17:06:04 +01:00
Ahmet Arslan 52062565a9 [DOCS] Correct DFI docs regarding stop word removal (#53836)
The documentation of DFI should recommend *not* to [remove stop words][1], since DFI is good at scoring queries that contain common terms: `the wall`, `the sun`, `the who`, etc.

[1]:https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/search/similarities/DFISimilarity.html
2020-03-24 10:48:42 -04:00
Karen Metts 9da589c5fd [DOCS] Replace outdated Logstash monitoring link (#54032)
Replaces a link to Logstash OSS-only content with a link to the general Logstash monitoring topic.
2020-03-24 10:03:31 -04:00
Tanguy Leroux dea8a31480 Wait for Active license before running CCR API tests (#53966)
DocsClientYamlTestSuiteIT sometimes fails for CCR 
related tests because tests are started before the license 
is fully applied and active within the cluster. The first 
tests to be executed then fails with the error noticed 
in #53430. This can be easily reproduced locally by 
only running CCR docs tests.

This commit adds some @Before logic in 
DocsClientYamlTestSuiteIT so that it waits for the 
license to be active before running CCR tests.

Closes #53430
2020-03-24 14:29:45 +01:00
David Roberts 1421471556
[ML] Introduce a "starting" datafeed state for lazy jobs (#54065)
It is possible for ML jobs to open lazily if the "allow_lazy_open"
option in the job config is set to true.  Such jobs wait in the
"opening" state until a node has sufficient capacity to run them.

This commit fixes the bug that prevented datafeeds for jobs lazily
waiting assignment from being started.  The state of such datafeeds
is "starting", and they can be stopped by the stop datafeed API
while in this state with or without force.

Backport of #53918
2020-03-24 13:00:04 +00:00
Hendrik Muhs 7dcacf531f
[7.x][Transform][Rollup] add processing stats to record the ti… (#54027)
add 2 additional stats: processing time and processing total which capture the
time spent for processing results and how often it ran. The 2 new stats
correspond to the existing indexing and search stats. Together with indexing
and search this now allows the user to see the full picture, all 3 stages.
2020-03-24 09:22:02 +01:00
Jason Tedor e3ca124537
Introduce autoscaling decisions (#53934)
This is the first in a series of commits that will introduce the
autoscaling deciders framework. This commit introduces the basic
framework for representing autoscaling decisions.
2020-03-23 23:08:06 -04:00
Jim Ferenczi 9e3f7f4575
Add heuristics to compute pre_filter_shard_size when unspecified (#53873) (#54007)
This commit changes the pre_filter_shard_size default from 128 to unspecified.
This allows to apply heuristics based on the request and the target indices when deciding
whether the can match phase should run or not. When unspecified, this pr runs the can match phase
automatically if one of these conditions is met:
  * The request targets more than 128 shards.
  * The request contains read-only indices.
  * The primary sort of the query targets an indexed field.
Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value.

Closes #39835
2020-03-24 02:05:15 +01:00
Julie Tibshirani df7cfb3a5b
Remove the top-level 'mapping type' section. (#54035)
It seemed confusing for users that our top-level mapping page still had a
prominent section named 'Mapping Type'. This PR reworks the docs to remove this
reference and adds a note about types removal (similar to the note we added to
other APIs like put mapping).
2020-03-23 15:34:23 -07:00
James Rodewig 43199a8c82 [DOCS] Remove double space in WDG docs 2020-03-23 17:18:04 -04:00
James Rodewig 553d8a9ca9 [DOCS] Fix "letter case" typo
Changes "lettercase" to "letter case" in the `uppercase` token filter
docs.
2020-03-23 17:11:59 -04:00
Namgyu Kim bc2289c258 Add nori_number token filter in analysis-nori (#53583)
This change adds the `nori_number` token filter.
It also adds a `discard_punctuation` option in nori_tokenizer that should be used in conjunction with the new filter.
2020-03-23 19:53:34 +01:00
Paweł Krześniak c0534f4157 [DOCS] link fix (#53973)
Fix bad link in top_metrics.
2020-03-23 14:20:54 -04:00
Luca Cavanna 932a7e3112
Backport of async search changes (#53976)
* Get Async Search: omit _clusters section when empty (#53907)

The _clusters section is omitted by the search API whenever no remote clusters are searched. Async search should do the same, but Get Async Search returns a deserialized response, hence a weird `_clusters` section with all values set to `0` gets returned instead. In fact the recreated Clusters object is not the same object as the EMPTY constant, yet it has the same content.

This commit addresses this by changing the comparison in the `toXContent` method to not print out the section if the number of total clusters is `0`.

* Async search: remove version from response (#53960)

The goal of the version field was to quickly show when you can expect to find something new in the search response, compared to when nothing has changed. This can also be done by looking at the `_shards` section and `num_reduce_phases` returned with the search response. In fact when there has been one or more additional reduction of the results, you can expect new results in the search response. Otherwise, the `_shards` section could notify of additional failures of shards that have completed the query, but that is not a guarantee that their results will be exposed (only when the following partial reduction is performed their results will be available).

That said this commit clarifies this in the docs and removes the version field from the async search response

* Async Search: replicas to auto expand from 0 to 1 (#53964)

This way single node clusters that are green don't go yellow once async search is used, while
all the others still have one replica.

* [DOCS] address timing issue in async search docs tests (#53910)

The docs snippets for submit async search have proven difficult to test as it is not possible to guarantee that you get a response that is not final, even when providing `wait_for_completion=0`. In the docs we want to show though a proper long-running query, and its first response should be partial rather than final.

With this commit we adapt the docs snippets to show a partial response, and replace under the hood all that's needed to make the snippets tests succeed when we get a final response. Also, increased the timeout so we always get a final response.

Closes #53887
Closes #53891
2020-03-23 19:13:31 +01:00
Lisa Cawley c4260ba3c7 [DOCS] Fixes formatting in transform overview (#53900) 2020-03-23 10:21:54 -07:00
C.J. Jameson 5ac8b795ba [DOCS] Clarify routing enforcement in docs (#53945)
Removes a mention of the `_doc` mapping type that's
no longer applicable now that mapping types are
removed/deprecated.
2020-03-23 12:57:07 -04:00
Lisa Cawley 5f45780577 [DOCS] Adds data nanos transform limitation (#53826) 2020-03-23 09:49:21 -07:00
Lisa Cawley 8b3ef4ac21 [DOCS] Add generated_dest_index to preview transform API (#53905) 2020-03-23 09:43:44 -07:00
Marios Trivyzas af03200ad6
SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) (#47720) (#53972)
The function is extended to operate on intervals according to the PostgreSQL: https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes : #46632
(cherry picked from commit 2dc79505825fa75e0711dcfa8e9c69e8028fc979)

Co-authored-by: musteaf <gs_mustea@hotmail.com>
2020-03-23 15:05:16 +01:00
Dan Hermann 0528e2b06b
Document the deprecation of the 'auth.password' setting (#53822) 2020-03-23 08:58:25 -05:00
István Zoltán Szabó 534e734ad1 [DOCS] Adds painless transform examples (#53274)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-23 10:39:47 +01:00
Przemyslaw Gomulka 015ad019d5
[docs] Known issue about joda patterns on 7.6 (#53957) 2020-03-23 10:28:55 +01:00
Przemyslaw Gomulka 412e163cf6
[Doc] migration guide joda (#51986)
The joda to java.time migration requires users to upgrade their mappings. We allow them to still use 6.x created indices with joda patterns in 7 but ask them to upgrade their patterns in 7.x.
This migration guide is to help them understand how they could be affected and what needs to be changed in their mappings.
closes #51614
closes #51236
2020-03-23 08:29:01 +01:00
Mark Vieira 0cfe6d90cc
Mute async-search test 2020-03-20 11:35:24 -07:00
Luca Belluccini 3937854444 [DOCS] Clarify upgrade paths (#53417)
Unsupported upgrade paths:
- `6.8 to 7.0`.
Supported, but requires a full cluster restart:
- `6.0–6.7 directly to 7.x`
2020-03-20 18:05:07 +00:00
Lisa Cawley c2af015ee7 [DOCS] Updates list of transform aggs (#53820) 2020-03-20 10:28:23 -07:00
Alan Woodward a70016a6d6 TermsLookup fields should be marked as Required in docs (#53784)
The terms-lookup section of our terms query docs currently state that the
index, id and path fields are optional. They should be marked instead
as required.
2020-03-20 15:39:41 +00:00
Luca Cavanna d486bdefdd [DOCS] correct async search note
The sort optimization kicks in whenever results are sorted by field.
2020-03-20 15:58:19 +01:00
Luca Cavanna 03fca61fcb [DOCS] add docs for async search (#53675)
Relates to #49091

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-03-20 14:46:38 +01:00
James Rodewig 6c57681446 [DOCS] Add redirects for missing data stream API docs (#53866)
The following API spec files contain a link to a not-yet-created
docs page for the data stream APIs:

* [indices.create_data_stream.json][0]
* [indices.delete_data_stream.jsonn][1]
* [indices.get_data_streams.json][2]

The Elaticsearch-JS client uses these spec files to create their docs.
This created a broken link in the Elaticsearch-JS, which has broken
the docs build.

This PR adds a temporary redirect for the docs page. This redirect
should be removed when the actual API docs are added.

[0]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.create_data_stream.json
[1]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_stream.json
[2]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_streams.json

Relates to #53558.

CC @martijnvg
2020-03-20 09:41:47 -04:00
lgypro be3090138e [Docs] Fix typo in _analyze api docs (#53837) 2020-03-20 12:02:40 +01:00
István Zoltán Szabó 53f7e31462 [DOCS] Fixes typo in start datafeed API docs. (#53811) 2020-03-19 17:56:40 +01:00
István Zoltán Szabó 29583288b3 [DOCS] Adds performance considerations section to transforms overview (#53791)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-03-19 17:51:00 +01:00
Lisa Cawley 68f7036a4c [DOCS] Adds example links to transform tutorial (#53640) 2020-03-19 09:41:57 -07:00
István Zoltán Szabó 00203c35fe [DOCS] Changes seconds to milliseconds since the Epoch in AD docs. (#53797) 2020-03-19 15:42:58 +01:00
Dominic Page b0884baf46
Geo shape query vs geo point backport (#53774)
Backport to 7x

Enable geo_shape query to work on geo_point fields for shapes: circle, polygon, multipolygon, rectangle see: #48928
Co-Authored-By:  @iverase
2020-03-19 13:00:36 +01:00
James Rodewig 8f4a3eb07f [DOCS] Add token graph concept docs (#53339)
Adds conceptual docs for token graphs.
These docs cover:

* How a token graph is constructed from a token stream
* How synonyms and multi-position tokens impact token graphs
* How token graphs are used during search
* Why some token filters produce invalid token graphs

Also makes the following supporting changes:
* Adds anchors to the 'Anatomy of an Analyzer' docs for cross-linking
* Adds several SVGs for token graph diagrams
2020-03-19 07:43:18 -04:00
Leaf-Lin b649b0d273
Update jvm options doc
There's a typo with extra `
2020-03-19 12:00:01 +11:00
Benjamin Trent 415d73c27d
[Transform] renamed _cat/transform to _cat/transforms (#53743) (#53771)
renaming _cat/transform to  _cat/transforms for uniformity with the other _cat apis.
2020-03-18 19:54:03 -04:00
Lisa Cawley 268e512f0b [DOCS] Add transform nodes (#53698) 2020-03-18 15:26:37 -07:00
James Rodewig 0e2e06bd7e [DOCS] Remove incorrect parms from put index template API docs (#53750)
Removes the `flat_settings` and `timeout` query parameters from the JSON
spec and asciidoc docs for the put index template API.

These parameters are not supported by the API.
2020-03-18 14:36:27 -04:00
markharwood 598d4c1bf9
Formatting fix
Bullet points were not rendered correctly
2020-03-18 15:36:52 +00:00
Lisa Cawley c6e37c7662 [DOCS] Adds stub for cat transform API (#53737) 2020-03-18 08:31:10 -07:00
James Rodewig 0732475cb9 [DOCS] Streamline `analyzer` mapping parm def (#51874)
Simplifies the `analyzer` mapping parameter definition to remove
duplicated analysis content and examples.
2020-03-18 09:43:59 -04:00
James Rodewig e4b7af11ab [DOCS] Remove `light_bengali` stemmer (#53697)
Only the `bengali` stemmer is available in Lucene and surfaced through
Elasticsearch. This removes the incorrect `light_bengali` link in our
docs.
2020-03-18 08:34:27 -04:00
Dan Hermann 94ac979c66
Support array for all string ingest processors (#53694) 2020-03-18 07:07:49 -05:00
Ralph Ursprung 474dfbf9c7 [Docs] Fix highlighting in match-query example (#52426) 2020-03-18 11:53:14 +01:00
Tianlun Li e7ae9ae596 Deprecate delaying state recovery for master nodes (#53646)
It is useful to be able to delay state recovery until enough data nodes have
joined the cluster, since this gives the shard allocator a decent opportunity
to re-use as much existing data as possible. However we also have the option to
delay state recovery until a certain number of master-eligible nodes have
joined, and this is unnecessary: we require a majority of master-eligible nodes
for state recovery, and there is no advantage in waiting for more.

This commit deprecates the unnecessary settings in preparation for their
removal.

Relates #51806
2020-03-18 10:04:22 +00:00
Yash Joshi 85cc7a06c4 [Docs] Fix typo in range query (#53656) 2020-03-18 10:18:08 +01:00
Karen Metts 3c5437894e
Remove link to old settings 7.x (#53639) 2020-03-17 14:38:50 -04:00
David Turner 347b7594b7 Fix deprecation in history retention docs (#53655)
This commit adjusts a `deprecation[...]` message in the docs since such
messages must be on a single line. It also moves this message to the start of
the description of the deprecated setting as is the case with other such
messages.
2020-03-17 14:06:37 +00:00
Luca Cavanna c3d2417448
Cumulative backport of async search changes (#53635)
* Submit async search to work only with POST (#53368)

Currently the submit async search API can be called using both GET and POST at REST, but given that it submits a call and creates internal state, POST should be the only allowed method.

* Refine SearchProgressListener internal API (#53373)

The following cumulative improvements have been made:
- rename `onReduce` and `notifyReduce` to `onFinalReduce` and `notifyFinalReduce`
- add unit test for `SearchShard`
- on* methods in `SearchProgressListener` shouldn't need to be public as they should never be called directly, they only need to be overridden hence they can be made protected. They are actually called directly from a test which required some adapting, like making `AsyncSearchTask.Listener` class package private instead of private
- Instead of overriding `getProgressListener` in `AsyncSearchTask`, as it feels weird to override a getter method, added a specific method that allows to retrieve the Listener directly without needing to cast it. Made the getter and setter for the listener final in the base class.
- rename `SearchProgressListener#searchShards` methods to `buildSearchShards` and make it static given that it accesses no instance members
- make `SearchShard` and `SearchShardTask` classes final

* Move async search yaml tests to x-pack yaml test folder (#53537)

The yaml tests for async search currently sit in its qa folder. There is no reason though for them to live in a separate folder as they don't require particular setup. This commit moves them to the main folder together with the other x-pack yaml tests so that they will be run by the client test runners too.

* [DOCS] Add temporary redirect for async-search (#53454)

The following API spec files contain a link to a not-yet-created
async search docs page:

* [async_search.delete.json][0]
* [async_search.get.json][1]
* [async_search.submit.json][2]

The Elaticsearch-js client uses these spec files to create their docs.
This created a broken link in the Elaticsearch-js docs, which has broken
the docs build.

This PR adds a temporary redirect for the docs page. This redirect
should be removed when the actual API docs are added.

[0]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json
[1]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json
[2]: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-17 00:08:17 +01:00
Ryan Earle b4e203a735 [DOCS] Remove `force` as valid value for `version_type` (#53428)
The `force` value for the `version_type` parameter was deprecated in 6.8.

This removes the value from the parameter definition.
2020-03-16 16:50:58 -04:00
Tim Brooks 531cd5aaa4
Add documentation for remote cluster proxy mode (#52779)
This is related to #49067.
2020-03-16 14:18:29 -06:00
Lisa Cawley 278e3fce50 [DOCS] Add anchors for scripted metric aggregations (#53618) 2020-03-16 12:20:41 -07:00
Abhilash Bolla 508e5239b4 [DOCS] Remove unneeded anchor from ILM snippet (#53432) 2020-03-16 15:19:50 -04:00
Nik Everett f7482f794a
Improve top_metrics docs (#53521) (#53619)
* Removes experimental.
* Replaces `"v"` (for value) with `"m"` (for metric).
* Move the note about tiebreaking into the list of limitations of the
  sort.
* Explain how you ask for `metrics`.
* Clean up some wording.
* Link to the docs from `top_metrics`.

Closes #51813
2020-03-16 13:47:43 -04:00
Andrei Stefan 91ca9c5c33
QL: constant_keyword support (#53241) (#53602)
(cherry picked from commit d6cd4ce7849ba215407c8c5fa815c9b373fb8480)
2020-03-16 18:06:31 +02:00
James Rodewig e1eebea846
[DOCS] Reformat `remove_duplicates` token filter (#53608)
Makes the following changes to the `remove_duplicates` token filter
docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Adds custom analyzer example
2020-03-16 11:37:06 -04:00
markharwood 2c74f3e22c
Backport of new wildcard field type (#53590)
* New wildcard field optimised for wildcard queries (#49993)

Indexes values using size 3 ngrams and also stores the full original as a binary doc value.
Wildcard queries operate by using a cheap approximation query on the ngram field followed up by a more expensive verification query using an automaton on the binary doc values.  Also supports aggregations and sorting.
2020-03-16 15:07:13 +00:00
Mayya Sharipova a906f8a0e4
Highlighters skip ignored keyword values (#53408) (#53604)
Keyword field values with length more than ignore_above are not
indexed. But highlighters still were retrieving these values
from _source and were trying to highlight them. This sometimes lead to
errors if a field length exceeded  max_analyzed_offset. But also this
is an overall wrong behaviour to attempt to highlight something that was
ignored during indexing.

This PR checks if a keyword value was ignored because of its length,
and if yes, skips highlighting it.

Backport: #53408
Closes #43800
2020-03-16 11:06:25 -04:00
Benjamin Trent 4e43ede735
[ML] renaming inference processor field field_mappings to new name field_map (#53433) (#53502)
This renames the `inference` processor configuration field `field_mappings` to `field_map`.

`field_mappings` is now deprecated.
2020-03-13 15:40:57 -04:00
Tom Veasey 690099553c
[7.x][ML] Adds the class_assignment_objective parameter to classification (#53552)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-13 17:35:51 +00:00
Lisa Cawley b1d589f276 [DOCS] Adds operations_behind to transform stats (#53518) 2020-03-13 09:34:49 -07:00
Jim Ferenczi e7c393a518 Fix broken link in client documentation (#51834)
This change fixes the link to the RankFeatureQueryBuilder javadocs
in the documentation.

Closes #51800
2020-03-13 14:38:11 +01:00
James Rodewig 1e31031b73 [DOCS] Clarify `max_shingle_size` parm def (#53480)
Rewrites the `search_as_you_type` field datatype's `max_shingle_size`
mapping parameter to improve clarity and better communicate trade-offs
regarding index size.

Relates to [elastic/kibana#55161][0].

Closes #51774.

[0]: https://github.com/elastic/kibana/pull/55161#discussion_r368107177
2020-03-13 04:08:44 -04:00
Nik Everett 9dcd64c110
Preserve metric types in top_metrics (backport of #53288) (#53440)
This changes the `top_metrics` aggregation to return metrics in their
original type. Since it only supports numerics, that means that dates,
longs, and doubles will come back as stored, with their appropriate
formatter applied.
2020-03-12 17:17:09 -04:00
Jim Ferenczi 97621e7f65 Removes old Lucene's experimental flag from analyzer documentations (#53217)
This change removes the Lucene's experimental flag from the documentations of the following
tokenizer/filters:
  * Simple Pattern Split Tokenizer
  * Simple Pattern tokenizer
  * Flatten Graph Token Filter
  * Word Delimiter Graph Token Filter

The flag is still present in Lucene codebase but we're fully supporting these tokenizers/filters
in ES for a long time now so the docs flag is misleading.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-12 21:18:19 +01:00
Jason Tedor 5b08ea84c9
Add deprecation check for listener thread pool (#53438)
This commit adds a deprecation check for the listener thread pool
settings as these will be removed in 8.0.0.
2020-03-12 14:32:41 -04:00
István Zoltán Szabó 969164cc47 [DOCS] Adds a warning about reindexing docs with the same ID to the PUT DFA docs. (#53490) 2020-03-12 18:01:43 +01:00
James Rodewig af987fb2d4
[DOCS] Reduce content reuse in enrich docs (#53460)
Restructures the 'Update an enrich policy' section to:

* Migrate the content to the section. It was previously stored in the
  Put Enrich Policy API docs.
* Remove the warning tag admonition from the section content.
* Replace a reused section earlier in the "Set up an enrich processor"
  page with a link.

No substantive changes were made to the content.
2020-03-12 05:57:23 -04:00
Benjamin Trent 89668c5ea0
[ML][Inference] adds new default_field_map field to trained models (#53294) (#53419)
Adds a new `default_field_map` field to trained model config objects.

This allows the model creator to supply field map if it knows that there should be some map for inference to work directly against the training data.

The use case internally is having analytics jobs supply a field mapping for multi-field fields. This allows us to use the model "out of the box" on data where we trained on `foo.keyword` but the `_source` only references `foo`.
2020-03-11 13:49:39 -04:00
Ignacio Vera 562a9eff33
remove sneaked placeholder from histogram docs (#53391) (#53405) 2020-03-11 14:04:22 +01:00
James Rodewig 933a9c6fca
[DOCS] Reformat `word_delimiter` token filter (#53387)
Makes the following changes to the `word_delimiter` token filter docs:

* Adds a warning admonition recommending the `word_delimiter_graph`
  filter instead. This warning includes a link to the deprecated Lucene
  `WordDelimiterFilter`.
* Updates the description
* Adds detailed analyze snippet
* Adds custom analyzer and custom filter snippets
* Reorganizes and updates parameter documentation
2020-03-11 09:03:57 -04:00
Dimitris Athanasiou 0fd0516d0d
[7.x][ML] Rename data frame analytics maximum_number_trees to max_trees (#53300) (#53390)
Deprecates `maximum_number_trees` parameter of classification and
regression and replaces it with `max_trees`.

Backport of #53300
2020-03-11 12:45:27 +02:00
James Rodewig a9dd7773d2 [DOCS] Use keyword tokenizer in word delimiter graph examples (#53384)
In a tip admonition, we recommend using the `keyword` tokenizer with the
`word_delimiter_graph` token filter. However, we only use the
`whitespace` tokenizer in the example snippets. This updates those
snippets to use the `keyword` tokenizer instead.

Also corrects several spacing issues for arrays in these docs.
2020-03-11 04:46:33 -04:00
James Rodewig 166b5a92f6 [DOCS] Correct anchor in word delimiter graph token filter docs 2020-03-10 10:32:42 -04:00
James Rodewig d503bf9a45 [DOCS] Make token graph diagrams consistent (#53331)
Updates the SVG for a token graph to make the layout consistent with
other graphs. This means moving the text directly above the edge lines.
Previously, the text was above the edge line.
2020-03-10 06:53:28 -04:00
James Rodewig 5e3df18d56 [DOCS] Adds Beats tip to EQL search docs (#53292)
Adds a tip admonition to the basic example in the EQL search docs.

This tip lets users know they can set up a Beat to automatically
index data in ES, rather than manually indexing using the bulk or index
APIs.
2020-03-10 05:16:18 -04:00
James Rodewig 641eb69520 [DOCS] Document `nodes` cluster stats (#52813)
Documents the `nodes` response parameters returned by the
`_cluster/stats` API.

Also adds collapsible attributes for the `indices` and `nodes`
sections.
2020-03-10 05:05:05 -04:00
Jason Tedor 1860c57147
Deprecate the listener thread pool (#53266)
The listener thread pool is being removed from use in the server
codebase. This commit deprecates configuring the listener thread pool.
2020-03-09 16:56:01 -04:00
Julie Tibshirani c33afea9fb Small corrections to stored_fields docs. (#53247)
* Fix a reference to the 'field' option.
* Remove claim about detecting script fields.
* Specify that object fields will just be ignored.
2020-03-09 10:59:17 -07:00
Dominik Riemer f78bd00f57 [DOCS] Update StreamPipes link in Integrations docs (#53087)
Updates the link to [StreamPipes][0] after transition to Apache Software Foundation.

[0]: https://streampipes.apache.org
2020-03-09 11:07:57 -04:00
Greg Back 64ea94ac8e [DOCS] Update ES Python client doc links (#53092)
Replaces `rtfd.org` links with `readthedocs.org`, which is now the official URL.
2020-03-09 10:34:38 -04:00
Romain Gonord 00657901ec [DOCS] Fix "Wait for Snapshot" link in ILM actions reference (#53269)
Corrects an anchor for the `Wait for Snapshot` action, which previously linked to the `Delete` action.
2020-03-09 08:40:36 -04:00
Anton Dollmaier 35c8226419 [DOCS] Fix parameter formatting for GeoHash grid agg docs (#53032)
Adds missing colon (`:`) to the parameter definition list.
2020-03-09 08:16:40 -04:00
James Rodewig 28cb4a167d
[DOCS] Reformat `word_delimiter_graph` token filter (#53170) (#53272)
Makes the following changes to the `word_delimiter_graph` token filter
docs:

* Updates the Lucene experimental admonition.
* Updates description
* Adds analyze snippet
* Adds custom analyzer and custom filter snippets
* Reorganizes and updates parameter list
* Expands and updates section re: differences between `word_delimiter`
  and `word_delimiter_graph`
2020-03-09 06:45:44 -04:00
István Zoltán Szabó aafc0409a9 [DOCS] Makes the description clearer on how to use aggregations in an anomaly detection job (#53103)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-09 09:49:59 +01:00
Lisa Cawley 341417613e
[7.x][DOCS] Adds common definitions for security settings (#51017) (#53242)
Co-Authored-By: Tim Vernum <tim@adjective.org>
2020-03-06 16:28:54 -08:00
Gordon Brown ff9b8bda63
Implement hidden aliases (#52547)
This commit introduces hidden aliases. These are similar to hidden
indices, in that they are not visible by default, unless explicitly
specified by name or by indicating that hidden indices/aliases are
desired.

The new alias property, `is_hidden` is implemented similarly to
`is_write_index`, except that it must be consistent across all indices
with a given alias - that is, all indices with a given alias must
specify the alias as either hidden, or all specify it as non-hidden,
either explicitly or by omitting the `is_hidden` property.
2020-03-06 16:02:38 -07:00
Adam Canady a88d0c7ca3 [Docs] Correct examples for * and + in regexp-syntax.asciidoc (#53210) 2020-03-06 17:17:32 +01:00
István Zoltán Szabó bf3dcd4229 [DOCS] Adds deleting flag to the GET job stats API docs (#53223) 2020-03-06 16:04:20 +01:00
James Rodewig 9bb9f63364 [DOCS] Note that `trim` filter doesn't change offsets (#53220)
The [word delimiter graph token filter docs][0] note that the `trim`
filter changes the length of tokens without changing their offsets.

This explicitly mentions that in the `trim` filter docs.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html
2020-03-06 07:32:35 -05:00
James Rodewig 4bc6d2dbec [DOCS] Correct link for Lucene StopFilter 2020-03-05 14:52:25 -05:00
Mayya Sharipova 7e2a9f58ee
script_score query errors on negative scores (#53133)
7.5 and 7.6 had a regression that allowed for
script_score queries to have negative scores.
We have corrected this regression in #52478.
This is an addition to #52478 that adds
a test and release notes.
2020-03-05 14:23:39 -05:00
David Turner c2627aa22f Clarify futher the order for a rolling upgrade (#52964)
Expands the "master-ineligible then master-eligible" sentence into a list and
specifies that within these subsets the order doesn't matter.
2020-03-05 15:28:59 +00:00
István Zoltán Szabó 58ce56f6c8 [DOCS] Makes the naming convention of the DFA response objects coherent (#53172) 2020-03-05 16:26:57 +01:00
István Zoltán Szabó 48707ec55a [DOCS] Expands GET DFA stat API docs with response objects. (#53107) 2020-03-05 15:31:55 +01:00
Nik Everett 28df7ae5ed
Support multiple metrics in `top_metrics` agg (backport of #52965) (#53163)
This adds support for returning multiple metrics to the `top_metrics`
agg. It looks like:
```
POST /test/_search?filter_path=aggregations
{
  "aggs": {
    "tm": {
      "top_metrics": {
        "metrics": [
          {"field": "v"},
          {"field": "m"}
        ],
        "sort": {"s": "desc"}
      }
    }
  }
}
```
2020-03-05 08:12:01 -05:00
James Rodewig e46bb54c7b
[DOCS] Document `any` keyword in EQL syntax (#52821) (#53157)
Adds documentation for the `any` keyword to the EQL syntax docs.

Includes:

* Definition of an event category and its relationship to the event
   category field.
* Example matching all event categories using `any` keyword
* Example using `any` with `where true`
2020-03-05 05:02:47 -05:00
James Rodewig 801e50203e [DOCS] Add missing doc type to EQL search results 2020-03-04 10:26:11 -05:00
Yannick Welsch d1e7951e00 [DOCS] Add 7.6.1. release notes (#52874)
Adds the release notes for 7.6.1.
2020-03-04 15:47:54 +01:00
James Rodewig e3d3c3400c [DOCS] Update EQL default event category and timestamp values (#53102)
Updates the documented default `event_category_field` and `timestamp_field`
values for the EQL search API. Also updates related guidance in the
EQL requirement docs.

Relates to #53073.
2020-03-04 09:17:37 -05:00
James Rodewig 0c4bf64095 [DOCS] Fix several Asciidoctor double arrow replacements (#52827)
Per the [Asciidoctor docs][0], Asciidoctor replaces the following
syntax with double arrows in the rendered HTML:

* => renders as ⇒
* <= renders as ⇐

This escapes several unintended replacements, such as in the Painless
docs.

Where appropriate, it also replaces some double arrow instances with
single arrows for consistency.

[0]: https://asciidoctor.org/docs/user-manual/#replacements
2020-03-04 08:43:19 -05:00
James Rodewig 2b59f8ac34 [DOCS] Correct `hits.total.relation` response parm def (#52847)
Fixes a partially completed definition for the `hits.total.relation`
response parameter in the search API docs.
2020-03-04 08:23:34 -05:00
Aleksandr Maus b47bffba24
EQL: consistent naming for event type vs event category (#53073) (#53090)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Lisa Cawley 892f0d5848 [DOCS] Adds link in datafeed indices_options (#53067) 2020-03-03 10:31:32 -08:00
Lisa Cawley bb4bcbc54c [DOCS] Adds ignore_throttled to index conventions (#53063) 2020-03-03 10:23:56 -08:00
James Rodewig cf87724ff6
[DOCS] Reformat `stop` token filter (#53059)
Makes the following changes to the `stop` token filter docs:

* Updates description
* Adds a link to the related Lucene filter
* Adds detailed analyze snippet
* Updates custom analyzer and custom filter snippets
* Adds a list of predefined stop words by language

Co-authored-by: ScottieL <36999642+ScottieL@users.noreply.github.com>
2020-03-03 13:22:52 -05:00
István Zoltán Szabó 6cece3a709 [DOCS] Adds response body documentation to GET inference API (#53050) 2020-03-03 16:26:40 +01:00
Adrien Grand cb868d2f5e
Introduce a `constant_keyword` field. (#49713) (#53024)
This field is a specialization of the `keyword` field for the case when all
documents have the same value. It typically performs more efficiently than
keywords at query time by figuring out whether all or none of the documents
match at rewrite time, like `term` queries on `_index`.

The name is up for discussion. I liked including `keyword` in it, so that we
still have room for a `singleton_numeric` in the future. However I'm unsure
whether to call it `singleton`, `constant` or something else, any opinions?

For this field there is a choice between
 1. accepting values in `_source` when they are equal to the value configured
    in mappings, but rejecting mapping updates
 2. rejecting values in `_source` but then allowing updates to the value that
    is configured in the mapping
This commit implements option 1, so that it is possible to reindex from/to an
index that has the field mapped as a keyword with no changes to the source.

Backport of #49713
2020-03-03 16:01:47 +01:00
James Rodewig bcb68c860c [DOCS] Reorganize EQL requirements page 2020-03-03 07:02:30 -05:00
James Rodewig 5cffa14f45 [DOCS] Fix typo in EQL docs 2020-03-02 16:09:03 -05:00
Costin Leau 712e0c05cd EQL: Add implicit ordering on timestamp (#53004)
QL: Move Sort base class from SQL to QL
(cherry picked from commit 798015b7bbd565e9c4222724614baeb432c7c2b3)
2020-03-02 22:41:36 +02:00
Orhan Toy ad2f630795 [DOCS] Fix formatting of simulate ingest pipeline API docs (#52754) 2020-03-02 11:46:27 -05:00
István Zoltán Szabó a267849def [DOCS] Adds transform-settings page and its subpage to redirects (#52944)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-02 16:36:40 +01:00
Lisa Cawley 4fbe1b0550
[DOCS] Adds cat anomaly detectors API (#52866) (#52970) 2020-03-02 07:28:55 -08:00
Hendrik Muhs a328a8eaf1
[7.x][Transform] implement node.transform to control where to… (#52998)
implement transform node attributes to disable transform on certain nodes and
test which nodes are allowed to do remote connections

closes #52200
closes #50033
closes #48734

backport #52712
2020-03-02 16:10:57 +01:00
James Rodewig db64029919
[7.x] [DOCS] Add parameter examples to EQL search tutorial (#52953)
Makes the following updates to the EQL search tutorial:

* Adds an API response to the basic tutorial
* Adds an example using the `event_type_field` parm
* Adds an example using the `timestamp_field`parm
* Adds an example using the `query` parm
* Updates example dataset to support more EQL query variety
2020-03-02 10:08:03 -05:00
Aleksandr Maus 89ed857c79
EQL: Change request parameter query to filter and rule to query (#52971) (#53006)
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
James Rodewig d336faa0b0 [DOCS] Reformat trim token filter docs (#51649)
Makes the following changes to the `trim` token filter docs:

* Updates description
* Adds a link to the related Lucene filter
* Adds tip about removing whitespace using tokenizers
* Adds detailed analyze snippets
* Adds custom analyzer snippet
2020-03-02 07:48:23 -05:00
James Rodewig f5bccad847 [DOCS] Correct guidance for `index_options` mapping parm (#52899)
Adds a warning admonition stating that the `index_options` mapping
parameter is intended only for `text` fields.

Removes an outdated statement regarding default values for numeric
and other datatypes.
2020-03-02 07:39:35 -05:00
rhymes 7eb4c07f1f [DOCS] Fix typo in index and search analysis docs (#52988) 2020-03-02 07:25:01 -05:00
Yang Wang 712191f2af
[DOCS] Fix typo and ensure asterisks render properly (#52991) (#52992)
Fix a typo and rendering issue for doc test example.
2020-03-02 11:56:42 +11:00
Dimitris Athanasiou 85b4e45093
[7.x]ML] Parse and report memory usage for DF Analytics (#52778) (#52980)
Adds reporting of memory usage for data frame analytics jobs.
This commit introduces a new index pattern `.ml-stats-*` whose
first concrete index will be `.ml-stats-000001`. This index serves
to store instrumentation information for those jobs.

Backport of #52778 and #52958
2020-02-29 13:03:40 +02:00
Rory Hunter b1be7dcd2d Document how to change GC logging behaviour (#52879)
Closes #43990. Describe how to change the default GC settings without changing
the default `jvm.options`. Give examples using `jvm.options.d`, and
`ES_JAVA_OPTS` with Docker.
2020-02-28 21:27:45 +00:00
Martijn van Groningen 6aa9aaa2c6
Add validation for dynamic templates (#52890)
Backport of #51233 to the seven dot x branch.

Tries to load a `Mapper` instance for the mapping snippet of a dynamic template.
This should catch things like using an analyzer that is undefined or mapping attributes that are unused.

This is best effort:
* If `{{name}}` placeholder is used in the mapping snippet then validation is skipped.
* If `match_mapping_type` is not specified then validation is performed for all mapping types.
  If parsing succeeds with a single mapping type then this the dynamic mapping is considered valid.

If is detected that a dynamic template mapping snippet is invalid at mapping update time then the mapping update is failed for indices created on 8.0.0-alpha1 and later. For indices created on prior version a deprecation warning is omitted instead. In 7.x clusters the mapping update will never fail in case of an invalid dynamic template mapping snippet and a deprecation warning will always be omitted.

Closes #17411
Closes #24419

Co-authored-by: Adrien Grand <jpountz@gmail.com>
2020-02-28 10:35:04 +01:00
Nik Everett 1d1956ee93
Add size support to `top_metrics` (backport of #52662) (#52914)
This adds support for returning the top "n" metrics instead of just the
very top.

Relates to #51813
2020-02-27 16:12:52 -05:00
Benjamin Trent eac38e9847
[ML] Add indices_options to datafeed config and update (#52793) (#52905)
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index.

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.

closes https://github.com/elastic/elasticsearch/issues/48056
2020-02-27 13:43:25 -05:00
Nattachai Suteerapongpan 14f847cc8f [DOCS] Fix typo in task management API docs (#52881) 2020-02-27 11:31:11 -05:00
Josh Devins 68ba571f70
Adds recall@k metric to rank eval API (#52889)
This change adds the recall@k metric and refactors precision@k to match
the new metric.

Recall@k is an important metric to use for learning to rank (LTR)
use-cases. Candidate generation or first ranking phase ranking functions
are often optimized for high recall, in order to generate as many
relevant candidates in the top-k as possible for a second phase of
ranking. Adding this metric allows tuning that base query for LTR.

See: https://github.com/elastic/elasticsearch/issues/51676
Backports: https://github.com/elastic/elasticsearch/pull/52577
2020-02-27 16:04:24 +01:00
István Zoltán Szabó 8785f57dfe [DOCS] Reformats cat DFA API docs. (#52885) 2020-02-27 14:21:52 +01:00
István Zoltán Szabó 4a33352a94 [DOCS] Adds cat trained model API documentation (#52824) 2020-02-27 12:54:11 +01:00
Costin Leau 40bc06f6ad EQL: Hook engine to Elasticsearch (#52828)
Add query execution and return actual results returned from
Elasticsearch inside the tests

(cherry picked from commit 3e039282bf991af87604a6d4f8eada19d5e33842)
2020-02-27 11:22:22 +02:00
David Turner 69b78f7f8a "Adding nodes" instructions only work on localhost (#52677)
The introductory sections of the reference manual contains some simplified
instructions for adding a node to the cluster. Unfortunately they are a little
too simplified and only really work for clusters running on `localhost`. If you
try and follow these instructions for a distributed cluster then the new node
will, confusingly, auto-bootstrap itself into a distinct one-node cluster.

Multiple nodes running on localhost is a valid config, of course, but we should
spell out that these instructions are really only for experimentation and that
it takes a bit more work to add nodes to a distributed cluster. This commit
does so.

Also, the "important config" instructions for discovery say that you MUST set
`discovery.seed_hosts` whereas in fact it is fine to ignore this setting and
use a dynamic discovery mechanism instead. This commit weakens this statement
and links to the docs for dynamic discovery mechanisms.

Finally, this section is also overloaded with some technical details that are
not important for this context and are adequately covered elsewhere, and
completely fails to note that the default discovery port is 9300. This commit
addresses this.
2020-02-27 09:18:37 +00:00
James Rodewig f5253d20f7 [DOCS] Update term vectors snippet to prevent CI failure (#52819)
Adds the `?refresh=wait_for` query argument to an index API snippet in
the term vectors API docs.

This should ensure the document is indexed and available before a
subsequent term vectors API request executes.

Fixes #52814.
2020-02-26 12:41:40 -05:00
Lisa Cawley b788ec7157 [DOCS] Adds cat datafeeds API (#52738) 2020-02-26 09:28:57 -08:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests (#52114) (#52798)
* Smarter copying of the rest specs and tests (#52114)

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
Bogdan Pintea 304e1e69b8 remove references to the SQL API from ODBC config (#52765)
Remove reference to an "SQL API" which could suggest that one needs to
treat this in a special way when configuring the ODBC driver.

(cherry picked from commit 451c341e0193b542409e8891ec2a31e62529a5e7)
2020-02-26 13:39:54 +01:00
István Zoltán Szabó f57422bbfd [DOCS] Adds cat data frame analytics API (#52764)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-26 11:10:42 +01:00
Lisa Cawley 05f1cd74a6 [DOCS] Fixes monitoring links (#52790) 2020-02-25 18:08:23 -08:00
Lisa Cawley 924f0bd243 [DOCS] Updates custom rules example (#52731) 2020-02-25 09:32:52 -08:00
Andrei Stefan 51c6aefa55
SQL: Use calendar_interval of 1d for HISTOGRAMs with 1 DAY intervals (#52749) (#52771)
(cherry picked from commit 556f5fa33be88570c4f8550cb8f784323d26a707)
2020-02-25 18:44:02 +02:00
Pius 563f033511 Update ilm-settings.asciidoc (#51577) 2020-02-25 10:18:55 -05:00
bellengao d2db16e046 [DOCS] Correct policy name in ILM docs example (#52354)
Updates an example snippet to use a consistent policy name.
2020-02-25 09:36:22 -05:00
David Pilato 6c6ab8fa47 [DOS] Fix typo in CSV processor docs (#52649)
Corrects an example array in a snippet of the CSV processor docs.
2020-02-25 08:48:50 -05:00
bellengao 49f37989c4 [DOCS] Fix typo in ingest node docs (#52671) 2020-02-25 07:57:52 -05:00
David Roberts cf122d13b8 [ML] Use event.timezone in file_structure_finder ingest pipeline (#52720)
This is because beat.timezone was renamed to event.timezone in
elastic/beats#9458
2020-02-25 12:33:53 +00:00
James Rodewig 9b05f6a668 [DOCS] Add admonition for app using cat APIs (#52727)
Adds an explicit "important" admonition discouraging apps from using
cat APIs.

cat APIs are intended for human consumption via the command line or
Kibana console only. They are not intended for consumption by
applications.
2020-02-25 07:20:33 -05:00
James Rodewig 1a14ae4e1b [DOCS] Document `include_in_*` nested mapping parms (#52648)
Adds documentation for the `include_in_parent` and `include_in_root`
mapping parameters for the `nested` mapping datatype.
2020-02-25 07:13:49 -05:00
Adrien Grand 5f81906fcf Discourage from opting in for the `niofs` store. (#52638)
Indices open with the `niofs` store type load much more data on-heap than
indices open with the `mmapfs` store type. This limitation is now documented
and examples have been updated to show how to update settings to use the
`mmapfs` store type rather than `niofs`.
2020-02-25 08:54:11 +01:00
Adrien Grand 9b0ddc1c03 Clarify the resiliency trade-off of disabling replicas to speed up indexing. (#52714)
We should be more explicit about the downsides of disabling replicas and
explain that users should be ready to re-do the entire load in case of
issues mid-way.
2020-02-25 08:54:10 +01:00
Adrien Grand 5ce66b8b3c Document how CCR may be used to speed up indexing. (#52717)
One architecture that we have recommended to several users to speed up
indexing involved using CCR to prevent searching from stealing resources
from indexing.
2020-02-25 08:54:10 +01:00
Bob Blank 28d4b71947
Clarified http.max_content_length description (#52329)
Adding "greater than" based on discussion with @jasontedor for clarity.
2020-02-24 21:01:14 -05:00
Andrei Stefan ed6b10bc03
SQL: use a calendar interval for histograms over 1 month intervals (#52586) (#52715)
(cherry picked from commit 928b11a34ec92d90d082abdf4fa09f7ce1d7c0c4)
2020-02-25 01:41:51 +02:00
Julie Tibshirani ba0401ecfd Correct the name of the search timeout parameter. (#52733)
The request body parameter is called 'timeout', not 'search_timeout'.
2020-02-24 14:59:06 -08:00
lcawl c6e35b460e [DOCS] Adds anchor for custom rules 2020-02-24 11:39:15 -08:00
Mayya Sharipova 034b1c0ba3
Correct boost calculation in script_score query (#52478) (#52724)
Before boost in script_score query was wrongly applied only to the subquery.
This commit makes sure that the boost is applied to the whole score
that comes out of script.

Closes #48465
2020-02-24 13:48:21 -05:00
Przemko Robakowski e72cb79476
Add docs for errors in GetAlias API (#51850) (#52716)
Closes #31499

Co-authored-by: Maxim <timonin.maksim@mail.ru>
2020-02-24 18:22:09 +01:00
James Rodewig 5e48811585 [DOCS] Document CCS-supported APIs (#52708)
Explicitly notes the Elasticsearch API endpoints that support CCS.

This should deter users from attempting to use CCS with other API
endpoints, such as `GET <index>/_doc/<_id>`.
2020-02-24 09:59:08 -05:00
Ignacio Vera ba9d3c6389
Add support for multipoint shape queries (#52564) (#52705) 2020-02-24 13:46:51 +01:00
James Rodewig 98bcf06bae [DOCS] Correct multi search API docs (#52523)
* Adds an example request to the top of the page.
* Relocates several parameters erroneously listed under "Request body"
to the appropriate "Query parameters" section.
* Updates the "Request body" section to better document the NDJSON
  structure of msearch requests.
2020-02-24 07:43:10 -05:00
Marios Trivyzas c03f51f68f
[Docs] Clarify default value for `allow_no_indices` (#52635) (#52697)
Add default value to each one of the usages of `allow_no_indices`
since it differs between different APIs.

Relates to: #52534

(cherry picked from commit 2eb986488ac326d6da6ab8ad0203a94e08684a36)
2020-02-24 11:57:32 +01:00
Benjamin Trent afd90647c9
[ML] Adds feature importance to option to inference processor (#52218) (#52666)
This adds machine learning model feature importance calculations to the inference processor.

The new flag in the configuration matches the analytics parameter name: `num_top_feature_importance_values`
Example:
```
"inference": {
   "field_mappings": {},
   "model_id": "my_model",
   "inference_config": {
      "regression": {
         "num_top_feature_importance_values": 3
      }
   }
}
```

This will write to the document as follows:
```
"inference" : {
   "feature_importance" : {
      "FlightTimeMin" : -76.90955548511226,
      "FlightDelayType" : 114.13514762158526,
      "DistanceMiles" : 13.731580450792187
   },
   "predicted_value" : 108.33165831875137,
   "model_id" : "my_model"
}
```

This is done through calculating the [SHAP values](https://arxiv.org/abs/1802.03888).

It requires that models have populated `number_samples` for each tree node. This is not available to models that were created before 7.7.

Additionally, if the inference config is requesting feature_importance, and not all nodes have been upgraded yet, it will not allow the pipeline to be created. This is to safe-guard in a mixed-version environment where only some ingest nodes have been upgraded.

NOTE: the algorithm is a Java port of the one laid out in ml-cpp: https://github.com/elastic/ml-cpp/blob/master/lib/maths/CTreeShapFeatureImportance.cc

usability blocked by: https://github.com/elastic/ml-cpp/pull/991
2020-02-21 18:42:31 -05:00