Commit Graph

7515 Commits

Author SHA1 Message Date
Julie Tibshirani 4f4c4a8713 Add a reference on returning fields during a search. (#57500)
This PR adds a section to the new 'run a search' reference that explains
the options for returning fields. Previously each option was only listed as a
separate request parameter and it was hard to know what was available.
2020-06-03 09:41:48 -07:00
James Rodewig 80faafdfc7
[DOCS] Add clear scroll API reference docs (#57367) (#57611) 2020-06-03 11:58:16 -04:00
Marios Trivyzas a674844893
SQL: Implement TRIM function (#57518) (#57593)
Add `TRIM` function which combines the functionality of both
`LTRIM` and `RTRIM` by stripping both leading and trailing
whitespaces.

Refers to #41195

(cherry picked from commit 6c86c919e12f0c4cb5e39d129aa65ab3e274268f)
2020-06-03 15:19:48 +02:00
William Brafford 6e67f1b3dc
[DOCS] Add release notes for 7.7.1 (#57566)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-06-03 08:54:32 -04:00
Tal Levy d638ecda68
fix missing _doc type from geo-shape doc test (#57556) 2020-06-02 13:44:08 -07:00
Jess dc12a687be
[DOCS] Edit validation section of dynamic templates docs (#57510) 2020-06-02 16:28:06 -04:00
Tal Levy f360020048 Update geo_point docs for geo_shape queries (#57487)
This commit highlights the ability for geo_point fields to be
used in geo_shape queries. It also adds an explicit geo_point
example in the geo_shape query documentation

Closes #56927.
2020-06-02 12:56:54 -07:00
James Rodewig a2e44a0c76
[DOCS] Refactor admons for multi-parameter options (#57491) (#57540)
Several APIs support options that can be specified as a query parameter or a
request body parameter.

Currently, this is documented using notes, which can get rather lengthy. This
replaces those multiple notes with a single note and a footnote.
2020-06-02 12:12:29 -04:00
James Rodewig 5950d1aec8 [DOCS] Fix heading capitalization in CCS docs 2020-06-02 11:13:13 -04:00
James Rodewig 641ed484d8
[DOCS] EQL: Add `dev` admonition to EQL pages (#57531) (#57533)
Adds the `dev` admonition to EQL features, which are in development
under a feature flag.
2020-06-02 11:03:12 -04:00
James Rodewig 808835ac1c
[DOCS] Add scroll API reference docs (#57153) (#57528)
Changes:

* Adds API reference docs for the scroll API
* Documents several related parameters in the search API docs
2020-06-02 10:11:12 -04:00
James Rodewig fd6dabf158
[DOCS] EQL: Fix hits param for sequences (#57410) (#57524) 2020-06-02 09:38:00 -04:00
Marios Trivyzas b8a13de20f
SQL: Implement TOP as an alternative to LIMIT (#57428) (#57507)
Add basic support for `TOP X` as a synonym to LIMIT X which is used
by [MS-SQL server](https://docs.microsoft.com/en-us/sql/t-sql/queries/top-transact-sql?view=sql-server-ver15),
e.g.:

```
SELECT TOP 5 a, b, c FROM test
```

TOP in SQL server also supports the `PERCENTAGE` and `WITH TIES`
keywords which this implementation doesn't.

Don't allow usage of both TOP and LIMIT in the same query.

Refers to #41195

(cherry picked from commit 2f5ab81b9ad884434d1faa60f4391f966ede73e8)
2020-06-02 10:53:42 +02:00
Julie Tibshirani e434c481dc Avoid unnecessary use of stored_fields in our docs. (#57488)
Generally we don't advocate for using `stored_fields`, and we're interested in
eventually removing the need for this parameter. So it's best to avoid using
stored fields in our docs examples when it's not actually necessary.

Individual changes:
* Avoid using 'stored_fields' in our docs.
* When defining script fields in top-hits, de-emphasize stored fields.
2020-06-01 17:31:42 -07:00
Lisa Cawley db5bf92acf
[7.x][DOCS] Replace docdir attribute with es-repo-dir (#57489) (#57494) 2020-06-01 16:42:53 -07:00
James Rodewig 6592c3856d [DOCS] Fix deep paging recommendations
Corrects recommendation to reference the `search_after` parameter,
not API.

Also corrects a typo and whitespace inconsistencies in the search docs.
2020-06-01 18:04:04 -04:00
Lisa Cawley a1514c9ffe
[DOCS] Replaces docdir attributes in ML APIs (#57390) (#57467) 2020-06-01 13:46:15 -07:00
James Rodewig 994781ff36
[DOCS] Add search pagination docs (#56785) (#57477)
Reworks the `from / size` content to `Paginate search results`.

Moves those docs from the request body search API page (slated for
deletion) to the `Run a search` tutorial docs.

Also adds some notes to the `from` and `size` param docs.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-01 16:43:06 -04:00
James Rodewig ab8ae7cf25
[DOCS] Combine search API and URI search API reference docs (#55884) (#57469)
The search API and URI search pages document the same `_search` API.
This combines the documentation from each page under the search API
docs.
2020-06-01 15:53:40 -04:00
Benjamin Trent 1aea9d5f49
Adding transform docs for geotile_grid (#57000) (#57474)
transforms and composite aggs support geotile_grid as a source. This adds documentation explaining that support.
2020-06-01 15:46:37 -04:00
Zachary Tong 4dc12633cf
Better description of real-memory breaker changes to aggs (#57306)
The old description mentions a setting that we ended up not merging.
The periodic real-memory checks are automatic and do not require
the user to configure any setting.
2020-06-01 14:58:20 -04:00
James Rodewig 50a8779c94
[DOCS] Create top-level "Search your data" page (#56058) (#57463)
**Goal**

Create a top-level search section. This will let us clean up our search
API reference docs, particularly content from [`Request body search`][0].

**Changes**

* Creates a top-level `Search your data` page. This page is designed to
  house concept and tutorial docs related to search.

* Creates a `Run a search` page under `Search your data`. For now, This
  contains a basic search tutorial. The goal is to add content from
  [`Request body search`][0] to this in the future.

* Relocates `Long-running searches` and `Search across clusters` under
  `Search your data`. Increments several headings in that content.

* Reorders the top-level TOC to move `Search your data` higher. Also
  moves the `Query DSL`, `EQL`, and `SQL access` chapters immediately
  after.

Relates to #48194

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-body.html
2020-06-01 14:55:26 -04:00
James Rodewig 12813d5918
[DOCS] Document `dynamic` and `static` setting types (#56919) (#57457) 2020-06-01 14:19:52 -04:00
James Rodewig cde5b7d2b3
[DOCS] Relocate discovery module content (#56611) (#57454)
* Moves `Discovery and cluster formation` content from `Modules` to
`Set up Elasticsearch`.

* Combines `Adding and removing nodes` with `Adding nodes to your
  cluster`. Adds related redirect.

* Removes and redirects the `Modules` page.

* Rewrites parts of `Discovery and cluster formation` to remove `module`
  references and meta references to the section.
2020-06-01 14:13:13 -04:00
James Rodewig 6934264162
[DOCS] Relocate `shard allocation` module content (#56535) (#57448) 2020-06-01 13:15:08 -04:00
Lisa Cawley b5542e0480
[DOCS] Adds ml-cpp PRs to release notes (#57444) 2020-06-01 09:56:32 -07:00
James Rodewig 5c12a1f063
[DOCS] Relocate `local gateway` setting content (#56448) (#57443)
Moves `local gateway` setting content to `Configuring Elasticsearch`
from `Modules`.

Relates to #53303.
2020-06-01 12:40:15 -04:00
James Rodewig 2ea5372eb7
[DOCS] Relocate `HTTP` module content (#56386) (#57439)
Moves `HTTP` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Replaces `http` with `HTTP` throughout
* Replaces `HTTP module` with `HTTP layer` throughout
* Increments several headings
* Adds explicit anchors to several headings
* Removes several unneeded `[float]` attributes

Closes #53306

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-06-01 12:22:55 -04:00
James Rodewig 4dd05ccf9e
[DOCS] Relocate `node` module content (#56376) (#57435)
Moves `node` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Retitles page `Node settings`
* Increments several headings
* Removes several unneeded `[float]` attributes
* Replaces remaining `[float]` attributes with `[discrete]`
* Updates the `Other node settings` section

Relates to #53307

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-06-01 12:04:15 -04:00
James Rodewig 131ce74840
[DOCS] Relocate thread pools content (#55814) (#57432)
Moves [thread pools content][0] from [Modules][1] to
[Configuring Elasticsearch][2].

Supporting changes:
* Changes page title to "Thread pools"
* Increments several headings
* Removes several unneeded `[float]` attributes
* Updates the anchors of several headings

Relates to #53307

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-threadpool.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html

Co-authored-by: debadair <debadair@elastic.co>
2020-06-01 11:46:16 -04:00
James Rodewig 462a39bc2a
[7.x] [DOCS] Relocate network settings content (#55865) (#57429)
* [DOCS] Relocate network settings content (#55865)

Moves [network settings content][0] from [Modules][1] to the
[Configuring Elasticsearch][2] section.

Supporting changes:
* Changes page title and headings to sentence case
* Increments several headings
* Removes several unneeded `[float]` and `[horizontal]` attributes
* Removes several unneeded blank links in definition lists

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-network.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html

* tweak for consistency
2020-06-01 11:27:27 -04:00
James Rodewig 1fcd73cd34
[DOCS] Relocate `transport` module content (#55472) (#57425)
Moves `transport` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Adds explicit anchors to several headings
* Changes several headings to sentence case
* Increments several headings
* Removes several unneeded `[float]` attributes
* Replaces `transport module` with `transport layer`

Relates to #53305

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-06-01 10:56:28 -04:00
James Rodewig de5de4bf58
[DOCS] Relocate "Remote Clusters" page (#55172) (#57422)
Relocates the "Remote Clusters" documentation from the "Modules" section to the "Set up Elasticsearch" section.

Supporting changes:
* Reorders the "Bootstrap checks for X-Pack" section to immediately follow the "Bootstrap checks"chapter.
* Removes an outdated X-Pack `idef` from the "Remote Clusters" intro.
2020-06-01 10:28:34 -04:00
James Rodewig 8c0e7c8a6e
[DOCS] Relocate "Plugins" page (#54974) (#57418) 2020-06-01 10:07:50 -04:00
James Rodewig 43ef469570
[DOCS] Relocate `indices` module content (#54903) (#57413)
Moves `indices` content from the [Modules][0] section to the [Configuring
Elasticsearch][1] section.

Also removes the [Indices][2] landing page and adds a related redirect.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-indices.html
2020-06-01 09:44:32 -04:00
Florian Kelbert 25f69547d9 [DOCS] Fix typo in rank feature query docs (#57401) 2020-06-01 08:42:45 -04:00
Nik Everett 07c76f2894
Update date_histogram docs (#56922) (#57387)
* Make it more clear that you can use `month` or `1M`.
* Explain rounding rules
* Consistently use "time zone" instead of "timezone". It looks like both
  are right but I see "time zone" much more. And the parameter in
  elasticsearch is `time_zone` so we may as well line up.

Closes #56760

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-29 17:40:40 -04:00
James Rodewig ebe433343f [DOCS] Correct link for ESMS redirect 2020-05-29 16:43:04 -04:00
Tomasz Elendt a7c36c8af5 Support multiple tokens on LHS in stemmer_override rules (#56113) (#56484)
This commit adds support for rules with multiple tokens on LHS, also
known as "contraction rules", into stemmer override token
filter. Contraction rules are handy into translating multiple
inflected words into the same root form. One side effect of this change is
that it brings stemmer override rules format closer to synonym rules
format so that it makes it easier to translate one into another.

This change also makes stemmer override rules parser more strict so
that it should catch more errors which were previously accepted.

Closes #56113
2020-05-29 22:34:31 +02:00
Benjamin Trent 35d5126cea
[7.x] [ML] adds new for_export flag to GET _ml/inference API (#57351) (#57368)
* [ML] adds new for_export flag to GET _ml/inference API (#57351)

Adds a new boolean flag, `for_export` to the `GET _ml/inference/<model_id>` API.

This flag is useful for moving models between clusters.
2020-05-29 14:01:08 -04:00
James Rodewig 7dbf5baf60 [DOCS] Remove Elastic Stack Monitoring Service docs (#57371) 2020-05-29 13:16:57 -04:00
Gordon Brown 1d5e476256
Add expand_wildcards to _cat/indices and _cat/aliases docs (#56964)
This commit adds the `expand_wildcards` parameter documentation to the
`_cat/indices` and `_cat/aliases` docs, as those APIs now support
`expand_wildcards`. Additionally, clarifies the `expand_wildcards` docs with
respect to hidden indices.
2020-05-29 11:01:53 -06:00
Benjamin Trent c8374dc9f3
[ML] add max_model_memory parameter to forecast request (#57254) (#57355)
This adds a max_model_memory setting to forecast requests. 
This setting can take a string value that is formatted according to byte sizes (i.e. "50mb", "150mb").

The default value is `20mb`.

There is a HARD limit at `500mb` which will throw an error if used.

If the limit is larger than 40% the anomaly job's configured model limit, the forecast limit is reduced to be strictly lower than that value. This reduction is logged and audited.

related native change: https://github.com/elastic/ml-cpp/pull/1238

closes: https://github.com/elastic/elasticsearch/issues/56420
2020-05-29 11:16:08 -04:00
Marios Trivyzas b2651323fd
SQL: Implement TIME_PARSE function for parsing strings into TIME values (#55223) (#57342)
Implement TIME_PARSE(<time_str>, <pattern_str>) function
which allows to parse a time string according to the specified
pattern into a time object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Closes #54963

Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: Patrick Jiang(白泽) <patrickjiang0530@gmail.com>

(cherry picked from commit 1fe1188d449cad7d0782a202372edc52a4014135)
2020-05-29 15:48:37 +02:00
Adam Locke 11ae062877
[7.6] [DOCS] Explain flood stage watermark (#57321)
* [7.7] [DOCS] Explain flood stage watermark

* Adding id to fix cross-linking issue.
2020-05-28 19:29:32 -04:00
James Rodewig 47024ccb3c
[DOCS] Add verify snapshot repository API docs (#57253) (#57288) 2020-05-28 09:48:04 -04:00
Martijn van Groningen 225ccd1cfa
Ensure template exists when creating data stream (#57275)
Backporting #56888 to 7.x branch.

Limit the creation of data streams only for namespaces that have a composable template with a data stream definition.

This way we ensure that mappings/settings have been specified and will be used at data stream creation and data stream rollover.

Also remove `timestamp_field` parameter from create data stream request and
let the create data stream api resolve the timestamp field
from the data stream definition snippet inside a composable template.

Relates to #53100
2020-05-28 15:08:25 +02:00
István Zoltán Szabó e1cab4feb4 [DOCS] Puts a link into the loss_function variable description (#56678) 2020-05-28 09:46:11 +02:00
James Rodewig a0ca0325fe
[DOCS] Reformat `min_hash` token filter docs (#57181) (#57246)
Changes:

* Rewrites description and adds a Lucene link
* Reformats the configurable parameters as a definition list
* Changes the `Theory` heading to `Using the min_hash token filter for
  similarity search`
* Adds some additional detail to the analyzer example
2020-05-27 15:08:23 -04:00
Lee Hinman c0f732b9f6
[7.x] Rename template V2 classes to ComposableTemplate (#57183) (#57232)
Backports the following commits to 7.x:

    Rename template V2 classes to ComposableTemplate (#57183)
2020-05-27 11:01:59 -06:00
Nik Everett 4d5be7c817
Save memory on numeric sig terms when not top (backport of #56789) (#57221)
This saves memory when running numeric significant terms which are not
at the top level by merging its collection into numeric terms and relying
on the optimization that we made in #55873.
2020-05-27 12:03:28 -04:00
Toby Sutor 2f6089c2b9 [DOCS] Add note about restoring snapshot versions (#55225)
As discussed at https://elastic.slack.com/archives/C0D1XEXEZ/p1586939752242300 is it not possible to restore snapshots taken on newer versions into clusters running lower versions. For example, a snapshot created in a 7.6.0 cluster cannot be restored on a 7.5.0 cluster. This needs to be documented.
2020-05-27 11:40:16 -04:00
James Rodewig 6dbe0022fa
[DOCS] Add redirects for 404 pages (#57161)
Adds several redirects for removed pages that return a
404 response.
2020-05-27 11:15:31 -04:00
James Rodewig f4a673c722
[DOCS] Add get snapshot repo API docs (#57150) (#57218) 2020-05-27 11:12:55 -04:00
James Rodewig 5ae650ecfd
[DOCS] Fix clarity of URL snapshot repo docs (#56920) (#57212)
Makes following changes to better clarify docs for read-only URL
snapshot repositories:

* Adds an example snippet for registering a URL repository
* Rewrites the protocols paragraph
* Adds a note to explicitly point out that only URLs using the `ftp`,
  `http`, `http`, and `jar` protocols do not need the `path.repo`
  setting.

Fixes #16280
2020-05-27 10:48:19 -04:00
James Rodewig ff63bda4be
[DOCS] Document index alias swaps are atomic (#55418) (#57209)
Co-authored-by: Tugberk Ugurlu <tugberk@outlook.com>
2020-05-27 10:23:51 -04:00
James Rodewig 2c2673d759
[DOCS] Fix remote cluster client node docs (#57176) (#57205)
Changes:

* Documents the `r` role in the _cat/nodes API docs

* Fixes a typo in the node docs

Co-authored-by: Nick Canzoneri <nickcanz@github.com>
2020-05-27 08:58:29 -04:00
István Zoltán Szabó 27f258711a [DOCS] Fixes formatting of admonition paragraph in PUT inference API docs. (#57196) 2020-05-27 13:43:55 +02:00
James Rodewig 9f1e3bc82b
[DOCS] Document impact of restore API's `include_global_state` param (#57067) (#57155)
Previously, the restore API snippet included a `include_global_state` value of `true`.

Some users copy and paste the code example verbatim, updating only the index and
snapshot value names. Running the snippet could inadvertently wipe out a
cluster's current ILM policies, index templates, and ingest pipelines.

This change updates the snippet to use a `include_global_state` value of
`false`. It also adds a callout that better describes impacts of
using a `include_global_state` argument of `true`.

Co-authored-by: Mike Wong <mike.wong@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: David Turner <david.turner@elastic.co>
2020-05-26 11:39:39 -04:00
Russ Cam 8fd73d1e4f [DOCS] Add .NET client link to Bulk API docs (#57095)
This commit adds a link in the Bulk API documentation to
the .NET client's BulkAllObservable type and associated methods.
2020-05-26 09:12:57 -04:00
István Zoltán Szabó 47bf95cee3 [DOCS] Improves navigation between forecast APIs and adds short description. (#57035) 2020-05-25 09:11:00 +02:00
Marios Trivyzas b91bae30b1
SQL: [Tests] Move JDBC integration tests to new module (#56872) (#57072)
Move the JDBC functionality integration tests from `:sql:qa` to a separate
module `:sql:qa:jdbc`. This way the tests are isolated from the rest of the
integration tests and they only depend to the `:sql:jdbc` module, thus
removing the danger of accidentally pulling in some dependency that may
hide bugs.

Moreover this is a preparation for #56722, so that we can run those tests
between different JDBC and ES node versions and ensure forward
compatibility.

Move the rest of existing tests inside a new `:sql:qa:server` project, so that
the `:sql:qa` becomes the parent project for both and one can run all the integration
tests by using this parent project.

(cherry picked from commit c09f4a04484b8a43934fe58fbc41bd90b7dbcc76)
2020-05-22 17:49:36 +02:00
James Rodewig b3426dd558
[DOCS] Add delete snapshot repo API docs (#57043)
Changes:

* Adds API reference docs for the delete snapshot repo API.

* Corrects an error in the delete snapshot repo API spec. Comma-separated
repository names are not supported.

* Relocates the existing delete snapshot repo API example docs.
2020-05-21 14:47:07 -04:00
James Rodewig a2de43d468
[DOCS] Reformat `shingle` token filter (#57040)
Changes:

* Rewrites description and adds Lucene link
* Adds analyze example
* Rewrites parameter documentation
* Updates custom analyzer and filter examples
* Adds anchor to `index.max_shingle_diff` index-level setting
2020-05-21 13:56:54 -04:00
István Zoltán Szabó 9b7356d6af [DOCS] Removes the Jobs section from the ML anomaly detection APIs page. (#57031) 2020-05-21 17:32:07 +02:00
James Rodewig 9455bc84d8 [DOCS] Fix anchor in put snapshot repo API docs 2020-05-21 09:36:49 -04:00
James Rodewig 5cb34d9a6e
[DOCS] Reformat `hunspell` token filter (#56955)
Changes:

* Rewrites description and adds Lucene link
* Adds analyze example
* Rewrites parameter documentation
* Updates custom analyzer example
* Rewrites related setting documentation
2020-05-20 14:47:53 -04:00
Brandon Morelli ec41d36c62
docs: update links to beats security docs (#56875) (#56953) 2020-05-20 11:28:39 -07:00
James Rodewig b7c6f0d02f
[DOCS] Fix default for `http.compression` setting (#56899) (#57002)
Elasticsearch enables HTTP compression by default. However, to mitigate
potential security risks like the BREACH attack, compression is disabled by
default if HTTPS is enabled.

This updates the `http.compression` setting definition accordingly and adds
additional context.

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-05-20 12:12:05 -04:00
Albert Zaharovits fd54ed328c DOCS Correct audit emit_node_id default value as false (#56995)
Since version 7, the `xpack.security.audit.logfile.emit_node_id` setting defaults
to `false`, yet the docs say otherwise. This commit fixes that.
2020-05-20 16:35:30 +03:00
Théophile Helleboid - chtitux a2c6d61ed5 [DOCS] Fix typo in search API `explain` param def (#56991)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-20 09:22:23 -04:00
Eddie Turizo 27673aa59c [DOCS] Fix ulimit value in system settings docs (#56973)
Correctly documents the ulimit value as 65535.

The limit was lowered as part of #37537.
2020-05-20 08:44:07 -04:00
Andrei Balici 19a336e8d3 Add `max_token_length` setting to the CharGroupTokenizer (#56860)
Adds `max_token_length` option to the CharGroupTokenizer.
Updates documentation as well to reflect the changes.

Closes #56676
2020-05-20 14:28:40 +02:00
James Rodewig cc12361a82 [DOCS] EQL: Fix whitespace in EQL snippet 2020-05-19 17:04:49 -04:00
James Rodewig f6d2688de2
[DOCS] Add JS client helper links to docs (#55216) (#56968)
Adds links for the Elasticsearch-js client to the bulk and scroll docs.

Co-authored-by: Tomas Della Vedova <delvedor@users.noreply.github.com>
2020-05-19 16:53:22 -04:00
Adam Locke cbd35e9a2b
[DOCS] Add links to `flattened` datatype (#56794) (#56963)
* Changes for #52239.

* Incorporating review feedback from Julie T. Also single-sourcing nexted options in the Mapping page and referencing them in the Nested page.

* Moving tip after the introduction and clarifying limits.

* Update docs/reference/mapping.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/mapping/types/nested.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-19 16:37:30 -04:00
James Rodewig 771ddbf083
[DOCS] EQL: Add sequence example to tutorial (#56965) (#56966)
Adds an example using the sequence syntax to the 'Run an EQL search'
tutorial.

Supplements other examples added with #56721
2020-05-19 16:14:57 -04:00
James Rodewig cc43d67eb1 [DOCS] Add leading slashes to EQL API examples 2020-05-19 15:38:37 -04:00
Adam Mohammed fd84fdf581 [DOCS] Fix typo in secure settings docs (#56778) 2020-05-19 12:04:25 -04:00
James Rodewig 0f233b1b0b
[DOCS] Remove outdated links for `similarity` mapping param args (#56925) 2020-05-19 11:04:53 -04:00
Lee Hinman e208925465
[7.x] Add template simulation API for simulating template composition (#56842) (#56924) 2020-05-19 08:12:21 -06:00
James Rodewig 342e713e2a
[DOCS] Fix fingerprint token filter's analyzer example (#56811) (#56943)
Co-authored-by: Abhilash Bolla <2282894+ivssh@users.noreply.github.com>
2020-05-19 09:30:00 -04:00
James Rodewig 22f54ba205 [DOCS] EQL: Fix API example headings 2020-05-18 16:29:29 -04:00
James Rodewig 74554f1ae8
[DOCS] Add put snapshot repo API docs (#56827) (#56900) 2020-05-18 08:55:22 -04:00
Benjamin Trent 297f864884
[ML] relax throttling on expired data cleanup (#56711) (#56895)
Throttling nightly cleanup as much as we do has been over cautious.

Night cleanup should be more lenient in its throttling. We still
keep the same batch size, but now the requests per second scale
with the number of data nodes. If we have more than 5 data nodes,
we don't throttle at all.

Additionally, the API now has `requests_per_second` and `timeout` set.
So users calling the API directly can set the throttling.

This commit also adds a new setting `xpack.ml.nightly_maintenance_requests_per_second`.
This will allow users to adjust throttling of the nightly maintenance.
2020-05-18 08:46:42 -04:00
James Rodewig e492c23944
[DOCS] Sort metric and pipeline agg docs (#56613) (#56846)
Co-authored-by: Gil Raphaelli <gil@elastic.co>
2020-05-15 17:15:53 -04:00
Gabriel Petrovay cb4d5f5042 Fixed calendar intervals documentation (#56666)
- the 1-letter intervals are not parseable (`m`, `h`, `d`, `w`,  `M`, `q`, `y`)
- fixed formatting broken by new lines
2020-05-15 16:55:57 -04:00
James Rodewig c50f86fbba
[DOCS] EQL: Document `case_sensitive` param (#56697) (#56818) 2020-05-15 11:47:19 -04:00
Dan Hermann 66871c5342
[7.x] Rename endpoint from plural "_data_streams" to singular "_data_stream" (#56825) 2020-05-15 10:27:53 -05:00
Dan Hermann 2a21d4d976
Docs for data stream REST APIs 2020-05-15 09:37:45 -05:00
James Rodewig 5e09762a27 [DOCS] EQL: Align comments in `between` fn examples 2020-05-15 09:20:45 -04:00
James Rodewig 24cd45345e [DOCS] EQL: Remove references to arrays/multi-value fields (#56772) 2020-05-15 09:09:07 -04:00
Benjamin Trent f71c305090
[7.x] [Transform] add support for terms agg in transforms (#56696) (#56809)
* [Transform] add support for terms agg in transforms (#56696)

This adds support for `terms` and `rare_terms` aggs in transforms. 

The default behavior is that the results are collapsed in the following manner:
`<AGG_NAME>.<BUCKET_NAME>.<SUBAGGS...>...`
Or if no sub aggs exist
`<AGG_NAME>.<BUCKET_NAME>.<_doc_count>`

The mapping is also defined as `flattened` by default. This is to avoid field explosion while still providing (limited) search and aggregation capabilities.
2020-05-15 08:08:43 -04:00
Tal Levy 5e90ff32f7
Add Normalize Pipeline Aggregation (#56399) (#56792)
This aggregation will perform normalizations of metrics
for a given series of data in the form of bucket values.

The aggregations supports the following normalizations

- rescale 0-1
- rescale 0-100
- percentage of sum
- mean normalization
- z-score normalization
- softmax normalization

To specify which normalization is to be used, it can be specified
in the normalize agg's `normalizer` field.

For example:

```
{
  "normalize": {
    "buckets_path": <>,
    "normalizer": "percent"
  }
}
```
2020-05-14 17:40:15 -07:00
Lee Hinman a73d7d9e2b
[7.x] Don't allow invalid template combinations (#56397) (#56795)
Backports the following commits to 7.x:

- Don't allow invalid template combinations (#56397)
2020-05-14 16:20:53 -06:00
James Rodewig 2a943a58a4
[DOCS] EQL: Document `number` function (#56770)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-14 15:44:04 -04:00
James Rodewig 2921747b23
[7.x] [DOCS] EQL: Document sequences (#56721) (#56774)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-14 11:51:40 -04:00
Lisa Cawley 6a8e10189f [DOCS] Add throttling based on configuration parameter (#56653) 2020-05-14 08:45:29 -07:00
David Roberts 4438115be0 [DOCS] Docs changes for overridden delimiter in find_file_structure (#56288)
Docs for #55735

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-14 09:25:21 +01:00
debadair 83e9ff42da
[DOCS] Added info about automatic config for Beats & Logstash. (#56317) (#56729)
* [DOCS] Added info about automatic config for Beats & Logstash.

* Update docs/reference/ilm/set-up-lifecycle-policy.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/ilm/set-up-lifecycle-policy.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/ilm/index.asciidoc

* Updated note in GS tutorial

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-13 19:27:22 -07:00
debadair 60f8a32dba
[DOCS] Add info about ILM and unallocated shards. (#56655) (#56724)
* [DOCS] Add info about ILM and unallocated shards.

* Incorporated review feedback.

* Update docs/reference/ilm/actions/ilm-allocate.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Fix xref

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-13 16:12:37 -07:00
Nik Everett 126619ae3c
Add list of defered aggregations to the profiler (backport of #56208) (#56682)
This adds a few things to the `breakdown` of the profiler:
* `histogram` aggregations now contain `total_buckets` which is the
  count of buckets that they collected. This could be useful when
  debugging a histogram inside of another bucketing agg that is fairly
  selective.
* All bucketing aggs that can delay their sub-aggregations will now add
  a list of delayed sub-aggregations. This is useful because we
  sometimes have fairly involved logic around which sub-aggregations get
  delayed and this will save you from having to guess.
* Aggregtations wrapped in the `MultiBucketAggregatorWrapper` can't
  accurately add anything to the breakdown. Instead they the wrapper
  adds a marker entry `"multi_bucket_aggregator_wrapper": true` so we
  can be quickly pick out such aggregations when debugging.

It also fixes a bug where `_count` breakdown entries were contributing
to the overall `time_in_nanos`. They didn't add a large amount of time
so it is unlikely that this caused a big problem, but I was there.

To support the arbitrary breakdown data this reworks the profiler so
that the `breakdown` can contain any data that is supported by
`StreamOutput#writeGenericValue(Object)` and
`XContentBuilder#value(Object)`.
2020-05-13 16:33:22 -04:00
Bogdan Pintea ee437bef27
Docs: forward port release docs of 7.7.0 (#56706)
Forward port the release docs of 7.7.0: breaking changes, release notes,
release highlights.
2020-05-13 20:08:14 +02:00
Julie Tibshirani a92d138c77 Correct the type of the 'analyzer' parameter in the _analyze docs. (#56650)
This optional parameter can only be a string. To test out a transient custom
analysis chain, users are expected to use the 'tokenizer', 'filter', and
'char_filter' parameters.
2020-05-13 11:05:06 -07:00
David Turner 26382dff19 Clarify doc count stats (#56665)
Today we report some statistics in terms of Lucene-level documents, which
differ from Elasticsearch-level documents in a number of ways and include
things like document tombstones which users cannot directly observe. This
commit clarifies the internal nature of these statistics.

Closes #56497
2020-05-13 15:07:44 +01:00
James Rodewig c859fafcbd [DOCS] Correct `query` datatype in enrich policy definition (#56224)
Corrects the datatype for the `query` property of an enrich policy
object. The `query` property is a query object, not a string.
2020-05-13 08:35:17 -04:00
Marios Trivyzas e781193cf9
SQL: Fix JDBC url pattern in docs and error message (#56612)
The docs pattern url was using `*` which means zero or many instead
of `?` which means zero or one. The pattern url returned in error
messages was not in sync with the one in the docs.

Fixes: #56476
(cherry picked from commit 1a5945c3962cdda21482f4b0b3e0ca508534c2c4)
2020-05-13 12:13:58 +02:00
Gabriel Petrovay ca586f2a8d [Docs] Correct formatting in datehistogram-aggregation.asciidoc (#56664) 2020-05-13 12:01:42 +02:00
debadair 6de6ec68f2
[DOCS] Extract the cron docs from Watcher docs and add to the API conventions. (#56313) (#56651)
* [DOCS] Promote cron expressions info from Watcher to a separate topic.

* Fix table error

* Fixed xref

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Incorporated review feedback

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-12 16:36:18 -07:00
CJ Cenizal 2704428a9c [Docs] Clarify that _ccr/info omits parameters from the response when the follower index is paused. (#55961) 2020-05-12 15:25:08 -07:00
debadair cabc963135
[DOCS] Add request body param descriptions for move to step (#56560) (#56644)
* [DOCS] Add request body param descriptions for move to step (#56560)

* [DOCS] Clarify definition of max_size (#56561)
2020-05-12 15:00:48 -07:00
James Rodewig a5154cc190 [DOCS] Correct setting type for `indices.query.bool.max_clause_count` (#56640)
#56449 incorrectly labelled this as a dynamic setting.

This corrects that error.
2020-05-12 16:26:18 -04:00
Jake Landis 5f5a648b9a
[7.x] remove the term 'system' from indicies doc (#56367) (#56375)
'system' indices will carry special meaning in the future this commit
removes the system from the name to avoid confusion. (technically
these indices will be hidden not system)
2020-05-12 14:44:50 -05:00
James Rodewig 54088a21d7
[DOCS] Collapse 7.8 breaking changes for security (#56622) 2020-05-12 15:30:28 -04:00
James Rodewig d247e8f7a6 [DOCS] Sort EQL search API params alphabetically 2020-05-12 13:52:18 -04:00
Andrei Stefan f0074e93a0
QL: case sensitive support in EQL (#56404) (#56597)
* QL: case sensitive support in EQL (#56404)
* adds a generic startsWith function to QL
* modifies the existent EQL startsWith function to be case sensitive
aware
* improves the existent EQL startsWith function to use a prefix query
when the function is used in a case sensitive context. Same improvement
is used in SQL's newly added STARTS_WITH function.
* adds case sensitivity to EQL configuration through a case_sensitive
parameter in the eql request, as established in #54411.
The case_sensitive parameter can be specified when running queries
(default is case insensitive)

(cherry picked from commit ee5a09ea840167566e34c28c8225dc38bc6a7ae8)
2020-05-12 16:56:18 +03:00
James Rodewig 8c457c884a
[DOCS] Add clean up snapshot repository API docs (#56519) 2020-05-12 09:54:49 -04:00
Ignacio Vera 222ee721ec
Add moving percentiles pipeline aggregation (#55441) (#56575)
Similar to what the moving function aggregation does, except merging windows of percentiles
sketches together instead of cumulatively merging final metrics
2020-05-12 11:35:23 +02:00
debadair 8f2c1cda2e
[DOCS] Added info about refresh interval & max_docs and step execution. Closes #49151. (#56315) (#56554) 2020-05-11 15:10:57 -07:00
Marc e0e7b89499 [DOCS] Add document update API link to concurrency control docs (#56481)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-11 17:37:32 -04:00
Lee Hinman 1337b35572
Remove prefer_v2_templates query string parameter (#56545)
This commit removes the `prefer_v2_templates` flag and setting. This was a brief setting that
allowed specifying whether V1 or V2 template should be used when an index is created. It has been
removed in favor of V2 templates always having priority.

Relates to #53101
Resolves #56528

This is not a breaking change because this flag was never in a released version.
2020-05-11 14:56:42 -06:00
James Rodewig 2be6d7b8b6
[DOCS] Relocate request body param docs to search API docs (#56436)
Moves documentation for the following request body parameters to the
search API reference docs:

* `explain`
* `query`
* `seq_no_primary_term`
* `version`

Removes documentation for these parameters from the Request body search
page[0].

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-body.html
2020-05-11 11:29:38 -04:00
Lisa Cawley 7ce0a25fbc [DOCS] Add UUID troubleshooting tip for stack monitoring (#55744) 2020-05-11 07:49:34 -07:00
Lisa Cawley 1474606b18 [DOCS] Clarify model snapshot retention properties (#56477) 2020-05-11 07:43:10 -07:00
James Rodewig ba67ab3b64
[DOCS] Add reference docs for `search.max_buckets` setting (#56449) (#56511)
Adds reference-style setting documentation for the `search.max_buckets`
setting.

This setting was previously only documented on the [bucket
aggregations][0] page.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-bucket.html
2020-05-11 09:45:09 -04:00
István Zoltán Szabó ebe1e4c4c4 [DOCS] Expands GET DFA stats API docs with new phases (#56407)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-11 09:26:15 +02:00
Hendrik Muhs c0985615aa [DOC] document transform settings and docs_per_second (#56178)
add documentation for throttling, added in #56007
2020-05-11 09:23:49 +02:00
debadair 6ae7327061
[DOCS] Align with ILM changes. (#55953) (#56455)
* [DOCS] Align with ILM changes.

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Incorporated review comments.
2020-05-08 14:22:27 -07:00
Jake Landis 95e5e9e598
[7.x] [DOCS] Update default value of index.name.time_format (#56453) (#56456)
Corrects the default value of index.name.time_format
2020-05-08 16:09:42 -05:00
Théophile Helleboid - chtitux 23e419a7aa SLM PUT: add precision on date math support in indices parameter (#55203)
It was not clear for me that `indices` parameter supports date math expression.

I think it may be worth to add the precision in the documentation.
2020-05-08 15:06:09 -06:00
Nicole Albee 5b708f846c
[DOCS] Fix broken link in the ilm-tutorial. (#56310) (#56311) (#56446) 2020-05-08 12:22:39 -05:00
István Zoltán Szabó ceb0b0dba3 [DOCS] Updated screenshots in transform ecommerce example. (#56359) 2020-05-08 07:46:15 +02:00
James Rodewig ea76b0c22b
[DOCS] Relocate search API's request body parameters (#56304)
Changes:
* Moves the document request body parameters for the search API
  from the Request body search page to the Search API reference page.

* Relocates a search request body example from the Request body search
  page to the Search API reference page.

* Adds a note to any duplicated query and request body parameters.
2020-05-07 11:00:03 -04:00
William Brafford 691044e67b
Add xpack setting deprecations to deprecation API (#56290)
* Add xpack setting deprecations to deprecation API

The deprecated settings showed up in the deprecation log file by
default, but I did not add them to the deprecation API. This commit
fixes that. Now if you use one of the deprecated basic feature
enablement settings, calling _monitoring/deprecations will inform you of
that fact.

* Remove incorrectly backported settings documents

It seems that I backported these docs to the wrong place in #56061,
in #55980, and in #56167. I hope they're in the right place now.

Co-authored-by: debadair <debadair@elastic.co>
2020-05-07 10:28:17 -04:00
James Rodewig 8e005db3e6
[DOCS] EQL: Document math functions (#55810) (#56337)
Documents the following EQL functions:

* `add`
* `divide`
* `module`
* `multiply`
* `subtract`
2020-05-07 09:18:43 -04:00
Paweł Krześniak e6dce13bda DOCS: minor formatting (#56263)
Removed extra back ticks. Please cherry-pick to other branches.
2020-05-06 13:41:47 -04:00
James Rodewig d29ea0b8ee
[DOCS] Update upgrade docs for 7.8 (#56281) 2020-05-06 11:47:39 -04:00
James Rodewig 2173a587af [DOCS] Move async search API docs after search request body API docs (#56244) 2020-05-06 09:03:23 -04:00
Luca Cavanna ef66018119 [DOCS] Async search: clarify behaviour when submit returns final results (#55934)
Closes #55636
2020-05-06 10:01:23 +02:00
James Rodewig 8686200a32 [DOCS] EQL: Document `concat` function (#56239)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-05 16:45:29 -04:00
James Rodewig dac4ed282e [DOCS] EQL: Add collapsible sections to EQL tutorial docs (#56235)
Adds collapsible sections to the snippet examples of the EQL tutorial
docs.

Also adds a leading slash to EQL API snippet examples.
2020-05-05 16:29:51 -04:00
James Rodewig e7df8b388e [DOCS] EQL: Add collapsible sections to EQL search API response (#56232)
Add collapsible sections to the response parameter docs
of the EQL search API.

Also clarifies some language regarding documents and
events.
2020-05-05 16:01:55 -04:00
Thiago Souza 1feb0a95b5 [DOCS] Correct get enrich policy API request example (#56207) 2020-05-05 12:37:54 -04:00
James Rodewig cd3663e5fa
[DOCS] EQL: Document `match` function (#56134) 2020-05-05 12:03:02 -04:00
James Rodewig 44414acd3b
[DOCS] EQL: Document nested field support (#56138)
Notes that you cannot use EQL in ES to search the values of `nested`
fields or their sub-fields. However, indices containing `nested` field
mappings are otherwise supported.
2020-05-05 11:46:06 -04:00
James Rodewig a528319827
[DOCS] Remove invalid search API query parameters (#55884) (#56212)
This is a backport of #55884 with redirects removed.

Changes:

* Adds an abbreviated title for the search API page.
* Removes the following invalid query parameters:
  * `analyzer`
  * `analyze_wildcard`
  * `default_operator`
  * `df`
  * `lenient`
  * `suggest_mode`
  * `suggest_size`
* Replaces the URI search page's query parameter docs with a xref
* Updates the headings of several examples
2020-05-05 11:10:34 -04:00
William Brafford 3499fa917c
Deprecated xpack "enable" settings should be no-ops (#55416) (#56167)
The following settings are now no-ops:

* xpack.flattened.enabled
* xpack.logstash.enabled
* xpack.rollup.enabled
* xpack.slm.enabled
* xpack.sql.enabled
* xpack.transform.enabled
* xpack.vectors.enabled

Since these settings no longer need to be checked, we can remove settings
parameters from a number of constructors and methods, and do so in this
commit.

We also update documentation to remove references to these settings.
2020-05-05 10:40:49 -04:00
Rory Hunter fd91617b19 Fix typo in docs (#56190)
Some dashes were swapped for question marks in bf4c364. Swap them back.
2020-05-05 15:28:32 +01:00
James Rodewig cc1af50214
[DOCS] Add tip for upgrade assistant (#55027)
Our upgrade docs don't mention the upgrade assistant, which can be
helpful when migrating across major versions. The docs also don't
mention deprecation logs, which can highlight other functionality that
may change.

This adds a related tip admonition to the upgrade docs.
2020-05-05 10:09:41 -04:00
David Roberts 7aa0daaabd
[7.x][ML] More advanced model snapshot retention options (#56194)
This PR implements the following changes to make ML model snapshot
retention more flexible in advance of adding a UI for the feature in
an upcoming release.

- The default for `model_snapshot_retention_days` for new jobs is now
  10 instead of 1
- There is a new job setting, `daily_model_snapshot_retention_after_days`,
  that defaults to 1 for new jobs and `model_snapshot_retention_days`
  for pre-7.8 jobs
- For days that are older than `model_snapshot_retention_days`, all
  model snapshots are deleted as before
- For days that are in between `daily_model_snapshot_retention_after_days`
  and `model_snapshot_retention_days` all but the first model snapshot
  for that day are deleted
- The `retain` setting of model snapshots is still respected to allow
  selected model snapshots to be retained indefinitely

Backport of #56125
2020-05-05 14:31:58 +01:00
James Rodewig 4dfdd46dc3 [DOCS] EQL: Remove case sensitivity from function docs (#55063)
Per #54411, we plan to handle case sensitivity via a parameter for the
EQL search API (with the possible exception of the `between` function).

This removes references and examples related to case sensitivity from
the EQL functions docs.
2020-05-05 09:26:49 -04:00
James Rodewig d04ac8b496 [DOCS] Correct `refresh` parm def for GET API (#56140) 2020-05-05 09:13:49 -04:00
James Rodewig bd17d098c4
[DOCS] Remove duplicate `index alias` link in glossary (#56143) 2020-05-05 09:07:19 -04:00
Dimitris Athanasiou 75dadb7a6d
[7.x][ML] Add loss_function to regression (#56118) (#56187)
Adds parameters `loss_function` and `loss_function_parameter`
to regression.

Backport of #56118
2020-05-05 14:59:51 +03:00
Hendrik Muhs e177a38504
[7.x][Transform] add throttling (#56007) (#56184)
add throttling to transform, throttling will slow down search requests by
delaying the execution based on a documents per second metric.

fixes #54862
2020-05-05 13:09:02 +02:00
Marios Trivyzas 363e994171
SQL: Fix DATETIME_PARSE behaviour regarding timezones (#56158) (#56182)
Previously, when the timezone was missing from the datetime string
and the pattern, UTC was used, instead of the session defined timezone.
Moreover, if a timezone was included in the datetime string and the
pattern then this timezone was used. To have a consistent behaviour
the resulting datetime will always be converted to the session defined
timezone, e.g.:
```
SELECT DATETIME_PARSE('2020-05-04 10:20:30.123 +02:00', 'HH:mm:ss dd/MM/uuuu VV') AS datetime;
```
with `time_zone` set to `-03:00` will result in
```
2020-05-04T05:20:40.123-03:00
```

Follows: #54960
(cherry picked from commit 8810ed03a209cc8fe1bad309a81e85b56a39da27)
2020-05-05 12:08:39 +02:00
Théophile Helleboid - chtitux 8a23da429a Docs fix node_id spec for secure settings reload API (#55712)
Fix docs typo for the `node_id` parameter in the secure settings reload API.
2020-05-05 11:21:02 +03:00
István Zoltán Szabó 9bcc975bd1 [DOCS] Simplifies footnote text in DFA APIs (#56105)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-05 09:05:08 +02:00
Lee Hinman 8fa14b333d
[7.x] Validate non-negative priorities for V2 index templates (#56139) (#56163)
Backports the following commits to 7.x:
 - Validate non-negative priorities for V2 index templates (#56139)
2020-05-04 16:19:13 -06:00
James Rodewig 922a80c3f4
[DOCS] Add collapsible sections to search API response (#55887) 2020-05-04 16:57:10 -04:00
Dan Hermann 9892813842
[7.x] Delay warning about missing x-pack (#56142)
* Delay warning about missing x-pack (#54265)

Currently, when monitoring is enabled in a freshly-installed cluster,
the non-master nodes log a warning message indicating that master may
not have x-pack installed. The message is often printed even when the
master does have x-pack installed but takes some time to setup the local
exporter for monitoring. This commit adds the local exporter setting
`wait_master.timeout` which defaults to 30 seconds. The setting
configures the time that the non-master nodes should wait for master to
setup monitoring. After the time elapses, they log a message to the user
about possible missing x-pack installation on master.

The logging of this warning was moved from `resolveBulk()` to
`openBulk()` since `resolveBulk()` is called only on cluster updates and
the message might not be logged until a new cluster update occurs.

Closes #40898
2020-05-04 14:16:18 -05:00
Lisa Cawley b816ab0c18
[DOCS] Synchs and links hyperparameter descriptions (#56131) 2020-05-04 10:37:26 -07:00
Julie Tibshirani 6b5cf1b031 For constant_keyword, make sure exists query handles missing values. (#55757)
It's possible for a constant_keyword to have a 'null' value before any documents
are seen that contain a value for the field. In this case, no documents have a
value for the field, and 'exists' queries should return no documents.
2020-05-04 09:41:52 -07:00
James Rodewig 4faf5a7916
[DOCS] Reformat `porter_stem` token filter (#56053)
Makes the following changes to the `porter_stem` token filter docs:

* Rewrites description and adds a Lucene link
* Adds detailed analyze example
* Adds an analyzer example
2020-05-04 10:39:17 -04:00
bellengao 722de7dd98 [Docs] Fix typo in match-bool-prefix-query doc (#56077) 2020-05-04 14:19:23 +02:00
bellengao 40f99119ae [Docs] Fix typo in getting-started-slm doc (#56075) 2020-05-04 14:18:00 +02:00
markharwood e197b6c45b
Analysis enhancement - add preserve_original setting in ngram-token-filter (#55432) (#56100)
Authored-by: Amit Khandelwal <amitmbm87@gmail.com>
2020-05-04 11:31:28 +01:00
Christos Soulios c65f828cb7
[7.x] Histogram field type support for ValueCount and Avg aggregations (#56099)
Backports #55933 to 7.x

Implements value_count and avg aggregations over Histogram fields as discussed in #53285

- value_count returns the sum of all counts array of the histograms
- avg computes a weighted average of the values array of the histogram by multiplying each value with its associated element in the counts array
2020-05-04 13:23:02 +03:00
Armin Braun 3a64ecb6bf
Allow Deleting Multiple Snapshots at Once (#55474) (#56083)
* Allow Deleting Multiple Snapshots at Once (#55474)

Adds deleting multiple snapshots in one go without significantly changing the mechanics of snapshot deletes otherwise.
This change does not yet allow mixing snapshot delete and abort. Abort is still only allowed for a single snapshot delete by exact name.
2020-05-03 20:30:58 +02:00
William Brafford d53c941c41
Make xpack.monitoring.enabled setting a no-op (#55617) (#56061)
* Make xpack.monitoring.enabled setting a no-op

This commit turns xpack.monitoring.enabled into a no-op. Mostly, this involved
removing the setting from the setup for integration tests. Monitoring may
introduce some complexity for test setup and teardown, so we should keep an eye
out for turbulence and failures

* Docs for making deprecated setting a no-op
2020-05-01 16:42:11 -04:00
David Turner 69f50fe79f Improve same-shard allocation explanations (#56010)
I see occasional confusion about the explanations emitted by the same-shard
allocation decider, particularly amongst new users setting up a single-node
cluster and trying to determine why their cluster has `yellow` health. For
example:

    the shard cannot be allocated to the same node on which a copy of the shard
    already exists

This is technically correct but it's quite a complicated sentence. Also, by
starting with "the shard cannot be allocated" it makes it sound like this is
the problem, whereas in fact this message is a good thing and users should
typically focus their attention elsewhere.

This commit simplifies the wording of these messages and makes them sound more
positive, for example:

    a copy of this shard is already allocated to this node
2020-05-01 10:07:14 +01:00
Vanessa Bell 9b005f70ef [Docs] Grammar improvements in disk-usage.asciidoc (#55620) 2020-04-30 19:31:56 +02:00
Karel Minarik 81dcc9aef7 [DOCS] Add link to the Go client bulk helper (#55872) 2020-04-30 13:24:00 -04:00
James Rodewig 61cf646f17
[DOCS] EQL: Add advantages to overview (#53452) (#56052)
Adds a concise list of EQL advantages, based on the "EQL Advantages"
section in the [EQL for the masses][0] blog post.

The intent is to inform users how EQL could benefit at a high level.

[0]: https://www.elastic.co/blog/eql-for-the-masses

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-04-30 13:19:31 -04:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API (#55732) (#56048)
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746
2020-04-30 12:53:36 -04:00
James Rodewig e4e02e133e
[DOCS] Remove approximate document counts example from term agg docs (#55442)
Removes an example from the "Document counts are approximate" section of the
terms agg documentation.

As #52377 details, the example was no longer accurate in 7.x or 6.8. Document
counts were more precise than the example presented.

We've opened issue #56025 to discuss re-adding an example later.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: AB Prashanth <panuradh@buffalo.edu>
2020-04-30 10:11:50 -04:00
William Brafford 273ff6a105
Make xpack.ilm.enabled setting a no-op (#55592) (#55980)
* Make xpack.ilm.enabled setting a no-op

* Add watcher setting to not use ILM

* Update documentation for no-op setting

* Remove NO_ILM ml index templates

* Remove unneeded setting from test setup

* Inline variable definitions for ML templates

* Use identical parameter names in templates

* New ILM/watcher setting falls back to old setting

* Add fallback unit test for watcher/ilm setting
2020-04-30 09:50:18 -04:00
Lisa Cawley 006e00ed0a
[DOCS] Adds documentation for secondary authorization headers (#55365) (#55986) 2020-04-29 16:29:38 -07:00
James Rodewig 65b47d20a6 [DOCS] Update attribute for multi arg footnotes (#55860) 2020-04-29 10:25:36 -04:00
James Rodewig 1808a1f36b [DOCS] EQL: Correct `cidrMatch` function heading (#55935) 2020-04-29 10:02:06 -04:00
István Zoltán Szabó 337dc45f5b [DOCS] Adds missing space and a relevant link to the slm execute API page (#55917)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-29 15:50:06 +02:00
James Rodewig bbf68de446 [DOCS] Correct Lucene link in `kstem` token filter docs 2020-04-29 09:30:37 -04:00
Luca Cavanna 8b05027bf0 [DOCS] Clarify async search response flags (#55574)
Relates to #55572
2020-04-29 15:22:05 +02:00
James Rodewig 767836c367
[DOCS] Reformat `kstem` token filter (#55823)
Makes the following changes to the `kstem` token filter docs:

* Rewrite description and adds a Lucene work
* Adds detailed analyze example
* Adds an analyzer example
2020-04-29 08:52:55 -04:00
Christos Soulios 02bf0c586a
[7.x] Histogram field type support for Sum aggregation (#55916)
Implements Sum aggregation over Histogram fields by summing the value of each bucket multiplied by their count as requested in #53285

Backports #55681 to 7.x
2020-04-29 15:06:12 +03:00
Henning Andersen f679880b80 [DOCS] Create index name required (#55886)
The name of the new index to create is required.

Relates #45749
2020-04-29 13:35:49 +02:00
István Zoltán Szabó e982cf4381 [DOCS] Makes the footnotes less verbose in configuring aggs page. (#55857) 2020-04-29 09:52:29 +02:00
debadair 8a662c7e62
ILM update backports (#55902)
* [DOCS] Rework conceptual info for ILM. (#52181)

* [DOCS] Rework conceptual info for ILM.

* Split the actions out of concepts.

* Added xpack role to actions.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review

* Edit actions for consistency and add action template. (#55632)

* Edit actions for consistency and add action template.

* Update docs/reference/ilm/actions/ilm-readonly.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review
2020-04-28 16:38:01 -07:00
Lee Hinman 4315a55a1c
[7.x] Initial documentation for index templates V2 (#55755) (#55898)
Backports the following commits to 7.x:
 - Initial documentation for index templates V2 (#55755)
2020-04-28 16:10:50 -06:00
Henning Andersen cab7bcc156
Disk decider respect watermarks for single data node (#55805) (#55847)
The disk decider had special handling for the single data node case,
allowing any allocation (skipping watermark checks) for such clusters.
This special handling can now be avoided via a setting.
2020-04-28 18:46:22 +02:00
Lee Hinman 777caf0725
[7.x] Add support for V2 index templates to /_cat/templates (#55829) (#55866)
Backports the following commits to 7.x:
 - Add support for V2 index templates to /_cat/templates (#55829)
2020-04-28 10:14:19 -06:00
Larry Gregory 47d252424b
Backport: Deprecate the kibana reserved user (#54967) (#55822) 2020-04-28 10:30:25 -04:00
James Rodewig ddc7305ac9 [DOCS] Correct search API's timeout parm default (#55855) 2020-04-28 09:44:50 -04:00
James Rodewig 386fb16409 [DOCS] SQL: Update link for supported regex in `RLIKE` docs (#55830)
The`RLIKE` function docs points users to [Java’s Pattern class doc][0]
for regular expression syntax. However, these docs include shorthand
character classes, such as `[\d]`, `[\s]`, and `[\w]`. These character
classes are not supported in Elasticsearch, which may confuse users.

This updates the SQL `RLIKE` docs to refer to the ES [regular expression
syntax docs][1], which only documents supported syntax.

[0]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/regexp-syntax.html

Relates to #55231
2020-04-28 09:25:51 -04:00
James Rodewig 452be22a4d [DOCS] Warn about searching across all fields wt. `query_string` (#55853)
Warn about potential performance impact when a large number of fields
is used with query string query and no default field.

Re-adds content from #35570.
That content was erroneously removed in #45296.

Co-authored-by: Peter Dyson <peter.dyson@geekpete.com>
2020-04-28 09:20:21 -04:00
Adrien Grand 58c3bb5ae1
Repurpose `ignore_throttled` to be only about frozen indices. (#55047) (#55852)
This has no practical impact on users since frozen indices are the only
throttled indices today. However this has an impact on upcoming features
that would use search throttling.

Filtering out throttled indices made sense a couple years ago, but as
we're now improving support for slow requests with `_async_search` and
exploring ways to reduce storage costs, this feature has most likely
become a trap, that we'd like to not have with upcoming features that
would use search throttling.

Relates #54058
2020-04-28 14:31:54 +02:00
Amit Khandelwal 126e4acca8 Expose `preserve_original` in `edge_ngram` token filter (#55766)
The Lucene `preserve_original` setting is currently not supported in the `edge_ngram`
token filter. This change adds it with a default value of `false`.

Closes #55767
2020-04-28 10:24:27 +02:00
István Zoltán Szabó a5cf4712e5 [DOCS] Changes feature importance links to point to the new page (#55531)
* [DOCS] Changes feature importance links to point to the new page.

* [DOCS] Fixes line breaks.
2020-04-28 09:03:43 +02:00
James Rodewig c16b1edae0 [DOCS] EQL: Fix whitespace in `stringContains` docs 2020-04-27 15:53:59 -04:00
James Rodewig 8df5cff9c1 [DOCS] Correct stemmer token filters anchor 2020-04-27 14:57:59 -04:00
James Rodewig 5b8a18c756 [DOCS] Correct stemmer token filter anchor 2020-04-27 14:51:51 -04:00
David Roberts 3ba44a5af8
[ML] Adding failed_category_count to model_size_stats (#55761)
The failed_category_count statistic records the number of times
categorization wanted to create a new category but couldn't
because the job had reached its model_memory_limit.

Backport of #55716
2020-04-25 10:36:49 +01:00
James Rodewig c1b0548db0
[DOCS] Document EQL search REST API (#52384) 2020-04-24 15:36:01 -04:00
James Rodewig 5981412bf7
[DOCS] EQL: Document `stringContains` function (#54968) 2020-04-24 15:09:05 -04:00
James Rodewig e4ebe55d04
[DOCS] EQL: Document `cidrMatch` function (#54216) (#55739) 2020-04-24 14:01:11 -04:00
James Rodewig e0a8adb5b2
[DOCS] Reformat `stemmer` token filter (#55693)
Makes the following changes to the `stemmer` token filter docs:

* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer example
* Adds a `language` value for the `estonian` stemmer
* Reorders the `language` values to show recommended algorithms first,
  followed by other values alphabetically
2020-04-24 11:25:01 -04:00
James Rodewig 96285b90c1
[DOCS] Add stemming concept docs (#55156)
Adds conceptual documentation for stemming, including:

* An overview of why stemming is helpful in search
* Algorithmic vs. dictionary stemming
* Token filters used to control stemming, such as `stemmer_override`, `keyword_marker`, and `conditional`
2020-04-24 11:01:28 -04:00
Christoph Büscher f95a741ad3
[Docs] Fix fuzziness example in match-query.asciidoc (#55715)
The example looks the same as in the previous section although it should use the
"fuzziness" parameter. This seems to be okay on 6.8 and master and was probably
only forgotten to port to 7.x branches.
2020-04-24 16:21:40 +02:00
Zachary Tong 715c90bf7d Aggs must specify a `field` or `script` (or both) (#52226)
This adds a validation to VSParserHelper to ensure that a field or
script or both are specified by the user.  This is technically
required today already, but throws an exception much deeper
in the agg framework and has a very unintuitive error for the user
(as well as eating more resources instead of failing early)
2020-04-23 19:23:41 -04:00
James Rodewig e74fdacabd
[DOCS] Add admonition for EQL exact matches on text fields (#53402) (#55670)
Adds a important admonition to the EQL syntax page noting that
the equal (`==`) operator should not be used to match `text` field
values.

Relates to #52709 and #53020
2020-04-23 10:59:50 -04:00
István Zoltán Szabó 5813dfdcc7
[7.x][DOCS] Adds ML related items to release highlights (#55652) 2020-04-23 11:58:32 +02:00
Lisa Cawley 314ca78e31
[7.x][DOCS] Update example and nesting in get data frame analytics job stats API (#55612) 2020-04-22 10:58:26 -07:00
James Rodewig 8d05d7dace
[DOCS] Add collapsible sections to 7.x breaking changes (#55334)
Adds collapsible sections and new format to the 7.x breaking changes.

Relates to #53229.
2020-04-22 10:56:38 -04:00
James Rodewig 6f9513915d
[DOCS] Add 'how to' doc about avoiding oversharding (#55480)
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
2020-04-22 10:44:16 -04:00
James Rodewig 414f9c98f3
[DOCS] Document missing bulk API response parameters (#55414)
Documents several parameters missing from the bulk API's response body
docs. Also moves several response-related chunks of text to the response
body section.

Relates to #55237
2020-04-22 09:48:03 -04:00
David Roberts 2dc5586afe
[ML] Add effective max model memory limit to ML info (#55581)
The ML info endpoint returns the max_model_memory_limit setting
if one is configured.  However, it is still possible to create
a job that cannot run anywhere in the current cluster because
no node in the cluster has enough memory to accommodate it.

This change adds an extra piece of information,
limits.effective_max_model_memory_limit, to the ML info
response that returns the biggest model memory limit that could
be run in the current cluster assuming no other jobs were
running.

The idea is that the ML UI will be able to warn users who try to
create jobs with higher model memory limits that their jobs will
not be able to start unless they add a bigger ML node to their
cluster.

Backport of #55529
2020-04-22 12:28:50 +01:00
David Roberts da5aeb8be7
[ML] Return assigned node in start/open job/datafeed response (#55570)
Adds a "node" field to the response from the following endpoints:

1. Open anomaly detection job
2. Start datafeed
3. Start data frame analytics job

If the job or datafeed is assigned to a node immediately then
this field will return the ID of that node.

In the case where a job or datafeed is opened or started lazily
the node field will contain an empty string.  Clients that want
to test whether a job or datafeed was opened or started lazily
can therefore check for this.

Backport of #55473
2020-04-22 12:06:53 +01:00
István Zoltán Szabó 0ce3406033 [DOCS] Provides further details on aggregations in datafeeds (#55462)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-22 08:54:52 +02:00
James Rodewig 777ffd5801
[DOCS] Add bulk API example with failures (#55412)
Adds an example for bulk API requests that include failures.
Also documents guidance on use the `filter_path` parameter
to narrow the bulk API response for errors.

Closes #55237
2020-04-21 16:22:23 -04:00
James Baiera 2a5f1f49a9
Add enrich metricset from 7.5 (#54791) (#55356)
Co-authored-by: Julien Guay <guay_j@yahoo.fr>
2020-04-21 12:39:08 -04:00
James Rodewig b9dfd12e7e
[DOCS] Remove 'Testing' chapter (#55270) (#55532)
Removes the 'Testing' chapter from the Elasticsearch Reference guide.

This chapter was originally written for so that users using the Java HLRC client could
use the same test classes when testing Elasticsearch in their own applications.
However, this is no longer the case or recommended.

Closes #55257.
2020-04-21 10:29:58 -04:00
Paul Sanwald 0f7917b94b
add release notes for 7.5.2 (#51259)
Adds release notes for 7.5.2
2020-04-21 08:19:46 -04:00
Benjamin Trent 24d41eb695
[ML] partitions model definitions into chunks (#55260) (#55484)
This paves the data layer way so that exceptionally large models are partitioned across multiple documents.

This change means that nodes before 7.8.0 will not be able to use trained inference models created on nodes on or after 7.8.0.

I chose the definition document limit to be 100. This *SHOULD* be plenty for any large model. One of the largest models that I have created so far had the following stats:
~314MB of inflated JSON, ~66MB when compressed, ~177MB of heap.
With the chunking sizes of `16 * 1024 * 1024` its compressed string could be partitioned to 5 documents.
Supporting models 20 times this size (compressed) seems adequate for now.
2020-04-20 16:08:54 -04:00
David Turner 8e618fdf10 Adjust docs for voting config exclusions API (#55006)
In #50836 we deprecated the existing voting config exclusions API and added a
new one. This commit adjust the docs to match.
2020-04-20 19:47:33 +01:00
Lee Hinman 9eddd2bcc9
[7.x] Add prefer_v2_templates flag and index setting (#55411) (#55476)
This commit adds a new querystring parameter on the following APIs:
- Index
- Update
- Bulk
- Create Index
- Rollover

These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference
creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be
changed to `true` for 8.0+ in subsequent work.

Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting.
This setting is used so that actions that automatically create a new index (things like rollover
initiated by ILM) will inherit the preference from the original index. This setting is dynamic so
that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias
performing periodic rollover.

This also adds support for sending this parameter to the High Level Rest Client.

Relates to #53101
2020-04-20 12:05:42 -06:00
jmceniery 99409e8c95 [DOCS] Remove Wikipedia link from `SUM_OF_SQUARES` SQL function docs (#52398)
Removed the link to Wikipedia as the function is not calculating the sum of squares in this way. More can be found here at this issue:

https://github.com/elastic/elasticsearch/issues/50416
2020-04-20 09:59:59 -04:00
Ben Skelker 74f55ec6fa [DOCS] Add `ip_range` datatype to core datatypes range list (#55446) 2020-04-20 08:55:09 -04:00
William Brafford 49e30b15a2
Deprecate disabling basic-license features (#54816) (#55405)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 15:04:17 -04:00
Andrei Dan ef338ee3d4
ILM DOCS: mention forcemerge is best effort (#54794) (#55401)
(cherry picked from commit 3fd05435c52dd265dbe1a40104e7dc7a335d50ae)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-17 15:42:23 +01:00
James Rodewig f87a3f0c48 [DOCS] Document analysis/mapping response for cluster stats API (#55054)
PR #51260 moved usage counts about mapping field types and analysis to
the `_cluster/stats` API.

This documents those stats in the response section of the cluster stats
API docs.
2020-04-17 08:44:10 -04:00
Adrien Grand 0cb6a1f089
Document the index corruption bug that gets fixed via Lucene 8.5.1. (#55232)
Using soft deletes on shrunk indices may cause corruption.
2020-04-17 13:37:37 +02:00
markharwood 7761b01a33
Remove normalizer support from wildcard field while we decide on approach for handling case insensitvity (#55294) (#55375)
Closes #55288
2020-04-17 11:43:26 +01:00
Marios Trivyzas f958e9abdc
SQL: Implement scripting inside aggs (#55241) (#55371)
Implement the use of scalar functions inside aggregate functions.
This allows for complex expressions inside aggregations, with or without
GROUBY as well as with or without a HAVING clause. e.g.:

```
SELECT MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) AS max, b
FROM test
GROUP BY b
HAVING MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) > 5
```

Scalar functions are still not allowed for `KURTOSIS` and `SKEWNESS` as
this is currently not implemented on the ElasticSearch side.

Fixes: #29980
Fixes: #36865
Fixes: #37271

(cherry picked from commit 506d1beea7abb2b45de793bba2e349090a78f2f9)
2020-04-17 12:41:22 +02:00
Lisa Cawley c7cf6e621d [DOCS] Remove text fields from classification dependent variables (#54849) 2020-04-16 13:40:28 -07:00
Lisa Cawley cf5278f771 [DOCS] Add ml-cpp PRs to 7.7 release notes (#55264)
Co-Authored-By: David Roberts <dave.roberts@elastic.co>
2020-04-16 11:28:34 -07:00
Julie Tibshirani d7cded8d7a
Fix updating include_in_parent/include_in_root of nested field. (#55326)
The main changes are:
1. Throw an error when updating `include_in_parent` or `include_in_root` attribute of nested field dynamically by the PUT mapping API.
2. Add a test for the change.

Closes #53792

Co-authored-by: bellengao <gbl_long@163.com>
2020-04-16 11:17:12 -07:00
James Rodewig f0b9be8b1b [DOCS] Reformat `flatten_graph` token filter (#54268)
* [DOCS] Reformat `flatten_graph` token filter

Makes the following changes to the `flatten_graph` token filter docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Adds analyzer example
2020-04-16 08:35:08 -04:00
Bogdan Pintea b88dd47de3 Docs: add the change log for 7.7 (#55019)
* Add the change log for 7.7

Add the change log for 7.7

* Update rel. notes to latest state (BC5)

Update the release notes to current state (i.e. BC5).

* Update docs/reference/release-notes/7.7.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-15 15:25:08 -04:00
Lisa Cawley f0b9578684 [DOCS] Removes transform performance note (#55177) 2020-04-15 10:42:52 -07:00
James Rodewig 4f2ab96f38 [DOCS] EQL: Document `indexOf` function (#55071) 2020-04-15 11:29:50 -04:00
James Rodewig 8d6f0f6a76 [DOCS] Document `max_concurrent_searches` default (#55116) 2020-04-15 10:04:23 -04:00
Benjamin Trent 8ff2cbf1a3
[7.x] [ML] adding prediction_field_type to inference config (#55128) (#55230)
* [ML] adding prediction_field_type to inference config (#55128)

Data frame analytics dynamically determines the classification field type. This field type then dictates the encoded JSON that is written to Elasticsearch. 

Inference needs to know about this field type so that it may provide the EXACT SAME predicted values as analytics. 

Here is added a new field `prediction_field_type` which indicates the desired type. Options are: `string` (DEFAULT), `number`, `boolean` (where close_to(1.0) == true, false otherwise). 

Analytics provides the default `prediction_field_type` when the model is created from the process.
2020-04-15 09:45:22 -04:00
Jake Landis 85139fad7e
[7.x] Advise a simpler curator migration (#54457) (#55188)
Advice for migrating from Curator should simply be to phase out curator managed indices, 
since curator will ignore ILM indices
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.7/ilm-and-curator.html#ilm-and-curator.

Co-authored-by: Jay Greenberg <PhaedrusTheGreek@users.noreply.github.com>
2020-04-15 07:55:31 -05:00
Lisa Cawley 2910d01179
[DOCS] Removes unshared sections from ml-shared.asciidoc (#55192) 2020-04-14 18:47:09 -07:00
Yang Wang f49354b7d7
Add migration notes for deprecating local parameter of get field mapping API (#55194)
This is a follow-up for #55099 to add migration notes about the deprecation of local parameter for get field mappings API.
2020-04-15 11:38:05 +10:00
Igor Motov 1754e50cbd
[7.x] Add analytics plugin usage stats to _xpack/usage (#54911) (#55162)
Adds analytics plugin usage stats to _xpack/usage.

Closes #54847
2020-04-14 17:03:14 -04:00
James Rodewig 12130843ca
[DOCS] Add maintenance releases to upgrade table (#55012)
Updates the supported upgrade path table in [Upgrade Elasticsearch][0]
to include a new row for maintenance releases. For example, this row
covers upgrading from 7.6.0 to 7.6.2.

The new table row only displays for releases greater than n.x.0. For
example, the new row will display for the 7.7.1 release but not the
7.7.0 release.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-upgrade.html
2020-04-14 11:28:55 -04:00
James Rodewig 3fbd8b371f [DOCS] Use consistent line breaks in EQL function docs 2020-04-14 10:17:45 -04:00
Yannick Welsch a610513ec7 Provide repository-level stats for searchable snapshots (#55051)
Provides basic repository-level stats that will allow us to get some insight into how many
requests are actually being made by the underlying SDK. Currently only tracks GET and LIST
calls for S3 repositories. Most of the code is unfortunately boiler plate to add a new endpoint
that will help us better understand some of the low-level dynamics of searchable snapshots.
2020-04-14 14:34:08 +02:00
lcawl fcd96db006 [DOCS] Edits create data frame analytics job API (#54751) 2020-04-13 10:43:52 -07:00
Nhat Nguyen 96bb1164f0 Support hierarchical task cancellation (#54757)
With this change, when a task is canceled, the task manager will cancel
not only its direct child tasks but all also its descendant tasks.

Closes #50990
2020-04-13 12:35:21 -04:00
Igor Motov 51c6f69e02
[7.x] Add support for filters to T-Test aggregation (#54980) (#55066)
Adds support for filters to T-Test aggregation. The filters can be used to
select populations based on some criteria and use values from the same or
different fields.

Closes #53692
2020-04-13 12:28:58 -04:00
James Rodewig 57d6493e29 [DOCS] EQL: Document `string` function (#55086) 2020-04-13 11:23:45 -04:00
Peter Dyson f0b6cf4c11 [DOCS] Note where ILM policies are stored and backup caveats (#54859) 2020-04-13 09:11:16 -06:00
Vishal Patel 16921ebbd8 [DOCS] Collapse nested objects in Explore API docs (#55067)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-13 09:27:03 -04:00
Ioannis Kakavas 7a8a66d9ae
[7.x] Fix ReloadSecureSettings API to consume password (#54771) (#55059)
The secure_settings_password was never taken into consideration in
the ReloadSecureSettings API. This commit fixes that and adds
necessary REST layer testing. Doing so, it also:

- Allows TestClusters to have a password protected keystore
so that it can be set for tests.
- Adds a parameter to the run task so that elastisearch can
be run with a password protected keystore from source.
2020-04-13 09:50:55 +03:00
Yang Wang 862799956c
Deprecate local parameter for get field mapping request (#55014) (#55099)
The usage of local parameter for GetFieldMappingRequest has been removed from the underlying transport action since v2.0.

This PR deprecates the parameter from rest layer. It will be removed in next major version.
2020-04-12 13:48:47 +10:00
James Rodewig 2655dfa2fe [DOCS] EQL: Reword field support for EQL functions (#55074)
Changes boilerplate sentence of "If using a field as the argument, this
parameter only supports..." to "...this parameter supports only...".

The latter is a bit more clear and readable.
2020-04-10 15:33:29 -04:00
Jason Tedor d1137ebdaa
Passthrough special characters in thread pool docs (#55080)
Some of these characters are special to Asciidoctor and they ruin the
rendering on this page. Instead, we use a macro to passthrough these
characters without Asciidoctor applying any subtitutions to them. This
commit then addresses some rendering issues in the thread pool docs.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-10 15:11:19 -04:00
Nik Everett b99a50bcb9
value_count Aggregation optimization (backport of #54854) (#55076)
We found some problems during the test.

Data: 200Million docs, 1 shard, 0 replica

    hits    |   avg   |   sum   | value_count |
----------- | ------- | ------- | ----------- |
     20,000 |   .038s |   .033s |       .063s |
    200,000 |   .127s |   .125s |       .334s |
  2,000,000 |   .789s |   .729s |      3.176s |
 20,000,000 |  4.200s |  3.239s |     22.787s |
200,000,000 | 21.000s | 22.000s |    154.917s |

The performance of `avg`, `sum` and other is very close when performing
statistics, but the performance of `value_count` has always been poor,
even not on an order of magnitude. Based on some common-sense knowledge,
we think that `value_count` and sum are similar operations, and the time
consumed should be the same. Therefore, we have discussed the agg
of `value_count`.

The principle of counting in es is to traverse the field of each
document. If the field is an ordinary value, the count value is
increased by 1. If it is an array type, the count value is increased
by n. However, the problem lies in traversing each document and taking
out the field, which changes from disk to an object in the Java
language. We summarize its current problems with Elasticsearch as:

- Number cast to string overhead, and GC problems caused by a large
  number of strings
- After the number type is converted to string, sorting and other
  unnecessary operations are performed

Here is the proof of type conversion overhead.

```
// Java long to string source code, getChars is very time-consuming.
public static String toString(long i) {
        int size = stringSize(i);
        if (COMPACT_STRINGS) {
            byte[] buf = new byte[size];
            getChars(i, size, buf);
            return new String(buf, LATIN1);
        } else {
            byte[] buf = new byte[size * 2];
            StringUTF16.getChars(i, size, buf);
            return new String(buf, UTF16);
        }
}
```

  test type  | average |  min |     max     |   sum
------------ | ------- | ---- | ----------- | -------
double->long |  32.2ns | 28ns |     0.024ms |  3.22s
long->double |  31.9ns | 28ns |     0.036ms |  3.19s
long->String | 163.8ns | 93ns |  1921    ms | 16.3s

particularly serious.

Our optimization code is actually very simple. It is to manage different
types separately, instead of uniformly converting to string unified
processing. We added type identification in ValueCountAggregator, and
made special treatment for number and geopoint types to cancel their
type conversion. Because the string type is reduced and the string
constant is reduced, the improvement effect is very obvious.

    hits    |   avg   |   sum   | value_count | value_count | value_count | value_count | value_count | value_count |
            |         |         |    double   |    double   |   keyword   |   keyword   |  geo_point  |  geo_point  |
            |         |         |   before    |    after    |   before    |    after    |   before    |    after    |
----------- | ------- | ------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
     20,000 |     38s |   .033s |       .063s |       .026s |       .030s |       .030s |       .038s |       .015s |
    200,000 |    127s |   .125s |       .334s |       .078s |       .116s |       .099s |       .278s |       .031s |
  2,000,000 |    789s |   .729s |      3.176s |       .439s |       .348s |       .386s |      3.365s |       .178s |
 20,000,000 |  4.200s |  3.239s |     22.787s |      2.700s |      2.500s |      2.600s |     25.192s |      1.278s |
200,000,000 | 21.000s | 22.000s |    154.917s |     18.990s |     19.000s |     20.000s |    168.971s |      9.093s |

- The results are more in line with common sense. `value_count` is about
  the same as `avg`, `sum`, etc., or even lower than these. Previously,
  `value_count` was much larger than avg and sum, and it was not even an
  order of magnitude when the amount of data was large.
- When calculating numeric types such as `double` and `long`, the
  performance is improved by about 8 to 9 times; when calculating the
  `geo_point` type, the performance is improved by 18 to 20 times.
2020-04-10 13:16:39 -04:00
James Rodewig c440754784 [DOCS] EQL: Document `wildcard` function (#54086) 2020-04-10 09:18:29 -04:00
oneoneonepig 356cc94889 [DOCS] Fix double quote typo in 7.0 breaking changes (#55040) 2020-04-10 09:11:51 -04:00
Jason Tedor 9eeae59a83
Clarify available processors (#54907)
The use of available processors, the terminology, and the settings
around it have evolved over time. This commit cleans up some places in
the codes and in the docs to adjust to the current terminology.
2020-04-10 08:48:27 -04:00
James Rodewig 51326432be [DOCS] Add query reference docs template (#52292) 2020-04-10 08:47:54 -04:00
James Rodewig d5a609a2e5 [DOCS] Add token filter reference docs template (#52290)
Creates a reusable template for token filter reference documentation.

Contributors can make a copy of this template and customize it when
documenting new token filters.
2020-04-10 08:45:10 -04:00
Marios Trivyzas bf0cadb602
SQL: Implement DATETIME_PARSE function for parsing strings (#54960) (#55035)
Implement DATETIME_PARSE(<datetime_str>, <pattern_str>) function
which allows to parse a datetime string according to the specified
pattern into a datetime object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Relates to #53714

(cherry picked from commit 3febcd8f3cdf9fdda4faf01f23a5f139f38b57e0)
2020-04-10 01:16:29 +02:00
Vishal Patel 51cb0c5c7b [DOCS] Collapse nested objects in cluster reroute docs (#54851) 2020-04-09 15:29:22 -04:00
István Zoltán Szabó 374f633b6e [DOCS] Adds link points to the data frame analytics supported fields (#55004)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-09 11:27:57 -07:00
James Rodewig c6cd8ca7c0
[DOCS] Update upgrade docs for 7.7 (#54978) 2020-04-08 16:23:08 -04:00
James Rodewig 964cf565c9
[DOCS] EQL: Document `between` function (#54950) 2020-04-08 13:49:15 -04:00
Théophile Helleboid - chtitux a8aa36d427 [DOCS] Fix typo in SLM retention docs (#54797) 2020-04-08 08:56:45 -04:00
Marios Trivyzas 6afd60b082
SQL: Implement DATETIME_FORMAT function for date/time formatting (#54832) (#54942)
Implement DATETIME_FORMAT(<date/datetime/time>, ) function
which allows for formatting a timestamp to the specified format. The
patterns allowed as those of java.time.format.DateTimeFormatter.

Related to #53714

(cherry picked from commit 72be0b54a9299e87e785469cdc9aafac2a48c046)
2020-04-08 13:45:47 +02:00
István Zoltán Szabó 3a3effedc2 [DOCS] Reworks some parts of EMM API docs (#54872)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-08 10:20:34 +02:00
Julie Tibshirani 475b210eec
Improve guidance on removing default mappings. (#54915)
In 7.x, an index template will fail to apply if it contains a `_default_`
mapping. Several users have expressed confusion over the fact that loading the
template doesn't show any default mappings. This docs change clarifies that in
order to see all mappings in the template, you must pass `include_type_name`.
2020-04-07 15:18:13 -07:00
James Rodewig 9569a8eb13 [DOCS] Add example to "avoid scripts" advice (#54719)
Adds a detailed example to the "Avoid scripts" section of the "Tune
for search speed" docs. The detail outlines how a script used to
transform indexed data can be moved to ingest.

The update also removes an outdated reference to supported script
languages.
2020-04-07 15:25:10 -04:00
Jason Tedor d1d478debf
Update docs to reflect node.processors (#54855)
We namespaced the previous setting "processors" into
"node.processors". This commit updates some of the documentation to
reflect this.
2020-04-07 13:06:14 -04:00
Lisa Cawley a7599031ae [DOCS] Adds tranform node to list of default types (#54850) 2020-04-07 08:49:05 -07:00
Ignacio Vera 076c199484
Add new point field. (#53804) (#54879)
This commit adds a new point field that is able to index arbitrary pair of values (x/y)
in the cartesian space. It only supports filtering using shape queries at the moment.
2020-04-07 15:28:50 +02:00
Tanguy Leroux 4d36917e52
Merge feature/searchable-snapshots branch into 7.x (#54803) (#54825)
This is a backport of #54803 for 7.x.

This pull request cherry picks the squashed commit from #54803 with the additional commits:

    6f50c92 which adjusts master code to 7.x
    a114549 to mute a failing ILM test (#54818)
    48cbca1 and 50186b2 that cleans up and fixes the previous test
    aae12bb that adds a missing feature flag (#54861)
    6f330e3 that adds missing serialization bits (#54864)
    bf72c02 that adjust the version in YAML tests
    a51955f that adds some plumbing for the transport client used in integration tests

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-07 13:28:53 +02:00
Ioannis Kakavas 3560c0cbf2
Remove `_xpack` from license API example (#54698) (#54763)
Resolves #54662
2020-04-07 09:51:37 +03:00
Lisa Cawley b3d5300968
[DOCS] Collapses sections in put snapshot lifecycle policy API (#54834) (#54840) 2020-04-06 13:46:56 -07:00
James Rodewig e9c3bfc8e5 [DOCS] Collapse nested objects in node stats API response (#54755)
Replaces dot notation with collapsed nested object formatting
per the [Elastic API reference template][0].

[0]:https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc
2020-04-06 15:19:54 -04:00
James Rodewig 548ad03941 [DOCS] Collapse nested objects in cluster stats API response (#54739)
Replaces dot notation with collapsed nested object formatting
per the [Elastic API reference template][0].

[0]:https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc
2020-04-06 13:11:46 -04:00
Igor Motov 2794572a35
[7.x] Add Student's t-test aggregation support (#54469) (#54737)
Adds t_test metric aggregation that can perform paired and unpaired two-sample
t-tests. In this PR support for filters in unpaired is still missing. It will
be added in a follow-up PR.

Relates to #53692
2020-04-06 11:36:47 -04:00
Nhat Nguyen 2fdbed7797 Broadcast cancellation to only nodes have outstanding child tasks (#54312)
Today when canceling a task we broadcast ban/unban requests to all nodes
in the cluster. This strategy does not scale well for hierarchical
cancellation. With this change, we will track outstanding child requests
and broadcast the cancellation to only nodes that have outstanding child
tasks. This change also prevents a parent task from sending child
requests once it got canceled.

Relates #50990
Supersedes #51157

Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2020-04-06 11:11:29 -04:00
István Zoltán Szabó 7dc1ba4273 [DOCS] Updates transform prerequisites (#54804) 2020-04-06 17:07:59 +02:00
Christoph Büscher def519ea70 [Docs] Correct date rounding example for `range` query (#51524)
Looking into #50237 I realized that two of the examples given in the
documentation around date math rounding for range queries on date fields using
`gt` and `lt` is slightly off by a nanosecond. This PR changes this to the
bounds that are currently parsed using these parameters.
2020-04-06 17:05:45 +02:00
István Zoltán Szabó 4cba1e6368 [DOCS] Changes kibana_user to kibana_admin in DFA API prerequisites. (#54806) 2020-04-06 15:46:18 +02:00
Jason Tedor 184c038f59
Add get autoscaling policy API (#54762)
This commit adds the get autoscaling policy API.
2020-04-04 18:04:25 -04:00
James Baiera 705e46d5c1
[DOCS] Remove unused cat tasks request parameters (#54539) (#54741) 2020-04-03 15:33:28 -04:00
Lisa Cawley de91d2aeea [DOCS] Collapse nested objects in CCR APIs (#54697) 2020-04-03 12:04:33 -07:00
Bogdan Pintea 7cef89e084 ODBC: Document the new VarcharLimit and EarlyExecution params (#54632)
* Document VarcharLimit and EarlyExecution params

Add the documentation for the newly added VarcharLimit and
EarlyExecution DSN attributes.

* Remove obsolete VersionChecking param

This param had been removed already along the #53082 work.

* Update docs/reference/sql/endpoints/odbc/configuration.asciidoc

fix typo

Co-Authored-By: Stuart Cam <stuart@codebrain.co.uk>

* Update docs/reference/sql/endpoints/odbc/configuration.asciidoc

fix typo

Co-Authored-By: Stuart Cam <stuart@codebrain.co.uk>
(cherry picked from commit f38761631a12b38f7f075635f7ac61dc96656cd7)
2020-04-03 14:46:39 +02:00
markharwood 2da2305587
Backport of lowercase normalizer PR #53882
A pre-configured normalizer for lower-casing.
Closes #53872
2020-04-03 11:43:40 +01:00
István Zoltán Szabó d025b90cd1 [DOCS] Makes PUT inference API docs collapsible (#54653)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-03 09:48:53 +02:00
Lisa Cawley 11afead21e [DOCS] Adds collapsible sections to rollup APIs (#54690) 2020-04-02 17:51:16 -07:00
Lisa Cawley b138dc4565
[DOCS] Add processing details to get transforms stats API (#54368) (#54595) 2020-04-02 16:12:57 -07:00
lcawl d3fa0ec730 [DOCS] Fixes typo in node settings 2020-04-02 16:01:54 -07:00
Lisa Cawley 98965116fe [DOCS] Clarify ML and transform settings on coordinating nodes (#54676) 2020-04-02 15:38:15 -07:00
Julie Tibshirani 5fb7602227
Disallow changing 'enabled' on the root mapper. (#54681)
In #33933 we disallowed changing the `enabled` parameter in object mappings.
However, the fix didn't cover the root object mapper. This PR adjusts the change
to also include the root mapper and clarifies the error message.
2020-04-02 15:28:48 -07:00
David Turner 4e083cd97d indices.recovery.max_bytes_per_sec may be per-node (#54633)
The `indices.recovery.max_bytes_per_sec` recovery bandwidth limit can differ
between nodes if it is not set dynamically, but today this is not obvious. This
commit adds a paragraph to its documentation clarifying how to set different
bandwidth limits on each node.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-02 18:15:41 +01:00
Benjamin Trent 4a1610265f
[7.x] [ML] add new inference_config field to trained model config (#54421) (#54647)
* [ML] add new inference_config field to trained model config (#54421)

A new field called `inference_config` is now added to the trained model config object. This new field allows for default inference settings from analytics or some external model builder.

The inference processor can still override whatever is set as the default in the trained model config.

* fixing for backport
2020-04-02 12:25:10 -04:00
Benjamin Trent 65233383f6
[7.x] [ML] prefer secondary authorization header for data[feed|frame] authz (#54121) (#54645)
* [ML] prefer secondary authorization header for data[feed|frame] authz (#54121)

Secondary authorization headers are to be used to facilitate Kibana spaces support + ML jobs/datafeeds.

Now on PUT/Update/Preview datafeed, and PUT data frame analytics the secondary authorization is preferred over the primary (if provided).

closes https://github.com/elastic/elasticsearch/issues/53801

* fixing for backport
2020-04-02 11:20:25 -04:00
qiye de8e0200fe [DOCS] Correct `shape` field release in 7.5 release highlights (#54631)
The `shape` field was added in 7.4, not 7.3.
This corrects a small error in the 7.5 release highlights.
2020-04-02 09:19:40 -04:00
Benjamin Trent eb31be0e71
[7.x] [ML] add num_matches and preferred_to_categories to category defintion objects (#54214) (#54639)
* [ML] add num_matches and preferred_to_categories to category defintion objects (#54214)

This adds two new fields to category definitions.

- `num_matches` indicating how many documents have been seen by this category
- `preferred_to_categories` indicating which other categories this particular category supersedes when messages are categorized.

These fields are only guaranteed to be up to date after a `_flush` or `_close`

native change: https://github.com/elastic/ml-cpp/pull/1062

* adjusting for backport
2020-04-02 09:09:19 -04:00
Jason Tedor 54ecb009bb
Add delete autoscaling policy API (#54601)
This commit adds an API for deleting autoscaling policies.
2020-04-02 09:05:12 -04:00
Aleh Zasypkin 161eac1942
[7.x] Switch to the most recent Kibana configuration format and SAML/OIDC endpoints. (#54624) 2020-04-02 11:59:11 +02:00
István Zoltán Szabó 3a8e880fe6 [DOCS] Adds snippet comparing two indices to the painless examples (#54563) 2020-04-02 08:46:25 +02:00
Russ Cam 110d9a7845 Correct description for mget API request URI index (#52305)
This commit corrects the description for the request URI index for the Multi Get (mget) API.
The index can only be a single index name (multiple or wildcard expressions not supported),
and acts as the index to use when "ids" are specified, or a document in the "docs" array does
not specify an index.

(cherry picked from commit aa4926ed7f91dfbf7973a01b1e4682e91dda11a9)
2020-04-02 09:08:34 +10:00
lcawl 949636944c [DOCS] Fixes shared attribute for feature importance 2020-04-01 14:52:08 -07:00
Jason Tedor ccecb78c98
Rename the policy in put autoscaling policy docs
The put autoscaling policy docs use a "hot" policy as an
example. Instead, this commit changes the name of this policy to
"my_autoscaling_policy".
2020-04-01 16:32:03 -04:00
Jason Tedor bd6b383926
Include autoscaling APIs in API docs (#54603)
This commit adds a top-level link to the autoscaling API reference page
to the API docs. Additionally, we add a conditional guard on the API
pages to only include them in development builds of the docs.
2020-04-01 15:44:13 -04:00
lcawl 2cd35bf696 [DOCS] Adds release highlights placeholder 2020-04-01 09:22:20 -07:00
Lisa Cawley f5ccf939d9 [DOCS] Clarifies API key breaking change (#54522) 2020-04-01 08:58:15 -07:00
James Rodewig 21abc311fd
[DOCS] Reformat `keyword_repeat` token filter (#54428) 2020-04-01 11:56:05 -04:00
James Rodewig 4982b720ef
[DOCS] EQL: Document `length` function (#54225) 2020-04-01 11:35:36 -04:00
James Rodewig b43eb5ac32
[DOCS] EQL: Document `endsWith` function (#54521) 2020-04-01 10:43:37 -04:00
Dan Hermann 11bfbd8bbf
Rename examples in ILM guide to avoid association with data streams (#54579) 2020-04-01 09:05:20 -05:00
István Zoltán Szabó 27f88fcdac [DOCS] Updates estimate model memory docs (#54574) 2020-04-01 15:55:25 +02:00
James Rodewig 95622d8782
[DOCS] EQL: Document `startsWith` function (#54518) (#54578) 2020-04-01 09:30:27 -04:00
James Rodewig 92d570d6f3
[DOCS] EQL: Add search/index speed tip for functions (#54346) (#54575)
EQL functions are an easy way for users to transform indexed data
at search time. However, using multiple functions can make
queries difficult to write and slows search speeds.

Users can circumvent this by indexing fields containing the transformed
data, but that usually slows index speeds.

This adds a related tip and example covering these tradeoffs.
2020-04-01 08:39:04 -04:00
bellengao e9c201b446 [DOCS] Correct field name in date_nanos doc (#54556) 2020-04-01 14:22:32 +02:00
Jason Tedor f670ae0bc8
Introduce autoscaling policies (#54473)
This commit is the first in a series of commits that introduces
autoscaling policies, and APIs for working with them. For now, we
introduce the basic infrastructure, and a single API for putting an
autoscaling policy. We will follow in rapid succession with APIs for
getting, and deleting autoscaling policies.
2020-04-01 08:12:26 -04:00
István Zoltán Szabó 325b8ec0ce [DOCS] Adds data_counts object to the GET DFA stats API (#54498) 2020-04-01 10:07:28 +02:00
Jason Tedor 5fcda57b37
Rename MetaData to Metadata in all of the places (#54519)
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 17:24:38 -04:00
James Rodewig 114894dd76 [DOCS] Add redirect for changed anchor ID (#54533)
The anchor ID for the snapshot repository plugins section in the docs
was recently changes from `_repository_plugins` to
`snapshots-repository-plugins`.

This adds a corresponding redirect so no links are broken.
2020-03-31 16:42:16 -04:00
James Rodewig 7401191019
[DOCS] Include 7.7.0 release notes (#54529)
Includes the 7.7.0 release notes so they render in the HTML docs.

Also removes a few legacy `coming[7.6.0]` tags.
2020-03-31 16:23:49 -04:00
Lisa Cawley 922ec8e961
[DOCS] Collapses nested objects in data frame analytics APIs (#54472) (#54526) 2020-03-31 12:51:04 -07:00
Glen Smith 0d4a001ef2 [DOCS] Clarify cluster health status during rolling upgrade (#40757)
Remove mention of the `yellow` and `red` starting
health status from the rolling upgrade docs.

Instead, we should emphasize that users wait 
for the node to recover with a health status of
`green` rather than the starting status.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-31 14:13:05 -04:00
Dimitris Athanasiou 0b25e3b66c
[7.x][ML] Fix DF analytics explain API request in docs (#54510) (#54514)
The explain API expects a data frame analytics config
as its request.

Backport of #54410
2020-03-31 18:56:52 +03:00
István Zoltán Szabó a5497cd9e0 [DOCS] Adds HTTP response count example to Painless examples (#54412) 2020-03-31 15:12:58 +02:00
István Zoltán Szabó eeb23e9e73 [DOCS] Adds description of analysis_stats object and its properties to GET DFA stats API docs (#53881)
Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-31 13:30:06 +02:00
James Rodewig ce7539ce6c
[DOCS] Fix broken deprecated macros (#54466)
Updates formatting for `deprecated` macros in the translog and
synced flush docs.

Previously, these macros were rendering literally.
2020-03-30 17:16:27 -04:00
Lisa Cawley 0fa1060ca4
[DOCS] Collapses content in machine learning APIs (#54234) (#54453) 2020-03-30 11:06:33 -07:00
James Rodewig ed1edb4964 [DOCS] Add missing word to keyword marker token filter docs 2020-03-30 10:52:14 -04:00
James Rodewig 21f362a2a8
[DOCS] Add a lowercase email example to keyword tokenizer docs (#53257) 2020-03-30 09:06:04 -04:00
Benjamin Trent 374e76d7cd
[Transform] fixing naming in HLRC and _cat to match API content (#54300) (#54408)
Fixing the naming of the HLRC values to match the ToXContent field names (i.e. the field names returned from an API call).

Also fixes the names in the _cat API as well.

closes #53946
2020-03-30 08:57:02 -04:00
István Zoltán Szabó 00eaa0ebe5 [DOCS] Changes scripted metric to filter aggs in transforms example (#54167) 2020-03-30 09:51:07 +02:00
Jason Tedor f0033783db
Deprecate node local storage setting (#54374)
This setting is not documented and has dubious value since it means
there can be nodes in the cluster (non-data and non-master nodes) that
do not have persistent node IDs. This does not have any use cases so
this commit removes the setting.
2020-03-28 14:36:41 -04:00
Gil Raphaelli 2984a54b7f [DOCS] Fix typos in top metrics agg docs (#54299) 2020-03-27 10:49:21 -04:00
AndyHunt66 2dd8946539 [DOCS] Remove redundant sentence in ingest processor docs (#54329) 2020-03-27 08:25:09 -04:00
Christoph Büscher f7ea794312 [Test] Don't expect specific scores in docs tests (#54297)
The failing suggester documentation test was expecting specific scores in the
test response, which is fragile implementation details that e.g. can change with
different lucene versions and generally shouldn't be done in documentation test.
Instead we usually replace the float values in the output response by the ones
in the actual response.

Closes #54257
2020-03-27 10:27:47 +01:00
Lisa Cawley 27cd5b343c
[DOCS] Augments cat transforms API (#53776) (#54232) 2020-03-26 07:56:46 -07:00
Jason Tedor d8f745736b
Clarify the remove keystore command can handle many (#54244)
The remove keystore command can handle multiple settings. In a few
places, we were not consistent about mentioning this. This commit
addreses this, in the CLI help, and the docs.
2020-03-26 08:49:43 -04:00
Luca Cavanna ff269160af Async search: rename REST parameters (#54198)
This commit renames wait_for_completion to wait_for_completion_timeout in submit async search and get async search.
Also it renames clean_on_completion to keep_on_completion and turns around its behaviour.

Closes #54069
2020-03-26 09:40:50 +01:00
István Zoltán Szabó 487b273286 [DOCS] Adds feature importance mapping subsection to inference processor docs (#54190) 2020-03-26 09:26:50 +01:00
Jason Tedor 6af89e62d1
Allow keystore add-file to handle multiple settings (#54240)
Today the keystore add-file command can only handle adding a single
setting/file pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add-file keystore command to accept adding multiple
settings in a single invocation.
2020-03-26 00:07:05 -04:00
Jason Tedor 4fbc0e9ab8
Complete keystore CLI options documentation (#54242)
The documentation was missing the long option for the force option, and
the short option for the stdin option. This commit addresses this by
adding these to the documentation.
2020-03-25 23:53:34 -04:00
Jason Tedor fe8257d981
Allow keystore add to handle multiple settings (#54229)
Today the keystore add command can only handle adding a single
setting/value pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add keystore command to accept adding multiple
settings in a single invocation.
2020-03-25 22:58:20 -04:00
James Rodewig 30a32040d3
[DOCS] EQL: Document `substring` function (#53867)
Adds documentation for the EQL `substring` function.

Supporting changes:

* Creates a new "EQL function reference" page
* Updates the title of the "EQL syntax reference" page for consistency
* Adds a brief "Functions" section to the EQL syntax docs
* Updates EQL limitations docs to state that only array functions are
  unsupported
2020-03-25 12:23:59 -04:00
Nik Everett 16e4bd50e2 Add breaking change note for #53669 2020-03-25 09:31:14 -04:00
James Rodewig 74051d68a8
[DOCS] Reformat `keyword_marker` token filter (#54076)
Makes the following changes to the `keyword_marker` token filter docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer and filter example
2020-03-25 09:26:06 -04:00
James Rodewig 2fdf6b2f96
[DOCS] Document missing data types for node stats API's response parameters (#53475)
Documents missing data types for several response parameters returned
by the node stats API.

Also adds several missing human-readable parameters returned by the API.
2020-03-25 08:42:58 -04:00
Jason Tedor 381d7586e4
Introduce formal role for remote cluster client (#54138)
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.

Relates #53924
2020-03-24 21:59:43 -04:00
David Roberts 7667004b20
[ML] Add a model memory estimation endpoint for anomaly detection (#54129)
A new endpoint for estimating anomaly detection job
model memory requirements:

POST _ml/anomaly_detectors/estimate_model_memory

Backport of #53507
2020-03-24 22:55:11 +00:00
Jim Ferenczi 55f2e8bff0 [DOCS] Add 7.6.2 release notes (#53720)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-24 22:42:25 +01:00
markharwood 6a60f85bba
Wildcard field - add normalizer support (#53851) (#54109)
Backport support for normalisation to wildcard field

Closes #53603
2020-03-24 17:37:47 +00:00
Lisa Cawley 88b1b2f36f [DOCS] Adds transform security privileges (#53908) 2020-03-24 09:35:45 -07:00
Tim Brooks caefa78513
Align remote info api with new settings (#54102)
Currently the remote info api has added a number of possible fields
(proxy, num_socket_connections, etc) that are available in proxy mode.
These fields are not aligned with what the settings are named. This
commit modifies this API to align with the settings.
2020-03-24 10:27:24 -06:00
Luca Cavanna 6b457abbd3 Async search: prevent users from overriding pre_filter_shard_size (#54088)
Submit async search forces pre_filter_shard_size for the underlying search that it creates.
With this commit we also prevent users from overriding such default as part of request validation.
2020-03-24 17:06:04 +01:00
Ahmet Arslan 52062565a9 [DOCS] Correct DFI docs regarding stop word removal (#53836)
The documentation of DFI should recommend *not* to [remove stop words][1], since DFI is good at scoring queries that contain common terms: `the wall`, `the sun`, `the who`, etc.

[1]:https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/search/similarities/DFISimilarity.html
2020-03-24 10:48:42 -04:00
Karen Metts 9da589c5fd [DOCS] Replace outdated Logstash monitoring link (#54032)
Replaces a link to Logstash OSS-only content with a link to the general Logstash monitoring topic.
2020-03-24 10:03:31 -04:00
David Roberts 1421471556
[ML] Introduce a "starting" datafeed state for lazy jobs (#54065)
It is possible for ML jobs to open lazily if the "allow_lazy_open"
option in the job config is set to true.  Such jobs wait in the
"opening" state until a node has sufficient capacity to run them.

This commit fixes the bug that prevented datafeeds for jobs lazily
waiting assignment from being started.  The state of such datafeeds
is "starting", and they can be stopped by the stop datafeed API
while in this state with or without force.

Backport of #53918
2020-03-24 13:00:04 +00:00
Hendrik Muhs 7dcacf531f
[7.x][Transform][Rollup] add processing stats to record the ti… (#54027)
add 2 additional stats: processing time and processing total which capture the
time spent for processing results and how often it ran. The 2 new stats
correspond to the existing indexing and search stats. Together with indexing
and search this now allows the user to see the full picture, all 3 stages.
2020-03-24 09:22:02 +01:00
Jason Tedor e3ca124537
Introduce autoscaling decisions (#53934)
This is the first in a series of commits that will introduce the
autoscaling deciders framework. This commit introduces the basic
framework for representing autoscaling decisions.
2020-03-23 23:08:06 -04:00
Jim Ferenczi 9e3f7f4575
Add heuristics to compute pre_filter_shard_size when unspecified (#53873) (#54007)
This commit changes the pre_filter_shard_size default from 128 to unspecified.
This allows to apply heuristics based on the request and the target indices when deciding
whether the can match phase should run or not. When unspecified, this pr runs the can match phase
automatically if one of these conditions is met:
  * The request targets more than 128 shards.
  * The request contains read-only indices.
  * The primary sort of the query targets an indexed field.
Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value.

Closes #39835
2020-03-24 02:05:15 +01:00
Julie Tibshirani df7cfb3a5b
Remove the top-level 'mapping type' section. (#54035)
It seemed confusing for users that our top-level mapping page still had a
prominent section named 'Mapping Type'. This PR reworks the docs to remove this
reference and adds a note about types removal (similar to the note we added to
other APIs like put mapping).
2020-03-23 15:34:23 -07:00
James Rodewig 43199a8c82 [DOCS] Remove double space in WDG docs 2020-03-23 17:18:04 -04:00
James Rodewig 553d8a9ca9 [DOCS] Fix "letter case" typo
Changes "lettercase" to "letter case" in the `uppercase` token filter
docs.
2020-03-23 17:11:59 -04:00
Paweł Krześniak c0534f4157 [DOCS] link fix (#53973)
Fix bad link in top_metrics.
2020-03-23 14:20:54 -04:00
Luca Cavanna 932a7e3112
Backport of async search changes (#53976)
* Get Async Search: omit _clusters section when empty (#53907)

The _clusters section is omitted by the search API whenever no remote clusters are searched. Async search should do the same, but Get Async Search returns a deserialized response, hence a weird `_clusters` section with all values set to `0` gets returned instead. In fact the recreated Clusters object is not the same object as the EMPTY constant, yet it has the same content.

This commit addresses this by changing the comparison in the `toXContent` method to not print out the section if the number of total clusters is `0`.

* Async search: remove version from response (#53960)

The goal of the version field was to quickly show when you can expect to find something new in the search response, compared to when nothing has changed. This can also be done by looking at the `_shards` section and `num_reduce_phases` returned with the search response. In fact when there has been one or more additional reduction of the results, you can expect new results in the search response. Otherwise, the `_shards` section could notify of additional failures of shards that have completed the query, but that is not a guarantee that their results will be exposed (only when the following partial reduction is performed their results will be available).

That said this commit clarifies this in the docs and removes the version field from the async search response

* Async Search: replicas to auto expand from 0 to 1 (#53964)

This way single node clusters that are green don't go yellow once async search is used, while
all the others still have one replica.

* [DOCS] address timing issue in async search docs tests (#53910)

The docs snippets for submit async search have proven difficult to test as it is not possible to guarantee that you get a response that is not final, even when providing `wait_for_completion=0`. In the docs we want to show though a proper long-running query, and its first response should be partial rather than final.

With this commit we adapt the docs snippets to show a partial response, and replace under the hood all that's needed to make the snippets tests succeed when we get a final response. Also, increased the timeout so we always get a final response.

Closes #53887
Closes #53891
2020-03-23 19:13:31 +01:00
Lisa Cawley c4260ba3c7 [DOCS] Fixes formatting in transform overview (#53900) 2020-03-23 10:21:54 -07:00
C.J. Jameson 5ac8b795ba [DOCS] Clarify routing enforcement in docs (#53945)
Removes a mention of the `_doc` mapping type that's
no longer applicable now that mapping types are
removed/deprecated.
2020-03-23 12:57:07 -04:00
Lisa Cawley 5f45780577 [DOCS] Adds data nanos transform limitation (#53826) 2020-03-23 09:49:21 -07:00
Lisa Cawley 8b3ef4ac21 [DOCS] Add generated_dest_index to preview transform API (#53905) 2020-03-23 09:43:44 -07:00
Marios Trivyzas af03200ad6
SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) (#47720) (#53972)
The function is extended to operate on intervals according to the PostgreSQL: https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes : #46632
(cherry picked from commit 2dc79505825fa75e0711dcfa8e9c69e8028fc979)

Co-authored-by: musteaf <gs_mustea@hotmail.com>
2020-03-23 15:05:16 +01:00
Dan Hermann 0528e2b06b
Document the deprecation of the 'auth.password' setting (#53822) 2020-03-23 08:58:25 -05:00
István Zoltán Szabó 534e734ad1 [DOCS] Adds painless transform examples (#53274)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-23 10:39:47 +01:00
Przemyslaw Gomulka 015ad019d5
[docs] Known issue about joda patterns on 7.6 (#53957) 2020-03-23 10:28:55 +01:00
Przemyslaw Gomulka 412e163cf6
[Doc] migration guide joda (#51986)
The joda to java.time migration requires users to upgrade their mappings. We allow them to still use 6.x created indices with joda patterns in 7 but ask them to upgrade their patterns in 7.x.
This migration guide is to help them understand how they could be affected and what needs to be changed in their mappings.
closes #51614
closes #51236
2020-03-23 08:29:01 +01:00
Mark Vieira 0cfe6d90cc
Mute async-search test 2020-03-20 11:35:24 -07:00
Luca Belluccini 3937854444 [DOCS] Clarify upgrade paths (#53417)
Unsupported upgrade paths:
- `6.8 to 7.0`.
Supported, but requires a full cluster restart:
- `6.0–6.7 directly to 7.x`
2020-03-20 18:05:07 +00:00
Lisa Cawley c2af015ee7 [DOCS] Updates list of transform aggs (#53820) 2020-03-20 10:28:23 -07:00
Alan Woodward a70016a6d6 TermsLookup fields should be marked as Required in docs (#53784)
The terms-lookup section of our terms query docs currently state that the
index, id and path fields are optional. They should be marked instead
as required.
2020-03-20 15:39:41 +00:00
Luca Cavanna d486bdefdd [DOCS] correct async search note
The sort optimization kicks in whenever results are sorted by field.
2020-03-20 15:58:19 +01:00
Luca Cavanna 03fca61fcb [DOCS] add docs for async search (#53675)
Relates to #49091

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-03-20 14:46:38 +01:00
James Rodewig 6c57681446 [DOCS] Add redirects for missing data stream API docs (#53866)
The following API spec files contain a link to a not-yet-created
docs page for the data stream APIs:

* [indices.create_data_stream.json][0]
* [indices.delete_data_stream.jsonn][1]
* [indices.get_data_streams.json][2]

The Elaticsearch-JS client uses these spec files to create their docs.
This created a broken link in the Elaticsearch-JS, which has broken
the docs build.

This PR adds a temporary redirect for the docs page. This redirect
should be removed when the actual API docs are added.

[0]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.create_data_stream.json
[1]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_stream.json
[2]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_streams.json

Relates to #53558.

CC @martijnvg
2020-03-20 09:41:47 -04:00
lgypro be3090138e [Docs] Fix typo in _analyze api docs (#53837) 2020-03-20 12:02:40 +01:00
István Zoltán Szabó 53f7e31462 [DOCS] Fixes typo in start datafeed API docs. (#53811) 2020-03-19 17:56:40 +01:00
István Zoltán Szabó 29583288b3 [DOCS] Adds performance considerations section to transforms overview (#53791)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-03-19 17:51:00 +01:00
Lisa Cawley 68f7036a4c [DOCS] Adds example links to transform tutorial (#53640) 2020-03-19 09:41:57 -07:00
István Zoltán Szabó 00203c35fe [DOCS] Changes seconds to milliseconds since the Epoch in AD docs. (#53797) 2020-03-19 15:42:58 +01:00
Dominic Page b0884baf46
Geo shape query vs geo point backport (#53774)
Backport to 7x

Enable geo_shape query to work on geo_point fields for shapes: circle, polygon, multipolygon, rectangle see: #48928
Co-Authored-By:  @iverase
2020-03-19 13:00:36 +01:00
James Rodewig 8f4a3eb07f [DOCS] Add token graph concept docs (#53339)
Adds conceptual docs for token graphs.
These docs cover:

* How a token graph is constructed from a token stream
* How synonyms and multi-position tokens impact token graphs
* How token graphs are used during search
* Why some token filters produce invalid token graphs

Also makes the following supporting changes:
* Adds anchors to the 'Anatomy of an Analyzer' docs for cross-linking
* Adds several SVGs for token graph diagrams
2020-03-19 07:43:18 -04:00
Leaf-Lin b649b0d273
Update jvm options doc
There's a typo with extra `
2020-03-19 12:00:01 +11:00
Benjamin Trent 415d73c27d
[Transform] renamed _cat/transform to _cat/transforms (#53743) (#53771)
renaming _cat/transform to  _cat/transforms for uniformity with the other _cat apis.
2020-03-18 19:54:03 -04:00
Lisa Cawley 268e512f0b [DOCS] Add transform nodes (#53698) 2020-03-18 15:26:37 -07:00
James Rodewig 0e2e06bd7e [DOCS] Remove incorrect parms from put index template API docs (#53750)
Removes the `flat_settings` and `timeout` query parameters from the JSON
spec and asciidoc docs for the put index template API.

These parameters are not supported by the API.
2020-03-18 14:36:27 -04:00
markharwood 598d4c1bf9
Formatting fix
Bullet points were not rendered correctly
2020-03-18 15:36:52 +00:00
Lisa Cawley c6e37c7662 [DOCS] Adds stub for cat transform API (#53737) 2020-03-18 08:31:10 -07:00
James Rodewig 0732475cb9 [DOCS] Streamline `analyzer` mapping parm def (#51874)
Simplifies the `analyzer` mapping parameter definition to remove
duplicated analysis content and examples.
2020-03-18 09:43:59 -04:00