Commit Graph

8519 Commits

Author SHA1 Message Date
James Rodewig 2be6d7b8b6
[DOCS] Relocate request body param docs to search API docs (#56436)
Moves documentation for the following request body parameters to the
search API reference docs:

* `explain`
* `query`
* `seq_no_primary_term`
* `version`

Removes documentation for these parameters from the Request body search
page[0].

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-body.html
2020-05-11 11:29:38 -04:00
Lisa Cawley 7ce0a25fbc [DOCS] Add UUID troubleshooting tip for stack monitoring (#55744) 2020-05-11 07:49:34 -07:00
Lisa Cawley 1474606b18 [DOCS] Clarify model snapshot retention properties (#56477) 2020-05-11 07:43:10 -07:00
James Rodewig ba67ab3b64
[DOCS] Add reference docs for `search.max_buckets` setting (#56449) (#56511)
Adds reference-style setting documentation for the `search.max_buckets`
setting.

This setting was previously only documented on the [bucket
aggregations][0] page.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-bucket.html
2020-05-11 09:45:09 -04:00
István Zoltán Szabó ebe1e4c4c4 [DOCS] Expands GET DFA stats API docs with new phases (#56407)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-11 09:26:15 +02:00
Hendrik Muhs c0985615aa [DOC] document transform settings and docs_per_second (#56178)
add documentation for throttling, added in #56007
2020-05-11 09:23:49 +02:00
debadair 6ae7327061
[DOCS] Align with ILM changes. (#55953) (#56455)
* [DOCS] Align with ILM changes.

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Incorporated review comments.
2020-05-08 14:22:27 -07:00
Jake Landis 95e5e9e598
[7.x] [DOCS] Update default value of index.name.time_format (#56453) (#56456)
Corrects the default value of index.name.time_format
2020-05-08 16:09:42 -05:00
Théophile Helleboid - chtitux 23e419a7aa SLM PUT: add precision on date math support in indices parameter (#55203)
It was not clear for me that `indices` parameter supports date math expression.

I think it may be worth to add the precision in the documentation.
2020-05-08 15:06:09 -06:00
Nicole Albee 5b708f846c
[DOCS] Fix broken link in the ilm-tutorial. (#56310) (#56311) (#56446) 2020-05-08 12:22:39 -05:00
István Zoltán Szabó ceb0b0dba3 [DOCS] Updated screenshots in transform ecommerce example. (#56359) 2020-05-08 07:46:15 +02:00
James Rodewig ea76b0c22b
[DOCS] Relocate search API's request body parameters (#56304)
Changes:
* Moves the document request body parameters for the search API
  from the Request body search page to the Search API reference page.

* Relocates a search request body example from the Request body search
  page to the Search API reference page.

* Adds a note to any duplicated query and request body parameters.
2020-05-07 11:00:03 -04:00
William Brafford 691044e67b
Add xpack setting deprecations to deprecation API (#56290)
* Add xpack setting deprecations to deprecation API

The deprecated settings showed up in the deprecation log file by
default, but I did not add them to the deprecation API. This commit
fixes that. Now if you use one of the deprecated basic feature
enablement settings, calling _monitoring/deprecations will inform you of
that fact.

* Remove incorrectly backported settings documents

It seems that I backported these docs to the wrong place in #56061,
in #55980, and in #56167. I hope they're in the right place now.

Co-authored-by: debadair <debadair@elastic.co>
2020-05-07 10:28:17 -04:00
James Rodewig 8e005db3e6
[DOCS] EQL: Document math functions (#55810) (#56337)
Documents the following EQL functions:

* `add`
* `divide`
* `module`
* `multiply`
* `subtract`
2020-05-07 09:18:43 -04:00
Ryan Ernst 33d6a55d1d
Create plugin for internalClusterTest task (#56067)
This commit creates a new gradle plugin to provide a separate task name
and source set for running ESIntegTestCase tests. The only project
converted to use the new plugin in this PR is server, as an example. The
remaining cases in x-pack will be handled in followups.

backport of #55896
2020-05-06 17:20:52 -07:00
Paweł Krześniak e6dce13bda DOCS: minor formatting (#56263)
Removed extra back ticks. Please cherry-pick to other branches.
2020-05-06 13:41:47 -04:00
James Rodewig d29ea0b8ee
[DOCS] Update upgrade docs for 7.8 (#56281) 2020-05-06 11:47:39 -04:00
Kamyar Ghajar d105e0ea9c [Docs] Update multi-search.asciidoc (#56284)
The documentation shows the wrong command for a multi-search async call.
2020-05-06 16:54:09 +02:00
James Rodewig 2173a587af [DOCS] Move async search API docs after search request body API docs (#56244) 2020-05-06 09:03:23 -04:00
Luca Cavanna ef66018119 [DOCS] Async search: clarify behaviour when submit returns final results (#55934)
Closes #55636
2020-05-06 10:01:23 +02:00
James Rodewig 8686200a32 [DOCS] EQL: Document `concat` function (#56239)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-05 16:45:29 -04:00
James Rodewig dac4ed282e [DOCS] EQL: Add collapsible sections to EQL tutorial docs (#56235)
Adds collapsible sections to the snippet examples of the EQL tutorial
docs.

Also adds a leading slash to EQL API snippet examples.
2020-05-05 16:29:51 -04:00
James Rodewig e7df8b388e [DOCS] EQL: Add collapsible sections to EQL search API response (#56232)
Add collapsible sections to the response parameter docs
of the EQL search API.

Also clarifies some language regarding documents and
events.
2020-05-05 16:01:55 -04:00
Andrei Dan a7968a1a5e
[7.x] HLRC: document index template v2 and component template APIs (#56136) (#56225)
This documents the index template v2 and component template APIs in the
high level rest client.

(cherry picked from commit 9bcf89b1e27613ab8887ce611ec2b0d1356cba8b)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-05-05 19:51:54 +01:00
Thiago Souza 1feb0a95b5 [DOCS] Correct get enrich policy API request example (#56207) 2020-05-05 12:37:54 -04:00
James Rodewig cd3663e5fa
[DOCS] EQL: Document `match` function (#56134) 2020-05-05 12:03:02 -04:00
James Rodewig 44414acd3b
[DOCS] EQL: Document nested field support (#56138)
Notes that you cannot use EQL in ES to search the values of `nested`
fields or their sub-fields. However, indices containing `nested` field
mappings are otherwise supported.
2020-05-05 11:46:06 -04:00
James Rodewig a528319827
[DOCS] Remove invalid search API query parameters (#55884) (#56212)
This is a backport of #55884 with redirects removed.

Changes:

* Adds an abbreviated title for the search API page.
* Removes the following invalid query parameters:
  * `analyzer`
  * `analyze_wildcard`
  * `default_operator`
  * `df`
  * `lenient`
  * `suggest_mode`
  * `suggest_size`
* Replaces the URI search page's query parameter docs with a xref
* Updates the headings of several examples
2020-05-05 11:10:34 -04:00
William Brafford 3499fa917c
Deprecated xpack "enable" settings should be no-ops (#55416) (#56167)
The following settings are now no-ops:

* xpack.flattened.enabled
* xpack.logstash.enabled
* xpack.rollup.enabled
* xpack.slm.enabled
* xpack.sql.enabled
* xpack.transform.enabled
* xpack.vectors.enabled

Since these settings no longer need to be checked, we can remove settings
parameters from a number of constructors and methods, and do so in this
commit.

We also update documentation to remove references to these settings.
2020-05-05 10:40:49 -04:00
Rory Hunter fd91617b19 Fix typo in docs (#56190)
Some dashes were swapped for question marks in bf4c364. Swap them back.
2020-05-05 15:28:32 +01:00
James Rodewig cc1af50214
[DOCS] Add tip for upgrade assistant (#55027)
Our upgrade docs don't mention the upgrade assistant, which can be
helpful when migrating across major versions. The docs also don't
mention deprecation logs, which can highlight other functionality that
may change.

This adds a related tip admonition to the upgrade docs.
2020-05-05 10:09:41 -04:00
David Roberts 7aa0daaabd
[7.x][ML] More advanced model snapshot retention options (#56194)
This PR implements the following changes to make ML model snapshot
retention more flexible in advance of adding a UI for the feature in
an upcoming release.

- The default for `model_snapshot_retention_days` for new jobs is now
  10 instead of 1
- There is a new job setting, `daily_model_snapshot_retention_after_days`,
  that defaults to 1 for new jobs and `model_snapshot_retention_days`
  for pre-7.8 jobs
- For days that are older than `model_snapshot_retention_days`, all
  model snapshots are deleted as before
- For days that are in between `daily_model_snapshot_retention_after_days`
  and `model_snapshot_retention_days` all but the first model snapshot
  for that day are deleted
- The `retain` setting of model snapshots is still respected to allow
  selected model snapshots to be retained indefinitely

Backport of #56125
2020-05-05 14:31:58 +01:00
James Rodewig 4dfdd46dc3 [DOCS] EQL: Remove case sensitivity from function docs (#55063)
Per #54411, we plan to handle case sensitivity via a parameter for the
EQL search API (with the possible exception of the `between` function).

This removes references and examples related to case sensitivity from
the EQL functions docs.
2020-05-05 09:26:49 -04:00
James Rodewig d04ac8b496 [DOCS] Correct `refresh` parm def for GET API (#56140) 2020-05-05 09:13:49 -04:00
James Rodewig bd17d098c4
[DOCS] Remove duplicate `index alias` link in glossary (#56143) 2020-05-05 09:07:19 -04:00
Dimitris Athanasiou 75dadb7a6d
[7.x][ML] Add loss_function to regression (#56118) (#56187)
Adds parameters `loss_function` and `loss_function_parameter`
to regression.

Backport of #56118
2020-05-05 14:59:51 +03:00
Hendrik Muhs e177a38504
[7.x][Transform] add throttling (#56007) (#56184)
add throttling to transform, throttling will slow down search requests by
delaying the execution based on a documents per second metric.

fixes #54862
2020-05-05 13:09:02 +02:00
Marios Trivyzas 363e994171
SQL: Fix DATETIME_PARSE behaviour regarding timezones (#56158) (#56182)
Previously, when the timezone was missing from the datetime string
and the pattern, UTC was used, instead of the session defined timezone.
Moreover, if a timezone was included in the datetime string and the
pattern then this timezone was used. To have a consistent behaviour
the resulting datetime will always be converted to the session defined
timezone, e.g.:
```
SELECT DATETIME_PARSE('2020-05-04 10:20:30.123 +02:00', 'HH:mm:ss dd/MM/uuuu VV') AS datetime;
```
with `time_zone` set to `-03:00` will result in
```
2020-05-04T05:20:40.123-03:00
```

Follows: #54960
(cherry picked from commit 8810ed03a209cc8fe1bad309a81e85b56a39da27)
2020-05-05 12:08:39 +02:00
Théophile Helleboid - chtitux 8a23da429a Docs fix node_id spec for secure settings reload API (#55712)
Fix docs typo for the `node_id` parameter in the secure settings reload API.
2020-05-05 11:21:02 +03:00
István Zoltán Szabó 9bcc975bd1 [DOCS] Simplifies footnote text in DFA APIs (#56105)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-05 09:05:08 +02:00
Lee Hinman 8fa14b333d
[7.x] Validate non-negative priorities for V2 index templates (#56139) (#56163)
Backports the following commits to 7.x:
 - Validate non-negative priorities for V2 index templates (#56139)
2020-05-04 16:19:13 -06:00
James Rodewig 922a80c3f4
[DOCS] Add collapsible sections to search API response (#55887) 2020-05-04 16:57:10 -04:00
Dan Hermann 9892813842
[7.x] Delay warning about missing x-pack (#56142)
* Delay warning about missing x-pack (#54265)

Currently, when monitoring is enabled in a freshly-installed cluster,
the non-master nodes log a warning message indicating that master may
not have x-pack installed. The message is often printed even when the
master does have x-pack installed but takes some time to setup the local
exporter for monitoring. This commit adds the local exporter setting
`wait_master.timeout` which defaults to 30 seconds. The setting
configures the time that the non-master nodes should wait for master to
setup monitoring. After the time elapses, they log a message to the user
about possible missing x-pack installation on master.

The logging of this warning was moved from `resolveBulk()` to
`openBulk()` since `resolveBulk()` is called only on cluster updates and
the message might not be logged until a new cluster update occurs.

Closes #40898
2020-05-04 14:16:18 -05:00
Lee Hinman 3cefe192a2
[7.x] Remove Index Templates V2 feature flag (#56123) (#56141)
Backports the following commits to 7.x:
 - Remove Index Templates V2 feature flag (#56123)
2020-05-04 13:15:51 -06:00
Lisa Cawley b816ab0c18
[DOCS] Synchs and links hyperparameter descriptions (#56131) 2020-05-04 10:37:26 -07:00
Julie Tibshirani 6b5cf1b031 For constant_keyword, make sure exists query handles missing values. (#55757)
It's possible for a constant_keyword to have a 'null' value before any documents
are seen that contain a value for the field. In this case, no documents have a
value for the field, and 'exists' queries should return no documents.
2020-05-04 09:41:52 -07:00
James Rodewig 4faf5a7916
[DOCS] Reformat `porter_stem` token filter (#56053)
Makes the following changes to the `porter_stem` token filter docs:

* Rewrites description and adds a Lucene link
* Adds detailed analyze example
* Adds an analyzer example
2020-05-04 10:39:17 -04:00
bellengao 722de7dd98 [Docs] Fix typo in match-bool-prefix-query doc (#56077) 2020-05-04 14:19:23 +02:00
bellengao 40f99119ae [Docs] Fix typo in getting-started-slm doc (#56075) 2020-05-04 14:18:00 +02:00
markharwood e197b6c45b
Analysis enhancement - add preserve_original setting in ngram-token-filter (#55432) (#56100)
Authored-by: Amit Khandelwal <amitmbm87@gmail.com>
2020-05-04 11:31:28 +01:00
Christos Soulios c65f828cb7
[7.x] Histogram field type support for ValueCount and Avg aggregations (#56099)
Backports #55933 to 7.x

Implements value_count and avg aggregations over Histogram fields as discussed in #53285

- value_count returns the sum of all counts array of the histograms
- avg computes a weighted average of the values array of the histogram by multiplying each value with its associated element in the counts array
2020-05-04 13:23:02 +03:00
Armin Braun 3a64ecb6bf
Allow Deleting Multiple Snapshots at Once (#55474) (#56083)
* Allow Deleting Multiple Snapshots at Once (#55474)

Adds deleting multiple snapshots in one go without significantly changing the mechanics of snapshot deletes otherwise.
This change does not yet allow mixing snapshot delete and abort. Abort is still only allowed for a single snapshot delete by exact name.
2020-05-03 20:30:58 +02:00
William Brafford d53c941c41
Make xpack.monitoring.enabled setting a no-op (#55617) (#56061)
* Make xpack.monitoring.enabled setting a no-op

This commit turns xpack.monitoring.enabled into a no-op. Mostly, this involved
removing the setting from the setup for integration tests. Monitoring may
introduce some complexity for test setup and teardown, so we should keep an eye
out for turbulence and failures

* Docs for making deprecated setting a no-op
2020-05-01 16:42:11 -04:00
David Turner 69f50fe79f Improve same-shard allocation explanations (#56010)
I see occasional confusion about the explanations emitted by the same-shard
allocation decider, particularly amongst new users setting up a single-node
cluster and trying to determine why their cluster has `yellow` health. For
example:

    the shard cannot be allocated to the same node on which a copy of the shard
    already exists

This is technically correct but it's quite a complicated sentence. Also, by
starting with "the shard cannot be allocated" it makes it sound like this is
the problem, whereas in fact this message is a good thing and users should
typically focus their attention elsewhere.

This commit simplifies the wording of these messages and makes them sound more
positive, for example:

    a copy of this shard is already allocated to this node
2020-05-01 10:07:14 +01:00
Vanessa Bell 9b005f70ef [Docs] Grammar improvements in disk-usage.asciidoc (#55620) 2020-04-30 19:31:56 +02:00
Karel Minarik 81dcc9aef7 [DOCS] Add link to the Go client bulk helper (#55872) 2020-04-30 13:24:00 -04:00
James Rodewig 61cf646f17
[DOCS] EQL: Add advantages to overview (#53452) (#56052)
Adds a concise list of EQL advantages, based on the "EQL Advantages"
section in the [EQL for the masses][0] blog post.

The intent is to inform users how EQL could benefit at a high level.

[0]: https://www.elastic.co/blog/eql-for-the-masses

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-04-30 13:19:31 -04:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API (#55732) (#56048)
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746
2020-04-30 12:53:36 -04:00
James Rodewig e4e02e133e
[DOCS] Remove approximate document counts example from term agg docs (#55442)
Removes an example from the "Document counts are approximate" section of the
terms agg documentation.

As #52377 details, the example was no longer accurate in 7.x or 6.8. Document
counts were more precise than the example presented.

We've opened issue #56025 to discuss re-adding an example later.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: AB Prashanth <panuradh@buffalo.edu>
2020-04-30 10:11:50 -04:00
William Brafford 273ff6a105
Make xpack.ilm.enabled setting a no-op (#55592) (#55980)
* Make xpack.ilm.enabled setting a no-op

* Add watcher setting to not use ILM

* Update documentation for no-op setting

* Remove NO_ILM ml index templates

* Remove unneeded setting from test setup

* Inline variable definitions for ML templates

* Use identical parameter names in templates

* New ILM/watcher setting falls back to old setting

* Add fallback unit test for watcher/ilm setting
2020-04-30 09:50:18 -04:00
Lisa Cawley 006e00ed0a
[DOCS] Adds documentation for secondary authorization headers (#55365) (#55986) 2020-04-29 16:29:38 -07:00
James Rodewig 65b47d20a6 [DOCS] Update attribute for multi arg footnotes (#55860) 2020-04-29 10:25:36 -04:00
James Rodewig 0cb5404925
[DOCS] Add Zabbix monitoring template to community monitoring integrations (#55782)
Co-authored-by: RogerTheUnicornHive <laris2@gmail.com>
2020-04-29 10:14:49 -04:00
James Rodewig 1808a1f36b [DOCS] EQL: Correct `cidrMatch` function heading (#55935) 2020-04-29 10:02:06 -04:00
István Zoltán Szabó 337dc45f5b [DOCS] Adds missing space and a relevant link to the slm execute API page (#55917)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-29 15:50:06 +02:00
James Rodewig bbf68de446 [DOCS] Correct Lucene link in `kstem` token filter docs 2020-04-29 09:30:37 -04:00
Luca Cavanna 8b05027bf0 [DOCS] Clarify async search response flags (#55574)
Relates to #55572
2020-04-29 15:22:05 +02:00
James Rodewig 767836c367
[DOCS] Reformat `kstem` token filter (#55823)
Makes the following changes to the `kstem` token filter docs:

* Rewrite description and adds a Lucene work
* Adds detailed analyze example
* Adds an analyzer example
2020-04-29 08:52:55 -04:00
Christos Soulios 02bf0c586a
[7.x] Histogram field type support for Sum aggregation (#55916)
Implements Sum aggregation over Histogram fields by summing the value of each bucket multiplied by their count as requested in #53285

Backports #55681 to 7.x
2020-04-29 15:06:12 +03:00
Henning Andersen f679880b80 [DOCS] Create index name required (#55886)
The name of the new index to create is required.

Relates #45749
2020-04-29 13:35:49 +02:00
István Zoltán Szabó e982cf4381 [DOCS] Makes the footnotes less verbose in configuring aggs page. (#55857) 2020-04-29 09:52:29 +02:00
debadair 8a662c7e62
ILM update backports (#55902)
* [DOCS] Rework conceptual info for ILM. (#52181)

* [DOCS] Rework conceptual info for ILM.

* Split the actions out of concepts.

* Added xpack role to actions.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review

* Edit actions for consistency and add action template. (#55632)

* Edit actions for consistency and add action template.

* Update docs/reference/ilm/actions/ilm-readonly.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review
2020-04-28 16:38:01 -07:00
Lee Hinman 4315a55a1c
[7.x] Initial documentation for index templates V2 (#55755) (#55898)
Backports the following commits to 7.x:
 - Initial documentation for index templates V2 (#55755)
2020-04-28 16:10:50 -06:00
Henning Andersen cab7bcc156
Disk decider respect watermarks for single data node (#55805) (#55847)
The disk decider had special handling for the single data node case,
allowing any allocation (skipping watermark checks) for such clusters.
This special handling can now be avoided via a setting.
2020-04-28 18:46:22 +02:00
Lee Hinman 777caf0725
[7.x] Add support for V2 index templates to /_cat/templates (#55829) (#55866)
Backports the following commits to 7.x:
 - Add support for V2 index templates to /_cat/templates (#55829)
2020-04-28 10:14:19 -06:00
Larry Gregory 47d252424b
Backport: Deprecate the kibana reserved user (#54967) (#55822) 2020-04-28 10:30:25 -04:00
James Rodewig ddc7305ac9 [DOCS] Correct search API's timeout parm default (#55855) 2020-04-28 09:44:50 -04:00
James Rodewig 386fb16409 [DOCS] SQL: Update link for supported regex in `RLIKE` docs (#55830)
The`RLIKE` function docs points users to [Java’s Pattern class doc][0]
for regular expression syntax. However, these docs include shorthand
character classes, such as `[\d]`, `[\s]`, and `[\w]`. These character
classes are not supported in Elasticsearch, which may confuse users.

This updates the SQL `RLIKE` docs to refer to the ES [regular expression
syntax docs][1], which only documents supported syntax.

[0]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/regexp-syntax.html

Relates to #55231
2020-04-28 09:25:51 -04:00
James Rodewig 452be22a4d [DOCS] Warn about searching across all fields wt. `query_string` (#55853)
Warn about potential performance impact when a large number of fields
is used with query string query and no default field.

Re-adds content from #35570.
That content was erroneously removed in #45296.

Co-authored-by: Peter Dyson <peter.dyson@geekpete.com>
2020-04-28 09:20:21 -04:00
Adrien Grand 58c3bb5ae1
Repurpose `ignore_throttled` to be only about frozen indices. (#55047) (#55852)
This has no practical impact on users since frozen indices are the only
throttled indices today. However this has an impact on upcoming features
that would use search throttling.

Filtering out throttled indices made sense a couple years ago, but as
we're now improving support for slow requests with `_async_search` and
exploring ways to reduce storage costs, this feature has most likely
become a trap, that we'd like to not have with upcoming features that
would use search throttling.

Relates #54058
2020-04-28 14:31:54 +02:00
Amit Khandelwal 126e4acca8 Expose `preserve_original` in `edge_ngram` token filter (#55766)
The Lucene `preserve_original` setting is currently not supported in the `edge_ngram`
token filter. This change adds it with a default value of `false`.

Closes #55767
2020-04-28 10:24:27 +02:00
István Zoltán Szabó a5cf4712e5 [DOCS] Changes feature importance links to point to the new page (#55531)
* [DOCS] Changes feature importance links to point to the new page.

* [DOCS] Fixes line breaks.
2020-04-28 09:03:43 +02:00
James Rodewig c16b1edae0 [DOCS] EQL: Fix whitespace in `stringContains` docs 2020-04-27 15:53:59 -04:00
James Rodewig 8df5cff9c1 [DOCS] Correct stemmer token filters anchor 2020-04-27 14:57:59 -04:00
James Rodewig 5b8a18c756 [DOCS] Correct stemmer token filter anchor 2020-04-27 14:51:51 -04:00
David Roberts 3ba44a5af8
[ML] Adding failed_category_count to model_size_stats (#55761)
The failed_category_count statistic records the number of times
categorization wanted to create a new category but couldn't
because the job had reached its model_memory_limit.

Backport of #55716
2020-04-25 10:36:49 +01:00
James Rodewig c1b0548db0
[DOCS] Document EQL search REST API (#52384) 2020-04-24 15:36:01 -04:00
James Rodewig 5981412bf7
[DOCS] EQL: Document `stringContains` function (#54968) 2020-04-24 15:09:05 -04:00
James Rodewig e4ebe55d04
[DOCS] EQL: Document `cidrMatch` function (#54216) (#55739) 2020-04-24 14:01:11 -04:00
James Rodewig e0a8adb5b2
[DOCS] Reformat `stemmer` token filter (#55693)
Makes the following changes to the `stemmer` token filter docs:

* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer example
* Adds a `language` value for the `estonian` stemmer
* Reorders the `language` values to show recommended algorithms first,
  followed by other values alphabetically
2020-04-24 11:25:01 -04:00
James Rodewig 96285b90c1
[DOCS] Add stemming concept docs (#55156)
Adds conceptual documentation for stemming, including:

* An overview of why stemming is helpful in search
* Algorithmic vs. dictionary stemming
* Token filters used to control stemming, such as `stemmer_override`, `keyword_marker`, and `conditional`
2020-04-24 11:01:28 -04:00
Christoph Büscher f95a741ad3
[Docs] Fix fuzziness example in match-query.asciidoc (#55715)
The example looks the same as in the previous section although it should use the
"fuzziness" parameter. This seems to be okay on 6.8 and master and was probably
only forgotten to port to 7.x branches.
2020-04-24 16:21:40 +02:00
Zachary Tong 715c90bf7d Aggs must specify a `field` or `script` (or both) (#52226)
This adds a validation to VSParserHelper to ensure that a field or
script or both are specified by the user.  This is technically
required today already, but throws an exception much deeper
in the agg framework and has a very unintuitive error for the user
(as well as eating more resources instead of failing early)
2020-04-23 19:23:41 -04:00
James Rodewig e74fdacabd
[DOCS] Add admonition for EQL exact matches on text fields (#53402) (#55670)
Adds a important admonition to the EQL syntax page noting that
the equal (`==`) operator should not be used to match `text` field
values.

Relates to #52709 and #53020
2020-04-23 10:59:50 -04:00
István Zoltán Szabó 5813dfdcc7
[7.x][DOCS] Adds ML related items to release highlights (#55652) 2020-04-23 11:58:32 +02:00
Lisa Cawley 314ca78e31
[7.x][DOCS] Update example and nesting in get data frame analytics job stats API (#55612) 2020-04-22 10:58:26 -07:00
James Rodewig 8d05d7dace
[DOCS] Add collapsible sections to 7.x breaking changes (#55334)
Adds collapsible sections and new format to the 7.x breaking changes.

Relates to #53229.
2020-04-22 10:56:38 -04:00
James Rodewig 6f9513915d
[DOCS] Add 'how to' doc about avoiding oversharding (#55480)
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
2020-04-22 10:44:16 -04:00
James Rodewig 414f9c98f3
[DOCS] Document missing bulk API response parameters (#55414)
Documents several parameters missing from the bulk API's response body
docs. Also moves several response-related chunks of text to the response
body section.

Relates to #55237
2020-04-22 09:48:03 -04:00
David Roberts 2dc5586afe
[ML] Add effective max model memory limit to ML info (#55581)
The ML info endpoint returns the max_model_memory_limit setting
if one is configured.  However, it is still possible to create
a job that cannot run anywhere in the current cluster because
no node in the cluster has enough memory to accommodate it.

This change adds an extra piece of information,
limits.effective_max_model_memory_limit, to the ML info
response that returns the biggest model memory limit that could
be run in the current cluster assuming no other jobs were
running.

The idea is that the ML UI will be able to warn users who try to
create jobs with higher model memory limits that their jobs will
not be able to start unless they add a bigger ML node to their
cluster.

Backport of #55529
2020-04-22 12:28:50 +01:00
David Roberts da5aeb8be7
[ML] Return assigned node in start/open job/datafeed response (#55570)
Adds a "node" field to the response from the following endpoints:

1. Open anomaly detection job
2. Start datafeed
3. Start data frame analytics job

If the job or datafeed is assigned to a node immediately then
this field will return the ID of that node.

In the case where a job or datafeed is opened or started lazily
the node field will contain an empty string.  Clients that want
to test whether a job or datafeed was opened or started lazily
can therefore check for this.

Backport of #55473
2020-04-22 12:06:53 +01:00
István Zoltán Szabó 0ce3406033 [DOCS] Provides further details on aggregations in datafeeds (#55462)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-22 08:54:52 +02:00
James Rodewig 777ffd5801
[DOCS] Add bulk API example with failures (#55412)
Adds an example for bulk API requests that include failures.
Also documents guidance on use the `filter_path` parameter
to narrow the bulk API response for errors.

Closes #55237
2020-04-21 16:22:23 -04:00
James Baiera 2a5f1f49a9
Add enrich metricset from 7.5 (#54791) (#55356)
Co-authored-by: Julien Guay <guay_j@yahoo.fr>
2020-04-21 12:39:08 -04:00
James Rodewig b9dfd12e7e
[DOCS] Remove 'Testing' chapter (#55270) (#55532)
Removes the 'Testing' chapter from the Elasticsearch Reference guide.

This chapter was originally written for so that users using the Java HLRC client could
use the same test classes when testing Elasticsearch in their own applications.
However, this is no longer the case or recommended.

Closes #55257.
2020-04-21 10:29:58 -04:00
Paul Sanwald 0f7917b94b
add release notes for 7.5.2 (#51259)
Adds release notes for 7.5.2
2020-04-21 08:19:46 -04:00
Benjamin Trent 24d41eb695
[ML] partitions model definitions into chunks (#55260) (#55484)
This paves the data layer way so that exceptionally large models are partitioned across multiple documents.

This change means that nodes before 7.8.0 will not be able to use trained inference models created on nodes on or after 7.8.0.

I chose the definition document limit to be 100. This *SHOULD* be plenty for any large model. One of the largest models that I have created so far had the following stats:
~314MB of inflated JSON, ~66MB when compressed, ~177MB of heap.
With the chunking sizes of `16 * 1024 * 1024` its compressed string could be partitioned to 5 documents.
Supporting models 20 times this size (compressed) seems adequate for now.
2020-04-20 16:08:54 -04:00
David Turner 8e618fdf10 Adjust docs for voting config exclusions API (#55006)
In #50836 we deprecated the existing voting config exclusions API and added a
new one. This commit adjust the docs to match.
2020-04-20 19:47:33 +01:00
Lee Hinman 9eddd2bcc9
[7.x] Add prefer_v2_templates flag and index setting (#55411) (#55476)
This commit adds a new querystring parameter on the following APIs:
- Index
- Update
- Bulk
- Create Index
- Rollover

These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference
creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be
changed to `true` for 8.0+ in subsequent work.

Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting.
This setting is used so that actions that automatically create a new index (things like rollover
initiated by ILM) will inherit the preference from the original index. This setting is dynamic so
that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias
performing periodic rollover.

This also adds support for sending this parameter to the High Level Rest Client.

Relates to #53101
2020-04-20 12:05:42 -06:00
jmceniery 99409e8c95 [DOCS] Remove Wikipedia link from `SUM_OF_SQUARES` SQL function docs (#52398)
Removed the link to Wikipedia as the function is not calculating the sum of squares in this way. More can be found here at this issue:

https://github.com/elastic/elasticsearch/issues/50416
2020-04-20 09:59:59 -04:00
Ben Skelker 74f55ec6fa [DOCS] Add `ip_range` datatype to core datatypes range list (#55446) 2020-04-20 08:55:09 -04:00
William Brafford 49e30b15a2
Deprecate disabling basic-license features (#54816) (#55405)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 15:04:17 -04:00
Andrei Dan ef338ee3d4
ILM DOCS: mention forcemerge is best effort (#54794) (#55401)
(cherry picked from commit 3fd05435c52dd265dbe1a40104e7dc7a335d50ae)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-17 15:42:23 +01:00
James Rodewig f87a3f0c48 [DOCS] Document analysis/mapping response for cluster stats API (#55054)
PR #51260 moved usage counts about mapping field types and analysis to
the `_cluster/stats` API.

This documents those stats in the response section of the cluster stats
API docs.
2020-04-17 08:44:10 -04:00
Adrien Grand 0cb6a1f089
Document the index corruption bug that gets fixed via Lucene 8.5.1. (#55232)
Using soft deletes on shrunk indices may cause corruption.
2020-04-17 13:37:37 +02:00
markharwood 7761b01a33
Remove normalizer support from wildcard field while we decide on approach for handling case insensitvity (#55294) (#55375)
Closes #55288
2020-04-17 11:43:26 +01:00
Marios Trivyzas f958e9abdc
SQL: Implement scripting inside aggs (#55241) (#55371)
Implement the use of scalar functions inside aggregate functions.
This allows for complex expressions inside aggregations, with or without
GROUBY as well as with or without a HAVING clause. e.g.:

```
SELECT MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) AS max, b
FROM test
GROUP BY b
HAVING MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) > 5
```

Scalar functions are still not allowed for `KURTOSIS` and `SKEWNESS` as
this is currently not implemented on the ElasticSearch side.

Fixes: #29980
Fixes: #36865
Fixes: #37271

(cherry picked from commit 506d1beea7abb2b45de793bba2e349090a78f2f9)
2020-04-17 12:41:22 +02:00
Lisa Cawley c7cf6e621d [DOCS] Remove text fields from classification dependent variables (#54849) 2020-04-16 13:40:28 -07:00
Lisa Cawley cf5278f771 [DOCS] Add ml-cpp PRs to 7.7 release notes (#55264)
Co-Authored-By: David Roberts <dave.roberts@elastic.co>
2020-04-16 11:28:34 -07:00
Julie Tibshirani d7cded8d7a
Fix updating include_in_parent/include_in_root of nested field. (#55326)
The main changes are:
1. Throw an error when updating `include_in_parent` or `include_in_root` attribute of nested field dynamically by the PUT mapping API.
2. Add a test for the change.

Closes #53792

Co-authored-by: bellengao <gbl_long@163.com>
2020-04-16 11:17:12 -07:00
James Rodewig f0b9be8b1b [DOCS] Reformat `flatten_graph` token filter (#54268)
* [DOCS] Reformat `flatten_graph` token filter

Makes the following changes to the `flatten_graph` token filter docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Adds analyzer example
2020-04-16 08:35:08 -04:00
Bogdan Pintea b88dd47de3 Docs: add the change log for 7.7 (#55019)
* Add the change log for 7.7

Add the change log for 7.7

* Update rel. notes to latest state (BC5)

Update the release notes to current state (i.e. BC5).

* Update docs/reference/release-notes/7.7.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-15 15:25:08 -04:00
Lisa Cawley f0b9578684 [DOCS] Removes transform performance note (#55177) 2020-04-15 10:42:52 -07:00
Ignacio Vera a677b63daa
Upgrade to lucene 8.5.1 release (#55229) (#55235)
Upgrade to lucene 8.5.1 release that contains a bug fix for a bug that might introduce index corruption when deleting data from an index that was previously shrunk.
2020-04-15 17:35:42 +02:00
James Rodewig 4f2ab96f38 [DOCS] EQL: Document `indexOf` function (#55071) 2020-04-15 11:29:50 -04:00
James Rodewig 8d6f0f6a76 [DOCS] Document `max_concurrent_searches` default (#55116) 2020-04-15 10:04:23 -04:00
Benjamin Trent 8ff2cbf1a3
[7.x] [ML] adding prediction_field_type to inference config (#55128) (#55230)
* [ML] adding prediction_field_type to inference config (#55128)

Data frame analytics dynamically determines the classification field type. This field type then dictates the encoded JSON that is written to Elasticsearch. 

Inference needs to know about this field type so that it may provide the EXACT SAME predicted values as analytics. 

Here is added a new field `prediction_field_type` which indicates the desired type. Options are: `string` (DEFAULT), `number`, `boolean` (where close_to(1.0) == true, false otherwise). 

Analytics provides the default `prediction_field_type` when the model is created from the process.
2020-04-15 09:45:22 -04:00
Jake Landis 85139fad7e
[7.x] Advise a simpler curator migration (#54457) (#55188)
Advice for migrating from Curator should simply be to phase out curator managed indices, 
since curator will ignore ILM indices
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.7/ilm-and-curator.html#ilm-and-curator.

Co-authored-by: Jay Greenberg <PhaedrusTheGreek@users.noreply.github.com>
2020-04-15 07:55:31 -05:00
Lisa Cawley 2910d01179
[DOCS] Removes unshared sections from ml-shared.asciidoc (#55192) 2020-04-14 18:47:09 -07:00
Yang Wang f49354b7d7
Add migration notes for deprecating local parameter of get field mapping API (#55194)
This is a follow-up for #55099 to add migration notes about the deprecation of local parameter for get field mappings API.
2020-04-15 11:38:05 +10:00
Igor Motov 1754e50cbd
[7.x] Add analytics plugin usage stats to _xpack/usage (#54911) (#55162)
Adds analytics plugin usage stats to _xpack/usage.

Closes #54847
2020-04-14 17:03:14 -04:00
James Rodewig 12130843ca
[DOCS] Add maintenance releases to upgrade table (#55012)
Updates the supported upgrade path table in [Upgrade Elasticsearch][0]
to include a new row for maintenance releases. For example, this row
covers upgrading from 7.6.0 to 7.6.2.

The new table row only displays for releases greater than n.x.0. For
example, the new row will display for the 7.7.1 release but not the
7.7.0 release.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-upgrade.html
2020-04-14 11:28:55 -04:00
James Rodewig 3fbd8b371f [DOCS] Use consistent line breaks in EQL function docs 2020-04-14 10:17:45 -04:00
Yannick Welsch a610513ec7 Provide repository-level stats for searchable snapshots (#55051)
Provides basic repository-level stats that will allow us to get some insight into how many
requests are actually being made by the underlying SDK. Currently only tracks GET and LIST
calls for S3 repositories. Most of the code is unfortunately boiler plate to add a new endpoint
that will help us better understand some of the low-level dynamics of searchable snapshots.
2020-04-14 14:34:08 +02:00
lcawl fcd96db006 [DOCS] Edits create data frame analytics job API (#54751) 2020-04-13 10:43:52 -07:00
Nhat Nguyen 96bb1164f0 Support hierarchical task cancellation (#54757)
With this change, when a task is canceled, the task manager will cancel
not only its direct child tasks but all also its descendant tasks.

Closes #50990
2020-04-13 12:35:21 -04:00
Igor Motov 51c6f69e02
[7.x] Add support for filters to T-Test aggregation (#54980) (#55066)
Adds support for filters to T-Test aggregation. The filters can be used to
select populations based on some criteria and use values from the same or
different fields.

Closes #53692
2020-04-13 12:28:58 -04:00
Jake Landis a2fafa6af4
[7.x] Lazy test cluster module and plugins (#54852) (#55087)
This change converts the module and plugin parameters
for testClusters to be lazy. Meaning that the values
are not resolved until they are actually used. This
removes the requirement to use project.afterEvaluate to
be able to resolve the bundle artifact.

Note - this does not completely remove the need for afterEvaluate
since it is still needed for the custom resource extension.
2020-04-13 10:53:35 -05:00
James Rodewig 57d6493e29 [DOCS] EQL: Document `string` function (#55086) 2020-04-13 11:23:45 -04:00
Peter Dyson f0b6cf4c11 [DOCS] Note where ILM policies are stored and backup caveats (#54859) 2020-04-13 09:11:16 -06:00
Vishal Patel 16921ebbd8 [DOCS] Collapse nested objects in Explore API docs (#55067)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-13 09:27:03 -04:00
Ioannis Kakavas 7a8a66d9ae
[7.x] Fix ReloadSecureSettings API to consume password (#54771) (#55059)
The secure_settings_password was never taken into consideration in
the ReloadSecureSettings API. This commit fixes that and adds
necessary REST layer testing. Doing so, it also:

- Allows TestClusters to have a password protected keystore
so that it can be set for tests.
- Adds a parameter to the run task so that elastisearch can
be run with a password protected keystore from source.
2020-04-13 09:50:55 +03:00
Yang Wang 862799956c
Deprecate local parameter for get field mapping request (#55014) (#55099)
The usage of local parameter for GetFieldMappingRequest has been removed from the underlying transport action since v2.0.

This PR deprecates the parameter from rest layer. It will be removed in next major version.
2020-04-12 13:48:47 +10:00
James Rodewig 2655dfa2fe [DOCS] EQL: Reword field support for EQL functions (#55074)
Changes boilerplate sentence of "If using a field as the argument, this
parameter only supports..." to "...this parameter supports only...".

The latter is a bit more clear and readable.
2020-04-10 15:33:29 -04:00
Jason Tedor d1137ebdaa
Passthrough special characters in thread pool docs (#55080)
Some of these characters are special to Asciidoctor and they ruin the
rendering on this page. Instead, we use a macro to passthrough these
characters without Asciidoctor applying any subtitutions to them. This
commit then addresses some rendering issues in the thread pool docs.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-10 15:11:19 -04:00
Nik Everett b99a50bcb9
value_count Aggregation optimization (backport of #54854) (#55076)
We found some problems during the test.

Data: 200Million docs, 1 shard, 0 replica

    hits    |   avg   |   sum   | value_count |
----------- | ------- | ------- | ----------- |
     20,000 |   .038s |   .033s |       .063s |
    200,000 |   .127s |   .125s |       .334s |
  2,000,000 |   .789s |   .729s |      3.176s |
 20,000,000 |  4.200s |  3.239s |     22.787s |
200,000,000 | 21.000s | 22.000s |    154.917s |

The performance of `avg`, `sum` and other is very close when performing
statistics, but the performance of `value_count` has always been poor,
even not on an order of magnitude. Based on some common-sense knowledge,
we think that `value_count` and sum are similar operations, and the time
consumed should be the same. Therefore, we have discussed the agg
of `value_count`.

The principle of counting in es is to traverse the field of each
document. If the field is an ordinary value, the count value is
increased by 1. If it is an array type, the count value is increased
by n. However, the problem lies in traversing each document and taking
out the field, which changes from disk to an object in the Java
language. We summarize its current problems with Elasticsearch as:

- Number cast to string overhead, and GC problems caused by a large
  number of strings
- After the number type is converted to string, sorting and other
  unnecessary operations are performed

Here is the proof of type conversion overhead.

```
// Java long to string source code, getChars is very time-consuming.
public static String toString(long i) {
        int size = stringSize(i);
        if (COMPACT_STRINGS) {
            byte[] buf = new byte[size];
            getChars(i, size, buf);
            return new String(buf, LATIN1);
        } else {
            byte[] buf = new byte[size * 2];
            StringUTF16.getChars(i, size, buf);
            return new String(buf, UTF16);
        }
}
```

  test type  | average |  min |     max     |   sum
------------ | ------- | ---- | ----------- | -------
double->long |  32.2ns | 28ns |     0.024ms |  3.22s
long->double |  31.9ns | 28ns |     0.036ms |  3.19s
long->String | 163.8ns | 93ns |  1921    ms | 16.3s

particularly serious.

Our optimization code is actually very simple. It is to manage different
types separately, instead of uniformly converting to string unified
processing. We added type identification in ValueCountAggregator, and
made special treatment for number and geopoint types to cancel their
type conversion. Because the string type is reduced and the string
constant is reduced, the improvement effect is very obvious.

    hits    |   avg   |   sum   | value_count | value_count | value_count | value_count | value_count | value_count |
            |         |         |    double   |    double   |   keyword   |   keyword   |  geo_point  |  geo_point  |
            |         |         |   before    |    after    |   before    |    after    |   before    |    after    |
----------- | ------- | ------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
     20,000 |     38s |   .033s |       .063s |       .026s |       .030s |       .030s |       .038s |       .015s |
    200,000 |    127s |   .125s |       .334s |       .078s |       .116s |       .099s |       .278s |       .031s |
  2,000,000 |    789s |   .729s |      3.176s |       .439s |       .348s |       .386s |      3.365s |       .178s |
 20,000,000 |  4.200s |  3.239s |     22.787s |      2.700s |      2.500s |      2.600s |     25.192s |      1.278s |
200,000,000 | 21.000s | 22.000s |    154.917s |     18.990s |     19.000s |     20.000s |    168.971s |      9.093s |

- The results are more in line with common sense. `value_count` is about
  the same as `avg`, `sum`, etc., or even lower than these. Previously,
  `value_count` was much larger than avg and sum, and it was not even an
  order of magnitude when the amount of data was large.
- When calculating numeric types such as `double` and `long`, the
  performance is improved by about 8 to 9 times; when calculating the
  `geo_point` type, the performance is improved by 18 to 20 times.
2020-04-10 13:16:39 -04:00
James Rodewig c440754784 [DOCS] EQL: Document `wildcard` function (#54086) 2020-04-10 09:18:29 -04:00
oneoneonepig 356cc94889 [DOCS] Fix double quote typo in 7.0 breaking changes (#55040) 2020-04-10 09:11:51 -04:00
Jason Tedor 9eeae59a83
Clarify available processors (#54907)
The use of available processors, the terminology, and the settings
around it have evolved over time. This commit cleans up some places in
the codes and in the docs to adjust to the current terminology.
2020-04-10 08:48:27 -04:00
James Rodewig 51326432be [DOCS] Add query reference docs template (#52292) 2020-04-10 08:47:54 -04:00
James Rodewig d5a609a2e5 [DOCS] Add token filter reference docs template (#52290)
Creates a reusable template for token filter reference documentation.

Contributors can make a copy of this template and customize it when
documenting new token filters.
2020-04-10 08:45:10 -04:00
Marios Trivyzas bf0cadb602
SQL: Implement DATETIME_PARSE function for parsing strings (#54960) (#55035)
Implement DATETIME_PARSE(<datetime_str>, <pattern_str>) function
which allows to parse a datetime string according to the specified
pattern into a datetime object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Relates to #53714

(cherry picked from commit 3febcd8f3cdf9fdda4faf01f23a5f139f38b57e0)
2020-04-10 01:16:29 +02:00
Vishal Patel 51cb0c5c7b [DOCS] Collapse nested objects in cluster reroute docs (#54851) 2020-04-09 15:29:22 -04:00
István Zoltán Szabó 374f633b6e [DOCS] Adds link points to the data frame analytics supported fields (#55004)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-09 11:27:57 -07:00
Mark Vieira dd73a14d11
Improve total build configuration time (#54611) (#54994)
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

- Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 16:47:02 -07:00
James Rodewig c6cd8ca7c0
[DOCS] Update upgrade docs for 7.7 (#54978) 2020-04-08 16:23:08 -04:00
James Rodewig 964cf565c9
[DOCS] EQL: Document `between` function (#54950) 2020-04-08 13:49:15 -04:00
Théophile Helleboid - chtitux a8aa36d427 [DOCS] Fix typo in SLM retention docs (#54797) 2020-04-08 08:56:45 -04:00
Marios Trivyzas 6afd60b082
SQL: Implement DATETIME_FORMAT function for date/time formatting (#54832) (#54942)
Implement DATETIME_FORMAT(<date/datetime/time>, ) function
which allows for formatting a timestamp to the specified format. The
patterns allowed as those of java.time.format.DateTimeFormatter.

Related to #53714

(cherry picked from commit 72be0b54a9299e87e785469cdc9aafac2a48c046)
2020-04-08 13:45:47 +02:00
István Zoltán Szabó 3a3effedc2 [DOCS] Reworks some parts of EMM API docs (#54872)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-08 10:20:34 +02:00
Julie Tibshirani 475b210eec
Improve guidance on removing default mappings. (#54915)
In 7.x, an index template will fail to apply if it contains a `_default_`
mapping. Several users have expressed confusion over the fact that loading the
template doesn't show any default mappings. This docs change clarifies that in
order to see all mappings in the template, you must pass `include_type_name`.
2020-04-07 15:18:13 -07:00
James Rodewig 9569a8eb13 [DOCS] Add example to "avoid scripts" advice (#54719)
Adds a detailed example to the "Avoid scripts" section of the "Tune
for search speed" docs. The detail outlines how a script used to
transform indexed data can be moved to ingest.

The update also removes an outdated reference to supported script
languages.
2020-04-07 15:25:10 -04:00
Jason Tedor d1d478debf
Update docs to reflect node.processors (#54855)
We namespaced the previous setting "processors" into
"node.processors". This commit updates some of the documentation to
reflect this.
2020-04-07 13:06:14 -04:00
Lisa Cawley a7599031ae [DOCS] Adds tranform node to list of default types (#54850) 2020-04-07 08:49:05 -07:00
Ignacio Vera 076c199484
Add new point field. (#53804) (#54879)
This commit adds a new point field that is able to index arbitrary pair of values (x/y)
in the cartesian space. It only supports filtering using shape queries at the moment.
2020-04-07 15:28:50 +02:00
Tanguy Leroux 4d36917e52
Merge feature/searchable-snapshots branch into 7.x (#54803) (#54825)
This is a backport of #54803 for 7.x.

This pull request cherry picks the squashed commit from #54803 with the additional commits:

    6f50c92 which adjusts master code to 7.x
    a114549 to mute a failing ILM test (#54818)
    48cbca1 and 50186b2 that cleans up and fixes the previous test
    aae12bb that adds a missing feature flag (#54861)
    6f330e3 that adds missing serialization bits (#54864)
    bf72c02 that adjust the version in YAML tests
    a51955f that adds some plumbing for the transport client used in integration tests

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-07 13:28:53 +02:00
Ioannis Kakavas 3560c0cbf2
Remove `_xpack` from license API example (#54698) (#54763)
Resolves #54662
2020-04-07 09:51:37 +03:00
Lisa Cawley b3d5300968
[DOCS] Collapses sections in put snapshot lifecycle policy API (#54834) (#54840) 2020-04-06 13:46:56 -07:00
James Rodewig e9c3bfc8e5 [DOCS] Collapse nested objects in node stats API response (#54755)
Replaces dot notation with collapsed nested object formatting
per the [Elastic API reference template][0].

[0]:https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc
2020-04-06 15:19:54 -04:00
James Rodewig 548ad03941 [DOCS] Collapse nested objects in cluster stats API response (#54739)
Replaces dot notation with collapsed nested object formatting
per the [Elastic API reference template][0].

[0]:https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc
2020-04-06 13:11:46 -04:00
Igor Motov 2794572a35
[7.x] Add Student's t-test aggregation support (#54469) (#54737)
Adds t_test metric aggregation that can perform paired and unpaired two-sample
t-tests. In this PR support for filters in unpaired is still missing. It will
be added in a follow-up PR.

Relates to #53692
2020-04-06 11:36:47 -04:00
Nhat Nguyen 2fdbed7797 Broadcast cancellation to only nodes have outstanding child tasks (#54312)
Today when canceling a task we broadcast ban/unban requests to all nodes
in the cluster. This strategy does not scale well for hierarchical
cancellation. With this change, we will track outstanding child requests
and broadcast the cancellation to only nodes that have outstanding child
tasks. This change also prevents a parent task from sending child
requests once it got canceled.

Relates #50990
Supersedes #51157

Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2020-04-06 11:11:29 -04:00
István Zoltán Szabó 7dc1ba4273 [DOCS] Updates transform prerequisites (#54804) 2020-04-06 17:07:59 +02:00
Christoph Büscher def519ea70 [Docs] Correct date rounding example for `range` query (#51524)
Looking into #50237 I realized that two of the examples given in the
documentation around date math rounding for range queries on date fields using
`gt` and `lt` is slightly off by a nanosecond. This PR changes this to the
bounds that are currently parsed using these parameters.
2020-04-06 17:05:45 +02:00
István Zoltán Szabó 4cba1e6368 [DOCS] Changes kibana_user to kibana_admin in DFA API prerequisites. (#54806) 2020-04-06 15:46:18 +02:00
Jason Tedor 184c038f59
Add get autoscaling policy API (#54762)
This commit adds the get autoscaling policy API.
2020-04-04 18:04:25 -04:00
James Baiera 705e46d5c1
[DOCS] Remove unused cat tasks request parameters (#54539) (#54741) 2020-04-03 15:33:28 -04:00
Lisa Cawley de91d2aeea [DOCS] Collapse nested objects in CCR APIs (#54697) 2020-04-03 12:04:33 -07:00
Bogdan Pintea 7cef89e084 ODBC: Document the new VarcharLimit and EarlyExecution params (#54632)
* Document VarcharLimit and EarlyExecution params

Add the documentation for the newly added VarcharLimit and
EarlyExecution DSN attributes.

* Remove obsolete VersionChecking param

This param had been removed already along the #53082 work.

* Update docs/reference/sql/endpoints/odbc/configuration.asciidoc

fix typo

Co-Authored-By: Stuart Cam <stuart@codebrain.co.uk>

* Update docs/reference/sql/endpoints/odbc/configuration.asciidoc

fix typo

Co-Authored-By: Stuart Cam <stuart@codebrain.co.uk>
(cherry picked from commit f38761631a12b38f7f075635f7ac61dc96656cd7)
2020-04-03 14:46:39 +02:00
markharwood 2da2305587
Backport of lowercase normalizer PR #53882
A pre-configured normalizer for lower-casing.
Closes #53872
2020-04-03 11:43:40 +01:00
István Zoltán Szabó d025b90cd1 [DOCS] Makes PUT inference API docs collapsible (#54653)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-03 09:48:53 +02:00
Lisa Cawley 11afead21e [DOCS] Adds collapsible sections to rollup APIs (#54690) 2020-04-02 17:51:16 -07:00
Lisa Cawley b138dc4565
[DOCS] Add processing details to get transforms stats API (#54368) (#54595) 2020-04-02 16:12:57 -07:00
lcawl d3fa0ec730 [DOCS] Fixes typo in node settings 2020-04-02 16:01:54 -07:00
Lisa Cawley 98965116fe [DOCS] Clarify ML and transform settings on coordinating nodes (#54676) 2020-04-02 15:38:15 -07:00
Julie Tibshirani 5fb7602227
Disallow changing 'enabled' on the root mapper. (#54681)
In #33933 we disallowed changing the `enabled` parameter in object mappings.
However, the fix didn't cover the root object mapper. This PR adjusts the change
to also include the root mapper and clarifies the error message.
2020-04-02 15:28:48 -07:00
David Turner 4e083cd97d indices.recovery.max_bytes_per_sec may be per-node (#54633)
The `indices.recovery.max_bytes_per_sec` recovery bandwidth limit can differ
between nodes if it is not set dynamically, but today this is not obvious. This
commit adds a paragraph to its documentation clarifying how to set different
bandwidth limits on each node.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-02 18:15:41 +01:00
Benjamin Trent 4a1610265f
[7.x] [ML] add new inference_config field to trained model config (#54421) (#54647)
* [ML] add new inference_config field to trained model config (#54421)

A new field called `inference_config` is now added to the trained model config object. This new field allows for default inference settings from analytics or some external model builder.

The inference processor can still override whatever is set as the default in the trained model config.

* fixing for backport
2020-04-02 12:25:10 -04:00
Benjamin Trent 65233383f6
[7.x] [ML] prefer secondary authorization header for data[feed|frame] authz (#54121) (#54645)
* [ML] prefer secondary authorization header for data[feed|frame] authz (#54121)

Secondary authorization headers are to be used to facilitate Kibana spaces support + ML jobs/datafeeds.

Now on PUT/Update/Preview datafeed, and PUT data frame analytics the secondary authorization is preferred over the primary (if provided).

closes https://github.com/elastic/elasticsearch/issues/53801

* fixing for backport
2020-04-02 11:20:25 -04:00
qiye de8e0200fe [DOCS] Correct `shape` field release in 7.5 release highlights (#54631)
The `shape` field was added in 7.4, not 7.3.
This corrects a small error in the 7.5 release highlights.
2020-04-02 09:19:40 -04:00
Benjamin Trent eb31be0e71
[7.x] [ML] add num_matches and preferred_to_categories to category defintion objects (#54214) (#54639)
* [ML] add num_matches and preferred_to_categories to category defintion objects (#54214)

This adds two new fields to category definitions.

- `num_matches` indicating how many documents have been seen by this category
- `preferred_to_categories` indicating which other categories this particular category supersedes when messages are categorized.

These fields are only guaranteed to be up to date after a `_flush` or `_close`

native change: https://github.com/elastic/ml-cpp/pull/1062

* adjusting for backport
2020-04-02 09:09:19 -04:00
Jason Tedor 54ecb009bb
Add delete autoscaling policy API (#54601)
This commit adds an API for deleting autoscaling policies.
2020-04-02 09:05:12 -04:00
Aleh Zasypkin 161eac1942
[7.x] Switch to the most recent Kibana configuration format and SAML/OIDC endpoints. (#54624) 2020-04-02 11:59:11 +02:00
István Zoltán Szabó 3a8e880fe6 [DOCS] Adds snippet comparing two indices to the painless examples (#54563) 2020-04-02 08:46:25 +02:00
Russ Cam 110d9a7845 Correct description for mget API request URI index (#52305)
This commit corrects the description for the request URI index for the Multi Get (mget) API.
The index can only be a single index name (multiple or wildcard expressions not supported),
and acts as the index to use when "ids" are specified, or a document in the "docs" array does
not specify an index.

(cherry picked from commit aa4926ed7f91dfbf7973a01b1e4682e91dda11a9)
2020-04-02 09:08:34 +10:00
lcawl 949636944c [DOCS] Fixes shared attribute for feature importance 2020-04-01 14:52:08 -07:00
Jason Tedor ccecb78c98
Rename the policy in put autoscaling policy docs
The put autoscaling policy docs use a "hot" policy as an
example. Instead, this commit changes the name of this policy to
"my_autoscaling_policy".
2020-04-01 16:32:03 -04:00
Jason Tedor bd6b383926
Include autoscaling APIs in API docs (#54603)
This commit adds a top-level link to the autoscaling API reference page
to the API docs. Additionally, we add a conditional guard on the API
pages to only include them in development builds of the docs.
2020-04-01 15:44:13 -04:00
lcawl 2cd35bf696 [DOCS] Adds release highlights placeholder 2020-04-01 09:22:20 -07:00
Lisa Cawley f5ccf939d9 [DOCS] Clarifies API key breaking change (#54522) 2020-04-01 08:58:15 -07:00