Commit Graph

7855 Commits

Author SHA1 Message Date
James Rodewig 838af15d29 [DOCS] Reformat compound word token filters ()
* Separates the compound token filters doc pages into separate token
  filter pages:
  * Dictionary decompounder token filter
  * Hyphenation decompounder token filter

* Adds analyze API examples for each compound token filter

* Adds a redirect for the removed compound token filters page

Co-Authored-By: debadair <debadair@elastic.co>
2019-11-13 09:36:52 -05:00
István Zoltán Szabó b55022b59f [DOCS] Adds test clause to the code snippets in the cluster restart page () 2019-11-13 14:36:44 +01:00
Julie Tibshirani 37fa3fb4ff
Ensure parameters are updated when merging flattened mappings. () ()
This PR makes the following two fixes around updating flattened fields:

* Make sure that the new value for ignore_above is immediately taken into
  affect. Previously we recorded the new value but did not use it when parsing
  documents.
* Allow depth_limit to be updated dynamically. It seems plausible that a user
  might want to tweak this setting as they encounter more data.
2019-11-12 21:50:39 -05:00
David Roberts 698ebd3d0a [TEST] Mute docs snippet test in close-job.asciidoc ()
Due to https://github.com/elastic/elasticsearch/pull/48583#issuecomment-552991325
2019-11-12 17:34:27 +00:00
Michael Basnight bc23bc5146 Add delete alias to the HLRC ()
The delete alias call is a rest only API call, but should still be added
to the rest client. This commit adds it as well as relevant tests.

Ref 
2019-11-12 11:02:53 -06:00
Orhan Toy 561351d2fc [Docs] Fix _count HTTP method () 2019-11-12 15:45:26 +01:00
István Zoltán Szabó fc145575c4 [DOCS] Creates a cluster restart documentation page ()
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-11-12 14:50:53 +01:00
James Rodewig 42e92616f6 [DOCS] Document indices response parameters for node stats API () 2019-11-12 08:35:35 -05:00
jimczi 0e82b5f59b add release notes for 7.5.0 2019-11-12 09:59:14 +01:00
Benjamin Trent 46ab1db54f
[7.x] [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results () ()
* [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results ()

[ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results ()

Related PR: https://github.com/elastic/ml-cpp/pull/809

* adjusting bwc version
2019-11-11 15:43:03 -05:00
István Zoltán Szabó c2f52015d3 [DOCS] Removes best practice about fields that are highly correlated to the dependent variable. () 2019-11-11 16:01:21 +01:00
István Zoltán Szabó 91888959e8 [DOCS] Extends analyzed_fields description in PUT DFA API docs. () 2019-11-11 15:55:12 +01:00
Patrick Maynard 4b85498617 [DOCS] Fix typo in search type docs () 2019-11-11 09:38:48 -05:00
James Rodewig dd92830801 [DOCS] Reformat condition token filter () 2019-11-11 08:49:44 -05:00
Arne Welzel f642baa9fb [DOCS] Remove extra "when" () 2019-11-11 10:11:02 +01:00
Yannick Welsch 87862868c6 Allow realtime get to read from translog ()
The realtime GET API currently has erratic performance in case where a document is accessed
that has just been indexed but not refreshed yet, as the implementation will currently force an
internal refresh in that case. Refreshing can be an expensive operation, and also will block the
thread that executes the GET operation, blocking other GETs to be processed. In case of
frequent access of recently indexed documents, this can lead to a refresh storm and terrible
GET performance.

While older versions of Elasticsearch (2.x and older) did not trigger refreshes and instead opted
to read from the translog in case of realtime GET API or update API, this was removed in 5.0
() to avoid inconsistencies between values that were returned from the translog and
those returned by the index. This was partially reverted in 6.3 () to allow _update and
upsert to read from the translog again as it was easier to guarantee consistency for these, and
also brought back more predictable performance characteristics of this API. Calls to the realtime
GET API, however, would still always do a refresh if necessary to return consistent results. This
means that users that were calling realtime GET APIs to coordinate updates on client side
(realtime GET + CAS for conditional index of updated doc) would still see very erratic
performance.

This PR (together with ) resolves the inconsistencies between reading from translog and
index. In particular it fixes the inconsistencies that happen when requesting stored fields, which
were not available when reading from translog. In case where stored fields are requested, this
PR will reparse the _source from the translog and derive the stored fields to be returned. With
this, it changes the realtime GET API to allow reading from the translog again, avoid refresh
storms and blocking the GET threadpool, and provide overall much better and predictable
performance for this API.
2019-11-09 17:47:50 +01:00
Julian Simioni 5e4501eb3f [Docs] Consolidate single example into a single line ()
The first example of splitting rules for the `word_delimiter` token filter was spread across two bullet points. This makes it look like they are two separate splitting rules.
2019-11-08 15:12:45 -05:00
Yannick Welsch af887be3e5 Hide orphaned tasks from follower stats ()
CCR follower stats can return information for persistent tasks that are in the process of being cleaned up. This is problematic for tests where CCR follower indices have been deleted, but their persistent follower task is only cleaned up asynchronously afterwards. If one of the following tests then accesses the follower stats, it might still get the stats for that follower task.

In addition, some tests were not cleaning up their auto-follow patterns, leaving orphaned patterns behind. Other tests cleaned up their auto-follow patterns. As always the same name was used, it just depended on the test execution order whether this led to a failure or not. This commit fixes the offensive tests, and will also automatically remove auto-follow-patterns at the end of tests, like we do for many other features.

Closes #48700
2019-11-08 13:56:53 +01:00
bellengao bdc7057d58 [DOCS] Correct typo in split index API docs () 2019-11-07 15:27:27 -05:00
bellengao 293902c6a5 [DOCS] Fix shard type in CCR overview doc ()
Closes 
2019-11-07 10:09:45 -05:00
Tanguy Leroux 552381d7f9 Add mention to Pause Auto-Follower API in Upgrade Clusters docs ()
Relates 
2019-11-06 09:48:44 -05:00
István Zoltán Szabó 3c9bd13dca [DOCS] Adds classification type DFA API docs and ml-shared.asciidoc () 2019-11-06 07:41:38 -05:00
István Zoltán Szabó 70765dfb05 [DOCS] Adds classification type evaluation docs to the DFA evaluation API () 2019-11-06 07:38:33 -05:00
glerb baabc21a04 [DOCS] Correct typo in Discovery docs () 2019-11-05 08:48:43 -05:00
James Rodewig 700a316bb3 [DOCS] Reformat decimal digit token filter docs () 2019-11-01 12:38:14 -04:00
James Rodewig 680999f246 [DOCS] List `indices.lifecycle.poll_interval` as cluster-level ()
Lists `indices.lifecycle.poll_interval` with other cluster-level ILM
settings.

Previously, it was included under index-level settings.
2019-11-01 11:54:46 -04:00
pulysak 9a0a7ab95a [DOCS] Fix typo in Index API reference docs () 2019-11-01 09:16:11 -04:00
Alexander Reelsen 80cde68af7 [DOCS] Remove unneeded // CONSOLE comments from snippets ()
Updates the docs README file to remove outdated `// CONSOLE` instructions
2019-11-01 09:04:24 -04:00
debadair b9f4b32892 [DOCS] Fix cross-doc link. ()
* [DOCS] Fix cross-doc link.

* Fixed xref
2019-10-31 18:59:17 -07:00
Lisa Cawley 40834c229f
[7.x][DOCS] Copies ESMS monitoring details to Elasticsearch Reference () 2019-10-31 18:22:08 -07:00
debadair 457379e74e [DOCS] Edited Docker install & tweaked Docker compose file. ()
* [DOCS] Edited Docker install & tweaked Docker compose file.

* Synced with Docker GS in SO

* Incorporated review comments
2019-10-31 18:12:39 -07:00
Tal Levy 4be54402de
[7.x] Add ingest info to Cluster Stats () ()
* Add ingest info to Cluster Stats ()

This commit enhances the ClusterStatsNodes response to include global
processor usage stats on a per-processor basis.

example output:

```
...
    "processor_stats": {
      "gsub": {
        "count": 0,
        "failed": 0
        "current": 0
        "time_in_millis": 0
      },
      "script": {
        "count": 0,
        "failed": 0
        "current": 0,
        "time_in_millis": 0
      }
    }
...
```

The purpose for this enhancement is to make it easier to collect stats on how specific processors are being used across the cluster beyond the current per-node usage statistics that currently exist in node stats.

Closes .

* fix BWC of ingest stats

The introduction of processor types into IngestStats had a bug.
It was set to `null` and set as the key to the map. This would
throw a NPE. This commit resolves this by setting all the processor
types from previous versions that are not serializing it out to
`_NOT_AVAILABLE`.
2019-10-31 14:36:54 -07:00
Deb Adair 6412d0f528 [DOCS] Remove coming tag from 7.4.2 RN backport. 2019-10-31 09:43:26 -07:00
Lisa Cawley b7559f23cc [DOCS] Fixes PR#48055 in release notes () 2019-10-31 07:37:44 -07:00
Peter Johnson 3f7aafa421 [DOCS] Fix typo in synonym token filter docs () 2019-10-31 09:12:24 -04:00
James Rodewig 3d5b1725a9 [DOCS] Remove unneeded filter from common grams analyze ex () 2019-10-31 09:08:14 -04:00
Brandon Morelli aa02174d53 [DOCS] Fix typo in ILM policy definition docs ()
Removes an extra "by".
2019-10-31 08:30:54 -04:00
Andrei Dan ffe5d5417f
ILM Make the `check-rollover-ready` step retryable () ()
This adds the infrastructure to be able to retry the execution of retryable
steps and makes the `check-rollover-ready` retryable as an initial step to
make the rollover action more resilient to transient errors.

(cherry picked from commit 454020ac8acb147eae97acb4ccd6fb470d1e5f48)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-10-31 11:28:55 +00:00
debadair a876760848 [DOCS] Add placeholder for 7.4.2 release notes () 2019-10-30 16:09:29 -07:00
Jason Tedor 13043219ac
Fix specification for cluster.remote.connect ()
The docs specify that cluster.remote.connect disables cross-cluster
search. This is correct, but not fully accurate as it disables any
functionality that relies on remote cluster connections: cross-cluster
search, remote data feeds, and cross-cluster replication. This commit
updates the docs to reflect this.
2019-10-30 11:26:15 -04:00
James Rodewig 0b062bbc82 [DOCS] Correct required file ext for user agent ingest processor ()
For the user agent ingest processor, custom regex files must end
with the `.yml` file extension.

This corrects the docs which said the `.yaml` extension was required.
2019-10-30 11:11:29 -04:00
Dan Hermann dbc05cd808
Add option to split processor for preserving trailing empty fields () 2019-10-30 08:25:03 -05:00
James Rodewig 77acbc4fa9 [DOCS] Reformat common grams token filter () 2019-10-30 08:40:56 -04:00
Yannick Welsch 356066ce6a Revert "Mute get-ccr-stats doctest ()"
This reverts commit f861927e8b8fc987949ce996a131a2d272b9646e.
2019-10-30 11:13:29 +01:00
Julie Tibshirani 89c65752dc
Update the signature of vector script functions. ()
Previously the functions accepted a doc values reference, whereas they now
accept the name of the vector field. Here's an example of how a vector function
was called before and after the change.

```
Before: cosineSimilarity(params.query_vector, doc['field'])
After:  cosineSimilarity(params.query_vector, 'field')
```

This seems more intuitive, since we don't allow direct access to vector doc
values and the the meaning of `doc['field']` is unclear.

The PR makes the following changes (broken into distinct commits):
* Add new function signatures of the form `function(params.query_vector,
'field')` and deprecates the old ones. Because Painless doesn't allow two
methods with the same name and number of arguments, we allow a generic `Object`
to be passed in to the function and decide on the behavior through an
`instanceof` check.
* Refactor the class bindings so that the document field is passed to the
constructor instead of the instance method. This allows us to avoid retrieving
the vector doc values on every function invocation, which gives a tiny speed-up
in benchmarks.

Note that this PR adds new signatures for the sparse vector functions too, even
though sparse vectors are deprecated. It seemed simplest to understand (for both
us and users) to keep everything symmetric between dense and sparse vectors.
2019-10-29 15:46:05 -07:00
James Rodewig 7002ce1e9c [DOCS] Replace `_uid` refs in reindex slicing docs ()
PR  removed the `_uid` field in favor of the `_id` field,
including for use in slicing.

This removes an outdated reference to `_uid` in our reindex docs.
2019-10-29 16:41:53 -04:00
Christoph Büscher 1de49d8a70 Remove Ranking Evaluation API experimental status ()
The API has been released long enough to remove the experimental status.
2019-10-29 20:57:39 +01:00
Lisa Cawley c6f4662038
[DOCS] Updates ML PRs in 7.4.1 release notes () 2019-10-29 09:35:11 -07:00
Daniel Andion d0cbbf9d58
SQL: [Docs] Typo in HAVING section ()
`HAVING` section code states `GROUP BY`  instead of the appropriate keyword.

(cherry picked from commit 9d505dc3db51e250fdf1b44e4d952dcd97bf1bc1)
2019-10-29 16:37:39 +01:00
lgypro abddf51672 [Docs] Fix syntax error leading to wrong doc ID ()
In order to index a document with id 2, the "&" should be replaced by "?"
2019-10-29 10:27:23 +01:00
Ian Danforth 82e25c4ac7 [Docs] Fix typo in suggesters search API doc () 2019-10-29 09:58:05 +01:00
Ian Danforth 4a076f5e92 [Doc] Fix typo in indices module docs () 2019-10-28 21:40:09 +01:00
Julie Tibshirani 605500df7e
Add sparse vector deprecation to 7.6 migration docs. ()
This note was accidentally omitted from the deprecation PR.
2019-10-28 11:57:20 -07:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop () ()
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Lisa Cawley 13ce179706
[DOCS] Re-enable code snippet testing in close anomaly detection job API () () 2019-10-28 08:42:09 -07:00
Michael Basnight 1ba57dbe08 [Docs] add missing snapshot restore reference () 2019-10-28 09:55:10 -05:00
Shaunak Kashyap d27a307379 [DOCS] Remove extraneous comma in Enrich Stats API's JSON response () 2019-10-25 12:35:50 -04:00
James Rodewig e9c8e4f6d1 [DOCS] Fix note format in index suggestion docs () 2019-10-25 11:31:47 -04:00
Christoph Büscher 055a0800eb [Docs] Mention reserved completion suggestion characters ()
We currently don't mention the three reserved characters anywhere. This change
adds a short note mentioning them

Closes 
2019-10-25 16:58:23 +02:00
Julie Tibshirani b2974e3816 Correct outdated information in _index docs. ()
This PR makes the following updates:
* Update the supported query types to include `prefix` and `wildcard`.
* Specify that queries accept index aliases.
* Clarify that when querying on a remote index name, the separator `:` must be
  present.
2019-10-24 11:02:25 -07:00
Hendrik Muhs 5ecfcdb162 update warning about index names after transform rename ()
update warning about index names after transform rename
2019-10-24 15:17:20 +02:00
Julie Tibshirani 4375316b9d Make sure to list the 7.5 migration docs. 2019-10-23 18:52:22 -07:00
Julie Tibshirani 2664cbd20b
Deprecate the sparse_vector field type. ()
We have not seen much adoption of this experimental field type, and don't see a
clear use case as it's currently designed. This PR deprecates the field type in
7.x. It will be removed from 8.0 in a follow-up PR.
2019-10-23 16:35:03 -07:00
James Rodewig 06dc1fbd96 [DOCS] Reformat ASCII folding token filter docs () 2019-10-23 15:06:55 -05:00
Jim Ferenczi 96556d72cc
Add a known issue to the release notes of 7.4.0 ()
A [bug](https://github.com/elastic/elasticsearch/issues/48358) in 7.4.0 prevents
the activation of the search slow log. This change adds an entry in the release
notes to warn users to not activate it in this version.

Relates 
2019-10-23 19:57:37 +02:00
James Rodewig 640d7416b1
[DOCS] Change prev version to 7.5 in upgrade docs () 2019-10-23 12:09:26 -05:00
James Rodewig 19afe3f84c [DOCS] Remove duplicate links for ingest processor overview () 2019-10-23 10:55:49 -05:00
James Rodewig 9c75f14a9f [DOCS] Reformat classic token filter docs () 2019-10-23 10:14:25 -05:00
James Rodewig 852622d970 [DOCS] Remove binary gendered language () 2019-10-23 09:37:12 -05:00
Alan Woodward 3be6135ceb Mute get-ccr-stats doctest ()
This test is failing frequently, due to 
2019-10-23 15:20:35 +01:00
Tom Callahan f573cd6a2c [DOCS] Add 7.4.1 Release Notes () 2019-10-23 08:22:00 -05:00
James Rodewig e45b0cd7e3 [DOCS] Sort cluster API docs alphabetically () 2019-10-22 12:28:39 -05:00
Brandon DeVault 2f162a98a0 [DOCS] Added the voting node.role value ()
* [DOCS] Added the voting node.role value 

With the elasticsearch.yml option "node.voting_only: true", if you perform a "_cat/nodes" there is a new option for `v` that will display if the node is a voting node.

* [DOCS] Added the voting node.role value

Updated "voting node" to read "voting-only node"
2019-10-22 15:26:05 +02:00
James Rodewig e931fcd331 [DOCS] Add placeholder for 7.4.1 release notes () 2019-10-22 07:53:28 -05:00
Alexandre Fonseca c41951c6b3 [Docs] Fix opType options in IndexRequest API example. () 2019-10-22 13:49:19 +02:00
Hendrik Muhs 0656207e9a [DOCS][Transform] document limitation regarding rolling upgrade with 7.2, 7.3 ()
adds a limitation about rolling upgrade from 7.2 or 7.3. and fixes a problem with renamed preferences
2019-10-22 09:01:45 +02:00
Martijn van Groningen c09b62d5bf
Backport: also validate source index at put enrich policy time ()
Backport of: 

This changes tests to create a valid
source index prior to creating the enrich policy.
2019-10-22 07:38:16 +02:00
James Rodewig f4fa61b2f2 [DOCS] Add 'Selecting gateway and seed nodes' section to CCS docs () 2019-10-21 11:14:23 -05:00
James Rodewig a66bb2c7ed [DOCS] Reformat CJK bigram and CJK width token filter docs () 2019-10-21 08:44:49 -05:00
Ignacio Vera b1224fca8c
upgrade to Lucene-8.3.0-snapshot-25968e3b75e () 2019-10-21 08:21:09 +02:00
Christoph Büscher 3ea666d694 Clarify mapping types that support ignore_malformed ()
The `ignore_malformed` setting only works on selected mapping types, otherwise
we throw an mapper_parsing_exception. We should add a list of all the mapping
types that support it, since the number of types not supporting it seems larger.

Closes 
2019-10-18 20:39:38 +02:00
Guido Lena Cota 1beeb3a83e [DOCS] Fix typos in Painless execute API docs() 2019-10-18 10:31:04 -04:00
Alexander Reelsen 66581d8158
update ingest-user-agent regexes.yml ()
This new regexes are from:
154eba17f5/regexes.yaml
2019-10-18 16:26:48 +02:00
James Rodewig 1d06be2f05 [DOCS] Note clause limit in `index.mapping.total_fields.limit` docs ()
Notes that you should increase the `indices.query.bool.max_clause_count`
setting if you increase the `index.mapping.total_fields.limit`
setting.

Closes .
2019-10-18 10:20:49 -04:00
markharwood fbd43196a8 [DOCS] Add TermVectors API reference to MLT query docs ()
Adds a reference the use of the TermVectors API following an issue raised about lack of flexibility in the MLT query: https://github.com/elastic/elasticsearch/issues/35509
2019-10-18 10:08:57 -04:00
Przemysław Witek 28f68fa221
Make num_top_classes parameter's default value equal to 2 () () 2019-10-17 18:43:15 +02:00
James Rodewig 4c6d56bef0 [DOCS] Expand put mapping API docs examples () 2019-10-17 09:31:52 -04:00
James Rodewig b4bf47cce4 [DOCS] Document ingest pipeline and processor node stats () 2019-10-17 09:19:45 -04:00
Anton b54d817f1d [DOCS] Fix typo in intervals query docs () 2019-10-17 09:15:55 -04:00
Hendrik Muhs 06f54428c2 update internal index name in documentation warning () 2019-10-17 15:09:47 +02:00
István Zoltán Szabó ecd62934f6 [DOCS] Applies rename of transform related roles and privileges () 2019-10-17 10:24:56 +02:00
Martijn van Groningen a5fe69c344
Include enrich into the info api as feature ()
This commit also fixes a bug, the enrich enabled setting
was not included in the list of settings.

Backport of 
2019-10-17 09:51:32 +02:00
Lee Hinman 5af66d79ef
Add SLM support to xpack usage and info APIs ()
* Add SLM support to xpack usage and info APIs

This is a backport of 

This adds the missing xpack usage and info information into the
`/_xpack` and `/_xpack/usage` APIs. The output now looks like:

```
GET /_xpack/usage
{
  ...
  "slm" : {
    "available" : true,
    "enabled" : true,
    "policy_count" : 1,
    "policy_stats" : {
      "retention_runs" : 0,
      ...
    }
  }
```

and

```
GET /_xpack
{
  ...
  "features" : {
    ...
    "slm" : {
      "available" : true,
      "enabled" : true
    },
    ...
  }
}
```

Relates to 

* Fix missing license
2019-10-16 21:06:27 -06:00
David Woods 73f5fef678 Add a note to query_string docs around spaces in field names. () 2019-10-16 13:28:55 -07:00
James Rodewig 8677653c5b [DOCS] Reformat apostrophe token filter docs () 2019-10-16 08:51:14 -04:00
François-Clément Brossard f501a4b2b5 Clarify low watermark documentation ()
Today the docs say that the low watermark has no effect on any shards that have
never been allocated, but this is confusing. Here "shard" means "replication
group" not "shard copy" but this conflicts with the "never been allocated"
qualifier since one allocates shard copies and not replication groups.

This commit removes the misleading words. A newly-created replication group
remains newly-created until one of its copies is assigned, which might be quite
some time later, but it seems better to leave this implicit.
2019-10-16 12:27:49 +01:00
Ian Danforth 28c1677341 [DOCS] Fix typo in percentile rank aggregation docs () 2019-10-15 15:56:45 -04:00
Wilder Pereira 8c73e215b2 [DOCS] Remove unneeded spaces from custom analyzer snippet () 2019-10-15 15:53:16 -04:00
James Rodewig 601a88bede [DOCS] Sort analyzers, tokenizers, and token filters alphabetically () 2019-10-15 15:47:25 -04:00
James Rodewig 3a7c2a4d17 [DOCS] Add `wait_for_completion` parm to execute enrich policy API docs () 2019-10-15 13:47:30 -04:00
Martijn van Groningen aff0c9babc
This commits merges () the enrich-7.x feature branch,
which is backport merge and adds a new ingest processor, named enrich processor,
that allows document being ingested to be enriched with data from other indices.

Besides a new enrich processor, this PR adds several APIs to manage an enrich policy.
An enrich policy is in charge of making the data from other indices available to the enrich processor in an efficient manner.

Related to 
2019-10-15 17:31:45 +02:00
James Rodewig 05b3de0110 [DOCS] Expand `updated` response parm in reindex API docs () 2019-10-15 11:17:09 -04:00
Marios Trivyzas 7fddf198b7 SQL: Implement DATEDIFF function ()
Implement DATEDIFF/TIMESTAMPDIFF function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/datediff-transact-sql?view=sql-server-2017
which allows a user to substract two date/datetime fields and return the
difference in the date/time unit specified.

Closes: 
(cherry picked from commit 745699f38dc8222670ffd65b66df33b5da39040b)
2019-10-15 15:12:11 +02:00
James Rodewig a61d12afea [DOCS] Add to 7.4 release notes () 2019-10-15 08:48:27 -04:00
Marios Trivyzas 6589617a51
SQL: Fix arg verification for DateAddProcessor ()
Previously, the safety check for the 2nd argument of the DateAddProcessor was
restricting it to Integer which was wrong since we allow all non-rational
numbers, so it's changed to a Number check as it's done in other cases.

Enhanced some tests regarding the check for an integer (non-rational
argument).

(cherry picked from commit 0516b6eaf5eb98fa5bd087c3fece80139a6b118e)
2019-10-15 12:52:11 +02:00
Martijn van Groningen 31e41d4ac2
fixed invalid reference 2019-10-15 10:45:35 +02:00
Tanguy Leroux fe15d9f784
Add documentation for Pause/Resume Auto-Follower APIs () ()
Relates 

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-10-15 08:38:49 +02:00
David Roberts 984323783e
[ML][7.x] Add lazy assignment job config option ()
This change adds:

- A new option, allow_lazy_open, to anomaly detection jobs
- A new option, allow_lazy_start, to data frame analytics jobs

Both work in the same way: they allow a job to be
opened/started even if no ML node exists that can
accommodate the job immediately. In this situation
the job waits in the opening/starting state until ML
node capacity is available. (The starting state for data
frame analytics jobs is new in this change.)

Additionally, the ML nightly maintenance tasks now
creates audit warnings for ML jobs that are unassigned.
This means that jobs that cannot be assigned to an ML
node for a very long time will show a yellow warning
triangle in the UI.

A final change is that it is now possible to close a job
that is not assigned to a node without using force.
This is because previously jobs that were open but
not assigned to a node were an aberration, whereas
after this change they'll be relatively common.
2019-10-15 06:55:11 +01:00
Martijn van Groningen cc4b6c43b3
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-15 07:23:47 +02:00
Gordon Brown 300ddfa3c1
SLM Start/Stop HLRC and docs ()
This commit adds HLRC support and documentation for the SLM Start and
Stop APIs, as well as updating existing documentation where appropriate.

This commit also ensures that the SLM APIs are properly included in the
HLRC documentation.
2019-10-14 16:56:31 -06:00
Martijn van Groningen 7fc9198d46
Change how `max_matches` affects `target_field` option. ()
Prior to this change the `target_field` would always be a json array
field in the document being ingested. This to take into account that
multiple enrich documents could be inserted into the `target_field`.

However the default `max_matches` is `1`. Meaning that by default
only a single enrich document would be added to `target_field` json
array field.

This commit changes this; if `max_matches` is set to `1` then the single
document would be added as a json object to the `target_field` and
if it is configured to a higher value then the enrich documents will be
added as a json array (even if a single enrich document happens to be
enriched).
2019-10-14 21:09:48 +02:00
Martijn van Groningen 7cc73f6193
Add HLRC support for enrich execute policy API ()
This PR also includes HLRC docs for the enrich stats api.

Relates to 
2019-10-14 19:55:48 +02:00
James Rodewig 170266765b [DOCS] Reformat docs for several snapshot lifecycle policy APIs () 2019-10-14 12:30:30 -04:00
Tanguy Leroux e4ea8b46b6
Add Pause/Resume Auto-Follower APIs to High Level REST Client ()
This commit adds support for Pause/Resume Auto-Follower APIs 
to the HLRC, with the documentation.

Relates 
2019-10-14 18:25:53 +02:00
David Roberts 1ca25bed38
[ML][7.x] Add option to stop datafeed that finds no data ()
Adds a new datafeed config option, max_empty_searches,
that tells a datafeed that has never found any data to stop
itself and close its associated job after a certain number
of real-time searches have returned no data.

Backport of 
2019-10-14 17:19:13 +01:00
James Rodewig 5f3ef2e09c [DOCS] Remove unsupported `local` and `master_timeout` parms from cat API docs () 2019-10-14 11:39:31 -04:00
Martijn van Groningen d4901a71d7
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-14 10:27:17 +02:00
Tanguy Leroux 742fa818b8
Add Pause/Resume Auto Follower APIs () ()
This commit adds two APIs that allow to pause and resume
CCR auto-follower patterns:

// pause auto-follower
POST /_ccr/auto_follow/my_pattern/pause

// resume auto-follower
POST /_ccr/auto_follow/my_pattern/resume

The ability to pause and resume auto-follow patterns can be
useful in some situations, including the rolling upgrades of
cluster using a bi-directional cross-cluster replication scheme
(see ).

This commit adds a new active flag to the AutoFollowPattern
and adapts the AutoCoordinator and AutoFollower classes so
that it stops to fetch remote's cluster state when all auto-follow
patterns associate to the remote cluster are paused.

When an auto-follower is paused, remote indices that match the
pattern are just ignored: they are not added to the pattern's
followed indices uids list that is maintained in the local cluster
state. This way, when the auto-follow pattern is resumed the
indices created in the remote cluster in the meantime will be
picked up again and added as new following indices. Indices
created and then deleted in the remote cluster will be ignored
as they won't be seen at all by the auto-follower pattern at
resume time.

Backport of  for 7.x
2019-10-13 09:22:51 +02:00
Yogesh Gaikwad ac209c142c
Remove uniqueness constraint for API key name and make it optional () ()
Since we cannot guarantee the uniqueness of the API key `name` this commit removes the constraint and makes this field optional.

Closes 
2019-10-12 22:22:16 +11:00
James Rodewig 256102042b
[DOCS] Reformat get snapshot lifecycle policy API docs () () 2019-10-11 13:00:48 -04:00
James Rodewig aa5689f5b3
[DOCS] Reformat put snapshot lifecycle policy API docs () () 2019-10-11 11:28:38 -04:00
David Turner ba62eb3dce Allow truncation of clean translog ()
Today the `elasticsearch-shard remove-corrupted-data` tool will only truncate a
translog it determines to be corrupt. However there may be other cases in which
it is desirable to truncate the translog, for instance if an operation in the
translog cannot be replayed for some reason other than corruption. This commit
adds a `--truncate-clean-translog` option to skip the corruption check on the
translog and blindly truncate it.
2019-10-11 15:48:12 +01:00
James Rodewig 58289a857d
[DOCS] Reformat reload search analyzers API docs () () 2019-10-11 09:40:49 -04:00
Przemysław Witek d210bfa888
[7.x] Add MlClientDocumentationIT tests for classification. () () 2019-10-11 10:19:55 +02:00
Matt Braymer-Hayes d315726c23 [DOCS] Fix gcloud command typo in discovery-gce.asciidoc. () 2019-10-10 14:21:46 -04:00
Marios Trivyzas 59b3294bc9 SQL: Implement DATEADD function ()
Implement DATEADD/TIMESTAMPADD function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/dateadd-transact-sql?view=sql-server-2017
which allows a user to add/subtract specified number of specified units
to/from a date/datetime field/expression.

Closes: 
(cherry picked from commit e624bc281bebb4bbe0b0c2e0a8cbc712e50097a8)
2019-10-10 16:22:13 +02:00
James Rodewig a8e027551a
[DOCS] Document `bytes` and `time` params in cat API docs () () 2019-10-10 09:48:40 -04:00
Martijn van Groningen 102016d571
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-10 14:44:05 +02:00
István Zoltán Szabó 9eac8bf2a8 [DOCS] Adds supported fields section to the PUT DFA API description () 2019-10-10 12:42:54 +02:00
David Turner ecb20ebc6c More bootstrap docs tweaks ()
Clarifies not to set `cluster.initial_master_nodes` on nodes that are joining
an existing cluster.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-10-10 09:55:30 +01:00
Jim Ferenczi dc39196ea4 Fix tag in the search request timeout option docs ()
and add missing parentheses `search_timeout` param
2019-10-10 10:35:44 +02:00
Martijn van Groningen aace42d38d
Add HLRC support for enrich stats API ()
This PR also includes HLRC docs for the enrich stats api.

Relates to 
2019-10-10 09:08:29 +02:00
István Zoltán Szabó 6f4b7e9a7f [DOCS] Extends the analyzed_fields description in the PUT DFA API docs () 2019-10-09 18:14:58 +02:00
James Rodewig 65f8294378 [DOCS] Add docs for `geo_match` enrich policy type () 2019-10-09 09:02:52 -04:00
Florian Kelbert 2abd9d53b6 [DOCS] Correct split API request format () 2019-10-09 08:28:41 -04:00
Andrei Stefan 75a7daae73 SQL: use calendar interval of 1y instead of fixed interval for grouping by YEAR and HISTOGRAMs ()
(cherry picked from commit 55f5463eee4ecea3537df4b34645f1d87472a802)
2019-10-09 11:51:35 +03:00
Martijn van Groningen da1e2ea461
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-09 09:06:13 +02:00
Jake Landis a6b0ae7f69
Fix bug in ingest node documentation () ()
The "Conditionals with the Pipeline Processor" incorrectly documents
how to create a pipeline of pipelines with a failure condition. The 
example as-is will always execute the fail processor. The change here
updates the documentation to correct guard the fail processor with an
if condition.
2019-10-08 17:23:38 -05:00
Ioannis Kakavas f4c884450f
Remove unmerged changes from release notes () 2019-10-08 18:14:41 +03:00
Lee Hinman ea4069ca63 Add Snapshot Lifecycle Retention documentation ()
* Add Snapshot Lifecycle Retention documentation

This commits adds API and general purpose documentation for SLM
retention.

Relates to 

* Fix docs tests

* Update default now that  has been merged

* Update docs/reference/ilm/apis/slm-api.asciidoc

Co-Authored-By: Gordon Brown <gordon.brown@elastic.co>

* Update docs/reference/ilm/apis/slm-api.asciidoc

Co-Authored-By: Gordon Brown <gordon.brown@elastic.co>

* Update docs with feedback
2019-10-08 08:47:08 -06:00
David Turner 11093197f1
Fix deprecation docs formatting ()
Relates 
2019-10-08 15:41:34 +02:00
James Rodewig 9b4eec9887 [DOCS] Fix errors in rollover index API docs () 2019-10-08 08:37:26 -04:00
David Turner bb5f750ab4 Deprecate include_relocations setting ()
Setting `cluster.routing.allocation.disk.include_relocations` to `false` is a
bad idea since it will lead to the kinds of overshoot that were otherwise fixed
in . This commit deprecates this setting so it can be removed in the next
major release.
2019-10-08 08:19:04 +01:00
Hendrik Muhs 5e0e54f455
[Transform] move root endpoint to _transform with BWC layer () ()
move the main endpoint to /_transform/ from /_data_frame/transforms/ with providing backwards compatibility and deprecation warnings
2019-10-08 08:59:01 +02:00
Ryan Ernst 8c6d1e0a08 Switch stored script example to script_score query ()
The example use of a scoring script was incorrectly using a filter
script query, which has no scoring, and thus no _score variable
avialable. This commit converts the example doc to using the newer
script_score query.
2019-10-07 17:08:57 -07:00
Lisa Cawley 39ef795085
[DOCS] Cleans up links to security content () () 2019-10-07 15:23:19 -07:00
Dimitris Athanasiou 7667ea5f6f
[7.x][ML] Additional outlier detection parameters () ()
Adds the following parameters to `outlier_detection`:

- `compute_feature_influence` (boolean): whether to compute or not
   feature influence scores
- `outlier_fraction` (double): the proportion of the data set assumed
   to be outlying prior to running outlier detection
- `standardization_enabled` (boolean): whether to apply standardization
   to the feature values

Backport of 
2019-10-07 18:21:33 +03:00
James Rodewig 176ca13c57 [DOCS] Correct deprecation note in mapping docs () 2019-10-07 09:37:52 -04:00
James Rodewig fc3cc30008
[DOCS] Reformat clear cache API docs () () 2019-10-07 09:36:30 -04:00
James Rodewig c03cdb4b15 [DOCS] Correct callouts in search template docs () 2019-10-07 09:25:32 -04:00
Yogesh Gaikwad 7c862fe71f
Add support to retrieve all API keys if user has privilege () ()
This commit adds support to retrieve all API keys if the authenticated
user is authorized to do so.
This removes the restriction of specifying one of the
parameters (like id, name, username and/or realm name)
when the `owner` is set to `false`.

Closes 
2019-10-07 23:58:21 +11:00
James Rodewig f93bb9dac5
[DOCS] Reformat type exists API docs () 2019-10-07 08:49:42 -04:00
Przemyslaw Gomulka 63bb4c91eb
Update deprecation logging doc with logger configuration () Backport#47508
Explicitly adds a configuration snippet to change logging level
2019-10-07 14:41:22 +02:00
Martijn van Groningen f2f2304c75
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-07 10:07:56 +02:00
debadair 41c04ef39c
[DOCS] Backporting API ref reformatting for document APIs ()
* [DOCS] Reformats bulk API. ()

* Reformats bulk API.

* Update docs/reference/docs/bulk.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Reformats mget API ()

* Reformats mget API

* Update docs/reference/docs/get.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Incorporated feedback.

* Reformats reindex API ()

* Reformats reindex API

* Incorporated review feedback.

* Reformats term vectors APIs ()

* Reformat termvectors APIs

* Reformats mtermvectors

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Incorporated review feedback.
2019-10-06 22:25:21 -07:00
debadair f765ded650 [DOCS] Comment out tag in Task Managment API Docs so it isn't rendered. ()
The tag for the shared content is being rendered in the output.
2019-10-05 12:48:51 -04:00
Mark Tozzi e404f7ea80
DocValueFormat implementation for date range fields () () 2019-10-04 17:21:17 -04:00
Lisa Cawley f35fcf7204
[DOCS] Adds security content in the Elasticsearch Reference () 2019-10-04 13:11:05 -07:00
James Rodewig 45f12d18fb
[DOCS] Reformat shrink index API docs () () 2019-10-04 14:00:18 -04:00
James Rodewig 0b4fb05540
[DOCS] Reformat refresh API docs () () 2019-10-04 13:50:09 -04:00
Lisa Cawley 9b3e5409c1
[7.x][DOCS] Copies security source files from stack-docs () 2019-10-04 08:19:10 -07:00
James Rodewig 3cc8081274 [DOCS] Correct headings for split index API docs 2019-10-04 11:07:41 -04:00
James Rodewig bb82addf35
[DOCS] Reformat split index API docs () () 2019-10-04 10:41:31 -04:00
István Zoltán Szabó a57cb5843f [DOCS] Fixes an attribute in the update datafeed API docs. () 2019-10-04 08:46:16 +02:00
Ryan Ernst f32692208e
Add explanations to script score queries () ()
While function scores using scripts do allow explanations, they are only
creatable with an expert plugin. This commit improves the situation for
the newer script score query by adding the ability to set the
explanation from the script itself.

To set the explanation, a user would check for `explanation != null` to
indicate an explanation is needed, and then call
`explanation.set("some description")`.
2019-10-03 21:05:05 -07:00
Lisa Cawley f81d9a4aa6
Update link to ls monitoring settings () () 2019-10-03 17:06:26 -07:00
James Rodewig b8e7f5069d
[DOCS] Reformat index recovery API docs () () 2019-10-03 16:43:26 -04:00
James Rodewig 56e7668c78 [DOCS] Reformat index shard stores API docs () 2019-10-03 15:57:35 -04:00
Francois-Clement Brossard dc02868671 [DOCS] Update painless statements with if/else example () 2019-10-03 15:23:54 -04:00
Paweł Krześniak edbc375916 [DOCS] Change index name in rollover ILM example ()
The warning section above the example tells that index name has to end with the digits but the example itself uses index name without digits which is confusing.
2019-10-03 09:26:10 -04:00
AndyHunt66 4a516f4ae2 [DOCS] Remove duplicated half-sentence from secure settings docs () 2019-10-03 08:41:06 -04:00
István Zoltán Szabó 9dcdbea228 [DOCS] Amends update datafeed API docs () 2019-10-03 13:13:18 +02:00
Russ Cam 3bb17a0c28 Remove string interning from release notes ()
This commit removes the string interning issue and PR from the 7.4.0 release notes. 

The PR https://github.com/elastic/elasticsearch/pull/41039 was closed and not merged.
2019-10-03 11:01:03 +02:00
Ioannis Kakavas d55378e8e7
Add realm uniqueness enforcement in 7.4 migration docs ()
Add the change to the migration docs as it was omitted in 
2019-10-03 11:03:29 +03:00
James Rodewig 6b96f53ea0 [DOCS] Change xrefs to external links in 7.4 rls highlights () 2019-10-02 17:03:21 -04:00
Lee Hinman 2e3eb4b24e
Add API to execute SLM retention on-demand () ()
* Add API to execute SLM retention on-demand ()

This is a backport of 

This commit adds the `/_slm/_execute_retention` API endpoint. This
endpoint kicks off SLM retention and then returns immediately.

This in particular allows us to run retention without scheduling it
(for entirely manual invocation) or perform a one-off cleanup.

This commit also includes HLRC for the new API, and fixes an issue
in SLMSnapshotBlockingIntegTests where retention invoked prior to the
test completing could resurrect an index the internal test cluster
cleanup had already deleted.

Resolves 
Relates to 
2019-10-02 12:29:04 -06:00
István Zoltán Szabó 033aa9cf9b [DOCS] Adds examples to the PUT dfa and the evaluate dfa APIs ()
* [DOCS] Adds examples to the PUT dfa and the evaluate dfa APIs.

* [DOCS] Removes extra lines from examples.

* Update docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Update docs/reference/ml/df-analytics/apis/put-dfanalytics.asciidoc

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* [DOCS] Explains examples.
2019-10-02 10:33:45 +02:00
David Turner 7739938930 Clarify that you cannot abort an upgrade ()
We do mention that rolling back an upgrade requires a restore from a snapshot,
but it's hidden at the bottom of the "preparing to upgrade" instructions on a
different page from the actual upgrade instructions. This commit duplicates the
preparatory instructions onto the pages containing the actual upgrade
instructions and rewords the point about rollbacks a bit.
2019-10-02 09:29:10 +01:00
Robin Clarke 98c1c2f650 Clearer language around upgrade sequence () 2019-10-02 09:25:50 +01:00
István Zoltán Szabó 1cecbd0cd3 [DOCS] Fine tunes update anomaly detection job API documentation ()
* [DOCS] Fine tunes update anomaly detection job API documentation.
* [DOCS] Removes delimiter to fix the table.
2019-10-02 10:06:49 +02:00
István Zoltán Szabó 6a9f04ee76 [DOCS] Fixes typos in the PUT dfa and the evaluate dfa documentation. () 2019-10-02 09:52:29 +02:00
James Rodewig 079bf887c0
[DOCS] Reorder index APIs alphabetically () () 2019-10-01 17:07:28 -04:00
James Rodewig 0179f93544
[DOCS] Reformat simulate pipeline API () () 2019-10-01 14:49:14 -04:00
James Rodewig aeb4edce3a
[DOCS] Reformat put pipeline API () () 2019-10-01 14:48:18 -04:00
lcawl 66116e39ba [DOCS] Edits ML release notes 2019-10-01 10:15:06 -07:00
James Rodewig e70220857d
[DOCS] Document cat tasks API () () 2019-10-01 12:22:50 -04:00
Mark Tozzi 5bdf25320a
Documentation notes for Range field histograms () () 2019-10-01 10:58:44 -04:00
Lisa Cawley 5ba543fd6c [DOCS] Adds machine learning PRs to release notes () 2019-10-01 10:17:41 -04:00
James Rodewig 2ca075dee4 [DOCS] Remove coming tags for 7.4.0 release () 2019-10-01 10:17:36 -04:00
Marios Trivyzas f792dbf239 SQL: Implement DATE_PART function ()
DATE_PART(<datetime unit>, <date/datetime>) is a function that allows
the user to extract the specified unit from a date/datetime field
similar to the EXTRACT (<datetime unit> FROM <date/datetime>) but
with different names and aliases for the units and it also provides more
options like `DATE_PART('tzoffset', datetimeField)`.

Implemented following the SQL server's spec: https://docs.microsoft.com/en-us/sql/t-sql/functions/datepart-transact-sql?view=sql-server-2017
with the difference that the <datetime unit> argument is either a
literal single quoted string or gets a value from a table field, whereas
in SQL server keywords are used (unquoted identifiers) and it's not
possible to use a value coming for a table column.

Closes: 
(cherry picked from commit ead743d3579eb753fd314d4a58fae205e465d72e)
2019-10-01 16:28:27 +03:00
István Zoltán Szabó 170b102ab5 [DOCS] Changes wording to move away from data frame terminology in the ES repo ()
* [DOCS] Changes wording to move away from data frame terminology in the ES repo.
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-10-01 08:08:17 +02:00
James Rodewig fd421bd12d
[7.x] [DOCS] Add response body parms to search API docs () () 2019-09-30 13:54:06 -04:00
Lisa Cawley 0c3ee0b15c
[DOCS] Moves Watcher content into Elasticsearch book () ()
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-30 10:18:50 -07:00
David Turner 41bc878738 Clarify rolling-upgrade docs ()
Note to upgrade the master-eligible nodes last, and note that
`cluster.initial_master_nodes` should not be set.
2019-09-30 17:02:58 +01:00
James Rodewig 024d1f2ab9
[DOCS] Reformat delete pipeline API () () 2019-09-30 11:38:46 -04:00
James Rodewig 312e32a3d7 [DOCS] Correct snippet in query string syntax 2019-09-30 11:30:33 -04:00
Andrew Naguib ae85a0e29a [DOCS] Note double backslashes (`\\`) are required to escape JSON chars () 2019-09-30 11:20:07 -04:00
James Rodewig 5990532cb7
[DOCS] Reformat flush API docs () () 2019-09-30 08:42:52 -04:00
Martijn van Groningen fe937ea4b8
Add config namespace in get policy api response ()
Currently the policy config is placed directly in the json object
of the toplevel `policies` array field. For example:

```
{
    "policies": [
        {
            "match": {
                "name" : "my-policy",
                "indices" : ["users"],
                "match_field" : "email",
                "enrich_fields" : [
                    "first_name",
                    "last_name",
                    "city",
                    "zip",
                    "state"
                ]
            }
        }
    ]
}
```

This change adds a `config` field in each policy json object:

```
{
    "policies": [
        {
            "config": {
                "match": {
                    "name" : "my-policy",
                    "indices" : ["users"],
                    "match_field" : "email",
                    "enrich_fields" : [
                        "first_name",
                        "last_name",
                        "city",
                        "zip",
                        "state"
                    ]
                }
            }
        }
    ]
}
```

This allows us in the future to add other information about policies
in the get policy api response.

The UI will consume this API to build an overview of all policies.
The UI may in the future include additional information about a policy
and the plan is to include that in the get policy api, so that this
information can be gathered in a single api call.

An example of the information that is likely to be added is:
* Last policy execution time
* The status of a policy (executing, executed, unexecuted)
* Information about the last failure if exists
2019-09-30 14:37:23 +02:00
Yannick Welsch 9dc90e41fc Remove "force" version type ()
It's been deprecated long ago and can be removed.

Relates to 

Closes 
2019-09-30 11:58:34 +02:00
Martijn van Groningen 66f72bcdbc
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-09-30 08:12:28 +02:00
Martijn van Groningen 36215bd33e
fixed docs issue 2019-09-30 08:04:18 +02:00
lcawl ab871add36 [DOCS] Skip code snippet testing in monitoring example 2019-09-27 17:11:24 -07:00
Lisa Cawley 62baa85632 [DOCS] Adds production monitoring page () 2019-09-27 15:01:54 -07:00
James Rodewig b98c57ab21
[DOCS] Document missing query parms for cat recovery API () ()
Documents the following query parameters for
the /_cat/recovery API:

* `active_only`
* `detailed`
* `index`
2019-09-27 16:31:10 -04:00
James Rodewig e01465eb88
[DOCS] Correct typo in ICU Analysis plugin description () () 2019-09-27 13:04:14 -04:00
James Rodewig b159305274
[DOCS] Add redirect for SLM API docs () () 2019-09-27 11:05:55 -04:00
Martijn van Groningen 7ffe2e7e63
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-09-27 14:42:11 +02:00
Marios Trivyzas 01623f9f1c
SQL: Add alias DATETRUNC to DATE_TRUNC function ()
To be on the safe side in terms of use cases also add the alias
DATETRUNC to the DATE_TRUNC function.

Follows: 

(cherry picked from commit 9ac223cb1fc66486f86e218fa785a32b61e9bacc)
2019-09-27 15:38:51 +03:00
István Zoltán Szabó 0ab7132c47 [DOCS] Reformats Profile API ()
* [DOCS] Reformats Profile API.

* [DOCS] Fixes failing docs test.
2019-09-27 11:14:14 +02:00
Colin Goodheart-Smithe 2cd6bed9c8
Updates 7.4.0 release notes 2019-09-27 09:20:08 +01:00
Pius 9c60e15ebd [DOCS] Add 'remote clusters' requirement link to CCR docs () 2019-09-26 16:15:21 -04:00
Nicole Albee 4714f10650 [DOCS] Remove coming tag from 7.2.0 release highlights () 2019-09-26 15:48:54 -04:00
Igor Motov ae202fda21 SQL: Add support for shape type ()
Enables support for Cartesian geometries shape type. We still need to
decide how to handle the distance function since it is currently using
the haversine distance formula and returns results in meters, which
doesn't make any sense for Cartesian geometries.

Closes 
Relates to 
2019-09-26 09:47:42 -04:00
James Rodewig 0c575dc1e8 [DOCS] Correct link to `index.store.preload` setting () 2019-09-26 08:57:09 -04:00
James Rodewig 3b626c2d56
[DOCS] Reformat get pipeline API () () 2019-09-26 08:51:12 -04:00
Gordon Brown 7ac647c365
Add support for POST requests to SLM Execute API ()
This commit adds support for POST requests to the SLM `_execute` API,
because POST is a more appropriate HTTP verb for this action as it is
not idempotent. The docs are also changed to favor POST over PUT,
although PUT is not removed or officially deprecated.
2019-09-25 16:15:10 -06:00
Andrei Dan 27520cac3b
ILM: parse origination date from index name () ()
* ILM: parse origination date from index name ()

Introduce the `index.lifecycle.parse_origination_date` setting that
indicates if the origination date should be parsed from the index name.
If set to true an index which doesn't match the expected format (namely
`indexName-{dateFormat}-optional_digits` will fail before being created.
The origination date will be parsed when initialising a lifecycle for an
index and it will be set as the `index.lifecycle.origination_date` for
that index.

A user set value for `index.lifecycle.origination_date` will always
override a possible parsable date from the index name.

(cherry picked from commit c363d27f0210733dad0c307d54fa224a92ddb569)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

* Drop usage of Map.of to be java 8 compliant
2019-09-25 21:44:16 +01:00
Lisa Cawley 287d96d1a1 [DOCS] Fix links to transform pages () 2019-09-25 13:09:47 -07:00
Gordon Brown a46eef9634
Change SLM stats format ()
Using arrays of objects with embedded IDs is preferred for new APIs over
using entity IDs as JSON keys.  This commit changes the SLM stats API to
use the preferred format.
2019-09-25 11:32:08 -06:00
Colin Goodheart-Smithe 35f056661f
Updates 7.4.0 release notes 2019-09-25 18:10:09 +01:00
James Rodewig 9ddc99de23 [DOCS] Reformat clone index API docs () 2019-09-25 12:37:53 -04:00
Lisa Cawley d5f396fe55
[DOCS] Adds transforms to Elasticsearch book () () 2019-09-25 08:11:37 -07:00
István Zoltán Szabó 74fd21f0b0 [DOCS] Reformats ranking evaluation API ()
* [DOCS] Reformats ranking evaluation API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-25 15:01:10 +02:00
István Zoltán Szabó 83365e94ba [DOCS] Reformat suggesters page. () 2019-09-25 14:42:16 +02:00
Paweł Krześniak b234d2cbd7 [Docs] Fix the numbering in the annotated_text example () 2019-09-25 11:29:01 +02:00
Hendrik Muhs e974f178b5 [Transform] rename data frame transform to transform for hlrc client ()
rename data frame transform to transform for hlrc
2019-09-25 08:31:43 +02:00
James Rodewig 61eef2fd31
[DOCS] Add multi-level nested query example to nested query docs () 2019-09-25 02:01:37 -04:00
Julie Tibshirani 4faba9cbbf Mention ip fields in the global ordinals docs. ()
Although they do not support eager_global_ordinals, ip fields use global
ordinals for certain aggregations like 'terms'.

This commit also corrects a reference to the sampler aggregation.
2019-09-24 12:39:11 -07:00
Lisa Cawley 1893f6adc2 [DOCS] Merge monitoring sections in Elasticsearch book () 2019-09-24 11:39:28 -07:00
David Roberts ffae769186
[TEST] Rename data-frames to transform in docs test cleanup ()
The renaming of the tests in  caused the
cleanup between tests to be skipped.

Backport of 
2019-09-24 17:38:09 +01:00
David Roberts 22dade8e1b
[DOCS] Add 7.4 breaking changes for transforms and data frame analytics ()
The PRs that made these changes are:

- https://github.com/elastic/elasticsearch/pull/44350
- https://github.com/elastic/elasticsearch/pull/45276
- https://github.com/elastic/elasticsearch/pull/45856

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-09-24 17:20:25 +01:00
Colin Goodheart-Smithe 56224068d4
Release highlights for 7.4.0 ()
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-24 09:25:13 +01:00
Julie Tibshirani 9124c94a6c
Add support for aliases in queries on _index. ()
Previously, queries on the _index field were not able to specify index aliases.
This was a regression in functionality compared to the 'indices' query that was
deprecated and removed in 6.0.

Now queries on _index can specify an alias, which is resolved to the concrete
index names when we check whether an index matches. To match a remote shard
target, the pattern needs to be of the form 'cluster:index' to match the
fully-qualified index name. Index aliases can be specified in the following query
types: term, terms, prefix, and wildcard.
2019-09-23 13:21:37 -07:00
James Rodewig 618fb31be8 [DOCS] Minor editorial changes to enrich docs 2019-09-23 13:25:34 -04:00
Lisa Cawley a815f8b930 [DOCS] Group rollup and transform content () 2019-09-23 08:55:25 -07:00
James Rodewig b09aba4c55 [DOCS] Reformat rollover index API docs () 2019-09-23 09:18:33 -04:00
Martijn van Groningen 0cfddca61d
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-09-23 09:46:05 +02:00
Lisa Cawley 875d864be6
[DOCS] Update data frame transform URLs () () 2019-09-20 15:57:43 -07:00
Javier Ruiz a5661ac03a [DOCS] Fix calendar interval typos for date histo agg () 2019-09-20 15:22:41 -04:00
Lisa Cawley 4c688df0ef [DOCS] Add security section to Elasticsearch book ()
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-09-20 11:34:37 -07:00
James Rodewig 99130114de
[DOCS] Correct several [source,console-result] snippets () () 2019-09-20 12:20:12 -04:00
Michael Basnight f1c7ed647b Allow comma separated ids in get enrich policy API ()
This commit changes the GET REST api so it will accept an optional comma
separated list of enrich policy ids. This change also modifies the
behavior of the GET API in that it will not error if it is passed a bad
enrich id anymore, but will instead just return an empty list.
2019-09-20 10:06:58 -05:00
István Zoltán Szabó fcea154f2e [DOCS] Reformats Field capabilities API ()
* [DOCS] Reformats Field capabilities API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-20 11:28:19 +02:00
István Zoltán Szabó 363075cf1d [DOCS] Reformats explain API ()
* [DOCS] Reformats explain API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-20 11:00:33 +02:00
Hendrik Muhs abe889af75
[7.5][Transform] rename classes in transform plugin ()
rename classes and settings in transform plugin, provide BWC for old settings
2019-09-20 10:43:00 +02:00
Colin Goodheart-Smithe 74d1588ec5
Updates release notes for 7.4.0 2019-09-20 08:10:47 +01:00
Julie Tibshirani 6ce4825594
Add vector function changes to 7.4 migration docs. ()
This note was accidentally omitted in the breaking change .
2019-09-19 20:26:52 -07:00
rikardbakkehaug 6b00e00bc6 [DOCS] Correct date math ex for `gt` and `gte` parms in `range` query docs () 2019-09-19 16:39:03 -04:00
Jason Tedor bd77626177
Add the ability to require an ingest pipeline ()
This commit adds the ability to require an ingest pipeline on an
index. Today we can have a default pipeline, but that could be
overridden by a request pipeline parameter. This commit introduces a new
index setting index.required_pipeline that acts similarly to
index.default_pipeline, except that it can not be overridden by a
request pipeline parameter. Additionally, a default pipeline and a
request pipeline can not both be set. The required pipeline can be set
to _none to ensure that no pipeline ever runs for index requests on that
index.
2019-09-19 16:37:45 -04:00
James Rodewig 251dbd8522 [DOCS] Remove `lowercase_terms` parm from term suggester docs () 2019-09-19 15:56:47 -04:00
James Rodewig 8307aa3b71 [DOCS] Update tagged region for index alias desc () 2019-09-19 15:30:33 -04:00
Takumasa Ochi 7a3054c5dc Fix typos in `match` in profile API ()
* Replace `matches` with correct `match`
* Use present tense consistently
* Replace `metric` with correct `match`
2019-09-19 16:07:52 +02:00
Armin Braun b32031ee03
Fix snapshot_restore_exception in Docs Tests () ()
We should only snapshot the index we're going to
restore in the next step. Otherwise, we will
potentially not get the correct response or
fail restoring outright due to internal indices
getting created concurrently when running against
the x-pack distribution.

Closes 
2019-09-19 16:05:51 +02:00
Russ Cam e467dddd4c Use snake casing for document field ()
This commit updates the search example to use snake casing for the document field. This is consistent with other examples
2019-09-19 14:27:00 +02:00
James Rodewig b73a9604c1
[DOCS] Separate and reformat synced flush API docs () () 2019-09-19 08:22:37 -04:00
LHearen e555ef2e86 [DOCS] make bulk page more readable ()
comma and semi-colon required to make it more readable
2019-09-19 14:20:41 +02:00
maidoo 0f2e72b7f8 [DOCS] Term dict memory usage is a 7.3.0 enhancement, not 7.2.0 ()
Closes 
2019-09-19 14:15:12 +02:00
Toby McLaughlin 7795e21ea3 [DOCS] Fix typo in GCS snapshot repo doc () 2019-09-19 14:06:37 +02:00
Luca Cavanna a6c9f8ffae [DOCS] mention automatic cancellation in search docs ()
Add a small note around automatic search cancellation which was missed as part of 

Relates to 
2019-09-19 09:35:30 +02:00
István Zoltán Szabó 3be51fbdf7 [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs and the evaluation API ()
* [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs.
Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>
Co-Authored-By: Tom Veasey <tveasey@users.noreply.github.com>
2019-09-19 09:23:18 +02:00
István Zoltán Szabó 65fffcc9c1 [DOCS]Changes failed transforms limitation. () 2019-09-19 09:04:41 +02:00
Lisa Cawley a32fe1618c [DOCS] Updates transform landing page () 2019-09-18 14:00:43 -07:00
James Rodewig b08b3741db [DOCS] Use versions file from elastic/docs () 2019-09-18 11:18:14 -04:00
James Rodewig d00bad9ab2 [DOCS] Reformat force merge API docs () 2019-09-18 10:10:51 -04:00
James Rodewig 6863b90272 [DOCS] Remove leveloffset for CCR docs () 2019-09-18 09:44:43 -04:00
István Zoltán Szabó e59be0354a [DOCS] Reformats validate API ()
* [DOCS] Reformats validate API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-18 14:31:17 +02:00
James Rodewig b6da5fa2f7 [DOCS] Correct `<enrich-policy>` parm description for comma-sep list () 2019-09-18 08:30:50 -04:00
István Zoltán Szabó d931665f91 [DOCS] Adds force parameter to Stop transforms API docs. () 2019-09-18 09:16:07 +02:00
Lee Hinman b85468d6ea
Add node setting for disabling SLM () ()
This adds the `xpack.slm.enabled` setting to allow disabling of SLM
functionality as well as its HTTP API endpoints.

Relates to 
2019-09-17 17:39:41 -06:00
Lisa Cawley 00c604a5a6 [DOCS] Augments ecommerce example () 2019-09-17 11:47:48 -07:00
James Rodewig 9dcf8a53a2 [DOCS] Add CCR docs to the Elasticsearch Reference Guide ()
Previously, cross-cluster replication (CCR) documentation was located in
the Stack Overview:
https://www.elastic.co/guide/en/elastic-stack-overview/master/xpack-ccr.html

This adds CCR documentation to the Elasticsearch Reference Guide with a
level offset for headings.

The level offset and CCR Stack Overview docs will be removed in later
commits.
2019-09-17 14:26:20 -04:00
Colin Goodheart-Smithe 9d6632d6fa
Updates 7.4.0 release notes 2019-09-17 09:07:56 +01:00
István Zoltán Szabó 595bf52927 [DOCS] Reformats count API ()
* [DOCS] Reformats count API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-17 09:54:19 +02:00
Lisa Cawley a9eb538e93 [DOCS] Removes data-frames folder () 2019-09-16 10:29:36 -07:00
Lisa Cawley d62e1a3d8b [DOCS] Fixes data frame analytics job terminology in HLRC () 2019-09-16 10:07:59 -07:00
lcawl 48099cb18c [DOCS] Fixes formatting of analyzer parameter 2019-09-16 10:03:23 -07:00
Lisa Cawley dddc9b3d73 [DOCS] Updates dataframe transform terminology () 2019-09-16 08:32:13 -07:00
István Zoltán Szabó fe8f33a8e1 [DOCS] Adds outlier detection params to the data frame analytics resources ()
* [DOCS] Adds outlier detection params to the data frame analytics resources.
Co-Authored-By: Tom Veasey <tveasey@users.noreply.github.com>
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-09-16 14:23:23 +02:00
Hendrik Muhs c8f52ec4ff
[Transform] Rename data frame plugin to transform: classes in xpack.core () ()
rename classes in xpack.core of transform plugin from "data frame transform" to "transform"
2019-09-16 13:39:22 +02:00
Costin Leau 16bed7f017 DOC: Fix typo
(cherry picked from commit d874f1139ef0335db181d932f1e1299e1c8e40b8)
2019-09-16 12:35:21 +03:00
Alexander Reelsen 011496ed5f Expose cache setting in UserAgentPlugin ()
The setting was not registered. Also documentation has been added.
2019-09-16 11:30:38 +02:00
Costin Leau e5eb4b49c9 DOC: Update section for SQuirrel SQL 4.0.0 ()
The just released SQuirrel SQL 4.0.0 provides an Elasticsearch driver
definition out of the box; update the documentation to reflect that.

(cherry picked from commit 3aa417ed74947e69f0ff605b1c210a0678a3cb9f)
2019-09-16 12:27:31 +03:00
Andrei Dan c57cca98b2
[ILM] Add date setting to calculate index age () ()
* [ILM] Add date setting to calculate index age

Add the `index.lifecycle.origination_date` to allow users to configure a
custom date that'll be used to calculate the index age for the phase
transmissions (as opposed to the default index creation date).

This could be useful for users to create an index with an "older"
origination date when indexing old data.

Relates to .

* [ILM] Don't override creation date on policy init

The initial approach we took was to override the lifecycle creation date
if the `index.lifecycle.origination_date` setting was set. This had the
disadvantage of the user not being able to update the `origination_date`
anymore once set.

This commit changes the way we makes use of the
`index.lifecycle.origination_date` setting by checking its value when
we calculate the index age (ie. at "read time") and, in case it's not
set, default to the index creation date.

* Make origination date setting index scope dynamic

* Document orignation date setting in ilm settings

(cherry picked from commit d5bd2bb77ee28c1978ab6679f941d7c02e389d32)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-09-16 08:50:28 +01:00
David Turner 272b0ecbdd Remove docs for proxy mode ()
We added docs for proxy mode in  but on reflection we should not be
documenting this setting since it does not play well with all proxies and we
can't recommend its use. This commit removes those docs and expands its Javadoc
instead.
2019-09-13 22:20:11 +01:00
James Rodewig 2831535cf9 [DOCS] Replace "// CONSOLE" comments with [source,console] () 2019-09-13 11:44:54 -04:00
James Rodewig 0def429bc1 [DOCS] Correct anchors and title for Update index settings API docs () 2019-09-13 09:45:52 -04:00
James Rodewig af7aba18d4 Fixed sample code for minhash ()
The sample code is wrong. Field type is required for the sample field.
I guess the intention was to give the sample field the name ```fingerprint```, mapping it as ```text``` using the custom analyzer ```my_analyzer```
2019-09-12 13:29:44 -04:00
James Rodewig 3b26657335 [DOCS] Remove coming tags for 7.3.2 release () 2019-09-12 13:26:11 -04:00
James Rodewig 411d4e9a93 [DOCS] Change // CONSOLE comments to [source,console] () 2019-09-12 10:27:35 -04:00
James Rodewig 60db630abd
[DOCS] Add missing mention of current version to snapshot docs () () 2019-09-12 08:47:22 -04:00
David Turner 5c85b0998b
Clarify that discovery ignores master-ineligibles ()
The changes in  mean that the discovery process can no longer use
master-ineligible nodes as a stepping-stone between master-eligible nodes.
This was normally an indication of a strange and possibly-fragile configuration
and was not recommended, but this commit adds a note to the breaking changes
docs to note that this kind of configuration is more obviously broken in recent
versions.
2019-09-12 11:07:34 +01:00
Jilles van Gurp 030d43a76a Add a few notes on Cancellable to the LLRC and HLRC docs. ()
Add a section to both the low level and high level client documentation on asynchronous usage and `Cancellable` added for  

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>
2019-09-12 09:45:29 +02:00
Luca Cavanna cfb186afaf Add support for cancelling async requests in low-level REST client ()
The low-level REST client exposes a `performRequestAsync` method that
allows to send async requests, but today it does not expose the ability
to cancel such requests. That is something that the underlying apache
async http client supports, and it makes sense for us to expose.

This commit adds a return value to the `performRequestAsync` method,
which is backwards compatible. A `Cancellable` object gets returned,
which exposes a `cancel` public method. When calling `cancel`, the
on-going request associated with the returned `Cancellable` instance
will be cancelled by calling its `abort` method. This works throughout
multiple retries, though some special care was needed for the case where
`cancel` is called between different attempts (when one attempt has
failed and the consecutive one has not been sent yet).

Note that cancelling a request on the client side does not automatically 
translate to cancelling the server side execution of it. That needs to be 
specifically implemented, which is on the work for the search API (see ).

Relates to 
2019-09-12 09:45:28 +02:00
Lisa Cawley ec5592ed76 [DOCS] Adds missing icons to Watcher HLRC APIs () 2019-09-11 16:35:15 -07:00
Lisa Cawley 4e371a220b [DOCS] Adds missing icons to ILM HLRC APIs () 2019-09-11 15:46:19 -07:00
Lisa Cawley c3b6e6fe96 [DOCS] Adds missing icons to CCR HLRC APIs () 2019-09-11 15:37:20 -07:00
Lisa Cawley 8200010e0a [DOCS] Adds missing icons to Graph HLRC APIs () 2019-09-11 15:22:10 -07:00
Lisa Cawley 769f42bdd7 [DOCS] Add missing icons to security HLRC APIs () 2019-09-11 15:05:31 -07:00
Lisa Cawley e609011754 [DOCS] Add missing icons to rollup HLRC APIs () 2019-09-11 11:50:22 -07:00
Marios Trivyzas d956509394 SQL: Implement DATE_TRUNC function ()
DATE_TRUNC(<truncate field>, <date/datetime>) is a function that allows
the user to truncate a timestamp to the specified field by zeroing out
the rest of the fields. The function is implemented according to the
spec from PostgreSQL: https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes: 
(cherry picked from commit b37e96712db1aace09f17b574eb02ff6b942a297)
2019-09-11 21:41:02 +03:00
Lisa Cawley fa54de7789 [DOCS] Add missing icons to transform HLRC APIs () 2019-09-11 11:22:58 -07:00
James Rodewig 35bf92cdac [DOCS] Reformat enrich stats API () 2019-09-11 13:52:50 -04:00
Thibault Richard 44c4412406 [DOCS] Fix backquote in the list of realm types () 2019-09-11 12:54:46 -04:00
James Rodewig 043471c643
[DOCS] Minor improvement to the nested aggregation docs () ()
* Minor improvement to the nested aggregation docs

* The attributes name and resellers.name were rather confusing,
  especially since the first one was dynamically mapped and not shown
  in the documentation (you had to read the test to see it). This
  change introduces a unique name for the nested attribute and adds
  the example document to the documentation.
* Change the index name from "index" to something more speaking.

* Update docs/reference/aggregations/bucket/nested-aggregation.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/aggregations/bucket/nested-aggregation.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/aggregations/bucket/nested-aggregation.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-11 12:06:42 -04:00
Lee Hinman dcdfc382c7 Fix ILM start and stop doc tests ()
This uses whatever the server retrieves, rather than hardcoded
"STOPPING" and "STOPPED" since the server may go to STOPPED before the
request is issued.

Resolves 
2019-09-11 09:54:41 -06:00
Lisa Cawley c0ec6ade4b
[DOCS] Adds transform content () () 2019-09-11 08:44:03 -07:00
Lisa Cawley dea472c6fb [DOCS] Adds machine learning PRs to release notes () 2019-09-11 08:30:59 -07:00
Christoph Büscher aa0c586b73 Deprecate `_field_names` disabling ()
Currently we allow `_field_names` fields to be disabled explicitely, but since
the overhead is negligible now we decided to keep it turned on by default and
deprecate the `enable` option on the field type. This change adds a deprecation
warning whenever this setting is used, going forward we want to ignore and finally
remove it.

Closes 
2019-09-11 14:58:08 +02:00
Martijn van Groningen 60ad099178
Add HLRC support for enrich get policy API. ()
Changed the signature of AbstractResponseTestCase#createServerTestInstance(...)
to include the randomly selected xcontent type. This is needed for the
creating a server response instance with a query which is represented as BytesReference.
Maybe this should go into a different change?

This PR also includes HLRC docs for the get policy api.

Relates to 
2019-09-11 14:42:50 +02:00
Martijn van Groningen a4b0f66919
Add enrich stats api ()
The enrich api returns enrich coordinator stats and
information about currently executing enrich policies.

The coordinator stats include per ingest node:
* The current number of search requests in the queue.
* The total number of outstanding remote requests that
  have been executed since node startup. Each remote
  request is likely to include multiple search requests.
  This depends on how much search requests are in the
  queue at the time when the remote request is performed.
* The number of current outstanding remote requests.
* The total number of search requests that `enrich`
  processors have executed since node startup.

The current execution policies stats include:
* The name of policy that is executing
* A full blow task info object that is executing the policy.

Relates to 
2019-09-11 13:40:24 +02:00
James Rodewig a27d075db4
[DOCS] Update "Enrich your data" tutorials ()
* Move enrich docs to separate file

* Rewrite enrich processor tutorial
2019-09-11 13:08:48 +02:00
James Rodewig d74d995382
[DOCS] Separate Enrich API Docs ()
* Add enrich policy common parameter

* Add enrich APIs to REST APIs index

* Add put enrich policy API docs

* Add get enrich policy API docs

* Add delete enrich policy API docs

* Add execute enrich policy API docs
2019-09-11 13:08:28 +02:00
Colin Goodheart-Smithe 31697c71bd
Updates 7.4.0 release notes 2019-09-11 10:51:04 +01:00
Colin Goodheart-Smithe ac52588a7a
Updates 7.4.0 release notes 2019-09-11 10:14:31 +01:00
Julie Tibshirani 10da998dfa Expand documentation around global ordinals. ()
This commit updates the eager_global_ordinals documentation to give more
background on what global ordinals are and when they are used. The docs also now
mention that global ordinal loading may be expensive, and describes the cases
where in which loading them can be avoided.
2019-09-10 11:04:07 -07:00
Abhilash Bolla 20e93bca6b Fixed grammar in pattern replace char filter docs. ()
Minor grammar fix in the pattern replace char filter docs.
2019-09-10 11:04:07 -07:00
Julie Tibshirani 1b9bd9a0a8 Use a literal block in the field data docs. ()
Currently we use `quote`, which renders a bit strangely on the website.
2019-09-10 11:04:07 -07:00
Lisa Cawley 7461259ba6 [DOCS] Adds missing icons to ML HLRC APIs () 2019-09-10 08:28:02 -07:00
Lee Hinman cdc3a260af
Add retention to Snapshot Lifecycle Management (backport of #4… ()
* Add retention to Snapshot Lifecycle Management ()

This commit adds retention to the existing Snapshot Lifecycle Management feature () as described in . This allows a user to configure SLM to automatically delete older snapshots based on a number of criteria.

An example policy would look like:

```
PUT /_slm/policy/snapshot-every-day
{
  "schedule": "0 30 2 * * ?",
  "name": "<production-snap-{now/d}>",
  "repository": "my-s3-repository",
  "config": {
    "indices": ["foo-*", "important"]
  },
  // Newly configured retention options
  "retention": {
    // Snapshots should be deleted after 14 days
    "expire_after": "14d",
    // Keep a maximum of thirty snapshots
    "max_count": 30,
    // Keep a minimum of the four most recent snapshots
    "min_count": 4
  }
}
```

SLM Retention is run on a scheduled configurable with the `slm.retention_schedule` setting, which supports cron expressions. Deletions are run for a configurable time bounded by the `slm.retention_duration` setting, which defaults to 1 hour.

Included in this work is a new SLM stats API endpoint available through

``` json
GET /_slm/stats
```

That returns statistics about snapshot taken and deleted, as well as successful retention runs, failures, and the time spent deleting snapshots.  has more information as well as an example of the output. These stats are also included when retrieving SLM policies via the API.

* Add base framework for snapshot retention ()

* Add base framework for snapshot retention

This adds a basic `SnapshotRetentionService` and `SnapshotRetentionTask`
to start as the basis for SLM's retention implementation.

Relates to 

* Remove extraneous 'public'

* Use a local var instead of reading class var repeatedly

* Add SnapshotRetentionConfiguration for retention configuration ()

* Add SnapshotRetentionConfiguration for retention configuration

This commit adds the `SnapshotRetentionConfiguration` class and its HLRC
counterpart to encapsulate the configuration for SLM retention.
Currently only a single parameter is supported as an example (we still
need to discuss the different options we want to support and their
names) to keep the size of the PR down. It also does not yet include version serialization checks
since the original SLM branch has not yet been merged.

Relates to 

* Fix REST tests

* Fix more documentation

* Use Objects.equals to avoid NPE

* Put `randomSnapshotLifecyclePolicy` in only one place

* Occasionally return retention with no configuration

* Implement SnapshotRetentionTask's snapshot filtering and delet… ()

* Implement SnapshotRetentionTask's snapshot filtering and deletion

This commit implements the snapshot filtering and deletion for
`SnapshotRetentionTask`. Currently only the expire-after age is used for
determining whether a snapshot is eligible for deletion.

Relates to 

* Fix deletes running on the wrong thread

* Handle missing or null policy in snap metadata differently

* Convert Tuple<String, List<SnapshotInfo>> to Map<String, List<SnapshotInfo>>

* Use the `OriginSettingClient` to work with security, enhance logging

* Prevent NPE in test by mocking Client

* Allow empty/missing SLM retention configuration ()

Semi-related to , this allows the `"retention"` configuration map
to be missing.

Relates to 

* Add min_count and max_count as SLM retention predicates ()

This adds the configuration options for `min_count` and `max_count` as
well as the logic for determining whether a snapshot meets this criteria
to SLM's retention feature.

These options are optional and one, two, or all three can be specified
in an SLM policy.

Relates to 

* Time-bound deletion of snapshots in retention delete function ()

* Time-bound deletion of snapshots in retention delete function

With a cluster that has a large number of snapshots, it's possible that
snapshot deletion can take a very long time (especially since deletes
currently have to happen in a serial fashion). To prevent snapshot
deletion from taking forever in a cluster and blocking other operations,
this commit adds a setting to allow configuring a maximum time to spend
deletion snapshots during retention. This dynamic setting defaults to 1
hour and is best-effort, meaning that it doesn't hard stop a deletion
at an hour mark, but ensures that once the time has passed, all
subsequent deletions are deferred until the next retention cycle.

Relates to 

* Wow snapshots suuuure can take a long time.

* Use a LongSupplier instead of actually sleeping

* Remove TestLogging annotation

* Remove rate limiting

* Add SLM metrics gathering and endpoint ()

* Add SLM metrics gathering and endpoint

This commit adds the infrastructure to gather metrics about the different SLM actions that a cluster
takes. These actions are stored in `SnapshotLifecycleStats` and perpetuated in cluster state. The
stats stored include the number of snapshots taken, failed, deleted, the number of retention runs,
as well as per-policy counts for snapshots taken, failed, and deleted. It also includes the amount
of time spent deleting snapshots from SLM retention.

This commit also adds an endpoint for retrieving all stats (further commits will expose this in the
SLM get-policy API) that looks like:

```
GET /_slm/stats
{
  "retention_runs" : 13,
  "retention_failed" : 0,
  "retention_timed_out" : 0,
  "retention_deletion_time" : "1.4s",
  "retention_deletion_time_millis" : 1404,
  "policy_metrics" : {
    "daily-snapshots2" : {
      "snapshots_taken" : 7,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 6,
      "snapshot_deletion_failures" : 0
    },
    "daily-snapshots" : {
      "snapshots_taken" : 12,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 12,
      "snapshot_deletion_failures" : 6
    }
  },
  "total_snapshots_taken" : 19,
  "total_snapshots_failed" : 0,
  "total_snapshots_deleted" : 18,
  "total_snapshot_deletion_failures" : 6
}
```

This does not yet include HLRC for this, as this commit is quite large on its own. That will be
added in a subsequent commit.

Relates to 

* Version qualify serialization

* Initialize counters outside constructor

* Use computeIfAbsent instead of being too verbose

* Move part of XContent generation into subclass

* Fix REST action for master merge

* Unused import

*  Record history of SLM retention actions ()

This commit records the deletion of snapshots by the retention component
of SLM into the SLM history index for the purposes of reviewing operations
taken by SLM and alerting.

* Retry SLM retention after currently running snapshot completes ()

* Retry SLM retention after currently running snapshot completes

This commit adds a ClusterStateObserver to wait until the currently
running snapshot is complete before proceeding with snapshot deletion.
SLM retention waits for the maximum allowed deletion time for the
snapshot to complete, however, the waiting time is not factored into
the limit on actual deletions.

Relates to 

* Increase timeout waiting for snapshot completion

* Apply patch

From 2374316f0d.patch

* Rename test variables

* [TEST] Be less strict for stats checking

* Skip SLM retention if ILM is STOPPING or STOPPED ()

This adds a check to ensure we take no action during SLM retention if
ILM is currently stopped or in the process of stopping.

Relates to 

* Check all actions preventing snapshot delete during retention ()

* Check all actions preventing snapshot delete during retention run

Previously we only checked to see if a snapshot was currently running,
but it turns out that more things can block snapshot deletion. This
changes the check to be a check for:

- a snapshot currently running
- a deletion already in progress
- a repo cleanup in progress
- a restore currently running

This was found by CI where a third party delete in a test caused SLM
retention deletion to throw an exception.

Relates to 

* Add unit test for okayToDeleteSnapshots

* Fix bug where SLM retention task would be scheduled on every node

* Enhance test logging

* Ignore if snapshot is already deleted

* Missing import

* Fix SnapshotRetentionServiceTests

* Expose SLM policy stats in get SLM policy API ()

This also adds support for the SLM stats endpoint to the high level rest client.

Retrieving a policy now looks like:

```json
{
  "daily-snapshots" : {
    "version": 1,
    "modified_date": "2019-04-23T01:30:00.000Z",
    "modified_date_millis": 1556048137314,
    "policy" : {
      "schedule": "0 30 1 * * ?",
      "name": "<daily-snap-{now/d}>",
      "repository": "my_repository",
      "config": {
        "indices": ["data-*", "important"],
        "ignore_unavailable": false,
        "include_global_state": false
      },
      "retention": {}
    },
    "stats": {
      "snapshots_taken": 0,
      "snapshots_failed": 0,
      "snapshots_deleted": 0,
      "snapshot_deletion_failures": 0
    },
    "next_execution": "2019-04-24T01:30:00.000Z",
    "next_execution_millis": 1556048160000
  }
}
```

Relates to 

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase ()

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase

This commit splits `SnapshotLifecycleIT` into two different tests.
`SnapshotLifecycleRestIT` which includes the tests that do not require
slow repositories, and `SLMSnapshotBlockingIntegTests` which is now an
integration test using `MockRepository` to simulate a snapshot being in
progress.

Relates to 
Resolves 

* Add error logging when exceptions are thrown

* Update serialization versions

* Fix type inference

* Use non-Cancellable HLRC return value

* Fix Client mocking in test

* Fix SLMSnapshotBlockingIntegTests for 7.x branch

* Update SnapshotRetentionTask for non-multi-repo snapshot retrieval

* Add serialization guards for SnapshotLifecyclePolicy
2019-09-10 09:08:09 -06:00
James Rodewig b59ecde041
[DOCS] [2 of 5] Change // CONSOLE comments to [source,console] () () 2019-09-09 13:38:14 -04:00
Zachary Tong b21e417181 [DOCS] Add 7.3.2 Release Notes () 2019-09-09 13:36:11 -04:00
James Rodewig e253ee6ba6
[DOCS] Change // CONSOLE comments to [source,console] () () 2019-09-09 12:35:50 -04:00
Suhel Khan d5529cb0bb [Docs] Fix typo in minimum-should-match.asciidoc () 2019-09-09 14:17:19 +02:00
David Turner cc092b1be1 Add support for OneZoneInfrequentAccess storage ()
The `repository-s3` plugin has supported a storage class of `onezone_ia` since
the SDK upgrade in , but we do not test or document this fact. This
commit adds this storage class to the docs and adds a test to ensure that the
documented storage classes are all accepted by S3 too.

Fixes 
2019-09-09 07:54:44 +01:00
Martijn van Groningen c057fce978
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-09-09 08:40:54 +02:00
James Rodewig a6cc0deaa0 [DOCS] Remove cat request from Index Segments API requests () 2019-09-06 16:47:00 -04:00
James Rodewig f04573f8e8
[DOCS] [5 of 5] Change // TESTRESPONSE comments to [source,console-results] () () 2019-09-06 16:09:09 -04:00
James Rodewig 61756597ae [DOCS] Correct definition for `allow_no_indices` parameter () 2019-09-06 14:11:16 -04:00
Anton 6ae1ae9c9a [Docs] Fix typo in field-names-field.asciidoc () 2019-09-06 18:04:28 +02:00
James Rodewig b05c0f333d [DOCS] Add index alias definition to glossary () 2019-09-06 11:35:55 -04:00
James Rodewig c46c57d439
[DOCS] Change // CONSOLE comments to [source,console] () () 2019-09-06 11:31:13 -04:00
James Rodewig 31b4e2f6df
[DOCS] Resort common-parms () () 2019-09-06 10:30:43 -04:00
James Rodewig bb7bff5e30
[DOCS] Replace "// TESTRESPONSE" magic comments with "[source,console-result] () () 2019-09-06 09:22:08 -04:00
István Zoltán Szabó 8208ffa666 [DOCS] Adds progress parameter description to the GET stats data frame analytics API doc. () 2019-09-06 15:18:57 +02:00
markharwood 323ec022be
Deprecate the "index.max_adjacency_matrix_filters" index setting ()
Following performance optimisations to the adjacency_matrix aggregation we no longer require this setting. Marked as deprecated and due for removal in 8.0

Related 
2019-09-06 13:59:47 +01:00
lois.左 27889b3d98 Fix typo in update API example () 2019-09-06 10:31:59 +02:00
Jim Ferenczi f2a6c88f83
Add a system property to ignore awareness attributes ()
This is a follow up of  for 7.x.
This change adds a system property called "es.routing.search_ignore_awareness_attributes" that when set to true will
effectively ignore allocation awareness attributes when routing search and get requests. This is now the default in 8.x so this
commit adds a way to opt-in to this new behavior in a minor version of 7.x.

Relates 
2019-09-06 09:29:27 +02:00
James Rodewig 327da31db4 [DOCS] Reformat index stats API docs () 2019-09-05 15:45:02 -04:00
Jason Tedor af1516a377
Add docs on upgrading the keystore ()
This commit adds a note to the docs regarding upgrading the keystore.
2019-09-05 14:41:31 -04:00
Lisa Cawley e078d62247 [DOCS] Identify reloadable Azure repository plugin settings () 2019-09-05 11:25:26 -07:00
Benjamin Trent bac5cb1c7a
[ML][Transforms] update supported aggs docs () () 2019-09-05 10:30:40 -05:00
James Rodewig 96a4dc21e0 [DOCS] Re-add versioning to put template docs ()
Adds documentation for index template versioning
accidentally removed with .
2019-09-05 10:18:36 -04:00
James Rodewig 1f36c4e50c
[DOCS] Replace "// CONSOLE" comments with [source,console] () () 2019-09-05 10:11:25 -04:00
James Rodewig d5597c9ba6 [DOCS] Reformat index segments API docs () 2019-09-05 08:34:59 -04:00
Adrien Grand 3f5f028e2c Fix release version of . 2019-09-05 14:07:00 +02:00
Przemyslaw Gomulka 9563d65d1e
Improve documentation for X-Opaque-ID () ()
this field can be present in search slow logs and deprecation logs. The
docs describes how to enable this functionality and what expect in logs.
closes 
2019-09-05 08:50:35 +02:00
Lisa Cawley b11968cf4d [DOCS] Synchs Watcher API titles with better HLRC titles () 2019-09-04 17:04:19 -07:00
Lisa Cawley da5618cbf4 [DOCS] Identify reloadable GCS repository plugin settings () 2019-09-04 16:28:44 -07:00
Lisa Cawley 9faf1d956f [DOCS] Identify reloadable S3 repository plugin settings () 2019-09-04 14:46:51 -07:00