Commit Graph

6633 Commits

Author SHA1 Message Date
István Zoltán Szabó 9600ab4f57 [DOCS] Adds recommendation on dedicated master-eligible nodes (#51674)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-01-31 12:56:58 +01:00
Sven Schliesing dc99acee66 [Docs] Fix typo in node-tool.asciidoc (#51667) 2020-01-31 10:36:21 +01:00
Yang Wang 77b00fc0c0
Add warnings for invalid realm order config (#51195) (#51515)
The changes are to help users prepare for migration to next major
release (v8.0.0) regarding to the break change of realm order config.

Warnings are added for when:
* A realm does not have an order config
* Multiple realms have the same order config

The warning messages are added to both deprecation API and loggings.
The main reasons for doing this are: 1) there is currently no automatic relay
between the two; 2) deprecation API is under basic and we need logging
for OSS.
2020-01-31 12:32:37 +11:00
Lisa Cawley 1a40ebfa67 [DOCS] Adds missing testenv attribute (#51719) 2020-01-30 16:15:17 -08:00
Lee Hinman b9faa0733d
[7.x] Rename ILM history index enablement setting (#51698) (#51705)
* Rename ILM history index enablement setting

The previous setting was `index.lifecycle.history_index_enabled`, this commit changes it to
`indices.lifecycle.history_index_enabled` to indicate this is not an index-level setting (it's node
level).
2020-01-30 15:27:44 -07:00
James Rodewig 36b2663e98 [DOCS] Add attribute for Lucene analysis links (#51687)
Adds a `lucene-analysis-docs` attribute for the Lucene `/analysis/`
javadocs directory. This should prevent typos and keep the docs DRY.
2020-01-30 11:24:01 -05:00
James Rodewig 4fcf5a9de4 [DOCS] Rewrite analysis intro (#51184)
* [DOCS] Rewrite analysis intro. Move index/search analysis content.

* Rewrites 'Text analysis' page intro as high-level definition.
  Adds guidance on when users should configure text analysis
* Rewrites and splits index/search analysis content:
  * Conceptual content -> 'Index and search analysis' under 'Concepts'
  * Task-based content -> 'Specify an analyzer' under 'Configure...'
* Adds detailed examples for when to use the same index/search analyzer
  and when not.
* Adds new example snippets for specifying search analyzers

* clarifications

* Add toc. Decrement headings.

* Reword 'When to configure' section

* Remove sentence from tip
2020-01-30 09:32:16 -05:00
Marios Trivyzas f373020349 SQL: Fix ORDER BY YEAR() function (#51562)
Previously, if YEAR() was used as and ORDER BY argument without being
wrapped with another scalar (e.g. YEAR(birth_date) + 10), no script
ordering was used but instead the underlying field (e.g. birth_date)
was used instead as a performance optimisation. This works correctly if
YEAR() is the only ORDER BY arg but if further args are used as tie
breakers for the ordering wrong results are produced. This is because
2 rows with the different birth_date but on the same year are not tied
as the underlying ordering is on birth_date and not on the
YEAR(birth_date), and the following ORDER BY args are ignored.

Remove this optimisation for YEAR() to avoid incorrect results in
such cases.

As a consequence another bug is revealed: scalar functions on top
of nested fields produce scripted sorting/filtering which is not yet
supported. In such cases no error was thrown but instead all values for
such nested fields were null and were passed to the script implementing
the sorting/filtering, producing incorrect results.

Detect such cases and throw a validation exception.

Fixes: #51224
(cherry picked from commit f41efd6753dc3650a7eabb3e07b02b3b32c5704c)
2020-01-30 15:29:36 +01:00
Nhat Nguyen f0fad5b622
Deprecate translog retention settings (#51588) (#51638)
This change deprecates the translog retention settings as they are
effectively ignored since 7.4.

Relates #50775
Relates #45473
2020-01-30 09:03:10 -05:00
Henning Andersen ca8601373a [DOCS] Task management API experimental status issue (#51634)
Add issue reference to documentation.

Relates #51628
2020-01-30 14:15:47 +01:00
Peter Dyson b5a2ee5be2 [DOCS] Fix minor typo affecting formatting (#51655) 2020-01-29 23:44:09 -08:00
Lisa Cawley 28f2f3dd02 [DOCS] Minor fixes in transform documentation (#51633) 2020-01-29 16:58:18 -08:00
Lisa Cawley fdf74f6ae4 [DOCS] Removes beta qualifiers from transform documentation (#51553) 2020-01-29 08:41:54 -08:00
Lisa Cawley 3f4156e95a
[DOCS] Adds release highlight for transforms (#51555) 2020-01-29 08:35:02 -08:00
Albert Zaharovits 90285ee907
Deprecate timeout.tcp_read AD/LDAP realm setting (#47305)
The timeout.tcp_read AD/LDAP realm setting, despite the low-level
allusion, controls the time interval the realms wait for a response for
a query (search or bind). If the connection to the server is synchronous
(un-pooled) the response timeout is analogous to the tcp read timeout.
But the tcp read timeout is irrelevant in the common case of a pooled
connection (when a Bind DN is specified).

The timeout.tcp_read qualifier is hereby deprecated in favor of
timeout.response.

In addition, the default value for both timeout.tcp_read and
timeout.response is that of timeout.ldap_search, instead of the 5s (but
the default for timeout.ldap_search is still 5s). The
timeout.ldap_search defines the server-controlled timeout of a search
request. There is no practical use case to have a smaller tcp_read
timeout compared to ldap_search (in this case the request would time-out
on the client but continue to be processed on the server). The proposed
change aims to simplify configuration so that the more common
configuration change, adjusting timeout.ldap_search up, has the expected
result (no timeout during searches) without any additional
modifications.

Closes #46028
2020-01-29 10:48:26 +02:00
Gordon Brown 89c2834b24
Deprecate creation of dot-prefixed index names except for hidden and system indices (#49959)
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered by
a plugin that extends SystemIndexPlugin. This is the first step
towards more thorough protections for system indices.

This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register .tasks as a system index.
2020-01-28 10:01:16 -07:00
James Rodewig 139305ffc8 [DOCS] Document `indices` cluster stats (#50527)
Documents the header and `indices` response parameters returned by the
`_cluster/stats` API.

Co-Authored-By: David Turner <david.turner@elastic.co>
2020-01-28 11:00:00 -05:00
Yannick Welsch fa212fe60b Stricter checks of setup and teardown in docs tests (#51430)
Adds extra checks due to 7.x backport
2020-01-28 16:52:23 +01:00
Yannick Welsch f6686345c9 Avoid unnecessary setup and teardown in docs tests (#51430)
The docs tests have recently been running much slower than before (see #49753).

The gist here is that with ILM/SLM we do a lot of unnecessary setup / teardown work on each
test. Compounded with the slightly slower cluster state storage mechanism, this causes the
tests to run much slower.

In particular, on RAMDisk, docs:check is taking

ES 7.4: 6:55 minutes
ES master: 16:09 minutes
ES with this commit: 6:52 minutes

on SSD, docs:check is taking

ES 7.4: ??? minutes
ES master: 32:20 minutes
ES with this commit: 11:21 minutes
2020-01-28 16:52:23 +01:00
James Rodewig 70e4ae3381 [DOCS] Reformat unique token filter docs (#50748)
* Updates the description
* Adds analyze, custom analyzer, and custom filter snippets
* Adds parameter documentation
2020-01-28 10:42:25 -05:00
David Roberts 550254ec7f [ML] Use CSV ingest processor in find_file_structure ingest pipeline (#51492)
Changes the find_file_structure response to include a CSV
ingest processor in the ingest pipeline it suggests.

Previously the Kibana file upload functionality parsed CSV
in the browser, but by parsing CSV in the ingest pipeline
it makes the Kibana file upload functionality more easily
interchangable with Filebeat such that the configurations
it creates can more easily be used to import data with the
same structure repeatedly in production.
2020-01-28 14:38:43 +00:00
William Brafford 9efa5be60e
Password-protected Keystore Feature Branch PR (#51123) (#51510)
* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
2020-01-28 05:32:32 -05:00
James Rodewig 65f49d0bba [DOCS] Add top-level EQL docs page. Adds EQL requirements page. (#51334)
* Creates a top-level page for EQL in the ES reference.
   This page contains a high-level introduction and will include a nav for other EQL docs pages as they're built.

* Creates a requirements page.
  This page outlines the fields needed to use EQL in ES.
2020-01-27 16:04:47 -05:00
James Rodewig 23b65390ab [DOCS] Add response snippets to 'Testing analyzers' page (#51427)
Adds response snippets to the `POST _analyze` snippets in the 'Testing
analyzers' page.

Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com>
2020-01-27 08:41:44 -05:00
David Turner 49bde5d286
Remove DEBUG-level default logging from actions (#51459)
In 2bb31fe (v0.6.0!) we added DEBUG-level logging to the default config of
action loggers "for easier debugging". This change to the default config lives
on to this day. It does not obviously make debugging any easier any more, but
it does result in a good deal of log noise sometimes. This commit removes this
special case from the default config.

Closes #51198
2020-01-27 10:50:10 +00:00
Lisa Cawley 931b22349f [DOCS] Adds http to elasticsearch-certutil command reference (#51188) 2020-01-24 09:59:07 -08:00
Mayya Sharipova a29deecbda Revert "Make it clear this is boost at index time (#51390)"
This reverts commit 3d5238bd95.
2020-01-24 11:05:42 -05:00
Jonas F. Henriksen 3d5238bd95 Make it clear this is boost at index time (#51390)
The way it was originally written, it sounds like 
we are boosting at query time.
 Of course, the effect is at query time, 
but the point here is that boosting is done at index time
2020-01-24 10:37:07 -05:00
Benjamin Trent 76660a5a4f
[7.x] [ML][Inference] add tags url param to GET (#51330) (#51404)
* [ML][Inference] add tags url param to GET (#51330)

Adds a new URL parameter, `tags` to the GET _ml/inference/<model_id> endpoint.

This parameter allows the list of models to be further reduced to those who contain all the provided tags.
2020-01-24 08:26:58 -05:00
István Zoltán Szabó 8bdf654cc7 [DOCS] Refines description. (#51400) 2020-01-24 13:34:25 +01:00
David Turner 40e7a826fc Allow decimal max_task_wait_time in docs (#51352)
The regex for the response to `GET _cat/health?v` in `getting-started.asciidoc`
requires `max_task_wait_time` to match `(-|\\d+(micros|ms|s))`, which doesn't
match times such as `3.9ms` that contain a decimal point. This commit adjusts
the regex to match times formatted like this too.

Fixes #47537
2020-01-24 08:59:43 +00:00
Lisa Cawley ec47698f7c [DOCS] Updates categorization examples with wizard screenshots (#51133) 2020-01-22 11:28:17 -08:00
Zachary Tong 83647101ef Update Release notes for BC2 2020-01-22 14:05:59 -05:00
Lisa Cawley 4590d4156a [DOCS] Clarify interval, frequency, and bucket span in ML APIs and example (#51280) 2020-01-22 08:15:46 -08:00
Igor Motov 08e9c673e5 Fix leftover mentions of method parameter in Percentile Aggs (#51272)
The method parameter is not used in the percentile aggs, instead
the method is determined by the presence of `hdr` or `tdigest`
objects.

Relates to #8324
2020-01-22 10:03:35 -05:00
David Kyle ca4b90a001
[ML] Calculate results and snapshot retention using latest bucket timestamps (#51061) (#51301)
The retention period is calculated relative to the last bucket result or snapshot
time rather than wall clock
2020-01-22 14:52:33 +00:00
Zachary Tong b38fdf9f94
Add release highlights for 7.6 (#51070)
Add release highlights for 7.6

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-01-22 09:42:57 -05:00
Russ Cam 86a50a24f3 [Docs] Including leading slash in range query doc example URLs (#51277) 2020-01-22 09:42:18 +01:00
Stuart Tettemer 41c15b438d
Scripting: Add char position of script errors (#51069) (#51266)
Add the character position of a scripting error to error responses.

The contents of the `position` field are experimental and subject to
change.  Currently, `offset` refers to the character location where the
error was encountered, `start` and `end` define a range of characters
that contain the error.

eg.
```
{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "y = x;",
          "     ^---- HERE"
        ],
        "script": "def x = new ArrayList(); Map y = x;",
        "lang": "painless",
        "position": {
          "offset": 33,
          "start": 29,
          "end": 35
        }
      }
```

Refs: #50993
2020-01-21 13:45:59 -07:00
István Zoltán Szabó 30d1587ad5 [DOCS] Fixes indentation in inference processor code snippet (#51252) 2020-01-21 16:22:16 +01:00
Lisa Cawley 79cf0894fa
[DOCS] Adds ML PRs to release notes (#51234) 2020-01-20 11:08:23 -08:00
Jason Tedor e20459c202
Exclude autoscaling docs from release docs (#51190)
Since autoscaling is currently only under development, this commit
causes the autoscaling docs to be excluded any time that release docs
are being built.
2020-01-20 10:52:47 -05:00
Andrei Stefan 2908b7e5fc
SQL: add support for passing query parameters in REST API calls (#51029) (#51222)
* REST PreparedStatement-like query parameters are now supported in the form of an array of non-object, non-array values where ES SQL parser will try to infer the data type of the value being passed as parameter.

(cherry picked from commit 45b8bf619aecb1c03d7bc0cf06928dcc36005a66)
2020-01-20 16:40:19 +02:00
István Zoltán Szabó 424b4ed4ea [DOCS] Expands the documentation of Node Query Cache (#51105)
Co-authored-by: debadair <debadair@elastic.co>
2020-01-20 11:13:29 +01:00
Jess 4b31ad1c0c [Docs] Small edits to Ranking Evaluation API docs (#51116)
Small updates to grammar, syntax, and unclear wordings.
2020-01-20 10:30:23 +01:00
István Zoltán Szabó e40580c24d [DOCS] Removes CCS limitation item from Transforms limitations. (#51151) 2020-01-20 09:44:11 +01:00
Jason Tedor 9ce4d2b901
Initial autoscaling commit (#51161)
This commit merely adds the skeleton for the autoscaling project, adding
the basics to include the autoscaling module in the default
distribution, opt-in to code formatting, and a placeholder for the docs.
2020-01-17 15:31:12 -05:00
István Zoltán Szabó 83c92cf7eb [DOCS] Adds text about data types to the categorization docs (#51145) 2020-01-17 10:00:20 -08:00
lcawl fee1a9528c [DOCS] Removes duplicate title 2020-01-17 09:49:42 -08:00
Jay Modi 107989df3e
Introduce hidden indices (#51164)
This change introduces a new feature for indices so that they can be
hidden from wildcard expansion. The feature is referred to as hidden
indices. An index can be marked hidden through the use of an index
setting, `index.hidden`, at creation time. One primary use case for
this feature is to have a construct that fits indices that are created
by the stack that contain data used for display to the user and/or
intended for querying by the user. The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

Hidden indices have the following properties:
* API calls for all indices (empty indices array, _all, or *) will not
  return hidden indices by default.
* Wildcard expansion will not return hidden indices by default unless
  the wildcard pattern begins with a `.`. This behavior is similar to
  shell expansion of wildcards.
* REST API calls can enable the expansion of wildcards to hidden
  indices with the `expand_wildcards` parameter. To expand wildcards
  to hidden indices, use the value `hidden` in conjunction with `open`
  and/or `closed`.
* Creation of a hidden index will ignore global index templates. A
  global index template is one with a match-all pattern.
* Index templates can make an index hidden, with the exception of a
  global index template.
* Accessing a hidden index directly requires no additional parameters.

Backport of #50452
2020-01-17 10:09:01 -07:00
Dimitris Athanasiou b70ebdeb96
[7.x][ML] DF Analytics _explain API should skip object fields (#51115) (#51147)
Object fields cannot be used as features. At the moment _explain
API includes them and even worse it allows it does not error when
an object field is excluded. This creates the expectation to the
user that all children fields will also be excluded while it's not
the case.

This commit omits object fields from the _explain API and also
adds an error if an object field is included or excluded.

Backport of #51115
2020-01-17 14:02:59 +02:00
James Rodewig 2353fe47fc [DOCS] Adds placeholder for 7.5.2 release notes (#51124)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-01-16 14:42:24 -05:00
James Rodewig b6bf64b969 [DOCS] Collapse node stats response sections (#51063)
elastic/docs#1687 added support for the `[%collapsible]` Asciidoc
attribute, which creates collapsible sections in the HTML output.

This PR makes two related changes to the nodes stats API documentation:

* Makes the response parameter sections collapsible. This allows users
  to more easily navigate the page without long walls of text.

* Reorders the response parameter sections to match the default order
  returned by the API.

Relates to #47524.
2020-01-16 13:19:29 -05:00
James Rodewig 7ef906fde8 [DOCS] Add tutorials section to analysis topic (#50809)
Adds a 'Configure text analysis' page to house tutorial content for the
analysis topic.

Also relocates the following pages as children as this new page:

* 'Test an analyzer'
* 'Configuring built-in analyzers'
* 'Create a custom analyzer'

I plan to add a tutorial for specifying index-time and search-time
analyzers to this section as part of a future PR.
2020-01-16 13:12:06 -05:00
James Rodewig ef26763ca9 [DOCS] Add concepts section to analysis topic (#50801)
This helps the topic better match the structure of
our machine learning docs, e.g.
https://www.elastic.co/guide/en/machine-learning/7.5/ml-concepts.html

This PR only includes the 'Anatomy of an analyzer' page as a 'Concepts'
child page, but I plan to add other concepts, such as 'Index time vs.
search time', with later PRs.
2020-01-16 13:00:39 -05:00
James Rodewig 1edaf2b101 [DOCS] Retitle analysis reference pages (#51071)
* Changes titles to sentence case.

* Appends pages with 'reference' to differentiate their content from
  conceptual overviews.

* Moves the 'Normalizers' page to end of the Analysis topic pages.
2020-01-16 12:30:51 -05:00
James Rodewig d590150ca2 [DOCS] Fix indent issue in similarity snippet (#51107)
Updates snippet to consistently use 2-space indentation. The snippet
previously used a mix of tab/5-space and 2-space indents.

Co-authored-by: Peter Johnson <wiz@wiz.co.nz>

Co-authored-by: Peter Johnson <peter@geocode.earth>
2020-01-16 11:00:15 -05:00
James Rodewig 1211772f6a [DOCS] Use same index in Cluster Allocation Explain docs (#50936)
Updates several example snippets in the Cluster Allocation Explain API
docs to consistently use the `my_index` index.

Previously, the snippets switches from `my_index` to `idx`, which could
confuse users.

Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com>

Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com>
2020-01-16 09:15:26 -05:00
Ted Timmons b345c7ff31 [Docs] Fix short alias for 'unassigned.for' (#51059)
The short alias for `unassigned.for` is `uf`, not 'ua'.
2020-01-16 12:10:30 +01:00
PND 1d391f7113 [Docs] Fix example output of edge n-gram token filter. (#51085) 2020-01-16 11:34:00 +01:00
Martijn van Groningen 02dfd71efa
Backport: Add pipeline name to ingest metadata (#51050)
Backport: #50467

This commit adds the name of the current pipeline to ingest metadata.
This pipeline name is accessible under the following key: '_ingest.pipeline'.

Example usage in pipeline:
PUT /_ingest/pipeline/2
{
    "processors": [
        {
            "set": {
                "field": "pipeline_name",
                "value": "{{_ingest.pipeline}}"
            }
        }
    ]
}

Closes #42106
2020-01-16 10:50:47 +01:00
Adrien Grand 45d7bdcfd7
Add analysis components and mapping types to the usage API. (#51062)
Knowing about used analysis components and mapping types would be incredibly
useful in order to know which ones may be deprecated or should get more love.

Some field types also act as a proxy to know about feature usage of some APIs
like the `percolator` or `completion` fields types for percolation and the
completion suggester, respectively.
2020-01-16 09:56:41 +01:00
Zachary Tong a62c9e4e69 Fix 7.6 release notes file name 2020-01-15 15:16:48 -05:00
Zachary Tong 8f48c8d312 Add 7.6.0 release notes 2020-01-15 14:10:37 -05:00
Christoph Büscher d291f189a8
Fix hardcoded version replacement in put-dfanalytics.asciidoc #51053
The version replacement for the code snippet should replace 7.6 with the current version,
but doesn't match because of a missing whitespace.

Closes #51052
2020-01-15 18:09:37 +01:00
taku333 65af0a0f0a [DOCS] Add 7.5.1 link to release notes overview (#51022) 2020-01-15 11:53:26 -05:00
Lee Hinman b14a949fa9 Add blurb about ILM-injected unfollow action (#51009)
These injected actions are harmless and safe to ignore for non-CCR indices.

Resolves #50548
2020-01-15 09:46:57 -07:00
Robin Clarke 5ea18cb4af [Docs] Fix sub-heading in start-stop-ilm.asciidoc (#51045)
Removed superfluous `=`.
2020-01-15 16:16:22 +01:00
Przemysław Witek b4a631277a
Add missing docs for new evaluation metrics (#50967) (#51041) 2020-01-15 15:53:42 +01:00
István Zoltán Szabó b570f417c2 [DOCS] Describes the relationship of the time-related settings in anomaly detection docs (#50959)
Co-Authored-By: David Roberts <dave.roberts@elastic.co>
2020-01-15 08:46:04 +01:00
Tim Vernum e41c0b1224
Deprecating kibana_user and kibana_dashboard_only_user roles (#50963)
This change adds a new `kibana_admin` role, and deprecates
the old `kibana_user` and`kibana_dashboard_only_user`roles.

The deprecation is implemented via a new reserved metadata
attribute, which can be consumed from the API and also triggers
deprecation logging when used (by a user authenticating to
Elasticsearch).

Some docs have been updated to avoid references to these
deprecated roles.

Backport of: #46456

Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
2020-01-15 11:07:19 +11:00
Yannick Welsch 4b0581f182 Remove custom metadata tool (#50813)
Adds a command-line tool to remove broken custom metadata from the cluster state.

Relates to #48701
2020-01-14 23:08:33 +01:00
James Rodewig a290762df1 [DOCS] Document `breakers`, `script`, and `discovery` node stats (#50509)
Documents the `breakers`, `script`, and `discovery` parameters returned
by the `_nodes/stats` API.
2020-01-14 16:51:50 -05:00
Christoph Büscher 2f13751bad
Deprecate and remove camel-case nGram and edgeNGram tokenizers (#50862) (#50991)
We deprecated and removed the camel-case versions of the nGram and edgeNGram
filters a while ago and we should do the same with the nGram and edgeNGram tokenizers.
This PR deprecates the use of these names in favour of ngram and edge_ngram in
7. Usage will be disallowed on new indices starting with 8 then.
2020-01-14 21:42:34 +01:00
lcawl 6848dee84b [DOCS] Fixes typo in keystore command 2020-01-14 11:57:02 -08:00
Tal Levy 9ee2e11181
[7.x] Adds support for geo-bounds filtering in geogrid aggregations (#50996)
* Adds support for geo-bounds filtering in geogrid aggregations (#50002)

It is fairly common to filter the geo point candidates in
geohash_grid and geotile_grid aggregations according to some
viewable bounding box. This change introduces the option of
specifying this filter directly in the tiling aggregation.

This is even more relevant to `geo_shape` where the bounds will restrict
the shape to be within the bounds

this optional `bounds` parameter is parsed in an equivalent fashion to
the bounds specified in the geo_bounding_box query.
2020-01-14 11:18:46 -08:00
Dimitris Athanasiou 1d8cb3c741
[7.x][ML] Add num_top_feature_importance_values param to regression and classi… (#50914) (#50976)
Adds a new parameter to regression and classification that enables computation
of importance for the top most important features. The computation of the importance
is based on SHAP (SHapley Additive exPlanations) method.

Backport of #50914
2020-01-14 16:46:09 +02:00
James Rodewig f028ab08d1 [DOCS] Use `s` parameter in cat API overview example (#50616)
Updates a snippet to use the `s` query string parameter rather than
piping the output to a separate `sort` command.

This ensures the snippet is tested and available in clients other than
curl (Kibana console, etc.).

Issue was originally raised by @hackaholic in #40926.
2020-01-14 08:22:07 -05:00
Yannick Welsch 22ba759e1f
Move metadata storage to Lucene (#50928)
* Move metadata storage to Lucene (#50907)

Today we split the on-disk cluster metadata across many files: one file for the metadata of each
index, plus one file for the global metadata and another for the manifest. Most metadata updates
only touch a few of these files, but some must write them all. If a node holds a large number of
indices then it's possible its disks are not fast enough to process a complete metadata update before timing out. In severe cases affecting master-eligible nodes this can prevent an election
from succeeding.

This commit uses Lucene as a metadata storage for the cluster state, and is a squashed version
of the following PRs that were targeting a feature branch:

* Introduce Lucene-based metadata persistence (#48733)

This commit introduces `LucenePersistedState` which master-eligible nodes
can use to persist the cluster metadata in a Lucene index rather than in
many separate files.

Relates #48701

* Remove per-index metadata without assigned shards (#49234)

Today on master-eligible nodes we maintain per-index metadata files for every
index. However, we also keep this metadata in the `LucenePersistedState`, and
only use the per-index metadata files for importing dangling indices. However
there is no point in importing a dangling index without any shard data, so we
do not need to maintain these extra files any more.

This commit removes per-index metadata files from nodes which do not hold any
shards of those indices.

Relates #48701

* Use Lucene exclusively for metadata storage (#50144)

This moves metadata persistence to Lucene for all node types. It also reenables BWC and adds
an interoperability layer for upgrades from prior versions.

This commit disables a number of tests related to dangling indices and command-line tools.
Those will be addressed in follow-ups.

Relates #48701

* Add command-line tool support for Lucene-based metadata storage (#50179)

Adds command-line tool support (unsafe-bootstrap, detach-cluster, repurpose, & shard
commands) for the Lucene-based metadata storage.

Relates #48701

* Use single directory for metadata (#50639)

Earlier PRs for #48701 introduced a separate directory for the cluster state. This is not needed
though, and introduces an additional unnecessary cognitive burden to the users.

Co-Authored-By: David Turner <david.turner@elastic.co>

* Add async dangling indices support (#50642)

Adds support for writing out dangling indices in an asynchronous way. Also provides an option to
avoid writing out dangling indices at all.

Relates #48701

* Fold node metadata into new node storage (#50741)

Moves node metadata to uses the new storage mechanism (see #48701) as the authoritative source.

* Write CS asynchronously on data-only nodes (#50782)

Writes cluster states out asynchronously on data-only nodes. The main reason for writing out
the cluster state at all is so that the data-only nodes can snap into a cluster, that they can do a
bit of bootstrap validation and so that the shard recovery tools work.
Cluster states that are written asynchronously have their voting configuration adapted to a non
existing configuration so that these nodes cannot mistakenly become master even if their node
role is changed back and forth.

Relates #48701

* Remove persistent cluster settings tool (#50694)

Adds the elasticsearch-node remove-settings tool to remove persistent settings from the on
disk cluster state in case where it contains incompatible settings that prevent the cluster from
forming.

Relates #48701

* Make cluster state writer resilient to disk issues (#50805)

Adds handling to make the cluster state writer resilient to disk issues. Relates to #48701

* Omit writing global metadata if no change (#50901)

Uses the same optimization for the new cluster state storage layer as the old one, writing global
metadata only when changed. Avoids writing out the global metadata if none of the persistent
fields changed. Speeds up server:integTest by ~10%.

Relates #48701

* DanglingIndicesIT should ensure node removed first (#50896)

These tests occasionally failed because the deletion was submitted before the
restarting node was removed from the cluster, causing the deletion not to be
fully acked. This commit fixes this by checking the restarting node has been
removed from the cluster.

Co-authored-by: David Turner <david.turner@elastic.co>

* fix tests

Co-authored-by: David Turner <david.turner@elastic.co>
2020-01-14 09:35:43 +01:00
Nhat Nguyen f0924e6d5b Remove outdated requirement of CCR (#50859)
With retention leases, users do not need to set 
index.soft_deletes.retention.operations. This change removes it from the
requirements of CCR
2020-01-13 20:00:23 -05:00
Nhat Nguyen fb32a55dd5 Deprecate synced flush (#50835)
A normal flush has the same effect as a synced flush on Elasticsearch 
7.6 or later. It's deprecated in 7.6 and will be removed in 8.0.

Relates #50776
2020-01-13 19:54:38 -05:00
Przemko Robakowski a18736b46d
[7.x] ILM action to wait for SLM policy execution (#50454) (#50943)
* ILM action to wait for SLM policy execution (#50454)

This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes #45067

* Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test

This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable.

Reletes to #50781

* Formatting changes

* Longer timeout

* Fix Map.of in Java8

* Unused import removed
2020-01-14 01:34:33 +01:00
Peter Dyson 4cb525d8d3 [DOCS] Array of index patterns is also valid source indices with transform (#50777) 2020-01-13 15:46:45 -08:00
Lee Hinman 91689e793d
[7.x] Refresh cached phase policy definition if possible on ne… (#50941)
* Refresh cached phase policy definition if possible on new policy

There are some cases when updating a policy does not change the
structure in a significant way. In these cases, we can reread the
policy definition for any indices using the updated policy.

This commit adds this refreshing to the `TransportPutLifecycleAction`
to allow this. It allows us to do things like change the configuration
values for a particular step, even when on that step (for example,
changing the rollover criteria while on the `check-rollover-ready` step).

There are more cases where the phase definition can be reread that just
the ones checked here (for example, removing an action that has already
been passed), and those will be added in subsequent work.

Relates to #48431
2020-01-13 14:31:41 -07:00
Lisa Cawley a82ddfb182 [DOCS] Adds elasticsearch-keystore command reference (#50872) 2020-01-13 13:08:21 -08:00
Nhat Nguyen 05f97d5e1b Revert "Deprecate synced flush (#50835)"
This reverts commit 1a32d7142a.
2020-01-13 11:41:03 -05:00
Nhat Nguyen 1a32d7142a
Deprecate synced flush (#50835)
A normal flush has the same effect as a synced flush on Elasticsearch 
7.6 or later. It's deprecated in 7.6 and will be removed in 8.0.

Relates #50776
2020-01-13 10:58:29 -05:00
Ioannis Kakavas ba37e3c4a0
Disable DiagnosticTrustManager in FIPS 140 (#49888)
This commit changes the default behavior for
xpack.security.ssl.diagnose.trust when running in a FIPS 140 JVM.

More specifically, when xpack.security.fips_mode.enabled is true:

- If xpack.security.ssl.diagnose.trust is not explicitly set, the
    default value of it becomes false and a log message is printed
    on info level, notifying of the fact that the TLS/SSL diagnostic
    messages are not enabled when in a FIPS 140 JVM.
- If xpack.security.ssl.diagnose.trust is explicitly set, the value of
    it is honored, even in FIPS mode.

This is relevant only for 7.x where we support Java 8 in which
SunJSSE can still be used as a FIPS 140 provider for TLS. SunJSSE
in FIPS mode, disallows the use of other TrustManager implementations
than the one shipped with SunJSSE.
2020-01-13 17:04:23 +02:00
junmuz 6718ce0f62 [DOCS] Correct typo in `ignore_malformed` mapping parm docs (#50780) 2020-01-13 09:49:53 -05:00
James Rodewig 4629a9714c [DOCS] Fix time_zone example in range query docs (#50830)
One of the example snippets in the range query docs was missing a
required 'T' in the `date` format. This adds the required 'T'.
2020-01-10 08:24:48 -05:00
debadair 83d961391b
[DOCS] Move snapshot-restore out of modules. (#49618) (#50829)
* [DOCS] Move snapshot-restore docs out of modules.

* [DOCS] Incorporates comments from @jrodewig.

* [DOCS] Fix snippet tests
2020-01-09 16:55:46 -08:00
Matt Braymer-Hayes 344c21813b [DOCS] Fix typo in refresh API docs (#50759) 2020-01-09 14:41:19 -05:00
Lisa Cawley ef1c14ad01 [DOCS] Update license expiry links (#50812) 2020-01-09 11:28:43 -08:00
Nik Everett 1d8e51f89d
Support offset in composite aggs (#50609) (#50808)
Adds support for the `offset` parameter to the `date_histogram` source
of composite aggs. The `offset` parameter is supported by the normal
`date_histogram` aggregation and is useful for folks that need to
measure things from, say, 6am one day to 6am the next day.

This is implemented by creating a new `Rounding` that knows how to
handle offsets and delegates to other rounding implementations. That
implementation doesn't fully implement the `Rounding` contract, namely
`nextRoundingValue`. That method isn't used by composite aggs so I can't
be sure that any implementation that I add will be correct. I propose to
leave it throwing `UnsupportedOperationException` until I need it.

Closes #48757
2020-01-09 14:11:24 -05:00
lcawl 8a5de4f56f [DOCS] Clarify detector_index property in ML APIs (#50723) 2020-01-09 08:34:34 -08:00
Benjamin Trent 3e014d39c2
[Transform] fail to start/put on missing pipeline (#50701) (#50795)
If a pipeline referenced by a transform does not exist, we should not allow the transform to be created. 

We do allow the pipeline existence check to be skipped with defer_validations, but if the pipeline still does not exist on `_start`, the pipeline will fail to start.

relates:  #50135
2020-01-09 10:33:22 -05:00
István Zoltán Szabó 4f150e4961
[7.x][DOCS] Moves analysis resources to PUT DFA API docs (#50793) 2020-01-09 16:21:35 +01:00
István Zoltán Szabó 71afeec7d0 Revert "[DOCS] Moves analysis resources to PUT DFA API docs (#50704)"
This reverts commit 4e1107d5d7.
2020-01-09 14:31:35 +01:00
István Zoltán Szabó 4e1107d5d7 [DOCS] Moves analysis resources to PUT DFA API docs (#50704)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-09 14:13:37 +01:00
István Zoltán Szabó acd73dda1c [DOCS] Improves find_file_structure documentation (#50743)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-09 11:20:29 +01:00