Commit Graph

6603 Commits

Author SHA1 Message Date
Rory Hunter cdb9862495
Clarify use of ES_JAVA_OPTS and Docker (#51984)
Backport of #51867.

Tweak the documentation around configuring the heap size when using
Docker, to state that:

- using `ES_JAVA_OPTS` is the preferred method
- Any `ES_JAVA_OPTS` overrides the defaults in `jvm.options`
- It's possible to bind-mount a custom `jvm.options`
2020-02-06 10:00:14 +00:00
Lisa Cawley 4c2dcf2bde [DOCS] Adds curl explanation to getting started content (#51963) 2020-02-05 19:02:21 -08:00
Przemko Robakowski 6332de40b4
Add empty_value parameter to CSV processor (#51567) (#51966)
* Add empty_value parameter to CSV processor

This change adds `empty_value` parameter to the CSV processor.
This value is used to fill empty fields. Fields will be skipped
if this parameter is ommited. This behavior is the same for both
quoted and unquoted fields.

* docs updated

* Fix compilation problem

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-02-05 23:35:52 +01:00
Lisa Cawley ea1d9e0803 [DOCS] Defines data frame transform stats API objects (#44197) 2020-02-05 12:30:55 -08:00
Lisa Cawley b93ebc29c1 [DOCS] Augments update license API (#51903) 2020-02-05 11:08:11 -08:00
Lee Hinman f8c8a10f05 Add documentation about ILM forcemerge with best_compression (#51893)
This adds the option to the parameter list and a warning about the index being unavailable during
the close and open operations.

Relates to #49974
2020-02-05 09:37:41 -07:00
James Rodewig b70cbc97aa [DOCS] Add EQL syntax page (#51821)
Adds documentation for basic EQL syntax.

Joins, sequences, and other syntax to be added as its supported
in future development.

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-02-05 08:14:07 -05:00
David Kyle 289d4f4f4d [ML] Remove stray field from inference docs (#51870)
model_info_field is not a valid option
2020-02-05 10:50:51 +00:00
Adrien Grand ad9d2f1922
Move analysis/mappings stats to cluster-stats. (#51875)
Closes #51138
2020-02-05 11:02:25 +01:00
debadair c0156cbb5d
Backporting updates to ILM org, overview, & GS (#51898)
* [DOCS] Align with ILM API docs (#48705)

* [DOCS] Reconciled with Snapshot/Restore reorg

* [DOCS] Split off ILM overview to a separate topic. (#51287)

* [DOCS} Split off overview to a separate topic.

* [DOCS] Incorporated feedback from @jrodewig.

* [DOCS] Edit ILM GS tutorial (#51513)

* [DOCS] Edit ILM GS tutorial

* [DOCS] Incorporated review feedback from @andreidan.

* [DOCS] Removed test link & fixed anchor & title.

* Update docs/reference/ilm/getting-started-ilm.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Fixed glossary merge error.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-02-04 16:45:18 -08:00
Florian Kelbert 43a7aadd46 [DOCS] Remove unneeded comma from CSV processor example (#51859) 2020-02-04 09:26:20 -05:00
baifan 60a53f2897 [DOCS] Fix `disk.used_percent` typo in `_cat/nodes` docs (#51854)
Corrects an example for the `disk.used_percent` parameter in `_cat/nodes` API.
2020-02-04 09:15:56 -05:00
Grzegorz Banasiak 87b126bbfc [DOCS] Fix index_prefixes link in 'faster prefix queries' docs (#51833)
Fixes a link in 'faster prefix queries' which incorrectly redirects to index_phrases mapping parameter description instead of index_prefixes.
2020-02-04 08:40:18 -05:00
William Brafford ba2810f23d
Use standard format for reload settings API (#51560) (#51828)
* Use standard format for reload settings API

The reload-secure-settings API page was not reorganized for the standard
API format, so this commit is reorganizing the page and adding some
links to the page in related documentation.

* Fix broken links

* Reorder examples to correctly check API response

* Note that only certain settings are reloadable

* [DOCS] Edits layout

* [DOCS] Removes unnecessary callouts

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-03 18:07:26 -05:00
Dan Hermann 4083eae0b7
[7.x] Secure password for monitoring HTTP exporter (#51775)
Adds a secure and reloadable SECURE_AUTH_PASSWORD setting to allow keystore entries in the form "xpack.monitoring.exporters.*.auth.secure_password" to securely supply passwords for monitoring HTTP exporters. Also deprecates the insecure `AUTH_PASSWORD` setting.
2020-02-03 07:42:30 -06:00
James Rodewig 1545c2ab26 [DOCS] Document node stats response meta (#51263)
Documents several metadata-related parameters returned by the
`GET _nodes/stats` API.
2020-02-03 08:33:57 -05:00
Darren LaCasse 480e9238a4 [DOCS] Remove extra word (#51757) 2020-01-31 10:30:06 -08:00
Zachary Tong 3147453600 Update Release Notes for BC4 2020-01-31 11:41:44 -05:00
Christoph Büscher 86f3b47299
Make `date_range` query rounding consistent with `date` (#50237) (#51741)
Currently the rounding used in range queries can behave differently for `date`
and `date_range` as explained in #50009. The behaviour on `date` fields is
the one we document in https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#range-query-date-math-rounding.
This change adapts the rounding behaviour for RangeType.DATE so it uses the
same logic as the `date` for the `date_range` type.

Backport of #50237
2020-01-31 15:35:05 +01:00
István Zoltán Szabó dfc9f2330c [DOCS] Adds PUT inference API docs (#51231)
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-31 13:13:34 +01:00
István Zoltán Szabó 9600ab4f57 [DOCS] Adds recommendation on dedicated master-eligible nodes (#51674)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-01-31 12:56:58 +01:00
Sven Schliesing dc99acee66 [Docs] Fix typo in node-tool.asciidoc (#51667) 2020-01-31 10:36:21 +01:00
Yang Wang 77b00fc0c0
Add warnings for invalid realm order config (#51195) (#51515)
The changes are to help users prepare for migration to next major
release (v8.0.0) regarding to the break change of realm order config.

Warnings are added for when:
* A realm does not have an order config
* Multiple realms have the same order config

The warning messages are added to both deprecation API and loggings.
The main reasons for doing this are: 1) there is currently no automatic relay
between the two; 2) deprecation API is under basic and we need logging
for OSS.
2020-01-31 12:32:37 +11:00
Lisa Cawley 1a40ebfa67 [DOCS] Adds missing testenv attribute (#51719) 2020-01-30 16:15:17 -08:00
Lee Hinman b9faa0733d
[7.x] Rename ILM history index enablement setting (#51698) (#51705)
* Rename ILM history index enablement setting

The previous setting was `index.lifecycle.history_index_enabled`, this commit changes it to
`indices.lifecycle.history_index_enabled` to indicate this is not an index-level setting (it's node
level).
2020-01-30 15:27:44 -07:00
James Rodewig 36b2663e98 [DOCS] Add attribute for Lucene analysis links (#51687)
Adds a `lucene-analysis-docs` attribute for the Lucene `/analysis/`
javadocs directory. This should prevent typos and keep the docs DRY.
2020-01-30 11:24:01 -05:00
James Rodewig 4fcf5a9de4 [DOCS] Rewrite analysis intro (#51184)
* [DOCS] Rewrite analysis intro. Move index/search analysis content.

* Rewrites 'Text analysis' page intro as high-level definition.
  Adds guidance on when users should configure text analysis
* Rewrites and splits index/search analysis content:
  * Conceptual content -> 'Index and search analysis' under 'Concepts'
  * Task-based content -> 'Specify an analyzer' under 'Configure...'
* Adds detailed examples for when to use the same index/search analyzer
  and when not.
* Adds new example snippets for specifying search analyzers

* clarifications

* Add toc. Decrement headings.

* Reword 'When to configure' section

* Remove sentence from tip
2020-01-30 09:32:16 -05:00
Marios Trivyzas f373020349 SQL: Fix ORDER BY YEAR() function (#51562)
Previously, if YEAR() was used as and ORDER BY argument without being
wrapped with another scalar (e.g. YEAR(birth_date) + 10), no script
ordering was used but instead the underlying field (e.g. birth_date)
was used instead as a performance optimisation. This works correctly if
YEAR() is the only ORDER BY arg but if further args are used as tie
breakers for the ordering wrong results are produced. This is because
2 rows with the different birth_date but on the same year are not tied
as the underlying ordering is on birth_date and not on the
YEAR(birth_date), and the following ORDER BY args are ignored.

Remove this optimisation for YEAR() to avoid incorrect results in
such cases.

As a consequence another bug is revealed: scalar functions on top
of nested fields produce scripted sorting/filtering which is not yet
supported. In such cases no error was thrown but instead all values for
such nested fields were null and were passed to the script implementing
the sorting/filtering, producing incorrect results.

Detect such cases and throw a validation exception.

Fixes: #51224
(cherry picked from commit f41efd6753dc3650a7eabb3e07b02b3b32c5704c)
2020-01-30 15:29:36 +01:00
Nhat Nguyen f0fad5b622
Deprecate translog retention settings (#51588) (#51638)
This change deprecates the translog retention settings as they are
effectively ignored since 7.4.

Relates #50775
Relates #45473
2020-01-30 09:03:10 -05:00
Henning Andersen ca8601373a [DOCS] Task management API experimental status issue (#51634)
Add issue reference to documentation.

Relates #51628
2020-01-30 14:15:47 +01:00
Peter Dyson b5a2ee5be2 [DOCS] Fix minor typo affecting formatting (#51655) 2020-01-29 23:44:09 -08:00
Lisa Cawley 28f2f3dd02 [DOCS] Minor fixes in transform documentation (#51633) 2020-01-29 16:58:18 -08:00
Lisa Cawley fdf74f6ae4 [DOCS] Removes beta qualifiers from transform documentation (#51553) 2020-01-29 08:41:54 -08:00
Lisa Cawley 3f4156e95a
[DOCS] Adds release highlight for transforms (#51555) 2020-01-29 08:35:02 -08:00
Albert Zaharovits 90285ee907
Deprecate timeout.tcp_read AD/LDAP realm setting (#47305)
The timeout.tcp_read AD/LDAP realm setting, despite the low-level
allusion, controls the time interval the realms wait for a response for
a query (search or bind). If the connection to the server is synchronous
(un-pooled) the response timeout is analogous to the tcp read timeout.
But the tcp read timeout is irrelevant in the common case of a pooled
connection (when a Bind DN is specified).

The timeout.tcp_read qualifier is hereby deprecated in favor of
timeout.response.

In addition, the default value for both timeout.tcp_read and
timeout.response is that of timeout.ldap_search, instead of the 5s (but
the default for timeout.ldap_search is still 5s). The
timeout.ldap_search defines the server-controlled timeout of a search
request. There is no practical use case to have a smaller tcp_read
timeout compared to ldap_search (in this case the request would time-out
on the client but continue to be processed on the server). The proposed
change aims to simplify configuration so that the more common
configuration change, adjusting timeout.ldap_search up, has the expected
result (no timeout during searches) without any additional
modifications.

Closes #46028
2020-01-29 10:48:26 +02:00
Gordon Brown 89c2834b24
Deprecate creation of dot-prefixed index names except for hidden and system indices (#49959)
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered by
a plugin that extends SystemIndexPlugin. This is the first step
towards more thorough protections for system indices.

This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register .tasks as a system index.
2020-01-28 10:01:16 -07:00
James Rodewig 139305ffc8 [DOCS] Document `indices` cluster stats (#50527)
Documents the header and `indices` response parameters returned by the
`_cluster/stats` API.

Co-Authored-By: David Turner <david.turner@elastic.co>
2020-01-28 11:00:00 -05:00
Yannick Welsch fa212fe60b Stricter checks of setup and teardown in docs tests (#51430)
Adds extra checks due to 7.x backport
2020-01-28 16:52:23 +01:00
Yannick Welsch f6686345c9 Avoid unnecessary setup and teardown in docs tests (#51430)
The docs tests have recently been running much slower than before (see #49753).

The gist here is that with ILM/SLM we do a lot of unnecessary setup / teardown work on each
test. Compounded with the slightly slower cluster state storage mechanism, this causes the
tests to run much slower.

In particular, on RAMDisk, docs:check is taking

ES 7.4: 6:55 minutes
ES master: 16:09 minutes
ES with this commit: 6:52 minutes

on SSD, docs:check is taking

ES 7.4: ??? minutes
ES master: 32:20 minutes
ES with this commit: 11:21 minutes
2020-01-28 16:52:23 +01:00
James Rodewig 70e4ae3381 [DOCS] Reformat unique token filter docs (#50748)
* Updates the description
* Adds analyze, custom analyzer, and custom filter snippets
* Adds parameter documentation
2020-01-28 10:42:25 -05:00
David Roberts 550254ec7f [ML] Use CSV ingest processor in find_file_structure ingest pipeline (#51492)
Changes the find_file_structure response to include a CSV
ingest processor in the ingest pipeline it suggests.

Previously the Kibana file upload functionality parsed CSV
in the browser, but by parsing CSV in the ingest pipeline
it makes the Kibana file upload functionality more easily
interchangable with Filebeat such that the configurations
it creates can more easily be used to import data with the
same structure repeatedly in production.
2020-01-28 14:38:43 +00:00
William Brafford 9efa5be60e
Password-protected Keystore Feature Branch PR (#51123) (#51510)
* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
2020-01-28 05:32:32 -05:00
James Rodewig 65f49d0bba [DOCS] Add top-level EQL docs page. Adds EQL requirements page. (#51334)
* Creates a top-level page for EQL in the ES reference.
   This page contains a high-level introduction and will include a nav for other EQL docs pages as they're built.

* Creates a requirements page.
  This page outlines the fields needed to use EQL in ES.
2020-01-27 16:04:47 -05:00
James Rodewig 23b65390ab [DOCS] Add response snippets to 'Testing analyzers' page (#51427)
Adds response snippets to the `POST _analyze` snippets in the 'Testing
analyzers' page.

Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com>
2020-01-27 08:41:44 -05:00
David Turner 49bde5d286
Remove DEBUG-level default logging from actions (#51459)
In 2bb31fe (v0.6.0!) we added DEBUG-level logging to the default config of
action loggers "for easier debugging". This change to the default config lives
on to this day. It does not obviously make debugging any easier any more, but
it does result in a good deal of log noise sometimes. This commit removes this
special case from the default config.

Closes #51198
2020-01-27 10:50:10 +00:00
Lisa Cawley 931b22349f [DOCS] Adds http to elasticsearch-certutil command reference (#51188) 2020-01-24 09:59:07 -08:00
Mayya Sharipova a29deecbda Revert "Make it clear this is boost at index time (#51390)"
This reverts commit 3d5238bd95.
2020-01-24 11:05:42 -05:00
Jonas F. Henriksen 3d5238bd95 Make it clear this is boost at index time (#51390)
The way it was originally written, it sounds like 
we are boosting at query time.
 Of course, the effect is at query time, 
but the point here is that boosting is done at index time
2020-01-24 10:37:07 -05:00
Benjamin Trent 76660a5a4f
[7.x] [ML][Inference] add tags url param to GET (#51330) (#51404)
* [ML][Inference] add tags url param to GET (#51330)

Adds a new URL parameter, `tags` to the GET _ml/inference/<model_id> endpoint.

This parameter allows the list of models to be further reduced to those who contain all the provided tags.
2020-01-24 08:26:58 -05:00
István Zoltán Szabó 8bdf654cc7 [DOCS] Refines description. (#51400) 2020-01-24 13:34:25 +01:00