Commit Graph

797 Commits

Author SHA1 Message Date
Andrei Dan 6e80de0e34
DOCS: general overview of data tiers and roles (#63086) (#63422)
This adds general overview documentation for data tiers,
the data tiers specific node roles, and their application in
ILM.

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: debadair <debadair@elastic.co>
(cherry picked from commit d588cab74722bfb1d3ca0fea15d10c66af937306)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-10-07 17:31:15 +01:00
Howard a914d8bc90
[DOCS] Remove duplicate disk.threshold_enabled setting (#62925) 2020-09-29 09:13:21 -04:00
Jay Modi cb1dc5260f
Dedicated threadpool for system index writes (#62792)
This commit adds a dedicated threadpool for system index write
operations. The dedicated resources for system index writes serves as
a means to ensure that user activity does not block important system
operations from occurring such as the management of users and roles.

Backport of #61655
2020-09-22 15:31:38 -06:00
Adam Locke 56fbfabeda
[DOCS] Add remote node as a node role (#62730) (#62776)
* Adding remote node as a node role.

* Incorporating reviewer feedback.
2020-09-22 12:02:22 -04:00
James Rodewig c4d80ab7be
[DOCS] Clarify `http.max_content_length` def (#62562) (#62571) 2020-09-17 13:13:37 -04:00
Adam Locke 5dc0de04fb
[DOCS] Clarifying remote clusters based on feedback from Support (#62335) (#62394)
* Clarifying remote clusters based on feedback from Support.

* Apply suggestions from code review

* Making additional editorial changes.
2020-09-15 12:02:43 -04:00
Varun Sharma 65ec94f8a3
[DOCS] Fix node roles typo (#62307) (#62306) 2020-09-14 10:17:30 -04:00
James Rodewig 9f1f468cef
[DOCS] Document dynamic discovery settings (#61420) (#62002) 2020-09-04 11:36:34 -04:00
James Rodewig 7e2903d888
[DOCS] Document dynamic index mgmt and buffer settings (#61753) (#61996) 2020-09-04 10:40:55 -04:00
James Rodewig 129b233156
[DOCS] Document dynamic cluster settings (#61760) (#61817)
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-09-01 16:04:23 -04:00
James Rodewig 1f24fc03a0
[DOCS] Document dynamic cluster-lvl shard alloc settings (#61338) (#61735) 2020-08-31 11:19:57 -04:00
James Rodewig 580ef8eb0c
[DOCS] Document static field cache settings (#61424) (#61606) 2020-08-26 17:29:15 -04:00
debadair 82585107aa
updated shard limit doc (#56496) (#61509)
* updated shard limit doc

As the documentation was not so clear. I have updated saying this limit includes open indices with unassigned primaries and replicas count towards the limit.

* [DOCS] Incorporated edits.

Co-authored-by: Deb Adair <debadair@elastic.co>

Co-authored-by: gadekishore <50092970+gadekishore@users.noreply.github.com>
2020-08-25 14:24:47 -07:00
James Rodewig e3d23c34ab
[DOCS] Document static HTTP settings (#61429) (#61536) 2020-08-25 11:27:05 -04:00
James Rodewig 17b5a0d25e
[DOCS] Combine `Search your data` files (#61477) (#61486)
No-op changes to:

* Move `Search your data` source files into the same directory
* Rename `Search your data` source files based on page ID
* Remove unneeded includes
* Remove the `Request` dir
2020-08-24 13:08:00 -04:00
James Rodewig dc9d613280
[DOCS] Document dynamic circuit breaker settings (#61334) (#61335) 2020-08-19 11:13:46 -04:00
István Zoltán Szabó 7596bb7454
[DOCS] Clarifies node.roles settings (#61266) (#61274) 2020-08-18 17:59:34 +02:00
Adam Locke a0af82c213
[7.x] [DOCS] Update CCR docs to focus on Kibana. (#61237)
* First crack at rewriting the CCR introduction.

* Emphasizing Kibana in configuring CCR (part one).

* Many more edits, plus new files.

* Fixing test case.

* Removing overview page and consolidating that information in the main page.

* Adding redirects for moved and deleted pages.

* Removing, consolidating, and adding redirects.

* Fixing duplicate ID in redirects and removing outdated reference.

* Adding test case and steps for recreating a follower index.

* Adding steps for managing CCR tasks in Kibana.

* Adding tasks for managing auto-follow patterns.

* Fixing glossary link.

* Fixing glossary link, again.

* Updating the upgrade information and other stuff.

* Apply suggestions from code review

* Incorporating review feedback.

* Adding more edits.

* Fixing link reference.

* Adding use cases for #59812.

* Incorporating feedback from reviewers.

* Apply suggestions from code review

* Incorporating more review comments.

* Condensing some of the steps for accessing Kibana.

* Incorporating small changes from reviewers.
2020-08-17 16:58:13 -04:00
James Rodewig 60876a0e32
[DOCS] Replace Wikipedia links with attribute (#61171) (#61209) 2020-08-17 11:27:04 -04:00
István Zoltán Szabó bc9170387a
[DOCS] Adds clarification to node roles (#61206) (#61211) 2020-08-17 16:15:53 +02:00
David Turner 0549c40ac1
Minor network docs fixes (#60905)
Followup to #60216, fixing the formatting of
`transport.tcp.reuse_address` and clarifying some wording around the
distinction between the transport and HTTP layers.
2020-08-13 13:06:09 +01:00
Yannick Welsch 25404cbe3d Provide option to allow writes when master is down (#60605)
Elasticsearch currently blocks writes by default when a master is unavailable. The cluster.no_master_block setting allows
a user to change this behavior to also block reads when a master is unavailable. This PR introduces a way to now also still
allow writes when a master is offline. Writes will continue to work as long as routing table changes are not needed (as
those require the master for consistency), or if dynamic mapping updates are not required (as again, these require the
master for consistency).

Eventually we should switch the default of cluster.no_master_block to this new mode.
2020-08-12 16:56:45 +02:00
Jay Modi 2fa6448a15
System index reads in separate threadpool (#60927)
This commit introduces a new thread pool, `system_read`, which is
intended for use by system indices for all read operations (get and
search). The `system_read` pool is a fixed thread pool with a maximum
number of threads equal to lesser of half of the available processors
or 5. Given the combination of both get and read operations in this
thread pool, the queue size has been set to 2000. The motivation for
this change is to allow system read operations to be serviced in spite
of the number of user searches.

In order to avoid a significant performance hit due to pattern matching
on all search requests, a new metadata flag is added to mark indices
as system or non-system. Previously created system indices will have
flag added to their metadata upon upgrade to a version with this
capability.

Additionally, this change also introduces a new class, `SystemIndices`,
which encapsulates logic around system indices. Currently, the class
provides a method to check if an index is a system index and a method
to find a matching index descriptor given the name of an index.

Relates #50251
Relates #37867
Backport of #57936
2020-08-11 12:16:34 -06:00
David Turner f44c28b595
Deprecate and ignore join timeout (#60872)
There is no point in timing out a join attempt any more once a cluster
is entirely in 7.x. Timing out and retrying with the same master is
pointless, and an in-flight join attempt to one master no longer blocks
attempts to join other masters. This commit deprecates this unnecessary
setting and removes its effect from the joining process.

Relates #60873 which removes this setting in master.
2020-08-10 13:57:41 +01:00
James Rodewig 788778c139
[DOCS] Move inner hits content to separate page (#60840) (#60843)
Moves inner hits content from the deprecated 'Request Body Search'
chapter to a separate page.
2020-08-06 14:06:01 -04:00
James Rodewig 26d51089da
[DOCS] Replace `twitter` dataset in docs (#60604) (#60609) 2020-08-03 13:31:19 -04:00
James Rodewig 5a2c6f0d4f
[DOCS] http -> https, remove outdated plugin docs (#60380) (#60545)
Plugin discovery documentation contained information about installing
Elasticsearch 2.0 and installing an oracle JDK, both of which is no
longer valid.

While noticing that the instructions used cleartext HTTP to install
packages, this commit replaces HTTPs links instead of HTTP where possible.

In addition a few community links have been removed, as they do not seem
to exist anymore.

Co-authored-by: Alexander Reelsen <alexander@reelsen.net>
2020-07-31 16:16:31 -04:00
James Rodewig 771e9f142a
[DOCS] Move search pagination content to one page (#60515) (#60525) 2020-07-31 12:40:40 -04:00
David Turner cf0cab614d Clarify remote clusters' use of transport layer (#60268)
Today there are a few places in the transport layer docs where we talk
about communication between nodes _within a cluster_. We also use the
transport layer for remote cluster connections, and these statements
also apply there, but this is not clear from today's docs. This commit
generalises these statements to make it clear that they apply to remote
cluster connections too.

It also adds a link from the docs on configuring TCP retries to the
(deeply-buried) docs on preserving long-lived connections.
2020-07-29 13:04:10 +01:00
Julie Tibshirani c7bfb5de41
Add search `fields` parameter to support high-level field retrieval. (#60258)
This feature adds a new `fields` parameter to the search request, which
consults both the document `_source` and the mappings to fetch fields in a
consistent way. The PR merges the `field-retrieval` feature branch.

Addresses #49028 and #55363.
2020-07-28 10:58:20 -07:00
Yannick Welsch a55c869aab Properly document keepalive and other tcp options (#60216)
Keepalive options are not well-documented (only in transport section, although also available at http and network level).

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-07-28 11:10:04 +02:00
Yannick Welsch ffe114b890 Set specific keepalive options by default on supported platforms (#59278)
keepalives tell any intermediate devices that the connection remains alive, which helps with overzealous firewalls that are
killing idle connections. keepalives are enabled by default in Elasticsearch, but use system defaults for their
configuration, which often times do not have reasonable defaults (e.g. 7200s for TCP_KEEP_IDLE) in the context of
distributed systems such as Elasticsearch.

This PR sets the socket-level keep_alive options for network.tcp.{keep_idle,keep_interval} to 5 minutes on configurations
that support it (>= Java 11 & (MacOS || Linux)) and where the system defaults are set to something higher than 5
minutes. This helps keep the connections alive while not interfering with system defaults or user-specified settings
unless they are deemed to be set too high by providing better out-of-the-box defaults.
2020-07-28 11:10:04 +02:00
James Rodewig aba785cb6e
[DOCS] Update my-index examples (#60132) (#60248)
Changes the following example index names to `my-index-000001` for consistency:

* `my-index`
* `my_index`
* `myindex`
2020-07-27 15:58:26 -04:00
James Rodewig 988e8c8fc6
[DOCS] Swap `[float]` for `[discrete]` (#60134)
Changes instances of `[float]` in our docs for `[discrete]`.

Asciidoctor prefers the `[discrete]` tag for floating headings:
https://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/#blocks
2020-07-23 12:42:33 -04:00
James Rodewig 293cb8d48c
[DOCS] Fix typo in thread pools docs (#59944) (#60019)
Fix typo where available processors should be allocated processors.

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-21 17:04:36 -04:00
Tim Brooks ed315442ac
Update thread pool docs about WRITE queue size (#59643)
This commit updates the thread pool documentation to reflect the change
in the WRITE thread pool default queue size.
2020-07-21 12:38:03 -06:00
Nik Everett fe10141108
Document supported scenarios for CCS (#58120) (#59886)
Documents the supported scenarios for CCS.

Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-20 10:41:53 -04:00
debadair 7d20d32a8c
Update node.asciidoc (#59201) (#59479)
TIP block was missing due to the lack of line break prior to the "TIP"

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-13 16:51:14 -07:00
Rory Hunter 5debd09808 Dangling indices documentation (#58751)
Part of #48366. Add documentation for the dangling indices
API added in #58176.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-09 14:02:23 +01:00
Nhat Nguyen ef5c397c0f
Sending operations concurrently in peer recovery (#58018)
Today, we send operations in phase2 of peer recoveries batch by batch
sequentially. Normally that's okay as we should have a fairly small of
operations in phase 2 due to the file-based threshold. However, if
phase1 takes a lot of time and we are actively indexing, then phase2 can
have a lot of operations to replay.

With this change, we will send multiple batches concurrently (defaults
to 1) to reduce the recovery time.

Backport of #58018
2020-07-07 22:03:31 -04:00
Lisa Cawley 2e71db71b6
[DOCS] Clarifies transform node settings (#59023) (#59192) 2020-07-07 13:54:54 -07:00
David Turner 8f4f844e6e Add docs for filesystem health checks (#59134)
Documents the feature and settings introduced in #52680.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-07 14:14:58 +01:00
Adam Locke 20d04081ec
[7.x] [DOCS] Add supported ESS settings to ES docs (#57953) (#58981)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 19:40:45 -04:00
David Turner 822b7421ce Forbid read-only-allow-delete block in blocks API (#58727)
The read-only-allow-delete block is not really under the user's control
since Elasticsearch adds/removes it automatically. This commit removes
support for it from the new API for adding blocks to indices that was
introduced in #58094.
2020-07-01 13:18:26 +01:00
Yannick Welsch 15c85b29fd
Account for recovery throttling when restoring snapshot (#58658) (#58811)
Restoring from a snapshot (which is a particular form of recovery) does not currently take recovery throttling into account
(i.e. the `indices.recovery.max_bytes_per_sec` setting). While restores are subject to their own throttling (repository
setting `max_restore_bytes_per_sec`), this repository setting does not allow for values to be configured differently on a
per-node basis. As restores are very similar in nature to peer recoveries (streaming bytes to the node), it makes sense to
configure throttling in a single place.

The `max_restore_bytes_per_sec` setting is also changed to default to unlimited now, whereas previously it was set to
`40mb`, which is the current default of `indices.recovery.max_bytes_per_sec`). This means that no behavioral change
will be observed by clusters where the recovery and restore settings were not adapted.

Relates https://github.com/elastic/elasticsearch/issues/57023

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-01 12:19:29 +02:00
Jason Tedor 52ad5842a9
Introduce node.roles setting (#58512)
Today we have individual settings for configuring node roles such as
node.data and node.master. Additionally, roles are pluggable and we have
used this to introduce roles such as node.ml and node.voting_only. As
the number of roles is growing, managing these becomes harder for the
user. For example, to create a master-only node, today a user has to
configure:
 - node.data: false
 - node.ingest: false
 - node.remote_cluster_client: false
 - node.ml: false

at a minimum if they are relying on defaults, but also add:
 - node.master: true
 - node.transform: false
 - node.voting_only: false

If they want to be explicit. This is also challenging in cases where a
user wants to have configure a coordinating-only node which requires
disabling all roles, a list which we are adding to, requiring the user
to keep checking whether a node has acquired any of these roles.

This commit addresses this by adding a list setting node.roles for which
a user has explicit control over the list of roles that a node has. If
the setting is configured, the node has exactly the roles in the list,
and not any additional roles. This means to configure a master-only
node, the setting is merely 'node.roles: [master]', and to configure a
coordinating-only node, the setting is merely: 'node.roles: []'.

With this change we deprecate the existing 'node.*' settings such as
'node.data'.
2020-06-25 14:14:51 -04:00
Stuart Tettemer 20abba8433
Scripting: Deprecate general cache settings (#55753) (#58283)
Backport: ef543b0
2020-06-18 11:54:23 -06:00
Stuart Tettemer 01795d1925
Revert "Scripting: Deprecate general cache settings (#55753)" (#58201)
This reverts commit 88e8b34fc2.
2020-06-16 14:58:18 -06:00
Stuart Tettemer 88e8b34fc2
Scripting: Deprecate general cache settings (#55753)
Backport: ef543b0
2020-06-16 13:06:59 -06:00
bellengao 19042ddf3b [DOCS] Fix typo in remote-clusters doc (#57820) 2020-06-09 11:28:30 +02:00