Commit Graph

31 Commits

Author SHA1 Message Date
Aron Szanto 316cb42b21 Update shards_allocation.asciidoc ()
Slight language and consistency updates in shard balancing heuristics
2017-08-03 11:27:02 +02:00
Clinton Gormley 3e568f52c1 Fixed asciidoc formatting 2017-07-27 15:55:52 +02:00
Jason Tedor e165c405ac Add an underscore to flood stage setting
This is a minor nitty bikeshedding change that renames the suffix of the
disk flood stage setting to "flood_stage" from "floodstage".

Relates 
2017-07-11 22:02:00 -04:00
Herman Schaaf 977712f977 Change small typo in shards_allocation.asciidoc () 2017-07-11 11:25:49 +02:00
Jason Tedor 8148e25087 Fix disk allocator docs
This commit fixes the disk allocator docs which were broken due to the
inadvertent removal of some docs snippet markup.
2017-07-07 22:11:09 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates 
2017-07-07 19:54:36 -04:00
Clinton Gormley ca12b1f2a6 Tidied up the disk allocator docs 2017-07-06 12:16:53 +02:00
Simon Willnauer 6e5cc424a8 Switch indices read-only if a node runs out of disk space ()
Today when we run out of disk all kinds of crazy things can happen
and nodes are becoming hard to maintain once out of disk is hit.
While we try to move shards away if we hit watermarks this might not
be possible in many situations. Based on the discussion in 
this change monitors disk utilization and adds a flood-stage watermark
that causes all indices that are allocated on a node hitting the flood-stage
mark to be switched read-only (with the option to be deleted). This allows users to react on the low disk
situation while subsequent write requests will be rejected. Users can switch
individual indices read-write once the situation is sorted out. There is no
automatic read-write switch once the node has enough space. This requires
user interaction.

The flood-stage watermark is set to `95%` utilization by default.

Closes 
2017-07-05 22:18:23 +02:00
Simon Willnauer 1cae850cf5 Add a cluster block that allows to delete indices that are read-only ()
Today when an index is `read-only` the index is also blocked from
being deleted which sometimes is undesired since in-order to make
changes to a cluster indices must be deleted to free up space. This is
a likely scenario in a hosted environment when disk-space is limited to switch
indices read-only but allow deletions to free up space.
2017-05-16 17:34:37 +02:00
Boaz Leskes 70a3ac1767 Add a note about `cluster.routing.allocation.node_concurrent_recoveries` ()
Closes 
2017-02-14 14:14:41 +02:00
Ali Beyad 51bfecc7cb [DOCS] fixes word usage in allocation awareness docs 2016-11-25 11:47:40 -05:00
Joshua Rich 63f484ffa3 Docs: Cluster Allocation Filtering
Put more emphasis on the fact that multiple values can be specified
and move examples after explanation of settings.
2016-10-17 11:02:33 +11:00
Nik Everett 3ed3e5e660 Convert more docs to CONSOLE
* plugins/discovery-azure-class.asciidoc
* reference/cluster.asciidoc
* reference/modules/cluster/misc.asciidoc
* reference/modules/indices/request_cache.asciidoc

After this is merged there will be no unconvereted snippets outside
of `reference`.

Related to 
2016-09-21 09:36:21 -04:00
Ali Beyad f608e6c6cf Improves the documentation for the ()
`cluster.routing.allocation.cluster_concurrent_rebalance` setting,
clarifying in which shard allocation situations the rebalance limit
takes effect.

Closes 
2016-09-16 16:06:18 -04:00
Jason Tedor 412c61c402 Update logger names in docs
In 7560101ec7, the Elasticsearch logger
names were modified to be their fully-qualified class name (with some
exceptions for special loggers like the slow logs and the transport
tracer). This commit updates the docs accordingly.

Relates 
2016-09-14 08:08:49 -04:00
Brandon Wulf 6b7d40929c Switch example from inclusion to exclusion.
Page is explaining allocation exclusion- example should be about exclusion as well.
2016-07-28 21:54:22 -04:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates 
2016-05-19 14:08:08 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE () 2016-05-09 15:42:23 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes . That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Ali Beyad 67c0734bf3 Update misc.asciidoc
Added documentation for the cluster.indices.tombstones.size property for maximum tombstones in the cluster state.
2016-05-04 15:21:47 -04:00
Simon Willnauer ad24653948 update allocation_awareness.asciidoc to also use 'node.attr' namespace 2016-03-30 13:52:45 +02:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Clinton Gormley d5f8f92559 Update shards_allocation.asciidoc
Closes https://github.com/elastic/elasticsearch/issues/16554
2016-02-13 15:16:42 +01:00
Dongjoon Hyun 21ea552070 Fix typos in docs. 2016-02-09 02:07:32 -08:00
Simon Willnauer f5e4cd4616 Remove recovery threadpools and throttle outgoing recoveries on the master
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
of primaries can get overloaded due to too many recoveries. To still keep that at bay
we limit the number of threads that are sending files to the target to overcome this problem.

The right solution here is to also throttle the outgoing recoveries that are today unbounded on
the master and don't start the recovery until we have enough resources on both source and target nodes.

The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools
that are hard to configure. This commit removes the concurrent streamns notion completely and sends files
in the thread that drives the recovery simplifying the recovery code considerably.
Outgoing recoveries are not throttled on the master via a allocation decider.
2015-12-22 14:59:43 +01:00
Yannick Welsch 3a442db9bd Allocate primary shards based on allocation ids
Closes 
2015-12-17 15:55:50 +01:00
Masaru Hasegawa 5ae00a6129 Take initializing shards into consideration during awareness allocation
It makes decision consistent.
Fixes 
2015-09-11 13:13:36 +09:00
Simon Willnauer 66b78341e4 Add note about multi data path and disk threshold deciders
Prior to 2.0 we summed up the available space on all disk on a node
due to the raid-0 like behavior. Now we don't do this anymore and use the
min & max disk space to make decisions.

Closes 
2015-08-31 16:23:54 +02:00
Clinton Gormley 2b512f1f29 Docs: Use "js" instead of "json" and "sh" instead of "shell" for source highlighting 2015-07-14 18:14:09 +02:00
Boaz Leskes 41f8c96fed Docs: clarification of allocation awareness w.r.t. rack failures
Closes 
2015-06-29 11:57:32 +02:00
Clinton Gormley f123a53d72 Docs: Refactored modules and index modules sections 2015-06-22 23:49:45 +02:00