diff --git a/docs/reference/how-to.asciidoc b/docs/reference/how-to.asciidoc index 3055d129bfc..8d11b947cc3 100644 --- a/docs/reference/how-to.asciidoc +++ b/docs/reference/how-to.asciidoc @@ -25,4 +25,4 @@ include::how-to/search-speed.asciidoc[] include::how-to/disk-usage.asciidoc[] -include::how-to/avoid-oversharding.asciidoc[] \ No newline at end of file +include::how-to/size-your-shards.asciidoc[] \ No newline at end of file diff --git a/docs/reference/how-to/avoid-oversharding.asciidoc b/docs/reference/how-to/avoid-oversharding.asciidoc deleted file mode 100644 index 7e620df196c..00000000000 --- a/docs/reference/how-to/avoid-oversharding.asciidoc +++ /dev/null @@ -1,148 +0,0 @@ -[[avoid-oversharding]] -== Avoid oversharding - -In some cases, reducing the number of shards in a cluster while maintaining the -same amount of data leads to a more effective use of system resources -(CPU, RAM, IO). In these situations, we consider the cluster _oversharded_. - -The number of shards where this inflection point occurs depends on a variety -of factors, including: - -* available hardware -* indexing load -* data volume -* the types of queries executed against the clusters -* the rate of these queries being issued -* the volume of data being queried - -Testing against production data with production queries on production hardware -is the only way to calibrate optimal shard sizes. Shard sizes of tens of GB -are commonly used, and this may be a useful starting point from which to -experiment. {kib}'s {kibana-ref}/elasticsearch-metrics.html[{es} monitoring] -provides a useful view of historical cluster performance when evaluating the -impact of different shard sizes. - -[discrete] -[[oversharding-inefficient]] -=== Why oversharding is inefficient - -Each segment has metadata that needs to be kept in heap memory. These include -lists of fields, the number of documents, and terms dictionaries. As a shard -grows in size, the size of its segments generally grow because smaller segments -are <> into fewer, larger segments. This typically -reduces the amount of heap required by a shard’s segment metadata for a given -data volume. At a bare minimum shards should be at least larger than 1GB to -make the most efficient use of memory. - -However, even though shards start to be more memory efficient at around 1GB, -a cluster full of 1GB shards will likely still perform poorly. This is because -having many small shards can also have a negative impact on search and -indexing operations. Each query or indexing operation is executed in a single -thread per shard of indices being queried or indexed to. The node receiving -a request from a client becomes responsible for distributing that request to -the appropriate shards as well as reducing the results from those individual -shards into a single response. Even assuming that a cluster has sufficient -<> available to immediately -process the requested action against all shards required by the request, the -overhead associated with making network requests to the nodes holding those -shards and with having to merge the results of results from many small shards -can lead to increased latency. This in turn can lead to exhaustion of the -threadpool and, as a result, decreased throughput. - -[discrete] -[[reduce-shard-counts-increase-shard-size]] -=== How to reduce shard counts and increase shard size - -Try these methods to reduce oversharding. - -[discrete] -[[reduce-shards-for-new-indices]] -==== Reduce the number of shards for new indices - -You can specify the `index.number_of_shards` setting for new indices created -with the <> or as part of -<> for indices automatically created by -<>. - -You can override the `index.number_of_shards` when rolling over an index -using the <>. - -[discrete] -[[create-larger-shards-by-increasing-rollover-thresholds]] -==== Create larger shards by increasing rollover thresholds - -You can roll over indices using the -<> or by specifying the -<> in an {ilm-init} policy. If using an -{ilm-init} policy, increase the rollover condition thresholds (`max_age`, - `max_docs`, `max_size`) to allow the indices to grow to a larger size - before being rolled over, which creates larger shards. - -Take special note of any empty indices. These may be managed by an {ilm-init} -policy that is rolling over the indices because the `max_age` threshold is met. -In this case, you may need to adjust the policy to make use of the `max_docs` -or `max_size` properties to prevent the creation of these empty indices. One -example where this may happen is if one or more {beats} stop sending data. If -the {ilm-init}-managed indices for those {beats} are configured to roll over - daily, then new, empty indices will be generated each day. Empty indices can - be identified using the <>. - -[discrete] -[[create-larger-shards-with-index-patterns]] -==== Create larger shards by using index patterns spanning longer time periods - -Creating indices covering longer time periods reduces index and shard counts -while increasing index sizes. For example, instead of daily indices, you can -create monthly, or even yearly indices. - -If creating indices using {ls}, the -{logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index[index] -property of the {es} output can be modified to a -<> covering a longer time period. -For example, use `logstash-%{+YYYY.MM}` instead of `logstash-%{+YYYY.MM.dd}` -to create monthly, rather than daily, indices. {beats} also lets you change the -date math expression defined in the `index` property of the {es} output, such -as for {filebeat-ref}/elasticsearch-output.html#index-option-es[Filebeat]. - -[discrete] -[[shrink-existing-index-to-fewer-shards]] -==== Shrink an existing index to fewer shards - -You can use the <> to shrink an -existing index down to fewer shards. - -<> also has a -<> available for indices in the warm phase. - -[discrete] -[[reindex-an-existing-index-to-fewer-shards]] -==== Reindex an existing index to fewer shards - -You can use the <> to reindex from an existing index -to a new index with fewer shards. After the data has been reindexed, the -oversharded index can be deleted. - -[discrete] -[[reindex-indices-from-shorter-periods-into-longer-periods]] -==== Reindex indices from shorter periods into longer periods - -You can use the <> to reindex multiple small indices -covering shorter time periods into a larger index covering a longer time period. -For example, daily indices from October with naming patterns such as -`foo-2019.10.11` could be combined into a monthly `foo-2019.10` index, -like this: - -[source,console] --------------------------------------------------- -POST /_reindex -{ - "source": { - "index": "foo-2019.10.*" - }, - "dest": { - "index": "foo-2019.10" - } -} --------------------------------------------------- - - diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc new file mode 100644 index 00000000000..68629f47013 --- /dev/null +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -0,0 +1,288 @@ +[[size-your-shards]] +== How to size your shards +++++ +Size your shards +++++ + +To protect against hardware failure and increase capacity, {es} stores copies of +an index’s data across multiple shards on multiple nodes. The number and size of +these shards can have a significant impact on your cluster's health. One common +problem is _oversharding_, a situation in which a cluster with a large number of +shards becomes unstable. + +[discrete] +[[create-a-sharding-strategy]] +=== Create a sharding strategy + +The best way to prevent oversharding and other shard-related issues +is to create a sharding strategy. A sharding strategy helps you determine and +maintain the optimal number of shards for your cluster while limiting the size +of those shards. + +Unfortunately, there is no one-size-fits-all sharding strategy. A strategy that +works in one environment may not scale in another. A good sharding strategy must +account for your infrastructure, use case, and performance expectations. + +The best way to create a sharding strategy is to benchmark your production data +on production hardware using the same queries and indexing loads you'd see in +production. For our recommended methodology, watch the +https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing[quantitative +cluster sizing video]. As you test different shard configurations, use {kib}'s +{kibana-ref}/elasticsearch-metrics.html[{es} monitoring tools] to track your +cluster's stability and performance. + +The following sections provide some reminders and guidelines you should consider +when designing your sharding strategy. If your cluster has shard-related +problems, see <>. + +[discrete] +[[shard-sizing-considerations]] +=== Sizing considerations + +Keep the following things in mind when building your sharding strategy. + +[discrete] +[[single-thread-per-shard]] +==== Searches run on a single thread per shard + +Most searches hit multiple shards. Each shard runs the search on a single +CPU thread. While a shard can run multiple concurrent searches, searches across a +large number of shards can deplete a node's <>. This can result in low throughput and slow search speeds. + +[discrete] +[[each-shard-has-overhead]] +==== Each shard has overhead + +Every shard uses memory and CPU resources. In most cases, a small +set of large shards uses fewer resources than many small shards. + +Segments play a big role in a shard's resource usage. Most shards contain +several segments, which store its index data. {es} keeps segment metadata in +<> so it can be quickly retrieved for searches. As a +shard grows, its segments are <> into fewer, larger +segments. This decreases the number of segments, which means less metadata is +kept in heap memory. + +[discrete] +[[shard-auto-balance]] +==== {es} automatically balances shards within a data tier + +A cluster's nodes are grouped into data tiers. Within each tier, {es} +attempts to spread an index's shards across as many nodes as possible. When you +add a new node or a node fails, {es} automatically rebalances the index's shards +across the tier's remaining nodes. + +[discrete] +[[shard-size-best-practices]] +=== Best practices + +Where applicable, use the following best practices as starting points for your +sharding strategy. + +[discrete] +[[delete-indices-not-documents]] +==== Delete indices, not documents + +Deleted documents aren't immediately removed from {es}'s file system. +Instead, {es} marks the document as deleted on each related shard. The marked +document will continue to use resources until it's removed during a periodic +<>. + +When possible, delete entire indices instead. {es} can immediately remove +deleted indices directly from the file system and free up resources. + +[discrete] +[[use-ds-ilm-for-time-series]] +==== Use data streams and {ilm-init} for time series data + +<> let you store time series data across multiple, +time-based backing indices. You can use <> to automatically manage these backing indices. + +[role="screenshot"] +image:images/ilm/index-lifecycle-policies.png[] + +One advantage of this setup is +<>, which creates +a new write index when the current one meets a defined `max_age`, `max_docs`, or +`max_size` threshold. You can use these thresholds to create indices based on +your retention intervals. When an index is no longer needed, you can use +{ilm-init} to automatically delete it and free up resources. + +{ilm-init} also makes it easy to change your sharding strategy over time: + +* *Want to decrease the shard count for new indices?* + +Change the <> setting in the +data stream's <>. + +* *Want larger shards?* + +Increase your {ilm-init} policy's <>. + +* *Need indices that span shorter intervals?* + +Offset the increased shard count by deleting older indices sooner. You can do +this by lowering the `min_age` threshold for your policy's +<>. + +Every new backing index is an opportunity to further tune your strategy. + +[discrete] +[[shard-size-recommendation]] +==== Aim for shard sizes between 10GB and 50GB + +Shards larger than 50GB may make a cluster less likely to recover from failure. +When a node fails, {es} rebalances the node's shards across the data tier's +remaining nodes. Shards larger than 50GB can be harder to move across a network +and may tax node resources. + +[discrete] +[[shard-count-recommendation]] +==== Aim for 20 shards or fewer per GB of heap memory + +The number of shards a node can hold is proportional to the node's +<>. For example, a node with 30GB of heap memory should +have at most 600 shards. The further below this limit you can keep your nodes, +the better. If you find your nodes exceeding more than 20 shards per GB, +consider adding another node. You can use the <> to +check the number of shards per node. + +[source,console] +---- +GET _cat/shards +---- +// TEST[setup:my_index] + +To use compressed pointers and save memory, we +recommend each node have a maximum heap size of 32GB or 50% of the node's +available memory, whichever is lower. See <>. + + +[discrete] +[[avoid-node-hotspots]] +==== Avoid node hotspots + +If too many shards are allocated to a specific node, the node can become a +hotspot. For example, if a single node contains too many shards for an index +with a high indexing volume, the node is likely to have issues. + +To prevent hotspots, use the +<> index +setting to explicitly limit the number of shards on a single node. You can +configure `index.routing.allocation.total_shards_per_node` using the +<>. + +[source,console] +-------------------------------------------------- +PUT /my-index-000001/_settings +{ + "index" : { + "routing.allocation.total_shards_per_node" : 5 + } +} +-------------------------------------------------- +// TEST[setup:my_index] + + +[discrete] +[[fix-an-oversharded-cluster]] +=== Fix an oversharded cluster + +If your cluster is experiencing stability issues due to oversharded indices, +you can use one or more of the following methods to fix them. + +[discrete] +[[reindex-indices-from-shorter-periods-into-longer-periods]] +==== Create time-based indices that cover longer periods + +For time series data, you can create indices that cover longer time intervals. +For example, instead of daily indices, you can create indices on a monthly or +yearly basis. + +If you're using {ilm-init}, you can do this by increasing the `max_age` +threshold for the <>. + +If your retention policy allows it, you can also create larger indices by +omitting a `max_age` threshold and using `max_docs` and/or `max_size` +thresholds instead. + +[discrete] +[[delete-empty-indices]] +==== Delete empty or unneeded indices + +If you're using {ilm-init} and roll over indices based on a `max_age` threshold, +you can inadvertently create indices with no documents. These empty indices +provide no benefit but still consume resources. + +You can find these empty indices using the <>. + +[source,console] +---- +GET /_cat/count/my-index-000001?v +---- +// TEST[setup:my_index] + +Once you have a list of empty indices, you can delete them using the +<>. You can also delete any other +unneeded indices. + +[source,console] +---- +DELETE /my-index-* +---- +// TEST[setup:my_index] + +[discrete] +[[force-merge-during-off-peak-hours]] +==== Force merge during off-peak hours + +If you no longer write to an index, you can use the <> to <> smaller segments into larger ones. +This can reduce shard overhead and improve search speeds. However, force merges +are resource-intensive. If possible, run the force merge during off-peak hours. + +[source,console] +---- +POST /my-index-000001/_forcemerge +---- +// TEST[setup:my_index] + +[discrete] +[[shrink-existing-index-to-fewer-shards]] +==== Shrink an existing index to fewer shards + +If you no longer write to an index, you can use the +<> to reduce its shard count. + +[source,console] +---- +POST /my-index-000001/_shrink/my-shrunken-index-000001 +---- +// TEST[s/^/PUT my-index-000001\n{"settings":{"index.number_of_shards":2,"blocks.write":true}}\n/] + +{ilm-init} also has a <> for indices in the +warm phase. + +[discrete] +[[combine-smaller-indices]] +==== Combine smaller indices + +You can also use the <> to combine indices +with similar mappings into a single large index. For time series data, you could +reindex indices for short time periods into a new index covering a +longer period. For example, you could reindex daily indices from October with a +shared index pattern, such as `my-index-2099.10.11`, into a monthly +`my-index-2099.10` index. After the reindex, delete the smaller indices. + +[source,console] +---- +POST /_reindex +{ + "source": { + "index": "my-index-2099.10.*" + }, + "dest": { + "index": "my-index-2099.10" + } +} +---- diff --git a/docs/reference/index-modules/allocation/total_shards.asciidoc b/docs/reference/index-modules/allocation/total_shards.asciidoc index 612540608d3..4a5844f80ba 100644 --- a/docs/reference/index-modules/allocation/total_shards.asciidoc +++ b/docs/reference/index-modules/allocation/total_shards.asciidoc @@ -9,6 +9,7 @@ shards evenly. The following _dynamic_ setting allows you to specify a hard limit on the total number of shards from a single index allowed per node: +[[total-shards-per-node]] `index.routing.allocation.total_shards_per_node`:: The maximum number of shards (replicas and primaries) that will be diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 1e25d9bf0b3..d5d3beec3b7 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -1173,3 +1173,8 @@ For other searchable snapshot APIs, see <>. === Point in time API See <>. + +[role="exclude",id="avoid-oversharding"] +=== Avoid oversharding + +See <>.