honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/reference/modules/indices/circuit_breaker.asciidoc

118 lines
4.3 KiB
Plaintext
Raw Normal View History

Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
[[circuit-breaker]]
=== Circuit Breaker
Elasticsearch contains multiple circuit breakers used to prevent operations from
causing an OutOfMemoryError. Each breaker specifies a limit for how much memory
it can use. Additionally, there is a parent-level breaker that specifies the
total amount of memory that can be used across all breakers.
These settings can be dynamically updated on a live cluster with the
<<cluster-update-settings,cluster-update-settings>> API.
[[parent-circuit-breaker]]
[float]
==== Parent circuit breaker
Circuit-break based on real memory usage With this commit we introduce a new circuit-breaking strategy to the parent circuit breaker. Contrary to the current implementation which only accounts for memory reserved via child circuit breakers, the new strategy measures real heap memory usage at the time of reservation. This allows us to be much more aggressive with the circuit breaker limit so we bump it to 95% by default. The new strategy is turned on by default and can be controlled with the new cluster setting `indices.breaker.total.userealmemory`. Note that we turn it off for all integration tests with an internal test cluster because it leads to spurious test failures which are of no value (we cannot fully control heap memory usage in tests). All REST tests, however, will make use of the real memory circuit breaker. Relates #31767
2018-07-13 04:08:28 -04:00
The parent-level breaker can be configured with the following settings:
`indices.breaker.total.use_real_memory`::
Whether the parent breaker should take real memory usage into account (`true`) or only
consider the amount that is reserved by child circuit breakers (`false`). Defaults to `true`.
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
`indices.breaker.total.limit`::
Circuit-break based on real memory usage With this commit we introduce a new circuit-breaking strategy to the parent circuit breaker. Contrary to the current implementation which only accounts for memory reserved via child circuit breakers, the new strategy measures real heap memory usage at the time of reservation. This allows us to be much more aggressive with the circuit breaker limit so we bump it to 95% by default. The new strategy is turned on by default and can be controlled with the new cluster setting `indices.breaker.total.userealmemory`. Note that we turn it off for all integration tests with an internal test cluster because it leads to spurious test failures which are of no value (we cannot fully control heap memory usage in tests). All REST tests, however, will make use of the real memory circuit breaker. Relates #31767
2018-07-13 04:08:28 -04:00
Starting limit for overall parent breaker, defaults to 70% of JVM heap if
`indices.breaker.total.use_real_memory` is `false`. If `indices.breaker.total.use_real_memory`
is `true`, defaults to 95% of the JVM heap.
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
[[fielddata-circuit-breaker]]
[float]
==== Field data circuit breaker
The field data circuit breaker allows Elasticsearch to estimate the amount of
memory a field will require to be loaded into memory. It can then prevent the
field data loading by raising an exception. By default the limit is configured
lower fielddata circuit breaker's default limit (#27162) * Lower fielddata circuit breaker default limit Lower fielddata circuit breaker default limit from 60% to 40% as we have moved to doc_values for most of the cases. * merge master in * update tests * update docs
2018-12-11 05:30:58 -05:00
to 40% of the maximum JVM heap. It can be configured with the following
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
parameters:
`indices.breaker.fielddata.limit`::
lower fielddata circuit breaker's default limit (#27162) * Lower fielddata circuit breaker default limit Lower fielddata circuit breaker default limit from 60% to 40% as we have moved to doc_values for most of the cases. * merge master in * update tests * update docs
2018-12-11 05:30:58 -05:00
Limit for fielddata breaker, defaults to 40% of JVM heap
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
`indices.breaker.fielddata.overhead`::
A constant that all field data estimations are multiplied with to determine a
final estimation. Defaults to 1.03
[[request-circuit-breaker]]
[float]
==== Request circuit breaker
The request circuit breaker allows Elasticsearch to prevent per-request data
structures (for example, memory used for calculating aggregations during a
request) from exceeding a certain amount of memory.
`indices.breaker.request.limit`::
Circuit break on aggregation bucket numbers with request breaker This adds new circuit breaking with the "request" breaker, which adds circuit breaks based on the number of buckets created during aggregations. It consists of incrementing during AggregatorBase creation This also bumps the REQUEST breaker to 60% of the JVM heap now. The output when circuit breaking an aggregation looks like: ```json { "shard" : 0, "index" : "i", "node" : "a5AvjUn_TKeTNYl0FyBW2g", "reason" : { "type" : "exception", "reason" : "java.util.concurrent.ExecutionException: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]];", "caused_by" : { "type" : "execution_exception", "reason" : "QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [myagg]>] would be larger than limit of [104857600/100mb]];", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]", "bytes_wanted" : 104860781, "bytes_limit" : 104857600 } } } } ``` Relates to #14046
2015-10-23 00:21:18 -04:00
Limit for request breaker, defaults to 60% of JVM heap
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
`indices.breaker.request.overhead`::
A constant that all request estimations are multiplied with to determine a
final estimation. Defaults to 1
Limit request size on transport level With this commit we limit the size of all in-flight requests on transport level. The size is guarded by a circuit breaker and is based on the content size of each request. By default we use 100% of available heap meaning that the parent circuit breaker will limit the maximum available size. This value can be changed by adjusting the setting network.breaker.inflight_requests.limit Relates #16011
2016-04-13 03:54:59 -04:00
[[in-flight-circuit-breaker]]
[float]
==== In flight requests circuit breaker
The in flight requests circuit breaker allows Elasticsearch to limit the memory usage of all
currently active incoming requests on transport or HTTP level from exceeding a certain amount of
Account for XContent overhead in in-flight breaker So far the in-flight request circuit breaker has only accounted for the on-the-wire representation of a request. However, we convert the raw request into XContent internally which increases the overhead. Therefore, we increase the value of the corresponding setting `network.breaker.inflight_requests.overhead` from one to two. While this value is still rather conservative (we assume that the representation as structured objects has no overhead compared to the byte[]), it is closer to reality than the current value. Relates #31613
2018-07-03 03:17:16 -04:00
memory on a node. The memory usage is based on the content length of the request itself. This
circuit breaker also considers that memory is not only needed for representing the raw request but
also as a structured object which is reflected by default overhead.
Limit request size on transport level With this commit we limit the size of all in-flight requests on transport level. The size is guarded by a circuit breaker and is based on the content size of each request. By default we use 100% of available heap meaning that the parent circuit breaker will limit the maximum available size. This value can be changed by adjusting the setting network.breaker.inflight_requests.limit Relates #16011
2016-04-13 03:54:59 -04:00
`network.breaker.inflight_requests.limit`::
Limit for in flight requests breaker, defaults to 100% of JVM heap. This means that it is bound
by the limit configured for the parent circuit breaker.
`network.breaker.inflight_requests.overhead`::
A constant that all in flight requests estimations are multiplied with to determine a
Account for XContent overhead in in-flight breaker So far the in-flight request circuit breaker has only accounted for the on-the-wire representation of a request. However, we convert the raw request into XContent internally which increases the overhead. Therefore, we increase the value of the corresponding setting `network.breaker.inflight_requests.overhead` from one to two. While this value is still rather conservative (we assume that the representation as structured objects has no overhead compared to the byte[]), it is closer to reality than the current value. Relates #31613
2018-07-03 03:17:16 -04:00
final estimation. Defaults to 2.
Limit request size on transport level With this commit we limit the size of all in-flight requests on transport level. The size is guarded by a circuit breaker and is based on the content size of each request. By default we use 100% of available heap meaning that the parent circuit breaker will limit the maximum available size. This value can be changed by adjusting the setting network.breaker.inflight_requests.limit Relates #16011
2016-04-13 03:54:59 -04:00
Add accounting circuit breaker and track segment memory usage (#27116) * Add accounting circuit breaker and track segment memory usage This commit adds a new circuit breaker "accounting" that is used for tracking the memory usage of non-request-tied memory users. It also adds tracking for the amount of Lucene segment memory used by a shard as a user of the new circuit breaker. The Lucene segment memory is updated when the shard refreshes, and removed when the shard relocates away from a node or is deleted. It should also be noted that all tracking for segment memory uses `addWithoutBreaking` so as not to fail the shard if a limit is reached. The `accounting` breaker has a default limit of 100% and will contribute to the parent breaker limit. Resolves #27044
2017-12-01 09:59:45 -05:00
[[accounting-circuit-breaker]]
[float]
==== Accounting requests circuit breaker
[Docs] Fix typo in circuit breaker docs (#29659) The previous description had a part that didn't fit and was probably from a copy/paste of the in flight requests description above.
2018-05-22 10:43:45 -04:00
The accounting circuit breaker allows Elasticsearch to limit the memory
Add accounting circuit breaker and track segment memory usage (#27116) * Add accounting circuit breaker and track segment memory usage This commit adds a new circuit breaker "accounting" that is used for tracking the memory usage of non-request-tied memory users. It also adds tracking for the amount of Lucene segment memory used by a shard as a user of the new circuit breaker. The Lucene segment memory is updated when the shard refreshes, and removed when the shard relocates away from a node or is deleted. It should also be noted that all tracking for segment memory uses `addWithoutBreaking` so as not to fail the shard if a limit is reached. The `accounting` breaker has a default limit of 100% and will contribute to the parent breaker limit. Resolves #27044
2017-12-01 09:59:45 -05:00
usage of things held in memory that are not released when a request is
completed. This includes things like the Lucene segment memory.
[DOCS] Fixes accounting setting names (#30863) The documentation for the account circuit breaker listed the settings for it's limit and overhead to be `network.breaker.accounting.limit` and `network.breaker.accounting.overhead` when in `HieratchyCircuitBreakerService` it seems the settings are actually `indices.breaker.accounting.limit` and `indices.breaker.accounting.overhead`.
2018-06-04 04:19:38 -04:00
`indices.breaker.accounting.limit`::
Add accounting circuit breaker and track segment memory usage (#27116) * Add accounting circuit breaker and track segment memory usage This commit adds a new circuit breaker "accounting" that is used for tracking the memory usage of non-request-tied memory users. It also adds tracking for the amount of Lucene segment memory used by a shard as a user of the new circuit breaker. The Lucene segment memory is updated when the shard refreshes, and removed when the shard relocates away from a node or is deleted. It should also be noted that all tracking for segment memory uses `addWithoutBreaking` so as not to fail the shard if a limit is reached. The `accounting` breaker has a default limit of 100% and will contribute to the parent breaker limit. Resolves #27044
2017-12-01 09:59:45 -05:00
Limit for accounting breaker, defaults to 100% of JVM heap. This means that it is bound
by the limit configured for the parent circuit breaker.
[DOCS] Fixes accounting setting names (#30863) The documentation for the account circuit breaker listed the settings for it's limit and overhead to be `network.breaker.accounting.limit` and `network.breaker.accounting.overhead` when in `HieratchyCircuitBreakerService` it seems the settings are actually `indices.breaker.accounting.limit` and `indices.breaker.accounting.overhead`.
2018-06-04 04:19:38 -04:00
`indices.breaker.accounting.overhead`::
Add accounting circuit breaker and track segment memory usage (#27116) * Add accounting circuit breaker and track segment memory usage This commit adds a new circuit breaker "accounting" that is used for tracking the memory usage of non-request-tied memory users. It also adds tracking for the amount of Lucene segment memory used by a shard as a user of the new circuit breaker. The Lucene segment memory is updated when the shard refreshes, and removed when the shard relocates away from a node or is deleted. It should also be noted that all tracking for segment memory uses `addWithoutBreaking` so as not to fail the shard if a limit is reached. The `accounting` breaker has a default limit of 100% and will contribute to the parent breaker limit. Resolves #27044
2017-12-01 09:59:45 -05:00
A constant that all accounting estimations are multiplied with to determine a
final estimation. Defaults to 1
Circuit break the number of inline scripts compiled per minute When compiling many dynamically changing scripts, parameterized scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>) should be preferred. This enforces a limit to the number of scripts that can be compiled within a minute. A new dynamic setting is added - `script.max_compilations_per_minute`, which defaults to 15. If more dynamic scripts are sent, a user will get the following exception: ```json { "error" : { "root_cause" : [ { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } ], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed_shards" : [ { "shard" : 0, "index" : "i", "node" : "a5V1eXcZRYiIk8lecjZ4Jw", "reason" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } } ], "caused_by" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } }, "status" : 500 } ``` This also fixes a bug in `ScriptService` where requests being executed concurrently on a single node could cause a script to be compiled multiple times (many in the case of a powerful node with many shards) due to no synchronization between checking the cache and compiling the script. There is now synchronization so that a script being compiled will only be compiled once regardless of the number of concurrent searches on a node. Relates to #19396
2016-07-26 15:36:29 -04:00
[[script-compilation-circuit-breaker]]
Limit request size on transport level With this commit we limit the size of all in-flight requests on transport level. The size is guarded by a circuit breaker and is based on the content size of each request. By default we use 100% of available heap meaning that the parent circuit breaker will limit the maximum available size. This value can be changed by adjusting the setting network.breaker.inflight_requests.limit Relates #16011
2016-04-13 03:54:59 -04:00
[float]
Circuit break the number of inline scripts compiled per minute When compiling many dynamically changing scripts, parameterized scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>) should be preferred. This enforces a limit to the number of scripts that can be compiled within a minute. A new dynamic setting is added - `script.max_compilations_per_minute`, which defaults to 15. If more dynamic scripts are sent, a user will get the following exception: ```json { "error" : { "root_cause" : [ { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } ], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed_shards" : [ { "shard" : 0, "index" : "i", "node" : "a5V1eXcZRYiIk8lecjZ4Jw", "reason" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } } ], "caused_by" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } }, "status" : 500 } ``` This also fixes a bug in `ScriptService` where requests being executed concurrently on a single node could cause a script to be compiled multiple times (many in the case of a powerful node with many shards) due to no synchronization between checking the cache and compiling the script. There is now synchronization so that a script being compiled will only be compiled once regardless of the number of concurrent searches on a node. Relates to #19396
2016-07-26 15:36:29 -04:00
==== Script compilation circuit breaker
Slightly different than the previous memory-based circuit breaker, the script
compilation circuit breaker limits the number of inline script compilations
within a period of time.
See the "prefer-parameters" section of the <<modules-scripting-using,scripting>>
documentation for more information.
ScriptService: Replace max compilation per minute setting with max compilation rate (#26399) The current script service has a script compilation limit for a one minute window. This is set to a small default value of 15. Instead of increasing that default value, this commit introduces a new setting that allows to configure a rate per time unit, so that the script service can deal with bursts better. The new setting is named `script.max_compilations_rate`, requires a nonnegative number and a positive time value. The default is `75/5m`, which is equivalent to the existing 15 per minute.
2017-09-01 04:15:27 -04:00
`script.max_compilations_rate`::
Circuit break the number of inline scripts compiled per minute When compiling many dynamically changing scripts, parameterized scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>) should be preferred. This enforces a limit to the number of scripts that can be compiled within a minute. A new dynamic setting is added - `script.max_compilations_per_minute`, which defaults to 15. If more dynamic scripts are sent, a user will get the following exception: ```json { "error" : { "root_cause" : [ { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } ], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed_shards" : [ { "shard" : 0, "index" : "i", "node" : "a5V1eXcZRYiIk8lecjZ4Jw", "reason" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } } ], "caused_by" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } }, "status" : 500 } ``` This also fixes a bug in `ScriptService` where requests being executed concurrently on a single node could cause a script to be compiled multiple times (many in the case of a powerful node with many shards) due to no synchronization between checking the cache and compiling the script. There is now synchronization so that a script being compiled will only be compiled once regardless of the number of concurrent searches on a node. Relates to #19396
2016-07-26 15:36:29 -04:00
ScriptService: Replace max compilation per minute setting with max compilation rate (#26399) The current script service has a script compilation limit for a one minute window. This is set to a small default value of 15. Instead of increasing that default value, this commit introduces a new setting that allows to configure a rate per time unit, so that the script service can deal with bursts better. The new setting is named `script.max_compilations_rate`, requires a nonnegative number and a positive time value. The default is `75/5m`, which is equivalent to the existing 15 per minute.
2017-09-01 04:15:27 -04:00
Limit for the number of unique dynamic scripts within a certain interval
that are allowed to be compiled. Defaults to 75/5m, meaning 75 every 5
minutes.