2015-06-22 17:49:45 -04:00
|
|
|
[[circuit-breaker]]
|
|
|
|
=== Circuit Breaker
|
|
|
|
|
|
|
|
Elasticsearch contains multiple circuit breakers used to prevent operations from
|
|
|
|
causing an OutOfMemoryError. Each breaker specifies a limit for how much memory
|
|
|
|
it can use. Additionally, there is a parent-level breaker that specifies the
|
|
|
|
total amount of memory that can be used across all breakers.
|
|
|
|
|
|
|
|
These settings can be dynamically updated on a live cluster with the
|
|
|
|
<<cluster-update-settings,cluster-update-settings>> API.
|
|
|
|
|
|
|
|
[[parent-circuit-breaker]]
|
|
|
|
[float]
|
|
|
|
==== Parent circuit breaker
|
|
|
|
|
|
|
|
The parent-level breaker can be configured with the following setting:
|
|
|
|
|
|
|
|
`indices.breaker.total.limit`::
|
|
|
|
|
|
|
|
Starting limit for overall parent breaker, defaults to 70% of JVM heap.
|
|
|
|
|
|
|
|
[[fielddata-circuit-breaker]]
|
|
|
|
[float]
|
|
|
|
==== Field data circuit breaker
|
|
|
|
The field data circuit breaker allows Elasticsearch to estimate the amount of
|
|
|
|
memory a field will require to be loaded into memory. It can then prevent the
|
|
|
|
field data loading by raising an exception. By default the limit is configured
|
|
|
|
to 60% of the maximum JVM heap. It can be configured with the following
|
|
|
|
parameters:
|
|
|
|
|
|
|
|
`indices.breaker.fielddata.limit`::
|
|
|
|
|
|
|
|
Limit for fielddata breaker, defaults to 60% of JVM heap
|
|
|
|
|
|
|
|
`indices.breaker.fielddata.overhead`::
|
|
|
|
|
|
|
|
A constant that all field data estimations are multiplied with to determine a
|
|
|
|
final estimation. Defaults to 1.03
|
|
|
|
|
|
|
|
[[request-circuit-breaker]]
|
|
|
|
[float]
|
|
|
|
==== Request circuit breaker
|
|
|
|
|
|
|
|
The request circuit breaker allows Elasticsearch to prevent per-request data
|
|
|
|
structures (for example, memory used for calculating aggregations during a
|
|
|
|
request) from exceeding a certain amount of memory.
|
|
|
|
|
|
|
|
`indices.breaker.request.limit`::
|
|
|
|
|
Circuit break on aggregation bucket numbers with request breaker
This adds new circuit breaking with the "request" breaker, which adds
circuit breaks based on the number of buckets created during
aggregations. It consists of incrementing during AggregatorBase creation
This also bumps the REQUEST breaker to 60% of the JVM heap now.
The output when circuit breaking an aggregation looks like:
```json
{
"shard" : 0,
"index" : "i",
"node" : "a5AvjUn_TKeTNYl0FyBW2g",
"reason" : {
"type" : "exception",
"reason" : "java.util.concurrent.ExecutionException: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]];",
"caused_by" : {
"type" : "execution_exception",
"reason" : "QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [myagg]>] would be larger than limit of [104857600/100mb]];",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]",
"bytes_wanted" : 104860781,
"bytes_limit" : 104857600
}
}
}
}
```
Relates to #14046
2015-10-23 00:21:18 -04:00
|
|
|
Limit for request breaker, defaults to 60% of JVM heap
|
2015-06-22 17:49:45 -04:00
|
|
|
|
|
|
|
`indices.breaker.request.overhead`::
|
|
|
|
|
|
|
|
A constant that all request estimations are multiplied with to determine a
|
|
|
|
final estimation. Defaults to 1
|
|
|
|
|
2016-04-13 03:54:59 -04:00
|
|
|
[[in-flight-circuit-breaker]]
|
|
|
|
[float]
|
|
|
|
==== In flight requests circuit breaker
|
|
|
|
|
|
|
|
The in flight requests circuit breaker allows Elasticsearch to limit the memory usage of all
|
|
|
|
currently active incoming requests on transport or HTTP level from exceeding a certain amount of
|
|
|
|
memory on a node. The memory usage is based on the content length of the request itself.
|
|
|
|
|
|
|
|
`network.breaker.inflight_requests.limit`::
|
|
|
|
|
|
|
|
Limit for in flight requests breaker, defaults to 100% of JVM heap. This means that it is bound
|
|
|
|
by the limit configured for the parent circuit breaker.
|
|
|
|
|
|
|
|
`network.breaker.inflight_requests.overhead`::
|
|
|
|
|
|
|
|
A constant that all in flight requests estimations are multiplied with to determine a
|
|
|
|
final estimation. Defaults to 1
|
|
|
|
|
2017-12-01 09:59:45 -05:00
|
|
|
[[accounting-circuit-breaker]]
|
|
|
|
[float]
|
|
|
|
==== Accounting requests circuit breaker
|
|
|
|
|
2018-05-22 10:43:45 -04:00
|
|
|
The accounting circuit breaker allows Elasticsearch to limit the memory
|
2017-12-01 09:59:45 -05:00
|
|
|
usage of things held in memory that are not released when a request is
|
|
|
|
completed. This includes things like the Lucene segment memory.
|
|
|
|
|
|
|
|
`network.breaker.accounting.limit`::
|
|
|
|
|
|
|
|
Limit for accounting breaker, defaults to 100% of JVM heap. This means that it is bound
|
|
|
|
by the limit configured for the parent circuit breaker.
|
|
|
|
|
|
|
|
`network.breaker.accounting.overhead`::
|
|
|
|
|
|
|
|
A constant that all accounting estimations are multiplied with to determine a
|
|
|
|
final estimation. Defaults to 1
|
|
|
|
|
Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.
If more dynamic scripts are sent, a user will get the following
exception:
```json
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "i",
"node" : "a5V1eXcZRYiIk8lecjZ4Jw",
"reason" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
}
],
"caused_by" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
},
"status" : 500
}
```
This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.
Relates to #19396
2016-07-26 15:36:29 -04:00
|
|
|
[[script-compilation-circuit-breaker]]
|
2016-04-13 03:54:59 -04:00
|
|
|
[float]
|
Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.
If more dynamic scripts are sent, a user will get the following
exception:
```json
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "i",
"node" : "a5V1eXcZRYiIk8lecjZ4Jw",
"reason" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
}
],
"caused_by" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
},
"status" : 500
}
```
This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.
Relates to #19396
2016-07-26 15:36:29 -04:00
|
|
|
==== Script compilation circuit breaker
|
|
|
|
|
|
|
|
Slightly different than the previous memory-based circuit breaker, the script
|
|
|
|
compilation circuit breaker limits the number of inline script compilations
|
|
|
|
within a period of time.
|
|
|
|
|
|
|
|
See the "prefer-parameters" section of the <<modules-scripting-using,scripting>>
|
|
|
|
documentation for more information.
|
|
|
|
|
2017-09-01 04:15:27 -04:00
|
|
|
`script.max_compilations_rate`::
|
Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.
If more dynamic scripts are sent, a user will get the following
exception:
```json
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "i",
"node" : "a5V1eXcZRYiIk8lecjZ4Jw",
"reason" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
}
],
"caused_by" : {
"type" : "general_script_exception",
"reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
"caused_by" : {
"type" : "circuit_breaking_exception",
"reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
"bytes_wanted" : 0,
"bytes_limit" : 0
}
}
},
"status" : 500
}
```
This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.
Relates to #19396
2016-07-26 15:36:29 -04:00
|
|
|
|
2017-09-01 04:15:27 -04:00
|
|
|
Limit for the number of unique dynamic scripts within a certain interval
|
|
|
|
that are allowed to be compiled. Defaults to 75/5m, meaning 75 every 5
|
|
|
|
minutes.
|