Support requests_per_second=-1 to mean no throttling in reindex

This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes #19089
This commit is contained in:
Nik Everett 2016-06-27 12:28:51 -04:00
parent 7861548786
commit d573541f66
16 changed files with 115 additions and 82 deletions

View File

@ -40,7 +40,7 @@ That will return something like this:
"search": 0 "search": 0
}, },
"throttled_millis": 0, "throttled_millis": 0,
"requests_per_second": "unlimited", "requests_per_second": -1.0,
"throttled_until_millis": 0, "throttled_until_millis": 0,
"total": 119, "total": 119,
"failures" : [ ] "failures" : [ ]
@ -164,15 +164,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>. <<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc) `requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
and throttles the number of requests per second that the delete by query issues. `1000`, etc) and throttles the number of requests per second that the delete-by-query
The throttling is done waiting between bulk batches so that it can manipulate issues or it can be set to `-1` to disabled throttling. The throttling is done
the scroll timeout. The wait time is the difference between the time it took the waiting between bulk batches so that it can manipulate the scroll timeout. The
batch to complete and the time `requests_per_second * requests_in_the_batch`. wait time is the difference between the time it took the batch to complete and
Since the batch isn't broken into multiple bulk requests large batch sizes will the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
cause Elasticsearch to create many requests and then wait for a while before broken into multiple bulk requests large batch sizes will cause Elasticsearch
starting the next set. This is "bursty" instead of "smooth". The default is to create many requests and then wait for a while before starting the next set.
`unlimited` which is also the only non-number value that it accepts. This is "bursty" instead of "smooth". The default is `-1`.
[float] [float]
=== Response body === Response body
@ -326,14 +326,15 @@ using the `_rethrottle` API:
[source,js] [source,js]
-------------------------------------------------- --------------------------------------------------
POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=unlimited POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=-1
-------------------------------------------------- --------------------------------------------------
// CONSOLE // CONSOLE
The `task_id` can be found using the tasks API above. The `task_id` can be found using the tasks API above.
Just like when setting it on the `_delete_by_query` API `requests_per_second` Just like when setting it on the `_delete_by_query` API `requests_per_second`
can be either `unlimited` to disable throttling or any decimal number like `1.7` can be either `-1` to disable throttling or any decimal number
or `12` to throttle to that level. Rethrottling that speeds up the query takes like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
effect immediately but rethrotting that slows down the query will take effect query takes effect immediately but rethrotting that slows down the query will
on after completing the current batch. This prevents scroll timeouts. take effect on after completing the current batch. This prevents scroll
timeouts.

View File

@ -39,7 +39,7 @@ That will return something like this:
"search": 0 "search": 0
}, },
"throttled_millis": 0, "throttled_millis": 0,
"requests_per_second": "unlimited", "requests_per_second": -1.0,
"throttled_until_millis": 0, "throttled_until_millis": 0,
"total": 120, "total": 120,
"failures" : [ ] "failures" : [ ]
@ -439,15 +439,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>. <<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc) `requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
and throttles the number of requests per second that the reindex issues. The `1000`, etc) and throttles the number of requests per second that the reindex
throttling is done waiting between bulk batches so that it can manipulate the issues or it can be set to `-1` to disabled throttling. The throttling is done
scroll timeout. The wait time is the difference between the time it took the waiting between bulk batches so that it can manipulate the scroll timeout. The
batch to complete and the time `requests_per_second * requests_in_the_batch`. wait time is the difference between the time it took the batch to complete and
Since the batch isn't broken into multiple bulk requests large batch sizes will the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
cause Elasticsearch to create many requests and then wait for a while before broken into multiple bulk requests large batch sizes will cause Elasticsearch
starting the next set. This is "bursty" instead of "smooth". The default is to create many requests and then wait for a while before starting the next set.
`unlimited` which is also the only non-number value that it accepts. This is "bursty" instead of "smooth". The default is `-1`.
[float] [float]
[[docs-reindex-response-body]] [[docs-reindex-response-body]]
@ -613,18 +613,18 @@ the `_rethrottle` API:
[source,js] [source,js]
-------------------------------------------------- --------------------------------------------------
POST _reindex/taskid:1/_rethrottle?requests_per_second=unlimited POST _reindex/taskid:1/_rethrottle?requests_per_second=-1
-------------------------------------------------- --------------------------------------------------
// CONSOLE // CONSOLE
The `task_id` can be found using the tasks API above. The `task_id` can be found using the tasks API above.
Just like when setting it on the `_reindex` API `requests_per_second` can be Just like when setting it on the `_reindex` API `requests_per_second`
either `unlimited` to disable throttling or any decimal number like `1.7` or can be either `-1` to disable throttling or any decimal number
`12` to throttle to that level. Rethrottling that speeds up the query takes like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
effect immediately but rethrotting that slows down the query will take effect query takes effect immediately but rethrotting that slows down the query will
on after completing the current batch. This prevents scroll timeouts. take effect on after completing the current batch. This prevents scroll
timeouts.
[float] [float]
=== Reindex to change the name of a field === Reindex to change the name of a field

View File

@ -32,7 +32,7 @@ That will return something like this:
"search": 0 "search": 0
}, },
"throttled_millis": 0, "throttled_millis": 0,
"requests_per_second": "unlimited", "requests_per_second": -1.0,
"throttled_until_millis": 0, "throttled_until_millis": 0,
"total": 120, "total": 120,
"failures" : [ ] "failures" : [ ]
@ -221,15 +221,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>. <<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc) `requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
and throttles the number of requests per second that the update by query issues. `1000`, etc) and throttles the number of requests per second that the update-by-query
The throttling is done waiting between bulk batches so that it can manipulate issues or it can be set to `-1` to disabled throttling. The throttling is done
the scroll timeout. The wait time is the difference between the time it took the waiting between bulk batches so that it can manipulate the scroll timeout. The
batch to complete and the time `requests_per_second * requests_in_the_batch`. wait time is the difference between the time it took the batch to complete and
Since the batch isn't broken into multiple bulk requests large batch sizes will the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
cause Elasticsearch to create many requests and then wait for a while before broken into multiple bulk requests large batch sizes will cause Elasticsearch
starting the next set. This is "bursty" instead of "smooth". The default is to create many requests and then wait for a while before starting the next set.
`unlimited` which is also the only non-number value that it accepts. This is "bursty" instead of "smooth". The default is `-1`.
[float] [float]
[[docs-update-by-query-response-body]] [[docs-update-by-query-response-body]]
@ -391,18 +391,18 @@ using the `_rethrottle` API:
[source,js] [source,js]
-------------------------------------------------- --------------------------------------------------
POST _update_by_query/taskid:1/_rethrottle?requests_per_second=unlimited POST _update_by_query/taskid:1/_rethrottle?requests_per_second=-1
-------------------------------------------------- --------------------------------------------------
// CONSOLE // CONSOLE
The `task_id` can be found using the tasks API above. The `task_id` can be found using the tasks API above.
Just like when setting it on the `_update_by_query` API `requests_per_second` Just like when setting it on the `_update_by_query` API `requests_per_second`
can be either `unlimited` to disable throttling or any decimal number like `1.7` can be either `-1` to disable throttling or any decimal number
or `12` to throttle to that level. Rethrottling that speeds up the query takes like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
effect immediately but rethrotting that slows down the query will take effect query takes effect immediately but rethrotting that slows down the query will
on after completing the current batch. This prevents scroll timeouts. take effect on after completing the current batch. This prevents scroll
timeouts.
[float] [float]
[[picking-up-a-new-property]] [[picking-up-a-new-property]]

View File

@ -137,20 +137,20 @@ public abstract class AbstractBaseReindexRestHandler<
if (requestsPerSecondString == null) { if (requestsPerSecondString == null) {
return null; return null;
} }
if ("unlimited".equals(requestsPerSecondString)) {
return Float.POSITIVE_INFINITY;
}
float requestsPerSecond; float requestsPerSecond;
try { try {
requestsPerSecond = Float.parseFloat(requestsPerSecondString); requestsPerSecond = Float.parseFloat(requestsPerSecondString);
} catch (NumberFormatException e) { } catch (NumberFormatException e) {
throw new IllegalArgumentException( throw new IllegalArgumentException(
"[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling.", e); "[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.", e);
}
if (requestsPerSecond == -1) {
return Float.POSITIVE_INFINITY;
} }
if (requestsPerSecond <= 0) { if (requestsPerSecond <= 0) {
// We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of "unlimited" // We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of -1
throw new IllegalArgumentException( throw new IllegalArgumentException(
"[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling."); "[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.");
} }
return requestsPerSecond; return requestsPerSecond;
} }

View File

@ -217,7 +217,7 @@ public class BulkByScrollTask extends CancellableTask {
} }
builder.endObject(); builder.endObject();
builder.timeValueField("throttled_millis", "throttled", throttled); builder.timeValueField("throttled_millis", "throttled", throttled);
builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? "unlimited" : requestsPerSecond); builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? -1 : requestsPerSecond);
if (reasonCancelled != null) { if (reasonCancelled != null) {
builder.field("canceled", reasonCancelled); builder.field("canceled", reasonCancelled);
} }
@ -324,7 +324,7 @@ public class BulkByScrollTask extends CancellableTask {
} }
/** /**
* The number of requests per second to which to throttle the request. 0 means unlimited. * The number of requests per second to which to throttle the request. Float.POSITIVE_INFINITY means unlimited.
*/ */
public float getRequestsPerSecond() { public float getRequestsPerSecond() {
return requestsPerSecond; return requestsPerSecond;

View File

@ -285,7 +285,7 @@ public class BulkByScrollTaskTests extends ESTestCase {
public void testXContentRepresentationOfUnlimitedRequestsPerSecon() throws IOException { public void testXContentRepresentationOfUnlimitedRequestsPerSecon() throws IOException {
XContentBuilder builder = JsonXContent.contentBuilder(); XContentBuilder builder = JsonXContent.contentBuilder();
task.getStatus().toXContent(builder, ToXContent.EMPTY_PARAMS); task.getStatus().toXContent(builder, ToXContent.EMPTY_PARAMS);
assertThat(builder.string(), containsString("\"requests_per_second\":\"unlimited\"")); assertThat(builder.string(), containsString("\"requests_per_second\":-1"));
} }
public void testPerfectlyThrottledBatchTime() { public void testPerfectlyThrottledBatchTime() {

View File

@ -68,7 +68,7 @@
--- ---
"requests_per_second cannot be an empty string": "requests_per_second cannot be an empty string":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query: delete_by_query:
requests_per_second: "" requests_per_second: ""
index: test index: test
@ -79,7 +79,7 @@
--- ---
"requests_per_second cannot be negative": "requests_per_second cannot be negative":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query: delete_by_query:
requests_per_second: -12 requests_per_second: -12
index: test index: test
@ -90,10 +90,22 @@
--- ---
"requests_per_second cannot be zero": "requests_per_second cannot be zero":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query: delete_by_query:
requests_per_second: 0 requests_per_second: 0
index: test index: test
body: body:
query: query:
match_all: {} match_all: {}
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query:
requests_per_second: unlimited
index: test
body:
query:
match_all: {}

View File

@ -40,7 +40,7 @@
- lt: {throttled_millis: 4000} - lt: {throttled_millis: 4000}
--- ---
"requests_per_second supports unlimited which turns off throttling": "requests_per_second supports -1 which turns off throttling":
- do: - do:
indices.create: indices.create:
index: test index: test
@ -69,7 +69,7 @@
delete_by_query: delete_by_query:
index: test index: test
scroll_size: 1 scroll_size: 1
requests_per_second: unlimited requests_per_second: -1
body: body:
query: query:
match_all: {} match_all: {}
@ -79,7 +79,7 @@
- match: {throttled_millis: 0} - match: {throttled_millis: 0}
--- ---
"Rethrottle": "Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request # and a small batch size on the request
- do: - do:
@ -121,7 +121,7 @@
- do: - do:
reindex.rethrottle: reindex.rethrottle:
requests_per_second: unlimited requests_per_second: -1
task_id: $task task_id: $task
- do: - do:

View File

@ -152,7 +152,7 @@
--- ---
"requests_per_second cannot be an empty string": "requests_per_second cannot be an empty string":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex: reindex:
requests_per_second: "" requests_per_second: ""
body: body:
@ -164,7 +164,7 @@
--- ---
"requests_per_second cannot be negative": "requests_per_second cannot be negative":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex: reindex:
requests_per_second: -12 requests_per_second: -12
body: body:
@ -176,7 +176,7 @@
--- ---
"requests_per_second cannot be zero": "requests_per_second cannot be zero":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex: reindex:
requests_per_second: 0 requests_per_second: 0
body: body:
@ -185,6 +185,18 @@
dest: dest:
index: dest index: dest
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex:
requests_per_second: unlimited
body:
source:
index: test
dest:
index: dest
--- ---
"reindex without source gives useful error message": "reindex without source gives useful error message":
- do: - do:

View File

@ -50,7 +50,7 @@
- is_false: task - is_false: task
--- ---
"requests_per_second supports unlimited to turn off throttling": "requests_per_second supports -1 to turn off throttling":
- do: - do:
indices.create: indices.create:
index: source index: source
@ -81,7 +81,7 @@
- do: - do:
reindex: reindex:
requests_per_second: unlimited requests_per_second: -1
body: body:
source: source:
index: source index: source
@ -97,7 +97,7 @@
- is_false: task - is_false: task
--- ---
"Rethrottle": "Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request # and a small batch size on the request
- do: - do:
@ -143,7 +143,7 @@
- do: - do:
reindex.rethrottle: reindex.rethrottle:
requests_per_second: unlimited requests_per_second: -1
task_id: $task task_id: $task
- do: - do:

View File

@ -76,7 +76,7 @@
--- ---
"requests_per_second cannot be an empty string": "requests_per_second cannot be an empty string":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query: update_by_query:
requests_per_second: "" requests_per_second: ""
index: test index: test
@ -84,7 +84,7 @@
--- ---
"requests_per_second cannot be negative": "requests_per_second cannot be negative":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query: update_by_query:
requests_per_second: -12 requests_per_second: -12
index: test index: test
@ -92,7 +92,15 @@
--- ---
"requests_per_second cannot be zero": "requests_per_second cannot be zero":
- do: - do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./ catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query:
requests_per_second: 0
index: test
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query: update_by_query:
requests_per_second: 0 requests_per_second: 0
index: test index: test

View File

@ -36,7 +36,7 @@
- lt: {throttled_millis: 4000} - lt: {throttled_millis: 4000}
--- ---
"requests_per_second supports unlimited which turns off throttling": "requests_per_second supports -1 which turns off throttling":
- do: - do:
indices.create: indices.create:
index: test index: test
@ -65,13 +65,13 @@
update_by_query: update_by_query:
index: test index: test
scroll_size: 1 scroll_size: 1
requests_per_second: unlimited requests_per_second: -1
- match: {batches: 3} - match: {batches: 3}
- match: {updated: 3} - match: {updated: 3}
- match: {throttled_millis: 0} - match: {throttled_millis: 0}
--- ---
"Rethrottle": "Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request # and a small batch size on the request
- do: - do:
@ -109,7 +109,7 @@
- do: - do:
reindex.rethrottle: reindex.rethrottle:
requests_per_second: unlimited requests_per_second: -1
task_id: $task task_id: $task
- do: - do:

View File

@ -195,7 +195,7 @@
"requests_per_second": { "requests_per_second": {
"type": "float", "type": "float",
"default": 0, "default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle." "description": "The throttle for this request in sub-requests per second. -1 means set no throttle."
} }
} }
}, },

View File

@ -29,7 +29,7 @@
"requests_per_second": { "requests_per_second": {
"type": "float", "type": "float",
"default": 0, "default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle." "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
} }
} }
}, },

View File

@ -15,7 +15,7 @@
"requests_per_second": { "requests_per_second": {
"type": "float", "type": "float",
"required": true, "required": true,
"description": "The throttle to set on this request in sub-requests per second. 0 means set no throttle. As does \"unlimited\". Otherwise it must be a float." "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
} }
} }
}, },

View File

@ -203,7 +203,7 @@
"requests_per_second": { "requests_per_second": {
"type": "float", "type": "float",
"default": 0, "default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle." "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
} }
} }
}, },