honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Support requests_per_second=-1 to mean no throttling in reindex 

This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes #19089
This commit is contained in:
Nik Everett 2016-06-27 12:28:51 -04:00
parent 7861548786
commit d573541f66
16 changed files with 115 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
docs/reference/docs/delete-by-query.asciidoc
View File

@ -40,7 +40,7 @@ That will return something like this:
"search": 0
},
"throttled_millis": 0,
"requests_per_second": "unlimited",
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"total": 119,
"failures" : [ ]
@ -164,15 +164,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
and throttles the number of requests per second that the delete by query issues.
The throttling is done waiting between bulk batches so that it can manipulate
the scroll timeout. The wait time is the difference between the time it took the
batch to complete and the time `requests_per_second * requests_in_the_batch`.
Since the batch isn't broken into multiple bulk requests large batch sizes will
cause Elasticsearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth". The default is
`unlimited` which is also the only non-number value that it accepts.
`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
`1000`, etc) and throttles the number of requests per second that the delete-by-query
issues or it can be set to `-1` to disabled throttling. The throttling is done
waiting between bulk batches so that it can manipulate the scroll timeout. The
wait time is the difference between the time it took the batch to complete and
the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
broken into multiple bulk requests large batch sizes will cause Elasticsearch
to create many requests and then wait for a while before starting the next set.
This is "bursty" instead of "smooth". The default is `-1`.
[float]
=== Response body
@ -326,14 +326,15 @@ using the `_rethrottle` API:
[source,js]
--------------------------------------------------
POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=unlimited
POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=-1
--------------------------------------------------
// CONSOLE
The `task_id` can be found using the tasks API above.
Just like when setting it on the `_delete_by_query` API `requests_per_second`
can be either `unlimited` to disable throttling or any decimal number like `1.7`
or `12` to throttle to that level. Rethrottling that speeds up the query takes
effect immediately but rethrotting that slows down the query will take effect
on after completing the current batch. This prevents scroll timeouts.
can be either `-1` to disable throttling or any decimal number
like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
query takes effect immediately but rethrotting that slows down the query will
take effect on after completing the current batch. This prevents scroll
timeouts.

34
docs/reference/docs/reindex.asciidoc
View File

@ -39,7 +39,7 @@ That will return something like this:
"search": 0
},
"throttled_millis": 0,
"requests_per_second": "unlimited",
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"total": 120,
"failures" : [ ]
@ -439,15 +439,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
and throttles the number of requests per second that the reindex issues. The
throttling is done waiting between bulk batches so that it can manipulate the
scroll timeout. The wait time is the difference between the time it took the
batch to complete and the time `requests_per_second * requests_in_the_batch`.
Since the batch isn't broken into multiple bulk requests large batch sizes will
cause Elasticsearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth". The default is
`unlimited` which is also the only non-number value that it accepts.
`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
`1000`, etc) and throttles the number of requests per second that the reindex
issues or it can be set to `-1` to disabled throttling. The throttling is done
waiting between bulk batches so that it can manipulate the scroll timeout. The
wait time is the difference between the time it took the batch to complete and
the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
broken into multiple bulk requests large batch sizes will cause Elasticsearch
to create many requests and then wait for a while before starting the next set.
This is "bursty" instead of "smooth". The default is `-1`.
[float]
[[docs-reindex-response-body]]
@ -613,18 +613,18 @@ the `_rethrottle` API:
[source,js]
--------------------------------------------------
POST _reindex/taskid:1/_rethrottle?requests_per_second=unlimited
POST _reindex/taskid:1/_rethrottle?requests_per_second=-1
--------------------------------------------------
// CONSOLE
The `task_id` can be found using the tasks API above.
Just like when setting it on the `_reindex` API `requests_per_second` can be
either `unlimited` to disable throttling or any decimal number like `1.7` or
`12` to throttle to that level. Rethrottling that speeds up the query takes
effect immediately but rethrotting that slows down the query will take effect
on after completing the current batch. This prevents scroll timeouts.
Just like when setting it on the `_reindex` API `requests_per_second`
can be either `-1` to disable throttling or any decimal number
like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
query takes effect immediately but rethrotting that slows down the query will
take effect on after completing the current batch. This prevents scroll
timeouts.
[float]
=== Reindex to change the name of a field

32
docs/reference/docs/update-by-query.asciidoc
View File

@ -32,7 +32,7 @@ That will return something like this:
"search": 0
},
"throttled_millis": 0,
"requests_per_second": "unlimited",
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"total": 120,
"failures" : [ ]
@ -221,15 +221,15 @@ request. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>.
`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
and throttles the number of requests per second that the update by query issues.
The throttling is done waiting between bulk batches so that it can manipulate
the scroll timeout. The wait time is the difference between the time it took the
batch to complete and the time `requests_per_second * requests_in_the_batch`.
Since the batch isn't broken into multiple bulk requests large batch sizes will
cause Elasticsearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth". The default is
`unlimited` which is also the only non-number value that it accepts.
`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
`1000`, etc) and throttles the number of requests per second that the update-by-query
issues or it can be set to `-1` to disabled throttling. The throttling is done
waiting between bulk batches so that it can manipulate the scroll timeout. The
wait time is the difference between the time it took the batch to complete and
the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
broken into multiple bulk requests large batch sizes will cause Elasticsearch
to create many requests and then wait for a while before starting the next set.
This is "bursty" instead of "smooth". The default is `-1`.
[float]
[[docs-update-by-query-response-body]]
@ -391,18 +391,18 @@ using the `_rethrottle` API:
[source,js]
--------------------------------------------------
POST _update_by_query/taskid:1/_rethrottle?requests_per_second=unlimited
POST _update_by_query/taskid:1/_rethrottle?requests_per_second=-1
--------------------------------------------------
// CONSOLE
The `task_id` can be found using the tasks API above.
Just like when setting it on the `_update_by_query` API `requests_per_second`
can be either `unlimited` to disable throttling or any decimal number like `1.7`
or `12` to throttle to that level. Rethrottling that speeds up the query takes
effect immediately but rethrotting that slows down the query will take effect
on after completing the current batch. This prevents scroll timeouts.
can be either `-1` to disable throttling or any decimal number
like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
query takes effect immediately but rethrotting that slows down the query will
take effect on after completing the current batch. This prevents scroll
timeouts.
[float]
[[picking-up-a-new-property]]

12
modules/reindex/src/main/java/org/elasticsearch/index/reindex/AbstractBaseReindexRestHandler.java
View File

@ -137,20 +137,20 @@ public abstract class AbstractBaseReindexRestHandler<
if (requestsPerSecondString == null) {
return null;
}
if ("unlimited".equals(requestsPerSecondString)) {
return Float.POSITIVE_INFINITY;
}
float requestsPerSecond;
try {
requestsPerSecond = Float.parseFloat(requestsPerSecondString);
} catch (NumberFormatException e) {
throw new IllegalArgumentException(
"[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling.", e);
"[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.", e);
}
if (requestsPerSecond == -1) {
return Float.POSITIVE_INFINITY;
}
if (requestsPerSecond <= 0) {
// We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of "unlimited"
// We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of -1
throw new IllegalArgumentException(
"[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling.");
"[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.");
}
return requestsPerSecond;
}

4
modules/reindex/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java
View File

@ -217,7 +217,7 @@ public class BulkByScrollTask extends CancellableTask {
}
builder.endObject();
builder.timeValueField("throttled_millis", "throttled", throttled);
builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? "unlimited" : requestsPerSecond);
builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? -1 : requestsPerSecond);
if (reasonCancelled != null) {
builder.field("canceled", reasonCancelled);
}
@ -324,7 +324,7 @@ public class BulkByScrollTask extends CancellableTask {
}
/**
* The number of requests per second to which to throttle the request. 0 means unlimited.
* The number of requests per second to which to throttle the request. Float.POSITIVE_INFINITY means unlimited.
*/
public float getRequestsPerSecond() {
return requestsPerSecond;

2
modules/reindex/src/test/java/org/elasticsearch/index/reindex/BulkByScrollTaskTests.java
View File

@ -285,7 +285,7 @@ public class BulkByScrollTaskTests extends ESTestCase {
public void testXContentRepresentationOfUnlimitedRequestsPerSecon() throws IOException {
XContentBuilder builder = JsonXContent.contentBuilder();
task.getStatus().toXContent(builder, ToXContent.EMPTY_PARAMS);
assertThat(builder.string(), containsString("\"requests_per_second\":\"unlimited\""));
assertThat(builder.string(), containsString("\"requests_per_second\":-1"));
}
public void testPerfectlyThrottledBatchTime() {

18
modules/reindex/src/test/resources/rest-api-spec/test/delete_by_query/20_validation.yaml
View File

@ -68,7 +68,7 @@
---
"requests_per_second cannot be an empty string":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query:
requests_per_second: ""
index: test
@ -79,7 +79,7 @@
---
"requests_per_second cannot be negative":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query:
requests_per_second: -12
index: test
@ -90,10 +90,22 @@
---
"requests_per_second cannot be zero":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query:
requests_per_second: 0
index: test
body:
query:
match_all: {}
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
delete_by_query:
requests_per_second: unlimited
index: test
body:
query:
match_all: {}

8
modules/reindex/src/test/resources/rest-api-spec/test/delete_by_query/70_throttle.yaml
View File

@ -40,7 +40,7 @@
- lt: {throttled_millis: 4000}
---
"requests_per_second supports unlimited which turns off throttling":
"requests_per_second supports -1 which turns off throttling":
- do:
indices.create:
index: test
@ -69,7 +69,7 @@
delete_by_query:
index: test
scroll_size: 1
requests_per_second: unlimited
requests_per_second: -1
body:
query:
match_all: {}
@ -79,7 +79,7 @@
- match: {throttled_millis: 0}
---
"Rethrottle":
"Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request
- do:
@ -121,7 +121,7 @@
- do:
reindex.rethrottle:
requests_per_second: unlimited
requests_per_second: -1
task_id: $task
- do:

18
modules/reindex/src/test/resources/rest-api-spec/test/reindex/20_validation.yaml
View File

@ -152,7 +152,7 @@
---
"requests_per_second cannot be an empty string":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex:
requests_per_second: ""
body:
@ -164,7 +164,7 @@
---
"requests_per_second cannot be negative":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex:
requests_per_second: -12
body:
@ -176,7 +176,7 @@
---
"requests_per_second cannot be zero":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex:
requests_per_second: 0
body:
@ -185,6 +185,18 @@
dest:
index: dest
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
reindex:
requests_per_second: unlimited
body:
source:
index: test
dest:
index: dest
---
"reindex without source gives useful error message":
- do:

8
modules/reindex/src/test/resources/rest-api-spec/test/reindex/70_throttle.yaml
View File

@ -50,7 +50,7 @@
- is_false: task
---
"requests_per_second supports unlimited to turn off throttling":
"requests_per_second supports -1 to turn off throttling":
- do:
indices.create:
index: source
@ -81,7 +81,7 @@
- do:
reindex:
requests_per_second: unlimited
requests_per_second: -1
body:
source:
index: source
@ -97,7 +97,7 @@
- is_false: task
---
"Rethrottle":
"Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request
- do:
@ -143,7 +143,7 @@
- do:
reindex.rethrottle:
requests_per_second: unlimited
requests_per_second: -1
task_id: $task
- do:

14
modules/reindex/src/test/resources/rest-api-spec/test/update_by_query/20_validation.yaml
View File

@ -76,7 +76,7 @@
---
"requests_per_second cannot be an empty string":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query:
requests_per_second: ""
index: test
@ -84,7 +84,7 @@
---
"requests_per_second cannot be negative":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query:
requests_per_second: -12
index: test
@ -92,7 +92,15 @@
---
"requests_per_second cannot be zero":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query:
requests_per_second: 0
index: test
---
"requests_per_second cannot be unlimited":
- do:
catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
update_by_query:
requests_per_second: 0
index: test

8
modules/reindex/src/test/resources/rest-api-spec/test/update_by_query/60_throttle.yaml
View File

@ -36,7 +36,7 @@
- lt: {throttled_millis: 4000}
---
"requests_per_second supports unlimited which turns off throttling":
"requests_per_second supports -1 which turns off throttling":
- do:
indices.create:
index: test
@ -65,13 +65,13 @@
update_by_query:
index: test
scroll_size: 1
requests_per_second: unlimited
requests_per_second: -1
- match: {batches: 3}
- match: {updated: 3}
- match: {throttled_millis: 0}
---
"Rethrottle":
"Rethrottle to -1 which turns off throttling":
# Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
# and a small batch size on the request
- do:
@ -109,7 +109,7 @@
- do:
reindex.rethrottle:
requests_per_second: unlimited
requests_per_second: -1
task_id: $task
- do:

2
rest-api-spec/src/main/resources/rest-api-spec/api/delete_by_query.json
View File

@ -195,7 +195,7 @@
"requests_per_second": {
"type": "float",
"default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
"description": "The throttle for this request in sub-requests per second. -1 means set no throttle."
}
}
},

2
rest-api-spec/src/main/resources/rest-api-spec/api/reindex.json
View File

@ -29,7 +29,7 @@
"requests_per_second": {
"type": "float",
"default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
"description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
}
}
},

2
rest-api-spec/src/main/resources/rest-api-spec/api/reindex.rethrottle.json
View File

@ -15,7 +15,7 @@
"requests_per_second": {
"type": "float",
"required": true,
"description": "The throttle to set on this request in sub-requests per second. 0 means set no throttle. As does \"unlimited\". Otherwise it must be a float."
"description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
}
}
},

2
rest-api-spec/src/main/resources/rest-api-spec/api/update_by_query.json
View File

@ -203,7 +203,7 @@
"requests_per_second": {
"type": "float",
"default": 0,
"description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
"description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
}
}
},