[DOCS] Add throttling based on configuration parameter (#56653)

This commit is contained in:
Lisa Cawley 2020-05-14 08:38:57 -07:00 committed by lcawl
parent fb5e6329b7
commit 6a8e10189f
5 changed files with 55 additions and 46 deletions

View File

@ -940,15 +940,13 @@ default value is `60s`.
end::sync-time-delay[]
tag::transform-settings[]
Defines the settings of the {transform}.
Defines optional {transform} settings.
end::transform-settings[]
tag::transform-settings-docs-per-second[]
Set `docs-per-second` to throttle the rate transform is issuing search and
index requests. Throttling slows down search requests by adding a wait time
between searches. For the number of docs, the number of input documents is taken.
Throttling can be disabled by setting it to `null`.
The default is no throttling, value `null`.
Specifies a limit on the number of input documents per second. This setting
throttles the {transform} by adding a wait time between search requests. The
default value is `null`, which disables throttling.
end::transform-settings-docs-per-second[]
tag::transform-settings-max-page-search-size[]

View File

@ -128,6 +128,23 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=pivot-group-by]
====
//End pivot
//Begin settings
`settings`::
(Optional, object)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
+
.Properties of `settings`
[%collapsible%open]
====
`docs_per_second`:::
(Optional, float)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
`max_page_search_size`:::
(Optional, integer)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
====
//End settings
//Begin source
`source`::
(Required, object)
@ -185,23 +202,6 @@ delays.
====
//End sync
//Begin settings
`settings`::
(Optional, object)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
+
.Properties of `settings`
[%collapsible%open]
====
`docs_per_second`:::
(Optional, float)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
`max_page_search_size`:::
(Optional, integer)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
====
//End settings
[[put-transform-example]]
==== {api-examples-title}

View File

@ -102,6 +102,23 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-pipeline]
(Optional, <<time-units, time units>>)
include::{docdir}/rest-api/common-parms.asciidoc[tag=frequency]
//Begin settings
`settings`::
(Optional, object)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
+
.Properties of `settings`
[%collapsible%open]
====
`docs_per_second`:::
(Optional, float)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
`max_page_search_size`:::
(Optional, integer)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
====
//End settings
//Begin source
`source`::
(Optional, object)
@ -159,23 +176,6 @@ delays.
====
//End sync
//Begin settings
`settings`::
(Optional, object)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
+
.Properties of `settings`
[%collapsible%open]
====
`docs_per_second`:::
(Optional, float)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
`max_page_search_size`:::
(Optional, integer)
include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
====
//End settings
[[update-transform-example]]
==== {api-examples-title}

View File

@ -49,11 +49,7 @@ requires enough resources to support both the composite aggregation search and
the indexing of its results.
TIP: If the cluster experiences unsuitable performance degradation due to the
{transform}, stop the {transform}. Consider whether you can apply a
source query to the {transform} to reduce the scope of data it
processes. Also consider whether the cluster has sufficient resources in place
to support both the composite aggregation search and the indexing of its
results.
{transform}, stop the {transform} and refer to <<transform-performance>>.
[discrete]
[[ml-transform-checkpoint-errors]]

View File

@ -72,4 +72,19 @@ If your {transform} must process a lot of historic data, it has high resource
usage initially--particularly during the first checkpoint.
For better performance, make sure that your search aggregations and queries are
optimized and that your {transform} is processing only necessary data.
optimized and that your {transform} is processing only necessary data. Consider
whether you can apply a source query to the {transform} to reduce the scope of
data it processes. Also consider whether the cluster has sufficient resources in
place to support both the composite aggregation search and the indexing of its
results.
If you prefer to spread out the impact on your cluster (at the cost of a slower
{transform}), you can throttle the rate at which it performs search and index
requests. Set the `docs_per_second` limit when you <<put-transform,create>> or
<<update-transform,update>> your {transform}. If you want to calculate the
current rate, use the following information from the
{ref}/get-transform-stats.html[get {transform} stats API]:
```
documents_processed / search_time_in_ms * 1000
```