[ML] adding delayed_data_check_config to datafeed update docs (#42095) (#42626)

* [ML] adding delayed_data_check_config to datafeed update docs

* [DOCS] Edits delayed data configuration details
This commit is contained in:
Benjamin Trent 2019-05-28 11:36:30 -04:00 committed by GitHub
parent 130c832e10
commit d06618a70d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 29 additions and 19 deletions

View File

@ -61,12 +61,12 @@ A {dfeed} resource has the following properties:
`delayed_data_check_config`:: `delayed_data_check_config`::
(object) Specifies whether the data feed checks for missing data and (object) Specifies whether the data feed checks for missing data and
and the size of the window. For example: the size of the window. For example:
`{"enabled": true, "check_window": "1h"}` See `{"enabled": true, "check_window": "1h"}` See
<<ml-datafeed-delayed-data-check-config>>. <<ml-datafeed-delayed-data-check-config>>.
[[ml-datafeed-chunking-config]] [[ml-datafeed-chunking-config]]
==== Chunking Configuration Objects ==== Chunking configuration objects
{dfeeds-cap} might be required to search over long time periods, for several months {dfeeds-cap} might be required to search over long time periods, for several months
or years. This search is split into time chunks in order to ensure the load or years. This search is split into time chunks in order to ensure the load
@ -88,31 +88,33 @@ A chunking configuration object has the following properties:
For example: `3h`. For example: `3h`.
[[ml-datafeed-delayed-data-check-config]] [[ml-datafeed-delayed-data-check-config]]
==== Delayed Data Check Configuration Objects ==== Delayed data check configuration objects
The {dfeed} can optionally search over indices that have already been read in The {dfeed} can optionally search over indices that have already been read in
an effort to find if any data has since been added to the index. If missing data an effort to determine whether any data has subsequently been added to the index.
is found, it is a good indication that the `query_delay` option is set too low and If missing data is found, it is a good indication that the `query_delay` option
the data is being indexed after the {dfeed} has passed that moment in time. See is set too low and the data is being indexed after the {dfeed} has passed that
moment in time. See
{stack-ov}/ml-delayed-data-detection.html[Working with delayed data]. {stack-ov}/ml-delayed-data-detection.html[Working with delayed data].
This check only runs on real-time {dfeeds} This check runs only on real-time {dfeeds}.
The configuration object has the following properties: The configuration object has the following properties:
`enabled`:: `enabled`::
(boolean) Should the {dfeed} periodically check for data being indexed after reading. (boolean) Specifies whether the {dfeed} periodically checks for delayed data.
Defaults to `true` Defaults to `true`.
`check_window`:: `check_window`::
(time units) The window of time before the latest finalized bucket that should be searched (time units) The window of time that is searched for late data. This window of
for late data. Defaults to `null` which causes an appropriate `check_window` to be calculated time ends with the latest finalized bucket. It defaults to `null`, which
when the real-time {dfeed} runs. causes an appropriate `check_window` to be calculated when the real-time
The default `check_window` span calculation is the max between `2h` or `8 * bucket_span`. {dfeed} runs. In particular, the default `check_window` span calculation is
based on the maximum of `2h` or `8 * bucket_span`.
[float] [float]
[[ml-datafeed-counts]] [[ml-datafeed-counts]]
==== {dfeed-cap} Counts ==== {dfeed-cap} counts
The get {dfeed} statistics API provides information about the operational The get {dfeed} statistics API provides information about the operational
progress of a {dfeed}. All of these properties are informational; you cannot progress of a {dfeed}. All of these properties are informational; you cannot

View File

@ -45,6 +45,11 @@ IMPORTANT: You must use {kib} or this API to create a {dfeed}. Do not put a {df
(object) Specifies how data searches are split into time chunks. (object) Specifies how data searches are split into time chunks.
See <<ml-datafeed-chunking-config>>. See <<ml-datafeed-chunking-config>>.
`delayed_data_check_config`::
(object) Specifies whether the data feed checks for missing data and
the size of the window. See
<<ml-datafeed-delayed-data-check-config>>.
`frequency`:: `frequency`::
(time units) The interval at which scheduled queries are made while the {dfeed} (time units) The interval at which scheduled queries are made while the {dfeed}
runs in real time. The default value is either the bucket span for short runs in real time. The default value is either the bucket span for short
@ -82,10 +87,6 @@ IMPORTANT: You must use {kib} or this API to create a {dfeed}. Do not put a {df
(unsigned integer) The `size` parameter that is used in {es} searches. (unsigned integer) The `size` parameter that is used in {es} searches.
The default value is `1000`. The default value is `1000`.
`delayed_data_check_config`::
(object) Specifies if and with how large a window should the data feed check
for missing data. See <<ml-datafeed-delayed-data-check-config>>.
For more information about these properties, For more information about these properties,
see <<ml-datafeed-resource>>. see <<ml-datafeed-resource>>.

View File

@ -14,7 +14,10 @@ Updates certain properties of a {dfeed}.
`POST _ml/datafeeds/<feed_id>/_update` `POST _ml/datafeeds/<feed_id>/_update`
//===== Description ===== Description
NOTE: If you update the `delayed_data_check_config` property, you must stop and
start the {dfeed} for the change to be applied.
==== Path Parameters ==== Path Parameters
@ -32,6 +35,10 @@ The following properties can be updated after the {dfeed} is created:
`chunking_config`:: `chunking_config`::
(object) Specifies how data searches are split into time chunks. (object) Specifies how data searches are split into time chunks.
See <<ml-datafeed-chunking-config>>. See <<ml-datafeed-chunking-config>>.
`delayed_data_check_config`::
(object) Specifies whether the data feed checks for missing data and
the size of the window. See <<ml-datafeed-delayed-data-check-config>>.
`frequency`:: `frequency`::
(time units) The interval at which scheduled queries are made while the (time units) The interval at which scheduled queries are made while the