* [ML] adding delayed_data_check_config to datafeed update docs * [DOCS] Edits delayed data configuration details
This commit is contained in:
parent
130c832e10
commit
d06618a70d
|
@ -61,12 +61,12 @@ A {dfeed} resource has the following properties:
|
||||||
|
|
||||||
`delayed_data_check_config`::
|
`delayed_data_check_config`::
|
||||||
(object) Specifies whether the data feed checks for missing data and
|
(object) Specifies whether the data feed checks for missing data and
|
||||||
and the size of the window. For example:
|
the size of the window. For example:
|
||||||
`{"enabled": true, "check_window": "1h"}` See
|
`{"enabled": true, "check_window": "1h"}` See
|
||||||
<<ml-datafeed-delayed-data-check-config>>.
|
<<ml-datafeed-delayed-data-check-config>>.
|
||||||
|
|
||||||
[[ml-datafeed-chunking-config]]
|
[[ml-datafeed-chunking-config]]
|
||||||
==== Chunking Configuration Objects
|
==== Chunking configuration objects
|
||||||
|
|
||||||
{dfeeds-cap} might be required to search over long time periods, for several months
|
{dfeeds-cap} might be required to search over long time periods, for several months
|
||||||
or years. This search is split into time chunks in order to ensure the load
|
or years. This search is split into time chunks in order to ensure the load
|
||||||
|
@ -88,31 +88,33 @@ A chunking configuration object has the following properties:
|
||||||
For example: `3h`.
|
For example: `3h`.
|
||||||
|
|
||||||
[[ml-datafeed-delayed-data-check-config]]
|
[[ml-datafeed-delayed-data-check-config]]
|
||||||
==== Delayed Data Check Configuration Objects
|
==== Delayed data check configuration objects
|
||||||
|
|
||||||
The {dfeed} can optionally search over indices that have already been read in
|
The {dfeed} can optionally search over indices that have already been read in
|
||||||
an effort to find if any data has since been added to the index. If missing data
|
an effort to determine whether any data has subsequently been added to the index.
|
||||||
is found, it is a good indication that the `query_delay` option is set too low and
|
If missing data is found, it is a good indication that the `query_delay` option
|
||||||
the data is being indexed after the {dfeed} has passed that moment in time. See
|
is set too low and the data is being indexed after the {dfeed} has passed that
|
||||||
|
moment in time. See
|
||||||
{stack-ov}/ml-delayed-data-detection.html[Working with delayed data].
|
{stack-ov}/ml-delayed-data-detection.html[Working with delayed data].
|
||||||
|
|
||||||
This check only runs on real-time {dfeeds}
|
This check runs only on real-time {dfeeds}.
|
||||||
|
|
||||||
The configuration object has the following properties:
|
The configuration object has the following properties:
|
||||||
|
|
||||||
`enabled`::
|
`enabled`::
|
||||||
(boolean) Should the {dfeed} periodically check for data being indexed after reading.
|
(boolean) Specifies whether the {dfeed} periodically checks for delayed data.
|
||||||
Defaults to `true`
|
Defaults to `true`.
|
||||||
|
|
||||||
`check_window`::
|
`check_window`::
|
||||||
(time units) The window of time before the latest finalized bucket that should be searched
|
(time units) The window of time that is searched for late data. This window of
|
||||||
for late data. Defaults to `null` which causes an appropriate `check_window` to be calculated
|
time ends with the latest finalized bucket. It defaults to `null`, which
|
||||||
when the real-time {dfeed} runs.
|
causes an appropriate `check_window` to be calculated when the real-time
|
||||||
The default `check_window` span calculation is the max between `2h` or `8 * bucket_span`.
|
{dfeed} runs. In particular, the default `check_window` span calculation is
|
||||||
|
based on the maximum of `2h` or `8 * bucket_span`.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-datafeed-counts]]
|
[[ml-datafeed-counts]]
|
||||||
==== {dfeed-cap} Counts
|
==== {dfeed-cap} counts
|
||||||
|
|
||||||
The get {dfeed} statistics API provides information about the operational
|
The get {dfeed} statistics API provides information about the operational
|
||||||
progress of a {dfeed}. All of these properties are informational; you cannot
|
progress of a {dfeed}. All of these properties are informational; you cannot
|
||||||
|
|
|
@ -45,6 +45,11 @@ IMPORTANT: You must use {kib} or this API to create a {dfeed}. Do not put a {df
|
||||||
(object) Specifies how data searches are split into time chunks.
|
(object) Specifies how data searches are split into time chunks.
|
||||||
See <<ml-datafeed-chunking-config>>.
|
See <<ml-datafeed-chunking-config>>.
|
||||||
|
|
||||||
|
`delayed_data_check_config`::
|
||||||
|
(object) Specifies whether the data feed checks for missing data and
|
||||||
|
the size of the window. See
|
||||||
|
<<ml-datafeed-delayed-data-check-config>>.
|
||||||
|
|
||||||
`frequency`::
|
`frequency`::
|
||||||
(time units) The interval at which scheduled queries are made while the {dfeed}
|
(time units) The interval at which scheduled queries are made while the {dfeed}
|
||||||
runs in real time. The default value is either the bucket span for short
|
runs in real time. The default value is either the bucket span for short
|
||||||
|
@ -82,10 +87,6 @@ IMPORTANT: You must use {kib} or this API to create a {dfeed}. Do not put a {df
|
||||||
(unsigned integer) The `size` parameter that is used in {es} searches.
|
(unsigned integer) The `size` parameter that is used in {es} searches.
|
||||||
The default value is `1000`.
|
The default value is `1000`.
|
||||||
|
|
||||||
`delayed_data_check_config`::
|
|
||||||
(object) Specifies if and with how large a window should the data feed check
|
|
||||||
for missing data. See <<ml-datafeed-delayed-data-check-config>>.
|
|
||||||
|
|
||||||
For more information about these properties,
|
For more information about these properties,
|
||||||
see <<ml-datafeed-resource>>.
|
see <<ml-datafeed-resource>>.
|
||||||
|
|
||||||
|
|
|
@ -14,7 +14,10 @@ Updates certain properties of a {dfeed}.
|
||||||
|
|
||||||
`POST _ml/datafeeds/<feed_id>/_update`
|
`POST _ml/datafeeds/<feed_id>/_update`
|
||||||
|
|
||||||
//===== Description
|
===== Description
|
||||||
|
|
||||||
|
NOTE: If you update the `delayed_data_check_config` property, you must stop and
|
||||||
|
start the {dfeed} for the change to be applied.
|
||||||
|
|
||||||
==== Path Parameters
|
==== Path Parameters
|
||||||
|
|
||||||
|
@ -32,6 +35,10 @@ The following properties can be updated after the {dfeed} is created:
|
||||||
`chunking_config`::
|
`chunking_config`::
|
||||||
(object) Specifies how data searches are split into time chunks.
|
(object) Specifies how data searches are split into time chunks.
|
||||||
See <<ml-datafeed-chunking-config>>.
|
See <<ml-datafeed-chunking-config>>.
|
||||||
|
|
||||||
|
`delayed_data_check_config`::
|
||||||
|
(object) Specifies whether the data feed checks for missing data and
|
||||||
|
the size of the window. See <<ml-datafeed-delayed-data-check-config>>.
|
||||||
|
|
||||||
`frequency`::
|
`frequency`::
|
||||||
(time units) The interval at which scheduled queries are made while the
|
(time units) The interval at which scheduled queries are made while the
|
||||||
|
|
Loading…
Reference in New Issue