[DOCS] Updates dataframe transform terminology (#46642)

This commit is contained in:
Lisa Cawley 2019-09-16 08:28:19 -07:00 committed by lcawl
parent 92acc732de
commit dddc9b3d73
33 changed files with 357 additions and 357 deletions

View File

@ -5,12 +5,12 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Delete {dataframe-transform} API === Delete {transform} API
Deletes an existing {dataframe-transform}. Deletes an existing {transform}.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Delete {dataframe-transform} request ==== Delete {transform} request
A +{request}+ object requires a non-null `id`. A +{request}+ object requires a non-null `id`.
@ -18,14 +18,14 @@ A +{request}+ object requires a non-null `id`.
--------------------------------------------------- ---------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
--------------------------------------------------- ---------------------------------------------------
<1> Constructing a new request referencing an existing {dataframe-transform} <1> Constructing a new request referencing an existing {transform}
<2> Sets the optional argument `force`. When `true`, the {dataframe-transform} <2> Sets the optional argument `force`. When `true`, the {transform}
is deleted regardless of its current state. The default value is `false`, is deleted regardless of its current state. The default value is `false`,
meaning that only `stopped` {dataframe-transforms} can be deleted. meaning that only `stopped` {transforms} can be deleted.
include::../execution.asciidoc[] include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ object acknowledges the Data Frame Transform deletion. The returned +{response}+ object acknowledges the {transform} deletion.

View File

@ -5,22 +5,22 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Get {dataframe-transform} API === Get {transform} API
Retrieves configuration information about one or more {dataframe-transforms}. Retrieves configuration information about one or more {transforms}.
The API accepts a +{request}+ object and returns a +{response}+. The API accepts a +{request}+ object and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Get {dataframe-transform} request ==== Get {transform} request
A +{request}+ requires either a {dataframe-transform} ID, a comma separated list A +{request}+ requires either a {transform} ID, a comma separated list
of ids or the special wildcard `_all` to get all {dataframe-transforms}. of ids or the special wildcard `_all` to get all {transforms}.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> Constructing a new GET request referencing an existing {dataframe-transform} <1> Constructing a new GET request referencing an existing {transform}
==== Optional arguments ==== Optional arguments
@ -31,9 +31,9 @@ The following arguments are optional.
include-tagged::{doc-tests-file}[{api}-request-options] include-tagged::{doc-tests-file}[{api}-request-options]
-------------------------------------------------- --------------------------------------------------
<1> The page parameters `from` and `size`. `from` specifies the number of <1> The page parameters `from` and `size`. `from` specifies the number of
{dataframe-transforms} to skip. `size` specifies the maximum number of {transforms} to skip. `size` specifies the maximum number of
{dataframe-transforms} to get. Defaults to `0` and `100` respectively. {transforms} to get. Defaults to `0` and `100` respectively.
<2> Whether to ignore if a wildcard expression matches no transforms. <2> Whether to ignore if a wildcard expression matches no {transforms}.
include::../execution.asciidoc[] include::../execution.asciidoc[]
@ -41,7 +41,7 @@ include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ contains the requested {dataframe-transforms}. The returned +{response}+ contains the requested {transforms}.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------

View File

@ -5,22 +5,22 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Get {dataframe-transform} stats API === Get {transform} stats API
Retrieves the operational statistics of one or more {dataframe-transforms}. Retrieves the operational statistics of one or more {transforms}.
The API accepts a +{request}+ object and returns a +{response}+. The API accepts a +{request}+ object and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Get {dataframe-transform} stats request ==== Get {transform} stats request
A +{request}+ requires a data frame transform id or the special wildcard `_all` A +{request}+ requires a {transform} id or the special wildcard `_all`
to get the statistics for all {dataframe-transforms}. to get the statistics for all {transforms}.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> Constructing a new GET Stats request referencing an existing {dataframe-transform} <1> Constructing a new GET Stats request referencing an existing {transform}
==== Optional arguments ==== Optional arguments
@ -31,10 +31,10 @@ The following arguments are optional.
include-tagged::{doc-tests-file}[{api}-request-options] include-tagged::{doc-tests-file}[{api}-request-options]
-------------------------------------------------- --------------------------------------------------
<1> The page parameters `from` and `size`. `from` specifies the number of <1> The page parameters `from` and `size`. `from` specifies the number of
{dataframe-transform} stats to skip. {transform} stats to skip.
`size` specifies the maximum number of {dataframe-transform} stats to get. `size` specifies the maximum number of {transform} stats to get.
Defaults to `0` and `100` respectively. Defaults to `0` and `100` respectively.
<2> Whether to ignore if a wildcard expression matches no transforms. <2> Whether to ignore if a wildcard expression matches no {transforms}.
include::../execution.asciidoc[] include::../execution.asciidoc[]
@ -42,15 +42,15 @@ include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ contains the requested {dataframe-transform} statistics. The returned +{response}+ contains the requested {transform} statistics.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-response] include-tagged::{doc-tests-file}[{api}-response]
-------------------------------------------------- --------------------------------------------------
<1> The response contains a list of `DataFrameTransformStats` objects <1> The response contains a list of `DataFrameTransformStats` objects
<2> The running state of the transform, for example `started`, `indexing`, etc. <2> The running state of the {transform}, for example `started`, `indexing`, etc.
<3> The overall transform statistics recording the number of documents indexed etc. <3> The overall {transform} statistics recording the number of documents indexed etc.
<4> The progress of the current run in the transform. Supplies the number of docs left until the next checkpoint <4> The progress of the current run in the {transform}. Supplies the number of docs left until the next checkpoint
and the total number of docs expected. and the total number of docs expected.
<5> The assigned node information if the task is currently assigned to a node and running. <5> The assigned node information if the task is currently assigned to a node and running.

View File

@ -5,16 +5,16 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Preview {dataframe-transform} API === Preview {transform} API
Previews the results of a {dataframe-transform}. Previews the results of a {transform}.
The API accepts a +{request}+ object as a request and returns a +{response}+. The API accepts a +{request}+ object as a request and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Preview {dataframe-transform} request ==== Preview {transform} request
A +{request}+ takes a single argument: a valid {dataframe-transform} config. A +{request}+ takes a single argument: a valid {transform} config.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
@ -22,7 +22,7 @@ include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> The source config from which the data should be gathered <1> The source config from which the data should be gathered
<2> The pivot config used to transform the data <2> The pivot config used to transform the data
<3> The configuration of the {dataframe-transform} to preview <3> The configuration of the {transform} to preview
include::../execution.asciidoc[] include::../execution.asciidoc[]

View File

@ -5,14 +5,14 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Put {dataframe-transform} API === Put {transform} API
Creates a new {dataframe-transform}. Creates a new {transform}.
The API accepts a +{request}+ object as a request and returns a +{response}+. The API accepts a +{request}+ object as a request and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Put {dataframe-transform} request ==== Put {transform} request
A +{request}+ requires the following argument: A +{request}+ requires the following argument:
@ -20,28 +20,28 @@ A +{request}+ requires the following argument:
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> The configuration of the {dataframe-transform} to create <1> The configuration of the {transform} to create
<2> Whether or not to wait to run deferrable validations until `_start` is called. <2> Whether or not to wait to run deferrable validations until `_start` is called.
This option should be used with care as the created {dataframe-transform} will run This option should be used with care as the created {transform} will run
with the privileges of the user creating it. Meaning, if they do not have privileges, with the privileges of the user creating it. Meaning, if they do not have privileges,
such an error will not be visible until `_start` is called. such an error will not be visible until `_start` is called.
[id="{upid}-{api}-config"] [id="{upid}-{api}-config"]
==== {dataframe-transform-cap} configuration ==== {transform-cap} configuration
The `DataFrameTransformConfig` object contains all the details about the The `DataFrameTransformConfig` object contains all the details about the
{dataframe-transform} configuration and contains the following arguments: {transform} configuration and contains the following arguments:
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-config] include-tagged::{doc-tests-file}[{api}-config]
-------------------------------------------------- --------------------------------------------------
<1> The {dataframe-transform} ID <1> The {transform} ID
<2> The source indices and query from which to gather data <2> The source indices and query from which to gather data
<3> The destination index and optional pipeline <3> The destination index and optional pipeline
<4> How often to check for updates to the source indices <4> How often to check for updates to the source indices
<5> The PivotConfig <5> The PivotConfig
<6> Optional free text description of the transform <6> Optional free text description of the {transform}
[id="{upid}-{api}-query-config"] [id="{upid}-{api}-query-config"]
@ -84,8 +84,8 @@ include-tagged::{doc-tests-file}[{api}-pivot-config]
-------------------------------------------------- --------------------------------------------------
<1> The `GroupConfig` to use in the pivot <1> The `GroupConfig` to use in the pivot
<2> The aggregations to use <2> The aggregations to use
<3> The maximum paging size for the transform when pulling data <3> The maximum paging size for the {transform} when pulling data
from the source. The size dynamically adjusts as the transform from the source. The size dynamically adjusts as the {transform}
is running to recover from and prevent OOM issues. is running to recover from and prevent OOM issues.
===== GroupConfig ===== GroupConfig
@ -121,4 +121,4 @@ include::../execution.asciidoc[]
==== Response ==== Response
The returned +{response}+ acknowledges the successful creation of The returned +{response}+ acknowledges the successful creation of
the new {dataframe-transform} or an error if the configuration is invalid. the new {transform} or an error if the configuration is invalid.

View File

@ -5,13 +5,13 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Start {dataframe-transform} API === Start {transform} API
Starts a {dataframe-transform}. Starts a {transform}.
It accepts a +{request}+ object and responds with a +{response}+ object. It accepts a +{request}+ object and responds with a +{response}+ object.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Start {dataframe-transform} request ==== Start {transform} request
A +{request}+ object requires a non-null `id`. A +{request}+ object requires a non-null `id`.
@ -20,7 +20,7 @@ A +{request}+ object requires a non-null `id`.
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
--------------------------------------------------- ---------------------------------------------------
<1> Constructing a new start request referencing an existing <1> Constructing a new start request referencing an existing
{dataframe-transform} {transform}
==== Optional arguments ==== Optional arguments
@ -30,11 +30,11 @@ The following arguments are optional.
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request-options] include-tagged::{doc-tests-file}[{api}-request-options]
-------------------------------------------------- --------------------------------------------------
<1> Controls the amount of time to wait until the {dataframe-transform} starts. <1> Controls the amount of time to wait until the {transform} starts.
include::../execution.asciidoc[] include::../execution.asciidoc[]
==== Response ==== Response
The returned +{response}+ object acknowledges the {dataframe-transform} has The returned +{response}+ object acknowledges the {transform} has
started. started.

View File

@ -5,13 +5,13 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Stop {dataframe-transform} API === Stop {transform} API
Stops a started {dataframe-transform}. Stops a started {transform}.
It accepts a +{request}+ object and responds with a +{response}+ object. It accepts a +{request}+ object and responds with a +{response}+ object.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Stop {dataframe-transform} request ==== Stop {transform} request
A +{request}+ object requires a non-null `id`. `id` can be a comma separated A +{request}+ object requires a non-null `id`. `id` can be a comma separated
list of IDs or a single ID. Wildcards, `*` and `_all` are also accepted. list of IDs or a single ID. Wildcards, `*` and `_all` are also accepted.
@ -21,7 +21,7 @@ list of IDs or a single ID. Wildcards, `*` and `_all` are also accepted.
--------------------------------------------------- ---------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
--------------------------------------------------- ---------------------------------------------------
<1> Constructing a new stop request referencing an existing {dataframe-transform} <1> Constructing a new stop request referencing an existing {transform}
==== Optional arguments ==== Optional arguments
@ -32,11 +32,11 @@ The following arguments are optional.
include-tagged::{doc-tests-file}[{api}-request-options] include-tagged::{doc-tests-file}[{api}-request-options]
-------------------------------------------------- --------------------------------------------------
<1> If true wait for the data frame task to stop before responding <1> If true wait for the data frame task to stop before responding
<2> Controls the amount of time to wait until the {dataframe-transform} stops. <2> Controls the amount of time to wait until the {transform} stops.
<3> Whether to ignore if a wildcard expression matches no transforms. <3> Whether to ignore if a wildcard expression matches no {transforms}.
include::../execution.asciidoc[] include::../execution.asciidoc[]
==== Response ==== Response
The returned +{response}+ object acknowledges the {dataframe-transform} has stopped. The returned +{response}+ object acknowledges the {transform} has stopped.

View File

@ -5,14 +5,14 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Update {dataframe-transform} API === Update {transform} API
Updates an existing {dataframe-transform}. Updates an existing {transform}.
The API accepts a +{request}+ object as a request and returns a +{response}+. The API accepts a +{request}+ object as a request and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Update {dataframe-transform} request ==== Update {transform} request
A +{request}+ requires the following argument: A +{request}+ requires the following argument:
@ -20,18 +20,18 @@ A +{request}+ requires the following argument:
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> The update configuration with which to update the {dataframe-transform}. <1> The update configuration with which to update the {transform}.
<2> The ID of the configuration to update. <2> The ID of the configuration to update.
<3> Whether or not to wait to run deferrable validations until `_start` is called. <3> Whether or not to wait to run deferrable validations until `_start` is called.
This option should be used with care as the created {dataframe-transform} will run This option should be used with care as the created {transform} will run
with the privileges of the user creating it. Meaning, if they do not have privileges, with the privileges of the user creating it. Meaning, if they do not have privileges,
such an error will not be visible until `_start` is called. such an error will not be visible until `_start` is called.
[id="{upid}-{api}-config"] [id="{upid}-{api}-config"]
==== {dataframe-transform-cap} update configuration ==== {transform-cap} update configuration
The `DataFrameTransformConfigUpdate` object contains all the details about updated The `DataFrameTransformConfigUpdate` object contains all the details about updated
{dataframe-transform} configuration and contains the following arguments: {transform} configuration and contains the following arguments:
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
@ -40,13 +40,13 @@ include-tagged::{doc-tests-file}[{api}-config]
<1> The source indices and query from which to gather data. <1> The source indices and query from which to gather data.
<2> The destination index and optional pipeline. <2> The destination index and optional pipeline.
<3> How often to check for updates to the source indices. <3> How often to check for updates to the source indices.
<4> How to keep the {dataframe-transform} in sync with incoming data. <4> How to keep the {transform} in sync with incoming data.
<5> Optional free text description of the transform. <5> Optional free text description of the {transform}.
include::../execution.asciidoc[] include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ contains the updated {dataframe-transform} configuration The returned +{response}+ contains the updated {transform} configuration
or an error if the update failed or is invalid. or an error if the update failed or is invalid.

View File

@ -5,13 +5,13 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Put Data Frame Analytics API === Put {dfanalytics} API
The Put Data Frame Analytics API is used to create a new {dataframe-analytics-config}. The Put {dfanalytics} API is used to create a new {dataframe-analytics-config}.
The API accepts a +{request}+ object as a request and returns a +{response}+. The API accepts a +{request}+ object as a request and returns a +{response}+.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Put Data Frame Analytics Request ==== Put {dfanalytics} request
A +{request}+ requires the following argument: A +{request}+ requires the following argument:
@ -19,12 +19,12 @@ A +{request}+ requires the following argument:
-------------------------------------------------- --------------------------------------------------
include-tagged::{doc-tests-file}[{api}-request] include-tagged::{doc-tests-file}[{api}-request]
-------------------------------------------------- --------------------------------------------------
<1> The configuration of the {dataframe-job} to create <1> The configuration of the {dfanalytics-job} to create
[id="{upid}-{api}-config"] [id="{upid}-{api}-config"]
==== Data Frame Analytics Configuration ==== {dfanalytics-cap} configuration
The `DataFrameAnalyticsConfig` object contains all the details about the {dataframe-job} The `DataFrameAnalyticsConfig` object contains all the details about the {dfanalytics-job}
configuration and contains the following arguments: configuration and contains the following arguments:
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
@ -64,7 +64,7 @@ include-tagged::{doc-tests-file}[{api}-query-config]
==== DestinationConfig ==== DestinationConfig
The index to which data should be written by the {dataframe-job}. The index to which data should be written by the {dfanalytics-job}.
["source","java",subs="attributes,callouts,macros"] ["source","java",subs="attributes,callouts,macros"]
-------------------------------------------------- --------------------------------------------------
@ -78,7 +78,7 @@ include-tagged::{doc-tests-file}[{api}-dest-config]
The analysis to be performed. The analysis to be performed.
Currently, the supported analyses include : +OutlierDetection+, +Regression+. Currently, the supported analyses include : +OutlierDetection+, +Regression+.
===== Outlier Detection ===== Outlier detection
+OutlierDetection+ analysis can be created in one of two ways: +OutlierDetection+ analysis can be created in one of two ways:

View File

@ -5,13 +5,13 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Start Data Frame Analytics API === Start {dfanalytics} API
The Start Data Frame Analytics API is used to start an existing {dataframe-analytics-config}. The start {dfanalytics} API is used to start an existing {dataframe-analytics-config}.
It accepts a +{request}+ object and responds with a +{response}+ object. It accepts a +{request}+ object and responds with a +{response}+ object.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Start Data Frame Analytics Request ==== Start {dfanalytics} Request
A +{request}+ object requires a {dataframe-analytics-config} id. A +{request}+ object requires a {dataframe-analytics-config} id.
@ -26,4 +26,4 @@ include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ object acknowledges the {dataframe-job} has started. The returned +{response}+ object acknowledges the {dfanalytics-job} has started.

View File

@ -5,13 +5,13 @@
-- --
[role="xpack"] [role="xpack"]
[id="{upid}-{api}"] [id="{upid}-{api}"]
=== Stop Data Frame Analytics API === Stop {dfanalytics} API
The Stop Data Frame Analytics API is used to stop a running {dataframe-analytics-config}. The stop {dfanalytics} API is used to stop a running {dataframe-analytics-config}.
It accepts a +{request}+ object and responds with a +{response}+ object. It accepts a +{request}+ object and responds with a +{response}+ object.
[id="{upid}-{api}-request"] [id="{upid}-{api}-request"]
==== Stop Data Frame Analytics Request ==== Stop {dfanalytics} Request
A +{request}+ object requires a {dataframe-analytics-config} id. A +{request}+ object requires a {dataframe-analytics-config} id.
@ -27,4 +27,4 @@ include::../execution.asciidoc[]
[id="{upid}-{api}-response"] [id="{upid}-{api}-response"]
==== Response ==== Response
The returned +{response}+ object acknowledges the {dataframe-job} has stopped. The returned +{response}+ object acknowledges the {dfanalytics-job} has stopped.

View File

@ -577,12 +577,12 @@ include::ilm/remove_lifecycle_policy_from_index.asciidoc[]
[role="xpack"] [role="xpack"]
[[_data_frame_transform_apis]] [[_data_frame_transform_apis]]
== {dataframe-transform-cap} APIs == {transform-cap} APIs
:upid: {mainid}-dataframe :upid: {mainid}-dataframe
:doc-tests-file: {doc-tests}/DataFrameTransformDocumentationIT.java :doc-tests-file: {doc-tests}/DataFrameTransformDocumentationIT.java
The Java High Level REST Client supports the following {dataframe-transform} The Java High Level REST Client supports the following {transform}
APIs: APIs:
* <<{upid}-get-data-frame-transform>> * <<{upid}-get-data-frame-transform>>

View File

@ -1,26 +1,26 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[delete-data-frame-transform]] [[delete-data-frame-transform]]
=== Delete {dataframe-transforms} API === Delete {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Delete {dataframe-transforms}</titleabbrev> <titleabbrev>Delete {transforms}</titleabbrev>
++++ ++++
Deletes an existing {dataframe-transform}. Deletes an existing {transform}.
beta[] beta[]
[[delete-data-frame-transform-request]] [[delete-data-frame-transform-request]]
==== {api-request-title} ==== {api-request-title}
`DELETE _data_frame/transforms/<data_frame_transform_id>` `DELETE _data_frame/transforms/<transform_id>`
[[delete-data-frame-transform-prereqs]] [[delete-data-frame-transform-prereqs]]
==== {api-prereq-title} ==== {api-prereq-title}
* Before you can delete the {dataframe-transform}, you must stop it. * Before you can delete the {transform}, you must stop it.
* If the {es} {security-features} are enabled, you must have * If the {es} {security-features} are enabled, you must have
`manage_data_frame_transforms` cluster privileges to use this API. The built-in `manage_data_frame_transforms` cluster privileges to use this API. The built-in
`data_frame_transforms_admin` role has these privileges. For more information, `data_frame_transforms_admin` role has these privileges. For more information,
@ -31,15 +31,15 @@ see {stack-ov}/security-privileges.html[Security privileges] and
[[delete-data-frame-transform-path-parms]] [[delete-data-frame-transform-path-parms]]
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<transform_id>`::
(Required, string) Identifier for the {dataframe-transform}. (Required, string) Identifier for the {transform}.
[[delete-data-frame-transform-query-parms]] [[delete-data-frame-transform-query-parms]]
==== {api-query-parms-title} ==== {api-query-parms-title}
`force`:: `force`::
(Optional, boolean) When `true`, the {dataframe-transform} is deleted regardless of its (Optional, boolean) When `true`, the {transform} is deleted regardless of its
current state. The default value is `false`, meaning that the {dataframe-transform} must be current state. The default value is `false`, meaning that the {transform} must be
`stopped` before it can be deleted. `stopped` before it can be deleted.
[[delete-data-frame-transform-examples]] [[delete-data-frame-transform-examples]]
@ -51,7 +51,7 @@ DELETE _data_frame/transforms/ecommerce_transform
-------------------------------------------------- --------------------------------------------------
// TEST[skip:setup kibana sample data] // TEST[skip:setup kibana sample data]
When the {dataframe-transform} is deleted, you receive the following results: When the {transform} is deleted, you receive the following results:
[source,console-result] [source,console-result]
---- ----

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[get-data-frame-transform-stats]] [[get-data-frame-transform-stats]]
=== Get {dataframe-transform} statistics API === Get {transform} statistics API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Get {dataframe-transform} statistics</titleabbrev> <titleabbrev>Get {transform} statistics</titleabbrev>
++++ ++++
Retrieves usage information for {dataframe-transforms}. Retrieves usage information for {transforms}.
beta[] beta[]
@ -16,9 +16,9 @@ beta[]
[[get-data-frame-transform-stats-request]] [[get-data-frame-transform-stats-request]]
==== {api-request-title} ==== {api-request-title}
`GET _data_frame/transforms/<data_frame_transform_id>/_stats` `GET _data_frame/transforms/<transform_id>/_stats`
`GET _data_frame/transforms/<data_frame_transform_id>,<data_frame_transform_id>/_stats` + `GET _data_frame/transforms/<transform_id>,<transform_id>/_stats` +
`GET _data_frame/transforms/_stats` + `GET _data_frame/transforms/_stats` +
@ -40,21 +40,21 @@ see {stack-ov}/security-privileges.html[Security privileges] and
[[get-data-frame-transform-stats-desc]] [[get-data-frame-transform-stats-desc]]
==== {api-description-title} ==== {api-description-title}
You can get statistics for multiple {dataframe-transforms} in a single API You can get statistics for multiple {transforms} in a single API
request by using a comma-separated list of identifiers or a wildcard expression. request by using a comma-separated list of identifiers or a wildcard expression.
You can get statistics for all {dataframe-transforms} by using `_all`, by You can get statistics for all {transforms} by using `_all`, by
specifying `*` as the `<data_frame_transform_id>`, or by omitting the specifying `*` as the `<transform_id>`, or by omitting the
`<data_frame_transform_id>`. `<transform_id>`.
[[get-data-frame-transform-stats-path-parms]] [[get-data-frame-transform-stats-path-parms]]
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<transform_id>`::
(Optional, string) Identifier for the {dataframe-transform}. It can be a (Optional, string) Identifier for the {transform}. It can be a
{dataframe-transform} identifier or a wildcard expression. If you do not {transform} identifier or a wildcard expression. If you do not
specify one of these options, the API returns information for all specify one of these options, the API returns information for all
{dataframe-transforms}. {transforms}.
[[get-data-frame-transform-stats-query-parms]] [[get-data-frame-transform-stats-query-parms]]
@ -64,7 +64,7 @@ specifying `*` as the `<data_frame_transform_id>`, or by omitting the
(Optional, boolean) Specifies what to do when the request: (Optional, boolean) Specifies what to do when the request:
+ +
-- --
* Contains wildcard expressions and there are no {dataframe-transforms} that match. * Contains wildcard expressions and there are no {transforms} that match.
* Contains the `_all` string or no identifiers and there are no matches. * Contains the `_all` string or no identifiers and there are no matches.
* Contains wildcard expressions and there are only partial matches. * Contains wildcard expressions and there are only partial matches.
@ -75,17 +75,17 @@ are no matches or only partial matches.
-- --
`from`:: `from`::
(Optional, integer) Skips the specified number of {dataframe-transforms}. The (Optional, integer) Skips the specified number of {transforms}. The
default value is `0`. default value is `0`.
`size`:: `size`::
(Optional, integer) Specifies the maximum number of {dataframe-transforms} to obtain. The default value is `100`. (Optional, integer) Specifies the maximum number of {transforms} to obtain. The default value is `100`.
[[get-data-frame-transform-stats-response]] [[get-data-frame-transform-stats-response]]
==== {api-response-body-title} ==== {api-response-body-title}
`transforms`:: `transforms`::
(array) An array of statistics objects for {dataframe-transforms}, which are (array) An array of statistics objects for {transforms}, which are
sorted by the `id` value in ascending order. sorted by the `id` value in ascending order.
[[get-data-frame-transform-stats-response-codes]] [[get-data-frame-transform-stats-response-codes]]
@ -98,7 +98,7 @@ are no matches or only partial matches.
[[get-data-frame-transform-stats-example]] [[get-data-frame-transform-stats-example]]
==== Examples ==== Examples
The following example skips for the first five {dataframe-transforms} and The following example skips for the first five {transforms} and
gets usage information for a maximum of ten results: gets usage information for a maximum of ten results:
[source,console] [source,console]
@ -108,7 +108,7 @@ GET _data_frame/transforms/_stats?from=5&size=10
// TEST[skip:todo] // TEST[skip:todo]
The following example gets usage information for the `ecommerce_transform` The following example gets usage information for the `ecommerce_transform`
{dataframe-transform}: {transform}:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[get-data-frame-transform]] [[get-data-frame-transform]]
=== Get {dataframe-transforms} API === Get {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Get {dataframe-transforms}</titleabbrev> <titleabbrev>Get {transforms}</titleabbrev>
++++ ++++
Retrieves configuration information for {dataframe-transforms}. Retrieves configuration information for {transforms}.
beta[] beta[]
@ -37,9 +37,9 @@ see {stack-ov}/security-privileges.html[Security privileges] and
[[get-data-frame-transform-desc]] [[get-data-frame-transform-desc]]
==== {api-description-title} ==== {api-description-title}
You can get information for multiple {dataframe-transforms} in a single API You can get information for multiple {transforms} in a single API
request by using a comma-separated list of identifiers or a wildcard expression. request by using a comma-separated list of identifiers or a wildcard expression.
You can get information for all {dataframe-transforms} by using `_all`, by You can get information for all {transforms} by using `_all`, by
specifying `*` as the `<data_frame_transform_id>`, or by omitting the specifying `*` as the `<data_frame_transform_id>`, or by omitting the
`<data_frame_transform_id>`. `<data_frame_transform_id>`.
@ -47,10 +47,10 @@ specifying `*` as the `<data_frame_transform_id>`, or by omitting the
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<data_frame_transform_id>`::
(Optional, string) Identifier for the {dataframe-transform}. It can be a (Optional, string) Identifier for the {transform}. It can be a
{dataframe-transform} identifier or a wildcard expression. If you do not {transform} identifier or a wildcard expression. If you do not
specify one of these options, the API returns information for all specify one of these options, the API returns information for all
{dataframe-transforms}. {transforms}.
[[get-data-frame-transform-query-parms]] [[get-data-frame-transform-query-parms]]
==== {api-query-parms-title} ==== {api-query-parms-title}
@ -59,7 +59,7 @@ specifying `*` as the `<data_frame_transform_id>`, or by omitting the
(Optional, boolean) Specifies what to do when the request: (Optional, boolean) Specifies what to do when the request:
+ +
-- --
* Contains wildcard expressions and there are no {dataframe-transforms} that match. * Contains wildcard expressions and there are no {transforms} that match.
* Contains the `_all` string or no identifiers and there are no matches. * Contains the `_all` string or no identifiers and there are no matches.
* Contains wildcard expressions and there are only partial matches. * Contains wildcard expressions and there are only partial matches.
@ -70,17 +70,17 @@ are no matches or only partial matches.
-- --
`from`:: `from`::
(Optional, integer) Skips the specified number of {dataframe-transforms}. The (Optional, integer) Skips the specified number of {transforms}. The
default value is `0`. default value is `0`.
`size`:: `size`::
(Optional, integer) Specifies the maximum number of {dataframe-transforms} to obtain. The default value is `100`. (Optional, integer) Specifies the maximum number of {transforms} to obtain. The default value is `100`.
[[get-data-frame-transform-response]] [[get-data-frame-transform-response]]
==== {api-response-body-title} ==== {api-response-body-title}
`transforms`:: `transforms`::
(array) An array of transform resources, which are sorted by the `id` value in (array) An array of {transform} resources, which are sorted by the `id` value in
ascending order. See <<data-frame-transform-resource>>. ascending order. See <<data-frame-transform-resource>>.
[[get-data-frame-transform-response-codes]] [[get-data-frame-transform-response-codes]]
@ -93,7 +93,7 @@ are no matches or only partial matches.
[[get-data-frame-transform-example]] [[get-data-frame-transform-example]]
==== {api-examples-title} ==== {api-examples-title}
The following example retrieves information about a maximum of ten transforms: The following example retrieves information about a maximum of ten {transforms}:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -102,7 +102,7 @@ GET _data_frame/transforms?size=10
// TEST[skip:setup kibana sample data] // TEST[skip:setup kibana sample data]
The following example gets configuration information for the The following example gets configuration information for the
`ecommerce_transform` {dataframe-transform}: `ecommerce_transform` {transform}:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------

View File

@ -1,9 +1,9 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[data-frame-apis]] [[data-frame-apis]]
== {dataframe-transform-cap} APIs == {transform-cap} APIs
See also {stack-ov}/ml-dataframes.html[{dataframe-transforms-cap}]. See also {stack-ov}/ml-dataframes.html[{transforms-cap}].
* <<put-data-frame-transform>> * <<put-data-frame-transform>>
* <<update-data-frame-transform>> * <<update-data-frame-transform>>

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[preview-data-frame-transform]] [[preview-data-frame-transform]]
=== Preview {dataframe-transforms} API === Preview {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Preview {dataframe-transforms}</titleabbrev> <titleabbrev>Preview {transforms}</titleabbrev>
++++ ++++
Previews a {dataframe-transform}. Previews a {transform}.
beta[] beta[]
@ -24,7 +24,7 @@ beta[]
`manage_data_frame_transforms` cluster privileges to use this API. The built-in `manage_data_frame_transforms` cluster privileges to use this API. The built-in
`data_frame_transforms_admin` role has these privileges. You must also have `data_frame_transforms_admin` role has these privileges. You must also have
`read` and `view_index_metadata` privileges on the source index for the `read` and `view_index_metadata` privileges on the source index for the
{dataframe-transform}. For more information, see {transform}. For more information, see
{stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/security-privileges.html[Security privileges] and
{stack-ov}/built-in-roles.html[Built-in roles]. {stack-ov}/built-in-roles.html[Built-in roles].
@ -32,7 +32,7 @@ beta[]
==== {api-description-title} ==== {api-description-title}
This API generates a preview of the results that you will get when you run the This API generates a preview of the results that you will get when you run the
<<put-data-frame-transform,create {dataframe-transforms} API>> with the same <<put-data-frame-transform,create {transforms} API>> with the same
configuration. It returns a maximum of 100 results. The calculations are based configuration. It returns a maximum of 100 results. The calculations are based
on all the current data in the source index. on all the current data in the source index.
@ -45,7 +45,7 @@ on all the current data in the source index.
`index`::: `index`:::
(Required, string or array) The _source indices_ for the (Required, string or array) The _source indices_ for the
{dataframe-transform}. It can be a single index, an index pattern (for {transform}. It can be a single index, an index pattern (for
example, `"myindex*"`), or an array of indices (for example, example, `"myindex*"`), or an array of indices (for example,
`["index1", "index2"]`). `["index1", "index2"]`).
@ -63,7 +63,7 @@ on all the current data in the source index.
`preview`:: `preview`::
(array) An array of documents. In particular, they are the JSON (array) An array of documents. In particular, they are the JSON
representation of the documents that would be created in the destination index representation of the documents that would be created in the destination index
by the {dataframe-transform}. by the {transform}.
==== {api-examples-title} ==== {api-examples-title}

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[put-data-frame-transform]] [[put-data-frame-transform]]
=== Create {dataframe-transforms} API === Create {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Create {dataframe-transforms}</titleabbrev> <titleabbrev>Create {transforms}</titleabbrev>
++++ ++++
Instantiates a {dataframe-transform}. Instantiates a {transform}.
beta[] beta[]
@ -31,30 +31,30 @@ information, see {stack-ov}/security-privileges.html[Security privileges] and
[[put-data-frame-transform-desc]] [[put-data-frame-transform-desc]]
==== {api-description-title} ==== {api-description-title}
This API defines a {dataframe-transform}, which copies data from source indices, This API defines a {transform}, which copies data from source indices,
transforms it, and persists it into an entity-centric destination index. The transforms it, and persists it into an entity-centric destination index. The
entities are defined by the set of `group_by` fields in the `pivot` object. You entities are defined by the set of `group_by` fields in the `pivot` object. You
can also think of the destination index as a two-dimensional tabular data can also think of the destination index as a two-dimensional tabular data
structure (known as a {dataframe}). The ID for each document in the structure (known as a {dataframe}). The ID for each document in the
{dataframe} is generated from a hash of the entity, so there is a unique row {dataframe} is generated from a hash of the entity, so there is a unique row
per entity. For more information, see per entity. For more information, see
{stack-ov}/ml-dataframes.html[{dataframe-transforms-cap}]. {stack-ov}/ml-dataframes.html[{transforms-cap}].
When the {dataframe-transform} is created, a series of validations occur to When the {transform} is created, a series of validations occur to
ensure its success. For example, there is a check for the existence of the ensure its success. For example, there is a check for the existence of the
source indices and a check that the destination index is not part of the source source indices and a check that the destination index is not part of the source
index pattern. You can use the `defer_validation` parameter to skip these index pattern. You can use the `defer_validation` parameter to skip these
checks. checks.
Deferred validations are always run when the {dataframe-transform} is started, Deferred validations are always run when the {transform} is started,
with the exception of privilege checks. When {es} {security-features} are with the exception of privilege checks. When {es} {security-features} are
enabled, the {dataframe-transform} remembers which roles the user that created enabled, the {transform} remembers which roles the user that created
it had at the time of creation and uses those same roles. If those roles do not it had at the time of creation and uses those same roles. If those roles do not
have the required privileges on the source and destination indices, the have the required privileges on the source and destination indices, the
{dataframe-transform} fails when it attempts unauthorized operations. {transform} fails when it attempts unauthorized operations.
IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. IMPORTANT: You must use {kib} or this API to create a {transform}.
Do not put a {dataframe-transform} directly into any Do not put a {transform} directly into any
`.data-frame-internal*` indices using the Elasticsearch index API. `.data-frame-internal*` indices using the Elasticsearch index API.
If {es} {security-features} are enabled, do not give users any If {es} {security-features} are enabled, do not give users any
privileges on `.data-frame-internal*` indices. privileges on `.data-frame-internal*` indices.
@ -63,7 +63,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}.
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<data_frame_transform_id>`::
(Required, string) Identifier for the {dataframe-transform}. This identifier (Required, string) Identifier for the {transform}. This identifier
can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and
underscores. It must start and end with alphanumeric characters. underscores. It must start and end with alphanumeric characters.
@ -73,28 +73,28 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}.
`defer_validation`:: `defer_validation`::
(Optional, boolean) When `true`, deferrable validations are not run. This (Optional, boolean) When `true`, deferrable validations are not run. This
behavior may be desired if the source index does not exist until after the behavior may be desired if the source index does not exist until after the
{dataframe-transform} is created. {transform} is created.
[[put-data-frame-transform-request-body]] [[put-data-frame-transform-request-body]]
==== {api-request-body-title} ==== {api-request-body-title}
`description`:: `description`::
(Optional, string) Free text description of the {dataframe-transform}. (Optional, string) Free text description of the {transform}.
`dest`:: `dest`::
(Required, object) Required. The destination configuration, which has the (Required, object) Required. The destination configuration, which has the
following properties: following properties:
`index`::: `index`:::
(Required, string) The _destination index_ for the {dataframe-transform}. (Required, string) The _destination index_ for the {transform}.
`pipeline`::: `pipeline`:::
(Optional, string) The unique identifier for a <<pipeline,pipeline>>. (Optional, string) The unique identifier for a <<pipeline,pipeline>>.
`frequency`:: `frequency`::
(Optional, <<time-units, time units>>) The interval between checks for changes in the source (Optional, <<time-units, time units>>) The interval between checks for changes in the source
indices when the {dataframe-transform} is running continuously. Also determines indices when the {transform} is running continuously. Also determines
the retry interval in the event of transient failures while the {dataframe-transform} is the retry interval in the event of transient failures while the {transform} is
searching or indexing. The minimum value is `1s` and the maximum is `1h`. The searching or indexing. The minimum value is `1s` and the maximum is `1h`. The
default value is `1m`. default value is `1m`.
@ -108,7 +108,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}.
`index`::: `index`:::
(Required, string or array) The _source indices_ for the (Required, string or array) The _source indices_ for the
{dataframe-transform}. It can be a single index, an index pattern (for {transform}. It can be a single index, an index pattern (for
example, `"myindex*"`), or an array of indices (for example, example, `"myindex*"`), or an array of indices (for example,
`["index1", "index2"]`). `["index1", "index2"]`).
@ -119,7 +119,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}.
`sync`:: `sync`::
(Optional, object) Defines the properties required to run continuously. (Optional, object) Defines the properties required to run continuously.
`time`::: `time`:::
(Required, object) Specifies that the {dataframe-transform} uses a time (Required, object) Specifies that the {transform} uses a time
field to synchronize the source and destination indices. field to synchronize the source and destination indices.
`field`:::: `field`::::
(Required, string) The date field that is used to identify new documents (Required, string) The date field that is used to identify new documents
@ -185,7 +185,7 @@ PUT _data_frame/transforms/ecommerce_transform
-------------------------------------------------- --------------------------------------------------
// TEST[setup:kibana_sample_data_ecommerce] // TEST[setup:kibana_sample_data_ecommerce]
When the transform is created, you receive the following results: When the {transform} is created, you receive the following results:
[source,console-result] [source,console-result]
---- ----

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[start-data-frame-transform]] [[start-data-frame-transform]]
=== Start {dataframe-transforms} API === Start {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Start {dataframe-transforms}</titleabbrev> <titleabbrev>Start {transforms}</titleabbrev>
++++ ++++
Starts one or more {dataframe-transforms}. Starts one or more {transforms}.
beta[] beta[]
@ -23,38 +23,38 @@ beta[]
* If the {es} {security-features} are enabled, you must have * If the {es} {security-features} are enabled, you must have
`manage_data_frame_transforms` cluster privileges to use this API. You must also `manage_data_frame_transforms` cluster privileges to use this API. You must also
have `view_index_metadata` privileges on the source index for the have `view_index_metadata` privileges on the source index for the
{dataframe-transform}. For more information, see {transform}. For more information, see
{stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/security-privileges.html[Security privileges] and
{stack-ov}/built-in-roles.html[Built-in roles]. {stack-ov}/built-in-roles.html[Built-in roles].
[[start-data-frame-transform-desc]] [[start-data-frame-transform-desc]]
==== {api-description-title} ==== {api-description-title}
When you start a {dataframe-transform}, it creates the destination index if it When you start a {transform}, it creates the destination index if it
does not already exist. The `number_of_shards` is set to `1` and the does not already exist. The `number_of_shards` is set to `1` and the
`auto_expand_replicas` is set to `0-1`. `auto_expand_replicas` is set to `0-1`.
The transform deduces the mapping definitions from the source indices. For The {transform} deduces the mapping definitions from the source indices. For
scripted fields, it uses <<dynamic-mapping,dynamic mappings>>. If a field in the scripted fields, it uses <<dynamic-mapping,dynamic mappings>>. If a field in the
destination index is created by `scripted_metric` or `bucket_script` destination index is created by `scripted_metric` or `bucket_script`
aggregations, the transform uses dynamic mappings unless a template exists or aggregations, the {transform} uses dynamic mappings unless a template exists or
the destination index already exists. Mapping definitions in the destination the destination index already exists. Mapping definitions in the destination
index take precedence over dynamic mappings and templates. index take precedence over dynamic mappings and templates.
When the {dataframe-transform} starts, a series of validations occur to ensure When the {transform} starts, a series of validations occur to ensure
its success. If you deferred validation when you created the its success. If you deferred validation when you created the
{dataframe-transform}, they occur when you start the transform--with the {transform}, they occur when you start the {transform}--with the
exception of privilege checks. When {es} {security-features} are enabled, the exception of privilege checks. When {es} {security-features} are enabled, the
{dataframe-transform} remembers which roles the user that created it had at the {transform} remembers which roles the user that created it had at the
time of creation and uses those same roles. If those roles do not have the time of creation and uses those same roles. If those roles do not have the
required privileges on the source and destination indices, the required privileges on the source and destination indices, the
{dataframe-transform} fails when it attempts unauthorized operations. {transform} fails when it attempts unauthorized operations.
[[start-data-frame-transform-path-parms]] [[start-data-frame-transform-path-parms]]
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<data_frame_transform_id>`::
(Required, string) Identifier for the {dataframe-transform}. This identifier (Required, string) Identifier for the {transform}. This identifier
can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and
underscores. It must start and end with alphanumeric characters. underscores. It must start and end with alphanumeric characters.
@ -67,7 +67,7 @@ POST _data_frame/transforms/ecommerce_transform/_start
-------------------------------------------------- --------------------------------------------------
// TEST[skip:set up kibana samples] // TEST[skip:set up kibana samples]
When the {dataframe-transform} starts, you receive the following results: When the {transform} starts, you receive the following results:
[source,console-result] [source,console-result]
---- ----

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[stop-data-frame-transform]] [[stop-data-frame-transform]]
=== Stop {dataframe-transforms} API === Stop {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Stop {dataframe-transforms}</titleabbrev> <titleabbrev>Stop {transforms}</titleabbrev>
++++ ++++
Stops one or more {dataframe-transforms}. Stops one or more {transforms}.
beta[] beta[]
@ -33,16 +33,16 @@ see {stack-ov}/security-privileges.html[Security privileges] and
[[stop-data-frame-transform-desc]] [[stop-data-frame-transform-desc]]
==== {api-description-title} ==== {api-description-title}
You can stop multiple {dataframe-transforms} in a single API request by using a You can stop multiple {transforms} in a single API request by using a
comma-separated list of {dataframe-transforms} or a wildcard expression. comma-separated list of {transforms} or a wildcard expression.
All {dataframe-transforms} can be stopped by using `_all` or `*` as the All {transforms} can be stopped by using `_all` or `*` as the
`<data_frame_transform_id>`. `<data_frame_transform_id>`.
[[stop-data-frame-transform-path-parms]] [[stop-data-frame-transform-path-parms]]
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<data_frame_transform_id>`::
(Required, string) Identifier for the {dataframe-transform}. This identifier (Required, string) Identifier for the {transform}. This identifier
can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and
underscores. It must start and end with alphanumeric characters. underscores. It must start and end with alphanumeric characters.
@ -53,15 +53,15 @@ All {dataframe-transforms} can be stopped by using `_all` or `*` as the
(Optional, boolean) Specifies what to do when the request: (Optional, boolean) Specifies what to do when the request:
+ +
-- --
* Contains wildcard expressions and there are no {dataframe-transforms} that match. * Contains wildcard expressions and there are no {transforms} that match.
* Contains the `_all` string or no identifiers and there are no matches. * Contains the `_all` string or no identifiers and there are no matches.
* Contains wildcard expressions and there are only partial matches. * Contains wildcard expressions and there are only partial matches.
The default value is `true`, which returns a successful acknowledgement message The default value is `true`, which returns a successful acknowledgement message
when there are no matches. When there are only partial matches, the API stops when there are no matches. When there are only partial matches, the API stops
the appropriate {dataframe-transforms}. For example, if the request contains the appropriate {transforms}. For example, if the request contains
`test-id1*,test-id2*` as the identifiers and there are no {dataframe-transforms} `test-id1*,test-id2*` as the identifiers and there are no {transforms}
that match `test-id2*`, the API nonetheless stops the {dataframe-transforms} that match `test-id2*`, the API nonetheless stops the {transforms}
that match `test-id1*`. that match `test-id1*`.
If this parameter is `false`, the request returns a `404` status code when there If this parameter is `false`, the request returns a `404` status code when there
@ -70,10 +70,10 @@ are no matches or only partial matches.
`timeout`:: `timeout`::
(Optional, time value) If `wait_for_completion=true`, the API blocks for (at (Optional, time value) If `wait_for_completion=true`, the API blocks for (at
maximum) the specified duration while waiting for the transform to stop. If maximum) the specified duration while waiting for the {transform} to stop. If
more than `timeout` time has passed, the API throws a timeout exception. Even more than `timeout` time has passed, the API throws a timeout exception. Even
if a timeout exception is thrown, the stop request is still processing and if a timeout exception is thrown, the stop request is still processing and
eventually moves the transform to `STOPPED`. The timeout simply means the API eventually moves the {transform} to `STOPPED`. The timeout simply means the API
call itself timed out while waiting for the status change. Defaults to `30s` call itself timed out while waiting for the status change. Defaults to `30s`
`wait_for_completion`:: `wait_for_completion`::
@ -97,7 +97,7 @@ POST _data_frame/transforms/ecommerce_transform/_stop
-------------------------------------------------- --------------------------------------------------
// TEST[skip:set up kibana samples] // TEST[skip:set up kibana samples]
When the {dataframe-transform} stops, you receive the following results: When the {transform} stops, you receive the following results:
[source,console-result] [source,console-result]
---- ----

View File

@ -1,9 +1,9 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[data-frame-transform-resource]] [[data-frame-transform-resource]]
=== {dataframe-transform-cap} resources === {transform-cap} resources
{dataframe-transform-cap} resources relate to the <<data-frame-apis>>. {transform-cap} resources relate to the <<data-frame-apis>>.
For more information, see For more information, see
{stack-ov}/ecommerce-dataframes.html[Transforming your data with {dataframes}]. {stack-ov}/ecommerce-dataframes.html[Transforming your data with {dataframes}].
@ -13,42 +13,42 @@ For more information, see
==== {api-definitions-title} ==== {api-definitions-title}
`description`:: `description`::
(string) A description of the {dataframe-transform}. (string) A description of the {transform}.
`dest`:: `dest`::
(object) The destination for the {dataframe-transform}. See (object) The destination for the {transform}. See
<<data-frame-transform-dest>>. <<data-frame-transform-dest>>.
`frequency`:: `frequency`::
(time units) The interval between checks for changes in the source indices (time units) The interval between checks for changes in the source indices
when the {dataframe-transform} is running continuously. Also determines the when the {transform} is running continuously. Also determines the
retry interval in the event of transient failures while the {dataframe-transform} is retry interval in the event of transient failures while the {transform} is
searching or indexing. The minimum value is `1s` and the maximum is `1h`. The searching or indexing. The minimum value is `1s` and the maximum is `1h`. The
default value is `1m`. default value is `1m`.
`id`:: `id`::
(string) A unique identifier for the {dataframe-transform}. (string) A unique identifier for the {transform}.
`pivot`:: `pivot`::
(object) The method for transforming the data. See (object) The method for transforming the data. See
<<data-frame-transform-pivot>>. <<data-frame-transform-pivot>>.
`source`:: `source`::
(object) The source of the data for the {dataframe-transform}. See (object) The source of the data for the {transform}. See
<<data-frame-transform-source>>. <<data-frame-transform-source>>.
[[data-frame-transform-dest]] [[data-frame-transform-dest]]
==== Dest objects ==== Dest objects
{dataframe-transform-cap} resources contain `dest` objects. For example, when {transform-cap} resources contain `dest` objects. For example, when
you create a {dataframe-transform}, you must define its destination. you create a {transform}, you must define its destination.
[discrete] [discrete]
[[data-frame-transform-dest-properties]] [[data-frame-transform-dest-properties]]
===== {api-definitions-title} ===== {api-definitions-title}
`index`:: `index`::
(string) The _destination index_ for the {dataframe-transform}. (string) The _destination index_ for the {transform}.
`pipeline`:: `pipeline`::
(string) The unique identifier for a <<pipeline,pipeline>>. (string) The unique identifier for a <<pipeline,pipeline>>.
@ -56,15 +56,15 @@ you create a {dataframe-transform}, you must define its destination.
[[data-frame-transform-source]] [[data-frame-transform-source]]
==== Source objects ==== Source objects
{dataframe-transform-cap} resources contain `source` objects. For example, when {transform-cap} resources contain `source` objects. For example, when
you create a {dataframe-transform}, you must define its source. you create a {transform}, you must define its source.
[discrete] [discrete]
[[data-frame-transform-source-properties]] [[data-frame-transform-source-properties]]
===== {api-definitions-title} ===== {api-definitions-title}
`index`:: `index`::
(string or array) The _source indices_ for the {dataframe-transform}. It can (string or array) The _source indices_ for the {transform}. It can
be a single index, an index pattern (for example, `"myindex*"`), or an array be a single index, an index pattern (for example, `"myindex*"`), or an array
of indices (for example, `["index1", "index2"]`). of indices (for example, `["index1", "index2"]`).
@ -75,7 +75,7 @@ you create a {dataframe-transform}, you must define its source.
[[data-frame-transform-pivot]] [[data-frame-transform-pivot]]
==== Pivot objects ==== Pivot objects
{dataframe-transform-cap} resources contain `pivot` objects, which define the {transform-cap} resources contain `pivot` objects, which define the
pivot function `group by` fields and the aggregation to reduce the data. pivot function `group by` fields and the aggregation to reduce the data.
[discrete] [discrete]
@ -100,7 +100,7 @@ pivot function `group by` fields and the aggregation to reduce the data.
* {ref}/search-aggregations-pipeline-bucket-script-aggregation.html[Bucket Script] * {ref}/search-aggregations-pipeline-bucket-script-aggregation.html[Bucket Script]
* {ref}/search-aggregations-pipeline-bucket-selector-aggregation.html[Bucket Selector] * {ref}/search-aggregations-pipeline-bucket-selector-aggregation.html[Bucket Selector]
IMPORTANT: {dataframe-transforms-cap} support a subset of the functionality in IMPORTANT: {transforms-cap} support a subset of the functionality in
composite aggregations. See composite aggregations. See
{stack-ov}/dataframe-limitations.html[{dataframe-cap} limitations]. {stack-ov}/dataframe-limitations.html[{dataframe-cap} limitations].
@ -126,4 +126,4 @@ composite aggregations. See
==== {api-examples-title} ==== {api-examples-title}
See the See the
<<put-data-frame-transform-example,create {dataframe-transforms} API examples>>. <<put-data-frame-transform-example,create {transforms} API examples>>.

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[update-data-frame-transform]] [[update-data-frame-transform]]
=== Update {dataframe-transforms} API === Update {transforms} API
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Update {dataframe-transforms}</titleabbrev> <titleabbrev>Update {transforms}</titleabbrev>
++++ ++++
Updates an existing {dataframe-transform}. Updates an existing {transform}.
beta[] beta[]
@ -31,16 +31,16 @@ information, see {stack-ov}/security-privileges.html[Security privileges] and
[[update-data-frame-transform-desc]] [[update-data-frame-transform-desc]]
==== {api-description-title} ==== {api-description-title}
This API updates an existing {dataframe-transform}. All settings except description do not This API updates an existing {transform}. All settings except description do not
take effect until after the {dataframe-transform} starts the next checkpoint. This is take effect until after the {transform} starts the next checkpoint. This is
so there is consistency with the pivoted data in each checkpoint. so there is consistency with the pivoted data in each checkpoint.
IMPORTANT: When {es} {security-features} are enabled, your {dataframe-transform} IMPORTANT: When {es} {security-features} are enabled, your {transform}
remembers which roles the user who updated it had at the time of update and remembers which roles the user who updated it had at the time of update and
runs with those privileges. runs with those privileges.
IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. IMPORTANT: You must use {kib} or this API to update a {transform}.
Do not update a {dataframe-transform} directly via Do not update a {transform} directly via
`.data-frame-internal*` indices using the Elasticsearch index API. `.data-frame-internal*` indices using the Elasticsearch index API.
If {es} {security-features} are enabled, do not give users any If {es} {security-features} are enabled, do not give users any
privileges on `.data-frame-internal*` indices. privileges on `.data-frame-internal*` indices.
@ -49,7 +49,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}.
==== {api-path-parms-title} ==== {api-path-parms-title}
`<data_frame_transform_id>`:: `<data_frame_transform_id>`::
(Required, string) Identifier for the {dataframe-transform}. This identifier (Required, string) Identifier for the {transform}. This identifier
can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and
underscores. It must start and end with alphanumeric characters. underscores. It must start and end with alphanumeric characters.
@ -59,29 +59,29 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}.
`defer_validation`:: `defer_validation`::
(Optional, boolean) When `true`, deferrable validations are not run. This (Optional, boolean) When `true`, deferrable validations are not run. This
behavior may be desired if the source index does not exist until after the behavior may be desired if the source index does not exist until after the
{dataframe-transform} is updated. {transform} is updated.
[[update-data-frame-transform-request-body]] [[update-data-frame-transform-request-body]]
==== {api-request-body-title} ==== {api-request-body-title}
`description`:: `description`::
(Optional, string) Free text description of the {dataframe-transform}. (Optional, string) Free text description of the {transform}.
`dest`:: `dest`::
(Optional, object) The destination configuration, which has the (Optional, object) The destination configuration, which has the
following properties: following properties:
`index`::: `index`:::
(Required, string) The _destination index_ for the {dataframe-transform}. (Required, string) The _destination index_ for the {transform}.
`pipeline`::: `pipeline`:::
(Optional, string) The unique identifier for a <<pipeline,pipeline>>. (Optional, string) The unique identifier for a <<pipeline,pipeline>>.
`frequency`:: `frequency`::
(Optional, <<time-units, time units>>) The interval between checks for changes (Optional, <<time-units, time units>>) The interval between checks for changes
in the source indices when the {dataframe-transform} is running continuously. in the source indices when the {transform} is running continuously.
Also determines the retry interval in the event of transient failures while Also determines the retry interval in the event of transient failures while
the {dataframe-transform} is searching or indexing. The minimum value is `1s` the {transform} is searching or indexing. The minimum value is `1s`
and the maximum is `1h`. The default value is `1m`. and the maximum is `1h`. The default value is `1m`.
`source`:: `source`::
@ -90,7 +90,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}.
`index`::: `index`:::
(Required, string or array) The _source indices_ for the (Required, string or array) The _source indices_ for the
{dataframe-transform}. It can be a single index, an index pattern (for {transform}. It can be a single index, an index pattern (for
example, `"myindex*"`), or an array of indices (for example, example, `"myindex*"`), or an array of indices (for example,
`["index1", "index2"]`). `["index1", "index2"]`).
@ -101,7 +101,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}.
`sync`:: `sync`::
(Optional, object) Defines the properties required to run continuously. (Optional, object) Defines the properties required to run continuously.
`time`::: `time`:::
(Required, object) Specifies that the {dataframe-transform} uses a time (Required, object) Specifies that the {transform} uses a time
field to synchronize the source and destination indices. field to synchronize the source and destination indices.
`field`:::: `field`::::
(Required, string) The date field that is used to identify new documents (Required, string) The date field that is used to identify new documents
@ -151,7 +151,7 @@ POST _data_frame/transforms/simple-kibana-ecomm-pivot/_update
-------------------------------------------------- --------------------------------------------------
// TEST[setup:simple_kibana_continuous_pivot] // TEST[setup:simple_kibana_continuous_pivot]
When the transform is updated, you receive the updated configuration: When the {transform} is updated, you receive the updated configuration:
[source,console-result] [source,console-result]
---- ----

View File

@ -8,7 +8,7 @@ These resource definitions are used in APIs related to {ml-features} and
* <<ml-datafeed-resource,{dfeeds-cap}>> * <<ml-datafeed-resource,{dfeeds-cap}>>
* <<ml-datafeed-counts,{dfeed-cap} counts>> * <<ml-datafeed-counts,{dfeed-cap} counts>>
* <<ml-dfanalytics-resources,{dfanalytics-cap}>> * <<ml-dfanalytics-resources,{dfanalytics-cap}>>
* <<data-frame-transform-resource,{dataframe-transforms-cap}>> * <<data-frame-transform-resource,{transforms-cap}>>
* <<ml-evaluate-dfanalytics-resources,Evaluate {dfanalytics}>> * <<ml-evaluate-dfanalytics-resources,Evaluate {dfanalytics}>>
* <<ml-job-resource,{anomaly-jobs-cap}>> * <<ml-job-resource,{anomaly-jobs-cap}>>
* <<ml-jobstats,{anomaly-jobs-cap} statistics>> * <<ml-jobstats,{anomaly-jobs-cap} statistics>>

View File

@ -14,7 +14,7 @@ not be included yet.
* <<cat, cat APIs>> * <<cat, cat APIs>>
* <<cluster, Cluster APIs>> * <<cluster, Cluster APIs>>
* <<ccr-apis,{ccr-cap} APIs>> * <<ccr-apis,{ccr-cap} APIs>>
* <<data-frame-apis,{dataframe-transform-cap} APIs>> * <<data-frame-apis,{transform-cap} APIs>>
* <<docs, Document APIs>> * <<docs, Document APIs>>
* <<graph-explore-api,Graph Explore API>> * <<graph-explore-api,Graph Explore API>>
* <<indices, Index APIs>> * <<indices, Index APIs>>

View File

@ -1,13 +1,13 @@
[role="xpack"] [role="xpack"]
[[data-frames-settings]] [[data-frames-settings]]
=== {dataframe-transforms-cap} settings in Elasticsearch === {transforms-cap} settings in Elasticsearch
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>{dataframe-transforms-cap} settings</titleabbrev> <titleabbrev>{transforms-cap} settings</titleabbrev>
++++ ++++
You do not need to configure any settings to use {dataframe-transforms}. It is enabled by default. You do not need to configure any settings to use {transforms}. It is enabled by default.
All of these settings can be added to the `elasticsearch.yml` configuration file. All of these settings can be added to the `elasticsearch.yml` configuration file.
The dynamic settings can also be updated across a cluster with the The dynamic settings can also be updated across a cluster with the
@ -18,23 +18,23 @@ file.
[float] [float]
[[general-data-frames-settings]] [[general-data-frames-settings]]
==== General {dataframe-transforms} settings ==== General {transforms} settings
`xpack.data_frame.enabled`:: `xpack.data_frame.enabled`::
Set to `true` (default) to enable {dataframe-transforms} on the node. + Set to `true` (default) to enable {transforms} on the node. +
+ +
If set to `false` in `elasticsearch.yml`, the {dataframe-transform} APIs are disabled on the node. If set to `false` in `elasticsearch.yml`, the {transform} APIs are disabled on the node.
Therefore the node cannot start or administrate transforms or receive transport (internal) Therefore the node cannot start or administrate {transform} or receive transport (internal)
communication requests related to {dataframe-transform} APIs. communication requests related to {transform} APIs.
+ +
IMPORTANT: If you want to use {dataframe-transform} features in your cluster, you must have IMPORTANT: If you want to use {transform} features in your cluster, you must have
`xpack.data_frame.enabled` set to `true` on all master-eligible nodes. This is the `xpack.data_frame.enabled` set to `true` on all master-eligible nodes. This is the
default behavior. default behavior.
`xpack.data_frame.num_transform_failure_retries` (<<cluster-update-settings,Dynamic>>):: `xpack.data_frame.num_transform_failure_retries` (<<cluster-update-settings,Dynamic>>)::
The number of times that a {dataframe-transform} retries when it experiences a The number of times that a {transform} retries when it experiences a
non-fatal error. Once the number of retries is exhausted, the {dataframe-transform} non-fatal error. Once the number of retries is exhausted, the {transform}
task will be marked as `failed`. The default value is `10` with a valid minimum of `0` task will be marked as `failed`. The default value is `10` with a valid minimum of `0`
and maximum of `100`. and maximum of `100`.
If a {dataframe-transform} is already running, it will have to be restarted If a {transform} is already running, it will have to be restarted
to use the changed setting. to use the changed setting.

View File

@ -2,7 +2,7 @@
[[df-api-quickref]] [[df-api-quickref]]
== API quick reference == API quick reference
All {dataframe-transform} endpoints have the following base: All {transform} endpoints have the following base:
[source,js] [source,js]
---- ----
@ -10,12 +10,12 @@ All {dataframe-transform} endpoints have the following base:
---- ----
// NOTCONSOLE // NOTCONSOLE
* {ref}/put-data-frame-transform.html[Create {dataframe-transforms}] * {ref}/put-data-frame-transform.html[Create {transforms}]
* {ref}/delete-data-frame-transform.html[Delete {dataframe-transforms}] * {ref}/delete-data-frame-transform.html[Delete {transforms}]
* {ref}/get-data-frame-transform.html[Get {dataframe-transforms}] * {ref}/get-data-frame-transform.html[Get {transforms}]
* {ref}/get-data-frame-transform-stats.html[Get {dataframe-transforms} statistics] * {ref}/get-data-frame-transform-stats.html[Get {transforms} statistics]
* {ref}/preview-data-frame-transform.html[Preview {dataframe-transforms}] * {ref}/preview-data-frame-transform.html[Preview {transforms}]
* {ref}/start-data-frame-transform.html[Start {dataframe-transforms}] * {ref}/start-data-frame-transform.html[Start {transforms}]
* {ref}/stop-data-frame-transform.html[Stop {dataframe-transforms}] * {ref}/stop-data-frame-transform.html[Stop {transforms}]
For the full list, see {ref}/data-frame-apis.html[{dataframe-transform-cap} APIs]. For the full list, see {ref}/data-frame-apis.html[{transform-cap} APIs].

View File

@ -1,24 +1,24 @@
[role="xpack"] [role="xpack"]
[[ml-transform-checkpoints]] [[ml-transform-checkpoints]]
== How {dataframe-transform} checkpoints work == How {transform} checkpoints work
++++ ++++
<titleabbrev>How checkpoints work</titleabbrev> <titleabbrev>How checkpoints work</titleabbrev>
++++ ++++
beta[] beta[]
Each time a {dataframe-transform} examines the source indices and creates or Each time a {transform} examines the source indices and creates or
updates the destination index, it generates a _checkpoint_. updates the destination index, it generates a _checkpoint_.
If your {dataframe-transform} runs only once, there is logically only one If your {transform} runs only once, there is logically only one
checkpoint. If your {dataframe-transform} runs continuously, however, it creates checkpoint. If your {transform} runs continuously, however, it creates
checkpoints as it ingests and transforms new source data. checkpoints as it ingests and transforms new source data.
To create a checkpoint, the {cdataframe-transform}: To create a checkpoint, the {ctransform}:
. Checks for changes to source indices. . Checks for changes to source indices.
+ +
Using a simple periodic timer, the {dataframe-transform} checks for changes to Using a simple periodic timer, the {transform} checks for changes to
the source indices. This check is done based on the interval defined in the the source indices. This check is done based on the interval defined in the
transform's `frequency` property. transform's `frequency` property.
+ +
@ -27,32 +27,32 @@ then it waits for the next timer.
. Identifies which entities have changed. . Identifies which entities have changed.
+ +
The {dataframe-transform} searches to see which entities have changed since the The {transform} searches to see which entities have changed since the
last time it checked. The transform's `sync` configuration object identifies a last time it checked. The `sync` configuration object in the {transform}
time field in the source indices. The transform uses the values in that field to identifies a time field in the source indices. The {transform} uses the values
synchronize the source and destination indices. in that field to synchronize the source and destination indices.
. Updates the destination index (the {dataframe}) with the changed entities. . Updates the destination index (the {dataframe}) with the changed entities.
+ +
-- --
The {dataframe-transform} applies changes related to either new or changed The {transform} applies changes related to either new or changed
entities to the destination index. The set of changed entities is paginated. For entities to the destination index. The set of changed entities is paginated. For
each page, the {dataframe-transform} performs a composite aggregation using a each page, the {transform} performs a composite aggregation using a
`terms` query. After all the pages of changes have been applied, the checkpoint `terms` query. After all the pages of changes have been applied, the checkpoint
is complete. is complete.
-- --
This checkpoint process involves both search and indexing activity on the This checkpoint process involves both search and indexing activity on the
cluster. We have attempted to favor control over performance while developing cluster. We have attempted to favor control over performance while developing
{dataframe-transforms}. We decided it was preferable for the {transforms}. We decided it was preferable for the
{dataframe-transform} to take longer to complete, rather than to finish quickly {transform} to take longer to complete, rather than to finish quickly
and take precedence in resource consumption. That being said, the cluster still and take precedence in resource consumption. That being said, the cluster still
requires enough resources to support both the composite aggregation search and requires enough resources to support both the composite aggregation search and
the indexing of its results. the indexing of its results.
TIP: If the cluster experiences unsuitable performance degradation due to the TIP: If the cluster experiences unsuitable performance degradation due to the
{dataframe-transform}, stop the transform. Consider whether you can apply a {transform}, stop the {transform}. Consider whether you can apply a
source query to the {dataframe-transform} to reduce the scope of data it source query to the {transform} to reduce the scope of data it
processes. Also consider whether the cluster has sufficient resources in place processes. Also consider whether the cluster has sufficient resources in place
to support both the composite aggregation search and the indexing of its to support both the composite aggregation search and the indexing of its
results. results.
@ -61,28 +61,28 @@ results.
[[ml-transform-checkpoint-errors]] [[ml-transform-checkpoint-errors]]
==== Error handling ==== Error handling
Failures in {dataframe-transforms} tend to be related to searching or indexing. Failures in {transforms} tend to be related to searching or indexing.
To increase the resiliency of {dataframe-transforms}, the cursor positions of To increase the resiliency of {transforms}, the cursor positions of
the aggregated search and the changed entities search are tracked in memory and the aggregated search and the changed entities search are tracked in memory and
persisted periodically. persisted periodically.
Checkpoint failures can be categorized as follows: Checkpoint failures can be categorized as follows:
* Temporary failures: The checkpoint is retried. If 10 consecutive failures * Temporary failures: The checkpoint is retried. If 10 consecutive failures
occur, the {dataframe-transform} has a failed status. For example, this occur, the {transform} has a failed status. For example, this
situation might occur when there are shard failures and queries return only situation might occur when there are shard failures and queries return only
partial results. partial results.
* Irrecoverable failures: The {dataframe-transform} immediately fails. For * Irrecoverable failures: The {transform} immediately fails. For
example, this situation occurs when the source index is not found. example, this situation occurs when the source index is not found.
* Adjustment failures: The {dataframe-transform} retries with adjusted settings. * Adjustment failures: The {transform} retries with adjusted settings.
For example, if a parent circuit breaker memory errors occur during the For example, if a parent circuit breaker memory errors occur during the
composite aggregation, the transform receives partial results. The aggregated composite aggregation, the {transform} receives partial results. The aggregated
search is retried with a smaller number of buckets. This retry is performed at search is retried with a smaller number of buckets. This retry is performed at
the interval defined in the transform's `frequency` property. If the search the interval defined in the `frequency` property for the {transform}. If the
is retried to the point where it reaches a minimal number of buckets, an search is retried to the point where it reaches a minimal number of buckets, an
irrecoverable failure occurs. irrecoverable failure occurs.
If the node running the {dataframe-transforms} fails, the transform restarts If the node running the {transforms} fails, the {transform} restarts
from the most recent persisted cursor position. This recovery process might from the most recent persisted cursor position. This recovery process might
repeat some of the work the transform had already done, but it ensures data repeat some of the work the {transform} had already done, but it ensures data
consistency. consistency.

View File

@ -1,14 +1,14 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[dataframe-examples]] [[dataframe-examples]]
== {dataframe-transform-cap} examples == {transform-cap} examples
++++ ++++
<titleabbrev>Examples</titleabbrev> <titleabbrev>Examples</titleabbrev>
++++ ++++
beta[] beta[]
These examples demonstrate how to use {dataframe-transforms} to derive useful These examples demonstrate how to use {transforms} to derive useful
insights from your data. All the examples use one of the insights from your data. All the examples use one of the
{kibana-ref}/add-sample-data.html[{kib} sample datasets]. For a more detailed, {kibana-ref}/add-sample-data.html[{kib} sample datasets]. For a more detailed,
step-by-step example, see step-by-step example, see
@ -67,7 +67,7 @@ gives more context to the final results.
NOTE: In the example above, condensed JSON formatting has been used for easier NOTE: In the example above, condensed JSON formatting has been used for easier
readability of the pivot object. readability of the pivot object.
The preview {dataframe-transforms} API enables you to see the layout of the The preview {transforms} API enables you to see the layout of the
{dataframe} in advance, populated with some sample values. For example: {dataframe} in advance, populated with some sample values. For example:
[source,js] [source,js]
@ -195,7 +195,7 @@ or flight stats for any of the featured destination or origin airports.
[[example-clientips]] [[example-clientips]]
=== Finding suspicious client IPs by using scripted metrics === Finding suspicious client IPs by using scripted metrics
With {dataframe-transforms}, you can use With {transforms}, you can use
{ref}/search-aggregations-metrics-scripted-metric-aggregation.html[scripted {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[scripted
metric aggregations] on your data. These aggregations are flexible and make metric aggregations] on your data. These aggregations are flexible and make
it possible to perform very complex processing. Let's use scripted metrics to it possible to perform very complex processing. Let's use scripted metrics to
@ -278,8 +278,8 @@ POST _data_frame/transforms/_preview
---------------------------------- ----------------------------------
// TEST[skip:setup kibana sample data] // TEST[skip:setup kibana sample data]
<1> This range query limits the transform to documents that are within the last <1> This range query limits the {transform} to documents that are within the last
30 days at the point in time the {dataframe-transform} checkpoint is processed. 30 days at the point in time the {transform} checkpoint is processed.
For batch {dataframes} this occurs once. For batch {dataframes} this occurs once.
<2> This is the destination index for the {dataframe}. It is ignored by <2> This is the destination index for the {dataframe}. It is ignored by
`_preview`. `_preview`.

View File

@ -5,10 +5,10 @@
beta[] beta[]
<<ml-dataframes,{dataframe-transforms-cap}>> enable you to retrieve information <<ml-dataframes,{transforms-cap}>> enable you to retrieve information
from an {es} index, transform it, and store it in another index. Let's use the from an {es} index, transform it, and store it in another index. Let's use the
{kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can {kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can
pivot and summarize your data with {dataframe-transforms}. pivot and summarize your data with {transforms}.
. If the {es} {security-features} are enabled, obtain a user ID with sufficient . If the {es} {security-features} are enabled, obtain a user ID with sufficient
@ -16,7 +16,7 @@ privileges to complete these steps.
+ +
-- --
You need `manage_data_frame_transforms` cluster privileges to preview and create You need `manage_data_frame_transforms` cluster privileges to preview and create
{dataframe-transforms}. Members of the built-in `data_frame_transforms_admin` {transforms}. Members of the built-in `data_frame_transforms_admin`
role have these privileges. role have these privileges.
You also need `read` and `view_index_metadata` index privileges on the source You also need `read` and `view_index_metadata` index privileges on the source
@ -51,10 +51,10 @@ at least one aggregation. You can preview what the transformed data will look
like, so go ahead and play with it! like, so go ahead and play with it!
For example, go to *Machine Learning* > *Data Frames* in {kib} and use the For example, go to *Machine Learning* > *Data Frames* in {kib} and use the
wizard to create a {dataframe-transform}: wizard to create a {transform}:
[role="screenshot"] [role="screenshot"]
image::images/ecommerce-pivot1.jpg["Creating a simple {dataframe-transform} in {kib}"] image::images/ecommerce-pivot1.jpg["Creating a simple {transform} in {kib}"]
In this case, we grouped the data by customer ID and calculated the sum of In this case, we grouped the data by customer ID and calculated the sum of
products each customer purchased. products each customer purchased.
@ -71,7 +71,7 @@ We'll accomplish this by using the
on the `order_id` field: on the `order_id` field:
[role="screenshot"] [role="screenshot"]
image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {dataframe-transform} in {kib}"] image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {transform} in {kib}"]
TIP: If you're interested in a subset of the data, you can optionally include a TIP: If you're interested in a subset of the data, you can optionally include a
{ref}/search-request-body.html#request-body-search-query[query] element. In this {ref}/search-request-body.html#request-body-search-query[query] element. In this
@ -81,7 +81,7 @@ If you want to use more complex queries, you can create your {dataframe} from a
{kibana-ref}/save-open-search.html[saved search]. {kibana-ref}/save-open-search.html[saved search].
If you prefer, you can use the If you prefer, you can use the
{ref}/preview-data-frame-transform.html[preview {dataframe-transforms} API]: {ref}/preview-data-frame-transform.html[preview {transforms} API]:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -134,28 +134,28 @@ POST _data_frame/transforms/_preview
-- --
. When you are satisfied with what you see in the preview, create the . When you are satisfied with what you see in the preview, create the
{dataframe-transform}. {transform}.
+ +
-- --
.. Supply a job ID and the name of the target (or _destination_) index. .. Supply a job ID and the name of the target (or _destination_) index.
.. Decide whether you want the {dataframe-transform} to run once or continuously. .. Decide whether you want the {transform} to run once or continuously.
-- --
+ +
-- --
Since this sample data index is unchanging, let's use the default behavior and Since this sample data index is unchanging, let's use the default behavior and
just run the {dataframe-transform} once. just run the {transform} once.
[role="screenshot"] [role="screenshot"]
image::images/ecommerce-batch.jpg["Specifying the {dataframe-transform} options in {kib}"] image::images/ecommerce-batch.jpg["Specifying the {transform} options in {kib}"]
If you want to try it out, however, go ahead and click on *Continuous mode*. If you want to try it out, however, go ahead and click on *Continuous mode*.
You must choose a field that the {dataframe-transform} can use to check which You must choose a field that the {transform} can use to check which
entities have changed. In general, it's a good idea to use the ingest timestamp entities have changed. In general, it's a good idea to use the ingest timestamp
field. In this example, however, you can use the `order_date` field. field. In this example, however, you can use the `order_date` field.
If you prefer, you can use the If you prefer, you can use the
{ref}/put-data-frame-transform.html[create {dataframe-transforms} API]. For {ref}/put-data-frame-transform.html[create {transforms} API]. For
example: example:
[source,console] [source,console]
@ -215,23 +215,23 @@ PUT _data_frame/transforms/ecommerce-customer-transform
// TEST[skip:setup kibana sample data] // TEST[skip:setup kibana sample data]
-- --
. Start the {dataframe-transform}. . Start the {transform}.
+ +
-- --
TIP: Even though resource utilization is automatically adjusted based on the TIP: Even though resource utilization is automatically adjusted based on the
cluster load, a {dataframe-transform} increases search and indexing load on your cluster load, a {transform} increases search and indexing load on your
cluster while it runs. If you're experiencing an excessive load, however, you cluster while it runs. If you're experiencing an excessive load, however, you
can stop it. can stop it.
You can start, stop, and manage {dataframe-transforms} in {kib}: You can start, stop, and manage {transforms} in {kib}:
[role="screenshot"] [role="screenshot"]
image::images/dataframe-transforms.jpg["Managing {dataframe-transforms} in {kib}"] image::images/dataframe-transforms.jpg["Managing {transforms} in {kib}"]
Alternatively, you can use the Alternatively, you can use the
{ref}/start-data-frame-transform.html[start {dataframe-transforms}] and {ref}/start-data-frame-transform.html[start {transforms}] and
{ref}/stop-data-frame-transform.html[stop {dataframe-transforms}] APIs. For {ref}/stop-data-frame-transform.html[stop {transforms}] APIs. For
example: example:
[source,console] [source,console]
@ -252,8 +252,8 @@ image::images/ecommerce-results.jpg["Exploring the new index in {kib}"]
-- --
TIP: If you do not want to keep the {dataframe-transform}, you can delete it in TIP: If you do not want to keep the {transform}, you can delete it in
{kib} or use the {kib} or use the
{ref}/delete-data-frame-transform.html[delete {dataframe-transform} API]. When {ref}/delete-data-frame-transform.html[delete {transform} API]. When
you delete a {dataframe-transform}, its destination index and {kib} index you delete a {transform}, its destination index and {kib} index
patterns remain. patterns remain.

View File

@ -1,6 +1,6 @@
[role="xpack"] [role="xpack"]
[[ml-dataframes]] [[ml-dataframes]]
= {dataframe-transforms-cap} = {transforms-cap}
[partintro] [partintro]
-- --
@ -65,7 +65,7 @@ create a {dataframe} to cache results. Thus, each user doesn't need to run the
aggregation query. aggregation query.
Though there are multiple ways to create {dataframes}, this content pertains Though there are multiple ways to create {dataframes}, this content pertains
to one specific method: _{dataframe-transforms}_. to one specific method: _{transforms}_.
* <<ml-transform-overview>> * <<ml-transform-overview>>
* <<df-api-quickref>> * <<df-api-quickref>>

View File

@ -1,6 +1,6 @@
[role="xpack"] [role="xpack"]
[[dataframe-limitations]] [[dataframe-limitations]]
== {dataframe-transform-cap} limitations == {transform-cap} limitations
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Limitations</titleabbrev> <titleabbrev>Limitations</titleabbrev>
@ -13,14 +13,14 @@ the Elastic {dataframe} feature:
[float] [float]
[[df-compatibility-limitations]] [[df-compatibility-limitations]]
=== Beta {dataframe-transforms} do not have guaranteed backwards or forwards compatibility === Beta {transforms} do not have guaranteed backwards or forwards compatibility
Whilst {dataframe-transforms} are beta, it is not guaranteed that a Whilst {transforms} are beta, it is not guaranteed that a
{dataframe-transform} created in a previous version of the {stack} will be able {transform} created in a previous version of the {stack} will be able
to start and operate in a future version. Neither can support be provided for to start and operate in a future version. Neither can support be provided for
{dataframe-transform} tasks to be able to operate in a cluster with mixed node {transform} tasks to be able to operate in a cluster with mixed node
versions. versions.
Please note that the output of a {dataframe-transform} is persisted to a Please note that the output of a {transform} is persisted to a
destination index. This is a normal {es} index and is not affected by the beta destination index. This is a normal {es} index and is not affected by the beta
status. status.
@ -29,7 +29,7 @@ status.
=== {dataframe-cap} UI will not work during a rolling upgrade from 7.2 === {dataframe-cap} UI will not work during a rolling upgrade from 7.2
If your cluster contains mixed version nodes, for example during a rolling If your cluster contains mixed version nodes, for example during a rolling
upgrade from 7.2 to a newer version, and {dataframe-transforms} have been upgrade from 7.2 to a newer version, and {transforms} have been
created in 7.2, the {dataframe} UI will not work. Please wait until all nodes created in 7.2, the {dataframe} UI will not work. Please wait until all nodes
have been upgraded to the newer version before using the {dataframe} UI. have been upgraded to the newer version before using the {dataframe} UI.
@ -46,23 +46,23 @@ table.
[[df-ccs-limitations]] [[df-ccs-limitations]]
=== {ccs-cap} is not supported === {ccs-cap} is not supported
{ccs-cap} is not supported for {dataframe-transforms}. {ccs-cap} is not supported for {transforms}.
[float] [float]
[[df-kibana-limitations]] [[df-kibana-limitations]]
=== Up to 1,000 {dataframe-transforms} are supported === Up to 1,000 {transforms} are supported
A single cluster will support up to 1,000 {dataframe-transforms}. A single cluster will support up to 1,000 {transforms}.
When using the When using the
{ref}/get-data-frame-transform.html[GET {dataframe-transforms} API] a total {ref}/get-data-frame-transform.html[GET {transforms} API] a total
`count` of transforms is returned. Use the `size` and `from` parameters to `count` of {transforms} is returned. Use the `size` and `from` parameters to
enumerate through the full list. enumerate through the full list.
[float] [float]
[[df-aggresponse-limitations]] [[df-aggresponse-limitations]]
=== Aggregation responses may be incompatible with destination index mappings === Aggregation responses may be incompatible with destination index mappings
When a {dataframe-transform} is first started, it will deduce the mappings When a {transform} is first started, it will deduce the mappings
required for the destination index. This process is based on the field types of required for the destination index. This process is based on the field types of
the source index and the aggregations used. If the fields are derived from the source index and the aggregations used. If the fields are derived from
{ref}/search-aggregations-metrics-scripted-metric-aggregation.html[`scripted_metrics`] {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[`scripted_metrics`]
@ -72,15 +72,15 @@ deduced mappings may be incompatible with the actual data. For example, numeric
overflows might occur or dynamically mapped fields might contain both numbers overflows might occur or dynamically mapped fields might contain both numbers
and strings. Please check {es} logs if you think this may have occurred. As a and strings. Please check {es} logs if you think this may have occurred. As a
workaround, you may define custom mappings prior to starting the workaround, you may define custom mappings prior to starting the
{dataframe-transform}. For example, {transform}. For example,
{ref}/indices-create-index.html[create a custom destination index] or {ref}/indices-create-index.html[create a custom destination index] or
{ref}/indices-templates.html[define an index template]. {ref}/indices-templates.html[define an index template].
[float] [float]
[[df-batch-limitations]] [[df-batch-limitations]]
=== Batch {dataframe-transforms} may not account for changed documents === Batch {transforms} may not account for changed documents
A batch {dataframe-transform} uses a A batch {transform} uses a
{ref}/search-aggregations-bucket-composite-aggregation.html[composite aggregation] {ref}/search-aggregations-bucket-composite-aggregation.html[composite aggregation]
which allows efficient pagination through all buckets. Composite aggregations which allows efficient pagination through all buckets. Composite aggregations
do not yet support a search context, therefore if the source data is changed do not yet support a search context, therefore if the source data is changed
@ -91,8 +91,8 @@ results may not include these changes.
[[df-consistency-limitations]] [[df-consistency-limitations]]
=== {cdataframe-cap} consistency does not account for deleted or updated documents === {cdataframe-cap} consistency does not account for deleted or updated documents
While the process for {cdataframe-transforms} allows the continual recalculation While the process for {transforms} allows the continual recalculation
of the {dataframe-transform} as new data is being ingested, it does also have of the {transform} as new data is being ingested, it does also have
some limitations. some limitations.
Changed entities will only be identified if their time field Changed entities will only be identified if their time field
@ -106,7 +106,7 @@ composite aggregation performed in consecutive checkpoint processing will search
over different source data, and entities that only existed in the deleted index over different source data, and entities that only existed in the deleted index
will not be removed from the {dataframe} destination index. will not be removed from the {dataframe} destination index.
Depending on your use case, you may wish to recreate the {dataframe-transform} Depending on your use case, you may wish to recreate the {transform}
entirely after deletions. Alternatively, if your use case is tolerant to entirely after deletions. Alternatively, if your use case is tolerant to
historical archiving, you may wish to include a max ingest timestamp in your historical archiving, you may wish to include a max ingest timestamp in your
aggregation. This will allow you to exclude results that have not been recently aggregation. This will allow you to exclude results that have not been recently
@ -115,9 +115,9 @@ updated when viewing the {dataframe} destination index.
[float] [float]
[[df-deletion-limitations]] [[df-deletion-limitations]]
=== Deleting a {dataframe-transform} does not delete the {dataframe} destination index or {kib} index pattern === Deleting a {transform} does not delete the {dataframe} destination index or {kib} index pattern
When deleting a {dataframe-transform} using `DELETE _data_frame/transforms/index` When deleting a {transform} using `DELETE _data_frame/transforms/index`
neither the {dataframe} destination index nor the {kib} index pattern, should neither the {dataframe} destination index nor the {kib} index pattern, should
one have been created, are deleted. These objects must be deleted separately. one have been created, are deleted. These objects must be deleted separately.
@ -125,9 +125,9 @@ one have been created, are deleted. These objects must be deleted separately.
[[df-aggregation-page-limitations]] [[df-aggregation-page-limitations]]
=== Handling dynamic adjustment of aggregation page size === Handling dynamic adjustment of aggregation page size
During the development of {dataframe-transforms}, control was favoured over During the development of {transforms}, control was favoured over
performance. In the design considerations, it is preferred for the performance. In the design considerations, it is preferred for the
{dataframe-transform} to take longer to complete quietly in the background {transform} to take longer to complete quietly in the background
rather than to finish quickly and take precedence in resource consumption. rather than to finish quickly and take precedence in resource consumption.
Composite aggregations are well suited for high cardinality data enabling Composite aggregations are well suited for high cardinality data enabling
@ -135,21 +135,21 @@ pagination through results. If a {ref}/circuit-breaker.html[circuit breaker]
memory exception occurs when performing the composite aggregated search then we memory exception occurs when performing the composite aggregated search then we
try again reducing the number of buckets requested. This circuit breaker is try again reducing the number of buckets requested. This circuit breaker is
calculated based upon all activity within the cluster, not just activity from calculated based upon all activity within the cluster, not just activity from
{dataframe-transforms}, so it therefore may only be a temporary resource {transforms}, so it therefore may only be a temporary resource
availability issue. availability issue.
For a batch {dataframe-transform}, the number of buckets requested is only ever For a batch {transform}, the number of buckets requested is only ever
adjusted downwards. The lowering of value may result in a longer duration for the adjusted downwards. The lowering of value may result in a longer duration for the
transform checkpoint to complete. For {cdataframes}, the number of {transform} checkpoint to complete. For {cdataframes}, the number of
buckets requested is reset back to its default at the start of every checkpoint buckets requested is reset back to its default at the start of every checkpoint
and it is possible for circuit breaker exceptions to occur repeatedly in the and it is possible for circuit breaker exceptions to occur repeatedly in the
{es} logs. {es} logs.
The {dataframe-transform} retrieves data in batches which means it calculates The {transform} retrieves data in batches which means it calculates
several buckets at once. Per default this is 500 buckets per search/index several buckets at once. Per default this is 500 buckets per search/index
operation. The default can be changed using `max_page_search_size` and the operation. The default can be changed using `max_page_search_size` and the
minimum value is 10. If failures still occur once the number of buckets minimum value is 10. If failures still occur once the number of buckets
requested has been reduced to its minimum, then the {dataframe-transform} will requested has been reduced to its minimum, then the {transform} will
be set to a failed state. be set to a failed state.
[float] [float]
@ -158,7 +158,7 @@ be set to a failed state.
For each checkpoint, entities are identified that have changed since the last For each checkpoint, entities are identified that have changed since the last
time the check was performed. This list of changed entities is supplied as a time the check was performed. This list of changed entities is supplied as a
{ref}/query-dsl-terms-query.html[terms query] to the {dataframe-transform} {ref}/query-dsl-terms-query.html[terms query] to the {transform}
composite aggregation, one page at a time. Then updates are applied to the composite aggregation, one page at a time. Then updates are applied to the
destination index for each page of entities. destination index for each page of entities.
@ -170,10 +170,10 @@ The index setting
{ref}/index-modules.html#dynamic-index-settings[`index.max_terms_count`] defines {ref}/index-modules.html#dynamic-index-settings[`index.max_terms_count`] defines
the maximum number of terms that can be used in a terms query. The default value the maximum number of terms that can be used in a terms query. The default value
is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the
transform will fail. {transform} will fail.
Using smaller values for `max_page_search_size` may result in a longer duration Using smaller values for `max_page_search_size` may result in a longer duration
for the transform checkpoint to complete. for the {transform} checkpoint to complete.
[float] [float]
[[df-scheduling-limitations]] [[df-scheduling-limitations]]
@ -183,22 +183,22 @@ A {cdataframe} periodically checks for changes to source data. The functionality
of the scheduler is currently limited to a basic periodic timer which can be of the scheduler is currently limited to a basic periodic timer which can be
within the `frequency` range from 1s to 1h. The default is 1m. This is designed within the `frequency` range from 1s to 1h. The default is 1m. This is designed
to run little and often. When choosing a `frequency` for this timer consider to run little and often. When choosing a `frequency` for this timer consider
your ingest rate along with the impact that the {dataframe-transform} your ingest rate along with the impact that the {transform}
search/index operations has other users in your cluster. Also note that retries search/index operations has other users in your cluster. Also note that retries
occur at `frequency` interval. occur at `frequency` interval.
[float] [float]
[[df-failed-limitations]] [[df-failed-limitations]]
=== Handling of failed {dataframe-transforms} === Handling of failed {transforms}
Failed {dataframe-transforms} remain as a persistent task and should be handled Failed {transforms} remain as a persistent task and should be handled
appropriately, either by deleting it or by resolving the root cause of the appropriately, either by deleting it or by resolving the root cause of the
failure and re-starting. failure and re-starting.
When using the API to delete a failed {dataframe-transform}, first stop it using When using the API to delete a failed {transform}, first stop it using
`_stop?force=true`, then delete it. `_stop?force=true`, then delete it.
If starting a failed {dataframe-transform}, after the root cause has been If starting a failed {transform}, after the root cause has been
resolved, the `_start?force=true` parameter must be specified. resolved, the `_start?force=true` parameter must be specified.
[float] [float]
@ -208,7 +208,7 @@ resolved, the `_start?force=true` parameter must be specified.
After a document is indexed, there is a very small delay until it is available After a document is indexed, there is a very small delay until it is available
to search. to search.
A {cdataframe-transform} periodically checks for changed entities between the A {ctransform} periodically checks for changed entities between the
time since it last checked and `now` minus `sync.time.delay`. This time window time since it last checked and `now` minus `sync.time.delay`. This time window
moves without overlapping. If the timestamp of a recently indexed document falls moves without overlapping. If the timestamp of a recently indexed document falls
within this time window but this document is not yet available to search then within this time window but this document is not yet available to search then

View File

@ -1,6 +1,6 @@
[role="xpack"] [role="xpack"]
[[ml-transform-overview]] [[ml-transform-overview]]
== {dataframe-transform-cap} overview == {transform-cap} overview
++++ ++++
<titleabbrev>Overview</titleabbrev> <titleabbrev>Overview</titleabbrev>
++++ ++++
@ -19,8 +19,8 @@ you to summarize this data, bringing it into an organized, more
analysis-friendly format. For example, you can summarize all the purchases of a analysis-friendly format. For example, you can summarize all the purchases of a
single customer. single customer.
You can create {dataframes} by using {dataframe-transforms}. You can create {dataframes} by using {transforms}.
{dataframe-transforms-cap} enable you to define a pivot, which is a set of {transforms-cap} enable you to define a pivot, which is a set of
features that transform the index into a different, more digestible format. features that transform the index into a different, more digestible format.
Pivoting results in a summary of your data, which is the {dataframe}. Pivoting results in a summary of your data, which is the {dataframe}.
@ -33,19 +33,19 @@ The second step is deciding how you want to aggregate the grouped data. When
using aggregations, you practically ask questions about the index. There are using aggregations, you practically ask questions about the index. There are
different types of aggregations, each with its own purpose and output. To learn different types of aggregations, each with its own purpose and output. To learn
more about the supported aggregations and group-by fields, see more about the supported aggregations and group-by fields, see
{ref}/data-frame-transform-resource.html[{dataframe-transform-cap} resources]. {ref}/data-frame-transform-resource.html[{transform-cap} resources].
As an optional step, you can also add a query to further limit the scope of the As an optional step, you can also add a query to further limit the scope of the
aggregation. aggregation.
The {dataframe-transform} performs a composite aggregation that The {transform} performs a composite aggregation that
paginates through all the data defined by the source index query. The output of paginates through all the data defined by the source index query. The output of
the aggregation is stored in a destination index. Each time the the aggregation is stored in a destination index. Each time the
{dataframe-transform} queries the source index, it creates a _checkpoint_. You {transform} queries the source index, it creates a _checkpoint_. You
can decide whether you want the {dataframe-transform} to run once (batch can decide whether you want the {transform} to run once (batch
{dataframe-transform}) or continuously ({cdataframe-transform}). A batch {transform}) or continuously ({transform}). A batch
{dataframe-transform} is a single operation that has a single checkpoint. {transform} is a single operation that has a single checkpoint.
{cdataframe-transforms-cap} continually increment and process checkpoints as new {ctransforms-cap} continually increment and process checkpoints as new
source data is ingested. source data is ingested.
.Example .Example
@ -57,7 +57,7 @@ information (name, gender, location, etc). Your dataset contains all the transac
from last year. from last year.
If you want to check the sales in the different categories in your last fiscal If you want to check the sales in the different categories in your last fiscal
year, define a {dataframe-transform} that groups the data by the product year, define a {transform} that groups the data by the product
categories (women's shoes, men's clothing, etc.) and the order date. Use the categories (women's shoes, men's clothing, etc.) and the order date. Use the
last year as the interval for the order date. Then add a sum aggregation on the last year as the interval for the order date. Then add a sum aggregation on the
ordered quantity. The result is a {dataframe} that shows the number of sold ordered quantity. The result is a {dataframe} that shows the number of sold
@ -66,6 +66,6 @@ items in every product category in the last year.
[role="screenshot"] [role="screenshot"]
image::images/ml-dataframepivot.jpg["Example of a data frame pivot in {kib}"] image::images/ml-dataframepivot.jpg["Example of a data frame pivot in {kib}"]
IMPORTANT: The {dataframe-transform} leaves your source index intact. It IMPORTANT: The {transform} leaves your source index intact. It
creates a new index that is dedicated to the {dataframe}. creates a new index that is dedicated to the {dataframe}.

View File

@ -1,5 +1,5 @@
[[dataframe-troubleshooting]] [[dataframe-troubleshooting]]
== Troubleshooting {dataframe-transforms} == Troubleshooting {transforms}
[subs="attributes"] [subs="attributes"]
++++ ++++
<titleabbrev>Troubleshooting</titleabbrev> <titleabbrev>Troubleshooting</titleabbrev>
@ -9,21 +9,21 @@ Use the information in this section to troubleshoot common problems.
include::{stack-repo-dir}/help.asciidoc[tag=get-help] include::{stack-repo-dir}/help.asciidoc[tag=get-help]
If you encounter problems with your {dataframe-transforms}, you can gather more If you encounter problems with your {transforms}, you can gather more
information from the following files and APIs: information from the following files and APIs:
* Lightweight audit messages are stored in `.data-frame-notifications-*`. Search * Lightweight audit messages are stored in `.data-frame-notifications-*`. Search
by your `transform_id`. by your `transform_id`.
* The * The
{ref}/get-data-frame-transform-stats.html[get {dataframe-transform} statistics API] {ref}/get-data-frame-transform-stats.html[get {transform} statistics API]
provides information about the transform status and failures. provides information about the {transform} status and failures.
* If the {dataframe-transform} exists as a task, you can use the * If the {transform} exists as a task, you can use the
{ref}/tasks.html[task management API] to gather task information. For example: {ref}/tasks.html[task management API] to gather task information. For example:
`GET _tasks?actions=data_frame/transforms*&detailed`. Typically, the task exists `GET _tasks?actions=data_frame/transforms*&detailed`. Typically, the task exists
when the transform is in a started or failed state. when the {transform} is in a started or failed state.
* The {es} logs from the node that was running the {dataframe-transform} might * The {es} logs from the node that was running the {transform} might
also contain useful information. You can identify the node from the notification also contain useful information. You can identify the node from the notification
messages. Alternatively, if the task still exists, you can get that information messages. Alternatively, if the task still exists, you can get that information
from the get {dataframe-transform} statistics API. For more information, see from the get {transform} statistics API. For more information, see
{ref}/logging.html[Logging configuration]. {ref}/logging.html[Logging configuration].