diff --git a/docs/java-rest/high-level/dataframe/delete_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/delete_data_frame.asciidoc index 480fd4f41f2..d4e2f93df18 100644 --- a/docs/java-rest/high-level/dataframe/delete_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/delete_data_frame.asciidoc @@ -5,12 +5,12 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Delete {dataframe-transform} API +=== Delete {transform} API -Deletes an existing {dataframe-transform}. +Deletes an existing {transform}. [id="{upid}-{api}-request"] -==== Delete {dataframe-transform} request +==== Delete {transform} request A +{request}+ object requires a non-null `id`. @@ -18,14 +18,14 @@ A +{request}+ object requires a non-null `id`. --------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] --------------------------------------------------- -<1> Constructing a new request referencing an existing {dataframe-transform} -<2> Sets the optional argument `force`. When `true`, the {dataframe-transform} +<1> Constructing a new request referencing an existing {transform} +<2> Sets the optional argument `force`. When `true`, the {transform} is deleted regardless of its current state. The default value is `false`, -meaning that only `stopped` {dataframe-transforms} can be deleted. +meaning that only `stopped` {transforms} can be deleted. include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ object acknowledges the Data Frame Transform deletion. +The returned +{response}+ object acknowledges the {transform} deletion. diff --git a/docs/java-rest/high-level/dataframe/get_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/get_data_frame.asciidoc index e3a4b4bb609..e36ff37a2d2 100644 --- a/docs/java-rest/high-level/dataframe/get_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/get_data_frame.asciidoc @@ -5,22 +5,22 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Get {dataframe-transform} API +=== Get {transform} API -Retrieves configuration information about one or more {dataframe-transforms}. +Retrieves configuration information about one or more {transforms}. The API accepts a +{request}+ object and returns a +{response}+. [id="{upid}-{api}-request"] -==== Get {dataframe-transform} request +==== Get {transform} request -A +{request}+ requires either a {dataframe-transform} ID, a comma separated list -of ids or the special wildcard `_all` to get all {dataframe-transforms}. +A +{request}+ requires either a {transform} ID, a comma separated list +of ids or the special wildcard `_all` to get all {transforms}. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- -<1> Constructing a new GET request referencing an existing {dataframe-transform} +<1> Constructing a new GET request referencing an existing {transform} ==== Optional arguments @@ -31,9 +31,9 @@ The following arguments are optional. include-tagged::{doc-tests-file}[{api}-request-options] -------------------------------------------------- <1> The page parameters `from` and `size`. `from` specifies the number of -{dataframe-transforms} to skip. `size` specifies the maximum number of -{dataframe-transforms} to get. Defaults to `0` and `100` respectively. -<2> Whether to ignore if a wildcard expression matches no transforms. +{transforms} to skip. `size` specifies the maximum number of +{transforms} to get. Defaults to `0` and `100` respectively. +<2> Whether to ignore if a wildcard expression matches no {transforms}. include::../execution.asciidoc[] @@ -41,7 +41,7 @@ include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ contains the requested {dataframe-transforms}. +The returned +{response}+ contains the requested {transforms}. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- diff --git a/docs/java-rest/high-level/dataframe/get_data_frame_stats.asciidoc b/docs/java-rest/high-level/dataframe/get_data_frame_stats.asciidoc index 0c0ef02a4e4..76b06f99eb1 100644 --- a/docs/java-rest/high-level/dataframe/get_data_frame_stats.asciidoc +++ b/docs/java-rest/high-level/dataframe/get_data_frame_stats.asciidoc @@ -5,22 +5,22 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Get {dataframe-transform} stats API +=== Get {transform} stats API -Retrieves the operational statistics of one or more {dataframe-transforms}. +Retrieves the operational statistics of one or more {transforms}. The API accepts a +{request}+ object and returns a +{response}+. [id="{upid}-{api}-request"] -==== Get {dataframe-transform} stats request +==== Get {transform} stats request -A +{request}+ requires a data frame transform id or the special wildcard `_all` -to get the statistics for all {dataframe-transforms}. +A +{request}+ requires a {transform} id or the special wildcard `_all` +to get the statistics for all {transforms}. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- -<1> Constructing a new GET Stats request referencing an existing {dataframe-transform} +<1> Constructing a new GET Stats request referencing an existing {transform} ==== Optional arguments @@ -31,10 +31,10 @@ The following arguments are optional. include-tagged::{doc-tests-file}[{api}-request-options] -------------------------------------------------- <1> The page parameters `from` and `size`. `from` specifies the number of -{dataframe-transform} stats to skip. -`size` specifies the maximum number of {dataframe-transform} stats to get. +{transform} stats to skip. +`size` specifies the maximum number of {transform} stats to get. Defaults to `0` and `100` respectively. -<2> Whether to ignore if a wildcard expression matches no transforms. +<2> Whether to ignore if a wildcard expression matches no {transforms}. include::../execution.asciidoc[] @@ -42,15 +42,15 @@ include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ contains the requested {dataframe-transform} statistics. +The returned +{response}+ contains the requested {transform} statistics. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-response] -------------------------------------------------- <1> The response contains a list of `DataFrameTransformStats` objects -<2> The running state of the transform, for example `started`, `indexing`, etc. -<3> The overall transform statistics recording the number of documents indexed etc. -<4> The progress of the current run in the transform. Supplies the number of docs left until the next checkpoint +<2> The running state of the {transform}, for example `started`, `indexing`, etc. +<3> The overall {transform} statistics recording the number of documents indexed etc. +<4> The progress of the current run in the {transform}. Supplies the number of docs left until the next checkpoint and the total number of docs expected. <5> The assigned node information if the task is currently assigned to a node and running. diff --git a/docs/java-rest/high-level/dataframe/preview_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/preview_data_frame.asciidoc index 057f38a19ef..d2c1233dac0 100644 --- a/docs/java-rest/high-level/dataframe/preview_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/preview_data_frame.asciidoc @@ -5,16 +5,16 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Preview {dataframe-transform} API +=== Preview {transform} API -Previews the results of a {dataframe-transform}. +Previews the results of a {transform}. The API accepts a +{request}+ object as a request and returns a +{response}+. [id="{upid}-{api}-request"] -==== Preview {dataframe-transform} request +==== Preview {transform} request -A +{request}+ takes a single argument: a valid {dataframe-transform} config. +A +{request}+ takes a single argument: a valid {transform} config. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- @@ -22,7 +22,7 @@ include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- <1> The source config from which the data should be gathered <2> The pivot config used to transform the data -<3> The configuration of the {dataframe-transform} to preview +<3> The configuration of the {transform} to preview include::../execution.asciidoc[] diff --git a/docs/java-rest/high-level/dataframe/put_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/put_data_frame.asciidoc index aa27124df51..c6983414484 100644 --- a/docs/java-rest/high-level/dataframe/put_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/put_data_frame.asciidoc @@ -5,14 +5,14 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Put {dataframe-transform} API +=== Put {transform} API -Creates a new {dataframe-transform}. +Creates a new {transform}. The API accepts a +{request}+ object as a request and returns a +{response}+. [id="{upid}-{api}-request"] -==== Put {dataframe-transform} request +==== Put {transform} request A +{request}+ requires the following argument: @@ -20,28 +20,28 @@ A +{request}+ requires the following argument: -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- -<1> The configuration of the {dataframe-transform} to create +<1> The configuration of the {transform} to create <2> Whether or not to wait to run deferrable validations until `_start` is called. -This option should be used with care as the created {dataframe-transform} will run +This option should be used with care as the created {transform} will run with the privileges of the user creating it. Meaning, if they do not have privileges, such an error will not be visible until `_start` is called. [id="{upid}-{api}-config"] -==== {dataframe-transform-cap} configuration +==== {transform-cap} configuration The `DataFrameTransformConfig` object contains all the details about the -{dataframe-transform} configuration and contains the following arguments: +{transform} configuration and contains the following arguments: ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-config] -------------------------------------------------- -<1> The {dataframe-transform} ID +<1> The {transform} ID <2> The source indices and query from which to gather data <3> The destination index and optional pipeline <4> How often to check for updates to the source indices <5> The PivotConfig -<6> Optional free text description of the transform +<6> Optional free text description of the {transform} [id="{upid}-{api}-query-config"] @@ -84,8 +84,8 @@ include-tagged::{doc-tests-file}[{api}-pivot-config] -------------------------------------------------- <1> The `GroupConfig` to use in the pivot <2> The aggregations to use -<3> The maximum paging size for the transform when pulling data -from the source. The size dynamically adjusts as the transform +<3> The maximum paging size for the {transform} when pulling data +from the source. The size dynamically adjusts as the {transform} is running to recover from and prevent OOM issues. ===== GroupConfig @@ -121,4 +121,4 @@ include::../execution.asciidoc[] ==== Response The returned +{response}+ acknowledges the successful creation of -the new {dataframe-transform} or an error if the configuration is invalid. +the new {transform} or an error if the configuration is invalid. diff --git a/docs/java-rest/high-level/dataframe/start_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/start_data_frame.asciidoc index 68f9c99b9b0..ef7d83815e5 100644 --- a/docs/java-rest/high-level/dataframe/start_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/start_data_frame.asciidoc @@ -5,13 +5,13 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Start {dataframe-transform} API +=== Start {transform} API -Starts a {dataframe-transform}. +Starts a {transform}. It accepts a +{request}+ object and responds with a +{response}+ object. [id="{upid}-{api}-request"] -==== Start {dataframe-transform} request +==== Start {transform} request A +{request}+ object requires a non-null `id`. @@ -20,7 +20,7 @@ A +{request}+ object requires a non-null `id`. include-tagged::{doc-tests-file}[{api}-request] --------------------------------------------------- <1> Constructing a new start request referencing an existing -{dataframe-transform} +{transform} ==== Optional arguments @@ -30,11 +30,11 @@ The following arguments are optional. -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request-options] -------------------------------------------------- -<1> Controls the amount of time to wait until the {dataframe-transform} starts. +<1> Controls the amount of time to wait until the {transform} starts. include::../execution.asciidoc[] ==== Response -The returned +{response}+ object acknowledges the {dataframe-transform} has +The returned +{response}+ object acknowledges the {transform} has started. \ No newline at end of file diff --git a/docs/java-rest/high-level/dataframe/stop_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/stop_data_frame.asciidoc index b5b066c573a..fe6dbee9c42 100644 --- a/docs/java-rest/high-level/dataframe/stop_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/stop_data_frame.asciidoc @@ -5,13 +5,13 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Stop {dataframe-transform} API +=== Stop {transform} API -Stops a started {dataframe-transform}. +Stops a started {transform}. It accepts a +{request}+ object and responds with a +{response}+ object. [id="{upid}-{api}-request"] -==== Stop {dataframe-transform} request +==== Stop {transform} request A +{request}+ object requires a non-null `id`. `id` can be a comma separated list of IDs or a single ID. Wildcards, `*` and `_all` are also accepted. @@ -21,7 +21,7 @@ list of IDs or a single ID. Wildcards, `*` and `_all` are also accepted. --------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] --------------------------------------------------- -<1> Constructing a new stop request referencing an existing {dataframe-transform} +<1> Constructing a new stop request referencing an existing {transform} ==== Optional arguments @@ -32,11 +32,11 @@ The following arguments are optional. include-tagged::{doc-tests-file}[{api}-request-options] -------------------------------------------------- <1> If true wait for the data frame task to stop before responding -<2> Controls the amount of time to wait until the {dataframe-transform} stops. -<3> Whether to ignore if a wildcard expression matches no transforms. +<2> Controls the amount of time to wait until the {transform} stops. +<3> Whether to ignore if a wildcard expression matches no {transforms}. include::../execution.asciidoc[] ==== Response -The returned +{response}+ object acknowledges the {dataframe-transform} has stopped. \ No newline at end of file +The returned +{response}+ object acknowledges the {transform} has stopped. \ No newline at end of file diff --git a/docs/java-rest/high-level/dataframe/update_data_frame.asciidoc b/docs/java-rest/high-level/dataframe/update_data_frame.asciidoc index 17fd9a9944c..e1313e08a69 100644 --- a/docs/java-rest/high-level/dataframe/update_data_frame.asciidoc +++ b/docs/java-rest/high-level/dataframe/update_data_frame.asciidoc @@ -5,14 +5,14 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Update {dataframe-transform} API +=== Update {transform} API -Updates an existing {dataframe-transform}. +Updates an existing {transform}. The API accepts a +{request}+ object as a request and returns a +{response}+. [id="{upid}-{api}-request"] -==== Update {dataframe-transform} request +==== Update {transform} request A +{request}+ requires the following argument: @@ -20,18 +20,18 @@ A +{request}+ requires the following argument: -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- -<1> The update configuration with which to update the {dataframe-transform}. +<1> The update configuration with which to update the {transform}. <2> The ID of the configuration to update. <3> Whether or not to wait to run deferrable validations until `_start` is called. -This option should be used with care as the created {dataframe-transform} will run +This option should be used with care as the created {transform} will run with the privileges of the user creating it. Meaning, if they do not have privileges, such an error will not be visible until `_start` is called. [id="{upid}-{api}-config"] -==== {dataframe-transform-cap} update configuration +==== {transform-cap} update configuration The `DataFrameTransformConfigUpdate` object contains all the details about updated -{dataframe-transform} configuration and contains the following arguments: +{transform} configuration and contains the following arguments: ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- @@ -40,13 +40,13 @@ include-tagged::{doc-tests-file}[{api}-config] <1> The source indices and query from which to gather data. <2> The destination index and optional pipeline. <3> How often to check for updates to the source indices. -<4> How to keep the {dataframe-transform} in sync with incoming data. -<5> Optional free text description of the transform. +<4> How to keep the {transform} in sync with incoming data. +<5> Optional free text description of the {transform}. include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ contains the updated {dataframe-transform} configuration +The returned +{response}+ contains the updated {transform} configuration or an error if the update failed or is invalid. diff --git a/docs/java-rest/high-level/ml/put-data-frame-analytics.asciidoc b/docs/java-rest/high-level/ml/put-data-frame-analytics.asciidoc index efddcd2a855..a4f0a66138a 100644 --- a/docs/java-rest/high-level/ml/put-data-frame-analytics.asciidoc +++ b/docs/java-rest/high-level/ml/put-data-frame-analytics.asciidoc @@ -5,13 +5,13 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Put Data Frame Analytics API +=== Put {dfanalytics} API -The Put Data Frame Analytics API is used to create a new {dataframe-analytics-config}. +The Put {dfanalytics} API is used to create a new {dataframe-analytics-config}. The API accepts a +{request}+ object as a request and returns a +{response}+. [id="{upid}-{api}-request"] -==== Put Data Frame Analytics Request +==== Put {dfanalytics} request A +{request}+ requires the following argument: @@ -19,12 +19,12 @@ A +{request}+ requires the following argument: -------------------------------------------------- include-tagged::{doc-tests-file}[{api}-request] -------------------------------------------------- -<1> The configuration of the {dataframe-job} to create +<1> The configuration of the {dfanalytics-job} to create [id="{upid}-{api}-config"] -==== Data Frame Analytics Configuration +==== {dfanalytics-cap} configuration -The `DataFrameAnalyticsConfig` object contains all the details about the {dataframe-job} +The `DataFrameAnalyticsConfig` object contains all the details about the {dfanalytics-job} configuration and contains the following arguments: ["source","java",subs="attributes,callouts,macros"] @@ -64,7 +64,7 @@ include-tagged::{doc-tests-file}[{api}-query-config] ==== DestinationConfig -The index to which data should be written by the {dataframe-job}. +The index to which data should be written by the {dfanalytics-job}. ["source","java",subs="attributes,callouts,macros"] -------------------------------------------------- @@ -78,7 +78,7 @@ include-tagged::{doc-tests-file}[{api}-dest-config] The analysis to be performed. Currently, the supported analyses include : +OutlierDetection+, +Regression+. -===== Outlier Detection +===== Outlier detection +OutlierDetection+ analysis can be created in one of two ways: diff --git a/docs/java-rest/high-level/ml/start-data-frame-analytics.asciidoc b/docs/java-rest/high-level/ml/start-data-frame-analytics.asciidoc index 79a239133ab..559c590f3b5 100644 --- a/docs/java-rest/high-level/ml/start-data-frame-analytics.asciidoc +++ b/docs/java-rest/high-level/ml/start-data-frame-analytics.asciidoc @@ -5,13 +5,13 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Start Data Frame Analytics API +=== Start {dfanalytics} API -The Start Data Frame Analytics API is used to start an existing {dataframe-analytics-config}. +The start {dfanalytics} API is used to start an existing {dataframe-analytics-config}. It accepts a +{request}+ object and responds with a +{response}+ object. [id="{upid}-{api}-request"] -==== Start Data Frame Analytics Request +==== Start {dfanalytics} Request A +{request}+ object requires a {dataframe-analytics-config} id. @@ -26,4 +26,4 @@ include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ object acknowledges the {dataframe-job} has started. \ No newline at end of file +The returned +{response}+ object acknowledges the {dfanalytics-job} has started. \ No newline at end of file diff --git a/docs/java-rest/high-level/ml/stop-data-frame-analytics.asciidoc b/docs/java-rest/high-level/ml/stop-data-frame-analytics.asciidoc index 41c15b76131..1cd2cab1baf 100644 --- a/docs/java-rest/high-level/ml/stop-data-frame-analytics.asciidoc +++ b/docs/java-rest/high-level/ml/stop-data-frame-analytics.asciidoc @@ -5,13 +5,13 @@ -- [role="xpack"] [id="{upid}-{api}"] -=== Stop Data Frame Analytics API +=== Stop {dfanalytics} API -The Stop Data Frame Analytics API is used to stop a running {dataframe-analytics-config}. +The stop {dfanalytics} API is used to stop a running {dataframe-analytics-config}. It accepts a +{request}+ object and responds with a +{response}+ object. [id="{upid}-{api}-request"] -==== Stop Data Frame Analytics Request +==== Stop {dfanalytics} Request A +{request}+ object requires a {dataframe-analytics-config} id. @@ -27,4 +27,4 @@ include::../execution.asciidoc[] [id="{upid}-{api}-response"] ==== Response -The returned +{response}+ object acknowledges the {dataframe-job} has stopped. \ No newline at end of file +The returned +{response}+ object acknowledges the {dfanalytics-job} has stopped. \ No newline at end of file diff --git a/docs/java-rest/high-level/supported-apis.asciidoc b/docs/java-rest/high-level/supported-apis.asciidoc index ec2995eba2f..abc9469590d 100644 --- a/docs/java-rest/high-level/supported-apis.asciidoc +++ b/docs/java-rest/high-level/supported-apis.asciidoc @@ -577,12 +577,12 @@ include::ilm/remove_lifecycle_policy_from_index.asciidoc[] [role="xpack"] [[_data_frame_transform_apis]] -== {dataframe-transform-cap} APIs +== {transform-cap} APIs :upid: {mainid}-dataframe :doc-tests-file: {doc-tests}/DataFrameTransformDocumentationIT.java -The Java High Level REST Client supports the following {dataframe-transform} +The Java High Level REST Client supports the following {transform} APIs: * <<{upid}-get-data-frame-transform>> diff --git a/docs/reference/data-frames/apis/delete-transform.asciidoc b/docs/reference/data-frames/apis/delete-transform.asciidoc index 557a8ffc59b..1717e595020 100644 --- a/docs/reference/data-frames/apis/delete-transform.asciidoc +++ b/docs/reference/data-frames/apis/delete-transform.asciidoc @@ -1,26 +1,26 @@ [role="xpack"] [testenv="basic"] [[delete-data-frame-transform]] -=== Delete {dataframe-transforms} API +=== Delete {transforms} API [subs="attributes"] ++++ -Delete {dataframe-transforms} +Delete {transforms} ++++ -Deletes an existing {dataframe-transform}. +Deletes an existing {transform}. beta[] [[delete-data-frame-transform-request]] ==== {api-request-title} -`DELETE _data_frame/transforms/` +`DELETE _data_frame/transforms/` [[delete-data-frame-transform-prereqs]] ==== {api-prereq-title} -* Before you can delete the {dataframe-transform}, you must stop it. +* Before you can delete the {transform}, you must stop it. * If the {es} {security-features} are enabled, you must have `manage_data_frame_transforms` cluster privileges to use this API. The built-in `data_frame_transforms_admin` role has these privileges. For more information, @@ -31,15 +31,15 @@ see {stack-ov}/security-privileges.html[Security privileges] and [[delete-data-frame-transform-path-parms]] ==== {api-path-parms-title} -``:: - (Required, string) Identifier for the {dataframe-transform}. +``:: + (Required, string) Identifier for the {transform}. [[delete-data-frame-transform-query-parms]] ==== {api-query-parms-title} `force`:: -(Optional, boolean) When `true`, the {dataframe-transform} is deleted regardless of its -current state. The default value is `false`, meaning that the {dataframe-transform} must be +(Optional, boolean) When `true`, the {transform} is deleted regardless of its +current state. The default value is `false`, meaning that the {transform} must be `stopped` before it can be deleted. [[delete-data-frame-transform-examples]] @@ -51,7 +51,7 @@ DELETE _data_frame/transforms/ecommerce_transform -------------------------------------------------- // TEST[skip:setup kibana sample data] -When the {dataframe-transform} is deleted, you receive the following results: +When the {transform} is deleted, you receive the following results: [source,console-result] ---- diff --git a/docs/reference/data-frames/apis/get-transform-stats.asciidoc b/docs/reference/data-frames/apis/get-transform-stats.asciidoc index 537a3263600..2ebe3c27993 100644 --- a/docs/reference/data-frames/apis/get-transform-stats.asciidoc +++ b/docs/reference/data-frames/apis/get-transform-stats.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[get-data-frame-transform-stats]] -=== Get {dataframe-transform} statistics API +=== Get {transform} statistics API [subs="attributes"] ++++ -Get {dataframe-transform} statistics +Get {transform} statistics ++++ -Retrieves usage information for {dataframe-transforms}. +Retrieves usage information for {transforms}. beta[] @@ -16,9 +16,9 @@ beta[] [[get-data-frame-transform-stats-request]] ==== {api-request-title} -`GET _data_frame/transforms//_stats` +`GET _data_frame/transforms//_stats` -`GET _data_frame/transforms/,/_stats` + +`GET _data_frame/transforms/,/_stats` + `GET _data_frame/transforms/_stats` + @@ -40,21 +40,21 @@ see {stack-ov}/security-privileges.html[Security privileges] and [[get-data-frame-transform-stats-desc]] ==== {api-description-title} -You can get statistics for multiple {dataframe-transforms} in a single API +You can get statistics for multiple {transforms} in a single API request by using a comma-separated list of identifiers or a wildcard expression. -You can get statistics for all {dataframe-transforms} by using `_all`, by -specifying `*` as the ``, or by omitting the -``. +You can get statistics for all {transforms} by using `_all`, by +specifying `*` as the ``, or by omitting the +``. [[get-data-frame-transform-stats-path-parms]] ==== {api-path-parms-title} -``:: - (Optional, string) Identifier for the {dataframe-transform}. It can be a - {dataframe-transform} identifier or a wildcard expression. If you do not +``:: + (Optional, string) Identifier for the {transform}. It can be a + {transform} identifier or a wildcard expression. If you do not specify one of these options, the API returns information for all - {dataframe-transforms}. + {transforms}. [[get-data-frame-transform-stats-query-parms]] @@ -64,7 +64,7 @@ specifying `*` as the ``, or by omitting the (Optional, boolean) Specifies what to do when the request: + -- -* Contains wildcard expressions and there are no {dataframe-transforms} that match. +* Contains wildcard expressions and there are no {transforms} that match. * Contains the `_all` string or no identifiers and there are no matches. * Contains wildcard expressions and there are only partial matches. @@ -75,17 +75,17 @@ are no matches or only partial matches. -- `from`:: - (Optional, integer) Skips the specified number of {dataframe-transforms}. The + (Optional, integer) Skips the specified number of {transforms}. The default value is `0`. `size`:: - (Optional, integer) Specifies the maximum number of {dataframe-transforms} to obtain. The default value is `100`. + (Optional, integer) Specifies the maximum number of {transforms} to obtain. The default value is `100`. [[get-data-frame-transform-stats-response]] ==== {api-response-body-title} `transforms`:: - (array) An array of statistics objects for {dataframe-transforms}, which are + (array) An array of statistics objects for {transforms}, which are sorted by the `id` value in ascending order. [[get-data-frame-transform-stats-response-codes]] @@ -98,7 +98,7 @@ are no matches or only partial matches. [[get-data-frame-transform-stats-example]] ==== Examples -The following example skips for the first five {dataframe-transforms} and +The following example skips for the first five {transforms} and gets usage information for a maximum of ten results: [source,console] @@ -108,7 +108,7 @@ GET _data_frame/transforms/_stats?from=5&size=10 // TEST[skip:todo] The following example gets usage information for the `ecommerce_transform` -{dataframe-transform}: +{transform}: [source,console] -------------------------------------------------- diff --git a/docs/reference/data-frames/apis/get-transform.asciidoc b/docs/reference/data-frames/apis/get-transform.asciidoc index 25b1421917a..7b4b7b8df6a 100644 --- a/docs/reference/data-frames/apis/get-transform.asciidoc +++ b/docs/reference/data-frames/apis/get-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[get-data-frame-transform]] -=== Get {dataframe-transforms} API +=== Get {transforms} API [subs="attributes"] ++++ -Get {dataframe-transforms} +Get {transforms} ++++ -Retrieves configuration information for {dataframe-transforms}. +Retrieves configuration information for {transforms}. beta[] @@ -37,9 +37,9 @@ see {stack-ov}/security-privileges.html[Security privileges] and [[get-data-frame-transform-desc]] ==== {api-description-title} -You can get information for multiple {dataframe-transforms} in a single API +You can get information for multiple {transforms} in a single API request by using a comma-separated list of identifiers or a wildcard expression. -You can get information for all {dataframe-transforms} by using `_all`, by +You can get information for all {transforms} by using `_all`, by specifying `*` as the ``, or by omitting the ``. @@ -47,10 +47,10 @@ specifying `*` as the ``, or by omitting the ==== {api-path-parms-title} ``:: - (Optional, string) Identifier for the {dataframe-transform}. It can be a - {dataframe-transform} identifier or a wildcard expression. If you do not + (Optional, string) Identifier for the {transform}. It can be a + {transform} identifier or a wildcard expression. If you do not specify one of these options, the API returns information for all - {dataframe-transforms}. + {transforms}. [[get-data-frame-transform-query-parms]] ==== {api-query-parms-title} @@ -59,7 +59,7 @@ specifying `*` as the ``, or by omitting the (Optional, boolean) Specifies what to do when the request: + -- -* Contains wildcard expressions and there are no {dataframe-transforms} that match. +* Contains wildcard expressions and there are no {transforms} that match. * Contains the `_all` string or no identifiers and there are no matches. * Contains wildcard expressions and there are only partial matches. @@ -70,17 +70,17 @@ are no matches or only partial matches. -- `from`:: - (Optional, integer) Skips the specified number of {dataframe-transforms}. The + (Optional, integer) Skips the specified number of {transforms}. The default value is `0`. `size`:: - (Optional, integer) Specifies the maximum number of {dataframe-transforms} to obtain. The default value is `100`. + (Optional, integer) Specifies the maximum number of {transforms} to obtain. The default value is `100`. [[get-data-frame-transform-response]] ==== {api-response-body-title} `transforms`:: - (array) An array of transform resources, which are sorted by the `id` value in + (array) An array of {transform} resources, which are sorted by the `id` value in ascending order. See <>. [[get-data-frame-transform-response-codes]] @@ -93,7 +93,7 @@ are no matches or only partial matches. [[get-data-frame-transform-example]] ==== {api-examples-title} -The following example retrieves information about a maximum of ten transforms: +The following example retrieves information about a maximum of ten {transforms}: [source,console] -------------------------------------------------- @@ -102,7 +102,7 @@ GET _data_frame/transforms?size=10 // TEST[skip:setup kibana sample data] The following example gets configuration information for the -`ecommerce_transform` {dataframe-transform}: +`ecommerce_transform` {transform}: [source,console] -------------------------------------------------- diff --git a/docs/reference/data-frames/apis/index.asciidoc b/docs/reference/data-frames/apis/index.asciidoc index 3abc123f354..e2b5172a6a3 100644 --- a/docs/reference/data-frames/apis/index.asciidoc +++ b/docs/reference/data-frames/apis/index.asciidoc @@ -1,9 +1,9 @@ [role="xpack"] [testenv="basic"] [[data-frame-apis]] -== {dataframe-transform-cap} APIs +== {transform-cap} APIs -See also {stack-ov}/ml-dataframes.html[{dataframe-transforms-cap}]. +See also {stack-ov}/ml-dataframes.html[{transforms-cap}]. * <> * <> diff --git a/docs/reference/data-frames/apis/preview-transform.asciidoc b/docs/reference/data-frames/apis/preview-transform.asciidoc index fde9cc7696f..0652dbb867a 100644 --- a/docs/reference/data-frames/apis/preview-transform.asciidoc +++ b/docs/reference/data-frames/apis/preview-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[preview-data-frame-transform]] -=== Preview {dataframe-transforms} API +=== Preview {transforms} API [subs="attributes"] ++++ -Preview {dataframe-transforms} +Preview {transforms} ++++ -Previews a {dataframe-transform}. +Previews a {transform}. beta[] @@ -24,7 +24,7 @@ beta[] `manage_data_frame_transforms` cluster privileges to use this API. The built-in `data_frame_transforms_admin` role has these privileges. You must also have `read` and `view_index_metadata` privileges on the source index for the -{dataframe-transform}. For more information, see +{transform}. For more information, see {stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/built-in-roles.html[Built-in roles]. @@ -32,7 +32,7 @@ beta[] ==== {api-description-title} This API generates a preview of the results that you will get when you run the -<> with the same +<> with the same configuration. It returns a maximum of 100 results. The calculations are based on all the current data in the source index. @@ -45,7 +45,7 @@ on all the current data in the source index. `index`::: (Required, string or array) The _source indices_ for the - {dataframe-transform}. It can be a single index, an index pattern (for + {transform}. It can be a single index, an index pattern (for example, `"myindex*"`), or an array of indices (for example, `["index1", "index2"]`). @@ -63,7 +63,7 @@ on all the current data in the source index. `preview`:: (array) An array of documents. In particular, they are the JSON representation of the documents that would be created in the destination index - by the {dataframe-transform}. + by the {transform}. ==== {api-examples-title} diff --git a/docs/reference/data-frames/apis/put-transform.asciidoc b/docs/reference/data-frames/apis/put-transform.asciidoc index f919afa1de3..284a27157e9 100644 --- a/docs/reference/data-frames/apis/put-transform.asciidoc +++ b/docs/reference/data-frames/apis/put-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[put-data-frame-transform]] -=== Create {dataframe-transforms} API +=== Create {transforms} API [subs="attributes"] ++++ -Create {dataframe-transforms} +Create {transforms} ++++ -Instantiates a {dataframe-transform}. +Instantiates a {transform}. beta[] @@ -31,30 +31,30 @@ information, see {stack-ov}/security-privileges.html[Security privileges] and [[put-data-frame-transform-desc]] ==== {api-description-title} -This API defines a {dataframe-transform}, which copies data from source indices, +This API defines a {transform}, which copies data from source indices, transforms it, and persists it into an entity-centric destination index. The entities are defined by the set of `group_by` fields in the `pivot` object. You can also think of the destination index as a two-dimensional tabular data structure (known as a {dataframe}). The ID for each document in the {dataframe} is generated from a hash of the entity, so there is a unique row per entity. For more information, see -{stack-ov}/ml-dataframes.html[{dataframe-transforms-cap}]. +{stack-ov}/ml-dataframes.html[{transforms-cap}]. -When the {dataframe-transform} is created, a series of validations occur to +When the {transform} is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use the `defer_validation` parameter to skip these checks. -Deferred validations are always run when the {dataframe-transform} is started, +Deferred validations are always run when the {transform} is started, with the exception of privilege checks. When {es} {security-features} are -enabled, the {dataframe-transform} remembers which roles the user that created +enabled, the {transform} remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the -{dataframe-transform} fails when it attempts unauthorized operations. +{transform} fails when it attempts unauthorized operations. -IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. - Do not put a {dataframe-transform} directly into any +IMPORTANT: You must use {kib} or this API to create a {transform}. + Do not put a {transform} directly into any `.data-frame-internal*` indices using the Elasticsearch index API. If {es} {security-features} are enabled, do not give users any privileges on `.data-frame-internal*` indices. @@ -63,7 +63,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. ==== {api-path-parms-title} ``:: - (Required, string) Identifier for the {dataframe-transform}. This identifier + (Required, string) Identifier for the {transform}. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters. @@ -73,28 +73,28 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. `defer_validation`:: (Optional, boolean) When `true`, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the - {dataframe-transform} is created. + {transform} is created. [[put-data-frame-transform-request-body]] ==== {api-request-body-title} `description`:: - (Optional, string) Free text description of the {dataframe-transform}. + (Optional, string) Free text description of the {transform}. `dest`:: (Required, object) Required. The destination configuration, which has the following properties: `index`::: - (Required, string) The _destination index_ for the {dataframe-transform}. + (Required, string) The _destination index_ for the {transform}. `pipeline`::: (Optional, string) The unique identifier for a <>. `frequency`:: (Optional, <>) The interval between checks for changes in the source - indices when the {dataframe-transform} is running continuously. Also determines - the retry interval in the event of transient failures while the {dataframe-transform} is + indices when the {transform} is running continuously. Also determines + the retry interval in the event of transient failures while the {transform} is searching or indexing. The minimum value is `1s` and the maximum is `1h`. The default value is `1m`. @@ -108,7 +108,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. `index`::: (Required, string or array) The _source indices_ for the - {dataframe-transform}. It can be a single index, an index pattern (for + {transform}. It can be a single index, an index pattern (for example, `"myindex*"`), or an array of indices (for example, `["index1", "index2"]`). @@ -119,7 +119,7 @@ IMPORTANT: You must use {kib} or this API to create a {dataframe-transform}. `sync`:: (Optional, object) Defines the properties required to run continuously. `time`::: - (Required, object) Specifies that the {dataframe-transform} uses a time + (Required, object) Specifies that the {transform} uses a time field to synchronize the source and destination indices. `field`:::: (Required, string) The date field that is used to identify new documents @@ -185,7 +185,7 @@ PUT _data_frame/transforms/ecommerce_transform -------------------------------------------------- // TEST[setup:kibana_sample_data_ecommerce] -When the transform is created, you receive the following results: +When the {transform} is created, you receive the following results: [source,console-result] ---- diff --git a/docs/reference/data-frames/apis/start-transform.asciidoc b/docs/reference/data-frames/apis/start-transform.asciidoc index 5a8735a9b7e..a633bbf9961 100644 --- a/docs/reference/data-frames/apis/start-transform.asciidoc +++ b/docs/reference/data-frames/apis/start-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[start-data-frame-transform]] -=== Start {dataframe-transforms} API +=== Start {transforms} API [subs="attributes"] ++++ -Start {dataframe-transforms} +Start {transforms} ++++ -Starts one or more {dataframe-transforms}. +Starts one or more {transforms}. beta[] @@ -23,38 +23,38 @@ beta[] * If the {es} {security-features} are enabled, you must have `manage_data_frame_transforms` cluster privileges to use this API. You must also have `view_index_metadata` privileges on the source index for the -{dataframe-transform}. For more information, see +{transform}. For more information, see {stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/built-in-roles.html[Built-in roles]. [[start-data-frame-transform-desc]] ==== {api-description-title} -When you start a {dataframe-transform}, it creates the destination index if it +When you start a {transform}, it creates the destination index if it does not already exist. The `number_of_shards` is set to `1` and the `auto_expand_replicas` is set to `0-1`. -The transform deduces the mapping definitions from the source indices. For +The {transform} deduces the mapping definitions from the source indices. For scripted fields, it uses <>. If a field in the destination index is created by `scripted_metric` or `bucket_script` -aggregations, the transform uses dynamic mappings unless a template exists or +aggregations, the {transform} uses dynamic mappings unless a template exists or the destination index already exists. Mapping definitions in the destination index take precedence over dynamic mappings and templates. -When the {dataframe-transform} starts, a series of validations occur to ensure +When the {transform} starts, a series of validations occur to ensure its success. If you deferred validation when you created the -{dataframe-transform}, they occur when you start the transform--with the +{transform}, they occur when you start the {transform}--with the exception of privilege checks. When {es} {security-features} are enabled, the -{dataframe-transform} remembers which roles the user that created it had at the +{transform} remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the -{dataframe-transform} fails when it attempts unauthorized operations. +{transform} fails when it attempts unauthorized operations. [[start-data-frame-transform-path-parms]] ==== {api-path-parms-title} ``:: - (Required, string) Identifier for the {dataframe-transform}. This identifier + (Required, string) Identifier for the {transform}. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters. @@ -67,7 +67,7 @@ POST _data_frame/transforms/ecommerce_transform/_start -------------------------------------------------- // TEST[skip:set up kibana samples] -When the {dataframe-transform} starts, you receive the following results: +When the {transform} starts, you receive the following results: [source,console-result] ---- diff --git a/docs/reference/data-frames/apis/stop-transform.asciidoc b/docs/reference/data-frames/apis/stop-transform.asciidoc index 3ce9c56a254..d7d6b0f6d28 100644 --- a/docs/reference/data-frames/apis/stop-transform.asciidoc +++ b/docs/reference/data-frames/apis/stop-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[stop-data-frame-transform]] -=== Stop {dataframe-transforms} API +=== Stop {transforms} API [subs="attributes"] ++++ -Stop {dataframe-transforms} +Stop {transforms} ++++ -Stops one or more {dataframe-transforms}. +Stops one or more {transforms}. beta[] @@ -33,16 +33,16 @@ see {stack-ov}/security-privileges.html[Security privileges] and [[stop-data-frame-transform-desc]] ==== {api-description-title} -You can stop multiple {dataframe-transforms} in a single API request by using a -comma-separated list of {dataframe-transforms} or a wildcard expression. -All {dataframe-transforms} can be stopped by using `_all` or `*` as the +You can stop multiple {transforms} in a single API request by using a +comma-separated list of {transforms} or a wildcard expression. +All {transforms} can be stopped by using `_all` or `*` as the ``. [[stop-data-frame-transform-path-parms]] ==== {api-path-parms-title} ``:: - (Required, string) Identifier for the {dataframe-transform}. This identifier + (Required, string) Identifier for the {transform}. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters. @@ -53,15 +53,15 @@ All {dataframe-transforms} can be stopped by using `_all` or `*` as the (Optional, boolean) Specifies what to do when the request: + -- -* Contains wildcard expressions and there are no {dataframe-transforms} that match. +* Contains wildcard expressions and there are no {transforms} that match. * Contains the `_all` string or no identifiers and there are no matches. * Contains wildcard expressions and there are only partial matches. The default value is `true`, which returns a successful acknowledgement message when there are no matches. When there are only partial matches, the API stops -the appropriate {dataframe-transforms}. For example, if the request contains -`test-id1*,test-id2*` as the identifiers and there are no {dataframe-transforms} -that match `test-id2*`, the API nonetheless stops the {dataframe-transforms} +the appropriate {transforms}. For example, if the request contains +`test-id1*,test-id2*` as the identifiers and there are no {transforms} +that match `test-id2*`, the API nonetheless stops the {transforms} that match `test-id1*`. If this parameter is `false`, the request returns a `404` status code when there @@ -70,10 +70,10 @@ are no matches or only partial matches. `timeout`:: (Optional, time value) If `wait_for_completion=true`, the API blocks for (at - maximum) the specified duration while waiting for the transform to stop. If + maximum) the specified duration while waiting for the {transform} to stop. If more than `timeout` time has passed, the API throws a timeout exception. Even if a timeout exception is thrown, the stop request is still processing and - eventually moves the transform to `STOPPED`. The timeout simply means the API + eventually moves the {transform} to `STOPPED`. The timeout simply means the API call itself timed out while waiting for the status change. Defaults to `30s` `wait_for_completion`:: @@ -97,7 +97,7 @@ POST _data_frame/transforms/ecommerce_transform/_stop -------------------------------------------------- // TEST[skip:set up kibana samples] -When the {dataframe-transform} stops, you receive the following results: +When the {transform} stops, you receive the following results: [source,console-result] ---- diff --git a/docs/reference/data-frames/apis/transformresource.asciidoc b/docs/reference/data-frames/apis/transformresource.asciidoc index f51074bd456..155aa5a7160 100644 --- a/docs/reference/data-frames/apis/transformresource.asciidoc +++ b/docs/reference/data-frames/apis/transformresource.asciidoc @@ -1,9 +1,9 @@ [role="xpack"] [testenv="basic"] [[data-frame-transform-resource]] -=== {dataframe-transform-cap} resources +=== {transform-cap} resources -{dataframe-transform-cap} resources relate to the <>. +{transform-cap} resources relate to the <>. For more information, see {stack-ov}/ecommerce-dataframes.html[Transforming your data with {dataframes}]. @@ -13,42 +13,42 @@ For more information, see ==== {api-definitions-title} `description`:: - (string) A description of the {dataframe-transform}. + (string) A description of the {transform}. `dest`:: - (object) The destination for the {dataframe-transform}. See + (object) The destination for the {transform}. See <>. `frequency`:: (time units) The interval between checks for changes in the source indices - when the {dataframe-transform} is running continuously. Also determines the - retry interval in the event of transient failures while the {dataframe-transform} is + when the {transform} is running continuously. Also determines the + retry interval in the event of transient failures while the {transform} is searching or indexing. The minimum value is `1s` and the maximum is `1h`. The default value is `1m`. `id`:: - (string) A unique identifier for the {dataframe-transform}. + (string) A unique identifier for the {transform}. `pivot`:: (object) The method for transforming the data. See <>. `source`:: - (object) The source of the data for the {dataframe-transform}. See + (object) The source of the data for the {transform}. See <>. [[data-frame-transform-dest]] ==== Dest objects -{dataframe-transform-cap} resources contain `dest` objects. For example, when -you create a {dataframe-transform}, you must define its destination. +{transform-cap} resources contain `dest` objects. For example, when +you create a {transform}, you must define its destination. [discrete] [[data-frame-transform-dest-properties]] ===== {api-definitions-title} `index`:: - (string) The _destination index_ for the {dataframe-transform}. + (string) The _destination index_ for the {transform}. `pipeline`:: (string) The unique identifier for a <>. @@ -56,15 +56,15 @@ you create a {dataframe-transform}, you must define its destination. [[data-frame-transform-source]] ==== Source objects -{dataframe-transform-cap} resources contain `source` objects. For example, when -you create a {dataframe-transform}, you must define its source. +{transform-cap} resources contain `source` objects. For example, when +you create a {transform}, you must define its source. [discrete] [[data-frame-transform-source-properties]] ===== {api-definitions-title} `index`:: - (string or array) The _source indices_ for the {dataframe-transform}. It can + (string or array) The _source indices_ for the {transform}. It can be a single index, an index pattern (for example, `"myindex*"`), or an array of indices (for example, `["index1", "index2"]`). @@ -75,7 +75,7 @@ you create a {dataframe-transform}, you must define its source. [[data-frame-transform-pivot]] ==== Pivot objects -{dataframe-transform-cap} resources contain `pivot` objects, which define the +{transform-cap} resources contain `pivot` objects, which define the pivot function `group by` fields and the aggregation to reduce the data. [discrete] @@ -100,7 +100,7 @@ pivot function `group by` fields and the aggregation to reduce the data. * {ref}/search-aggregations-pipeline-bucket-script-aggregation.html[Bucket Script] * {ref}/search-aggregations-pipeline-bucket-selector-aggregation.html[Bucket Selector] -IMPORTANT: {dataframe-transforms-cap} support a subset of the functionality in +IMPORTANT: {transforms-cap} support a subset of the functionality in composite aggregations. See {stack-ov}/dataframe-limitations.html[{dataframe-cap} limitations]. @@ -126,4 +126,4 @@ composite aggregations. See ==== {api-examples-title} See the -<>. +<>. diff --git a/docs/reference/data-frames/apis/update-transform.asciidoc b/docs/reference/data-frames/apis/update-transform.asciidoc index 9cbd5d639f9..06dbc70f2a4 100644 --- a/docs/reference/data-frames/apis/update-transform.asciidoc +++ b/docs/reference/data-frames/apis/update-transform.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[update-data-frame-transform]] -=== Update {dataframe-transforms} API +=== Update {transforms} API [subs="attributes"] ++++ -Update {dataframe-transforms} +Update {transforms} ++++ -Updates an existing {dataframe-transform}. +Updates an existing {transform}. beta[] @@ -31,16 +31,16 @@ information, see {stack-ov}/security-privileges.html[Security privileges] and [[update-data-frame-transform-desc]] ==== {api-description-title} -This API updates an existing {dataframe-transform}. All settings except description do not -take effect until after the {dataframe-transform} starts the next checkpoint. This is +This API updates an existing {transform}. All settings except description do not +take effect until after the {transform} starts the next checkpoint. This is so there is consistency with the pivoted data in each checkpoint. -IMPORTANT: When {es} {security-features} are enabled, your {dataframe-transform} +IMPORTANT: When {es} {security-features} are enabled, your {transform} remembers which roles the user who updated it had at the time of update and runs with those privileges. -IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. - Do not update a {dataframe-transform} directly via +IMPORTANT: You must use {kib} or this API to update a {transform}. + Do not update a {transform} directly via `.data-frame-internal*` indices using the Elasticsearch index API. If {es} {security-features} are enabled, do not give users any privileges on `.data-frame-internal*` indices. @@ -49,7 +49,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. ==== {api-path-parms-title} ``:: - (Required, string) Identifier for the {dataframe-transform}. This identifier + (Required, string) Identifier for the {transform}. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters. @@ -59,29 +59,29 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. `defer_validation`:: (Optional, boolean) When `true`, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the - {dataframe-transform} is updated. + {transform} is updated. [[update-data-frame-transform-request-body]] ==== {api-request-body-title} `description`:: - (Optional, string) Free text description of the {dataframe-transform}. + (Optional, string) Free text description of the {transform}. `dest`:: (Optional, object) The destination configuration, which has the following properties: `index`::: - (Required, string) The _destination index_ for the {dataframe-transform}. + (Required, string) The _destination index_ for the {transform}. `pipeline`::: (Optional, string) The unique identifier for a <>. `frequency`:: (Optional, <>) The interval between checks for changes - in the source indices when the {dataframe-transform} is running continuously. + in the source indices when the {transform} is running continuously. Also determines the retry interval in the event of transient failures while - the {dataframe-transform} is searching or indexing. The minimum value is `1s` + the {transform} is searching or indexing. The minimum value is `1s` and the maximum is `1h`. The default value is `1m`. `source`:: @@ -90,7 +90,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. `index`::: (Required, string or array) The _source indices_ for the - {dataframe-transform}. It can be a single index, an index pattern (for + {transform}. It can be a single index, an index pattern (for example, `"myindex*"`), or an array of indices (for example, `["index1", "index2"]`). @@ -101,7 +101,7 @@ IMPORTANT: You must use {kib} or this API to update a {dataframe-transform}. `sync`:: (Optional, object) Defines the properties required to run continuously. `time`::: - (Required, object) Specifies that the {dataframe-transform} uses a time + (Required, object) Specifies that the {transform} uses a time field to synchronize the source and destination indices. `field`:::: (Required, string) The date field that is used to identify new documents @@ -151,7 +151,7 @@ POST _data_frame/transforms/simple-kibana-ecomm-pivot/_update -------------------------------------------------- // TEST[setup:simple_kibana_continuous_pivot] -When the transform is updated, you receive the updated configuration: +When the {transform} is updated, you receive the updated configuration: [source,console-result] ---- diff --git a/docs/reference/rest-api/defs.asciidoc b/docs/reference/rest-api/defs.asciidoc index efeb6c60923..0fb37b573a2 100644 --- a/docs/reference/rest-api/defs.asciidoc +++ b/docs/reference/rest-api/defs.asciidoc @@ -8,7 +8,7 @@ These resource definitions are used in APIs related to {ml-features} and * <> * <> * <> -* <> +* <> * <> * <> * <> diff --git a/docs/reference/rest-api/index.asciidoc b/docs/reference/rest-api/index.asciidoc index b80df9a93e8..e6179ffed22 100644 --- a/docs/reference/rest-api/index.asciidoc +++ b/docs/reference/rest-api/index.asciidoc @@ -14,7 +14,7 @@ not be included yet. * <> * <> * <> -* <> +* <> * <> * <> * <> diff --git a/docs/reference/settings/data-frames-settings.asciidoc b/docs/reference/settings/data-frames-settings.asciidoc index e550063f7b4..dc000fc414f 100644 --- a/docs/reference/settings/data-frames-settings.asciidoc +++ b/docs/reference/settings/data-frames-settings.asciidoc @@ -1,13 +1,13 @@ [role="xpack"] [[data-frames-settings]] -=== {dataframe-transforms-cap} settings in Elasticsearch +=== {transforms-cap} settings in Elasticsearch [subs="attributes"] ++++ -{dataframe-transforms-cap} settings +{transforms-cap} settings ++++ -You do not need to configure any settings to use {dataframe-transforms}. It is enabled by default. +You do not need to configure any settings to use {transforms}. It is enabled by default. All of these settings can be added to the `elasticsearch.yml` configuration file. The dynamic settings can also be updated across a cluster with the @@ -18,23 +18,23 @@ file. [float] [[general-data-frames-settings]] -==== General {dataframe-transforms} settings +==== General {transforms} settings `xpack.data_frame.enabled`:: -Set to `true` (default) to enable {dataframe-transforms} on the node. + +Set to `true` (default) to enable {transforms} on the node. + + -If set to `false` in `elasticsearch.yml`, the {dataframe-transform} APIs are disabled on the node. -Therefore the node cannot start or administrate transforms or receive transport (internal) -communication requests related to {dataframe-transform} APIs. +If set to `false` in `elasticsearch.yml`, the {transform} APIs are disabled on the node. +Therefore the node cannot start or administrate {transform} or receive transport (internal) +communication requests related to {transform} APIs. + -IMPORTANT: If you want to use {dataframe-transform} features in your cluster, you must have +IMPORTANT: If you want to use {transform} features in your cluster, you must have `xpack.data_frame.enabled` set to `true` on all master-eligible nodes. This is the default behavior. `xpack.data_frame.num_transform_failure_retries` (<>):: -The number of times that a {dataframe-transform} retries when it experiences a -non-fatal error. Once the number of retries is exhausted, the {dataframe-transform} +The number of times that a {transform} retries when it experiences a +non-fatal error. Once the number of retries is exhausted, the {transform} task will be marked as `failed`. The default value is `10` with a valid minimum of `0` and maximum of `100`. -If a {dataframe-transform} is already running, it will have to be restarted +If a {transform} is already running, it will have to be restarted to use the changed setting. diff --git a/docs/reference/transform/api-quickref.asciidoc b/docs/reference/transform/api-quickref.asciidoc index 7750331a027..31dabfa687a 100644 --- a/docs/reference/transform/api-quickref.asciidoc +++ b/docs/reference/transform/api-quickref.asciidoc @@ -2,7 +2,7 @@ [[df-api-quickref]] == API quick reference -All {dataframe-transform} endpoints have the following base: +All {transform} endpoints have the following base: [source,js] ---- @@ -10,12 +10,12 @@ All {dataframe-transform} endpoints have the following base: ---- // NOTCONSOLE -* {ref}/put-data-frame-transform.html[Create {dataframe-transforms}] -* {ref}/delete-data-frame-transform.html[Delete {dataframe-transforms}] -* {ref}/get-data-frame-transform.html[Get {dataframe-transforms}] -* {ref}/get-data-frame-transform-stats.html[Get {dataframe-transforms} statistics] -* {ref}/preview-data-frame-transform.html[Preview {dataframe-transforms}] -* {ref}/start-data-frame-transform.html[Start {dataframe-transforms}] -* {ref}/stop-data-frame-transform.html[Stop {dataframe-transforms}] +* {ref}/put-data-frame-transform.html[Create {transforms}] +* {ref}/delete-data-frame-transform.html[Delete {transforms}] +* {ref}/get-data-frame-transform.html[Get {transforms}] +* {ref}/get-data-frame-transform-stats.html[Get {transforms} statistics] +* {ref}/preview-data-frame-transform.html[Preview {transforms}] +* {ref}/start-data-frame-transform.html[Start {transforms}] +* {ref}/stop-data-frame-transform.html[Stop {transforms}] -For the full list, see {ref}/data-frame-apis.html[{dataframe-transform-cap} APIs]. +For the full list, see {ref}/data-frame-apis.html[{transform-cap} APIs]. diff --git a/docs/reference/transform/checkpoints.asciidoc b/docs/reference/transform/checkpoints.asciidoc index 808ce071ede..4c41b876b23 100644 --- a/docs/reference/transform/checkpoints.asciidoc +++ b/docs/reference/transform/checkpoints.asciidoc @@ -1,24 +1,24 @@ [role="xpack"] [[ml-transform-checkpoints]] -== How {dataframe-transform} checkpoints work +== How {transform} checkpoints work ++++ How checkpoints work ++++ beta[] -Each time a {dataframe-transform} examines the source indices and creates or +Each time a {transform} examines the source indices and creates or updates the destination index, it generates a _checkpoint_. -If your {dataframe-transform} runs only once, there is logically only one -checkpoint. If your {dataframe-transform} runs continuously, however, it creates +If your {transform} runs only once, there is logically only one +checkpoint. If your {transform} runs continuously, however, it creates checkpoints as it ingests and transforms new source data. -To create a checkpoint, the {cdataframe-transform}: +To create a checkpoint, the {ctransform}: . Checks for changes to source indices. + -Using a simple periodic timer, the {dataframe-transform} checks for changes to +Using a simple periodic timer, the {transform} checks for changes to the source indices. This check is done based on the interval defined in the transform's `frequency` property. + @@ -27,32 +27,32 @@ then it waits for the next timer. . Identifies which entities have changed. + -The {dataframe-transform} searches to see which entities have changed since the -last time it checked. The transform's `sync` configuration object identifies a -time field in the source indices. The transform uses the values in that field to -synchronize the source and destination indices. +The {transform} searches to see which entities have changed since the +last time it checked. The `sync` configuration object in the {transform} +identifies a time field in the source indices. The {transform} uses the values +in that field to synchronize the source and destination indices. . Updates the destination index (the {dataframe}) with the changed entities. + -- -The {dataframe-transform} applies changes related to either new or changed +The {transform} applies changes related to either new or changed entities to the destination index. The set of changed entities is paginated. For -each page, the {dataframe-transform} performs a composite aggregation using a +each page, the {transform} performs a composite aggregation using a `terms` query. After all the pages of changes have been applied, the checkpoint is complete. -- This checkpoint process involves both search and indexing activity on the cluster. We have attempted to favor control over performance while developing -{dataframe-transforms}. We decided it was preferable for the -{dataframe-transform} to take longer to complete, rather than to finish quickly +{transforms}. We decided it was preferable for the +{transform} to take longer to complete, rather than to finish quickly and take precedence in resource consumption. That being said, the cluster still requires enough resources to support both the composite aggregation search and the indexing of its results. TIP: If the cluster experiences unsuitable performance degradation due to the -{dataframe-transform}, stop the transform. Consider whether you can apply a -source query to the {dataframe-transform} to reduce the scope of data it +{transform}, stop the {transform}. Consider whether you can apply a +source query to the {transform} to reduce the scope of data it processes. Also consider whether the cluster has sufficient resources in place to support both the composite aggregation search and the indexing of its results. @@ -61,28 +61,28 @@ results. [[ml-transform-checkpoint-errors]] ==== Error handling -Failures in {dataframe-transforms} tend to be related to searching or indexing. -To increase the resiliency of {dataframe-transforms}, the cursor positions of +Failures in {transforms} tend to be related to searching or indexing. +To increase the resiliency of {transforms}, the cursor positions of the aggregated search and the changed entities search are tracked in memory and persisted periodically. Checkpoint failures can be categorized as follows: * Temporary failures: The checkpoint is retried. If 10 consecutive failures -occur, the {dataframe-transform} has a failed status. For example, this +occur, the {transform} has a failed status. For example, this situation might occur when there are shard failures and queries return only partial results. -* Irrecoverable failures: The {dataframe-transform} immediately fails. For +* Irrecoverable failures: The {transform} immediately fails. For example, this situation occurs when the source index is not found. -* Adjustment failures: The {dataframe-transform} retries with adjusted settings. +* Adjustment failures: The {transform} retries with adjusted settings. For example, if a parent circuit breaker memory errors occur during the -composite aggregation, the transform receives partial results. The aggregated +composite aggregation, the {transform} receives partial results. The aggregated search is retried with a smaller number of buckets. This retry is performed at -the interval defined in the transform's `frequency` property. If the search -is retried to the point where it reaches a minimal number of buckets, an +the interval defined in the `frequency` property for the {transform}. If the +search is retried to the point where it reaches a minimal number of buckets, an irrecoverable failure occurs. -If the node running the {dataframe-transforms} fails, the transform restarts +If the node running the {transforms} fails, the {transform} restarts from the most recent persisted cursor position. This recovery process might -repeat some of the work the transform had already done, but it ensures data +repeat some of the work the {transform} had already done, but it ensures data consistency. diff --git a/docs/reference/transform/dataframe-examples.asciidoc b/docs/reference/transform/dataframe-examples.asciidoc index 31abb4787a9..6c03ad3ecb3 100644 --- a/docs/reference/transform/dataframe-examples.asciidoc +++ b/docs/reference/transform/dataframe-examples.asciidoc @@ -1,14 +1,14 @@ [role="xpack"] [testenv="basic"] [[dataframe-examples]] -== {dataframe-transform-cap} examples +== {transform-cap} examples ++++ Examples ++++ beta[] -These examples demonstrate how to use {dataframe-transforms} to derive useful +These examples demonstrate how to use {transforms} to derive useful insights from your data. All the examples use one of the {kibana-ref}/add-sample-data.html[{kib} sample datasets]. For a more detailed, step-by-step example, see @@ -67,7 +67,7 @@ gives more context to the final results. NOTE: In the example above, condensed JSON formatting has been used for easier readability of the pivot object. -The preview {dataframe-transforms} API enables you to see the layout of the +The preview {transforms} API enables you to see the layout of the {dataframe} in advance, populated with some sample values. For example: [source,js] @@ -195,7 +195,7 @@ or flight stats for any of the featured destination or origin airports. [[example-clientips]] === Finding suspicious client IPs by using scripted metrics -With {dataframe-transforms}, you can use +With {transforms}, you can use {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[scripted metric aggregations] on your data. These aggregations are flexible and make it possible to perform very complex processing. Let's use scripted metrics to @@ -278,8 +278,8 @@ POST _data_frame/transforms/_preview ---------------------------------- // TEST[skip:setup kibana sample data] -<1> This range query limits the transform to documents that are within the last -30 days at the point in time the {dataframe-transform} checkpoint is processed. +<1> This range query limits the {transform} to documents that are within the last +30 days at the point in time the {transform} checkpoint is processed. For batch {dataframes} this occurs once. <2> This is the destination index for the {dataframe}. It is ignored by `_preview`. diff --git a/docs/reference/transform/ecommerce-example.asciidoc b/docs/reference/transform/ecommerce-example.asciidoc index 3404ee37758..01841cbd806 100644 --- a/docs/reference/transform/ecommerce-example.asciidoc +++ b/docs/reference/transform/ecommerce-example.asciidoc @@ -5,10 +5,10 @@ beta[] -<> enable you to retrieve information +<> enable you to retrieve information from an {es} index, transform it, and store it in another index. Let's use the {kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can -pivot and summarize your data with {dataframe-transforms}. +pivot and summarize your data with {transforms}. . If the {es} {security-features} are enabled, obtain a user ID with sufficient @@ -16,7 +16,7 @@ privileges to complete these steps. + -- You need `manage_data_frame_transforms` cluster privileges to preview and create -{dataframe-transforms}. Members of the built-in `data_frame_transforms_admin` +{transforms}. Members of the built-in `data_frame_transforms_admin` role have these privileges. You also need `read` and `view_index_metadata` index privileges on the source @@ -51,10 +51,10 @@ at least one aggregation. You can preview what the transformed data will look like, so go ahead and play with it! For example, go to *Machine Learning* > *Data Frames* in {kib} and use the -wizard to create a {dataframe-transform}: +wizard to create a {transform}: [role="screenshot"] -image::images/ecommerce-pivot1.jpg["Creating a simple {dataframe-transform} in {kib}"] +image::images/ecommerce-pivot1.jpg["Creating a simple {transform} in {kib}"] In this case, we grouped the data by customer ID and calculated the sum of products each customer purchased. @@ -71,7 +71,7 @@ We'll accomplish this by using the on the `order_id` field: [role="screenshot"] -image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {dataframe-transform} in {kib}"] +image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {transform} in {kib}"] TIP: If you're interested in a subset of the data, you can optionally include a {ref}/search-request-body.html#request-body-search-query[query] element. In this @@ -81,7 +81,7 @@ If you want to use more complex queries, you can create your {dataframe} from a {kibana-ref}/save-open-search.html[saved search]. If you prefer, you can use the -{ref}/preview-data-frame-transform.html[preview {dataframe-transforms} API]: +{ref}/preview-data-frame-transform.html[preview {transforms} API]: [source,console] -------------------------------------------------- @@ -134,28 +134,28 @@ POST _data_frame/transforms/_preview -- . When you are satisfied with what you see in the preview, create the -{dataframe-transform}. +{transform}. + -- .. Supply a job ID and the name of the target (or _destination_) index. -.. Decide whether you want the {dataframe-transform} to run once or continuously. +.. Decide whether you want the {transform} to run once or continuously. -- + -- Since this sample data index is unchanging, let's use the default behavior and -just run the {dataframe-transform} once. +just run the {transform} once. [role="screenshot"] -image::images/ecommerce-batch.jpg["Specifying the {dataframe-transform} options in {kib}"] +image::images/ecommerce-batch.jpg["Specifying the {transform} options in {kib}"] If you want to try it out, however, go ahead and click on *Continuous mode*. -You must choose a field that the {dataframe-transform} can use to check which +You must choose a field that the {transform} can use to check which entities have changed. In general, it's a good idea to use the ingest timestamp field. In this example, however, you can use the `order_date` field. If you prefer, you can use the -{ref}/put-data-frame-transform.html[create {dataframe-transforms} API]. For +{ref}/put-data-frame-transform.html[create {transforms} API]. For example: [source,console] @@ -215,23 +215,23 @@ PUT _data_frame/transforms/ecommerce-customer-transform // TEST[skip:setup kibana sample data] -- -. Start the {dataframe-transform}. +. Start the {transform}. + -- TIP: Even though resource utilization is automatically adjusted based on the -cluster load, a {dataframe-transform} increases search and indexing load on your +cluster load, a {transform} increases search and indexing load on your cluster while it runs. If you're experiencing an excessive load, however, you can stop it. -You can start, stop, and manage {dataframe-transforms} in {kib}: +You can start, stop, and manage {transforms} in {kib}: [role="screenshot"] -image::images/dataframe-transforms.jpg["Managing {dataframe-transforms} in {kib}"] +image::images/dataframe-transforms.jpg["Managing {transforms} in {kib}"] Alternatively, you can use the -{ref}/start-data-frame-transform.html[start {dataframe-transforms}] and -{ref}/stop-data-frame-transform.html[stop {dataframe-transforms}] APIs. For +{ref}/start-data-frame-transform.html[start {transforms}] and +{ref}/stop-data-frame-transform.html[stop {transforms}] APIs. For example: [source,console] @@ -252,8 +252,8 @@ image::images/ecommerce-results.jpg["Exploring the new index in {kib}"] -- -TIP: If you do not want to keep the {dataframe-transform}, you can delete it in +TIP: If you do not want to keep the {transform}, you can delete it in {kib} or use the -{ref}/delete-data-frame-transform.html[delete {dataframe-transform} API]. When -you delete a {dataframe-transform}, its destination index and {kib} index +{ref}/delete-data-frame-transform.html[delete {transform} API]. When +you delete a {transform}, its destination index and {kib} index patterns remain. diff --git a/docs/reference/transform/index.asciidoc b/docs/reference/transform/index.asciidoc index 2a45e1709dd..7242e850d3f 100644 --- a/docs/reference/transform/index.asciidoc +++ b/docs/reference/transform/index.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[ml-dataframes]] -= {dataframe-transforms-cap} += {transforms-cap} [partintro] -- @@ -65,7 +65,7 @@ create a {dataframe} to cache results. Thus, each user doesn't need to run the aggregation query. Though there are multiple ways to create {dataframes}, this content pertains -to one specific method: _{dataframe-transforms}_. +to one specific method: _{transforms}_. * <> * <> diff --git a/docs/reference/transform/limitations.asciidoc b/docs/reference/transform/limitations.asciidoc index c019018bce5..6e5e4b54e9a 100644 --- a/docs/reference/transform/limitations.asciidoc +++ b/docs/reference/transform/limitations.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[dataframe-limitations]] -== {dataframe-transform-cap} limitations +== {transform-cap} limitations [subs="attributes"] ++++ Limitations @@ -13,14 +13,14 @@ the Elastic {dataframe} feature: [float] [[df-compatibility-limitations]] -=== Beta {dataframe-transforms} do not have guaranteed backwards or forwards compatibility +=== Beta {transforms} do not have guaranteed backwards or forwards compatibility -Whilst {dataframe-transforms} are beta, it is not guaranteed that a -{dataframe-transform} created in a previous version of the {stack} will be able +Whilst {transforms} are beta, it is not guaranteed that a +{transform} created in a previous version of the {stack} will be able to start and operate in a future version. Neither can support be provided for -{dataframe-transform} tasks to be able to operate in a cluster with mixed node +{transform} tasks to be able to operate in a cluster with mixed node versions. -Please note that the output of a {dataframe-transform} is persisted to a +Please note that the output of a {transform} is persisted to a destination index. This is a normal {es} index and is not affected by the beta status. @@ -29,7 +29,7 @@ status. === {dataframe-cap} UI will not work during a rolling upgrade from 7.2 If your cluster contains mixed version nodes, for example during a rolling -upgrade from 7.2 to a newer version, and {dataframe-transforms} have been +upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2, the {dataframe} UI will not work. Please wait until all nodes have been upgraded to the newer version before using the {dataframe} UI. @@ -46,23 +46,23 @@ table. [[df-ccs-limitations]] === {ccs-cap} is not supported -{ccs-cap} is not supported for {dataframe-transforms}. +{ccs-cap} is not supported for {transforms}. [float] [[df-kibana-limitations]] -=== Up to 1,000 {dataframe-transforms} are supported +=== Up to 1,000 {transforms} are supported -A single cluster will support up to 1,000 {dataframe-transforms}. +A single cluster will support up to 1,000 {transforms}. When using the -{ref}/get-data-frame-transform.html[GET {dataframe-transforms} API] a total -`count` of transforms is returned. Use the `size` and `from` parameters to +{ref}/get-data-frame-transform.html[GET {transforms} API] a total +`count` of {transforms} is returned. Use the `size` and `from` parameters to enumerate through the full list. [float] [[df-aggresponse-limitations]] === Aggregation responses may be incompatible with destination index mappings -When a {dataframe-transform} is first started, it will deduce the mappings +When a {transform} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of the source index and the aggregations used. If the fields are derived from {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[`scripted_metrics`] @@ -72,15 +72,15 @@ deduced mappings may be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. Please check {es} logs if you think this may have occurred. As a workaround, you may define custom mappings prior to starting the -{dataframe-transform}. For example, +{transform}. For example, {ref}/indices-create-index.html[create a custom destination index] or {ref}/indices-templates.html[define an index template]. [float] [[df-batch-limitations]] -=== Batch {dataframe-transforms} may not account for changed documents +=== Batch {transforms} may not account for changed documents -A batch {dataframe-transform} uses a +A batch {transform} uses a {ref}/search-aggregations-bucket-composite-aggregation.html[composite aggregation] which allows efficient pagination through all buckets. Composite aggregations do not yet support a search context, therefore if the source data is changed @@ -91,8 +91,8 @@ results may not include these changes. [[df-consistency-limitations]] === {cdataframe-cap} consistency does not account for deleted or updated documents -While the process for {cdataframe-transforms} allows the continual recalculation -of the {dataframe-transform} as new data is being ingested, it does also have +While the process for {transforms} allows the continual recalculation +of the {transform} as new data is being ingested, it does also have some limitations. Changed entities will only be identified if their time field @@ -106,7 +106,7 @@ composite aggregation performed in consecutive checkpoint processing will search over different source data, and entities that only existed in the deleted index will not be removed from the {dataframe} destination index. -Depending on your use case, you may wish to recreate the {dataframe-transform} +Depending on your use case, you may wish to recreate the {transform} entirely after deletions. Alternatively, if your use case is tolerant to historical archiving, you may wish to include a max ingest timestamp in your aggregation. This will allow you to exclude results that have not been recently @@ -115,9 +115,9 @@ updated when viewing the {dataframe} destination index. [float] [[df-deletion-limitations]] -=== Deleting a {dataframe-transform} does not delete the {dataframe} destination index or {kib} index pattern +=== Deleting a {transform} does not delete the {dataframe} destination index or {kib} index pattern -When deleting a {dataframe-transform} using `DELETE _data_frame/transforms/index` +When deleting a {transform} using `DELETE _data_frame/transforms/index` neither the {dataframe} destination index nor the {kib} index pattern, should one have been created, are deleted. These objects must be deleted separately. @@ -125,9 +125,9 @@ one have been created, are deleted. These objects must be deleted separately. [[df-aggregation-page-limitations]] === Handling dynamic adjustment of aggregation page size -During the development of {dataframe-transforms}, control was favoured over +During the development of {transforms}, control was favoured over performance. In the design considerations, it is preferred for the -{dataframe-transform} to take longer to complete quietly in the background +{transform} to take longer to complete quietly in the background rather than to finish quickly and take precedence in resource consumption. Composite aggregations are well suited for high cardinality data enabling @@ -135,21 +135,21 @@ pagination through results. If a {ref}/circuit-breaker.html[circuit breaker] memory exception occurs when performing the composite aggregated search then we try again reducing the number of buckets requested. This circuit breaker is calculated based upon all activity within the cluster, not just activity from -{dataframe-transforms}, so it therefore may only be a temporary resource +{transforms}, so it therefore may only be a temporary resource availability issue. -For a batch {dataframe-transform}, the number of buckets requested is only ever +For a batch {transform}, the number of buckets requested is only ever adjusted downwards. The lowering of value may result in a longer duration for the -transform checkpoint to complete. For {cdataframes}, the number of +{transform} checkpoint to complete. For {cdataframes}, the number of buckets requested is reset back to its default at the start of every checkpoint and it is possible for circuit breaker exceptions to occur repeatedly in the {es} logs. -The {dataframe-transform} retrieves data in batches which means it calculates +The {transform} retrieves data in batches which means it calculates several buckets at once. Per default this is 500 buckets per search/index operation. The default can be changed using `max_page_search_size` and the minimum value is 10. If failures still occur once the number of buckets -requested has been reduced to its minimum, then the {dataframe-transform} will +requested has been reduced to its minimum, then the {transform} will be set to a failed state. [float] @@ -158,7 +158,7 @@ be set to a failed state. For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a -{ref}/query-dsl-terms-query.html[terms query] to the {dataframe-transform} +{ref}/query-dsl-terms-query.html[terms query] to the {transform} composite aggregation, one page at a time. Then updates are applied to the destination index for each page of entities. @@ -170,10 +170,10 @@ The index setting {ref}/index-modules.html#dynamic-index-settings[`index.max_terms_count`] defines the maximum number of terms that can be used in a terms query. The default value is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the -transform will fail. +{transform} will fail. Using smaller values for `max_page_search_size` may result in a longer duration -for the transform checkpoint to complete. +for the {transform} checkpoint to complete. [float] [[df-scheduling-limitations]] @@ -183,22 +183,22 @@ A {cdataframe} periodically checks for changes to source data. The functionality of the scheduler is currently limited to a basic periodic timer which can be within the `frequency` range from 1s to 1h. The default is 1m. This is designed to run little and often. When choosing a `frequency` for this timer consider -your ingest rate along with the impact that the {dataframe-transform} +your ingest rate along with the impact that the {transform} search/index operations has other users in your cluster. Also note that retries occur at `frequency` interval. [float] [[df-failed-limitations]] -=== Handling of failed {dataframe-transforms} +=== Handling of failed {transforms} -Failed {dataframe-transforms} remain as a persistent task and should be handled +Failed {transforms} remain as a persistent task and should be handled appropriately, either by deleting it or by resolving the root cause of the failure and re-starting. -When using the API to delete a failed {dataframe-transform}, first stop it using +When using the API to delete a failed {transform}, first stop it using `_stop?force=true`, then delete it. -If starting a failed {dataframe-transform}, after the root cause has been +If starting a failed {transform}, after the root cause has been resolved, the `_start?force=true` parameter must be specified. [float] @@ -208,7 +208,7 @@ resolved, the `_start?force=true` parameter must be specified. After a document is indexed, there is a very small delay until it is available to search. -A {cdataframe-transform} periodically checks for changed entities between the +A {ctransform} periodically checks for changed entities between the time since it last checked and `now` minus `sync.time.delay`. This time window moves without overlapping. If the timestamp of a recently indexed document falls within this time window but this document is not yet available to search then diff --git a/docs/reference/transform/overview.asciidoc b/docs/reference/transform/overview.asciidoc index c0a7856e283..ddf9dd2057d 100644 --- a/docs/reference/transform/overview.asciidoc +++ b/docs/reference/transform/overview.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[ml-transform-overview]] -== {dataframe-transform-cap} overview +== {transform-cap} overview ++++ Overview ++++ @@ -19,8 +19,8 @@ you to summarize this data, bringing it into an organized, more analysis-friendly format. For example, you can summarize all the purchases of a single customer. -You can create {dataframes} by using {dataframe-transforms}. -{dataframe-transforms-cap} enable you to define a pivot, which is a set of +You can create {dataframes} by using {transforms}. +{transforms-cap} enable you to define a pivot, which is a set of features that transform the index into a different, more digestible format. Pivoting results in a summary of your data, which is the {dataframe}. @@ -33,19 +33,19 @@ The second step is deciding how you want to aggregate the grouped data. When using aggregations, you practically ask questions about the index. There are different types of aggregations, each with its own purpose and output. To learn more about the supported aggregations and group-by fields, see -{ref}/data-frame-transform-resource.html[{dataframe-transform-cap} resources]. +{ref}/data-frame-transform-resource.html[{transform-cap} resources]. As an optional step, you can also add a query to further limit the scope of the aggregation. -The {dataframe-transform} performs a composite aggregation that +The {transform} performs a composite aggregation that paginates through all the data defined by the source index query. The output of the aggregation is stored in a destination index. Each time the -{dataframe-transform} queries the source index, it creates a _checkpoint_. You -can decide whether you want the {dataframe-transform} to run once (batch -{dataframe-transform}) or continuously ({cdataframe-transform}). A batch -{dataframe-transform} is a single operation that has a single checkpoint. -{cdataframe-transforms-cap} continually increment and process checkpoints as new +{transform} queries the source index, it creates a _checkpoint_. You +can decide whether you want the {transform} to run once (batch +{transform}) or continuously ({transform}). A batch +{transform} is a single operation that has a single checkpoint. +{ctransforms-cap} continually increment and process checkpoints as new source data is ingested. .Example @@ -57,7 +57,7 @@ information (name, gender, location, etc). Your dataset contains all the transac from last year. If you want to check the sales in the different categories in your last fiscal -year, define a {dataframe-transform} that groups the data by the product +year, define a {transform} that groups the data by the product categories (women's shoes, men's clothing, etc.) and the order date. Use the last year as the interval for the order date. Then add a sum aggregation on the ordered quantity. The result is a {dataframe} that shows the number of sold @@ -66,6 +66,6 @@ items in every product category in the last year. [role="screenshot"] image::images/ml-dataframepivot.jpg["Example of a data frame pivot in {kib}"] -IMPORTANT: The {dataframe-transform} leaves your source index intact. It +IMPORTANT: The {transform} leaves your source index intact. It creates a new index that is dedicated to the {dataframe}. diff --git a/docs/reference/transform/troubleshooting.asciidoc b/docs/reference/transform/troubleshooting.asciidoc index 4ea0dd8cc83..485512ff743 100644 --- a/docs/reference/transform/troubleshooting.asciidoc +++ b/docs/reference/transform/troubleshooting.asciidoc @@ -1,5 +1,5 @@ [[dataframe-troubleshooting]] -== Troubleshooting {dataframe-transforms} +== Troubleshooting {transforms} [subs="attributes"] ++++ Troubleshooting @@ -9,21 +9,21 @@ Use the information in this section to troubleshoot common problems. include::{stack-repo-dir}/help.asciidoc[tag=get-help] -If you encounter problems with your {dataframe-transforms}, you can gather more +If you encounter problems with your {transforms}, you can gather more information from the following files and APIs: * Lightweight audit messages are stored in `.data-frame-notifications-*`. Search by your `transform_id`. * The -{ref}/get-data-frame-transform-stats.html[get {dataframe-transform} statistics API] -provides information about the transform status and failures. -* If the {dataframe-transform} exists as a task, you can use the +{ref}/get-data-frame-transform-stats.html[get {transform} statistics API] +provides information about the {transform} status and failures. +* If the {transform} exists as a task, you can use the {ref}/tasks.html[task management API] to gather task information. For example: `GET _tasks?actions=data_frame/transforms*&detailed`. Typically, the task exists -when the transform is in a started or failed state. -* The {es} logs from the node that was running the {dataframe-transform} might +when the {transform} is in a started or failed state. +* The {es} logs from the node that was running the {transform} might also contain useful information. You can identify the node from the notification messages. Alternatively, if the task still exists, you can get that information -from the get {dataframe-transform} statistics API. For more information, see +from the get {transform} statistics API. For more information, see {ref}/logging.html[Logging configuration].