parent
ba01540d7e
commit
9ba017f699
|
@ -11,7 +11,7 @@
|
||||||
include::rollup/index.asciidoc[tag=rollup-intro]
|
include::rollup/index.asciidoc[tag=rollup-intro]
|
||||||
* <<transforms,Transforming your data>>
|
* <<transforms,Transforming your data>>
|
||||||
+
|
+
|
||||||
include::transform/index.asciidoc[tag=transform-intro]
|
include::transform/transforms.asciidoc[tag=transform-intro]
|
||||||
|
|
||||||
--
|
--
|
||||||
|
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[transform-api-quickref]]
|
[[transform-api-quickref]]
|
||||||
=== API quick reference
|
= API quick reference
|
||||||
|
|
||||||
All {transform} endpoints have the following base:
|
All {transform} endpoints have the following base:
|
||||||
|
|
||||||
|
@ -10,13 +10,13 @@ _transform/
|
||||||
----
|
----
|
||||||
// NOTCONSOLE
|
// NOTCONSOLE
|
||||||
|
|
||||||
* {ref}/put-transform.html[Create {transforms}]
|
* <<put-transform,Create {transforms}>>
|
||||||
* {ref}/delete-transform.html[Delete {transforms}]
|
* <<delete-transform,Delete {transforms}>>
|
||||||
* {ref}/get-transform.html[Get {transforms}]
|
* <<get-transform,Get {transforms}>>
|
||||||
* {ref}/get-transform-stats.html[Get {transforms} statistics]
|
* <<get-transform-stats,Get {transforms} statistics>>
|
||||||
* {ref}/preview-transform.html[Preview {transforms}]
|
* <<preview-transform,Preview {transforms}>>
|
||||||
* {ref}/start-transform.html[Start {transforms}]
|
* <<start-transform,Start {transforms}>>
|
||||||
* {ref}/stop-transform.html[Stop {transforms}]
|
* <<stop-transform,Stop {transforms}>>
|
||||||
* {ref}/update-transform.html[Update {transforms}]
|
* <<update-transform,Update {transforms}>>
|
||||||
|
|
||||||
For the full list, see {ref}/transform-apis.html[{transform-cap} APIs].
|
For the full list, see <<transform-apis>>.
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[delete-transform]]
|
[[delete-transform]]
|
||||||
=== Delete {transform} API
|
= Delete {transform} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,12 +11,12 @@
|
||||||
Deletes an existing {transform}.
|
Deletes an existing {transform}.
|
||||||
|
|
||||||
[[delete-transform-request]]
|
[[delete-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`DELETE _transform/<transform_id>`
|
`DELETE _transform/<transform_id>`
|
||||||
|
|
||||||
[[delete-transform-prereqs]]
|
[[delete-transform-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
* Before you can delete the {transform}, you must stop it.
|
* Before you can delete the {transform}, you must stop it.
|
||||||
|
|
||||||
|
@ -31,14 +31,14 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[delete-transform-path-parms]]
|
[[delete-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Required, string)
|
(Required, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[[delete-transform-query-parms]]
|
[[delete-transform-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`force`::
|
`force`::
|
||||||
(Optional, boolean) When `true`, the {transform} is deleted regardless of its
|
(Optional, boolean) When `true`, the {transform} is deleted regardless of its
|
||||||
|
@ -46,7 +46,7 @@ current state. The default value is `false`, meaning that the {transform} must b
|
||||||
`stopped` before it can be deleted.
|
`stopped` before it can be deleted.
|
||||||
|
|
||||||
[[delete-transform-examples]]
|
[[delete-transform-examples]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[get-transform-stats]]
|
[[get-transform-stats]]
|
||||||
=== Get {transform} statistics API
|
= Get {transform} statistics API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -12,7 +12,7 @@ Retrieves usage information for {transforms}.
|
||||||
|
|
||||||
|
|
||||||
[[get-transform-stats-request]]
|
[[get-transform-stats-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`GET _transform/<transform_id>/_stats`
|
`GET _transform/<transform_id>/_stats`
|
||||||
|
|
||||||
|
@ -26,7 +26,7 @@ Retrieves usage information for {transforms}.
|
||||||
|
|
||||||
|
|
||||||
[[get-transform-stats-prereqs]]
|
[[get-transform-stats-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
privileges:
|
privileges:
|
||||||
|
@ -39,7 +39,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[get-transform-stats-desc]]
|
[[get-transform-stats-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
You can get statistics for multiple {transforms} in a single API
|
You can get statistics for multiple {transforms} in a single API
|
||||||
request by using a comma-separated list of identifiers or a wildcard expression.
|
request by using a comma-separated list of identifiers or a wildcard expression.
|
||||||
|
@ -49,7 +49,7 @@ specifying `*` as the `<transform_id>`, or by omitting the
|
||||||
|
|
||||||
|
|
||||||
[[get-transform-stats-path-parms]]
|
[[get-transform-stats-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Optional, string)
|
(Optional, string)
|
||||||
|
@ -57,7 +57,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id-wildcard]
|
||||||
|
|
||||||
|
|
||||||
[[get-transform-stats-query-parms]]
|
[[get-transform-stats-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`allow_no_match`::
|
`allow_no_match`::
|
||||||
(Optional, boolean)
|
(Optional, boolean)
|
||||||
|
@ -73,7 +73,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=size-transforms]
|
||||||
|
|
||||||
[role="child_attributes"]
|
[role="child_attributes"]
|
||||||
[[get-transform-stats-response]]
|
[[get-transform-stats-response]]
|
||||||
==== {api-response-body-title}
|
== {api-response-body-title}
|
||||||
|
|
||||||
The API returns an array of statistics objects for {transforms}, which are
|
The API returns an array of statistics objects for {transforms}, which are
|
||||||
sorted by the `id` value in ascending order. All of these properties are
|
sorted by the `id` value in ascending order. All of these properties are
|
||||||
|
@ -251,14 +251,14 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=trigger-count]
|
||||||
//End stats
|
//End stats
|
||||||
|
|
||||||
[[get-transform-stats-response-codes]]
|
[[get-transform-stats-response-codes]]
|
||||||
==== {api-response-codes-title}
|
== {api-response-codes-title}
|
||||||
|
|
||||||
`404` (Missing resources)::
|
`404` (Missing resources)::
|
||||||
If `allow_no_match` is `false`, this code indicates that there are no
|
If `allow_no_match` is `false`, this code indicates that there are no
|
||||||
resources that match the request or only partial matches for the request.
|
resources that match the request or only partial matches for the request.
|
||||||
|
|
||||||
[[get-transform-stats-example]]
|
[[get-transform-stats-example]]
|
||||||
==== Examples
|
== Examples
|
||||||
|
|
||||||
The following example skips for the first five {transforms} and
|
The following example skips for the first five {transforms} and
|
||||||
gets usage information for a maximum of ten results:
|
gets usage information for a maximum of ten results:
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[get-transform]]
|
[[get-transform]]
|
||||||
=== Get {transforms} API
|
= Get {transforms} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,7 +11,7 @@
|
||||||
Retrieves configuration information for {transforms}.
|
Retrieves configuration information for {transforms}.
|
||||||
|
|
||||||
[[get-transform-request]]
|
[[get-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`GET _transform/<transform_id>` +
|
`GET _transform/<transform_id>` +
|
||||||
|
|
||||||
|
@ -24,7 +24,7 @@ Retrieves configuration information for {transforms}.
|
||||||
`GET _transform/*`
|
`GET _transform/*`
|
||||||
|
|
||||||
[[get-transform-prereqs]]
|
[[get-transform-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
privileges:
|
privileges:
|
||||||
|
@ -36,7 +36,7 @@ The built-in `transform_user` role has this privilege.
|
||||||
For more information, see <<security-privileges>> and <<built-in-roles>>.
|
For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
[[get-transform-desc]]
|
[[get-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
You can get information for multiple {transforms} in a single API
|
You can get information for multiple {transforms} in a single API
|
||||||
request by using a comma-separated list of identifiers or a wildcard expression.
|
request by using a comma-separated list of identifiers or a wildcard expression.
|
||||||
|
@ -44,14 +44,14 @@ You can get information for all {transforms} by using `_all`, by
|
||||||
specifying `*` as the `<transform_id>`, or by omitting the `<transform_id>`.
|
specifying `*` as the `<transform_id>`, or by omitting the `<transform_id>`.
|
||||||
|
|
||||||
[[get-transform-path-parms]]
|
[[get-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Optional, string)
|
(Optional, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id-wildcard]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id-wildcard]
|
||||||
|
|
||||||
[[get-transform-query-parms]]
|
[[get-transform-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`allow_no_match`::
|
`allow_no_match`::
|
||||||
(Optional, boolean)
|
(Optional, boolean)
|
||||||
|
@ -66,7 +66,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=from-transforms]
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=size-transforms]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=size-transforms]
|
||||||
|
|
||||||
[[get-transform-response]]
|
[[get-transform-response]]
|
||||||
==== {api-response-body-title}
|
== {api-response-body-title}
|
||||||
|
|
||||||
The API returns an array of {transform} resources, which are sorted by the `id`
|
The API returns an array of {transform} resources, which are sorted by the `id`
|
||||||
value in ascending order. For the full list of properties, see
|
value in ascending order. For the full list of properties, see
|
||||||
|
@ -81,14 +81,14 @@ This property is informational; you cannot change its value.
|
||||||
created.
|
created.
|
||||||
|
|
||||||
[[get-transform-response-codes]]
|
[[get-transform-response-codes]]
|
||||||
==== {api-response-codes-title}
|
== {api-response-codes-title}
|
||||||
|
|
||||||
`404` (Missing resources)::
|
`404` (Missing resources)::
|
||||||
If `allow_no_match` is `false`, this code indicates that there are no
|
If `allow_no_match` is `false`, this code indicates that there are no
|
||||||
resources that match the request or only partial matches for the request.
|
resources that match the request or only partial matches for the request.
|
||||||
|
|
||||||
[[get-transform-example]]
|
[[get-transform-example]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
The following example retrieves information about a maximum of ten {transforms}:
|
The following example retrieves information about a maximum of ten {transforms}:
|
||||||
|
|
||||||
|
|
|
@ -1,29 +1,17 @@
|
||||||
[role="xpack"]
|
include::transform-apis.asciidoc[leveloffset=+1]
|
||||||
[testenv="basic"]
|
|
||||||
[[transform-apis]]
|
|
||||||
== {transform-cap} APIs
|
|
||||||
|
|
||||||
* <<put-transform>>
|
|
||||||
* <<delete-transform>>
|
|
||||||
* <<get-transform>>
|
|
||||||
* <<get-transform-stats>>
|
|
||||||
* <<preview-transform>>
|
|
||||||
* <<start-transform>>
|
|
||||||
* <<stop-transform>>
|
|
||||||
* <<update-transform>>
|
|
||||||
|
|
||||||
//CREATE
|
//CREATE
|
||||||
include::put-transform.asciidoc[]
|
include::put-transform.asciidoc[leveloffset=+2]
|
||||||
//DELETE
|
//DELETE
|
||||||
include::delete-transform.asciidoc[]
|
include::delete-transform.asciidoc[leveloffset=+2]
|
||||||
//GET
|
//GET
|
||||||
include::get-transform.asciidoc[]
|
include::get-transform.asciidoc[leveloffset=+2]
|
||||||
include::get-transform-stats.asciidoc[]
|
include::get-transform-stats.asciidoc[leveloffset=+2]
|
||||||
//PREVIEW
|
//PREVIEW
|
||||||
include::preview-transform.asciidoc[]
|
include::preview-transform.asciidoc[leveloffset=+2]
|
||||||
//START
|
//START
|
||||||
include::start-transform.asciidoc[]
|
include::start-transform.asciidoc[leveloffset=+2]
|
||||||
//STOP
|
//STOP
|
||||||
include::stop-transform.asciidoc[]
|
include::stop-transform.asciidoc[leveloffset=+2]
|
||||||
//UPDATE
|
//UPDATE
|
||||||
include::update-transform.asciidoc[]
|
include::update-transform.asciidoc[leveloffset=+2]
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[preview-transform]]
|
[[preview-transform]]
|
||||||
=== Preview {transform} API
|
= Preview {transform} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,12 +11,12 @@
|
||||||
Previews a {transform}.
|
Previews a {transform}.
|
||||||
|
|
||||||
[[preview-transform-request]]
|
[[preview-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`POST _transform/_preview`
|
`POST _transform/_preview`
|
||||||
|
|
||||||
[[preview-transform-prereq]]
|
[[preview-transform-prereq]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
privileges:
|
privileges:
|
||||||
|
@ -30,7 +30,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[preview-transform-desc]]
|
[[preview-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
This API generates a preview of the results that you will get when you run the
|
This API generates a preview of the results that you will get when you run the
|
||||||
<<put-transform,create {transforms} API>> with the same
|
<<put-transform,create {transforms} API>> with the same
|
||||||
|
@ -49,7 +49,7 @@ or an index template with your preferred mappings before you start the
|
||||||
|
|
||||||
[role="child_attributes"]
|
[role="child_attributes"]
|
||||||
[[preview-transform-request-body]]
|
[[preview-transform-request-body]]
|
||||||
==== {api-request-body-title}
|
== {api-request-body-title}
|
||||||
|
|
||||||
|
|
||||||
`description`::
|
`description`::
|
||||||
|
@ -164,7 +164,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-max
|
||||||
|
|
||||||
[role="child_attributes"]
|
[role="child_attributes"]
|
||||||
[[preview-transform-response]]
|
[[preview-transform-response]]
|
||||||
==== {api-response-body-title}
|
== {api-response-body-title}
|
||||||
|
|
||||||
`preview`::
|
`preview`::
|
||||||
(array) An array of documents. In particular, they are the JSON representation
|
(array) An array of documents. In particular, they are the JSON representation
|
||||||
|
@ -190,7 +190,7 @@ of the documents that would be created in the destination index by the
|
||||||
====
|
====
|
||||||
//End generated_dest_index
|
//End generated_dest_index
|
||||||
|
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[put-transform]]
|
[[put-transform]]
|
||||||
=== Create {transform} API
|
= Create {transform} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,12 +11,12 @@
|
||||||
Instantiates a {transform}.
|
Instantiates a {transform}.
|
||||||
|
|
||||||
[[put-transform-request]]
|
[[put-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`PUT _transform/<transform_id>`
|
`PUT _transform/<transform_id>`
|
||||||
|
|
||||||
[[put-transform-prereqs]]
|
[[put-transform-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
built-in roles and privileges:
|
built-in roles and privileges:
|
||||||
|
@ -32,7 +32,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[put-transform-desc]]
|
[[put-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
This API defines a {transform}, which copies data from source indices,
|
This API defines a {transform}, which copies data from source indices,
|
||||||
transforms it, and persists it into an entity-centric destination index. The
|
transforms it, and persists it into an entity-centric destination index. The
|
||||||
|
@ -64,14 +64,14 @@ IMPORTANT: You must use {kib} or this API to create a {transform}.
|
||||||
`.data-frame-internal*` indices.
|
`.data-frame-internal*` indices.
|
||||||
|
|
||||||
[[put-transform-path-parms]]
|
[[put-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Required, string)
|
(Required, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[[put-transform-query-parms]]
|
[[put-transform-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`defer_validation`::
|
`defer_validation`::
|
||||||
(Optional, boolean) When `true`, deferrable validations are not run. This
|
(Optional, boolean) When `true`, deferrable validations are not run. This
|
||||||
|
@ -80,7 +80,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[role="child_attributes"]
|
[role="child_attributes"]
|
||||||
[[put-transform-request-body]]
|
[[put-transform-request-body]]
|
||||||
==== {api-request-body-title}
|
== {api-request-body-title}
|
||||||
|
|
||||||
`description`::
|
`description`::
|
||||||
(Optional, string) Free text description of the {transform}.
|
(Optional, string) Free text description of the {transform}.
|
||||||
|
@ -204,7 +204,7 @@ delays.
|
||||||
|
|
||||||
|
|
||||||
[[put-transform-example]]
|
[[put-transform-example]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[start-transform]]
|
[[start-transform]]
|
||||||
=== Start {transform} API
|
= Start {transform} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,12 +11,12 @@
|
||||||
Starts one or more {transforms}.
|
Starts one or more {transforms}.
|
||||||
|
|
||||||
[[start-transform-request]]
|
[[start-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`POST _transform/<transform_id>/_start`
|
`POST _transform/<transform_id>/_start`
|
||||||
|
|
||||||
[[start-transform-prereqs]]
|
[[start-transform-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
built-in roles and privileges:
|
built-in roles and privileges:
|
||||||
|
@ -28,7 +28,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[start-transform-desc]]
|
[[start-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
When you start a {transform}, it creates the destination index if it
|
When you start a {transform}, it creates the destination index if it
|
||||||
does not already exist. The `number_of_shards` is set to `1` and the
|
does not already exist. The `number_of_shards` is set to `1` and the
|
||||||
|
@ -51,14 +51,14 @@ required privileges on the source and destination indices, the
|
||||||
{transform} fails when it attempts unauthorized operations.
|
{transform} fails when it attempts unauthorized operations.
|
||||||
|
|
||||||
[[start-transform-path-parms]]
|
[[start-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Required, string)
|
(Required, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[[start-transform-example]]
|
[[start-transform-example]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[stop-transform]]
|
[[stop-transform]]
|
||||||
=== Stop {transforms} API
|
= Stop {transforms} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -12,7 +12,7 @@ Stops one or more {transforms}.
|
||||||
|
|
||||||
|
|
||||||
[[stop-transform-request]]
|
[[stop-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`POST _transform/<transform_id>/_stop` +
|
`POST _transform/<transform_id>/_stop` +
|
||||||
|
|
||||||
|
@ -22,7 +22,7 @@ Stops one or more {transforms}.
|
||||||
|
|
||||||
|
|
||||||
[[stop-transform-prereq]]
|
[[stop-transform-prereq]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
built-in roles and privileges:
|
built-in roles and privileges:
|
||||||
|
@ -35,7 +35,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[stop-transform-desc]]
|
[[stop-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
You can stop multiple {transforms} in a single API request by using a
|
You can stop multiple {transforms} in a single API request by using a
|
||||||
comma-separated list of {transforms} or a wildcard expression.
|
comma-separated list of {transforms} or a wildcard expression.
|
||||||
|
@ -44,14 +44,14 @@ All {transforms} can be stopped by using `_all` or `*` as the
|
||||||
|
|
||||||
|
|
||||||
[[stop-transform-path-parms]]
|
[[stop-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Required, string)
|
(Required, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[[stop-transform-query-parms]]
|
[[stop-transform-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`allow_no_match`::
|
`allow_no_match`::
|
||||||
(Optional, boolean)
|
(Optional, boolean)
|
||||||
|
@ -81,7 +81,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-match-transfo
|
||||||
stops as soon as possible. Defaults to `false`.
|
stops as soon as possible. Defaults to `false`.
|
||||||
|
|
||||||
[[stop-transform-response-codes]]
|
[[stop-transform-response-codes]]
|
||||||
==== {api-response-codes-title}
|
== {api-response-codes-title}
|
||||||
|
|
||||||
`404` (Missing resources)::
|
`404` (Missing resources)::
|
||||||
If `allow_no_match` is `false`, this code indicates that there are no
|
If `allow_no_match` is `false`, this code indicates that there are no
|
||||||
|
@ -89,7 +89,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-match-transfo
|
||||||
|
|
||||||
|
|
||||||
[[stop-transform-example]]
|
[[stop-transform-example]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -0,0 +1,13 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[testenv="basic"]
|
||||||
|
[[transform-apis]]
|
||||||
|
= {transform-cap} APIs
|
||||||
|
|
||||||
|
* <<put-transform>>
|
||||||
|
* <<delete-transform>>
|
||||||
|
* <<get-transform>>
|
||||||
|
* <<get-transform-stats>>
|
||||||
|
* <<preview-transform>>
|
||||||
|
* <<start-transform>>
|
||||||
|
* <<stop-transform>>
|
||||||
|
* <<update-transform>>
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[update-transform]]
|
[[update-transform]]
|
||||||
=== Update {transform} API
|
= Update {transform} API
|
||||||
|
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
|
@ -11,12 +11,12 @@
|
||||||
Updates certain properties of a {transform}.
|
Updates certain properties of a {transform}.
|
||||||
|
|
||||||
[[update-transform-request]]
|
[[update-transform-request]]
|
||||||
==== {api-request-title}
|
== {api-request-title}
|
||||||
|
|
||||||
`POST _transform/<transform_id>/_update`
|
`POST _transform/<transform_id>/_update`
|
||||||
|
|
||||||
[[update-transform-prereqs]]
|
[[update-transform-prereqs]]
|
||||||
==== {api-prereq-title}
|
== {api-prereq-title}
|
||||||
|
|
||||||
If the {es} {security-features} are enabled, you must have the following
|
If the {es} {security-features} are enabled, you must have the following
|
||||||
built-in roles and privileges:
|
built-in roles and privileges:
|
||||||
|
@ -31,7 +31,7 @@ For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||||||
|
|
||||||
|
|
||||||
[[update-transform-desc]]
|
[[update-transform-desc]]
|
||||||
==== {api-description-title}
|
== {api-description-title}
|
||||||
|
|
||||||
This API updates an existing {transform}. The list of properties that you can
|
This API updates an existing {transform}. The list of properties that you can
|
||||||
update is a subset of the list that you can define when you create a {transform}.
|
update is a subset of the list that you can define when you create a {transform}.
|
||||||
|
@ -58,14 +58,14 @@ give users any privileges on `.data-frame-internal*` indices.
|
||||||
====
|
====
|
||||||
|
|
||||||
[[update-transform-path-parms]]
|
[[update-transform-path-parms]]
|
||||||
==== {api-path-parms-title}
|
== {api-path-parms-title}
|
||||||
|
|
||||||
`<transform_id>`::
|
`<transform_id>`::
|
||||||
(Required, string)
|
(Required, string)
|
||||||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[[update-transform-query-parms]]
|
[[update-transform-query-parms]]
|
||||||
==== {api-query-parms-title}
|
== {api-query-parms-title}
|
||||||
|
|
||||||
`defer_validation`::
|
`defer_validation`::
|
||||||
(Optional, boolean) When `true`, deferrable validations are not run. This
|
(Optional, boolean) When `true`, deferrable validations are not run. This
|
||||||
|
@ -74,7 +74,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-id]
|
||||||
|
|
||||||
[role="child_attributes"]
|
[role="child_attributes"]
|
||||||
[[update-transform-request-body]]
|
[[update-transform-request-body]]
|
||||||
==== {api-request-body-title}
|
== {api-request-body-title}
|
||||||
|
|
||||||
`description`::
|
`description`::
|
||||||
(Optional, string) Free text description of the {transform}.
|
(Optional, string) Free text description of the {transform}.
|
||||||
|
@ -177,7 +177,7 @@ delays.
|
||||||
//End sync
|
//End sync
|
||||||
|
|
||||||
[[update-transform-example]]
|
[[update-transform-example]]
|
||||||
==== {api-examples-title}
|
== {api-examples-title}
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[transform-checkpoints]]
|
[[transform-checkpoints]]
|
||||||
=== How {transform} checkpoints work
|
= How {transform} checkpoints work
|
||||||
++++
|
++++
|
||||||
<titleabbrev>How checkpoints work</titleabbrev>
|
<titleabbrev>How checkpoints work</titleabbrev>
|
||||||
++++
|
++++
|
||||||
|
@ -53,7 +53,7 @@ TIP: If the cluster experiences unsuitable performance degradation due to the
|
||||||
|
|
||||||
[discrete]
|
[discrete]
|
||||||
[[ml-transform-checkpoint-errors]]
|
[[ml-transform-checkpoint-errors]]
|
||||||
==== Error handling
|
== Error handling
|
||||||
|
|
||||||
Failures in {transforms} tend to be related to searching or indexing.
|
Failures in {transforms} tend to be related to searching or indexing.
|
||||||
To increase the resiliency of {transforms}, the cursor positions of
|
To increase the resiliency of {transforms}, the cursor positions of
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[ecommerce-transforms]]
|
[[ecommerce-transforms]]
|
||||||
=== Tutorial: Transforming the eCommerce sample data
|
= Tutorial: Transforming the eCommerce sample data
|
||||||
|
|
||||||
<<transforms,{transforms-cap}>> enable you to retrieve information
|
<<transforms,{transforms-cap}>> enable you to retrieve information
|
||||||
from an {es} index, transform it, and store it in another index. Let's use the
|
from an {es} index, transform it, and store it in another index. Let's use the
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[transform-examples]]
|
[[transform-examples]]
|
||||||
=== {transform-cap} examples
|
= {transform-cap} examples
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Examples</titleabbrev>
|
<titleabbrev>Examples</titleabbrev>
|
||||||
++++
|
++++
|
||||||
|
@ -17,7 +17,7 @@ step-by-step example, see
|
||||||
* <<example-clientips>>
|
* <<example-clientips>>
|
||||||
|
|
||||||
[[example-best-customers]]
|
[[example-best-customers]]
|
||||||
==== Finding your best customers
|
== Finding your best customers
|
||||||
|
|
||||||
In this example, we use the eCommerce orders sample dataset to find the
|
In this example, we use the eCommerce orders sample dataset to find the
|
||||||
customers who spent the most in our hypothetical webshop. Let's transform the
|
customers who spent the most in our hypothetical webshop. Let's transform the
|
||||||
|
@ -101,14 +101,14 @@ navigate data from a customer centric perspective. In some cases, it can even
|
||||||
make creating visualizations much simpler.
|
make creating visualizations much simpler.
|
||||||
|
|
||||||
[[example-airline]]
|
[[example-airline]]
|
||||||
==== Finding air carriers with the most delays
|
== Finding air carriers with the most delays
|
||||||
|
|
||||||
In this example, we use the Flights sample dataset to find out which air carrier
|
In this example, we use the Flights sample dataset to find out which air carrier
|
||||||
had the most delays. First, we filter the source data such that it excludes all
|
had the most delays. First, we filter the source data such that it excludes all
|
||||||
the cancelled flights by using a query filter. Then we transform the data to
|
the cancelled flights by using a query filter. Then we transform the data to
|
||||||
contain the distinct number of flights, the sum of delayed minutes, and the sum
|
contain the distinct number of flights, the sum of delayed minutes, and the sum
|
||||||
of the flight minutes by air carrier. Finally, we use a
|
of the flight minutes by air carrier. Finally, we use a
|
||||||
{ref}/search-aggregations-pipeline-bucket-script-aggregation.html[`bucket_script`]
|
<<search-aggregations-pipeline-bucket-script-aggregation,`bucket_script`>>
|
||||||
to determine what percentage of the flight time was actually delay.
|
to determine what percentage of the flight time was actually delay.
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
|
@ -188,7 +188,7 @@ or flight stats for any of the featured destination or origin airports.
|
||||||
|
|
||||||
|
|
||||||
[[example-clientips]]
|
[[example-clientips]]
|
||||||
==== Finding suspicious client IPs
|
== Finding suspicious client IPs
|
||||||
|
|
||||||
In this example, we use the web log sample dataset to identify suspicious client
|
In this example, we use the web log sample dataset to identify suspicious client
|
||||||
IPs. We transform the data such that the new index contains the sum of bytes and
|
IPs. We transform the data such that the new index contains the sum of bytes and
|
||||||
|
|
|
@ -1,32 +1,11 @@
|
||||||
[role="xpack"]
|
include::transforms.asciidoc[leveloffset=+1]
|
||||||
[[transforms]]
|
include::overview.asciidoc[leveloffset=+2]
|
||||||
== Transforming data
|
include::setup.asciidoc[leveloffset=+2]
|
||||||
|
include::usage.asciidoc[leveloffset=+2]
|
||||||
// tag::transform-intro[]
|
include::checkpoints.asciidoc[leveloffset=+2]
|
||||||
{transforms-cap} enable you to convert existing {es} indices into summarized
|
include::api-quickref.asciidoc[leveloffset=+2]
|
||||||
indices, which provide opportunities for new insights and analytics.
|
include::ecommerce-tutorial.asciidoc[leveloffset=+2]
|
||||||
// end::transform-intro[]
|
include::examples.asciidoc[leveloffset=+2]
|
||||||
For example, you can use {transforms} to pivot your data into entity-centric
|
include::painless-examples.asciidoc[leveloffset=+2]
|
||||||
indices that summarize the behavior of users or sessions or other entities in
|
include::troubleshooting.asciidoc[leveloffset=+2]
|
||||||
your data.
|
include::limitations.asciidoc[leveloffset=+2]
|
||||||
|
|
||||||
* <<transform-overview>>
|
|
||||||
* <<transform-setup>>
|
|
||||||
* <<transform-usage>>
|
|
||||||
* <<transform-api-quickref>>
|
|
||||||
* <<ecommerce-transforms>>
|
|
||||||
* <<transform-examples>>
|
|
||||||
* <<transform-painless-examples>>
|
|
||||||
* <<transform-troubleshooting>>
|
|
||||||
* <<transform-limitations>>
|
|
||||||
|
|
||||||
include::overview.asciidoc[]
|
|
||||||
include::setup.asciidoc[]
|
|
||||||
include::usage.asciidoc[]
|
|
||||||
include::checkpoints.asciidoc[]
|
|
||||||
include::api-quickref.asciidoc[]
|
|
||||||
include::ecommerce-tutorial.asciidoc[]
|
|
||||||
include::examples.asciidoc[]
|
|
||||||
include::painless-examples.asciidoc[]
|
|
||||||
include::troubleshooting.asciidoc[]
|
|
||||||
include::limitations.asciidoc[]
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[transform-limitations]]
|
[[transform-limitations]]
|
||||||
=== {transform-cap} limitations
|
= {transform-cap} limitations
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Limitations</titleabbrev>
|
<titleabbrev>Limitations</titleabbrev>
|
||||||
|
@ -9,77 +9,73 @@
|
||||||
The following limitations and known problems apply to the {version} release of
|
The following limitations and known problems apply to the {version} release of
|
||||||
the Elastic {transform} feature:
|
the Elastic {transform} feature:
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-ui-limitation]]
|
[[transform-ui-limitation]]
|
||||||
==== {transforms-cap} UI will not work during a rolling upgrade from 7.2
|
== {transforms-cap} UI will not work during a rolling upgrade from 7.2
|
||||||
|
|
||||||
If your cluster contains mixed version nodes, for example during a rolling
|
If your cluster contains mixed version nodes, for example during a rolling
|
||||||
upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2,
|
upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2,
|
||||||
the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all
|
the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all
|
||||||
nodes have been upgraded to the newer version before using the {transforms} UI.
|
nodes have been upgraded to the newer version before using the {transforms} UI.
|
||||||
|
|
||||||
[float]
|
[discrete]
|
||||||
[[transform-rolling-upgrade-limitation]]
|
[[transform-rolling-upgrade-limitation]]
|
||||||
==== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3
|
== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3
|
||||||
|
|
||||||
If your cluster contains mixed version nodes, for example during a rolling
|
If your cluster contains mixed version nodes, for example during a rolling
|
||||||
upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped will
|
upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped will
|
||||||
not be reassigned until the upgrade is complete. After the upgrade is done, {transforms}
|
not be reassigned until the upgrade is complete. After the upgrade is done, {transforms}
|
||||||
resume automatically; no action is required.
|
resume automatically; no action is required.
|
||||||
|
|
||||||
[float]
|
[discrete]
|
||||||
[[transform-datatype-limitations]]
|
[[transform-datatype-limitations]]
|
||||||
==== {dataframe-cap} data type limitation
|
== {dataframe-cap} data type limitation
|
||||||
|
|
||||||
{dataframes-cap} do not (yet) support fields containing arrays – in the UI or
|
{dataframes-cap} do not (yet) support fields containing arrays – in the UI or
|
||||||
the API. If you try to create one, the UI will fail to show the source index
|
the API. If you try to create one, the UI will fail to show the source index
|
||||||
table.
|
table.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-kibana-limitations]]
|
[[transform-kibana-limitations]]
|
||||||
==== Up to 1,000 {transforms} are supported
|
== Up to 1,000 {transforms} are supported
|
||||||
|
|
||||||
A single cluster will support up to 1,000 {transforms}. When using the
|
A single cluster will support up to 1,000 {transforms}. When using the
|
||||||
{ref}/get-transform.html[GET {transforms} API] a total `count` of {transforms}
|
<<get-transform,GET {transforms} API>> a total `count` of {transforms}
|
||||||
is returned. Use the `size` and `from` parameters to enumerate through the full
|
is returned. Use the `size` and `from` parameters to enumerate through the full
|
||||||
list.
|
list.
|
||||||
|
|
||||||
[float]
|
[discrete]
|
||||||
[[transform-aggresponse-limitations]]
|
[[transform-aggresponse-limitations]]
|
||||||
==== Aggregation responses may be incompatible with destination index mappings
|
== Aggregation responses may be incompatible with destination index mappings
|
||||||
|
|
||||||
When a {transform} is first started, it will deduce the mappings
|
When a {transform} is first started, it will deduce the mappings
|
||||||
required for the destination index. This process is based on the field types of
|
required for the destination index. This process is based on the field types of
|
||||||
the source index and the aggregations used. If the fields are derived from
|
the source index and the aggregations used. If the fields are derived from
|
||||||
{ref}/search-aggregations-metrics-scripted-metric-aggregation.html[`scripted_metrics`]
|
<<search-aggregations-metrics-scripted-metric-aggregation,`scripted_metrics`>>
|
||||||
or {ref}/search-aggregations-pipeline-bucket-script-aggregation.html[`bucket_scripts`],
|
or <<search-aggregations-pipeline-bucket-script-aggregation,`bucket_scripts`>>,
|
||||||
{ref}/dynamic-mapping.html[dynamic mappings] will be used. In some instances the
|
<<dynamic-mapping,dynamic mappings>> will be used. In some instances the
|
||||||
deduced mappings may be incompatible with the actual data. For example, numeric
|
deduced mappings may be incompatible with the actual data. For example, numeric
|
||||||
overflows might occur or dynamically mapped fields might contain both numbers
|
overflows might occur or dynamically mapped fields might contain both numbers
|
||||||
and strings. Please check {es} logs if you think this may have occurred. As a
|
and strings. Please check {es} logs if you think this may have occurred. As a
|
||||||
workaround, you may define custom mappings prior to starting the
|
workaround, you may define custom mappings prior to starting the
|
||||||
{transform}. For example,
|
{transform}. For example,
|
||||||
{ref}/indices-create-index.html[create a custom destination index] or
|
<<indices-create-index,create a custom destination index>> or
|
||||||
{ref}/indices-templates.html[define an index template].
|
<<indices-templates,define an index template>>.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-batch-limitations]]
|
[[transform-batch-limitations]]
|
||||||
==== Batch {transforms} may not account for changed documents
|
== Batch {transforms} may not account for changed documents
|
||||||
|
|
||||||
A batch {transform} uses a
|
A batch {transform} uses a
|
||||||
{ref}/search-aggregations-bucket-composite-aggregation.html[composite aggregation]
|
<<search-aggregations-bucket-composite-aggregation,composite aggregation>>
|
||||||
which allows efficient pagination through all buckets. Composite aggregations
|
which allows efficient pagination through all buckets. Composite aggregations
|
||||||
do not yet support a search context, therefore if the source data is changed
|
do not yet support a search context, therefore if the source data is changed
|
||||||
(deleted, updated, added) while the batch {dataframe} is in progress, then the
|
(deleted, updated, added) while the batch {dataframe} is in progress, then the
|
||||||
results may not include these changes.
|
results may not include these changes.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-consistency-limitations]]
|
[[transform-consistency-limitations]]
|
||||||
==== {ctransform-cap} consistency does not account for deleted or updated documents
|
== {ctransform-cap} consistency does not account for deleted or updated documents
|
||||||
|
|
||||||
While the process for {transforms} allows the continual recalculation of the
|
While the process for {transforms} allows the continual recalculation of the
|
||||||
{transform} as new data is being ingested, it does also have some limitations.
|
{transform} as new data is being ingested, it does also have some limitations.
|
||||||
|
@ -101,19 +97,17 @@ archiving, you may wish to include a max ingest timestamp in your aggregation.
|
||||||
This will allow you to exclude results that have not been recently updated when
|
This will allow you to exclude results that have not been recently updated when
|
||||||
viewing the destination index.
|
viewing the destination index.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-deletion-limitations]]
|
[[transform-deletion-limitations]]
|
||||||
==== Deleting a {transform} does not delete the destination index or {kib} index pattern
|
== Deleting a {transform} does not delete the destination index or {kib} index pattern
|
||||||
|
|
||||||
When deleting a {transform} using `DELETE _transform/index`
|
When deleting a {transform} using `DELETE _transform/index`
|
||||||
neither the destination index nor the {kib} index pattern, should one have been
|
neither the destination index nor the {kib} index pattern, should one have been
|
||||||
created, are deleted. These objects must be deleted separately.
|
created, are deleted. These objects must be deleted separately.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-aggregation-page-limitations]]
|
[[transform-aggregation-page-limitations]]
|
||||||
==== Handling dynamic adjustment of aggregation page size
|
== Handling dynamic adjustment of aggregation page size
|
||||||
|
|
||||||
During the development of {transforms}, control was favoured over performance.
|
During the development of {transforms}, control was favoured over performance.
|
||||||
In the design considerations, it is preferred for the {transform} to take longer
|
In the design considerations, it is preferred for the {transform} to take longer
|
||||||
|
@ -121,9 +115,9 @@ to complete quietly in the background rather than to finish quickly and take
|
||||||
precedence in resource consumption.
|
precedence in resource consumption.
|
||||||
|
|
||||||
Composite aggregations are well suited for high cardinality data enabling
|
Composite aggregations are well suited for high cardinality data enabling
|
||||||
pagination through results. If a {ref}/circuit-breaker.html[circuit breaker]
|
pagination through results. If a <<circuit-breaker,circuit breaker>> memory
|
||||||
memory exception occurs when performing the composite aggregated search then we
|
exception occurs when performing the composite aggregated search then we try
|
||||||
try again reducing the number of buckets requested. This circuit breaker is
|
again reducing the number of buckets requested. This circuit breaker is
|
||||||
calculated based upon all activity within the cluster, not just activity from
|
calculated based upon all activity within the cluster, not just activity from
|
||||||
{transforms}, so it therefore may only be a temporary resource
|
{transforms}, so it therefore may only be a temporary resource
|
||||||
availability issue.
|
availability issue.
|
||||||
|
@ -140,23 +134,21 @@ default can be changed using `max_page_search_size` and the minimum value is 10.
|
||||||
If failures still occur once the number of buckets requested has been reduced to
|
If failures still occur once the number of buckets requested has been reduced to
|
||||||
its minimum, then the {transform} will be set to a failed state.
|
its minimum, then the {transform} will be set to a failed state.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-dynamic-adjustments-limitations]]
|
[[transform-dynamic-adjustments-limitations]]
|
||||||
==== Handling dynamic adjustments for many terms
|
== Handling dynamic adjustments for many terms
|
||||||
|
|
||||||
For each checkpoint, entities are identified that have changed since the last
|
For each checkpoint, entities are identified that have changed since the last
|
||||||
time the check was performed. This list of changed entities is supplied as a
|
time the check was performed. This list of changed entities is supplied as a
|
||||||
{ref}/query-dsl-terms-query.html[terms query] to the {transform} composite
|
<<query-dsl-terms-query,terms query>> to the {transform} composite aggregation,
|
||||||
aggregation, one page at a time. Then updates are applied to the destination
|
one page at a time. Then updates are applied to the destination index for each
|
||||||
index for each page of entities.
|
page of entities.
|
||||||
|
|
||||||
The page `size` is defined by `max_page_search_size` which is also used to
|
The page `size` is defined by `max_page_search_size` which is also used to
|
||||||
define the number of buckets returned by the composite aggregation search. The
|
define the number of buckets returned by the composite aggregation search. The
|
||||||
default value is 500, the minimum is 10.
|
default value is 500, the minimum is 10.
|
||||||
|
|
||||||
The index setting
|
The index setting <<dynamic-index-settings,`index.max_terms_count`>> defines
|
||||||
{ref}/index-modules.html#dynamic-index-settings[`index.max_terms_count`] defines
|
|
||||||
the maximum number of terms that can be used in a terms query. The default value
|
the maximum number of terms that can be used in a terms query. The default value
|
||||||
is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the
|
is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the
|
||||||
{transform} will fail.
|
{transform} will fail.
|
||||||
|
@ -164,10 +156,9 @@ is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the
|
||||||
Using smaller values for `max_page_search_size` may result in a longer duration
|
Using smaller values for `max_page_search_size` may result in a longer duration
|
||||||
for the {transform} checkpoint to complete.
|
for the {transform} checkpoint to complete.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-scheduling-limitations]]
|
[[transform-scheduling-limitations]]
|
||||||
==== {ctransform-cap} scheduling limitations
|
== {ctransform-cap} scheduling limitations
|
||||||
|
|
||||||
A {ctransform} periodically checks for changes to source data. The functionality
|
A {ctransform} periodically checks for changes to source data. The functionality
|
||||||
of the scheduler is currently limited to a basic periodic timer which can be
|
of the scheduler is currently limited to a basic periodic timer which can be
|
||||||
|
@ -177,10 +168,9 @@ your ingest rate along with the impact that the {transform}
|
||||||
search/index operations has other users in your cluster. Also note that retries
|
search/index operations has other users in your cluster. Also note that retries
|
||||||
occur at `frequency` interval.
|
occur at `frequency` interval.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-failed-limitations]]
|
[[transform-failed-limitations]]
|
||||||
==== Handling of failed {transforms}
|
== Handling of failed {transforms}
|
||||||
|
|
||||||
Failed {transforms} remain as a persistent task and should be handled
|
Failed {transforms} remain as a persistent task and should be handled
|
||||||
appropriately, either by deleting it or by resolving the root cause of the
|
appropriately, either by deleting it or by resolving the root cause of the
|
||||||
|
@ -189,10 +179,9 @@ failure and re-starting.
|
||||||
When using the API to delete a failed {transform}, first stop it using
|
When using the API to delete a failed {transform}, first stop it using
|
||||||
`_stop?force=true`, then delete it.
|
`_stop?force=true`, then delete it.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[float]
|
|
||||||
[[transform-availability-limitations]]
|
[[transform-availability-limitations]]
|
||||||
==== {ctransforms-cap} may give incorrect results if documents are not yet available to search
|
== {ctransforms-cap} may give incorrect results if documents are not yet available to search
|
||||||
|
|
||||||
After a document is indexed, there is a very small delay until it is available
|
After a document is indexed, there is a very small delay until it is available
|
||||||
to search.
|
to search.
|
||||||
|
@ -207,8 +196,9 @@ If using a `sync.time.field` that represents the data ingest time and using a
|
||||||
zero second or very small `sync.time.delay`, then it is more likely that this
|
zero second or very small `sync.time.delay`, then it is more likely that this
|
||||||
issue will occur.
|
issue will occur.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[[transform-date-nanos]]
|
[[transform-date-nanos]]
|
||||||
==== Support for date nanoseconds data type
|
== Support for date nanoseconds data type
|
||||||
|
|
||||||
If your data uses the <<date_nanos,date nanosecond data type>>, aggregations
|
If your data uses the <<date_nanos,date nanosecond data type>>, aggregations
|
||||||
are nonetheless on millisecond resolution. This limitation also affects the
|
are nonetheless on millisecond resolution. This limitation also affects the
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[transform-overview]]
|
[[transform-overview]]
|
||||||
=== {transform-cap} overview
|
= {transform-cap} overview
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Overview</titleabbrev>
|
<titleabbrev>Overview</titleabbrev>
|
||||||
++++
|
++++
|
||||||
|
@ -60,9 +60,9 @@ image::images/pivot-preview.jpg["Example of a {transform} pivot in {kib}"]
|
||||||
IMPORTANT: The {transform} leaves your source index intact. It
|
IMPORTANT: The {transform} leaves your source index intact. It
|
||||||
creates a new index that is dedicated to the transformed data.
|
creates a new index that is dedicated to the transformed data.
|
||||||
|
|
||||||
|
[discrete]
|
||||||
[[transform-performance]]
|
[[transform-performance]]
|
||||||
==== Performance considerations
|
== Performance considerations
|
||||||
|
|
||||||
{transforms-cap} perform search aggregations on the source indices then index
|
{transforms-cap} perform search aggregations on the source indices then index
|
||||||
the results into the destination index. Therefore, a {transform} never takes
|
the results into the destination index. Therefore, a {transform} never takes
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[transform-painless-examples]]
|
[[transform-painless-examples]]
|
||||||
=== Painless examples for {transforms}
|
= Painless examples for {transforms}
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Painless examples for {transforms}</titleabbrev>
|
<titleabbrev>Painless examples for {transforms}</titleabbrev>
|
||||||
++++
|
++++
|
||||||
|
@ -22,10 +22,8 @@ NOTE: While the context of the following examples is the {transform} use case,
|
||||||
the Painless scripts in the snippets below can be used in other {es} search
|
the Painless scripts in the snippets below can be used in other {es} search
|
||||||
aggregations, too.
|
aggregations, too.
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-top-hits]]
|
[[painless-top-hits]]
|
||||||
==== Getting top hits by using scripted metric aggregation
|
== Getting top hits by using scripted metric aggregation
|
||||||
|
|
||||||
This snippet shows how to find the latest document, in other words the document
|
This snippet shows how to find the latest document, in other words the document
|
||||||
with the earliest timestamp. From a technical perspective, it helps to achieve
|
with the earliest timestamp. From a technical perspective, it helps to achieve
|
||||||
|
@ -103,10 +101,8 @@ You can retrieve the last value in a similar way:
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
// NOTCONSOLE
|
// NOTCONSOLE
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-time-features]]
|
[[painless-time-features]]
|
||||||
==== Getting time features by using aggregations
|
== Getting time features by using aggregations
|
||||||
|
|
||||||
This snippet shows how to extract time based features by using Painless in a
|
This snippet shows how to extract time based features by using Painless in a
|
||||||
{transform}. The snippet uses an index where `@timestamp` is defined as a `date`
|
{transform}. The snippet uses an index where `@timestamp` is defined as a `date`
|
||||||
|
@ -149,10 +145,8 @@ type field.
|
||||||
<7> Sets `date` based on the timestamp of the document.
|
<7> Sets `date` based on the timestamp of the document.
|
||||||
<8> Returns the month value from `date`.
|
<8> Returns the month value from `date`.
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-group-by]]
|
[[painless-group-by]]
|
||||||
==== Using Painless in `group_by`
|
== Using Painless in `group_by`
|
||||||
|
|
||||||
It is possible to base the `group_by` property of a {transform} on the output of
|
It is possible to base the `group_by` property of a {transform} on the output of
|
||||||
a script. The following example uses the {kib} sample web logs dataset. The goal
|
a script. The following example uses the {kib} sample web logs dataset. The goal
|
||||||
|
@ -287,14 +281,12 @@ them. The table below shows how normalization modifies the output of the
|
||||||
| "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1" | "firefox"
|
| "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1" | "firefox"
|
||||||
|===
|
|===
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-bucket-script]]
|
[[painless-bucket-script]]
|
||||||
==== Getting duration by using bucket script
|
== Getting duration by using bucket script
|
||||||
|
|
||||||
This example shows you how to get the duration of a session by client IP from a
|
This example shows you how to get the duration of a session by client IP from a
|
||||||
data log by using
|
data log by using
|
||||||
{ref}/search-aggregations-pipeline-bucket-script-aggregation.html[bucket script].
|
<<search-aggregations-pipeline-bucket-script-aggregation,bucket script>>.
|
||||||
The example uses the {kib} sample web logs dataset.
|
The example uses the {kib} sample web logs dataset.
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
|
@ -345,10 +337,8 @@ the buckets you want to use for the variable. In this particular case, `min` and
|
||||||
<3> Finally, the script substracts the start date of the session from the end
|
<3> Finally, the script substracts the start date of the session from the end
|
||||||
date which results in the duration of the session.
|
date which results in the duration of the session.
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-count-http]]
|
[[painless-count-http]]
|
||||||
==== Counting HTTP responses by using scripted metric aggregation
|
== Counting HTTP responses by using scripted metric aggregation
|
||||||
|
|
||||||
You can count the different HTTP response types in a web log data set by using
|
You can count the different HTTP response types in a web log data set by using
|
||||||
scripted metric aggregation as part of the {transform}. The example below
|
scripted metric aggregation as part of the {transform}. The example below
|
||||||
|
@ -405,10 +395,8 @@ properties of the `counts` object; error responses to the error counts, success
|
||||||
responses to the success counts, and other responses to the other counts.
|
responses to the success counts, and other responses to the other counts.
|
||||||
Finally, returns the `counts` array with the response counts.
|
Finally, returns the `counts` array with the response counts.
|
||||||
|
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-compare]]
|
[[painless-compare]]
|
||||||
==== Comparing indices by using scripted metric aggregations
|
== Comparing indices by using scripted metric aggregations
|
||||||
|
|
||||||
This example shows how to compare the content of two indices by a {transform}
|
This example shows how to compare the content of two indices by a {transform}
|
||||||
that uses a scripted metric aggregation.
|
that uses a scripted metric aggregation.
|
||||||
|
@ -475,9 +463,8 @@ not equal, than it reports back a `count_mismatch`. Then it iterates through all
|
||||||
the values of the two indices and compare them. If the values are equal, then it
|
the values of the two indices and compare them. If the values are equal, then it
|
||||||
returns a `match`, otherwise returns a `mismatch`.
|
returns a `match`, otherwise returns a `mismatch`.
|
||||||
|
|
||||||
[discrete]
|
|
||||||
[[painless-web-session]]
|
[[painless-web-session]]
|
||||||
==== Getting web session details by using scripted metric aggregation
|
== Getting web session details by using scripted metric aggregation
|
||||||
|
|
||||||
This example shows how to derive multiple features from a single transaction.
|
This example shows how to derive multiple features from a single transaction.
|
||||||
Let's take a look on the example source document from the data:
|
Let's take a look on the example source document from the data:
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[transform-setup]]
|
[[transform-setup]]
|
||||||
=== Set up {transforms}
|
= Set up {transforms}
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Setup</titleabbrev>
|
<titleabbrev>Setup</titleabbrev>
|
||||||
++++
|
++++
|
||||||
|
@ -13,7 +13,7 @@ To use the {transforms}, you must have the
|
||||||
|
|
||||||
[discrete]
|
[discrete]
|
||||||
[[transform-setup-nodes]]
|
[[transform-setup-nodes]]
|
||||||
==== {transform-cap} nodes
|
== {transform-cap} nodes
|
||||||
|
|
||||||
To use {transforms}, there must be at least one {transform} node in your cluster.
|
To use {transforms}, there must be at least one {transform} node in your cluster.
|
||||||
If you want to control which nodes run {transforms}, add or remove `transform`
|
If you want to control which nodes run {transforms}, add or remove `transform`
|
||||||
|
@ -22,7 +22,7 @@ from the `node.roles` setting on some nodes. For more information, see
|
||||||
|
|
||||||
[discrete]
|
[discrete]
|
||||||
[[transform-privileges]]
|
[[transform-privileges]]
|
||||||
==== Security privileges
|
== Security privileges
|
||||||
|
|
||||||
The {es} {security-features} provide <<built-in-roles,built-in roles>>
|
The {es} {security-features} provide <<built-in-roles,built-in roles>>
|
||||||
and <<security-privileges,privileges>> that make it easier to control
|
and <<security-privileges,privileges>> that make it easier to control
|
||||||
|
|
|
@ -0,0 +1,21 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[transforms]]
|
||||||
|
= Transforming data
|
||||||
|
|
||||||
|
// tag::transform-intro[]
|
||||||
|
{transforms-cap} enable you to convert existing {es} indices into summarized
|
||||||
|
indices, which provide opportunities for new insights and analytics.
|
||||||
|
// end::transform-intro[]
|
||||||
|
For example, you can use {transforms} to pivot your data into entity-centric
|
||||||
|
indices that summarize the behavior of users or sessions or other entities in
|
||||||
|
your data.
|
||||||
|
|
||||||
|
* <<transform-overview>>
|
||||||
|
* <<transform-setup>>
|
||||||
|
* <<transform-usage>>
|
||||||
|
* <<transform-api-quickref>>
|
||||||
|
* <<ecommerce-transforms>>
|
||||||
|
* <<transform-examples>>
|
||||||
|
* <<transform-painless-examples>>
|
||||||
|
* <<transform-troubleshooting>>
|
||||||
|
* <<transform-limitations>>
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[transform-troubleshooting]]
|
[[transform-troubleshooting]]
|
||||||
=== Troubleshooting {transforms}
|
= Troubleshooting {transforms}
|
||||||
[subs="attributes"]
|
[subs="attributes"]
|
||||||
++++
|
++++
|
||||||
<titleabbrev>Troubleshooting</titleabbrev>
|
<titleabbrev>Troubleshooting</titleabbrev>
|
||||||
|
@ -20,16 +20,15 @@ information from the following files and APIs:
|
||||||
|
|
||||||
* Lightweight audit messages are stored in `.transform-notifications-read`. Search
|
* Lightweight audit messages are stored in `.transform-notifications-read`. Search
|
||||||
by your `transform_id`.
|
by your `transform_id`.
|
||||||
* The
|
* The <<get-transform-stats,get {transform} statistics API>> provides
|
||||||
{ref}/get-transform-stats.html[get {transform} statistics API]
|
information about the {transform} status and failures.
|
||||||
provides information about the {transform} status and failures.
|
|
||||||
* If the {transform} exists as a task, you can use the
|
* If the {transform} exists as a task, you can use the
|
||||||
{ref}/tasks.html[task management API] to gather task information. For example:
|
<<tasks,task management API>> to gather task information. For example:
|
||||||
`GET _tasks?actions=data_frame/transforms*&detailed`. Typically, the task exists
|
`GET _tasks?actions=data_frame/transforms*&detailed`. Typically, the task exists
|
||||||
when the {transform} is in a started or failed state.
|
when the {transform} is in a started or failed state.
|
||||||
* The {es} logs from the node that was running the {transform} might
|
* The {es} logs from the node that was running the {transform} might
|
||||||
also contain useful information. You can identify the node from the notification
|
also contain useful information. You can identify the node from the notification
|
||||||
messages. Alternatively, if the task still exists, you can get that information
|
messages. Alternatively, if the task still exists, you can get that information
|
||||||
from the get {transform} statistics API. For more information, see
|
from the get {transform} statistics API. For more information, see
|
||||||
{ref}/logging.html[Logging configuration].
|
<<logging>>.
|
||||||
|
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="basic"]
|
[testenv="basic"]
|
||||||
[[transform-usage]]
|
[[transform-usage]]
|
||||||
=== When to use {transforms}
|
= When to use {transforms}
|
||||||
|
|
||||||
{es} aggregations are a powerful and flexible feature that enable you to
|
{es} aggregations are a powerful and flexible feature that enable you to
|
||||||
summarize and retrieve complex insights about your data. You can summarize
|
summarize and retrieve complex insights about your data. You can summarize
|
||||||
|
|
Loading…
Reference in New Issue