mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-03-26 09:58:28 +00:00
2a4380e0aa
In data frame transforms the same scheduler controls both retries in the event of search failures and gaps between checks for changes when the transform is running continuously. Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
128 lines
4.4 KiB
Plaintext
128 lines
4.4 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[data-frame-transform-resource]]
|
|
=== {dataframe-transform-cap} resources
|
|
|
|
{dataframe-transform-cap} resources relate to the <<data-frame-apis>>.
|
|
|
|
For more information, see
|
|
{stack-ov}/ecommerce-dataframes.html[Transforming your data with {dataframes}].
|
|
|
|
[discrete]
|
|
[[data-frame-transform-properties]]
|
|
==== {api-definitions-title}
|
|
|
|
`description`::
|
|
(string) A description of the {dataframe-transform}.
|
|
|
|
`dest`::
|
|
(object) The destination for the {dataframe-transform}. See
|
|
<<data-frame-transform-dest>>.
|
|
|
|
`frequency`::
|
|
(time units) The interval between checks for changes in the source indices
|
|
when the {dataframe-transform} is running continuously. Also determines the
|
|
retry interval in the event of transient failures while the {dataframe-transform} is
|
|
searching or indexing. The minimum value is `1s` and the maximum is `1h`. The
|
|
default value is `1m`.
|
|
|
|
`id`::
|
|
(string) A unique identifier for the {dataframe-transform}.
|
|
|
|
`pivot`::
|
|
(object) The method for transforming the data. See
|
|
<<data-frame-transform-pivot>>.
|
|
|
|
`source`::
|
|
(object) The source of the data for the {dataframe-transform}. See
|
|
<<data-frame-transform-source>>.
|
|
|
|
[[data-frame-transform-dest]]
|
|
==== Dest objects
|
|
|
|
{dataframe-transform-cap} resources contain `dest` objects. For example, when
|
|
you create a {dataframe-transform}, you must define its destination.
|
|
|
|
[discrete]
|
|
[[data-frame-transform-dest-properties]]
|
|
===== {api-definitions-title}
|
|
|
|
`index`::
|
|
(string) The _destination index_ for the {dataframe-transform}.
|
|
|
|
`pipeline`::
|
|
(string) The unique identifier for a <<pipeline,pipeline>>.
|
|
|
|
[[data-frame-transform-source]]
|
|
==== Source objects
|
|
|
|
{dataframe-transform-cap} resources contain `source` objects. For example, when
|
|
you create a {dataframe-transform}, you must define its source.
|
|
|
|
[discrete]
|
|
[[data-frame-transform-source-properties]]
|
|
===== {api-definitions-title}
|
|
|
|
`index`::
|
|
(string or array) The _source indices_ for the {dataframe-transform}. It can
|
|
be a single index, an index pattern (for example, `"myindex*"`), or an array
|
|
of indices (for example, `["index1", "index2"]`).
|
|
|
|
`query`::
|
|
(object) A query clause that retrieves a subset of data from the source index.
|
|
See <<query-dsl>>.
|
|
|
|
[[data-frame-transform-pivot]]
|
|
==== Pivot objects
|
|
|
|
{dataframe-transform-cap} resources contain `pivot` objects, which define the
|
|
pivot function `group by` fields and the aggregation to reduce the data.
|
|
|
|
[discrete]
|
|
[[data-frame-transform-pivot-properties]]
|
|
===== {api-definitions-title}
|
|
|
|
`aggregations` or `aggs`::
|
|
(object) Defines how to aggregate the grouped data. The following composite
|
|
aggregations are supported:
|
|
+
|
|
--
|
|
* {ref}/search-aggregations-metrics-avg-aggregation.html[Average]
|
|
* {ref}/search-aggregations-metrics-weight-avg-aggregation.html[Weighted Average]
|
|
* {ref}/search-aggregations-metrics-cardinality-aggregation.html[Cardinality]
|
|
* {ref}/search-aggregations-metrics-geocentroid-aggregation.html[Geo Centroid]
|
|
* {ref}/search-aggregations-metrics-max-aggregation.html[Max]
|
|
* {ref}/search-aggregations-metrics-min-aggregation.html[Min]
|
|
* {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[Scripted Metric]
|
|
* {ref}/search-aggregations-metrics-sum-aggregation.html[Sum]
|
|
* {ref}/search-aggregations-metrics-valuecount-aggregation.html[Value Count]
|
|
* {ref}/search-aggregations-pipeline-bucket-script-aggregation.html[Bucket Script]
|
|
|
|
IMPORTANT: {dataframe-transforms-cap} support a subset of the functionality in
|
|
composite aggregations. See
|
|
{stack-ov}/dataframe-limitations.html[{dataframe-cap} limitations].
|
|
|
|
--
|
|
|
|
`group_by`::
|
|
(object) Defines how to group the data. More than one grouping can be defined
|
|
per pivot. The following groupings are supported:
|
|
+
|
|
--
|
|
* {ref}/search-aggregations-bucket-composite-aggregation.html#_terms[Terms]
|
|
* {ref}/search-aggregations-bucket-composite-aggregation.html#_histogram[Histogram]
|
|
* {ref}/search-aggregations-bucket-composite-aggregation.html#_date_histogram[Date Histogram]
|
|
--
|
|
|
|
`max_page_search_size`::
|
|
(integer) Defines the initial page size to use for the composite aggregation
|
|
for each checkpoint. If circuit breaker exceptions occur, the page size is
|
|
dynamically adjusted to a lower value. The minimum value is `10` and the
|
|
maximum is `10,000`. The default value is `500`.
|
|
|
|
[[data-frame-transform-example]]
|
|
==== {api-examples-title}
|
|
|
|
See the
|
|
<<put-data-frame-transform-example,create {dataframe-transforms} API examples>>.
|