263 lines
7.8 KiB
Plaintext
263 lines
7.8 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[ecommerce-dataframes]]
|
|
=== Transforming the eCommerce sample data
|
|
|
|
beta[]
|
|
|
|
<<ml-dataframes,{dataframe-transforms-cap}>> enable you to retrieve information
|
|
from an {es} index, transform it, and store it in another index. Let's use the
|
|
{kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can
|
|
pivot and summarize your data with {dataframe-transforms}.
|
|
|
|
|
|
. If the {es} {security-features} are enabled, obtain a user ID with sufficient
|
|
privileges to complete these steps.
|
|
+
|
|
--
|
|
You need `manage_data_frame_transforms` cluster privileges to preview and create
|
|
{dataframe-transforms}. Members of the built-in `data_frame_transforms_admin`
|
|
role have these privileges.
|
|
|
|
You also need `read` and `view_index_metadata` index privileges on the source
|
|
index and `read`, `create_index`, and `index` privileges on the destination
|
|
index.
|
|
|
|
For more information, see <<security-privileges>> and <<built-in-roles>>.
|
|
--
|
|
|
|
. Choose your _source index_.
|
|
+
|
|
--
|
|
In this example, we'll use the eCommerce orders sample data. If you're not
|
|
already familiar with the `kibana_sample_data_ecommerce` index, use the
|
|
*Revenue* dashboard in {kib} to explore the data. Consider what insights you
|
|
might want to derive from this eCommerce data.
|
|
--
|
|
|
|
. Play with various options for grouping and aggregating the data.
|
|
+
|
|
--
|
|
For example, you might want to group the data by product ID and calculate the
|
|
total number of sales for each product and its average price. Alternatively, you
|
|
might want to look at the behavior of individual customers and calculate how
|
|
much each customer spent in total and how many different categories of products
|
|
they purchased. Or you might want to take the currencies or geographies into
|
|
consideration. What are the most interesting ways you can transform and
|
|
interpret this data?
|
|
|
|
_Pivoting_ your data involves using at least one field to group it and applying
|
|
at least one aggregation. You can preview what the transformed data will look
|
|
like, so go ahead and play with it!
|
|
|
|
For example, go to *Machine Learning* > *Data Frames* in {kib} and use the
|
|
wizard to create a {dataframe-transform}:
|
|
|
|
[role="screenshot"]
|
|
image::images/ecommerce-pivot1.jpg["Creating a simple {dataframe-transform} in {kib}"]
|
|
|
|
In this case, we grouped the data by customer ID and calculated the sum of
|
|
products each customer purchased.
|
|
|
|
Let's add some more aggregations to learn more about our customers' orders. For
|
|
example, let's calculate the total sum of their purchases, the maximum number of
|
|
products that they purchased in a single order, and their total number of orders.
|
|
We'll accomplish this by using the
|
|
{ref}/search-aggregations-metrics-sum-aggregation.html[`sum` aggregation] on the
|
|
`taxless_total_price` field, the
|
|
{ref}/search-aggregations-metrics-max-aggregation.html[`max` aggregation] on the
|
|
`total_quantity` field, and the
|
|
{ref}/search-aggregations-metrics-cardinality-aggregation.html[`cardinality` aggregation]
|
|
on the `order_id` field:
|
|
|
|
[role="screenshot"]
|
|
image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {dataframe-transform} in {kib}"]
|
|
|
|
TIP: If you're interested in a subset of the data, you can optionally include a
|
|
{ref}/search-request-body.html#request-body-search-query[query] element. In this
|
|
example, we've filtered the data so that we're only looking at orders with a
|
|
`currency` of `EUR`. Alternatively, we could group the data by that field too.
|
|
If you want to use more complex queries, you can create your {dataframe} from a
|
|
{kibana-ref}/save-open-search.html[saved search].
|
|
|
|
If you prefer, you can use the
|
|
{ref}/preview-data-frame-transform.html[preview {dataframe-transforms} API]:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST _data_frame/transforms/_preview
|
|
{
|
|
"source": {
|
|
"index": "kibana_sample_data_ecommerce",
|
|
"query": {
|
|
"bool": {
|
|
"filter": {
|
|
"term": {"currency": "EUR"}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"pivot": {
|
|
"group_by": {
|
|
"customer_id": {
|
|
"terms": {
|
|
"field": "customer_id"
|
|
}
|
|
}
|
|
},
|
|
"aggregations": {
|
|
"total_quantity.sum": {
|
|
"sum": {
|
|
"field": "total_quantity"
|
|
}
|
|
},
|
|
"taxless_total_price.sum": {
|
|
"sum": {
|
|
"field": "taxless_total_price"
|
|
}
|
|
},
|
|
"total_quantity.max": {
|
|
"max": {
|
|
"field": "total_quantity"
|
|
}
|
|
},
|
|
"order_id.cardinality": {
|
|
"cardinality": {
|
|
"field": "order_id"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[skip:set up sample data]
|
|
--
|
|
|
|
. When you are satisfied with what you see in the preview, create the
|
|
{dataframe-transform}.
|
|
+
|
|
--
|
|
.. Supply a job ID and the name of the target (or _destination_) index.
|
|
|
|
.. Decide whether you want the {dataframe-transform} to run once or continuously.
|
|
--
|
|
+
|
|
--
|
|
Since this sample data index is unchanging, let's use the default behavior and
|
|
just run the {dataframe-transform} once.
|
|
|
|
[role="screenshot"]
|
|
image::images/ecommerce-batch.jpg["Specifying the {dataframe-transform} options in {kib}"]
|
|
|
|
If you want to try it out, however, go ahead and click on *Continuous mode*.
|
|
You must choose a field that the {dataframe-transform} can use to check which
|
|
entities have changed. In general, it's a good idea to use the ingest timestamp
|
|
field. In this example, however, you can use the `order_date` field.
|
|
|
|
If you prefer, you can use the
|
|
{ref}/put-data-frame-transform.html[create {dataframe-transforms} API]. For
|
|
example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _data_frame/transforms/ecommerce-customer-transform
|
|
{
|
|
"source": {
|
|
"index": [
|
|
"kibana_sample_data_ecommerce"
|
|
],
|
|
"query": {
|
|
"bool": {
|
|
"filter": {
|
|
"term": {
|
|
"currency": "EUR"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"pivot": {
|
|
"group_by": {
|
|
"customer_id": {
|
|
"terms": {
|
|
"field": "customer_id"
|
|
}
|
|
}
|
|
},
|
|
"aggregations": {
|
|
"total_quantity.sum": {
|
|
"sum": {
|
|
"field": "total_quantity"
|
|
}
|
|
},
|
|
"taxless_total_price.sum": {
|
|
"sum": {
|
|
"field": "taxless_total_price"
|
|
}
|
|
},
|
|
"total_quantity.max": {
|
|
"max": {
|
|
"field": "total_quantity"
|
|
}
|
|
},
|
|
"order_id.cardinality": {
|
|
"cardinality": {
|
|
"field": "order_id"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"dest": {
|
|
"index": "ecommerce-customers"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[skip:setup kibana sample data]
|
|
--
|
|
|
|
. Start the {dataframe-transform}.
|
|
+
|
|
--
|
|
|
|
TIP: Even though resource utilization is automatically adjusted based on the
|
|
cluster load, a {dataframe-transform} increases search and indexing load on your
|
|
cluster while it runs. If you're experiencing an excessive load, however, you
|
|
can stop it.
|
|
|
|
You can start, stop, and manage {dataframe-transforms} in {kib}:
|
|
|
|
[role="screenshot"]
|
|
image::images/dataframe-transforms.jpg["Managing {dataframe-transforms} in {kib}"]
|
|
|
|
Alternatively, you can use the
|
|
{ref}/start-data-frame-transform.html[start {dataframe-transforms}] and
|
|
{ref}/stop-data-frame-transform.html[stop {dataframe-transforms}] APIs. For
|
|
example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST _data_frame/transforms/ecommerce-customer-transform/_start
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[skip:setup kibana sample data]
|
|
|
|
--
|
|
|
|
. Explore the data in your new index.
|
|
+
|
|
--
|
|
For example, use the *Discover* application in {kib}:
|
|
|
|
[role="screenshot"]
|
|
image::images/ecommerce-results.jpg["Exploring the new index in {kib}"]
|
|
|
|
--
|
|
|
|
TIP: If you do not want to keep the {dataframe-transform}, you can delete it in
|
|
{kib} or use the
|
|
{ref}/delete-data-frame-transform.html[delete {dataframe-transform} API]. When
|
|
you delete a {dataframe-transform}, its destination index and {kib} index
|
|
patterns remain.
|