mirror of https://github.com/apache/druid.git
[docs] update tutorial for Theta sketches (#16953)
* from start to step 3 of Ingest data using Theta sketche * updated upto "Query the Theta sketch column" * fixed sentence * another typo * using sql ingestion instead of batch-sql * waiting for explanations on DS_THETA * Revert "using sql ingestion instead of batch-sql" This reverts commitb95fcb9b32. * Revert "using sql ingestion instead of batch-sql" This reverts commitb95fcb9b32. * just copy and pasting to where I was * updated tutorial * fixing images, and removing unused * slightly updating explanatio * Update docs/tutorials/tutorial-sketches-theta.md * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * addressing comments in review * made filter clause consitent with other instances * Apply suggestions from code review Co-authored-by: Charles Smith <techdocsmith@gmail.com> Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> --------- Co-authored-by: Benedict Jin <asdf2014@apache.org> Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> Co-authored-by: Charles Smith <techdocsmith@gmail.com>
This commit is contained in:
Edgar Melendrez
GitHub
parent
9932f2e70a
commit
a67a3c8e0a
Binary file not shown.
|
Before Width: | Height: | Size: 179 KiB After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 159 KiB After Width: | Height: | Size: 138 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 211 KiB After Width: | Height: | Size: 86 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 143 KiB After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 123 KiB After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 69 KiB After Width: | Height: | Size: 102 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 84 KiB |
|
|
@ -36,7 +36,7 @@ Imagine you are interested in the number of visitors that watched episodes of a
|
||||||
|
|
||||||
There is no way to answer these questions by just looking at the aggregated numbers. You would have to go back to the detail data and scan every single row. If the data volume is high enough, this may take a very long time, meaning that an interactive data exploration is not possible.
|
There is no way to answer these questions by just looking at the aggregated numbers. You would have to go back to the detail data and scan every single row. If the data volume is high enough, this may take a very long time, meaning that an interactive data exploration is not possible.
|
||||||
|
|
||||||
An additional nuisance is that unique counts don't work well with rollups. For this example, it would be great if you could have just one row of data per 15 minute interval[^1], show, and episode. After all, you are not interested in the individual user IDs, just the unique counts.
|
An additional nuisance is that unique counts don't work well with rollups. For this example, it would be great if you could have just one row of data per 15 minute interval<sup>[^1]</sup>, show, and episode. After all, you are not interested in the individual user IDs, just the unique counts.
|
||||||
|
|
||||||
[^1]: Why 15 minutes and not just 1 hour? Intervals of 15 minutes work better with international timezones because those are not always aligned by hour. India, for instance, is 30 minutes off, and Nepal is even 45 minutes off. With 15 minute aggregates, you can get hourly sums for any of those timezones, too!
|
[^1]: Why 15 minutes and not just 1 hour? Intervals of 15 minutes work better with international timezones because those are not always aligned by hour. India, for instance, is 30 minutes off, and Nepal is even 45 minutes off. With 15 minute aggregates, you can get hourly sums for any of those timezones, too!
|
||||||
|
|
||||||
|
|
@ -60,9 +60,11 @@ In this tutorial, you will learn how to do the following:
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
For this tutorial, you should have already downloaded Druid as described in
|
Before proceeding, download Druid as described in the [single-machine quickstart](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
|
||||||
the [single-machine quickstart](index.md) and have it running on your local machine.
|
|
||||||
It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md).
|
It's helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md).
|
||||||
|
|
||||||
|
## Sample data
|
||||||
|
|
||||||
This tutorial works with the following data:
|
This tutorial works with the following data:
|
||||||
- **date**: a timestamp. In this case it's just dates but as mentioned earlier, a finer granularity makes sense in real life.
|
- **date**: a timestamp. In this case it's just dates but as mentioned earlier, a finer granularity makes sense in real life.
|
||||||
|
|
@ -95,103 +97,35 @@ date,uid,show,episode
|
||||||
|
|
||||||
## Ingest data using Theta sketches
|
## Ingest data using Theta sketches
|
||||||
|
|
||||||
1. Navigate to the **Load data** wizard in the web console.
|
Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the sample data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
|
||||||
2. Select `Paste data` as the data source and paste the given data:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
3. Leave the source type as `inline` and click **Apply** and **Next: Parse data**.
|
|
||||||
4. Parse the data as CSV, with included headers:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
5. Accept the default values in the **Parse time**, **Transform**, and **Filter** stages.
|
|
||||||
6. In the **Configure schema** stage, enable rollup and confirm your choice in the dialog. Then set the query granularity to `day`.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
7. Add the Theta sketch during this stage. Select **Add metric**.
|
|
||||||
8. Define the new metric as a Theta sketch with the following details:
|
|
||||||
* **Name**: `theta_uid`
|
|
||||||
* **Type**: `thetaSketch`
|
|
||||||
* **Field name**: `uid`
|
|
||||||
* **Size**: Accept the default value, `16384`.
|
|
||||||
* **Is input theta sketch**: Accept the default value, `False`.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
9. Click **Apply** to add the new metric to the data model.
|
|
||||||
|
|
||||||
|
|
||||||
10. You are not interested in individual user ID's, only the unique counts. Right now, `uid` is still in the data model. To remove it, click on the `uid` column in the data model and delete it using the trashcan icon on the right:
|
```sql
|
||||||
|
INSERT INTO "ts_tutorial"
|
||||||
|
WITH "source" AS (SELECT * FROM TABLE(
|
||||||
|
EXTERN(
|
||||||
11. For the remaining stages of the **Load data** wizard, set the following options:
|
'{"type":"inline","data":"date,uid,show,episode\n2022-05-19,alice,Game of Thrones,S1E1\n2022-05-19,alice,Game of Thrones,S1E2\n2022-05-19,alice,Game of Thrones,S1E1\n2022-05-19,bob,Bridgerton,S1E1\n2022-05-20,alice,Game of Thrones,S1E1\n2022-05-20,carol,Bridgerton,S1E2\n2022-05-20,dan,Bridgerton,S1E1\n2022-05-21,alice,Game of Thrones,S1E1\n2022-05-21,carol,Bridgerton,S1E1\n2022-05-21,erin,Game of Thrones,S1E1\n2022-05-21,alice,Bridgerton,S1E1\n2022-05-22,bob,Game of Thrones,S1E1\n2022-05-22,bob,Bridgerton,S1E1\n2022-05-22,carol,Bridgerton,S1E2\n2022-05-22,bob,Bridgerton,S1E1\n2022-05-22,erin,Game of Thrones,S1E1\n2022-05-22,erin,Bridgerton,S1E2\n2022-05-23,erin,Game of Thrones,S1E1\n2022-05-23,alice,Game of Thrones,S1E1"}',
|
||||||
* **Partition**: Set **Segment granularity** to `day`.
|
'{"type":"csv","findColumnsFromHeader":true}'
|
||||||
* **Tune**: Leave the default options.
|
)
|
||||||
* **Publish**: Set the datasource name to `ts_tutorial`.
|
) EXTEND ("date" VARCHAR, "show" VARCHAR, "episode" VARCHAR, "uid" VARCHAR))
|
||||||
|
SELECT
|
||||||
On the **Edit spec** page, your final input spec should match the following:
|
TIME_FLOOR(TIME_PARSE("date"), 'P1D') AS "__time",
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"type": "index_parallel",
|
|
||||||
"spec": {
|
|
||||||
"ioConfig": {
|
|
||||||
"type": "index_parallel",
|
|
||||||
"inputSource": {
|
|
||||||
"type": "inline",
|
|
||||||
"data": "date,uid,show,episode\n2022-05-19,alice,Game of Thrones,S1E1\n2022-05-19,alice,Game of Thrones,S1E2\n2022-05-19,alice,Game of Thrones,S1E1\n2022-05-19,bob,Bridgerton,S1E1\n2022-05-20,alice,Game of Thrones,S1E1\n2022-05-20,carol,Bridgerton,S1E2\n2022-05-20,dan,Bridgerton,S1E1\n2022-05-21,alice,Game of Thrones,S1E1\n2022-05-21,carol,Bridgerton,S1E1\n2022-05-21,erin,Game of Thrones,S1E1\n2022-05-21,alice,Bridgerton,S1E1\n2022-05-22,bob,Game of Thrones,S1E1\n2022-05-22,bob,Bridgerton,S1E1\n2022-05-22,carol,Bridgerton,S1E2\n2022-05-22,bob,Bridgerton,S1E1\n2022-05-22,erin,Game of Thrones,S1E1\n2022-05-22,erin,Bridgerton,S1E2\n2022-05-23,erin,Game of Thrones,S1E1\n2022-05-23,alice,Game of Thrones,S1E1"
|
|
||||||
},
|
|
||||||
"inputFormat": {
|
|
||||||
"type": "csv",
|
|
||||||
"findColumnsFromHeader": true
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"tuningConfig": {
|
|
||||||
"type": "index_parallel",
|
|
||||||
"partitionsSpec": {
|
|
||||||
"type": "hashed"
|
|
||||||
},
|
|
||||||
"forceGuaranteedRollup": true
|
|
||||||
},
|
|
||||||
"dataSchema": {
|
|
||||||
"dataSource": "ts_tutorial",
|
|
||||||
"timestampSpec": {
|
|
||||||
"column": "date",
|
|
||||||
"format": "auto"
|
|
||||||
},
|
|
||||||
"dimensionsSpec": {
|
|
||||||
"dimensions": [
|
|
||||||
"show",
|
"show",
|
||||||
"episode"
|
"episode",
|
||||||
]
|
COUNT(*) AS "count",
|
||||||
},
|
DS_THETA("uid") AS "theta_uid"
|
||||||
"granularitySpec": {
|
FROM "source"
|
||||||
"queryGranularity": "day",
|
GROUP BY 1, 2, 3
|
||||||
"rollup": true,
|
PARTITIONED BY DAY
|
||||||
"segmentGranularity": "day"
|
|
||||||
},
|
|
||||||
"metricsSpec": [
|
|
||||||
{
|
|
||||||
"name": "count",
|
|
||||||
"type": "count"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "thetaSketch",
|
|
||||||
"name": "theta_uid",
|
|
||||||
"fieldName": "uid"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Notice the `theta_uid` object in the `metricsSpec` list, that defines the `thetaSketch` aggregator on the `uid` column during ingestion.
|
Notice the `theta_uid` column in the `SELECT` statement. It defines the `thetaSketch` aggregator on the `uid` column during ingestion.
|
||||||
|
In this scenario you are not interested in individual user IDs, only the unique counts.
|
||||||
|
Instead you create Theta sketches on the values of `uid` using the `DS_THETA` function.
|
||||||
|
|
||||||
Click **Submit** to start the ingestion.
|
[`DS_THETA`](../development/extensions-core/datasketches-theta.md#aggregator) has an optional second parameter that controls the accuracy and size of the sketches.
|
||||||
|
|
||||||
|
The `GROUP BY` statement groups the entries for each episode of a show watched on the same day.
|
||||||
|
|
||||||
## Query the Theta sketch column
|
## Query the Theta sketch column
|
||||||
|
|
||||||
|
|
@ -209,36 +143,22 @@ Let's first see what the data looks like in Druid. Run the following SQL stateme
|
||||||
SELECT * FROM ts_tutorial
|
SELECT * FROM ts_tutorial
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The Theta sketch column `theta_uid` appears as a Base64-encoded string; behind it is a bitmap.
|
The Theta sketch column `theta_uid` appears as a Base64-encoded string; behind it is a bitmap.
|
||||||
|
|
||||||
The following query to compute the distinct counts of user IDs uses `APPROX_COUNT_DISTINCT_DS_THETA` and groups by the other dimensions:
|
The following query uses `THETA_SKETCH_ESTIMATE` to compute the distinct counts of user IDs and groups by the other dimensions:
|
||||||
```sql
|
|
||||||
SELECT __time,
|
|
||||||
"show",
|
|
||||||
"episode",
|
|
||||||
APPROX_COUNT_DISTINCT_DS_THETA(theta_uid) AS users
|
|
||||||
FROM ts_tutorial
|
|
||||||
GROUP BY 1, 2, 3
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
In the preceding query, `APPROX_COUNT_DISTINCT_DS_THETA` is equivalent to calling `DS_THETA` and `THETA_SKETCH_ESIMATE` as follows:
|
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT __time,
|
SELECT
|
||||||
|
__time,
|
||||||
"show",
|
"show",
|
||||||
"episode",
|
"episode",
|
||||||
THETA_SKETCH_ESTIMATE(DS_THETA(theta_uid)) AS users
|
THETA_SKETCH_ESTIMATE(theta_uid) AS users
|
||||||
FROM ts_tutorial
|
FROM ts_tutorial
|
||||||
GROUP BY 1, 2, 3
|
|
||||||
```
|
```
|
||||||
|
|
||||||
That is, `APPROX_COUNT_DISTINCT_DS_THETA` applies the following:
|
|
||||||
* `DS_THETA`: Creates a new Theta sketch from the column of Theta sketches
|
|
||||||
* `THETA_SKETCH_ESTIMATE`: Calculates the distinct count estimate from the output of `DS_THETA`
|
|
||||||
|
|
||||||
### Filtered metrics
|
### Filtered metrics
|
||||||
|
|
||||||
|
|
@ -247,7 +167,17 @@ Druid has the capability to use [filtered metrics](../querying/sql-aggregations.
|
||||||
In the case of Theta sketches, the filter clause has to be inserted between the aggregator and the estimator.
|
In the case of Theta sketches, the filter clause has to be inserted between the aggregator and the estimator.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
As an example, query the total unique users that watched _Bridgerton:_
|
As an example, query the total unique users that watched _Bridgerton_:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT APPROX_COUNT_DISTINCT_DS_THETA(theta_uid) FILTER(WHERE "show" = 'Bridgerton') AS users
|
||||||
|
FROM ts_tutorial
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
In the preceding query, `APPROX_COUNT_DISTINCT_DS_THETA` is equivalent to calling `DS_THETA` and `THETA_SKETCH_ESIMATE` as follows:
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT THETA_SKETCH_ESTIMATE(
|
SELECT THETA_SKETCH_ESTIMATE(
|
||||||
|
|
@ -256,7 +186,12 @@ SELECT THETA_SKETCH_ESTIMATE(
|
||||||
FROM ts_tutorial
|
FROM ts_tutorial
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The `APPROX_COUNT_DISTINCT_DS_THETA` function applies the following:
|
||||||
|
|
||||||
|
* `DS_THETA`: Creates a new Theta sketch from the column of Theta sketches.
|
||||||
|
* `THETA_SKETCH_ESTIMATE`: Calculates the distinct count estimate from the output of `DS_THETA`.
|
||||||
|
|
||||||
|
Note that the filter clause limits an aggregation query to only the rows that match the filter.
|
||||||
|
|
||||||
### Set operations
|
### Set operations
|
||||||
|
|
||||||
|
|
@ -274,7 +209,7 @@ SELECT THETA_SKETCH_ESTIMATE(
|
||||||
FROM ts_tutorial
|
FROM ts_tutorial
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Again, the set function is spliced in between the aggregator and the estimator.
|
Again, the set function is spliced in between the aggregator and the estimator.
|
||||||
|
|
||||||
|
|
@ -290,7 +225,7 @@ SELECT THETA_SKETCH_ESTIMATE(
|
||||||
FROM ts_tutorial
|
FROM ts_tutorial
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
And finally, there is `THETA_SKETCH_NOT` which computes the set difference of two or more segments.
|
And finally, there is `THETA_SKETCH_NOT` which computes the set difference of two or more segments.
|
||||||
The result describes how many visitors watched episode 1 of Bridgerton but not episode 2.
|
The result describes how many visitors watched episode 1 of Bridgerton but not episode 2.
|
||||||
|
|
@ -306,7 +241,7 @@ SELECT THETA_SKETCH_ESTIMATE(
|
||||||
FROM ts_tutorial
|
FROM ts_tutorial
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Conclusions
|
## Conclusions
|
||||||
|
|
||||||
|
|
@ -314,7 +249,7 @@ FROM ts_tutorial
|
||||||
- This allows us to use rollup and discard the individual values, just retaining statistical approximations in the sketches.
|
- This allows us to use rollup and discard the individual values, just retaining statistical approximations in the sketches.
|
||||||
- With Theta sketch set operations, affinity analysis is easier, for example, to answer questions such as which segments correlate or overlap by how much.
|
- With Theta sketch set operations, affinity analysis is easier, for example, to answer questions such as which segments correlate or overlap by how much.
|
||||||
|
|
||||||
## Further reading
|
## Learn more
|
||||||
|
|
||||||
See the following topics for more information:
|
See the following topics for more information:
|
||||||
* [Theta sketch](../development/extensions-core/datasketches-theta.md) for reference on ingestion and native queries on Theta sketches in Druid.
|
* [Theta sketch](../development/extensions-core/datasketches-theta.md) for reference on ingestion and native queries on Theta sketches in Druid.
|
||||||
|
|
@ -326,4 +261,3 @@ See the following topics for more information:
|
||||||
## Acknowledgments
|
## Acknowledgments
|
||||||
|
|
||||||
This tutorial is adapted from a [blog post](https://blog.hellmar-becker.de/2022/06/05/druid-data-cookbook-counting-unique-visitors-for-overlapping-segments/) by community member Hellmar Becker.
|
This tutorial is adapted from a [blog post](https://blog.hellmar-becker.de/2022/06/05/druid-data-cookbook-counting-unique-visitors-for-overlapping-segments/) by community member Hellmar Becker.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue