diff --git a/DataIngestion/dataformats.md b/DataIngestion/dataformats.md index a27ece5..607b870 100644 --- a/DataIngestion/dataformats.md +++ b/DataIngestion/dataformats.md @@ -132,10 +132,10 @@ TSV `inputFormat` 有以下组件： #### ORC > [!WARNING] -> 使用ORC输入格式之前，首先需要包含 [druid-orc-extensions](../Development/orc-extensions.md) +> 使用ORC输入格式之前，首先需要包含 [druid-orc-extensions](../development/orc-extensions.md) > [!WARNING] -> 如果您正在考虑从早于0.15.0的版本升级到0.15.0或更高版本，请仔细阅读 [从contrib扩展的迁移](../Development/orc-extensions.md#从contrib扩展迁移)。 +> 如果您正在考虑从早于0.15.0的版本升级到0.15.0或更高版本，请仔细阅读 [从contrib扩展的迁移](../development/orc-extensions.md#从contrib扩展迁移)。 一个加载ORC格式数据的 `inputFormat` 示例： ```json @@ -169,7 +169,7 @@ ORC `inputFormat` 有以下组件： #### Parquet > [!WARNING] -> 使用Parquet输入格式之前，首先需要包含 [druid-parquet-extensions](../Development/parquet-extensions.md) +> 使用Parquet输入格式之前，首先需要包含 [druid-parquet-extensions](../development/parquet-extensions.md) 一个加载Parquet格式数据的 `inputFormat` 示例： ```json @@ -277,7 +277,7 @@ Parquet `inputFormat` 有以下组件： > [!WARNING] > parser在 [本地批任务](native.md), [Kafka索引任务](kafka.md) 和 [Kinesis索引任务](kinesis.md) 中已经废弃，在这些类型的摄入方式中考虑使用 [inputFormat](#数据格式) -该部分列出来了所有默认的以及核心扩展中的解析器。对于社区的扩展解析器，请参见 [社区扩展列表](../Development/extensions.md#社区扩展) +该部分列出来了所有默认的以及核心扩展中的解析器。对于社区的扩展解析器，请参见 [社区扩展列表](../development/extensions.md#社区扩展) #### String Parser @@ -291,7 +291,7 @@ Parquet `inputFormat` 有以下组件： #### Avro Hadoop Parser > [!WARNING] -> 需要添加 [druid-avro-extensions](../Development/avro-extensions.md) 来使用 Avro Hadoop解析器 +> 需要添加 [druid-avro-extensions](../development/avro-extensions.md) 来使用 Avro Hadoop解析器 该解析器用于 [Hadoop批摄取](hadoopbased.md)。在 `ioConfig` 中，`inputSpec` 中的 `inputFormat` 必须设置为 `org.apache.druid.data.input.avro.AvroValueInputFormat`。您可能想在 `tuningConfig` 中的 `jobProperties` 选项设置Avro reader的schema， 例如：`"avro.schema.input.value.path": "/path/to/your/schema.avsc"` 或者 `"avro.schema.input.value": "your_schema_JSON_object"`。如果未设置Avro读取器的schema，则将使用Avro对象容器文件中的schema，详情可以参见 [avro规范](http://avro.apache.org/docs/1.7.7/spec.html#Schema+Resolution) @@ -339,10 +339,10 @@ Avro parseSpec可以包含使用"root"或"path"字段类型的 [flattenSpec](#fl #### ORC Hadoop Parser > [!WARNING] -> 需要添加 [druid-orc-extensions](../Development/orc-extensions.md) 来使用ORC Hadoop解析器 +> 需要添加 [druid-orc-extensions](../development/orc-extensions.md) 来使用ORC Hadoop解析器 > [!WARNING] -> 如果您正在考虑从早于0.15.0的版本升级到0.15.0或更高版本，请仔细阅读 [从contrib扩展的迁移](../Development/orc-extensions.md#从contrib扩展迁移)。 +> 如果您正在考虑从早于0.15.0的版本升级到0.15.0或更高版本，请仔细阅读 [从contrib扩展的迁移](../development/orc-extensions.md#从contrib扩展迁移)。 该解析器用于 [Hadoop批摄取](hadoopbased.md)。在 `ioConfig` 中，`inputSpec` 中的 `inputFormat` 必须设置为 `org.apache.orc.mapreduce.OrcInputFormat`。 @@ -564,7 +564,7 @@ Avro parseSpec可以包含使用"root"或"path"字段类型的 [flattenSpec](#fl #### Parquet Hadoop Parser > [!WARNING] -> 需要添加 [druid-parquet-extensions](../Development/parquet-extensions.md) 来使用Parquet Hadoop解析器 +> 需要添加 [druid-parquet-extensions](../development/parquet-extensions.md) 来使用Parquet Hadoop解析器 该解析器用于 [Hadoop批摄取](hadoopbased.md)。在 `ioConfig` 中，`inputSpec` 中的 `inputFormat` 必须设置为 `org.apache.druid.data.input.parquet.DruidParquetInputFormat`。 @@ -690,7 +690,7 @@ Parquet Hadoop 解析器支持自动字段发现，如果提供了一个带有 ` > 考虑在该解析器之上使用 [Parquet Hadoop Parser](#parquet-hadoop-parser) 来摄取Parquet文件。 两者之间的不同之处参见 [Parquet Hadoop解析器 vs Parquet Avro Hadoop解析器]() 部分 > [!WARNING] -> 使用Parquet Avro Hadoop Parser需要同时加入 [druid-parquet-extensions](../Development/parquet-extensions.md) 和 [druid-avro-extensions](../Development/avro-extensions.md) +> 使用Parquet Avro Hadoop Parser需要同时加入 [druid-parquet-extensions](../development/parquet-extensions.md) 和 [druid-avro-extensions](../development/avro-extensions.md) 该解析器用于 [Hadoop批摄取](hadoopbased.md), 该解析器首先将Parquet数据转换为Avro记录，然后再解析它们后摄入到Druid。在 `ioConfig` 中，`inputSpec` 中的 `inputFormat` 必须设置为 `org.apache.druid.data.input.parquet.DruidParquetAvroInputFormat`。 @@ -763,7 +763,7 @@ Parquet Avro Hadoop 解析器支持自动字段发现，如果提供了一个带 #### Avro Stream Parser > [!WARNING] -> 需要添加 [druid-avro-extensions](../Development/avro-extensions.md) 来使用Avro Stream解析器 +> 需要添加 [druid-avro-extensions](../development/avro-extensions.md) 来使用Avro Stream解析器 该解析器用于 [流式摄取](streamingest.md), 直接从一个流来读取数据。 @@ -909,7 +909,7 @@ Avro Bytes Decorder首先提取输入消息的 `subject` 和 `id`， 然后使 #### Protobuf Parser > [!WARNING] -> 需要添加 [druid-protobuf-extensions](../Development/protobuf-extensions.md) 来使用Protobuf解析器 +> 需要添加 [druid-protobuf-extensions](../development/protobuf-extensions.md) 来使用Protobuf解析器 此解析器用于 [流接收](streamingest.md)，并直接从流中读取协议缓冲区数据。 @@ -949,7 +949,7 @@ Avro Bytes Decorder首先提取输入消息的 `subject` 和 `id`， 然后使 } } ``` -有关更多详细信息和示例，请参见 [扩展说明](../Development/protobuf-extensions.md)。 +有关更多详细信息和示例，请参见 [扩展说明](../development/protobuf-extensions.md)。 ### ParseSpec @@ -1117,7 +1117,7 @@ JSON数据也可以包含多值维度。维度的多个值必须在接收的数 注意: JavaScript解析器必须完全解析数据，并在JS逻辑中以 `{key:value}` 格式返回。这意味着任何展平或解析多维值都必须在这里完成。 > [!WARNING] -> 默认情况下禁用基于JavaScript的功能。有关使用Druid的JavaScript功能的指南，包括如何启用它的说明，请参阅 [Druid JavaScript编程指南](../Development/JavaScript.md)。 +> 默认情况下禁用基于JavaScript的功能。有关使用Druid的JavaScript功能的指南，包括如何启用它的说明，请参阅 [Druid JavaScript编程指南](../development/JavaScript.md)。 #### 时间和维度解析规范 diff --git a/DataIngestion/datamanage.md b/DataIngestion/datamanage.md index d793ec6..1a78965 100644 --- a/DataIngestion/datamanage.md +++ b/DataIngestion/datamanage.md @@ -163,7 +163,7 @@ Druid使用 `ioConfig` 中的 `inputSpec` 来知道要接收的数据位于何 ### 删除数据 -Druid支持永久的将标记为"unused"状态（详情可见架构设计中的 [段的生命周期](../Design/Design.md#段生命周期)）的段删除掉 +Druid支持永久的将标记为"unused"状态（详情可见架构设计中的 [段的生命周期](../design/Design.md#段生命周期)）的段删除掉 杀死任务负责从元数据存储和深度存储中删除掉指定时间间隔内的不被使用的段 diff --git a/DataIngestion/faq.md b/DataIngestion/faq.md index 1c0f7c6..5fcfb6b 100644 --- a/DataIngestion/faq.md +++ b/DataIngestion/faq.md @@ -34,7 +34,7 @@ Druid会拒绝时间窗口之外的事件， 确认事件是否被拒绝了的 ### 摄取之后段存储在哪里 -段的存储位置由 `druid.storage.type` 配置决定的，Druid会将段上传到 [深度存储](../Design/Deepstorage.md)。 本地磁盘是默认的深度存储位置。 +段的存储位置由 `druid.storage.type` 配置决定的，Druid会将段上传到 [深度存储](../design/Deepstorage.md)。 本地磁盘是默认的深度存储位置。 ### 流摄取任务没有发生段切换递交 @@ -49,11 +49,11 @@ Druid会拒绝时间窗口之外的事件， 确认事件是否被拒绝了的 ### 如何让HDFS工作 -确保在类路径中包含 `druid-hdfs-storage` 和所有的hadoop配置、依赖项（可以通过在安装了hadoop的计算机上运行 `hadoop classpath`命令获得）。并且，提供必要的HDFS设置，如 [深度存储](../Design/Deepstorage.md) 中所述。 +确保在类路径中包含 `druid-hdfs-storage` 和所有的hadoop配置、依赖项（可以通过在安装了hadoop的计算机上运行 `hadoop classpath`命令获得）。并且，提供必要的HDFS设置，如 [深度存储](../design/Deepstorage.md) 中所述。 ### 没有在Historical进程中看到Druid段 -您可以查看位于 `:` 的Coordinator控制台, 确保您的段实际上已加载到 [Historical进程](../Design/Historical.md)中。如果段不存在，请检查Coordinator日志中有关复制错误容量的消息。不下载段的一个原因是，Historical进程的 `maxSize` 太小，使它们无法下载更多数据。您可以使用（例如）更改它： +您可以查看位于 `:` 的Coordinator控制台, 确保您的段实际上已加载到 [Historical进程](../design/Historical.md)中。如果段不存在，请检查Coordinator日志中有关复制错误容量的消息。不下载段的一个原因是，Historical进程的 `maxSize` 太小，使它们无法下载更多数据。您可以使用（例如）更改它： ```json -Ddruid.segmentCache.locations=[{"path":"/tmp/druid/storageLocation","maxSize":"500000000000"}] diff --git a/DataIngestion/hadoopbased.md b/DataIngestion/hadoopbased.md index 818bb92..9e44a52 100644 --- a/DataIngestion/hadoopbased.md +++ b/DataIngestion/hadoopbased.md @@ -13,7 +13,7 @@ ## 基于Hadoop的摄入 -Apache Druid当前支持通过一个Hadoop摄取任务来支持基于Apache Hadoop的批量索引任务， 这些任务被提交到 [Druid Overlord](../Design/Overlord.md)的一个运行实例上。详情可以查看 [基于Hadoop的摄取vs基于本地批摄取的对比](ingestion.md#批量摄取) 来了解基于Hadoop的摄取、本地简单批摄取、本地并行摄取三者的比较。 +Apache Druid当前支持通过一个Hadoop摄取任务来支持基于Apache Hadoop的批量索引任务， 这些任务被提交到 [Druid Overlord](../design/Overlord.md)的一个运行实例上。详情可以查看 [基于Hadoop的摄取vs基于本地批摄取的对比](ingestion.md#批量摄取) 来了解基于Hadoop的摄取、本地简单批摄取、本地并行摄取三者的比较。 运行一个基于Hadoop的批量摄取任务，首先需要编写一个如下的摄取规范， 然后提交到Overlord的 [`druid/indexer/v1/task`](../Operations/api.md#overlord) 接口，或者使用Druid软件包中自带的 `bin/post-index-task` 脚本。 @@ -388,7 +388,7 @@ Hadoop的 [MapReduce文档](https://hadoop.apache.org/docs/stable/hadoop-mapredu ```json classification=yarn-site,properties=[mapreduce.reduce.memory.mb=6144,mapreduce.reduce.java.opts=-server -Xms2g -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps,mapreduce.map.java.opts=758,mapreduce.map.java.opts=-server -Xms512m -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps,mapreduce.task.timeout=1800000] ``` -* 按照 [Hadoop连接配置](../GettingStarted/chapter-4.md#Hadoop连接配置) 指导，使用EMR master中 `/etc/hadoop/conf` 的XML文件。 +* 按照 [Hadoop连接配置](../tutorials/img/chapter-4.md#Hadoop连接配置) 指导，使用EMR master中 `/etc/hadoop/conf` 的XML文件。 ### Kerberized Hadoop集群 @@ -472,7 +472,7 @@ spec文件需要包含一个JSON对象，其中的内容与Hadoop索引任务中 | `password` | String | DB的密码 | 是 | | `segmentTable` | String | DB中使用的表 | 是 | -这些属性应该模仿您为 [Coordinator](../Design/Coordinator.md) 配置的内容。 +这些属性应该模仿您为 [Coordinator](../design/Coordinator.md) 配置的内容。 **segmentOutputPath配置** diff --git a/DataIngestion/ingestion.md b/DataIngestion/ingestion.md index be06539..a8bab72 100644 --- a/DataIngestion/ingestion.md +++ b/DataIngestion/ingestion.md @@ -16,7 +16,7 @@ Druid中的所有数据都被组织成*段*，这些段是数据文件，通常每个段最多有几百万行。在Druid中加载数据称为*摄取或索引*，它包括从源系统读取数据并基于该数据创建段。 -在大多数摄取方法中，加载数据的工作由Druid [MiddleManager](../Design/MiddleManager.md) 进程（或 [Indexer](../Design/Indexer.md) 进程）完成。一个例外是基于Hadoop的摄取，这项工作是使用Hadoop MapReduce作业在YARN上完成的（尽管MiddleManager或Indexer进程仍然参与启动和监视Hadoop作业）。一旦段被生成并存储在 [深层存储](../Design/Deepstorage.md) 中，它们将被Historical进程加载。有关如何在引擎下工作的更多细节，请参阅Druid设计文档的[存储设计](../Design/Design.md) 部分。 +在大多数摄取方法中，加载数据的工作由Druid [MiddleManager](../design/MiddleManager.md) 进程（或 [Indexer](../design/Indexer.md) 进程）完成。一个例外是基于Hadoop的摄取，这项工作是使用Hadoop MapReduce作业在YARN上完成的（尽管MiddleManager或Indexer进程仍然参与启动和监视Hadoop作业）。一旦段被生成并存储在 [深层存储](../design/Deepstorage.md) 中，它们将被Historical进程加载。有关如何在引擎下工作的更多细节，请参阅Druid设计文档的[存储设计](../design/Design.md) 部分。 ### 如何使用本文档 @@ -394,7 +394,7 @@ Druid以两种可能的方式来解释 `dimensionsSpec` : *normal* 和 *schemale ##### `granularitySpec` `granularitySpec` 位于 `dataSchema` -> `granularitySpec`, 用来配置以下操作： -1. 通过 `segmentGranularity` 来将数据源分区到 [时间块](../Design/Design.md#数据源和段) +1. 通过 `segmentGranularity` 来将数据源分区到 [时间块](../design/Design.md#数据源和段) 2. 如果需要的话，通过 `queryGranularity` 来截断时间戳 3. 通过 `interval` 来指定批摄取中应创建段的时间块 4. 通过 `rollup` 来指定是否在摄取时进行汇总 @@ -418,7 +418,7 @@ Druid以两种可能的方式来解释 `dimensionsSpec` : *normal* 和 *schemale | 字段 | 描述 | 默认值 | |-|-|-| | type | `uniform` 或者 `arbitrary` ，大多数时候使用 `uniform` | `uniform` | -| segmentGranularity | 数据源的 [时间分块](../Design/Design.md#数据源和段) 粒度。每个时间块可以创建多个段, 例如，当设置为 `day` 时，同一天的事件属于同一时间块，该时间块可以根据其他配置和输入大小进一步划分为多个段。这里可以提供任何粒度。请注意，同一时间块中的所有段应具有相同的段粒度。

如果 `type` 字段设置为 `arbitrary` 则忽略 | `day` | +| segmentGranularity | 数据源的 [时间分块](../design/Design.md#数据源和段) 粒度。每个时间块可以创建多个段, 例如，当设置为 `day` 时，同一天的事件属于同一时间块，该时间块可以根据其他配置和输入大小进一步划分为多个段。这里可以提供任何粒度。请注意，同一时间块中的所有段应具有相同的段粒度。

如果 `type` 字段设置为 `arbitrary` 则忽略 | `day` | | queryGranularity | 每个段内时间戳存储的分辨率, 必须等于或比 `segmentGranularity` 更细。这将是您可以查询的最细粒度，并且仍然可以查询到合理的结果。但是请注意，您仍然可以在比此粒度更粗的场景进行查询，例如 "`minute`"的值意味着记录将以分钟的粒度存储，并且可以在分钟的任意倍数（包括分钟、5分钟、小时等）进行查询。

这里可以提供任何 [粒度](../Querying/AggregationGranularity.md) 。使用 `none` 按原样存储时间戳，而不进行任何截断。请注意，即使将 `queryGranularity` 设置为 `none`，也将应用 `rollup`。 | `none` | | rollup | 是否在摄取时使用 [rollup](#rollup)。 注意：即使 `queryGranularity` 设置为 `none`，rollup也仍然是有效的，当数据具有相同的时间戳时数据将被汇总 | `true` | | interval | 描述应该创建段的时间块的间隔列表。如果 `type` 设置为`uniform`，则此列表将根据 `segmentGranularity` 进行拆分和舍入。如果 `type` 设置为 `arbitrary` ，则将按原样使用此列表。

如果该值不提供或者为空值，则批处理摄取任务通常会根据在输入数据中找到的时间戳来确定要输出的时间块。

如果指定，批处理摄取任务可以跳过确定分区阶段，这可能会导致更快的摄取。批量摄取任务也可以预先请求它们的所有锁，而不是逐个请求。批处理摄取任务将丢弃任何时间戳超出指定间隔的记录。

在任何形式的流摄取中忽略该配置。 | `null` | diff --git a/DataIngestion/kafka.md b/DataIngestion/kafka.md index 75556a0..6033f8d 100644 --- a/DataIngestion/kafka.md +++ b/DataIngestion/kafka.md @@ -186,7 +186,7 @@ curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http:/ Kafka索引服务同时支持通过 [`inputFormat`](dataformats.md#inputformat) 和 [`parser`](dataformats.md#parser) 来指定数据格式。 `inputFormat` 是一种新的且推荐的用于Kafka索引服务中指定数据格式的方式，但是很遗憾的是目前它还不支持过时的 `parser` 所有支持的所有格式（未来会支持）。 -`inputFormat` 支持的格式包括 [`csv`](dataformats.md#csv), [`delimited`](dataformats.md#TSV(Delimited)), [`json`](dataformats.md#json)。可以使用 `parser` 来读取 [`avro_stream`](dataformats.md#AvroStreamParser), [`protobuf`](dataformats.md#ProtobufParser), [`thrift`](../Development/thrift.md) 格式的数据。 +`inputFormat` 支持的格式包括 [`csv`](dataformats.md#csv), [`delimited`](dataformats.md#TSV(Delimited)), [`json`](dataformats.md#json)。可以使用 `parser` 来读取 [`avro_stream`](dataformats.md#AvroStreamParser), [`protobuf`](dataformats.md#ProtobufParser), [`thrift`](../development/overview.md) 格式的数据。 ### 操作 diff --git a/DataIngestion/native.md b/DataIngestion/native.md index 5c6ec5a..1db4413 100644 --- a/DataIngestion/native.md +++ b/DataIngestion/native.md @@ -39,7 +39,7 @@ Apache Druid当前支持两种类型的本地批量索引任务， `index_parall 传统的 [`firehose`](#firehoses%e5%b7%b2%e5%ba%9f%e5%bc%83) 支持其他一些云存储类型。下面的 `firehose` 类型也是可拆分的。请注意，`firehose` 只支持文本格式。 -* [`static-cloudfiles`](../Development/rackspacecloudfiles.md) +* [`static-cloudfiles`](../development/rackspacecloudfiles.md) 您可能需要考虑以下事项： * 您可能希望控制每个worker进程的输入数据量。这可以使用不同的配置进行控制，具体取决于并行摄取的阶段（有关更多详细信息，请参阅 [`partitionsSpec`](#partitionsspec)。对于从 `inputSource` 读取数据的任务，可以在 `tuningConfig` 中设置 [分割提示规范](#分割提示规范)。对于合并无序段的任务，可以在 `tuningConfig` 中设置`totalNumMergeTasks`。 @@ -235,7 +235,7 @@ PartitionsSpec用于描述辅助分区方法。您应该根据需要的rollup模 基于哈希分区的并行任务类似于 [MapReduce](https://en.wikipedia.org/wiki/MapReduce)。任务分为两个阶段运行，即 `部分段生成` 和 `部分段合并`。 -* 在 `部分段生成` 阶段，与MapReduce中的Map阶段一样，并行任务根据分割提示规范分割输入数据，并将每个分割分配给一个worker。每个worker（`partial_index_generate` 类型）从 `granularitySpec` 中的`segmentGranularity（主分区键）` 读取分配的分割，然后按`partitionsSpec` 中 `partitionDimensions（辅助分区键）`的哈希值对行进行分区。分区数据存储在 [MiddleManager](../Design/MiddleManager.md) 或 [Indexer](../Design/Indexer.md) 的本地存储中。 +* 在 `部分段生成` 阶段，与MapReduce中的Map阶段一样，并行任务根据分割提示规范分割输入数据，并将每个分割分配给一个worker。每个worker（`partial_index_generate` 类型）从 `granularitySpec` 中的`segmentGranularity（主分区键）` 读取分配的分割，然后按`partitionsSpec` 中 `partitionDimensions（辅助分区键）`的哈希值对行进行分区。分区数据存储在 [MiddleManager](../design/MiddleManager.md) 或 [Indexer](../design/Indexer.md) 的本地存储中。 * `部分段合并` 阶段类似于MapReduce中的Reduce阶段。并行任务生成一组新的worker（`partial_index_merge` 类型）来合并在前一阶段创建的分区数据。这里，分区数据根据要合并的时间块和分区维度的散列值进行洗牌；每个worker从多个MiddleManager/Indexer进程中读取落在同一时间块和同一散列值中的数据，并将其合并以创建最终段。最后，它们将最后的段一次推送到深层存储。 **基于单一维度范围分区** @@ -254,7 +254,7 @@ PartitionsSpec用于描述辅助分区方法。您应该根据需要的rollup模 在 `single-dim` 分区下，并行任务分为3个阶段进行，即 `部分维分布`、`部分段生成` 和 `部分段合并`。第一个阶段是收集一些统计数据以找到最佳分区，另外两个阶段是创建部分段并分别合并它们，就像在基于哈希的分区中那样。 * 在 `部分维度分布` 阶段，并行任务分割输入数据，并根据分割提示规范将其分配给worker。每个worker任务（`partial_dimension_distribution` 类型）读取分配的分割并为 `partitionDimension` 构建直方图。并行任务从worker任务收集这些直方图，并根据 `partitionDimension` 找到最佳范围分区，以便在分区之间均匀分布行。请注意，`targetRowsPerSegment` 或 `maxRowsPerSegment` 将用于查找最佳分区。 -* 在 `部分段生成` 阶段，并行任务生成新的worker任务（`partial_range_index_generate` 类型）以创建分区数据。每个worker任务都读取在前一阶段中创建的分割，根据 `granularitySpec` 中的`segmentGranularity（主分区键）`的时间块对行进行分区，然后根据在前一阶段中找到的范围分区对行进行分区。分区数据存储在 [MiddleManager](../Design/MiddleManager.md) 或 [Indexer](../Design/Indexer.md)的本地存储中。 +* 在 `部分段生成` 阶段，并行任务生成新的worker任务（`partial_range_index_generate` 类型）以创建分区数据。每个worker任务都读取在前一阶段中创建的分割，根据 `granularitySpec` 中的`segmentGranularity（主分区键）`的时间块对行进行分区，然后根据在前一阶段中找到的范围分区对行进行分区。分区数据存储在 [MiddleManager](../design/MiddleManager.md) 或 [Indexer](../design/Indexer.md)的本地存储中。 * 在 `部分段合并` 阶段，并行索引任务生成一组新的worker任务（`partial_index_generic_merge`类型）来合并在上一阶段创建的分区数据。这里，分区数据根据时间块和 `partitionDimension` 的值进行洗牌；每个工作任务从多个MiddleManager/Indexer进程中读取属于同一范围的同一分区中的段，并将它们合并以创建最后的段。最后，它们将最后的段推到深层存储。 > [!WARNING] @@ -654,7 +654,7 @@ PartitionsSpec用于描述辅助分区方法。您应该根据需要的rollup模 #### S3输入源 > [!WARNING] -> 您需要添加 [`druid-s3-extensions`](../Development/S3-compatible.md) 扩展以便使用S3输入源。 +> 您需要添加 [`druid-s3-extensions`](../development/S3-compatible.md) 扩展以便使用S3输入源。 S3输入源支持直接从S3读取对象。可以通过S3 URI字符串列表或S3位置前缀列表指定对象，该列表将尝试列出内容并摄取位置中包含的所有对象。S3输入源是可拆分的，可以由 [并行任务](#并行任务) 使用，其中 `index_parallel` 的每个worker任务将读取一个或多个对象。 @@ -734,7 +734,7 @@ S3对象： | `accessKeyId` | S3输入源访问密钥的 [Password Provider](../Operations/passwordproviders.md) 或纯文本字符串 | None | 如果 `secretAccessKey` 被提供的话，则为必须 | | `secretAccessKey` | S3输入源访问密钥的 [Password Provider](../Operations/passwordproviders.md) 或纯文本字符串 | None | 如果 `accessKeyId` 被提供的话，则为必须 | -**注意**： *如果 `accessKeyId` 和 `secretAccessKey` 未被指定的话， 则将使用默认的 [S3认证](../Development/S3-compatible.md#S3认证方式)* +**注意**： *如果 `accessKeyId` 和 `secretAccessKey` 未被指定的话， 则将使用默认的 [S3认证](../development/S3-compatible.md#S3认证方式)* #### 谷歌云存储输入源 diff --git a/DataIngestion/taskrefer.md b/DataIngestion/taskrefer.md index df43d4c..0f25789 100644 --- a/DataIngestion/taskrefer.md +++ b/DataIngestion/taskrefer.md @@ -21,7 +21,7 @@ 任务API主要在两个地方是可用的： -* [Overlord](../Design/Overlord.md) 进程提供HTTP API接口来进行提交任务、取消任务、检查任务状态、查看任务日志与报告等。 查看 [任务API文档](../Operations/api.md) 可以看到完整列表 +* [Overlord](../design/Overlord.md) 进程提供HTTP API接口来进行提交任务、取消任务、检查任务状态、查看任务日志与报告等。 查看 [任务API文档](../Operations/api.md) 可以看到完整列表 * Druid SQL包括了一个 [`sys.tasks`](../Querying/druidsql.md#系统Schema) ，保存了当前任务运行的信息。 此表是只读的，并且可以通过Overlord API查询完整信息的有限制的子集。 ### 任务报告 diff --git a/Development/index.md b/Development/index.md deleted file mode 100644 index 880f21d..0000000 --- a/Development/index.md +++ /dev/null @@ -1 +0,0 @@ -## 开发指南 \ No newline at end of file diff --git a/Development/thrift.md b/Development/thrift.md deleted file mode 100644 index 1eef3b8..0000000 --- a/Development/thrift.md +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/Querying/Aggregations.md b/Querying/Aggregations.md index a5fbb5a..b4ef4db 100644 --- a/Querying/Aggregations.md +++ b/Querying/Aggregations.md @@ -306,7 +306,7 @@ Double/Float/Long/String的ANY聚合器不能够使用在摄入规范中，只 ``` > [!WARNING] -> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../Development/JavaScript.md)。 +> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../development/JavaScript.md)。 ### 近似聚合(Approximate Aggregations) #### 唯一计数(Count distinct) diff --git a/Querying/dimensionspec.md b/Querying/dimensionspec.md index 46df9bc..6630b0c 100644 --- a/Querying/dimensionspec.md +++ b/Querying/dimensionspec.md @@ -60,7 +60,7 @@ 该项功能仅仅对多值维度是比较有用的。如果你在Apache Druid中有一个值为 ["v1","v2","v3"] 的行，当发送一个带有对维度值为"v1"进行[查询过滤](filters.md)的GroupBy/TopN查询, 在响应中，将会得到包含"v1","v2","v3"的三行数据。这个行为在大多数场景是不适合的。 -之所以会发生这种情况，是因为"查询过滤器"是在位图上内部使用的，并且只用于匹配要包含在查询结果处理中的行。对于多值维度，"查询过滤器"的行为类似于包含检查，它将匹配维度值为["v1"、"v2"、"v3"]的行。有关更多详细信息，请参阅[段](../Design/Segments.md)中"多值列"一节, 然后groupBy/topN处理管道"分解"所有多值维度，得到3行"v1"、"v2"和"v3"。 +之所以会发生这种情况，是因为"查询过滤器"是在位图上内部使用的，并且只用于匹配要包含在查询结果处理中的行。对于多值维度，"查询过滤器"的行为类似于包含检查，它将匹配维度值为["v1"、"v2"、"v3"]的行。有关更多详细信息，请参阅[段](../design/Segments.md)中"多值列"一节, 然后groupBy/topN处理管道"分解"所有多值维度，得到3行"v1"、"v2"和"v3"。 除了有效地选择要处理的行的"查询过滤器"之外，还可以使用带过滤的DimensionSpec来筛选多值维度值中的特定值。这些维度规范采用代理维度规范和筛选条件。从"分解"行中，查询结果中只返回与给定筛选条件匹配的行。 @@ -87,7 +87,7 @@ #### 带Lookup的DimensionSpec > [!WARNING] -> Lookups是一个[实验性的特性](../Development/experimental.md) +> Lookups是一个[实验性的特性](../development/experimental.md) 带Lookup的DimensionSpec可用于将lookup实现直接定义为维度规范。一般来说，有两种不同类型的查找实现。第一种是在查询时像map实现一样传递的。 @@ -296,7 +296,7 @@ null字符串被认定为长度为0 ``` > [!WARNING] -> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../Development/JavaScript.md)。 +> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../development/JavaScript.md)。 #### 已注册的Lookup提取函数 diff --git a/Querying/druidsql.md b/Querying/druidsql.md index c006050..753193a 100644 --- a/Querying/druidsql.md +++ b/Querying/druidsql.md @@ -131,7 +131,7 @@ Druid的原生类型系统允许字符串可能有多个值。这些 [多值维 在默认模式(`true`)下，Druid将NULL和空字符串互换处理，而不是根据SQL标准。在这种模式下，Druid SQL只部分支持NULL。例如，表达式 `col IS NULL` 和 `col = ''` 等效，如果 `col` 包含空字符串，则两者的计算结果都为true。类似地，如果`col1`是空字符串，则表达式 `COALESCE(col1，col2)` 将返回 `col2`。当 `COUNT(*)` 聚合器计算所有行时，`COUNT(expr)` 聚合器将计算expr既不为空也不为空字符串的行数。此模式中的数值列不可为空；任何空值或缺少的值都将被视为零。 -在SQL兼容模式(`false`)中，NULL的处理更接近SQL标准，该属性同时影响存储和查询，因此为了获得最佳行为，应该在接收时和查询时同时设置该属性。处理空值的能力会带来一些开销；有关更多详细信息，请参阅 [段文档](../Design/Segments.md#SQL兼容的空值处理)。 +在SQL兼容模式(`false`)中，NULL的处理更接近SQL标准，该属性同时影响存储和查询，因此为了获得最佳行为，应该在接收时和查询时同时设置该属性。处理空值的能力会带来一些开销；有关更多详细信息，请参阅 [段文档](../design/Segments.md#SQL兼容的空值处理)。 ### 聚合函数 @@ -148,14 +148,14 @@ Druid的原生类型系统允许字符串可能有多个值。这些 [多值维 | `MAX(expr)` | 取数字的最大值 | | `AVG(expr)` | 取平均值 | | `APPROX_COUNT_DISTINCT(expr)` | 唯一值的计数，该值可以是常规列或hyperUnique。这始终是近似值，而不考虑"useApproximateCountDistinct"的值。该函数使用了Druid内置的"cardinality"或"hyperUnique"聚合器。另请参见 `COUNT(DISTINCT expr)` | -| `APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])` | 唯一值的计数，该值可以是常规列或[HLL sketch](../Configuration/core-ext/datasketches-hll.md)。`lgk` 和 `tgtHllType` 参数在HLL Sketch文档中做了描述。 该值也始终是近似值，而不考虑"useApproximateCountDistinct"的值。另请参见 `COUNT(DISTINCT expr)`, 使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | -| `APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])` | 唯一值的计数，该值可以是常规列或[Theta sketch](../Configuration/core-ext/datasketches-theta.md)。`size` 参数在Theta Sketch文档中做了描述。 该值也始终是近似值，而不考虑"useApproximateCountDistinct"的值。另请参见 `COUNT(DISTINCT expr)`, 使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | -| `DS_HLL(expr, [lgK, tgtHllType])` | 在表达式的值上创建一个 [`HLL sketch`](../Configuration/core-ext/datasketches-hll.md), 该值可以是常规列或者包括HLL Sketch的列。`lgk` 和 `tgtHllType` 参数在HLL Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | -| `DS_THETA(expr, [size])` | 在表达式的值上创建一个[`Theta sketch`](../Configuration/core-ext/datasketches-theta.md)，该值可以是常规列或者包括Theta Sketch的列。`size` 参数在Theta Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | +| `APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])` | 唯一值的计数，该值可以是常规列或[HLL sketch](../Configuration/core-ext/datasketches-hll.md)。`lgk` 和 `tgtHllType` 参数在HLL Sketch文档中做了描述。 该值也始终是近似值，而不考虑"useApproximateCountDistinct"的值。另请参见 `COUNT(DISTINCT expr)`, 使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | +| `APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])` | 唯一值的计数，该值可以是常规列或[Theta sketch](../Configuration/core-ext/datasketches-theta.md)。`size` 参数在Theta Sketch文档中做了描述。 该值也始终是近似值，而不考虑"useApproximateCountDistinct"的值。另请参见 `COUNT(DISTINCT expr)`, 使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | +| `DS_HLL(expr, [lgK, tgtHllType])` | 在表达式的值上创建一个 [`HLL sketch`](../Configuration/core-ext/datasketches-hll.md), 该值可以是常规列或者包括HLL Sketch的列。`lgk` 和 `tgtHllType` 参数在HLL Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | +| `DS_THETA(expr, [size])` | 在表达式的值上创建一个[`Theta sketch`](../Configuration/core-ext/datasketches-theta.md)，该值可以是常规列或者包括Theta Sketch的列。`size` 参数在Theta Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | | `APPROX_QUANTILE(expr, probability, [resolution])` | 在数值表达式或者[近似图](../Configuration/core-ext/approximate-histograms.md) 表达式上计算近似分位数，"probability"应该是位于0到1之间（不包括1），"resolution"是用于计算的centroids，更高的resolution将会获得更精确的结果，默认值为50。使用该函数需要加载 [近似直方图扩展](../Configuration/core-ext/approximate-histograms.md) | -| `APPROX_QUANTILE_DS(expr, probability, [k])` | 在数值表达式或者 [Quantiles sketch](../Configuration/core-ext/datasketches-quantiles.md) 表达式上计算近似分位数，"probability"应该是位于0到1之间（不包括1）, `k`参数在Quantiles Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | +| `APPROX_QUANTILE_DS(expr, probability, [k])` | 在数值表达式或者 [Quantiles sketch](../Configuration/core-ext/datasketches-quantiles.md) 表达式上计算近似分位数，"probability"应该是位于0到1之间（不包括1）, `k`参数在Quantiles Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | | `APPROX_QUANTILE_FIXED_BUCKETS(expr, probability, numBuckets, lowerLimit, upperLimit, [outlierHandlingMode])` | 在数值表达式或者[fixed buckets直方图](../Configuration/core-ext/approximate-histograms.md) 表达式上计算近似分位数，"probability"应该是位于0到1之间（不包括1）, `numBuckets`, `lowerLimit`, `upperLimit` 和 `outlierHandlingMode` 参数在fixed buckets直方图文档中做了描述。 使用该函数需要加载 [近似直方图扩展](../Configuration/core-ext/approximate-histograms.md) | -| `DS_QUANTILES_SKETCH(expr, [k])` | 在表达式的值上创建一个[`Quantiles sketch`](../Configuration/core-ext/datasketches-quantiles.md)，该值可以是常规列或者包括Quantiles Sketch的列。`k`参数在Quantiles Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../Development/datasketches-extension.md) | +| `DS_QUANTILES_SKETCH(expr, [k])` | 在表达式的值上创建一个[`Quantiles sketch`](../Configuration/core-ext/datasketches-quantiles.md)，该值可以是常规列或者包括Quantiles Sketch的列。`k`参数在Quantiles Sketch文档中做了描述。使用该函数需要加载 [DataSketches扩展](../development/datasketches-extension.md) | | `BLOOM_FILTER(expr, numEntries)` | 根据`expr`生成的值计算bloom筛选器，其中`numEntries`在假阳性率增加之前具有最大数量的不同值。详细可以参见 [Bloom过滤器扩展](../Configuration/core-ext/bloom-filter.md) | | `TDIGEST_QUANTILE(expr, quantileFraction, [compression])` | 根据`expr`生成的值构建一个T-Digest sketch，并返回分位数的值。"compression"（默认值100）确定sketch的精度和大小。更高的compression意味着更高的精度，但更多的空间来存储sketch。有关更多详细信息，请参阅 [t-digest扩展文档](../Configuration/core-ext/tdigestsketch-quantiles.md) | | `TDIGEST_GENERATE_SKETCH(expr, [compression])` | 根据`expr`生成的值构建一个T-Digest sketch。"compression"（默认值100）确定sketch的精度和大小。更高的compression意味着更高的精度，但更多的空间来存储sketch。有关更多详细信息，请参阅 [t-digest扩展文档](../Configuration/core-ext/tdigestsketch-quantiles.md) | @@ -326,7 +326,7 @@ Druid的原生类型系统允许字符串可能有多个值。这些 [多值维 **HLL Sketch函数** -以下函数操作在 [DataSketches HLL sketches](../Configuration/core-ext/datasketches-hll.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../Development/datasketches-extension.md) +以下函数操作在 [DataSketches HLL sketches](../Configuration/core-ext/datasketches-hll.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../development/datasketches-extension.md) | 函数 | 描述 | |-|-| @@ -337,7 +337,7 @@ Druid的原生类型系统允许字符串可能有多个值。这些 [多值维 **Theta Sketch函数** -以下函数操作在 [theta sketches](../Configuration/core-ext/datasketches-theta.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../Development/datasketches-extension.md) +以下函数操作在 [theta sketches](../Configuration/core-ext/datasketches-theta.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../development/datasketches-extension.md) | 函数 | 描述 | |-|-| @@ -349,7 +349,7 @@ Druid的原生类型系统允许字符串可能有多个值。这些 [多值维 **Quantiles Sketch函数** -以下函数操作在 [quantiles sketches](../Configuration/core-ext/datasketches-quantiles.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../Development/datasketches-extension.md) +以下函数操作在 [quantiles sketches](../Configuration/core-ext/datasketches-quantiles.md) 之上，使用这些函数之前需要加载 [DataSketches扩展](../development/datasketches-extension.md) | 函数 | 描述 | |-|-| @@ -647,7 +647,7 @@ try (Connection connection = DriverManager.getConnection(url, connectionProperti **连接粘性** -Druid的JDBC服务不在Broker之间共享连接状态。这意味着，如果您使用JDBC并且有多个Druid Broker，您应该连接到一个特定的Broker，或者使用启用了粘性会话的负载平衡器。Druid Router进程在平衡JDBC请求时提供连接粘性，即使使用普通的非粘性负载平衡器，也可以用来实现必要的粘性。请参阅 [Router文档](../Design/Router.md) 以了解更多详细信息 +Druid的JDBC服务不在Broker之间共享连接状态。这意味着，如果您使用JDBC并且有多个Druid Broker，您应该连接到一个特定的Broker，或者使用启用了粘性会话的负载平衡器。Druid Router进程在平衡JDBC请求时提供连接粘性，即使使用普通的非粘性负载平衡器，也可以用来实现必要的粘性。请参阅 [Router文档](../design/Router.md) 以了解更多详细信息 注意：非JDBC的 [HTTP POST](#http-post) 是无状态的，不需要粘性 @@ -759,10 +759,10 @@ segments表提供了所有Druid段的详细信息，无论该段是否被发布 | `partition_num` | LONG | 分区号（整数，在数据源+间隔+版本中是唯一的；不一定是连续的） | | `num_replicas` | LONG | 当前正在服务的此段的副本数 | | `num_rows` | LONG | 当前段中的行数，如果查询时Broker未知，则此值可以为空 | -| `is_published` | LONG | 布尔值表示为long类型，其中1=true，0=false。1表示此段已发布到元数据存储且 `used=1`。详情查看 [架构页面](../Design/Design.md) | -| `is_available` | LONG | 布尔值表示为long类型，其中1=true，0=false。1表示此段当前由任何进程（Historical或Realtime）提供服务。详情查看 [架构页面](../Design/Design.md) | +| `is_published` | LONG | 布尔值表示为long类型，其中1=true，0=false。1表示此段已发布到元数据存储且 `used=1`。详情查看 [架构页面](../design/Design.md) | +| `is_available` | LONG | 布尔值表示为long类型，其中1=true，0=false。1表示此段当前由任何进程（Historical或Realtime）提供服务。详情查看 [架构页面](../design/Design.md) | | `is_realtime` | LONG | 布尔值表示为long类型，其中1=true，0=false。如果此段仅由实时任务提供服务，则为1；如果任何Historical进程正在为此段提供服务，则为0。 | -| `is_overshadowed` | LONG | 布尔值表示为long类型，其中1=true，0=false。如果此段已发布，并且被其他已发布的段完全覆盖则为1。目前，对于未发布的段，`is_overshadowed` 总是false，尽管这在未来可能会改变。可以通过过滤 `is_published=1` 和 `is_overshadowed=0` 来筛选"应该发布"的段。如果段最近被替换，它们可以短暂地被发布，也可以被掩盖，但还没有被取消发布。详情查看 [架构页面](../Design/Design.md) | +| `is_overshadowed` | LONG | 布尔值表示为long类型，其中1=true，0=false。如果此段已发布，并且被其他已发布的段完全覆盖则为1。目前，对于未发布的段，`is_overshadowed` 总是false，尽管这在未来可能会改变。可以通过过滤 `is_published=1` 和 `is_overshadowed=0` 来筛选"应该发布"的段。如果段最近被替换，它们可以短暂地被发布，也可以被掩盖，但还没有被取消发布。详情查看 [架构页面](../design/Design.md) | | `payload` | STRING | JSON序列化数据段负载 | 例如，要检索数据源"wikipedia"的所有段，请使用查询： diff --git a/Querying/filters.md b/Querying/filters.md index fd8f255..7640c5d 100644 --- a/Querying/filters.md +++ b/Querying/filters.md @@ -116,7 +116,7 @@ JavaScript函数需要一个维度值的参数，返回值要么是true或者fal JavaScript过滤器支持使用提取函数，详情可见 [带提取函数的过滤器](#带提取函数的过滤器) > [!WARNING] -> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../Development/JavaScript.md)。 +> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../development/JavaScript.md)。 ### **提取过滤器(Extraction Filter)** diff --git a/Querying/lookups.md b/Querying/lookups.md index 8b38b31..eb5b0ff 100644 --- a/Querying/lookups.md +++ b/Querying/lookups.md @@ -2,7 +2,7 @@ ## Lookups > [!WARNING] -> Lookups是一个 [实验性的特性](../Development/experimental.md) +> Lookups是一个 [实验性的特性](../development/experimental.md) Lookups是Apache Druid中的一个概念，在Druid中维度值(可选地)被新值替换，从而允许类似join的功能。在Druid中应用Lookup类似于在数据仓库中的联接维度表。有关详细信息，请参见 [维度说明](querydimensions.md)。在这些文档中，"key"是指要匹配的维度值，"value"是指其替换的目标值。所以如果你想把 `appid-12345` 映射到`Super Mega Awesome App`，那么键应该是 `appid-12345`，值就是 `Super Mega Awesome App`。 @@ -85,7 +85,7 @@ GROUP BY 1 ### 动态配置 > [!WARNING] -> 动态Lookup配置是一个 [实验特性](../Development/experimental.md), 不再支持静态配置。下面的文档说明了集群范围的配置，该配置可以通过Coordinator进行访问。配置通过服务器的"tier"概念传播。"tier"被定义为一个应该接收一组Lookup的服务集合。例如，您可以让所有Historical都是 `_default`，而Peon是它们所负责的数据源的各个层的一部分。Lookups的tier完全独立于Historical tiers。 +> 动态Lookup配置是一个 [实验特性](../development/experimental.md), 不再支持静态配置。下面的文档说明了集群范围的配置，该配置可以通过Coordinator进行访问。配置通过服务器的"tier"概念传播。"tier"被定义为一个应该接收一组Lookup的服务集合。例如，您可以让所有Historical都是 `_default`，而Peon是它们所负责的数据源的各个层的一部分。Lookups的tier完全独立于Historical tiers。 这些配置都可以通过以下URI模板来使用JSON获取到： diff --git a/Querying/makeNativeQueries.md b/Querying/makeNativeQueries.md index ccbc25f..40261ba 100644 --- a/Querying/makeNativeQueries.md +++ b/Querying/makeNativeQueries.md @@ -36,7 +36,7 @@ curl -X POST ':/druid/v2/?pretty' -H 'Content-Type:applica Druid的原生查询级别相对较低，与内部执行计算的方式密切相关。Druid查询被设计成轻量级的，并且非常快速地完成。这意味着对于更复杂的分析，或者构建更复杂的可视化，可能需要多个Druid查询。 -即使查询通常是向Broker或Router发出的，但是它们也可以被 [Historical进程](../Design/Historical.md) 和运行流摄取任务的 [peon(任务jvm)](../Design/Peons.md) 接受。如果您想查询由特定进程提供服务的特定段的结果，这可能很有价值。 +即使查询通常是向Broker或Router发出的，但是它们也可以被 [Historical进程](../design/Historical.md) 和运行流摄取任务的 [peon(任务jvm)](../design/Peons.md) 接受。如果您想查询由特定进程提供服务的特定段的结果，这可能很有价值。 ### 可用的查询 diff --git a/Querying/multi-value-dimensions.md b/Querying/multi-value-dimensions.md index 7c9a129..e8cb518 100644 --- a/Querying/multi-value-dimensions.md +++ b/Querying/multi-value-dimensions.md @@ -3,7 +3,7 @@ Apache Druid支持多值字符串维度。当输入字段中包括一个数组值而非单一值（例如，JSON数组，或者包括多个 `listDelimiter` 分割的TSV字段）时即可生成多值维度。 -本文档描述了对一个维度进行聚合时，多值维度上的GroupBy查询行为（TopN很类似）。对于多值维度的内部详细信息可以查看 [Segments](../Design/Segments.md) 文档的多值列部分。本文档中的示例都为 [原生Druid查询](makeNativeQueries.md)格式，对于多值维度在SQL中的使用情况请查阅 [Druid SQL 文档](druidsql.md) +本文档描述了对一个维度进行聚合时，多值维度上的GroupBy查询行为（TopN很类似）。对于多值维度的内部详细信息可以查看 [Segments](../design/Segments.md) 文档的多值列部分。本文档中的示例都为 [原生Druid查询](makeNativeQueries.md)格式，对于多值维度在SQL中的使用情况请查阅 [Druid SQL 文档](druidsql.md) [!WARNING] -> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../Development/JavaScript.md)。 +> 基于JavaScript的功能默认是禁用的。 如何启用它以及如何使用Druid JavaScript功能，参考 [JavaScript编程指南](../development/JavaScript.md)。 ### 超唯一基数后置聚合器(HyperUnique Cardinality post-aggregator) diff --git a/Querying/queryexecution.md b/Querying/queryexecution.md index 027d416..c3c9510 100644 --- a/Querying/queryexecution.md +++ b/Querying/queryexecution.md @@ -23,7 +23,7 @@ Druid的查询执行方法因查询的 [数据源类型](#数据源类型) 而 直接在 [表数据源](datasource.md#table) 上操作的查询使用由Broker进程引导的**分散-聚集**方法执行。过程如下： -1. Broker根据 `"interval"` 参数确定哪些 [段](../Design/Segments.md) 与查询相关。段总是按时间划分的，因此任何间隔与查询间隔重叠的段都可能是相关的。 +1. Broker根据 `"interval"` 参数确定哪些 [段](../design/Segments.md) 与查询相关。段总是按时间划分的，因此任何间隔与查询间隔重叠的段都可能是相关的。 2. 如果输入数据使用 [`single_dim` partitionsSpec](../DataIngestion/native.md#partitionsSpec) 按范围分区，并且过滤器与用于分区的维度匹配，则Broker还可以根据 `"filter"` 进一步修剪段列表。 3. Broker在删除了查询的段列表之后，将查询转发到当前为这些段提供服务的数据服务器（如Historical或者运行在MiddleManagers的任务）。 4. 对于除 [Scan](scan.md) 之外的所有查询类型，数据服务器并行处理每个段，并为每个段生成部分结果。所做的具体处理取决于查询类型。如果启用了 [查询缓存](querycached.md)，则可以缓存这些部分结果。对于Scan查询，段由单个线程按顺序处理，结果不被缓存。 diff --git a/SUMMARY.md b/SUMMARY.md index 44c6ff3..72bf859 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -16,7 +16,7 @@ * [新手入门]() * [Druid介绍](GettingStarted/chapter-1.md) * [快速开始](GettingStarted/chapter-2.md) - * [Docker](GettingStarted/Docker.md) + * [Docker](tutorials/docker.md) * [单服务器部署](GettingStarted/chapter-3.md) * [集群部署](GettingStarted/chapter-4.md) @@ -35,20 +35,20 @@ * [Kerberized HDFS存储](tutorials/chapter-12.md) * [架构设计]() - * [整体设计](Design/Design.md) - * [段设计](Design/Segments.md) - * [进程与服务](Design/Processes.md) - * [Coordinator](Design/Coordinator.md) - * [Overlord](Design/Overlord.md) - * [Historical](Design/Historical.md) - * [MiddleManager](Design/MiddleManager.md) - * [Broker](Design/Broker.md) - * [Router](Design/Router.md) - * [Indexer](Design/Indexer.md) - * [Peon](Design/Peons.md) - * [深度存储](Design/Deepstorage.md) - * [元数据存储](Design/Metadata.md) - * [Zookeeper](Design/Zookeeper.md) + * [整体设计](design/Design.md) + * [段设计](design/Segments.md) + * [进程与服务](design/Processes.md) + * [Coordinator](design/Coordinator.md) + * [Overlord](design/Overlord.md) + * [Historical](design/Historical.md) + * [MiddleManager](design/MiddleManager.md) + * [Broker](design/Broker.md) + * [Router](design/Router.md) + * [Indexer](design/Indexer.md) + * [Peon](design/Peons.md) + * [深度存储](design/Deepstorage.md) + * [元数据存储](design/Metadata.md) + * [Zookeeper](design/Zookeeper.md) * [数据摄取]() * [摄取概述](DataIngestion/ingestion.md) @@ -107,7 +107,7 @@ * [操作指南](Operations/index.md) * [开发指南]() - * [开发指南](Development/index.md) + * [开发指南](development/index.md) * [其他相关]() * [其他相关](misc/index.md) diff --git a/_sidebar.md b/_sidebar.md index c0bdd0c..5ff90f6 100644 --- a/_sidebar.md +++ b/_sidebar.md @@ -3,9 +3,9 @@ - [公众平台](CONTACT.md) - 开始使用 - - [从文件中载入数据](yong-zhou/ling-ling/mao-ping-li-cun/index.md) - - [从 Kafka 中载入数据](yong-zhou/ling-ling/tang-fu-cun/index.md) - - [从 Hadoop 中载入数据](yong-zhou/ling-ling/zhao-jia-wan-cun/index.md) + - [Druid 介绍](design/index.md) + - [快速开始](tutorials/index.md) + - [Docker 容器](tutorials/docker.md) - 设计（Design） - [JWT](jwt/README.md) @@ -16,7 +16,11 @@ - [面试问题和经验](interview/index.md) - [算法题](algorithm/index.md) - 查询（Querying） - + +- 开发（Development） + - [在 Druid 中进行开发](development/index.md) + - [创建扩展（extensions）](development/modules.md) + - 其他杂项（Misc） - [Druid 资源快速导航](misc/index.md) diff --git a/Design/Broker.md b/design/Broker.md similarity index 100% rename from Design/Broker.md rename to design/Broker.md diff --git a/Design/Coordinator.md b/design/Coordinator.md similarity index 100% rename from Design/Coordinator.md rename to design/Coordinator.md diff --git a/Design/Deepstorage.md b/design/Deepstorage.md similarity index 100% rename from Design/Deepstorage.md rename to design/Deepstorage.md diff --git a/Design/Design.md b/design/Design.md similarity index 94% rename from Design/Design.md rename to design/Design.md index 935d1e9..631c42a 100644 --- a/Design/Design.md +++ b/design/Design.md @@ -139,7 +139,7 @@ clarity-cloud0_2018-05-21T16:00:00.000Z_2018-05-21T17:00:00.000Z_2018-05-21T15:5 ### 查询处理 -查询首先进入[Broker](../Design/Broker.md), Broker首先鉴别哪些段可能与本次查询有关。 段的列表总是按照时间进行筛选和修剪的，当然也可能由其他属性，具体取决于数据源的分区方式。然后，Broker将确定哪些[Historical](../Design/Historical.md)和[MiddleManager](../Design/MiddleManager.md)为这些段提供服务、并向每个进程发送一个子查询。 Historical和MiddleManager进程接收查询、处理查询并返回结果，Broker将接收到的结果合并到一起形成最后的结果集返回给调用者。 +查询首先进入[Broker](/Broker.md), Broker首先鉴别哪些段可能与本次查询有关。 段的列表总是按照时间进行筛选和修剪的，当然也可能由其他属性，具体取决于数据源的分区方式。然后，Broker将确定哪些[Historical](/Historical.md)和[MiddleManager](/MiddleManager.md)为这些段提供服务、并向每个进程发送一个子查询。 Historical和MiddleManager进程接收查询、处理查询并返回结果，Broker将接收到的结果合并到一起形成最后的结果集返回给调用者。 Broker精简是Druid限制每个查询扫描数据量的一个重要方法，但不是唯一的方法。对于比Broker更细粒度级别的精简筛选器，每个段中的索引结构允许Druid在查看任何数据行之前，找出哪些行（如果有的话）与筛选器集匹配。一旦Druid知道哪些行与特定查询匹配，它就只访问该查询所需的特定列。在这些列中，Druid可以从一行跳到另一行，避免读取与查询过滤器不匹配的数据。 diff --git a/Design/Historical.md b/design/Historical.md similarity index 100% rename from Design/Historical.md rename to design/Historical.md diff --git a/Design/Indexer.md b/design/Indexer.md similarity index 98% rename from Design/Indexer.md rename to design/Indexer.md index 651a26f..367562b 100644 --- a/Design/Indexer.md +++ b/design/Indexer.md @@ -14,7 +14,7 @@ ## Indexer > [!WARNING] -> 索引器是一个可选的和[实验性](../Development/experimental.md)的功能, 其内存管理系统仍在开发中，并将在以后的版本中得到显著增强。 +> 索引器是一个可选的和[实验性](../development/experimental.md)的功能, 其内存管理系统仍在开发中，并将在以后的版本中得到显著增强。 Apache Druid索引器进程是MiddleManager + Peon任务执行系统的另一种可替代选择。索引器在单个JVM进程中作为单独的线程运行任务，而不是为每个任务派生单独的JVM进程。 diff --git a/Design/Metadata.md b/design/Metadata.md similarity index 100% rename from Design/Metadata.md rename to design/Metadata.md diff --git a/Design/MiddleManager.md b/design/MiddleManager.md similarity index 100% rename from Design/MiddleManager.md rename to design/MiddleManager.md diff --git a/Design/Overlord.md b/design/Overlord.md similarity index 100% rename from Design/Overlord.md rename to design/Overlord.md diff --git a/Design/Peons.md b/design/Peons.md similarity index 100% rename from Design/Peons.md rename to design/Peons.md diff --git a/Design/Processes.md b/design/Processes.md similarity index 98% rename from Design/Processes.md rename to design/Processes.md index 4f83542..d6a558d 100644 --- a/Design/Processes.md +++ b/design/Processes.md @@ -86,7 +86,7 @@ Data服务执行摄取作业并存储可查询数据。 [Indexer](./Indexer.md) 进程是MiddleManager和Peon的替代方法。Indexer在单个JVM进程中作为单个线程运行任务，而不是为每个任务派生单独的JVM进程。 -与MiddleManager + Peon系统相比，Indexer的设计更易于配置和部署，并且能够更好地实现跨任务的资源共享。Indexer是一种较新的功能，由于其内存管理系统仍在开发中，因此目前被指定为[实验性的特性](../Development/experimental.md)。它将在Druid的未来版本中继续成熟。 +与MiddleManager + Peon系统相比，Indexer的设计更易于配置和部署，并且能够更好地实现跨任务的资源共享。Indexer是一种较新的功能，由于其内存管理系统仍在开发中，因此目前被指定为[实验性的特性](../development/experimental.md)。它将在Druid的未来版本中继续成熟。 通常，您可以部署MiddleManagers或indexer，但不能同时部署两者。 diff --git a/Design/Router.md b/design/Router.md similarity index 99% rename from Design/Router.md rename to design/Router.md index a6093ce..af4bc7f 100644 --- a/Design/Router.md +++ b/design/Router.md @@ -98,7 +98,7 @@ Router有一个可配置的策略列表，用于选择将查询路由到哪个Br ``` > [!WARNING] -> 默认情况下禁用基于JavaScript的功能。有关使用Druid的JavaScript功能的指南，包括如何启用它的说明，请参阅[Druid JavaScript编程指南](../Development/JavaScript.md)。 +> 默认情况下禁用基于JavaScript的功能。有关使用Druid的JavaScript功能的指南，包括如何启用它的说明，请参阅[Druid JavaScript编程指南](../development/JavaScript.md)。 ### Avatica查询平衡 diff --git a/Design/Segments.md b/design/Segments.md similarity index 100% rename from Design/Segments.md rename to design/Segments.md diff --git a/Design/Zookeeper.md b/design/Zookeeper.md similarity index 100% rename from Design/Zookeeper.md rename to design/Zookeeper.md diff --git a/Design/img/druid-architecture.png b/design/img/druid-architecture.png similarity index 100% rename from Design/img/druid-architecture.png rename to design/img/druid-architecture.png diff --git a/Design/img/druid-column-types.png b/design/img/druid-column-types.png similarity index 100% rename from Design/img/druid-column-types.png rename to design/img/druid-column-types.png diff --git a/Design/img/druid-timeline.png b/design/img/druid-timeline.png similarity index 100% rename from Design/img/druid-timeline.png rename to design/img/druid-timeline.png diff --git a/design/index.md b/design/index.md new file mode 100644 index 0000000..d803a70 --- /dev/null +++ b/design/index.md @@ -0,0 +1,100 @@ +--- +id: index +title: "Introduction to Apache Druid" +--- + + + +## What is Druid? + +Apache Druid is a real-time analytics database designed for fast slice-and-dice analytics +("[OLAP](http://en.wikipedia.org/wiki/Online_analytical_processing)" queries) on large data sets. Druid is most often +used as a database for powering use cases where real-time ingest, fast query performance, and high uptime are important. +As such, Druid is commonly used for powering GUIs of analytical applications, or as a backend for highly-concurrent APIs +that need fast aggregations. Druid works best with event-oriented data. + +Common application areas for Druid include: + +- Clickstream analytics (web and mobile analytics) +- Network telemetry analytics (network performance monitoring) +- Server metrics storage +- Supply chain analytics (manufacturing metrics) +- Application performance metrics +- Digital marketing/advertising analytics +- Business intelligence / OLAP + +Druid's core architecture combines ideas from data warehouses, timeseries databases, and logsearch systems. Some of +Druid's key features are: + +1. **Columnar storage format.** Druid uses column-oriented storage, meaning it only needs to load the exact columns +needed for a particular query. This gives a huge speed boost to queries that only hit a few columns. In addition, each +column is stored optimized for its particular data type, which supports fast scans and aggregations. +2. **Scalable distributed system.** Druid is typically deployed in clusters of tens to hundreds of servers, and can +offer ingest rates of millions of records/sec, retention of trillions of records, and query latencies of sub-second to a +few seconds. +3. **Massively parallel processing.** Druid can process a query in parallel across the entire cluster. +4. **Realtime or batch ingestion.** Druid can ingest data either real-time (ingested data is immediately available for +querying) or in batches. +5. **Self-healing, self-balancing, easy to operate.** As an operator, to scale the cluster out or in, simply add or +remove servers and the cluster will rebalance itself automatically, in the background, without any downtime. If any +Druid servers fail, the system will automatically route around the damage until those servers can be replaced. Druid +is designed to run 24/7 with no need for planned downtimes for any reason, including configuration changes and software +updates. +6. **Cloud-native, fault-tolerant architecture that won't lose data.** Once Druid has ingested your data, a copy is +stored safely in [deep storage](architecture.html#deep-storage) (typically cloud storage, HDFS, or a shared filesystem). +Your data can be recovered from deep storage even if every single Druid server fails. For more limited failures affecting +just a few Druid servers, replication ensures that queries are still possible while the system recovers. +7. **Indexes for quick filtering.** Druid uses [Roaring](https://roaringbitmap.org/) or +[CONCISE](https://arxiv.org/pdf/1004.0403) compressed bitmap indexes to create indexes that power fast filtering and +searching across multiple columns. +8. **Time-based partitioning.** Druid first partitions data by time, and can additionally partition based on other fields. +This means time-based queries will only access the partitions that match the time range of the query. This leads to +significant performance improvements for time-based data. +9. **Approximate algorithms.** Druid includes algorithms for approximate count-distinct, approximate ranking, and +computation of approximate histograms and quantiles. These algorithms offer bounded memory usage and are often +substantially faster than exact computations. For situations where accuracy is more important than speed, Druid also +offers exact count-distinct and exact ranking. +10. **Automatic summarization at ingest time.** Druid optionally supports data summarization at ingestion time. This +summarization partially pre-aggregates your data, and can lead to big costs savings and performance boosts. + +## When should I use Druid? + +Druid is used by many companies of various sizes for many different use cases. Check out the +[Powered by Apache Druid](/druid-powered) page + +Druid is likely a good choice if your use case fits a few of the following descriptors: + +- Insert rates are very high, but updates are less common. +- Most of your queries are aggregation and reporting queries ("group by" queries). You may also have searching and +scanning queries. +- You are targeting query latencies of 100ms to a few seconds. +- Your data has a time component (Druid includes optimizations and design choices specifically related to time). +- You may have more than one table, but each query hits just one big distributed table. Queries may potentially hit more +than one smaller "lookup" table. +- You have high cardinality data columns (e.g. URLs, user IDs) and need fast counting and ranking over them. +- You want to load data from Kafka, HDFS, flat files, or object storage like Amazon S3. + +Situations where you would likely _not_ want to use Druid include: + +- You need low-latency updates of _existing_ records using a primary key. Druid supports streaming inserts, but not streaming updates (updates are done using +background batch jobs). +- You are building an offline reporting system where query latency is not very important. +- You want to do "big" joins (joining one big fact table to another big fact table) and you are okay with these queries +taking a long time to complete. diff --git a/Development/JavaScript.md b/development/JavaScript.md similarity index 100% rename from Development/JavaScript.md rename to development/JavaScript.md diff --git a/Development/S3-compatible.md b/development/S3-compatible.md similarity index 100% rename from Development/S3-compatible.md rename to development/S3-compatible.md diff --git a/Development/avro-extensions.md b/development/avro-extensions.md similarity index 100% rename from Development/avro-extensions.md rename to development/avro-extensions.md diff --git a/Development/datasketches-extension.md b/development/datasketches-extension.md similarity index 100% rename from Development/datasketches-extension.md rename to development/datasketches-extension.md diff --git a/Development/experimental.md b/development/experimental.md similarity index 100% rename from Development/experimental.md rename to development/experimental.md diff --git a/development/extensions-contrib/aliyun-oss-extensions.md b/development/extensions-contrib/aliyun-oss-extensions.md new file mode 100644 index 0000000..64fcd87 --- /dev/null +++ b/development/extensions-contrib/aliyun-oss-extensions.md @@ -0,0 +1,54 @@ +--- +id: aliyun-oss +title: "Aliyun OSS" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension. + +## Deep Storage + +[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss). + +To use aliyun OSS as deep storage, first config as below + +|Property|Description|Possible Values|Default| +|--------|---------------|-----------|-------| +|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.| +|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. | +|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.| + +if you want to use OSS as deep storage, use the configurations below + +|Property|Description|Possible Values|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.| +|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.| +|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | | + +To save index logs to OSS, apply the configurations below: + +|Property|Description|Possible Values|Default| +|--------|---------------|-----------|-------| +|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.| +|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.| +|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | | diff --git a/development/extensions-contrib/ambari-metrics-emitter.md b/development/extensions-contrib/ambari-metrics-emitter.md new file mode 100644 index 0000000..77443f3 --- /dev/null +++ b/development/extensions-contrib/ambari-metrics-emitter.md @@ -0,0 +1,99 @@ +--- +id: ambari-metrics-emitter +title: "Ambari Metrics Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `ambari-metrics-emitter` extension. + +## Introduction + +This extension emits Druid metrics to a ambari-metrics carbon server. +Events are sent after been [pickled](http://ambari-metrics.readthedocs.org/en/latest/feeding-carbon.html#the-pickle-protocol); the size of the batch is configurable. + +## Configuration + +All the configuration parameters for ambari-metrics emitter are under `druid.emitter.ambari-metrics`. + +|property|description|required?|default| +|--------|-----------|---------|-------| +|`druid.emitter.ambari-metrics.hostname`|The hostname of the ambari-metrics server.|yes|none| +|`druid.emitter.ambari-metrics.port`|The port of the ambari-metrics server.|yes|none| +|`druid.emitter.ambari-metrics.protocol`|The protocol used to send metrics to ambari metrics collector. One of http/https|no|http| +|`druid.emitter.ambari-metrics.trustStorePath`|Path to trustStore to be used for https|no|none| +|`druid.emitter.ambari-metrics.trustStoreType`|trustStore type to be used for https|no|none| +|`druid.emitter.ambari-metrics.trustStoreType`|trustStore password to be used for https|no|none| +|`druid.emitter.ambari-metrics.batchSize`|Number of events to send as one batch.|no|100| +|`druid.emitter.ambari-metrics.eventConverter`| Filter and converter of druid events to ambari-metrics timeline event(please see next section). |yes|none| +|`druid.emitter.ambari-metrics.flushPeriod` | Queue flushing period in milliseconds. |no|1 minute| +|`druid.emitter.ambari-metrics.maxQueueSize`| Maximum size of the queue used to buffer events. |no|`MAX_INT`| +|`druid.emitter.ambari-metrics.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.ambari-metrics.emitWaitTime` | wait time in milliseconds to try to send the event otherwise emitter will throwing event. |no|0| +|`druid.emitter.ambari-metrics.waitForEventTime` | waiting time in milliseconds if necessary for an event to become available. |no|1000 (1 sec)| + +### Druid to Ambari Metrics Timeline Event Converter + +Ambari Metrics Timeline Event Converter defines a mapping between druid metrics name plus dimensions to a timeline event metricName. +ambari-metrics metric path is organized using the following schema: +`.[].[]..` +Properly naming the metrics is critical to avoid conflicts, confusing data and potentially wrong interpretation later on. + +Example `druid.historical.hist-host1:8080.MyDataSourceName.GroupBy.query/time`: + + * `druid` -> namespace prefix + * `historical` -> service name + * `hist-host1:8080` -> druid hostname + * `MyDataSourceName` -> dimension value + * `GroupBy` -> dimension value + * `query/time` -> metric name + +We have two different implementation of event converter: + +#### Send-All converter + +The first implementation called `all`, will send all the druid service metrics events. +The path will be in the form `.[].[]..` +User has control of `.[].[].` + +```json + +druid.emitter.ambari-metrics.eventConverter={"type":"all", "namespacePrefix": "druid.test", "appName":"druid"} + +``` + +#### White-list based converter + +The second implementation called `whiteList`, will send only the white listed metrics and dimensions. +Same as for the `all` converter user has control of `.[].[].` +White-list based converter comes with the following default white list map located under resources in `./src/main/resources/defaultWhiteListMap.json` + +Although user can override the default white list map by supplying a property called `mapPath`. +This property is a String containing the path for the file containing **white list map JSON object**. +For example the following converter will read the map from the file `/pathPrefix/fileName.json`. + +```json + +druid.emitter.ambari-metrics.eventConverter={"type":"whiteList", "namespacePrefix": "druid.test", "ignoreHostname":true, "appName":"druid", "mapPath":"/pathPrefix/fileName.json"} + +``` + +**Druid emits a huge number of metrics we highly recommend to use the `whiteList` converter** diff --git a/development/extensions-contrib/cassandra.md b/development/extensions-contrib/cassandra.md new file mode 100644 index 0000000..d6ecc33 --- /dev/null +++ b/development/extensions-contrib/cassandra.md @@ -0,0 +1,30 @@ +--- +id: cassandra +title: "Apache Cassandra" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-cassandra-storage` extension. + +[Apache Cassandra](http://www.datastax.com/what-we-offer/products-services/datastax-enterprise/apache-cassandra) can also +be leveraged for deep storage. This requires some additional Druid configuration as well as setting up the necessary +schema within a Cassandra keystore. diff --git a/development/extensions-contrib/cloudfiles.md b/development/extensions-contrib/cloudfiles.md new file mode 100644 index 0000000..d6ccccc --- /dev/null +++ b/development/extensions-contrib/cloudfiles.md @@ -0,0 +1,98 @@ +--- +id: cloudfiles +title: "Rackspace Cloud Files" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-cloudfiles-extensions` extension. + +## Deep Storage + +[Rackspace Cloud Files](http://www.rackspace.com/cloud/files/) is another option for deep storage. This requires some additional Druid configuration. + +|Property|Possible Values|Description|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|cloudfiles||Must be set.| +|`druid.storage.region`||Rackspace Cloud Files region.|Must be set.| +|`druid.storage.container`||Rackspace Cloud Files container name.|Must be set.| +|`druid.storage.basePath`||Rackspace Cloud Files base path to use in the container.|Must be set.| +|`druid.storage.operationMaxRetries`||Number of tries before cancel a Rackspace operation.|10| +|`druid.cloudfiles.userName`||Rackspace Cloud username|Must be set.| +|`druid.cloudfiles.apiKey`||Rackspace Cloud API key.|Must be set.| +|`druid.cloudfiles.provider`|rackspace-cloudfiles-us,rackspace-cloudfiles-uk|Name of the provider depending on the region.|Must be set.| +|`druid.cloudfiles.useServiceNet`|true,false|Whether to use the internal service net.|true| + +## Firehose + + + +#### StaticCloudFilesFirehose + +This firehose ingests events, similar to the StaticAzureBlobStoreFirehose, but from Rackspace's Cloud Files. + +Data is newline delimited, with one JSON object per line and parsed as per the `InputRowParser` configuration. + +The storage account is shared with the one used for Rackspace's Cloud Files deep storage functionality, but blobs can be in a different region and container. + +As with the Azure blobstore, it is assumed to be gzipped if the extension ends in .gz + +This firehose is _splittable_ and can be used by [native parallel index tasks](../../ingestion/native-batch.md#parallel-task). +Since each split represents an object in this firehose, each worker task of `index_parallel` will read an object. + +Sample spec: + +```json +"firehose" : { + "type" : "static-cloudfiles", + "blobs": [ + { + "region": "DFW" + "container": "container", + "path": "/path/to/your/file.json" + }, + { + "region": "ORD" + "container": "anothercontainer", + "path": "/another/path.json" + } + ] +} +``` +This firehose provides caching and prefetching features. In IndexTask, a firehose can be read twice if intervals or +shardSpecs are not specified, and, in this case, caching can be useful. Prefetching is preferred when direct scan of objects is slow. + +|property|description|default|required?| +|--------|-----------|-------|---------| +|type|This should be `static-cloudfiles`.|N/A|yes| +|blobs|JSON array of Cloud Files blobs.|N/A|yes| +|maxCacheCapacityBytes|Maximum size of the cache space in bytes. 0 means disabling cache.|1073741824|no| +|maxCacheCapacityBytes|Maximum size of the cache space in bytes. 0 means disabling cache. Cached files are not removed until the ingestion task completes.|1073741824|no| +|maxFetchCapacityBytes|Maximum size of the fetch space in bytes. 0 means disabling prefetch. Prefetched files are removed immediately once they are read.|1073741824|no| +|fetchTimeout|Timeout for fetching a Cloud Files object.|60000|no| +|maxFetchRetry|Maximum retry for fetching a Cloud Files object.|3|no| + +Cloud Files Blobs: + +|property|description|default|required?| +|--------|-----------|-------|---------| +|container|Name of the Cloud Files container|N/A|yes| +|path|The path where data is located.|N/A|yes| diff --git a/development/extensions-contrib/distinctcount.md b/development/extensions-contrib/distinctcount.md new file mode 100644 index 0000000..c9ef545 --- /dev/null +++ b/development/extensions-contrib/distinctcount.md @@ -0,0 +1,99 @@ +--- +id: distinctcount +title: "DistinctCount Aggregator" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) the `druid-distinctcount` extension. + +Additionally, follow these steps: + +1. First, use a single dimension hash-based partition spec to partition data by a single dimension. For example visitor_id. This to make sure all rows with a particular value for that dimension will go into the same segment, or this might over count. +2. Second, use distinctCount to calculate the distinct count, make sure queryGranularity is divided exactly by segmentGranularity or else the result will be wrong. + +There are some limitations, when used with groupBy, the groupBy keys' numbers should not exceed maxIntermediateRows in every segment. If exceeded the result will be wrong. When used with topN, numValuesPerPass should not be too big. If too big the distinctCount will use a lot of memory and might cause the JVM to go our of memory. + +Example: + +## Timeseries query + +```json +{ + "queryType": "timeseries", + "dataSource": "sample_datasource", + "granularity": "day", + "aggregations": [ + { + "type": "distinctCount", + "name": "uv", + "fieldName": "visitor_id" + } + ], + "intervals": [ + "2016-03-01T00:00:00.000/2013-03-20T00:00:00.000" + ] +} +``` + +## TopN query + +```json +{ + "queryType": "topN", + "dataSource": "sample_datasource", + "dimension": "sample_dim", + "threshold": 5, + "metric": "uv", + "granularity": "all", + "aggregations": [ + { + "type": "distinctCount", + "name": "uv", + "fieldName": "visitor_id" + } + ], + "intervals": [ + "2016-03-06T00:00:00/2016-03-06T23:59:59" + ] +} +``` + +## GroupBy query + +```json +{ + "queryType": "groupBy", + "dataSource": "sample_datasource", + "dimensions": ["sample_dim"], + "granularity": "all", + "aggregations": [ + { + "type": "distinctCount", + "name": "uv", + "fieldName": "visitor_id" + } + ], + "intervals": [ + "2016-03-06T00:00:00/2016-03-06T23:59:59" + ] +} +``` diff --git a/development/extensions-contrib/gce-extensions.md b/development/extensions-contrib/gce-extensions.md new file mode 100644 index 0000000..687f6bb --- /dev/null +++ b/development/extensions-contrib/gce-extensions.md @@ -0,0 +1,103 @@ +--- +id: gce-extensions +title: "GCE Extensions" +--- + + + + +To use this Apache Druid (incubating) extension, make sure to [include](../../development/extensions.md#loading-extensions) `gce-extensions`. + +At the moment, this extension enables only Druid to autoscale instances in GCE. + +The extension manages the instances to be scaled up and down through the use of the [Managed Instance Groups](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances#resize_managed_group) +of GCE (MIG from now on). This choice has been made to ease the configuration of the machines and simplify their +management. + +For this reason, in order to use this extension, the user must have created +1. An instance template with the right machine type and image to bu used to run the MiddleManager +2. A MIG that has been configured to use the instance template created in the point above + +Moreover, in order to be able to rescale the machines in the MIG, the Overlord must run with a service account +guaranteeing the following two scopes from the [Compute Engine API](https://developers.google.com/identity/protocols/googlescopes#computev1) +- `https://www.googleapis.com/auth/cloud-platform` +- `https://www.googleapis.com/auth/compute` + +## Overlord Dynamic Configuration + +The Overlord can dynamically change worker behavior. + +The JSON object can be submitted to the Overlord via a POST request at: + +``` +http://:/druid/indexer/v1/worker +``` + +Optional Header Parameters for auditing the config change can also be specified. + +|Header Param Name| Description | Default | +|----------|-------------|---------| +|`X-Druid-Author`| author making the config change|""| +|`X-Druid-Comment`| comment describing the change being done|""| + +A sample worker config spec is shown below: + +```json +{ + "autoScaler": { + "envConfig" : { + "numInstances" : 1, + "projectId" : "super-project", + "zoneName" : "us-central-1", + "managedInstanceGroupName" : "druid-middlemanagers" + }, + "maxNumWorkers" : 4, + "minNumWorkers" : 2, + "type" : "gce" + } +} +``` + +The configuration of the autoscaler is quite simple and it is made of two levels only. + +The external level specifies the `type`—always `gce` in this case— and two numeric values, +the `maxNumWorkers` and `minNumWorkers` used to define the boundaries in between which the +number of instances must be at any time. + +The internal level is the `envConfig` and it is used to specify + +- The `numInstances` used to specify how many workers will be spawned at each +request to provision more workers. This is safe to be left to `1` +- The `projectId` used to specify the name of the project in which the MIG resides +- The `zoneName` used to identify in which zone of the worlds the MIG is +- The `managedInstanceGroupName` used to specify the MIG containing the instances created or +removed + +Please refer to the Overlord Dynamic Configuration section in the main [documentation](../../configuration/index.md) +for parameters other than the ones specified here, such as `selectStrategy` etc. + +## Known limitations + +- The module internally uses the [ListManagedInstances](https://cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/listManagedInstances) + call from the API and, while the documentation of the API states that the call can be paged through using the + `pageToken` argument, the responses to such call do not provide any `nextPageToken` to set such parameter. This means + that the extension can operate safely with a maximum of 500 MiddleManagers instances at any time (the maximum number + of instances to be returned for each call). + \ No newline at end of file diff --git a/development/extensions-contrib/graphite.md b/development/extensions-contrib/graphite.md new file mode 100644 index 0000000..5729315 --- /dev/null +++ b/development/extensions-contrib/graphite.md @@ -0,0 +1,117 @@ +--- +id: graphite +title: "Graphite Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `graphite-emitter` extension. + +## Introduction + +This extension emits druid metrics to a graphite carbon server. +Metrics can be sent by using [plaintext](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol) or [pickle](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol) protocol. +The pickle protocol is more efficient and supports sending batches of metrics (plaintext protocol send only one metric) in one request; batch size is configurable. + +## Configuration + +All the configuration parameters for graphite emitter are under `druid.emitter.graphite`. + +|property|description|required?|default| +|--------|-----------|---------|-------| +|`druid.emitter.graphite.hostname`|The hostname of the graphite server.|yes|none| +|`druid.emitter.graphite.port`|The port of the graphite server.|yes|none| +|`druid.emitter.graphite.batchSize`|Number of events to send as one batch (only for pickle protocol)|no|100| +|`druid.emitter.graphite.protocol`|Graphite protocol; available protocols: pickle, plaintext.|no|pickle| +|`druid.emitter.graphite.eventConverter`| Filter and converter of druid events to graphite event (please see next section).|yes|none| +|`druid.emitter.graphite.flushPeriod` | Queue flushing period in milliseconds. |no|1 minute| +|`druid.emitter.graphite.maxQueueSize`| Maximum size of the queue used to buffer events. |no|`MAX_INT`| +|`druid.emitter.graphite.alertEmitters`| List of emitters where alerts will be forwarded to. This is a JSON list of emitter names, e.g. `["logging", "http"]`|no| empty list (no forwarding)| +|`druid.emitter.graphite.requestLogEmitters`| List of emitters where request logs (i.e., query logging events sent to emitters when `druid.request.logging.type` is set to `emitter`) will be forwarded to. This is a JSON list of emitter names, e.g. `["logging", "http"]`|no| empty list (no forwarding)| +|`druid.emitter.graphite.emitWaitTime` | wait time in milliseconds to try to send the event otherwise emitter will throwing event. |no|0| +|`druid.emitter.graphite.waitForEventTime` | waiting time in milliseconds if necessary for an event to become available. |no|1000 (1 sec)| + +### Supported event types + +The graphite emitter only emits service metric events to graphite (See [Druid Metrics](../../operations/metrics.md) for a list of metrics). + +Alerts and request logs are not sent to graphite. These event types are not well represented in Graphite, which is more suited for timeseries views on numeric metrics, vs. storing non-numeric log events. + +Instead, alerts and request logs are optionally forwarded to other emitter implementations, specified by `druid.emitter.graphite.alertEmitters` and `druid.emitter.graphite.requestLogEmitters` respectively. + +### Druid to Graphite Event Converter + +Graphite Event Converter defines a mapping between druid metrics name plus dimensions to a Graphite metric path. +Graphite metric path is organized using the following schema: +`.[].[]..` +Properly naming the metrics is critical to avoid conflicts, confusing data and potentially wrong interpretation later on. + +Example `druid.historical.hist-host1_yahoo_com:8080.MyDataSourceName.GroupBy.query/time`: + + * `druid` -> namespace prefix + * `historical` -> service name + * `hist-host1.yahoo.com:8080` -> druid hostname + * `MyDataSourceName` -> dimension value + * `GroupBy` -> dimension value + * `query/time` -> metric name + +We have two different implementation of event converter: + +#### Send-All converter + +The first implementation called `all`, will send all the druid service metrics events. +The path will be in the form `.[].[]..` +User has control of `.[].[].` + +You can omit the hostname by setting `ignoreHostname=true` +`druid.SERVICE_NAME.dataSourceName.queryType.query/time` + +You can omit the service name by setting `ignoreServiceName=true` +`druid.HOSTNAME.dataSourceName.queryType.query/time` + +Elements in metric name by default are separated by "/", so graphite will create all metrics on one level. If you want to have metrics in the tree structure, you have to set `replaceSlashWithDot=true` +Original: `druid.HOSTNAME.dataSourceName.queryType.query/time` +Changed: `druid.HOSTNAME.dataSourceName.queryType.query.time` + + +```json + +druid.emitter.graphite.eventConverter={"type":"all", "namespacePrefix": "druid.test", "ignoreHostname":true, "ignoreServiceName":true} + +``` + +#### White-list based converter + +The second implementation called `whiteList`, will send only the white listed metrics and dimensions. +Same as for the `all` converter user has control of `.[].[].` +White-list based converter comes with the following default white list map located under resources in `./src/main/resources/defaultWhiteListMap.json` + +Although user can override the default white list map by supplying a property called `mapPath`. +This property is a String containing the path for the file containing **white list map JSON object**. +For example the following converter will read the map from the file `/pathPrefix/fileName.json`. + +```json + +druid.emitter.graphite.eventConverter={"type":"whiteList", "namespacePrefix": "druid.test", "ignoreHostname":true, "ignoreServiceName":true, "mapPath":"/pathPrefix/fileName.json"} + +``` + +**Druid emits a huge number of metrics we highly recommend to use the `whiteList` converter** diff --git a/development/extensions-contrib/influx.md b/development/extensions-contrib/influx.md new file mode 100644 index 0000000..e377bea --- /dev/null +++ b/development/extensions-contrib/influx.md @@ -0,0 +1,67 @@ +--- +id: influx +title: "InfluxDB Line Protocol Parser" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-influx-extensions`. + +This extension enables Druid to parse the [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v1.5/write_protocols/line_protocol_tutorial/), a popular text-based timeseries metric serialization format. + +## Line Protocol + +A typical line looks like this: + +```cpu,application=dbhost=prdb123,region=us-east-1 usage_idle=99.24,usage_user=0.55 1520722030000000000``` + +which contains four parts: + + - measurement: A string indicating the name of the measurement represented (e.g. cpu, network, web_requests) + - tags: zero or more key-value pairs (i.e. dimensions) + - measurements: one or more key-value pairs; values can be numeric, boolean, or string + - timestamp: nanoseconds since Unix epoch (the parser truncates it to milliseconds) + +The parser extracts these fields into a map, giving the measurement the key `measurement` and the timestamp the key `_ts`. The tag and measurement keys are copied verbatim, so users should take care to avoid name collisions. It is up to the ingestion spec to decide which fields should be treated as dimensions and which should be treated as metrics (typically tags correspond to dimensions and measurements correspond to metrics). + +The parser is configured like so: + +```json +"parser": { + "type": "string", + "parseSpec": { + "format": "influx", + "timestampSpec": { + "column": "__ts", + "format": "millis" + }, + "dimensionsSpec": { + "dimensionExclusions": [ + "__ts" + ] + }, + "whitelistMeasurements": [ + "cpu" + ] + } +``` + +The `whitelistMeasurements` field is an optional list of strings. If present, measurements that do not match one of the strings in the list will be ignored. diff --git a/development/extensions-contrib/influxdb-emitter.md b/development/extensions-contrib/influxdb-emitter.md new file mode 100644 index 0000000..3b1c84c --- /dev/null +++ b/development/extensions-contrib/influxdb-emitter.md @@ -0,0 +1,74 @@ +--- +id: influxdb-emitter +title: "InfluxDB Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-influxdb-emitter` extension. + +## Introduction + +This extension emits druid metrics to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) over HTTP. Currently this emitter only emits service metric events to InfluxDB (See [Druid metrics](../../operations/metrics.md) for a list of metrics). +When a metric event is fired it is added to a queue of events. After a configurable amount of time, the events on the queue are transformed to InfluxDB's line protocol +and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this point. The queue is also flushed as the emitter is shutdown. + +Note that authentication and authorization must be [enabled](https://docs.influxdata.com/influxdb/v1.7/administration/authentication_and_authorization/) on the InfluxDB server. + +## Configuration + +All the configuration parameters for the influxdb emitter are under `druid.emitter.influxdb`. + +|Property|Description|Required?|Default| +|--------|-----------|---------|-------| +|`druid.emitter.influxdb.hostname`|The hostname of the InfluxDB server.|Yes|N/A| +|`druid.emitter.influxdb.port`|The port of the InfluxDB server.|No|8086| +|`druid.emitter.influxdb.databaseName`|The name of the database in InfluxDB.|Yes|N/A| +|`druid.emitter.influxdb.maxQueueSize`|The size of the queue that holds events.|No|Integer.MAX_VALUE(=2^31-1)| +|`druid.emitter.influxdb.flushPeriod`|How often (in milliseconds) the events queue is parsed into Line Protocol and POSTed to InfluxDB.|No|60000| +|`druid.emitter.influxdb.flushDelay`|How long (in milliseconds) the scheduled method will wait until it first runs.|No|60000| +|`druid.emitter.influxdb.influxdbUserName`|The username for authenticating with the InfluxDB database.|Yes|N/A| +|`druid.emitter.influxdb.influxdbPassword`|The password of the database authorized user|Yes|N/A| +|`druid.emitter.influxdb.dimensionWhitelist`|A whitelist of metric dimensions to include as tags|No|`["dataSource","type","numMetrics","numDimensions","threshold","dimension","taskType","taskStatus","tier"]`| + +## InfluxDB Line Protocol + +An example of how this emitter parses a Druid metric event into InfluxDB's [line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_reference/) is given here: + +The syntax of the line protocol is : + +`[,=[,=]] =[,=] []` + +where timestamp is in nanoseconds since epoch. + +A typical service metric event as recorded by Druid's logging emitter is: `Event [{"feed":"metrics","timestamp":"2017-10-31T09:09:06.857Z","service":"druid/historical","host":"historical001:8083","version":"0.11.0-SNAPSHOT","metric":"query/cache/total/hits","value":34787256}]`. + +This event is parsed into line protocol according to these rules: + +* The measurement becomes druid_query since query is the first part of the metric. +* The tags are service=druid/historical, hostname=historical001, metric=druid_cache_total. (The metric tag is the middle part of the druid metric separated with _ and preceded by druid_. Another example would be if an event has metric=query/time then there is no middle part and hence no metric tag) +* The field is druid_hits since this is the last part of the metric. + +This gives the following String which can be POSTed to InfluxDB: `"druid_query,service=druid/historical,hostname=historical001,metric=druid_cache_total druid_hits=34787256 1509440946857000000"` + +The InfluxDB emitter has a white list of dimensions +which will be added as a tag to the line protocol string if the metric has a dimension from the white list. +The value of the dimension is sanitized such that every occurrence of a dot or whitespace is replaced with a `_` . diff --git a/development/extensions-contrib/kafka-emitter.md b/development/extensions-contrib/kafka-emitter.md new file mode 100644 index 0000000..15c975b --- /dev/null +++ b/development/extensions-contrib/kafka-emitter.md @@ -0,0 +1,54 @@ +--- +id: kafka-emitter +title: "Kafka Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `kafka-emitter` extension. + +## Introduction + +This extension emits Druid metrics to [Apache Kafka](https://kafka.apache.org) directly with JSON format.
+Currently, Kafka has not only their nice ecosystem but also consumer API readily available. +So, If you currently use Kafka, It's easy to integrate various tool or UI +to monitor the status of your Druid cluster with this extension. + +## Configuration + +All the configuration parameters for the Kafka emitter are under `druid.emitter.kafka`. + +|property|description|required?|default| +|--------|-----------|---------|-------| +|`druid.emitter.kafka.bootstrap.servers`|Comma-separated Kafka broker. (`[hostname:port],[hostname:port]...`)|yes|none| +|`druid.emitter.kafka.metric.topic`|Kafka topic name for emitter's target to emit service metric.|yes|none| +|`druid.emitter.kafka.alert.topic`|Kafka topic name for emitter's target to emit alert.|yes|none| +|`druid.emitter.kafka.producer.config`|JSON formatted configuration which user want to set additional properties to Kafka producer.|no|none| +|`druid.emitter.kafka.clusterName`|Optional value to specify name of your druid cluster. It can help make groups in your monitoring environment. |no|none| + +### Example + +``` +druid.emitter.kafka.bootstrap.servers=hostname1:9092,hostname2:9092 +druid.emitter.kafka.metric.topic=druid-metric +druid.emitter.kafka.alert.topic=druid-alert +druid.emitter.kafka.producer.config={"max.block.ms":10000} +``` diff --git a/development/extensions-contrib/materialized-view.md b/development/extensions-contrib/materialized-view.md new file mode 100644 index 0000000..0b2eb3d --- /dev/null +++ b/development/extensions-contrib/materialized-view.md @@ -0,0 +1,136 @@ +--- +id: materialized-view +title: "Materialized View" +--- + + + + +To use this Apache Druid feature, make sure to only load `materialized-view-selection` on Broker and load `materialized-view-maintenance` on Overlord. In addition, this feature currently requires a Hadoop cluster. + +This feature enables Druid to greatly improve the query performance, especially when the query dataSource has a very large number of dimensions but the query only required several dimensions. This feature includes two parts. One is `materialized-view-maintenance`, and the other is `materialized-view-selection`. + +## Materialized-view-maintenance +In materialized-view-maintenance, dataSources user ingested are called "base-dataSource". For each base-dataSource, we can submit `derivativeDataSource` supervisors to create and maintain other dataSources which we called "derived-dataSource". The dimensions and metrics of derived-dataSources are the subset of base-dataSource's. +The `derivativeDataSource` supervisor is used to keep the timeline of derived-dataSource consistent with base-dataSource. Each `derivativeDataSource` supervisor is responsible for one derived-dataSource. + +A sample derivativeDataSource supervisor spec is shown below: + +```json + { + "type": "derivativeDataSource", + "baseDataSource": "wikiticker", + "dimensionsSpec": { + "dimensions": [ + "isUnpatrolled", + "metroCode", + "namespace", + "page", + "regionIsoCode", + "regionName", + "user" + ] + }, + "metricsSpec": [ + { + "name": "count", + "type": "count" + }, + { + "name": "added", + "type": "longSum", + "fieldName": "added" + } + ], + "tuningConfig": { + "type": "hadoop" + } + } +``` + +**Supervisor Configuration** + +|Field|Description|Required| +|--------|-----------|---------| +|Type |The supervisor type. This should always be `derivativeDataSource`.|yes| +|baseDataSource |The name of base dataSource. This dataSource data should be already stored inside Druid, and the dataSource will be used as input data.|yes| +|dimensionsSpec |Specifies the dimensions of the data. These dimensions must be the subset of baseDataSource's dimensions.|yes| +|metricsSpec |A list of aggregators. These metrics must be the subset of baseDataSource's metrics. See [aggregations](../../querying/aggregations.md).|yes| +|tuningConfig |TuningConfig must be HadoopTuningConfig. See [Hadoop tuning config](../../ingestion/hadoop.html#tuningconfig).|yes| +|dataSource |The name of this derived dataSource. |no(default=baseDataSource-hashCode of supervisor)| +|hadoopDependencyCoordinates |A JSON array of Hadoop dependency coordinates that Druid will use, this property will override the default Hadoop coordinates. Once specified, Druid will look for those Hadoop dependencies from the location specified by druid.extensions.hadoopDependenciesDir |no| +|classpathPrefix |Classpath that will be prepended for the Peon process. |no| +|context |See below. |no| + +**Context** + +|Field|Description|Required| +|--------|-----------|---------| +|maxTaskCount |The max number of tasks the supervisor can submit simultaneously. |no(default=1)| + +## Materialized-view-selection + +In materialized-view-selection, we implement a new query type `view`. When we request a view query, Druid will try its best to optimize the query based on query dataSource and intervals. + +A sample view query spec is shown below: + +```json + { + "queryType": "view", + "query": { + "queryType": "groupBy", + "dataSource": "wikiticker", + "granularity": "all", + "dimensions": [ + "user" + ], + "limitSpec": { + "type": "default", + "limit": 1, + "columns": [ + { + "dimension": "added", + "direction": "descending", + "dimensionOrder": "numeric" + } + ] + }, + "aggregations": [ + { + "type": "longSum", + "name": "added", + "fieldName": "added" + } + ], + "intervals": [ + "2015-09-12/2015-09-13" + ] + } + } +``` + +There are 2 parts in a view query: + +|Field|Description|Required| +|--------|-----------|---------| +|queryType |The query type. This should always be view |yes| +|query |The real query of this `view` query. The real query must be [groupBy](../../querying/groupbyquery.md), [topN](../../querying/topnquery.md), or [timeseries](../../querying/timeseriesquery.md) type.|yes| + +**Note that Materialized View is currently designated as experimental. Please make sure the time of all processes are the same and increase monotonically. Otherwise, some unexpected errors may happen on query results.** diff --git a/development/extensions-contrib/momentsketch-quantiles.md b/development/extensions-contrib/momentsketch-quantiles.md new file mode 100644 index 0000000..3f3bff5 --- /dev/null +++ b/development/extensions-contrib/momentsketch-quantiles.md @@ -0,0 +1,125 @@ +--- +id: momentsketch-quantiles +title: "Moment Sketches for Approximate Quantiles module" +--- + + + + +This module provides aggregators for approximate quantile queries using the [momentsketch](https://github.com/stanford-futuredata/momentsketch) library. +The momentsketch provides coarse quantile estimates with less space and aggregation time overheads than traditional sketches, approaching the performance of counts and sums by reconstructing distributions from computed statistics. + +To use this Apache Druid extension, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-momentsketch"] +``` + +### Aggregator + +The result of the aggregation is a momentsketch that is the union of all sketches either built from raw data or read from the segments. + +The `momentSketch` aggregator operates over raw data while the `momentSketchMerge` aggregator should be used when aggregating precomputed sketches. + +```json +{ + "type" : , + "name" : , + "fieldName" : , + "k" : , + "compress" : + } +``` + +|property|description|required?| +|--------|-----------|---------| +|type|Type of aggregator desired. Either "momentSketch" or "momentSketchMerge" |yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the input field (can contain sketches or raw numeric values).|yes| +|k|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Usable range is generally [3,15] |no, defaults to 13.| +|compress|Flag for whether the aggregator compresses numeric values using arcsinh. Can improve robustness to skewed and long-tailed distributions, but reduces accuracy slightly on more uniform distributions.| no, defaults to true + +### Post Aggregators + +Users can query for a set of quantiles using the `momentSketchSolveQuantiles` post-aggregator on the sketches created by the `momentSketch` or `momentSketchMerge` aggregators. + +```json +{ + "type" : "momentSketchSolveQuantiles", + "name" : , + "field" : , + "fractions" : +} +``` + +Users can also query for the min/max of a distribution: + +```json +{ + "type" : "momentSketchMin" | "momentSketchMax", + "name" : , + "field" : , +} +``` + +### Example +As an example of a query with sketches pre-aggregated at ingestion time, one could set up the following aggregator at ingest: + +```json +{ + "type": "momentSketch", + "name": "sketch", + "fieldName": "value", + "k": 10, + "compress": true, +} +``` + +and make queries using the following aggregator + post-aggregator: + +```json +{ + "aggregations": [{ + "type": "momentSketchMerge", + "name": "sketch", + "fieldName": "sketch", + "k": 10, + "compress": true + }], + "postAggregations": [ + { + "type": "momentSketchSolveQuantiles", + "name": "quantiles", + "fractions": [0.1, 0.5, 0.9], + "field": { + "type": "fieldAccess", + "fieldName": "sketch" + } + }, + { + "type": "momentSketchMin", + "name": "min", + "field": { + "type": "fieldAccess", + "fieldName": "sketch" + } + }] +} +``` \ No newline at end of file diff --git a/development/extensions-contrib/moving-average-query.md b/development/extensions-contrib/moving-average-query.md new file mode 100644 index 0000000..3e9bb53 --- /dev/null +++ b/development/extensions-contrib/moving-average-query.md @@ -0,0 +1,349 @@ +--- +id: moving-average-query +title: "Moving Average Query" +--- + + + + +## Overview +**Moving Average Query** is an extension which provides support for [Moving Average](https://en.wikipedia.org/wiki/Moving_average) and other Aggregate [Window Functions](https://en.wikibooks.org/wiki/Structured_Query_Language/Window_functions) in Druid queries. + +These Aggregate Window Functions consume standard Druid Aggregators and outputs additional windowed aggregates called [Averagers](#averagers). + +#### High level algorithm + +Moving Average encapsulates the [groupBy query](../../querying/groupbyquery.md) (Or [timeseries](../../querying/timeseriesquery.md) in case of no dimensions) in order to rely on the maturity of these query types. + +It runs the query in two main phases: + +1. Runs an inner [groupBy](../../querying/groupbyquery.html) or [timeseries](../../querying/timeseriesquery.html) query to compute Aggregators (i.e. daily count of events). +2. Passes over aggregated results in Broker, in order to compute Averagers (i.e. moving 7 day average of the daily count). + +#### Main enhancements provided by this extension: +1. Functionality: Extending druid query functionality (i.e. initial introduction of Window Functions). +2. Performance: Improving performance of such moving aggregations by eliminating multiple segment scans. + +#### Further reading +[Moving Average](https://en.wikipedia.org/wiki/Moving_average) + +[Window Functions](https://en.wikibooks.org/wiki/Structured_Query_Language/Window_functions) + +[Analytic Functions](https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts) + + +## Operations +To use this extension, make sure to [load](../../development/extensions.md#loading-extensions) `druid-moving-average-query` only to the Broker. + +## Configuration +There are currently no configuration properties specific to Moving Average. + +## Limitations +* movingAverage is missing support for the following groupBy properties: `subtotalsSpec`, `virtualColumns`. +* movingAverage is missing support for the following timeseries properties: `descending`. +* movingAverage is missing support for [SQL-compatible null handling](https://github.com/apache/druid/issues/4349) (So setting druid.generic.useDefaultValueForNull in configuration will give an error). + +##Query spec: +* Most properties in the query spec derived from [groupBy query](../../querying/groupbyquery.md) / [timeseries](../../querying/timeseriesquery.md), see documentation for these query types. + +|property|description|required?| +|--------|-----------|---------| +|queryType|This String should always be "movingAverage"; this is the first thing Druid looks at to figure out how to interpret the query.|yes| +|dataSource|A String or Object defining the data source to query, very similar to a table in a relational database. See [DataSource](../../querying/datasource.md) for more information.|yes| +|dimensions|A JSON list of [DimensionSpec](../../querying/dimensionspecs.md) (Notice that property is optional)|no| +|limitSpec|See [LimitSpec](../../querying/limitspec.md)|no| +|having|See [Having](../../querying/having.md)|no| +|granularity|A period granularity; See [Period Granularities](../../querying/granularities.html#period-granularities)|yes| +|filter|See [Filters](../../querying/filters.md)|no| +|aggregations|Aggregations forms the input to Averagers; See [Aggregations](../../querying/aggregations.md)|yes| +|postAggregations|Supports only aggregations as input; See [Post Aggregations](../../querying/post-aggregations.md)|no| +|intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes| +|context|An additional JSON Object which can be used to specify certain flags.|no| +|averagers|Defines the moving average function; See [Averagers](#averagers)|yes| +|postAveragers|Support input of both averagers and aggregations; Syntax is identical to postAggregations (See [Post Aggregations](../../querying/post-aggregations.md))|no| + +## Averagers + +Averagers are used to define the Moving-Average function. Averagers are not limited to an average - they can also provide other types of window functions such as MAX()/MIN(). + +### Properties + +These are properties which are common to all Averagers: + +|property|description|required?| +|--------|-----------|---------| +|type|Averager type; See [Averager types](#averager-types)|yes| +|name|Averager name|yes| +|fieldName|Input name (An aggregation name)|yes| +|buckets|Number of lookback buckets (time periods), including current one. Must be >0|yes| +|cycleSize|Cycle size; Used to calculate day-of-week option; See [Cycle size (Day of Week)](#cycle-size-day-of-week)|no, defaults to 1| + + +### Averager types: + +* [Standard averagers](#standard-averagers): + * doubleMean + * doubleMeanNoNulls + * doubleSum + * doubleMax + * doubleMin + * longMean + * longMeanNoNulls + * longSum + * longMax + * longMin + +#### Standard averagers + +These averagers offer four functions: + +* Mean (Average) +* MeanNoNulls (Ignores empty buckets). +* Sum +* Max +* Min + +**Ignoring nulls**: +Using a MeanNoNulls averager is useful when the interval starts at the dataset beginning time. +In that case, the first records will ignore missing buckets and average won't be artificially low. +However, this also means that empty days in a sparse dataset will also be ignored. + +Example of usage: + +```json +{ "type" : "doubleMean", "name" : , "fieldName": } +``` + +### Cycle size (Day of Week) +This optional parameter is used to calculate over a single bucket within each cycle instead of all buckets. +A prime example would be weekly buckets, resulting in a Day of Week calculation. (Other examples: Month of year, Hour of day). + +I.e. when using these parameters: + +* *granularity*: period=P1D (daily) +* *buckets*: 28 +* *cycleSize*: 7 + +Within each output record, the averager will compute the result over the following buckets: current (#0), #7, #14, #21. +Whereas without specifying cycleSize it would have computed over all 28 buckets. + +## Examples + +All examples are based on the Wikipedia dataset provided in the Druid [tutorials](../../tutorials/index.md). + +### Basic example + +Calculating a 7-buckets moving average for Wikipedia edit deltas. + +Query syntax: + +```json +{ + "queryType": "movingAverage", + "dataSource": "wikipedia", + "granularity": { + "type": "period", + "period": "PT30M" + }, + "intervals": [ + "2015-09-12T00:00:00Z/2015-09-13T00:00:00Z" + ], + "aggregations": [ + { + "name": "delta30Min", + "fieldName": "delta", + "type": "longSum" + } + ], + "averagers": [ + { + "name": "trailing30MinChanges", + "fieldName": "delta30Min", + "type": "longMean", + "buckets": 7 + } + ] +} +``` + +Result: + +```json +[ { + "version" : "v1", + "timestamp" : "2015-09-12T00:30:00.000Z", + "event" : { + "delta30Min" : 30490, + "trailing30MinChanges" : 4355.714285714285 + } + }, { + "version" : "v1", + "timestamp" : "2015-09-12T01:00:00.000Z", + "event" : { + "delta30Min" : 96526, + "trailing30MinChanges" : 18145.14285714286 + } + }, { +... +... +... +}, { + "version" : "v1", + "timestamp" : "2015-09-12T23:00:00.000Z", + "event" : { + "delta30Min" : 119100, + "trailing30MinChanges" : 198697.2857142857 + } +}, { + "version" : "v1", + "timestamp" : "2015-09-12T23:30:00.000Z", + "event" : { + "delta30Min" : 177882, + "trailing30MinChanges" : 193890.0 + } +} +``` + +### Post averager example + +Calculating a 7-buckets moving average for Wikipedia edit deltas, plus a ratio between the current period and the moving average. + +Query syntax: + +```json +{ + "queryType": "movingAverage", + "dataSource": "wikipedia", + "granularity": { + "type": "period", + "period": "PT30M" + }, + "intervals": [ + "2015-09-12T22:00:00Z/2015-09-13T00:00:00Z" + ], + "aggregations": [ + { + "name": "delta30Min", + "fieldName": "delta", + "type": "longSum" + } + ], + "averagers": [ + { + "name": "trailing30MinChanges", + "fieldName": "delta30Min", + "type": "longMean", + "buckets": 7 + } + ], + "postAveragers" : [ + { + "name": "ratioTrailing30MinChanges", + "type": "arithmetic", + "fn": "/", + "fields": [ + { + "type": "fieldAccess", + "fieldName": "delta30Min" + }, + { + "type": "fieldAccess", + "fieldName": "trailing30MinChanges" + } + ] + } + ] +} +``` + +Result: + +```json +[ { + "version" : "v1", + "timestamp" : "2015-09-12T22:00:00.000Z", + "event" : { + "delta30Min" : 144269, + "trailing30MinChanges" : 204088.14285714287, + "ratioTrailing30MinChanges" : 0.7068955500319539 + } +}, { + "version" : "v1", + "timestamp" : "2015-09-12T22:30:00.000Z", + "event" : { + "delta30Min" : 242860, + "trailing30MinChanges" : 214031.57142857142, + "ratioTrailing30MinChanges" : 1.134692411867141 + } +}, { + "version" : "v1", + "timestamp" : "2015-09-12T23:00:00.000Z", + "event" : { + "delta30Min" : 119100, + "trailing30MinChanges" : 198697.2857142857, + "ratioTrailing30MinChanges" : 0.5994042624782422 + } +}, { + "version" : "v1", + "timestamp" : "2015-09-12T23:30:00.000Z", + "event" : { + "delta30Min" : 177882, + "trailing30MinChanges" : 193890.0, + "ratioTrailing30MinChanges" : 0.9174377224199288 + } +} ] +``` + + +### Cycle size example + +Calculating an average of every first 10-minutes of the last 3 hours: + +Query syntax: + +```json +{ + "queryType": "movingAverage", + "dataSource": "wikipedia", + "granularity": { + "type": "period", + "period": "PT10M" + }, + "intervals": [ + "2015-09-12T00:00:00Z/2015-09-13T00:00:00Z" + ], + "aggregations": [ + { + "name": "delta10Min", + "fieldName": "delta", + "type": "doubleSum" + } + ], + "averagers": [ + { + "name": "trailing10MinPerHourChanges", + "fieldName": "delta10Min", + "type": "doubleMeanNoNulls", + "buckets": 18, + "cycleSize": 6 + } + ] +} +``` diff --git a/development/extensions-contrib/opentsdb-emitter.md b/development/extensions-contrib/opentsdb-emitter.md new file mode 100644 index 0000000..0a3c688 --- /dev/null +++ b/development/extensions-contrib/opentsdb-emitter.md @@ -0,0 +1,62 @@ +--- +id: opentsdb-emitter +title: "OpenTSDB Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `opentsdb-emitter` extension. + +## Introduction + +This extension emits druid metrics to [OpenTSDB](https://github.com/OpenTSDB/opentsdb) over HTTP (Using `Jersey client`). And this emitter only emits service metric events to OpenTSDB (See [Druid metrics](../../operations/metrics.md) for a list of metrics). + +## Configuration + +All the configuration parameters for the OpenTSDB emitter are under `druid.emitter.opentsdb`. + +|property|description|required?|default| +|--------|-----------|---------|-------| +|`druid.emitter.opentsdb.host`|The host of the OpenTSDB server.|yes|none| +|`druid.emitter.opentsdb.port`|The port of the OpenTSDB server.|yes|none| +|`druid.emitter.opentsdb.connectionTimeout`|`Jersey client` connection timeout(in milliseconds).|no|2000| +|`druid.emitter.opentsdb.readTimeout`|`Jersey client` read timeout(in milliseconds).|no|2000| +|`druid.emitter.opentsdb.flushThreshold`|Queue flushing threshold.(Events will be sent as one batch)|no|100| +|`druid.emitter.opentsdb.maxQueueSize`|Maximum size of the queue used to buffer events.|no|1000| +|`druid.emitter.opentsdb.consumeDelay`|Queue consuming delay(in milliseconds). Actually, we use `ScheduledExecutorService` to schedule consuming events, so this `consumeDelay` means the delay between the termination of one execution and the commencement of the next. If your druid processes produce metric events fast, then you should decrease this `consumeDelay` or increase the `maxQueueSize`.|no|10000| +|`druid.emitter.opentsdb.metricMapPath`|JSON file defining the desired metrics and dimensions for every Druid metric|no|./src/main/resources/defaultMetrics.json| +|`druid.emitter.opentsdb.namespacePrefix`|Optional (string) prefix for metric names, for example the default metric name `query.count` with a namespacePrefix set to `druid` would be emitted as `druid.query.count` |no|null| + +### Druid to OpenTSDB Event Converter + +The OpenTSDB emitter will send only the desired metrics and dimensions which is defined in a JSON file. +If the user does not specify their own JSON file, a default file is used. All metrics are expected to be configured in the JSON file. Metrics which are not configured will be logged. +Desired metrics and dimensions is organized using the following schema:` : [ ]`
+e.g. + +```json +"query/time": [ + "dataSource", + "type" +] +``` + +For most use-cases, the default configuration is sufficient. diff --git a/development/extensions-contrib/redis-cache.md b/development/extensions-contrib/redis-cache.md new file mode 100644 index 0000000..e8e39e3 --- /dev/null +++ b/development/extensions-contrib/redis-cache.md @@ -0,0 +1,58 @@ +--- +id: redis-cache +title: "Druid Redis Cache" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-redis-cache` extension. + +A cache implementation for Druid based on [Redis](https://github.com/antirez/redis). + +Below are the configuration options known to this module. + +Note that just adding these properties does not enable the cache. You still need to add the `druid..cache.useCache` and `druid..cache.populateCache` properties for the processes you want to enable the cache on as described in the [cache configuration docs](../../configuration/index.html#cache-configuration). + +A possible configuration would be to keep the properties below in your `common.runtime.properties` file (present on all processes) and then add `druid..cache.useCache` and `druid..cache.populateCache` in the `runtime.properties` file of the process types you want to enable caching on. + +## Configuration + +|`common.runtime.properties`|Description|Default|Required| +|--------------------|-----------|-------|--------| +|`druid.cache.host`|Redis server host|None|yes| +|`druid.cache.port`|Redis server port|None|yes| +|`druid.cache.expiration`|Expiration(in milliseconds) for cache entries|24 * 3600 * 1000|no| +|`druid.cache.timeout`|Timeout(in milliseconds) for get cache entries from Redis|2000|no| +|`druid.cache.maxTotalConnections`|Max total connections to Redis|8|no| +|`druid.cache.maxIdleConnections`|Max idle connections to Redis|8|no| +|`druid.cache.minIdleConnections`|Min idle connections to Redis|0|no| + +## Enabling + +To enable the redis cache, include this module on the loadList and set `druid.cache.type` to `redis` in your properties. + +## Metrics + +In addition to the normal cache metrics, the redis cache implementation also reports the following in both `total` and `delta` + +|Metric|Description|Normal value| +|------|-----------|------------| +|`query/cache/redis/*/requests`|Count of requests to redis cache|whatever request to redis will increase request count by 1| diff --git a/development/extensions-contrib/sqlserver.md b/development/extensions-contrib/sqlserver.md new file mode 100644 index 0000000..103897f --- /dev/null +++ b/development/extensions-contrib/sqlserver.md @@ -0,0 +1,56 @@ +--- +id: sqlserver +title: "Microsoft SQLServer" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `sqlserver-metadata-storage` as an extension. + +## Setting up SQLServer + +1. Install Microsoft SQLServer + +2. Create a druid database and user + + Create the druid user + - Microsoft SQL Server Management Studio - Security - Logins - New Login... + - Create a druid user, enter `diurd` when prompted for the password. + + Create a druid database owned by the user we just created + - Databases - New Database + - Database Name: druid, Owner: druid + +3. Add the Microsoft JDBC library to the Druid classpath + - To ensure the com.microsoft.sqlserver.jdbc.SQLServerDriver class is loaded you will have to add the appropriate Microsoft JDBC library (sqljdbc*.jar) to the Druid classpath. + - For instance, if all jar files in your "druid/lib" directory are automatically added to your Druid classpath, then manually download the Microsoft JDBC drivers from ( https://www.microsoft.com/en-ca/download/details.aspx?id=11774) and drop it into my druid/lib directory. + +4. Configure your Druid metadata storage extension: + + Add the following parameters to your Druid configuration, replacing `` + with the location (host name and port) of the database. + + ```properties + druid.metadata.storage.type=sqlserver + druid.metadata.storage.connector.connectURI=jdbc:sqlserver://;databaseName=druid + druid.metadata.storage.connector.user=druid + druid.metadata.storage.connector.password=diurd + ``` diff --git a/development/extensions-contrib/statsd.md b/development/extensions-contrib/statsd.md new file mode 100644 index 0000000..7a6dd6b --- /dev/null +++ b/development/extensions-contrib/statsd.md @@ -0,0 +1,71 @@ +--- +id: statsd +title: "StatsD Emitter" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `statsd-emitter` extension. + +## Introduction + +This extension emits druid metrics to a StatsD server. +(https://github.com/etsy/statsd) +(https://github.com/armon/statsite) + +## Configuration + +All the configuration parameters for the StatsD emitter are under `druid.emitter.statsd`. + +|property|description|required?|default| +|--------|-----------|---------|-------| +|`druid.emitter.statsd.hostname`|The hostname of the StatsD server.|yes|none| +|`druid.emitter.statsd.port`|The port of the StatsD server.|yes|none| +|`druid.emitter.statsd.prefix`|Optional metric name prefix.|no|""| +|`druid.emitter.statsd.separator`|Metric name separator|no|.| +|`druid.emitter.statsd.includeHost`|Flag to include the hostname as part of the metric name.|no|false| +|`druid.emitter.statsd.dimensionMapPath`|JSON file defining the StatsD type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.statsd.blankHolder`|The blank character replacement as StatsD does not support path with blank character|no|"-"| +|`druid.emitter.statsd.dogstatsd`|Flag to enable [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/) support. Causes dimensions to be included as tags, not as a part of the metric name. `convertRange` fields will be ignored.|no|false| +|`druid.emitter.statsd.dogstatsdConstantTags`|If `druid.emitter.statsd.dogstatsd` is true, the tags in the JSON list of strings will be sent with every event.|no|[]| +|`druid.emitter.statsd.dogstatsdServiceAsTag`|If `druid.emitter.statsd.dogstatsd` and `druid.emitter.statsd.dogstatsdServiceAsTag` are true, druid service (e.g. `druid/broker`, `druid/coordinator`, etc) is reported as a tag (e.g. `druid_service:druid/broker`) instead of being included in metric name (e.g. `druid.broker.query.time`) and `druid` is used as metric prefix (e.g. `druid.query.time`).|no|false| +|`druid.emitter.statsd.dogstatsdEvents`|If `druid.emitter.statsd.dogstatsd` and `druid.emitter.statsd.dogstatsdEvents` are true, [Alert events](../../operations/alerts.html) are reported to DogStatsD.|no|false| + +### Druid to StatsD Event Converter + +Each metric sent to StatsD must specify a type, one of `[timer, counter, guage]`. StatsD Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +StatsD expects that metric values be integers. Druid emits some metrics with values between the range 0 and 1. To accommodate these metrics they are converted +into the range 0 to 100. This conversion can be enabled by setting the optional "convertRange" field true in the JSON mapping file. +If the user does not specify their own JSON file, a default mapping is used. All +metrics are expected to be mapped. Metrics which are not mapped will log an error. +StatsD metric path is organized using the following schema: +` : { "dimensions" : , "type" : , "convertRange" : true/false}` +e.g. +`query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer"}` + +For metrics which are emitted from multiple services with different dimensions, the metric name is prefixed with +the service name. +e.g. +`"coordinator-segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge" }, + "historical-segment/count" : { "dimensions" : ["dataSource", "tier", "priority"], "type" : "gauge" }` + +For most use-cases, the default mapping is sufficient. diff --git a/development/extensions-contrib/tdigestsketch-quantiles.md b/development/extensions-contrib/tdigestsketch-quantiles.md new file mode 100644 index 0000000..705bbc2 --- /dev/null +++ b/development/extensions-contrib/tdigestsketch-quantiles.md @@ -0,0 +1,151 @@ +--- +id: tdigestsketch-quantiles +title: "T-Digest Quantiles Sketch module" +--- + + + + +This module provides Apache Druid approximate sketch aggregators based on T-Digest. +T-Digest (https://github.com/tdunning/t-digest) is a popular data structure for accurate on-line accumulation of +rank-based statistics such as quantiles and trimmed means. +The data structure is also designed for parallel programming use cases like distributed aggregations or map reduce jobs by making combining two intermediate t-digests easy and efficient. + +The tDigestSketch aggregator is capable of generating sketches from raw numeric values as well as +aggregating/combining pre-generated T-Digest sketches generated using the tDigestSketch aggregator itself. +While one can generate sketches on the fly during the query time itself, it generally is more performant +to generate sketches during ingestion time itself and then combining them during query time. +The module also provides a postAggregator, quantilesFromTDigestSketch, that can be used to compute approximate +quantiles from T-Digest sketches generated by the tDigestSketch aggregator. + +To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-tdigestsketch"] +``` + +### Aggregator + +The result of the aggregation is a T-Digest sketch that is built ingesting numeric values from the raw data or from +combining pre-generated T-Digest sketches. + +```json +{ + "type" : "tDigestSketch", + "name" : , + "fieldName" : , + "compression": + } +``` + +Example: + +```json +{ + "type": "tDigestSketch", + "name": "sketch", + "fieldName": "session_duration", + "compression": 200 +} +``` + +```json +{ + "type": "tDigestSketch", + "name": "combined_sketch", + "fieldName": , + "compression": 200 +} +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "tDigestSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the input field containing raw numeric values or pre-generated T-Digest sketches.|yes| +|compression|Parameter that determines the accuracy and size of the sketch. Higher compression means higher accuracy but more space to store sketches.|no, defaults to 100| + + +### Post Aggregators + +#### Quantiles + +This returns an array of quantiles corresponding to a given array of fractions. + +```json +{ + "type" : "quantilesFromTDigestSketch", + "name": , + "field" : , + "fractions" : +} +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "quantilesFromTDigestSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|field|A field reference pointing to the field aggregated/combined T-Digest sketch.|yes| +|fractions|Non-empty array of fractions between 0 and 1|yes| + +Example: + +```json +{ + "queryType": "groupBy", + "dataSource": "test_datasource", + "granularity": "ALL", + "dimensions": [], + "aggregations": [{ + "type": "tDigestSketch", + "name": "merged_sketch", + "fieldName": "ingested_sketch", + "compression": 200 + }], + "postAggregations": [{ + "type": "quantilesFromTDigestSketch", + "name": "quantiles", + "fractions": [0, 0.5, 1], + "field": { + "type": "fieldAccess", + "fieldName": "merged_sketch" + } + }], + "intervals": ["2016-01-01T00:00:00.000Z/2016-01-31T00:00:00.000Z"] +} +``` + +Similar to quantilesFromTDigestSketch except it takes in a single fraction for computing quantile. + +```json +{ + "type" : "quantileFromTDigestSketch", + "name": , + "field" : , + "fraction" : +} +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "quantileFromTDigestSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|field|A field reference pointing to the field aggregated/combined T-Digest sketch.|yes| +|fraction|Decimal value between 0 and 1|yes| diff --git a/development/extensions-contrib/thrift.md b/development/extensions-contrib/thrift.md new file mode 100644 index 0000000..dd3f3db --- /dev/null +++ b/development/extensions-contrib/thrift.md @@ -0,0 +1,87 @@ +--- +id: thrift +title: "Thrift" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-thrift-extensions`. + +This extension enables Druid to ingest thrift compact data online (`ByteBuffer`) and offline (SequenceFile of type `` or LzoThriftBlock File). + +You may want to use another version of thrift, change the dependency in pom and compile yourself. + +## LZO Support + +If you plan to read LZO-compressed Thrift files, you will need to download version 0.4.19 of the [hadoop-lzo JAR](https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo/0.4.19) and place it in your `extensions/druid-thrift-extensions` directory. + +## Thrift Parser + + +| Field | Type | Description | Required | +| ----------- | ----------- | ---------------------------------------- | -------- | +| type | String | This should say `thrift` | yes | +| parseSpec | JSON Object | Specifies the timestamp and dimensions of the data. Should be a JSON parseSpec. | yes | +| thriftJar | String | path of thrift jar, if not provided, it will try to find the thrift class in classpath. Thrift jar in batch ingestion should be uploaded to HDFS first and configure `jobProperties` with `"tmpjars":"/path/to/your/thrift.jar"` | no | +| thriftClass | String | classname of thrift | yes | + +- Batch Ingestion example - `inputFormat` and `tmpjars` should be set. + +This is for batch ingestion using the HadoopDruidIndexer. The inputFormat of inputSpec in ioConfig could be one of `"org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat"` and `com.twitter.elephantbird.mapreduce.input.LzoThriftBlockInputFormat`. Be careful, when `LzoThriftBlockInputFormat` is used, thrift class must be provided twice. + +```json +{ + "type": "index_hadoop", + "spec": { + "dataSchema": { + "dataSource": "book", + "parser": { + "type": "thrift", + "jarPath": "book.jar", + "thriftClass": "org.apache.druid.data.input.thrift.Book", + "protocol": "compact", + "parseSpec": { + "format": "json", + ... + } + }, + "metricsSpec": [], + "granularitySpec": {} + }, + "ioConfig": { + "type": "hadoop", + "inputSpec": { + "type": "static", + "inputFormat": "org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat", + // "inputFormat": "com.twitter.elephantbird.mapreduce.input.LzoThriftBlockInputFormat", + "paths": "/user/to/some/book.seq" + } + }, + "tuningConfig": { + "type": "hadoop", + "jobProperties": { + "tmpjars":"/user/h_user_profile/du00/druid/test/book.jar", + // "elephantbird.class.for.MultiInputFormat" : "${YOUR_THRIFT_CLASS_NAME}" + } + } + } +} +``` diff --git a/development/extensions-contrib/time-min-max.md b/development/extensions-contrib/time-min-max.md new file mode 100644 index 0000000..7082233 --- /dev/null +++ b/development/extensions-contrib/time-min-max.md @@ -0,0 +1,104 @@ +--- +id: time-min-max +title: "Timestamp Min/Max aggregators" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-time-min-max`. + +These aggregators enable more precise calculation of min and max time of given events than `__time` column whose granularity is sparse, the same as query granularity. +To use this feature, a "timeMin" or "timeMax" aggregator must be included at indexing time. +They can apply to any columns that can be converted to timestamp, which include Long, DateTime, Timestamp, and String types. + +For example, when a data set consists of timestamp, dimension, and metric value like followings. + +``` +2015-07-28T01:00:00.000Z A 1 +2015-07-28T02:00:00.000Z A 1 +2015-07-28T03:00:00.000Z A 1 +2015-07-28T04:00:00.000Z B 1 +2015-07-28T05:00:00.000Z A 1 +2015-07-28T06:00:00.000Z B 1 +2015-07-29T01:00:00.000Z C 1 +2015-07-29T02:00:00.000Z C 1 +2015-07-29T03:00:00.000Z A 1 +2015-07-29T04:00:00.000Z A 1 +``` + +At ingestion time, timeMin and timeMax aggregator can be included as other aggregators. + +```json +{ + "type": "timeMin", + "name": "tmin", + "fieldName": "" +} +``` + +```json +{ + "type": "timeMax", + "name": "tmax", + "fieldName": "" +} +``` + +`name` is output name of aggregator and can be any string. `fieldName` is typically column specified in timestamp spec but can be any column that can be converted to timestamp. + +To query for results, the same aggregators "timeMin" and "timeMax" is used. + +```json +{ + "queryType": "groupBy", + "dataSource": "timeMinMax", + "granularity": "DAY", + "dimensions": ["product"], + "aggregations": [ + { + "type": "count", + "name": "count" + }, + { + "type": "timeMin", + "name": "", + "fieldName": "tmin" + }, + { + "type": "timeMax", + "name": "", + "fieldName": "tmax" + } + ], + "intervals": [ + "2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z" + ] +} +``` + +Then, result has min and max of timestamp, which is finer than query granularity. + +``` +2015-07-28T00:00:00.000Z A 4 2015-07-28T01:00:00.000Z 2015-07-28T05:00:00.000Z +2015-07-28T00:00:00.000Z B 2 2015-07-28T04:00:00.000Z 2015-07-28T06:00:00.000Z +2015-07-29T00:00:00.000Z A 2 2015-07-29T03:00:00.000Z 2015-07-29T04:00:00.000Z +2015-07-29T00:00:00.000Z C 2 2015-07-29T01:00:00.000Z 2015-07-29T02:00:00.000Z +``` diff --git a/development/extensions-core/approximate-histograms.md b/development/extensions-core/approximate-histograms.md new file mode 100644 index 0000000..508e327 --- /dev/null +++ b/development/extensions-core/approximate-histograms.md @@ -0,0 +1,320 @@ +--- +id: approximate-histograms +title: "Approximate Histogram aggregators" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-histogram` as an extension. + +The `druid-histogram` extension provides an approximate histogram aggregator and a fixed buckets histogram aggregator. + + + +## Approximate Histogram aggregator (Deprecated) + +> The Approximate Histogram aggregator is deprecated. Please use [DataSketches Quantiles](../extensions-core/datasketches-quantiles.md) instead which provides a superior distribution-independent algorithm with formal error guarantees. + +This aggregator is based on +[http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf](http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf) +to compute approximate histograms, with the following modifications: + +- some tradeoffs in accuracy were made in the interest of speed (see below) +- the sketch maintains the exact original data as long as the number of + distinct data points is fewer than the resolutions (number of centroids), + increasing accuracy when there are few data points, or when dealing with + discrete data points. You can find some of the details in [this post](https://metamarkets.com/2013/histograms/). + +Approximate histogram sketches are still experimental for a reason, and you +should understand the limitations of the current implementation before using +them. The approximation is heavily data-dependent, which makes it difficult to +give good general guidelines, so you should experiment and see what parameters +work well for your data. + +Here are a few things to note before using them: + +- As indicated in the original paper, there are no formal error bounds on the + approximation. In practice, the approximation gets worse if the distribution + is skewed. +- The algorithm is order-dependent, so results can vary for the same query, due + to variations in the order in which results are merged. +- In general, the algorithm only works well if the data that comes is randomly + distributed (i.e. if data points end up sorted in a column, approximation + will be horrible) +- We traded accuracy for aggregation speed, taking some shortcuts when adding + histograms together, which can lead to pathological cases if your data is + ordered in some way, or if your distribution has long tails. It should be + cheaper to increase the resolution of the sketch to get the accuracy you need. + +That being said, those sketches can be useful to get a first order approximation +when averages are not good enough. Assuming most rows in your segment store +fewer data points than the resolution of histogram, you should be able to use +them for monitoring purposes and detect meaningful variations with a few +hundred centroids. To get good accuracy readings on 95th percentiles with +millions of rows of data, you may want to use several thousand centroids, +especially with long tails, since that's where the approximation will be worse. + +### Creating approximate histogram sketches at ingestion time + +To use this feature, an "approxHistogram" or "approxHistogramFold" aggregator must be included at +indexing time. The ingestion aggregator can only apply to numeric values. If you use "approxHistogram" +then any input rows missing the value will be considered to have a value of 0, while with "approxHistogramFold" +such rows will be ignored. + +To query for results, an "approxHistogramFold" aggregator must be included in the +query. + +```json +{ + "type" : "approxHistogram or approxHistogramFold (at ingestion time), approxHistogramFold (at query time)", + "name" : , + "fieldName" : , + "resolution" : , + "numBuckets" : , + "lowerLimit" : , + "upperLimit" : +} +``` + +|Property |Description |Default | +|-------------------------|------------------------------|----------------------------------| +|`resolution` |Number of centroids (data points) to store. The higher the resolution, the more accurate results are, but the slower the computation will be.|50| +|`numBuckets` |Number of output buckets for the resulting histogram. Bucket intervals are dynamic, based on the range of the underlying data. Use a post-aggregator to have finer control over the bucketing scheme|7| +|`lowerLimit`/`upperLimit`|Restrict the approximation to the given range. The values outside this range will be aggregated into two centroids. Counts of values outside this range are still maintained. |-INF/+INF| +|`finalizeAsBase64Binary` |If true, the finalized aggregator value will be a Base64-encoded byte array containing the serialized form of the histogram. If false, the finalized aggregator value will be a JSON representation of the histogram.|false| + +## Fixed Buckets Histogram + +The fixed buckets histogram aggregator builds a histogram on a numeric column, with evenly-sized buckets across a specified value range. Values outside of the range are handled based on a user-specified outlier handling mode. + +This histogram supports the min/max/quantiles post-aggregators but does not support the bucketing post-aggregators. + +### When to use + +The accuracy/usefulness of the fixed buckets histogram is extremely data-dependent; it is provided to support special use cases where the user has a great deal of prior information about the data being aggregated and knows that a fixed buckets implementation is suitable. + +For general histogram and quantile use cases, the [DataSketches Quantiles Sketch](../extensions-core/datasketches-quantiles.md) extension is recommended. + +### Properties + + +|Property |Description |Default | +|-------------------------|------------------------------|----------------------------------| +|`type`|Type of the aggregator. Must `fixedBucketsHistogram`.|No default, must be specified| +|`name`|Column name for the aggregator.|No default, must be specified| +|`fieldName`|Column name of the input to the aggregator.|No default, must be specified| +|`lowerLimit`|Lower limit of the histogram. |No default, must be specified| +|`upperLimit`|Upper limit of the histogram. |No default, must be specified| +|`numBuckets`|Number of buckets for the histogram. The range [lowerLimit, upperLimit] will be divided into `numBuckets` intervals of equal size.|10| +|`outlierHandlingMode`|Specifies how values outside of [lowerLimit, upperLimit] will be handled. Supported modes are "ignore", "overflow", and "clip". See [outlier handling modes](#outlier-handling-modes) for more details.|No default, must be specified| +|`finalizeAsBase64Binary`|If true, the finalized aggregator value will be a Base64-encoded byte array containing the [serialized form](#serialization-formats) of the histogram. If false, the finalized aggregator value will be a JSON representation of the histogram.|false| + +An example aggregator spec is shown below: + +```json +{ + "type" : "fixedBucketsHistogram", + "name" : , + "fieldName" : , + "numBuckets" : , + "lowerLimit" : , + "upperLimit" : , + "outlierHandlingMode": +} +``` + +### Outlier handling modes + +The outlier handling mode specifies what should be done with values outside of the histogram's range. There are three supported modes: + +- `ignore`: Throw away outlier values. +- `overflow`: A count of outlier values will be tracked by the histogram, available in the `lowerOutlierCount` and `upperOutlierCount` fields. +- `clip`: Outlier values will be clipped to the `lowerLimit` or the `upperLimit` and included in the histogram. + +If you don't care about outliers, `ignore` is the cheapest option performance-wise. There is currently no difference in storage size among the modes. + +### Output fields + +The histogram aggregator's output object has the following fields: + +- `lowerLimit`: Lower limit of the histogram +- `upperLimit`: Upper limit of the histogram +- `numBuckets`: Number of histogram buckets +- `outlierHandlingMode`: Outlier handling mode +- `count`: Total number of values contained in the histogram, excluding outliers +- `lowerOutlierCount`: Count of outlier values below `lowerLimit`. Only used if the outlier mode is `overflow`. +- `upperOutlierCount`: Count of outlier values above `upperLimit`. Only used if the outlier mode is `overflow`. +- `missingValueCount`: Count of null values seen by the histogram. +- `max`: Max value seen by the histogram. This does not include outlier values. +- `min`: Min value seen by the histogram. This does not include outlier values. +- `histogram`: An array of longs with size `numBuckets`, containing the bucket counts + +### Ingesting existing histograms + +It is also possible to ingest existing fixed buckets histograms. The input must be a Base64 string encoding a byte array that contains a serialized histogram object. Both "full" and "sparse" formats can be used. Please see [Serialization formats](#serialization-formats) below for details. + +### Serialization formats + +#### Full serialization format + +This format includes the full histogram bucket count array in the serialization format. + +``` +byte: serialization version, must be 0x01 +byte: encoding mode, 0x01 for full +double: lowerLimit +double: upperLimit +int: numBuckets +byte: outlier handling mode (0x00 for `ignore`, 0x01 for `overflow`, and 0x02 for `clip`) +long: count, total number of values contained in the histogram, excluding outliers +long: lowerOutlierCount +long: upperOutlierCount +long: missingValueCount +double: max +double: min +array of longs: bucket counts for the histogram +``` + +#### Sparse serialization format + +This format represents the histogram bucket counts as (bucketNum, count) pairs. This serialization format is used when less than half of the histogram's buckets have values. + +``` +byte: serialization version, must be 0x01 +byte: encoding mode, 0x02 for sparse +double: lowerLimit +double: upperLimit +int: numBuckets +byte: outlier handling mode (0x00 for `ignore`, 0x01 for `overflow`, and 0x02 for `clip`) +long: count, total number of values contained in the histogram, excluding outliers +long: lowerOutlierCount +long: upperOutlierCount +long: missingValueCount +double: max +double: min +int: number of following (bucketNum, count) pairs +sequence of (int, long) pairs: + int: bucket number + count: bucket count +``` + +### Combining histograms with different bucketing schemes + +It is possible to combine two histograms with different bucketing schemes (lowerLimit, upperLimit, numBuckets) together. + +The bucketing scheme of the "left hand" histogram will be preserved (i.e., when running a query, the bucketing schemes specified in the query's histogram aggregators will be preserved). + +When merging, we assume that values are evenly distributed within the buckets of the "right hand" histogram. + +When the right-hand histogram contains outliers (when using `overflow` mode), we assume that all of the outliers counted in the right-hand histogram will be outliers in the left-hand histogram as well. + +For performance and accuracy reasons, we recommend avoiding aggregation of histograms with different bucketing schemes if possible. + +### Null handling + +If `druid.generic.useDefaultValueForNull` is false, null values will be tracked in the `missingValueCount` field of the histogram. + +If `druid.generic.useDefaultValueForNull` is true, null values will be added to the histogram as the default 0.0 value. + +## Histogram post-aggregators + +Post-aggregators are used to transform opaque approximate histogram sketches +into bucketed histogram representations, as well as to compute various +distribution metrics such as quantiles, min, and max. + +### Equal buckets post-aggregator + +Computes a visual representation of the approximate histogram with a given number of equal-sized bins. +Bucket intervals are based on the range of the underlying data. This aggregator is not supported for the fixed buckets histogram. + +```json +{ + "type": "equalBuckets", + "name": "", + "fieldName": "", + "numBuckets": +} +``` + +### Buckets post-aggregator + +Computes a visual representation given an initial breakpoint, offset, and a bucket size. + +Bucket size determines the width of the binning interval. + +Offset determines the value on which those interval bins align. + +This aggregator is not supported for the fixed buckets histogram. + +```json +{ + "type": "buckets", + "name": "", + "fieldName": "", + "bucketSize": , + "offset": +} +``` + +### Custom buckets post-aggregator + +Computes a visual representation of the approximate histogram with bins laid out according to the given breaks. + +This aggregator is not supported for the fixed buckets histogram. + +```json +{ "type" : "customBuckets", "name" : , "fieldName" : , + "breaks" : [ , , ... ] } +``` + +### min post-aggregator + +Returns the minimum value of the underlying approximate or fixed buckets histogram aggregator + +```json +{ "type" : "min", "name" : , "fieldName" : } +``` + +### max post-aggregator + +Returns the maximum value of the underlying approximate or fixed buckets histogram aggregator + +```json +{ "type" : "max", "name" : , "fieldName" : } +``` + +#### quantile post-aggregator + +Computes a single quantile based on the underlying approximate or fixed buckets histogram aggregator + +```json +{ "type" : "quantile", "name" : , "fieldName" : , + "probability" : } +``` + +#### quantiles post-aggregator + +Computes an array of quantiles based on the underlying approximate or fixed buckets histogram aggregator + +```json +{ "type" : "quantiles", "name" : , "fieldName" : , + "probabilities" : [ , , ... ] } +``` diff --git a/development/extensions-core/avro.md b/development/extensions-core/avro.md new file mode 100644 index 0000000..3ec4c70 --- /dev/null +++ b/development/extensions-core/avro.md @@ -0,0 +1,32 @@ +--- +id: avro +title: "Apache Avro" +--- + + + +## Avro extension + +This Apache Druid extension enables Druid to ingest and understand the Apache Avro data format. This extension provides +two Avro Parsers for stream ingestion and Hadoop batch ingestion. +See [Avro Hadoop Parser](../../ingestion/data-formats.md#avro-hadoop-parser) and [Avro Stream Parser](../../ingestion/data-formats.md#avro-stream-parser) +for more details about how to use these in an ingestion spec. + +Make sure to [include](../../development/extensions.md#loading-extensions) `druid-avro-extensions` as an extension. \ No newline at end of file diff --git a/development/extensions-core/azure.md b/development/extensions-core/azure.md new file mode 100644 index 0000000..ed05b42 --- /dev/null +++ b/development/extensions-core/azure.md @@ -0,0 +1,43 @@ +--- +id: azure +title: "Microsoft Azure" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-azure-extensions` extension. + +## Deep Storage + +[Microsoft Azure Storage](http://azure.microsoft.com/en-us/services/storage/) is another option for deep storage. This requires some additional Druid configuration. + +|Property|Description|Possible Values|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|azure||Must be set.| +|`druid.azure.account`||Azure Storage account name.|Must be set.| +|`druid.azure.key`||Azure Storage account key.|Must be set.| +|`druid.azure.container`||Azure Storage container name.|Must be set.| +|`druid.azure.prefix`|A prefix string that will be prepended to the blob names for the segments published to Azure deep storage| |""| +|`druid.azure.protocol`|the protocol to use|http or https|https| +|`druid.azure.maxTries`|Number of tries before canceling an Azure operation.| |3| +|`druid.azure.maxListingLength`|maximum number of input files matching a given prefix to retrieve at a time| |1024| + +See [Azure Services](http://azure.microsoft.com/en-us/pricing/free-trial/) for more information. diff --git a/development/extensions-core/bloom-filter.md b/development/extensions-core/bloom-filter.md new file mode 100644 index 0000000..602f1f2 --- /dev/null +++ b/development/extensions-core/bloom-filter.md @@ -0,0 +1,179 @@ +--- +id: bloom-filter +title: "Bloom Filter" +--- + + + + +This Apache Druid extension adds the ability to both construct bloom filters from query results, and filter query results by testing +against a bloom filter. Make sure to [include](../../development/extensions.md#loading-extensions) `druid-bloom-filter` as an +extension. + +A Bloom filter is a probabilistic data structure for performing a set membership check. A bloom filter is a good candidate +to use with Druid for cases where an explicit filter is impossible, e.g. filtering a query against a set of millions of + values. + +Following are some characteristics of Bloom filters: + +- Bloom filters are highly space efficient when compared to using a HashSet. +- Because of the probabilistic nature of bloom filters, false positive results are possible (element was not actually +inserted into a bloom filter during construction, but `test()` says true) +- False negatives are not possible (if element is present then `test()` will never say false). +- The false positive probability of this implementation is currently fixed at 5%, but increasing the number of entries +that the filter can hold can decrease this false positive rate in exchange for overall size. +- Bloom filters are sensitive to number of elements that will be inserted in the bloom filter. During the creation of bloom filter expected number of entries must be specified. If the number of insertions exceed + the specified initial number of entries then false positive probability will increase accordingly. + +This extension is currently based on `org.apache.hive.common.util.BloomKFilter` from `hive-storage-api`. Internally, +this implementation uses Murmur3 as the hash algorithm. + +To construct a BloomKFilter externally with Java to use as a filter in a Druid query: + +```java +BloomKFilter bloomFilter = new BloomKFilter(1500); +bloomFilter.addString("value 1"); +bloomFilter.addString("value 2"); +bloomFilter.addString("value 3"); +ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); +BloomKFilter.serialize(byteArrayOutputStream, bloomFilter); +String base64Serialized = Base64.encodeBase64String(byteArrayOutputStream.toByteArray()); +``` + +This string can then be used in the native or SQL Druid query. + +## Filtering queries with a Bloom Filter + +### JSON Specification of Bloom Filter +```json +{ + "type" : "bloom", + "dimension" : , + "bloomKFilter" : , + "extractionFn" : +} +``` + +|Property |Description |required? | +|-------------------------|------------------------------|----------------------------------| +|`type` |Filter Type. Should always be `bloom`|yes| +|`dimension` |The dimension to filter over. | yes | +|`bloomKFilter` |Base64 encoded Binary representation of `org.apache.hive.common.util.BloomKFilter`| yes | +|`extractionFn`|[Extraction function](../../querying/dimensionspecs.html#extraction-functions) to apply to the dimension values |no| + + +### Serialized Format for BloomKFilter + + Serialized BloomKFilter format: + + - 1 byte for the number of hash functions. + - 1 big endian int(That is how OutputStream works) for the number of longs in the bitset + - big endian longs in the BloomKFilter bitset + +Note: `org.apache.hive.common.util.BloomKFilter` provides a serialize method which can be used to serialize bloom filters to outputStream. + +### Filtering SQL Queries + +Bloom filters can be used in SQL `WHERE` clauses via the `bloom_filter_test` operator: + +```sql +SELECT COUNT(*) FROM druid.foo WHERE bloom_filter_test(, '') +``` + +### Expression and Virtual Column Support + +The bloom filter extension also adds a bloom filter [Druid expression](../../misc/math-expr.md) which shares syntax +with the SQL operator. + +```sql +bloom_filter_test(, '') +``` + +## Bloom Filter Query Aggregator + +Input for a `bloomKFilter` can also be created from a druid query with the `bloom` aggregator. Note that it is very +important to set a reasonable value for the `maxNumEntries` parameter, which is the maximum number of distinct entries +that the bloom filter can represent without increasing the false positive rate. It may be worth performing a query using +one of the unique count sketches to calculate the value for this parameter in order to build a bloom filter appropriate +for the query. + +### JSON Specification of Bloom Filter Aggregator + +```json +{ + "type": "bloom", + "name": , + "maxNumEntries": + "field": + } +``` + +|Property |Description |required? | +|-------------------------|------------------------------|----------------------------------| +|`type` |Aggregator Type. Should always be `bloom`|yes| +|`name` |Output field name |yes| +|`field` |[DimensionSpec](../../querying/dimensionspecs.md) to add to `org.apache.hive.common.util.BloomKFilter` | yes | +|`maxNumEntries` |Maximum number of distinct values supported by `org.apache.hive.common.util.BloomKFilter`, default `1500`| no | + +### Example + +```json +{ + "queryType": "timeseries", + "dataSource": "wikiticker", + "intervals": [ "2015-09-12T00:00:00.000/2015-09-13T00:00:00.000" ], + "granularity": "day", + "aggregations": [ + { + "type": "bloom", + "name": "userBloom", + "maxNumEntries": 100000, + "field": { + "type":"default", + "dimension":"user", + "outputType": "STRING" + } + } + ] +} +``` + +response + +```json +[{"timestamp":"2015-09-12T00:00:00.000Z","result":{"userBloom":"BAAAJhAAAA..."}}] +``` + +These values can then be set in the filter specification described above. + +Ordering results by a bloom filter aggregator, for example in a TopN query, will perform a comparatively expensive +linear scan _of the filter itself_ to count the number of set bits as a means of approximating how many items have been +added to the set. As such, ordering by an alternate aggregation is recommended if possible. + + +### SQL Bloom Filter Aggregator +Bloom filters can be computed in SQL expressions with the `bloom_filter` aggregator: + +```sql +SELECT BLOOM_FILTER(, ) FROM druid.foo WHERE dim2 = 'abc' +``` + +but requires the setting `druid.sql.planner.serializeComplexValues` to be set to `true`. Bloom filter results in a SQL + response are serialized into a base64 string, which can then be used in subsequent queries as a filter. diff --git a/development/extensions-core/datasketches-extension.md b/development/extensions-core/datasketches-extension.md new file mode 100644 index 0000000..07d1021 --- /dev/null +++ b/development/extensions-core/datasketches-extension.md @@ -0,0 +1,39 @@ +--- +id: datasketches-extension +title: "DataSketches extension" +--- + + + + +Apache Druid aggregators based on [Apache DataSketches](https://datasketches.apache.org/) library. Sketches are data structures implementing approximate streaming mergeable algorithms. Sketches can be ingested from the outside of Druid or built from raw data at ingestion time. Sketches can be stored in Druid segments as additive metrics. + +To use the datasketches aggregators, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-datasketches"] +``` + +The following modules are available: + +* [Theta sketch](datasketches-theta.html) - approximate distinct counting with set operations (union, intersection and set difference). +* [Tuple sketch](datasketches-tuple.html) - extension of Theta sketch to support values associated with distinct keys (arrays of numeric values in this specialized implementation). +* [Quantiles sketch](datasketches-quantiles.html) - approximate distribution of comparable values to obtain ranks, quantiles and histograms. This is a specialized implementation for numeric values. +* [HLL sketch](datasketches-hll.html) - approximate distinct counting using very compact HLL sketch. diff --git a/development/extensions-core/datasketches-hll.md b/development/extensions-core/datasketches-hll.md new file mode 100644 index 0000000..cc39e7e --- /dev/null +++ b/development/extensions-core/datasketches-hll.md @@ -0,0 +1,121 @@ +--- +id: datasketches-hll +title: "DataSketches HLL Sketch module" +--- + + + + +This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregator to produce a union of sketch columns in the same row. +You can use the HLL sketch aggregator on columns of any identifiers. It will return estimated cardinality of the column. + +To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-datasketches"] +``` + +### Aggregators + +``` +{ + "type" : "HLLSketchBuild", + "name" : , + "fieldName" : , + "lgK" : , + "tgtHllType" : , + "round": + } +``` + +``` +{ + "type" : "HLLSketchMerge", + "name" : , + "fieldName" : , + "lgK" : , + "tgtHllType" : , + "round": + } +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should be "HLLSketchBuild" or "HLLSketchMerge"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the input field.|yes| +|lgK|log2 of K that is the number of buckets in the sketch, parameter that controls the size and the accuracy. Must be a power of 2 from 4 to 21 inclusively.|no, defaults to 12| +|tgtHllType|The type of the target HLL sketch. Must be "HLL_4", "HLL_6" or "HLL_8" |no, defaults to "HLL_4"| +|round|Round off values to whole numbers. Only affects query-time behavior and is ignored at ingestion-time.|no, defaults to false| + +### Post Aggregators + +#### Estimate + +Returns the distinct count estimate as a double. + +``` +{ + "type" : "HLLSketchEstimate", + "name": , + "field" : , + "round" : +} +``` + +#### Estimate with bounds + +Returns a distinct count estimate and error bounds from an HLL sketch. +The result will be an array containing three double values: estimate, lower bound and upper bound. +The bounds are provided at a given number of standard deviations (optional, defaults to 1). +This must be an integer value of 1, 2 or 3 corresponding to approximately 68.3%, 95.4% and 99.7% confidence intervals. + +``` +{ + "type" : "HLLSketchEstimateWithBounds", + "name": , + "field" : , + "numStdDev" : +} +``` + +#### Union + +``` +{ + "type" : "HLLSketchUnion", + "name": , + "fields" : , + "lgK": , + "tgtHllType" : +} +``` + +#### Sketch to string + +Human-readable sketch summary for debugging. + +``` +{ + "type" : "HLLSketchToString", + "name": , + "field" : +} +``` diff --git a/development/extensions-core/datasketches-quantiles.md b/development/extensions-core/datasketches-quantiles.md new file mode 100644 index 0000000..031fb02 --- /dev/null +++ b/development/extensions-core/datasketches-quantiles.md @@ -0,0 +1,137 @@ +--- +id: datasketches-quantiles +title: "DataSketches Quantiles Sketch module" +--- + + + + +This module provides Apache Druid aggregators based on numeric quantiles DoublesSketch from [Apache DataSketches](https://datasketches.apache.org/) library. Quantiles sketch is a mergeable streaming algorithm to estimate the distribution of values, and approximately answer queries about the rank of a value, probability mass function of the distribution (PMF) or histogram, cumulative distribution function (CDF), and quantiles (median, min, max, 95th percentile and such). See [Quantiles Sketch Overview](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html). + +There are three major modes of operation: + +1. Ingesting sketches built outside of Druid (say, with Pig or Hive) +2. Building sketches from raw data during ingestion +3. Building sketches from raw data at query time + +To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-datasketches"] +``` + +### Aggregator + +The result of the aggregation is a DoublesSketch that is the union of all sketches either built from raw data or read from the segments. + +```json +{ + "type" : "quantilesDoublesSketch", + "name" : , + "fieldName" : , + "k": + } +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "quantilesDoublesSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the input field (can contain sketches or raw numeric values).|yes| +|k|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Must be a power of 2 from 2 to 32768. See the [Quantiles Accuracy](https://datasketches.apache.org/docs/Quantiles/QuantilesAccuracy.html) for details. |no, defaults to 128| + +### Post Aggregators + +#### Quantile + +This returns an approximation to the value that would be preceded by a given fraction of a hypothetical sorted version of the input stream. + +```json +{ + "type" : "quantilesDoublesSketchToQuantile", + "name": , + "field" : , + "fraction" : +} +``` + +#### Quantiles + +This returns an array of quantiles corresponding to a given array of fractions + +```json +{ + "type" : "quantilesDoublesSketchToQuantiles", + "name": , + "field" : , + "fractions" : +} +``` + +#### Histogram + +This returns an approximation to the histogram given an array of split points that define the histogram bins or a number of bins (not both). An array of m unique, monotonically increasing split points divide the real number line into m+1 consecutive disjoint intervals. The definition of an interval is inclusive of the left split point and exclusive of the right split point. If the number of bins is specified instead of split points, the interval between the minimum and maximum values is divided into the given number of equally-spaced bins. + +```json +{ + "type" : "quantilesDoublesSketchToHistogram", + "name": , + "field" : , + "splitPoints" : , + "numBins" : +} +``` + +#### Rank + +This returns an approximation to the rank of a given value that is the fraction of the distribution less than that value. + +```json +{ + "type" : "quantilesDoublesSketchToRank", + "name": , + "field" : , + "value" : +} +``` +#### CDF + +This returns an approximation to the Cumulative Distribution Function given an array of split points that define the edges of the bins. An array of m unique, monotonically increasing split points divide the real number line into m+1 consecutive disjoint intervals. The definition of an interval is inclusive of the left split point and exclusive of the right split point. The resulting array of fractions can be viewed as ranks of each split point with one additional rank that is always 1. + +```json +{ + "type" : "quantilesDoublesSketchToCDF", + "name": , + "field" : , + "splitPoints" : +} +``` + +#### Sketch Summary + +This returns a summary of the sketch that can be used for debugging. This is the result of calling toString() method. + +```json +{ + "type" : "quantilesDoublesSketchToString", + "name": , + "field" : +} +``` diff --git a/development/extensions-core/datasketches-theta.md b/development/extensions-core/datasketches-theta.md new file mode 100644 index 0000000..5b5564e --- /dev/null +++ b/development/extensions-core/datasketches-theta.md @@ -0,0 +1,284 @@ +--- +id: datasketches-theta +title: "DataSketches Theta Sketch module" +--- + + + + +This module provides Apache Druid aggregators based on Theta sketch from [Apache DataSketches](https://datasketches.apache.org/) library. Note that sketch algorithms are approximate; see details in the "Accuracy" section of the datasketches doc. +At ingestion time, this aggregator creates the Theta sketch objects which get stored in Druid segments. Logically speaking, a Theta sketch object can be thought of as a Set data structure. At query time, sketches are read and aggregated (set unioned) together. In the end, by default, you receive the estimate of the number of unique entries in the sketch object. Also, you can use post aggregators to do union, intersection or difference on sketch columns in the same row. +Note that you can use `thetaSketch` aggregator on columns which were not ingested using the same. It will return estimated cardinality of the column. It is recommended to use it at ingestion time as well to make querying faster. + +To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-datasketches"] +``` + +### Aggregators + +```json +{ + "type" : "thetaSketch", + "name" : , + "fieldName" : , + "isInputThetaSketch": false, + "size": 16384 + } +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "thetaSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the aggregator used at ingestion time.|yes| +|isInputThetaSketch|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| +|size|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object will retain. Higher size means higher accuracy but more space to store sketches. Note that after you index with a particular size, druid will persist sketch in segments and you will use size greater or equal to that at query time. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize.html) for details. In general, We recommend just sticking to default size. |no, defaults to 16384| + +### Post Aggregators + +#### Sketch Estimator + +```json +{ + "type" : "thetaSketchEstimate", + "name": , + "field" : +} +``` + +#### Sketch Operations + +```json +{ + "type" : "thetaSketchSetOp", + "name": , + "func": , + "fields" : , + "size": <16384 by default, must be max of size from sketches in fields input> +} +``` + +#### Sketch Summary + +This returns a summary of the sketch that can be used for debugging. This is the result of calling toString() method. + +```json +{ + "type" : "thetaSketchToString", + "name": , + "field" : +} +``` + +### Examples + +Assuming, you have a dataset containing (timestamp, product, user_id). You want to answer questions like + +How many unique users visited product A? +How many unique users visited both product A and product B? + +to answer above questions, you would index your data using following aggregator. + +```json +{ "type": "thetaSketch", "name": "user_id_sketch", "fieldName": "user_id" } +``` + +then, sample query for, How many unique users visited product A? + +```json +{ + "queryType": "groupBy", + "dataSource": "test_datasource", + "granularity": "ALL", + "dimensions": [], + "aggregations": [ + { "type": "thetaSketch", "name": "unique_users", "fieldName": "user_id_sketch" } + ], + "filter": { "type": "selector", "dimension": "product", "value": "A" }, + "intervals": [ "2014-10-19T00:00:00.000Z/2014-10-22T00:00:00.000Z" ] +} +``` + +sample query for, How many unique users visited both product A and B? + +```json +{ + "queryType": "groupBy", + "dataSource": "test_datasource", + "granularity": "ALL", + "dimensions": [], + "filter": { + "type": "or", + "fields": [ + {"type": "selector", "dimension": "product", "value": "A"}, + {"type": "selector", "dimension": "product", "value": "B"} + ] + }, + "aggregations": [ + { + "type" : "filtered", + "filter" : { + "type" : "selector", + "dimension" : "product", + "value" : "A" + }, + "aggregator" : { + "type": "thetaSketch", "name": "A_unique_users", "fieldName": "user_id_sketch" + } + }, + { + "type" : "filtered", + "filter" : { + "type" : "selector", + "dimension" : "product", + "value" : "B" + }, + "aggregator" : { + "type": "thetaSketch", "name": "B_unique_users", "fieldName": "user_id_sketch" + } + } + ], + "postAggregations": [ + { + "type": "thetaSketchEstimate", + "name": "final_unique_users", + "field": + { + "type": "thetaSketchSetOp", + "name": "final_unique_users_sketch", + "func": "INTERSECT", + "fields": [ + { + "type": "fieldAccess", + "fieldName": "A_unique_users" + }, + { + "type": "fieldAccess", + "fieldName": "B_unique_users" + } + ] + } + } + ], + "intervals": [ + "2014-10-19T00:00:00.000Z/2014-10-22T00:00:00.000Z" + ] +} +``` + +#### Retention Analysis Example + +Suppose you want to answer a question like, "How many unique users performed a specific action in a particular time period and also performed another specific action in a different time period?" + +e.g., "How many unique users signed up in week 1, and purchased something in week 2?" + +Using the `(timestamp, product, user_id)` example dataset, data would be indexed with the following aggregator, like in the example above: + +```json +{ "type": "thetaSketch", "name": "user_id_sketch", "fieldName": "user_id" } +``` + +The following query expresses: + +"Out of the unique users who visited Product A between 10/01/2014 and 10/07/2014, how many visited Product A again in the week of 10/08/2014 to 10/14/2014?" + +```json +{ + "queryType": "groupBy", + "dataSource": "test_datasource", + "granularity": "ALL", + "dimensions": [], + "filter": { + "type": "or", + "fields": [ + {"type": "selector", "dimension": "product", "value": "A"} + ] + }, + "aggregations": [ + { + "type" : "filtered", + "filter" : { + "type" : "and", + "fields" : [ + { + "type" : "selector", + "dimension" : "product", + "value" : "A" + }, + { + "type" : "interval", + "dimension" : "__time", + "intervals" : ["2014-10-01T00:00:00.000Z/2014-10-07T00:00:00.000Z"] + } + ] + }, + "aggregator" : { + "type": "thetaSketch", "name": "A_unique_users_week_1", "fieldName": "user_id_sketch" + } + }, + { + "type" : "filtered", + "filter" : { + "type" : "and", + "fields" : [ + { + "type" : "selector", + "dimension" : "product", + "value" : "A" + }, + { + "type" : "interval", + "dimension" : "__time", + "intervals" : ["2014-10-08T00:00:00.000Z/2014-10-14T00:00:00.000Z"] + } + ] + }, + "aggregator" : { + "type": "thetaSketch", "name": "A_unique_users_week_2", "fieldName": "user_id_sketch" + } + }, + ], + "postAggregations": [ + { + "type": "thetaSketchEstimate", + "name": "final_unique_users", + "field": + { + "type": "thetaSketchSetOp", + "name": "final_unique_users_sketch", + "func": "INTERSECT", + "fields": [ + { + "type": "fieldAccess", + "fieldName": "A_unique_users_week_1" + }, + { + "type": "fieldAccess", + "fieldName": "A_unique_users_week_2" + } + ] + } + } + ], + "intervals": ["2014-10-01T00:00:00.000Z/2014-10-14T00:00:00.000Z"] +} +``` diff --git a/development/extensions-core/datasketches-tuple.md b/development/extensions-core/datasketches-tuple.md new file mode 100644 index 0000000..b1150c5 --- /dev/null +++ b/development/extensions-core/datasketches-tuple.md @@ -0,0 +1,174 @@ +--- +id: datasketches-tuple +title: "DataSketches Tuple Sketch module" +--- + + + + +This module provides Apache Druid aggregators based on Tuple sketch from [Apache DataSketches](https://datasketches.apache.org/) library. ArrayOfDoublesSketch sketches extend the functionality of the count-distinct Theta sketches by adding arrays of double values associated with unique keys. + +To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: + +``` +druid.extensions.loadList=["druid-datasketches"] +``` + +### Aggregators + +```json +{ + "type" : "arrayOfDoublesSketch", + "name" : , + "fieldName" : , + "nominalEntries": , + "numberOfValues" : , + "metricColumns" : + } +``` + +|property|description|required?| +|--------|-----------|---------| +|type|This String should always be "arrayOfDoublesSketch"|yes| +|name|A String for the output (result) name of the calculation.|yes| +|fieldName|A String for the name of the input field.|yes| +|nominalEntries|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Must be a power of 2. See the [Theta sketch accuracy](https://datasketches.apache.org/docs/Theta/ThetaErrorTable.html) for details. |no, defaults to 16384| +|numberOfValues|Number of values associated with each distinct key. |no, defaults to 1| +|metricColumns|If building sketches from raw data, an array of names of the input columns containing numeric values to be associated with each distinct key.|no, defaults to empty array| + +### Post Aggregators + +#### Estimate of the number of distinct keys + +Returns a distinct count estimate from a given ArrayOfDoublesSketch. + +```json +{ + "type" : "arrayOfDoublesSketchToEstimate", + "name": , + "field" : +} +``` + +#### Estimate of the number of distinct keys with error bounds + +Returns a distinct count estimate and error bounds from a given ArrayOfDoublesSketch. The result will be three double values: estimate of the number of distinct keys, lower bound and upper bound. The bounds are provided at the given number of standard deviations (optional, defaults to 1). This must be an integer value of 1, 2 or 3 corresponding to approximately 68.3%, 95.4% and 99.7% confidence intervals. + +```json +{ + "type" : "arrayOfDoublesSketchToEstimateAndBounds", + "name": , + "field" : , + "numStdDevs", +} +``` + +#### Number of retained entries + +Returns the number of retained entries from a given ArrayOfDoublesSketch. + +```json +{ + "type" : "arrayOfDoublesSketchToNumEntries", + "name": , + "field" : +} +``` + +#### Mean values for each column + +Returns a list of mean values from a given ArrayOfDoublesSketch. The result will be N double values, where N is the number of double values kept in the sketch per key. + +```json +{ + "type" : "arrayOfDoublesSketchToMeans", + "name": , + "field" : +} +``` + +#### Variance values for each column + +Returns a list of variance values from a given ArrayOfDoublesSketch. The result will be N double values, where N is the number of double values kept in the sketch per key. + +```json +{ + "type" : "arrayOfDoublesSketchToVariances", + "name": , + "field" : +} +``` + +#### Quantiles sketch from a column + +Returns a quantiles DoublesSketch constructed from a given column of values from a given ArrayOfDoublesSketch using optional parameter k that determines the accuracy and size of the quantiles sketch. See [Quantiles Sketch Module](datasketches-quantiles.html) + +* The column number is 1-based and is optional (the default is 1). +* The parameter k is optional (the default is defined in the sketch library). +* The result is a quantiles sketch. + +```json +{ + "type" : "arrayOfDoublesSketchToQuantilesSketch", + "name": , + "field" : , + "column" : , + "k" : +} +``` + +#### Set Operations + +Returns a result of a specified set operation on the given array of sketches. Supported operations are: union, intersection and set difference (UNION, INTERSECT, NOT). + +```json +{ + "type" : "arrayOfDoublesSketchSetOp", + "name": , + "operation": <"UNION"|"INTERSECT"|"NOT">, + "fields" : , + "nominalEntries" : , + "numberOfValues" : +} +``` + +#### Student's t-test + +Performs Student's t-test and returns a list of p-values given two instances of ArrayOfDoublesSketch. The result will be N double values, where N is the number of double values kept in the sketch per key. See [t-test documentation](http://commons.apache.org/proper/commons-math/javadocs/api-3.4/org/apache/commons/math3/stat/inference/TTest.html). + +```json +{ + "type" : "arrayOfDoublesSketchTTest", + "name": , + "fields" : , +} +``` + +#### Sketch summary + +Returns a human-readable summary of a given ArrayOfDoublesSketch. This is a string returned by toString() method of the sketch. This can be useful for debugging. + +```json +{ + "type" : "arrayOfDoublesSketchToString", + "name": , + "field" : +} +``` diff --git a/development/extensions-core/druid-basic-security.md b/development/extensions-core/druid-basic-security.md new file mode 100644 index 0000000..892306e --- /dev/null +++ b/development/extensions-core/druid-basic-security.md @@ -0,0 +1,544 @@ +--- +id: druid-basic-security +title: "Basic Security" +--- + + + + +This Apache Druid extension adds: + +- an Authenticator which supports [HTTP Basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) using the Druid metadata store or LDAP as its credentials store +- an Authorizer which implements basic role-based access control for Druid metadata store or LDAP users and groups + +Make sure to [include](../../development/extensions.md#loading-extensions) `druid-basic-security` as an extension. + +Please see [Authentication and Authorization](../../design/auth.md) for more information on the extension interfaces being implemented. + +## Configuration + +The examples in the section will use "MyBasicMetadataAuthenticator", "MyBasicLDAPAuthenticator", "MyBasicMetadataAuthorizer", and "MyBasicLDAPAuthorizer" as names for the Authenticators and Authorizer. + +These properties are not tied to specific Authenticator or Authorizer instances. + +These configuration properties should be added to the common runtime properties file. + +### Properties +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.basic.common.pollingPeriod`|Defines in milliseconds how often processes should poll the Coordinator for the current Druid metadata store authenticator/authorizer state.|60000|No| +|`druid.auth.basic.common.maxRandomDelay`|Defines in milliseconds the amount of random delay to add to the pollingPeriod, to spread polling requests across time.|6000|No| +|`druid.auth.basic.common.maxSyncRetries`|Determines how many times a service will retry if the authentication/authorization Druid metadata store state sync with the Coordinator fails.|10|No| +|`druid.auth.basic.common.cacheDirectory`|If defined, snapshots of the basic Authenticator and Authorizer Druid metadata store caches will be stored on disk in this directory. If this property is defined, when a service is starting, it will attempt to initialize its caches from these on-disk snapshots, if the service is unable to initialize its state by communicating with the Coordinator.|null|No| + + +### Creating an Authenticator that uses the Druid metadata store to lookup and validate credentials +``` +druid.auth.authenticatorChain=["MyBasicMetadataAuthenticator"] + +druid.auth.authenticator.MyBasicMetadataAuthenticator.type=basic +druid.auth.authenticator.MyBasicMetadataAuthenticator.initialAdminPassword=password1 +druid.auth.authenticator.MyBasicMetadataAuthenticator.initialInternalClientPassword=password2 +druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialsValidator.type=metadata +druid.auth.authenticator.MyBasicMetadataAuthenticator.skipOnFailure=false +druid.auth.authenticator.MyBasicMetadataAuthenticator.authorizerName=MyBasicMetadataAuthorizer +``` + +To use the Basic authenticator, add an authenticator with type `basic` to the authenticatorChain. +The authenticator needs to also define a credentialsValidator with type 'metadata' or 'ldap'. +If credentialsValidator is not specified, type 'metadata' will be used as default. + +Configuration of the named authenticator is assigned through properties with the form: + +``` +druid.auth.authenticator.. +``` + +The authenticator configuration examples in the rest of this document will use "MyBasicMetadataAuthenticator" or "MyBasicLDAPAuthenticator" as the name of the authenticators being configured. + + +#### Properties for Druid metadata store user authentication +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.initialAdminPassword`|Initial [Password Provider](../../operations/password-provider.md) for the automatically created default admin user. If no password is specified, the default admin user will not be created. If the default admin user already exists, setting this property will not affect its password.|null|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.initialInternalClientPassword`|Initial [Password Provider](../../operations/password-provider.md) for the default internal system user, used for internal process communication. If no password is specified, the default internal system user will not be created. If the default internal system user already exists, setting this property will not affect its password.|null|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.enableCacheNotifications`|If true, the Coordinator will notify Druid processes whenever a configuration change to this Authenticator occurs, allowing them to immediately update their state without waiting for polling.|true|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.cacheNotificationTimeout`|The timeout in milliseconds for the cache notifications.|5000|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialIterations`|Number of iterations to use for password hashing.|10000|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialsValidator.type`|The type of credentials store (metadata) to validate requests credentials.|metadata|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.skipOnFailure`|If true and the request credential doesn't exists or isn't fully configured in the credentials store, the request will proceed to next Authenticator in the chain.|false|No| +|`druid.auth.authenticator.MyBasicMetadataAuthenticator.authorizerName`|Authorizer that requests should be directed to|N/A|Yes| + +#### Properties for LDAP user authentication +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.initialAdminPassword`|Initial [Password Provider](../../operations/password-provider.md) for the automatically created default admin user. If no password is specified, the default admin user will not be created. If the default admin user already exists, setting this property will not affect its password.|null|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.initialInternalClientPassword`|Initial [Password Provider](../../operations/password-provider.md) for the default internal system user, used for internal process communication. If no password is specified, the default internal system user will not be created. If the default internal system user already exists, setting this property will not affect its password.|null|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.enableCacheNotifications`|If true, the Coordinator will notify Druid processes whenever a configuration change to this Authenticator occurs, allowing them to immediately update their state without waiting for polling.|true|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.cacheNotificationTimeout`|The timeout in milliseconds for the cache notifications.|5000|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialIterations`|Number of iterations to use for password hashing.|10000|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.type`|The type of credentials store (ldap) to validate requests credentials.|metadata|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.url`|URL of the LDAP server.|null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.bindUser`|LDAP bind user username.|null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.bindPassword`|[Password Provider](../../operations/password-provider.md) LDAP bind user password.|null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.baseDn`|The point from where the LDAP server will search for users.|null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.userSearch`|The filter/expression to use for the search. For example, (&(sAMAccountName=%s)(objectClass=user))|null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.userAttribute`|The attribute id identifying the attribute that will be returned as part of the search. For example, sAMAccountName. |null|Yes| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.credentialVerifyDuration`|The duration in seconds for how long valid credentials are verifiable within the cache when not requested.|600|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.credentialMaxDuration`|The max duration in seconds for valid credentials that can reside in cache regardless of how often they are requested.|3600|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.credentialsValidator.credentialCacheSize`|The valid credentials cache size. The cache uses a LRU policy.|100|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.skipOnFailure`|If true and the request credential doesn't exists or isn't fully configured in the credentials store, the request will proceed to next Authenticator in the chain.|false|No| +|`druid.auth.authenticator.MyBasicLDAPAuthenticator.authorizerName`|Authorizer that requests should be directed to.|N/A|Yes| + +### Creating an Escalator + +``` +# Escalator +druid.escalator.type=basic +druid.escalator.internalClientUsername=druid_system +druid.escalator.internalClientPassword=password2 +druid.escalator.authorizerName=MyBasicMetadataAuthorizer +``` + +#### Properties +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.escalator.internalClientUsername`|The escalator will use this username for requests made as the internal system user.|n/a|Yes| +|`druid.escalator.internalClientPassword`|The escalator will use this [Password Provider](../../operations/password-provider.md) for requests made as the internal system user.|n/a|Yes| +|`druid.escalator.authorizerName`|Authorizer that requests should be directed to.|n/a|Yes| + + +### Creating an Authorizer +``` +druid.auth.authorizers=["MyBasicMetadataAuthorizer"] + +druid.auth.authorizer.MyBasicMetadataAuthorizer.type=basic +``` + +To use the Basic authorizer, add an authorizer with type `basic` to the authorizers list. + +Configuration of the named authorizer is assigned through properties with the form: + +``` +druid.auth.authorizer.. +``` + +The authorizer configuration examples in the rest of this document will use "MyBasicMetadataAuthorizer" or "MyBasicLDAPAuthorizer" as the name of the authenticators being configured. + +#### Properties for Druid metadata store user authorization +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.authorizer.MyBasicMetadataAuthorizer.enableCacheNotifications`|If true, the Coordinator will notify Druid processes whenever a configuration change to this Authorizer occurs, allowing them to immediately update their state without waiting for polling.|true|No| +|`druid.auth.authorizer.MyBasicMetadataAuthorizer.cacheNotificationTimeout`|The timeout in milliseconds for the cache notifications.|5000|No| +|`druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminUser`|The initial admin user with role defined in initialAdminRole property if specified, otherwise the default admin role will be assigned.|admin|No| +|`druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminRole`|The initial admin role to create if it doesn't already exists.|admin|No| +|`druid.auth.authorizer.MyBasicMetadataAuthorizer.roleProvider.type`|The type of role provider to authorize requests credentials.|metadata|No + +#### Properties for LDAP user authorization +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.enableCacheNotifications`|If true, the Coordinator will notify Druid processes whenever a configuration change to this Authorizer occurs, allowing them to immediately update their state without waiting for polling.|true|No| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.cacheNotificationTimeout`|The timeout in milliseconds for the cache notifications.|5000|No| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.initialAdminUser`|The initial admin user with role defined in initialAdminRole property if specified, otherwise the default admin role will be assigned.|admin|No| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.initialAdminRole`|The initial admin role to create if it doesn't already exists.|admin|No| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.initialAdminGroupMapping`|The initial admin group mapping with role defined in initialAdminRole property if specified, otherwise the default admin role will be assigned. The name of this initial admin group mapping will be set to adminGroupMapping|null|No| +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.roleProvider.type`|The type of role provider (ldap) to authorize requests credentials.|metadata|No +|`druid.auth.authorizer.MyBasicLDAPAuthorizer.roleProvider.groupFilters`|Array of LDAP group filters used to filter out the allowed set of groups returned from LDAP search. Filters can be begin with *, or end with ,* to provide configurational flexibility to limit or filter allowed set of groups available to LDAP Authorizer.|null|No| + +## Usage + +### Coordinator Security API +To use these APIs, a user needs read/write permissions for the CONFIG resource type with name "security". + +#### Authentication API + +Root path: `/druid-ext/basic-security/authentication` + +Each API endpoint includes {authenticatorName}, specifying which Authenticator instance is being configured. + +##### User/Credential Management +`GET(/druid-ext/basic-security/authentication/db/{authenticatorName}/users)` +Return a list of all user names. + +`GET(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName})` +Return the name and credentials information of the user with name {userName} + +`POST(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName})` +Create a new user with name {userName} + +`DELETE(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName})` +Delete the user with name {userName} + +`POST(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName}/credentials)` +Assign a password used for HTTP basic authentication for {userName} +Content: JSON password request object + +Example request body: + +``` +{ + "password": "helloworld" +} +``` + +##### Cache Load Status +`GET(/druid-ext/basic-security/authentication/loadStatus)` +Return the current load status of the local caches of the authentication Druid metadata store. + +#### Authorization API + +Root path: `/druid-ext/basic-security/authorization` + +Each API endpoint includes {authorizerName}, specifying which Authorizer instance is being configured. + +##### User Creation/Deletion +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/users)` +Return a list of all user names. + +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/users/{userName})` +Return the name and role information of the user with name {userName} + +Example output: + +```json +{ + "name": "druid2", + "roles": [ + "druidRole" + ] +} +``` + +This API supports the following flags: + +- `?full`: The response will also include the full information for each role currently assigned to the user. + +Example output: + +```json +{ + "name": "druid2", + "roles": [ + { + "name": "druidRole", + "permissions": [ + { + "resourceAction": { + "resource": { + "name": "A", + "type": "DATASOURCE" + }, + "action": "READ" + }, + "resourceNamePattern": "A" + }, + { + "resourceAction": { + "resource": { + "name": "C", + "type": "CONFIG" + }, + "action": "WRITE" + }, + "resourceNamePattern": "C" + } + ] + } + ] +} +``` + +The output format of this API when `?full` is specified is deprecated and in later versions will be switched to the output format used when both `?full` and `?simplifyPermissions` flag is set. + +The `resourceNamePattern` is a compiled version of the resource name regex. It is redundant and complicates the use of this API for clients such as frontends that edit the authorization configuration, as the permission format in this output does not match the format used for adding permissions to a role. + +- `?full?simplifyPermissions`: When both `?full` and `?simplifyPermissions` are set, the permissions in the output will contain only a list of `resourceAction` objects, without the extraneous `resourceNamePattern` field. + +```json +{ + "name": "druid2", + "roles": [ + { + "name": "druidRole", + "users": null, + "permissions": [ + { + "resource": { + "name": "A", + "type": "DATASOURCE" + }, + "action": "READ" + }, + { + "resource": { + "name": "C", + "type": "CONFIG" + }, + "action": "WRITE" + } + ] + } + ] +} +``` + +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/users/{userName})` +Create a new user with name {userName} + +`DELETE(/druid-ext/basic-security/authorization/db/{authorizerName}/users/{userName})` +Delete the user with name {userName} + +##### Group mapping Creation/Deletion +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings)` +Return a list of all group mappings. + +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings/{groupMappingName})` +Return the group mapping and role information of the group mapping with name {groupMappingName} + +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings/{groupMappingName})` +Create a new group mapping with name {groupMappingName} +Content: JSON group mapping object +Example request body: + +``` +{ + "name": "user", + "groupPattern": "CN=aaa,OU=aaa,OU=Groupings,DC=corp,DC=company,DC=com", + "roles": [ + "user" + ] +} +``` + +`DELETE(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings/{groupMappingName})` +Delete the group mapping with name {groupMappingName} + +#### Role Creation/Deletion +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/roles)` +Return a list of all role names. + +`GET(/druid-ext/basic-security/authorization/db/{authorizerName}/roles/{roleName})` +Return name and permissions for the role named {roleName}. + +Example output: + +```json +{ + "name": "druidRole2", + "permissions": [ + { + "resourceAction": { + "resource": { + "name": "E", + "type": "DATASOURCE" + }, + "action": "WRITE" + }, + "resourceNamePattern": "E" + } + ] +} +``` + +The default output format of this API is deprecated and in later versions will be switched to the output format used when the `?simplifyPermissions` flag is set. The `resourceNamePattern` is a compiled version of the resource name regex. It is redundant and complicates the use of this API for clients such as frontends that edit the authorization configuration, as the permission format in this output does not match the format used for adding permissions to a role. + +This API supports the following flags: + +- `?full`: The output will contain an extra `users` list, containing the users that currently have this role. + +```json +{"users":["druid"]} +``` + +- `?simplifyPermissions`: The permissions in the output will contain only a list of `resourceAction` objects, without the extraneous `resourceNamePattern` field. The `users` field will be null when `?full` is not specified. + +Example output: + +```json +{ + "name": "druidRole2", + "users": null, + "permissions": [ + { + "resource": { + "name": "E", + "type": "DATASOURCE" + }, + "action": "WRITE" + } + ] +} +``` + + +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/roles/{roleName})` +Create a new role with name {roleName}. +Content: username string + +`DELETE(/druid-ext/basic-security/authorization/db/{authorizerName}/roles/{roleName})` +Delete the role with name {roleName}. + + +#### Role Assignment +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/users/{userName}/roles/{roleName})` +Assign role {roleName} to user {userName}. + +`DELETE(/druid-ext/basic-security/authorization/db/{authorizerName}/users/{userName}/roles/{roleName})` +Unassign role {roleName} from user {userName} + +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings/{groupMappingName}/roles/{roleName})` +Assign role {roleName} to group mapping {groupMappingName}. + +`DELETE(/druid-ext/basic-security/authorization/db/{authorizerName}/groupMappings/{groupMappingName}/roles/{roleName})` +Unassign role {roleName} from group mapping {groupMappingName} + + +#### Permissions +`POST(/druid-ext/basic-security/authorization/db/{authorizerName}/roles/{roleName}/permissions)` +Set the permissions of {roleName}. This replaces the previous set of permissions on the role. + +Content: List of JSON Resource-Action objects, e.g.: + +``` +[ +{ + "resource": { + "name": "wiki.*", + "type": "DATASOURCE" + }, + "action": "READ" +}, +{ + "resource": { + "name": "wikiticker", + "type": "DATASOURCE" + }, + "action": "WRITE" +} +] +``` + +The "name" field for resources in the permission definitions are regexes used to match resource names during authorization checks. + +Please see [Defining permissions](#defining-permissions) for more details. + +##### Cache Load Status +`GET(/druid-ext/basic-security/authorization/loadStatus)` +Return the current load status of the local caches of the authorization Druid metadata store. + +## Default user accounts + +### Authenticator +If `druid.auth.authenticator..initialAdminPassword` is set, a default admin user named "admin" will be created, with the specified initial password. If this configuration is omitted, the "admin" user will not be created. + +If `druid.auth.authenticator..initialInternalClientPassword` is set, a default internal system user named "druid_system" will be created, with the specified initial password. If this configuration is omitted, the "druid_system" user will not be created. + + +### Authorizer + +Each Authorizer will always have a default "admin" and "druid_system" user with full privileges. + +## Defining permissions + +There are two action types in Druid: READ and WRITE + +There are three resource types in Druid: DATASOURCE, CONFIG, and STATE. + +### DATASOURCE +Resource names for this type are datasource names. Specifying a datasource permission allows the administrator to grant users access to specific datasources. + +### CONFIG +There are two possible resource names for the "CONFIG" resource type, "CONFIG" and "security". Granting a user access to CONFIG resources allows them to access the following endpoints. + +"CONFIG" resource name covers the following endpoints: + +|Endpoint|Process Type| +|--------|---------| +|`/druid/coordinator/v1/config`|coordinator| +|`/druid/indexer/v1/worker`|overlord| +|`/druid/indexer/v1/worker/history`|overlord| +|`/druid/worker/v1/disable`|middleManager| +|`/druid/worker/v1/enable`|middleManager| + +"security" resource name covers the following endpoint: + +|Endpoint|Process Type| +|--------|---------| +|`/druid-ext/basic-security/authentication`|coordinator| +|`/druid-ext/basic-security/authorization`|coordinator| + +### STATE +There is only one possible resource name for the "STATE" config resource type, "STATE". Granting a user access to STATE resources allows them to access the following endpoints. + +"STATE" resource name covers the following endpoints: + +|Endpoint|Process Type| +|--------|---------| +|`/druid/coordinator/v1`|coordinator| +|`/druid/coordinator/v1/rules`|coordinator| +|`/druid/coordinator/v1/rules/history`|coordinator| +|`/druid/coordinator/v1/servers`|coordinator| +|`/druid/coordinator/v1/tiers`|coordinator| +|`/druid/broker/v1`|broker| +|`/druid/v2/candidates`|broker| +|`/druid/indexer/v1/leader`|overlord| +|`/druid/indexer/v1/isLeader`|overlord| +|`/druid/indexer/v1/action`|overlord| +|`/druid/indexer/v1/workers`|overlord| +|`/druid/indexer/v1/scaling`|overlord| +|`/druid/worker/v1/enabled`|middleManager| +|`/druid/worker/v1/tasks`|middleManager| +|`/druid/worker/v1/task/{taskid}/shutdown`|middleManager| +|`/druid/worker/v1/task/{taskid}/log`|middleManager| +|`/druid/historical/v1`|historical| +|`/druid-internal/v1/segments/`|historical| +|`/druid-internal/v1/segments/`|peon| +|`/druid-internal/v1/segments/`|realtime| +|`/status`|all process types| + +### HTTP methods + +For information on what HTTP methods are supported on a particular request endpoint, please refer to the [API documentation](../../operations/api-reference.md). + +GET requires READ permission, while POST and DELETE require WRITE permission. + +### SQL Permissions + +Queries on Druid datasources require DATASOURCE READ permissions for the specified datasource. + +Queries on the [INFORMATION_SCHEMA tables](../../querying/sql.html#information-schema) will +return information about datasources that the caller has DATASOURCE READ access to. Other +datasources will be omitted. + +Queries on the [system schema tables](../../querying/sql.html#system-schema) require the following permissions: +- `segments`: Segments will be filtered based on DATASOURCE READ permissions. +- `servers`: The user requires STATE READ permissions. +- `server_segments`: The user requires STATE READ permissions and segments will be filtered based on DATASOURCE READ permissions. +- `tasks`: Tasks will be filtered based on DATASOURCE READ permissions. + +## Configuration Propagation + +To prevent excessive load on the Coordinator, the Authenticator and Authorizer user/role Druid metadata store state is cached on each Druid process. + +Each process will periodically poll the Coordinator for the latest Druid metadata store state, controlled by the `druid.auth.basic.common.pollingPeriod` and `druid.auth.basic.common.maxRandomDelay` properties. + +When a configuration update occurs, the Coordinator can optionally notify each process with the updated Druid metadata store state. This behavior is controlled by the `enableCacheNotifications` and `cacheNotificationTimeout` properties on Authenticators and Authorizers. + +Note that because of the caching, changes made to the user/role Druid metadata store may not be immediately reflected at each Druid process. diff --git a/development/extensions-core/druid-kerberos.md b/development/extensions-core/druid-kerberos.md new file mode 100644 index 0000000..7217f68 --- /dev/null +++ b/development/extensions-core/druid-kerberos.md @@ -0,0 +1,125 @@ +--- +id: druid-kerberos +title: "Kerberos" +--- + + + + +Apache Druid Extension to enable Authentication for Druid Processes using Kerberos. +This extension adds an Authenticator which is used to protect HTTP Endpoints using the simple and protected GSSAPI negotiation mechanism [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO). +Make sure to [include](../../development/extensions.md#loading-extensions) `druid-kerberos` as an extension. + + +## Configuration + +### Creating an Authenticator +``` +druid.auth.authenticatorChain=["MyKerberosAuthenticator"] + +druid.auth.authenticator.MyKerberosAuthenticator.type=kerberos +``` + +To use the Kerberos authenticator, add an authenticator with type `kerberos` to the authenticatorChain. The example above uses the name "MyKerberosAuthenticator" for the Authenticator. + +Configuration of the named authenticator is assigned through properties with the form: + +``` +druid.auth.authenticator.. +``` + +The configuration examples in the rest of this document will use "kerberos" as the name of the authenticator being configured. + +### Properties +|Property|Possible Values|Description|Default|required| +|--------|---------------|-----------|-------|--------| +|`druid.auth.authenticator.kerberos.serverPrincipal`|`HTTP/_HOST@EXAMPLE.COM`| SPNEGO service principal used by druid processes|empty|Yes| +|`druid.auth.authenticator.kerberos.serverKeytab`|`/etc/security/keytabs/spnego.service.keytab`|SPNego service keytab used by druid processes|empty|Yes| +|`druid.auth.authenticator.kerberos.authToLocal`|`RULE:[1:$1@$0](druid@EXAMPLE.COM)s/.*/druid DEFAULT`|It allows you to set a general rule for mapping principal names to local user names. It will be used if there is not an explicit mapping for the principal name that is being translated.|DEFAULT|No| +|`druid.auth.authenticator.kerberos.cookieSignatureSecret`|`secretString`| Secret used to sign authentication cookies. It is advisable to explicitly set it, if you have multiple druid nodes running on same machine with different ports as the Cookie Specification does not guarantee isolation by port.||No| +|`druid.auth.authenticator.kerberos.authorizerName`|Depends on available authorizers|Authorizer that requests should be directed to|Empty|Yes| + +As a note, it is required that the SPNego principal in use by the druid processes must start with HTTP (This specified by [RFC-4559](https://tools.ietf.org/html/rfc4559)) and must be of the form "HTTP/_HOST@REALM". +The special string _HOST will be replaced automatically with the value of config `druid.host` + +### `druid.auth.authenticator.kerberos.excludedPaths` + +In older releases, the Kerberos authenticator had an `excludedPaths` property that allowed the user to specify a list of paths where authentication checks should be skipped. This property has been removed from the Kerberos authenticator because the path exclusion functionality is now handled across all authenticators/authorizers by setting `druid.auth.unsecuredPaths`, as described in the [main auth documentation](../../design/auth.md). + +### Auth to Local Syntax +`druid.auth.authenticator.kerberos.authToLocal` allows you to set a general rules for mapping principal names to local user names. +The syntax for mapping rules is `RULE:\[n:string](regexp)s/pattern/replacement/g`. The integer n indicates how many components the target principal should have. If this matches, then a string will be formed from string, substituting the realm of the principal for $0 and the nth component of the principal for $n. e.g. if the principal was druid/admin then `\[2:$2$1suffix]` would result in the string `admindruidsuffix`. +If this string matches regexp, then the s//\[g] substitution command will be run over the string. The optional g will cause the substitution to be global over the string, instead of replacing only the first match in the string. +If required, multiple rules can be be joined by newline character and specified as a String. + +### Increasing HTTP Header size for large SPNEGO negotiate header +In Active Directory environment, SPNEGO token in the Authorization header includes PAC (Privilege Access Certificate) information, +which includes all security groups for the user. In some cases when the user belongs to many security groups the header to grow beyond what druid can handle by default. +In such cases, max request header size that druid can handle can be increased by setting `druid.server.http.maxRequestHeaderSize` (default 8Kb) and `druid.router.http.maxRequestBufferSize` (default 8Kb). + +## Configuring Kerberos Escalated Client + +Druid internal processes communicate with each other using an escalated http Client. A Kerberos enabled escalated HTTP Client can be configured by following properties - + + +|Property|Example Values|Description|Default|required| +|--------|---------------|-----------|-------|--------| +|`druid.escalator.type`|`kerberos`| Type of Escalator client used for internal process communication.|n/a|Yes| +|`druid.escalator.internalClientPrincipal`|`druid@EXAMPLE.COM`| Principal user name, used for internal process communication|n/a|Yes| +|`druid.escalator.internalClientKeytab`|`/etc/security/keytabs/druid.keytab`|Path to keytab file used for internal process communication|n/a|Yes| +|`druid.escalator.authorizerName`|`MyBasicAuthorizer`|Authorizer that requests should be directed to.|n/a|Yes| + +## Accessing Druid HTTP end points when kerberos security is enabled +1. To access druid HTTP endpoints via curl user will need to first login using `kinit` command as follows - + + ``` + kinit -k -t user@REALM.COM + ``` + +2. Once the login is successful verify that login is successful using `klist` command +3. Now you can access druid HTTP endpoints using curl command as follows - + + ``` + curl --negotiate -u:anyUser -b ~/cookies.txt -c ~/cookies.txt -X POST -H'Content-Type: application/json' + ``` + + e.g to send a query from file `query.json` to the Druid Broker use this command - + + ``` + curl --negotiate -u:anyUser -b ~/cookies.txt -c ~/cookies.txt -X POST -H'Content-Type: application/json' http://broker-host:port/druid/v2/?pretty -d @query.json + ``` + Note: Above command will authenticate the user first time using SPNego negotiate mechanism and store the authentication cookie in file. For subsequent requests the cookie will be used for authentication. + +## Accessing Coordinator or Overlord console from web browser +To access Coordinator/Overlord console from browser you will need to configure your browser for SPNego authentication as follows - + +1. Safari - No configurations required. +2. Firefox - Open firefox and follow these steps - + 1. Go to `about:config` and search for `network.negotiate-auth.trusted-uris`. + 2. Double-click and add the following values: `"http://druid-coordinator-hostname:ui-port"` and `"http://druid-overlord-hostname:port"` +3. Google Chrome - From the command line run following commands - + 1. `google-chrome --auth-server-whitelist="druid-coordinator-hostname" --auth-negotiate-delegate-whitelist="druid-coordinator-hostname"` + 2. `google-chrome --auth-server-whitelist="druid-overlord-hostname" --auth-negotiate-delegate-whitelist="druid-overlord-hostname"` +4. Internet Explorer - + 1. Configure trusted websites to include `"druid-coordinator-hostname"` and `"druid-overlord-hostname"` + 2. Allow negotiation for the UI website. + +## Sending Queries programmatically +Many HTTP client libraries, such as Apache Commons [HttpComponents](https://hc.apache.org/), already have support for performing SPNEGO authentication. You can use any of the available HTTP client library to communicate with druid cluster. diff --git a/development/extensions-core/druid-lookups.md b/development/extensions-core/druid-lookups.md new file mode 100644 index 0000000..acede46 --- /dev/null +++ b/development/extensions-core/druid-lookups.md @@ -0,0 +1,155 @@ +--- +id: druid-lookups +title: "Cached Lookup Module" +--- + + + + +> Please note that this is an experimental module and the development/testing still at early stage. Feel free to try it and give us your feedback. + +## Description +This Apache Druid module provides a per-lookup caching mechanism for JDBC data sources. +The main goal of this cache is to speed up the access to a high latency lookup sources and to provide a caching isolation for every lookup source. +Thus user can define various caching strategies or and implementation per lookup, even if the source is the same. +This module can be used side to side with other lookup module like the global cached lookup module. + +To use this extension please make sure to [include](../../development/extensions.md#loading-extensions) `druid-lookups-cached-single` as an extension. + +> If using JDBC, you will need to add your database's client JAR files to the extension's directory. +> For Postgres, the connector JAR is already included. +> For MySQL, you can get it from https://dev.mysql.com/downloads/connector/j/. +> Copy or symlink the downloaded file to `extensions/druid-lookups-cached-single` under the distribution root directory. + +## Architecture +Generally speaking this module can be divided into two main component, namely, the data fetcher layer and caching layer. + +### Data Fetcher layer + +First part is the data fetcher layer API `DataFetcher`, that exposes a set of fetch methods to fetch data from the actual Lookup dimension source. +For instance `JdbcDataFetcher` provides an implementation of `DataFetcher` that can be used to fetch key/value from a RDBMS via JDBC driver. +If you need new type of data fetcher, all you need to do, is to implement the interface `DataFetcher` and load it via another druid module. +### Caching layer + +This extension comes with two different caching strategies. First strategy is a poll based and the second is a load based. +#### Poll lookup cache + +The poll strategy cache strategy will fetch and swap all the pair of key/values periodically from the lookup source. +Hence, user should make sure that the cache can fit all the data. +The current implementation provides 2 type of poll cache, the first is on-heap (uses immutable map), while the second uses MapDB based off-heap map. +User can also implement a different lookup polling cache by implementing `PollingCacheFactory` and `PollingCache` interfaces. + +#### Loading lookup +Loading cache strategy will load the key/value pair upon request on the key it self, the general algorithm is load key if absent. +Once the key/value pair is loaded eviction will occur according to the cache eviction policy. +This module comes with two loading lookup implementation, the first is on-heap backed by a Guava cache implementation, the second is MapDB off-heap implementation. +Both implementations offer various eviction strategies. +Same for Loading cache, developer can implement a new type of loading cache by implementing `LookupLoadingCache` interface. + +## Configuration and Operation: + + +### Polling Lookup + +**Note that the current implementation of `offHeapPolling` and `onHeapPolling` will create two caches one to lookup value based on key and the other to reverse lookup the key from value** + +|Field|Type|Description|Required|default| +|-----|----|-----------|--------|-------| +|dataFetcher|JSON object|Specifies the lookup data fetcher type to use in order to fetch data|yes|null| +|cacheFactory|JSON Object|Cache factory implementation|no |onHeapPolling| +|pollPeriod|Period|polling period |no |null (poll once)| + + +##### Example of Polling On-heap Lookup +This example demonstrates a polling cache that will update its on-heap cache every 10 minutes + +```json +{ + "type":"pollingLookup", + "pollPeriod":"PT10M", + "dataFetcher":{ "type":"jdbcDataFetcher", "connectorConfig":"jdbc://mysql://localhost:3306/my_data_base", "table":"lookup_table_name", "keyColumn":"key_column_name", "valueColumn": "value_column_name"}, + "cacheFactory":{"type":"onHeapPolling"} +} + +``` + +##### Example Polling Off-heap Lookup +This example demonstrates an off-heap lookup that will be cached once and never swapped `(pollPeriod == null)` + +```json +{ + "type":"pollingLookup", + "dataFetcher":{ "type":"jdbcDataFetcher", "connectorConfig":"jdbc://mysql://localhost:3306/my_data_base", "table":"lookup_table_name", "keyColumn":"key_column_name", "valueColumn": "value_column_name"}, + "cacheFactory":{"type":"offHeapPolling"} +} + +``` + + +### Loading lookup + +|Field|Type|Description|Required|default| +|-----|----|-----------|--------|-------| +|dataFetcher|JSON object|Specifies the lookup data fetcher type to use in order to fetch data|yes|null| +|loadingCacheSpec|JSON Object|Lookup cache spec implementation|yes |null| +|reverseLoadingCacheSpec|JSON Object| Reverse lookup cache implementation|yes |null| + + +##### Example Loading On-heap Guava + +Guava cache configuration spec. + +|Field|Type|Description|Required|default| +|-----|----|-----------|--------|-------| +|concurrencyLevel|int|Allowed concurrency among update operations|no|4| +|initialCapacity|int|Initial capacity size|no |null| +|maximumSize|long| Specifies the maximum number of entries the cache may contain.|no |null (infinite capacity)| +|expireAfterAccess|long| Specifies the eviction time after last read in milliseconds.|no |null (No read-time-based eviction when set to null)| +|expireAfterWrite|long| Specifies the eviction time after last write in milliseconds.|no |null (No write-time-based eviction when set to null)| + +```json +{ + "type":"loadingLookup", + "dataFetcher":{ "type":"jdbcDataFetcher", "connectorConfig":"jdbc://mysql://localhost:3306/my_data_base", "table":"lookup_table_name", "keyColumn":"key_column_name", "valueColumn": "value_column_name"}, + "loadingCacheSpec":{"type":"guava"}, + "reverseLoadingCacheSpec":{"type":"guava", "maximumSize":500000, "expireAfterAccess":100000, "expireAfterAccess":10000} +} +``` + +##### Example Loading Off-heap MapDB + +Off heap cache is backed by [MapDB](http://www.mapdb.org/) implementation. MapDB is using direct memory as memory pool, please take that into account when limiting the JVM direct memory setup. + +|Field|Type|Description|Required|default| +|-----|----|-----------|--------|-------| +|maxStoreSize|double|maximal size of store in GB, if store is larger entries will start expiring|no |0| +|maxEntriesSize|long| Specifies the maximum number of entries the cache may contain.|no |0 (infinite capacity)| +|expireAfterAccess|long| Specifies the eviction time after last read in milliseconds.|no |0 (No read-time-based eviction when set to null)| +|expireAfterWrite|long| Specifies the eviction time after last write in milliseconds.|no |0 (No write-time-based eviction when set to null)| + + +```json +{ + "type":"loadingLookup", + "dataFetcher":{ "type":"jdbcDataFetcher", "connectorConfig":"jdbc://mysql://localhost:3306/my_data_base", "table":"lookup_table_name", "keyColumn":"key_column_name", "valueColumn": "value_column_name"}, + "loadingCacheSpec":{"type":"mapDb", "maxEntriesSize":100000}, + "reverseLoadingCacheSpec":{"type":"mapDb", "maxStoreSize":5, "expireAfterAccess":100000, "expireAfterAccess":10000} +} +``` diff --git a/development/extensions-core/druid-pac4j.md b/development/extensions-core/druid-pac4j.md new file mode 100644 index 0000000..3dea811 --- /dev/null +++ b/development/extensions-core/druid-pac4j.md @@ -0,0 +1,46 @@ +--- +id: druid-pac4j +title: "Druid pac4j based Security extension" +--- + + + + +Apache Druid Extension to enable [OpenID Connect](https://openid.net/connect/) based Authentication for Druid Processes using [pac4j](https://github.com/pac4j/pac4j) as the underlying client library. +This can be used with any authentication server that supports same e.g. [Okta](https://developer.okta.com/). +This extension should only be used at the router node to enable a group of users in existing authentication server to interact with Druid cluster, using the [Web Console](../../operations/druid-console.html). This extension does not support JDBC client authentication. + +## Configuration + +### Creating an Authenticator +``` +druid.auth.authenticatorChain=["pac4j"] +druid.auth.authenticator.pac4j.type=pac4j +``` + +### Properties +|Property|Description|Default|required| +|--------|---------------|-----------|-------| +|`druid.auth.pac4j.cookiePassphrase`|passphrase for encrypting the cookies used to manage authentication session with browser. It can be provided as plaintext string or The [Password Provider](../../operations/password-provider.md).|none|Yes| +|`druid.auth.pac4j.readTimeout`|Socket connect and read timeout duration used when communicating with authentication server|PT5S|No| +|`druid.auth.pac4j.enableCustomSslContext`|Whether to use custom SSLContext setup via [simple-client-sslcontext](simple-client-sslcontext.md) extension which must be added to extensions list when this property is set to true.|false|No| +|`druid.auth.pac4j.oidc.clientID`|OAuth Client Application id.|none|Yes| +|`druid.auth.pac4j.oidc.clientSecret`|OAuth Client Application secret. It can be provided as plaintext string or The [Password Provider](../../operations/password-provider.md).|none|Yes| +|`druid.auth.pac4j.oidc.discoveryURI`|discovery URI for fetching OP metadata [see this](http://openid.net/specs/openid-connect-discovery-1_0.html).|none|Yes| diff --git a/development/extensions-core/druid-ranger-security.md b/development/extensions-core/druid-ranger-security.md new file mode 100644 index 0000000..77f3eb9 --- /dev/null +++ b/development/extensions-core/druid-ranger-security.md @@ -0,0 +1,127 @@ +--- +id: druid-ranger-security +title: "Apache Ranger Security" +--- + + + +This Apache Druid extension adds an Authorizer which implements access control for Druid, backed by [Apache Ranger](https://ranger.apache.org/). Please see [Authentication and Authorization](../../design/auth.md) for more information on the basic facilities this extension provides. + +Make sure to [include](../../development/extensions.md#loading-extensions) `druid-ranger-security` as an extension. + +> The latest release of Apache Ranger is at the time of writing version 2.0. This version has a dependency on `log4j 1.2.17` which has a vulnerability if you configure it to use a `SocketServer` (CVE-2019-17571). Next to that, it also includes Kafka 2.0.0 which has 2 known vulnerabilities (CVE-2019-12399, CVE-2018-17196). Kafka can be used by the audit component in Ranger, but is not required. + +## Configuration + +Support for Apache Ranger authorization consists of three elements: +* configuring the extension in Apache Druid +* configuring the connection to Apache Ranger +* providing the service definition for Druid to Apache Ranger + +### Enabling the extension +Ensure that you have a valid authenticator chain and escalator set in your `common.runtime.properties`. For every authenticator your wish to use the authorizer for, set `druid.auth.authenticator..authorizerName` to the name you will give the authorizer, e.g. `ranger`. + +Then add the following and amend to your needs (in case you need to use multiple authorizers): + +``` +druid.auth.authorizers=["ranger"] +druid.auth.authorizer.ranger.type=ranger +``` + +The following is an example that showcases using `druid-basic-security` for authentication and `druid-ranger-security` for authorization. + +``` +druid.auth.authenticatorChain=["basic"] +druid.auth.authenticator.basic.type=basic +druid.auth.authenticator.basic.initialAdminPassword=password1 +druid.auth.authenticator.basic.initialInternalClientPassword=password2 +druid.auth.authenticator.basic.credentialsValidator.type=metadata +druid.auth.authenticator.basic.skipOnFailure=false +druid.auth.authenticator.basic.enableCacheNotifications=true +druid.auth.authenticator.basic.authorizerName=ranger + +druid.auth.authorizers=["ranger"] +druid.auth.authorizer.ranger.type=ranger + +# Escalator +druid.escalator.type=basic +druid.escalator.internalClientUsername=druid_system +druid.escalator.internalClientPassword=password2 +druid.escalator.authorizerName=ranger +``` + +> Contrary to the documentation of `druid-basic-auth` Ranger does not automatically provision a highly privileged system user, you will need to do this yourself. This system user in the case of `druid-basic-auth` is named `druid_system` and for the escalator it is configurable, as shown above. Make sure to take note of these user names and configure `READ` access to `state:STATE` and to `config:security` in your ranger policies, otherwise system services will not work properly. + +#### Properties to configure the extension in Apache Druid +|Property|Description|Default|required| +|--------|-----------|-------|--------| +|`druid.auth.ranger.keytab`|Defines the keytab to be used while authenticating against Apache Ranger to obtain policies and provide auditing|null|No| +|`druid.auth.ranger.principal`|Defines the principal to be used while authenticating against Apache Ranger to obtain policies and provide auditing|null|No| +|`druid.auth.ranger.use_ugi`|Determines if groups that the authenticated user belongs to should be obtained from Hadoop's `UserGroupInformation`|null|No| + +### Configuring the connection to Apache Ranger + +The Apache Ranger authorization extension will read several configuration files. Discussing the contents of those files is beyond the scope of this document. Depending on your needs you will need to create them. The minimum you will need to have is a `ranger-druid-security.xml` file that you will need to put in the classpath (e.g. `_common`). For auditing, the configuration is in `ranger-druid-audit.xml`. + +### Adding the service definition for Apache Druid to Apache Ranger + +At the time of writing of this document Apache Ranger (2.0) does not include an out of the box service and service definition for Druid. You can add the service definition to Apache Ranger by entering the following command: + +`curl -u : -d "@ranger-servicedef-druid.json" -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:6080/service/public/v2/api/servicedef/` + +You should get back `json` describing the service definition you just added. You can now go to the web interface of Apache Ranger which should now include a widget for "Druid". Click the plus sign and create the new service. Ensure your service name is equal to what you configured in `ranger-druid-security.xml`. + +#### Configuring Apache Ranger policies + +When installing a new Druid service in Apache Ranger for the first time, Ranger will provision the policies to allow the administrative user `read/write` access to all properties and data sources. You might want to limit this. Do not forget to add the correct policies for the `druid_system` user and the `internalClientUserName` of the escalator. + +> Loading new data sources requires `write` access to the `datasource` prior to the loading itself. So if you want to create a datasource `wikipedia` you are required to have an `allow` policy inside Apache Ranger before trying to load the spec. + +## Usage + +### HTTP methods + +For information on what HTTP methods are supported for a particular request endpoint, please refer to the [API documentation](../../operations/api-reference.md). + +GET requires READ permission, while POST and DELETE require WRITE permission. + +### SQL Permissions + +Queries on Druid datasources require DATASOURCE READ permissions for the specified datasource. + +Queries on the [INFORMATION_SCHEMA tables](../../querying/sql.html#information-schema) will return information about datasources that the caller has DATASOURCE READ access to. Other datasources will be omitted. + +Queries on the [system schema tables](../../querying/sql.html#system-schema) require the following permissions: +- `segments`: Segments will be filtered based on DATASOURCE READ permissions. +- `servers`: The user requires STATE READ permissions. +- `server_segments`: The user requires STATE READ permissions and segments will be filtered based on DATASOURCE READ permissions. +- `tasks`: Tasks will be filtered based on DATASOURCE READ permissions. + + +### Debugging + +If you face difficulty grasping why access is denied to certain elements, and the `audit` section in Apache Ranger does not give you any detail, you can enable debug logging for `org.apache.druid.security.ranger`. To do so add the following in your `log4j2.xml`: + +```xml + + + + +``` diff --git a/development/extensions-core/examples.md b/development/extensions-core/examples.md new file mode 100644 index 0000000..577ee30 --- /dev/null +++ b/development/extensions-core/examples.md @@ -0,0 +1,26 @@ +--- +id: examples +title: "Extension Examples" +--- + + + + +This extension was removed in Apache Druid 0.16.0. In prior versions, the extension provided obsolete facilities to ingest data from the Twitter 'Spritzer' data stream as well as the Wikipedia changes IRC channel. diff --git a/development/extensions-core/google.md b/development/extensions-core/google.md new file mode 100644 index 0000000..f3e9437 --- /dev/null +++ b/development/extensions-core/google.md @@ -0,0 +1,58 @@ +--- +id: google +title: "Google Cloud Storage" +--- + + + +## Google Cloud Storage Extension + +This extension allows you to do 2 things: +* [Ingest data](#reading-data-from-google-cloud-storage) from files stored in Google Cloud Storage. +* Write segments to [deep storage](#deep-storage) in GCS. + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-google-extensions` extension. + +### Required Configuration + +To configure connectivity to google cloud, run druid processes with `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_keyfile` in the environment. + +### Reading data from Google Cloud Storage + +The [Google Cloud Storage input source](../../ingestion/native-batch.md#google-cloud-storage-input-source) is supported by the [Parallel task](../../ingestion/native-batch.md#parallel-task) +to read objects directly from Google Cloud Storage. If you use the [Hadoop task](../../ingestion/hadoop.md), +you can read data from Google Cloud Storage by specifying the paths in your [`inputSpec`](../../ingestion/hadoop.md#inputspec). + +Objects can also be read directly from Google Cloud Storage via the [StaticGoogleBlobStoreFirehose](../../ingestion/native-batch.md#staticgoogleblobstorefirehose) + +### Deep Storage + +Deep storage can be written to Google Cloud Storage either via this extension or the [druid-hdfs-storage extension](../extensions-core/hdfs.md). + +#### Configuration + +To configure connectivity to google cloud, run druid processes with `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_keyfile` in the environment. + +|Property|Description|Possible Values|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|google||Must be set.| +|`druid.google.bucket`||Google Storage bucket name.|Must be set.| +|`druid.google.prefix`|A prefix string that will be prepended to the blob names for the segments published to Google deep storage| |""| +|`druid.google.maxListingLength`|maximum number of input files matching a given prefix to retrieve at a time| |1024| diff --git a/development/extensions-core/hdfs.md b/development/extensions-core/hdfs.md new file mode 100644 index 0000000..7114047 --- /dev/null +++ b/development/extensions-core/hdfs.md @@ -0,0 +1,169 @@ +--- +id: hdfs +title: "HDFS" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-hdfs-storage` as an extension and run druid processes with `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_keyfile` in the environment. + +## Deep Storage + +### Configuration for HDFS + +|Property|Possible Values|Description|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|hdfs||Must be set.| +|`druid.storage.storageDirectory`||Directory for storing segments.|Must be set.| +|`druid.hadoop.security.kerberos.principal`|`druid@EXAMPLE.COM`| Principal user name |empty| +|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path to keytab file|empty| + +Besides the above settings, you also need to include all Hadoop configuration files (such as `core-site.xml`, `hdfs-site.xml`) +in the Druid classpath. One way to do this is copying all those files under `${DRUID_HOME}/conf/_common`. + +If you are using the Hadoop ingestion, set your output directory to be a location on Hadoop and it will work. +If you want to eagerly authenticate against a secured hadoop/hdfs cluster you must set `druid.hadoop.security.kerberos.principal` and `druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job method that runs `kinit` command periodically. + +### Configuration for Cloud Storage + +You can also use the AWS S3 or the Google Cloud Storage as the deep storage via HDFS. + +#### Configuration for AWS S3 + +To use the AWS S3 as the deep storage, you need to configure `druid.storage.storageDirectory` properly. + +|Property|Possible Values|Description|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|hdfs| |Must be set.| +|`druid.storage.storageDirectory`|s3a://bucket/example/directory or s3n://bucket/example/directory|Path to the deep storage|Must be set.| + +You also need to include the [Hadoop AWS module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html), especially the `hadoop-aws.jar` in the Druid classpath. +Run the below command to install the `hadoop-aws.jar` file under `${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes. + +```bash +java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps -h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}"; +cp ${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar ${DRUID_HOME}/extensions/druid-hdfs-storage/ +``` + +Finally, you need to add the below properties in the `core-site.xml`. +For more configurations, see the [Hadoop AWS module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html). + +```xml + + fs.s3a.impl + org.apache.hadoop.fs.s3a.S3AFileSystem + The implementation class of the S3A Filesystem + + + + fs.AbstractFileSystem.s3a.impl + org.apache.hadoop.fs.s3a.S3A + The implementation class of the S3A AbstractFileSystem. + + + + fs.s3a.access.key + AWS access key ID. Omit for IAM role-based or provider-based authentication. + your access key + + + + fs.s3a.secret.key + AWS secret key. Omit for IAM role-based or provider-based authentication. + your secret key + +``` + +#### Configuration for Google Cloud Storage + +To use the Google Cloud Storage as the deep storage, you need to configure `druid.storage.storageDirectory` properly. + +|Property|Possible Values|Description|Default| +|--------|---------------|-----------|-------| +|`druid.storage.type`|hdfs||Must be set.| +|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the deep storage|Must be set.| + +All services that need to access GCS need to have the [GCS connector jar](https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage#other_sparkhadoop_clusters) in their class path. +Please read the [install instructions](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md) +to properly set up the necessary libraries and configurations. +One option is to place this jar in `${DRUID_HOME}/lib/` and `${DRUID_HOME}/extensions/druid-hdfs-storage/`. + +Finally, you need to configure the `core-site.xml` file with the filesystem +and authentication properties needed for GCS. You may want to copy the below +example properties. Please follow the instructions at +[https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md) +for more details. +For more configurations, [GCS core default](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/conf/gcs-core-default.xml) +and [GCS core template](https://github.com/GoogleCloudPlatform/bdutil/blob/master/conf/hadoop2/gcs-core-template.xml). + +```xml + + fs.gs.impl + com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem + The FileSystem for gs: (GCS) uris. + + + + fs.AbstractFileSystem.gs.impl + com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS + The AbstractFileSystem for gs: uris. + + + + google.cloud.auth.service.account.enable + true + + Whether to use a service account for GCS authorization. + Setting this property to `false` will disable use of service accounts for + authentication. + + + + + google.cloud.auth.service.account.json.keyfile + /path/to/keyfile + + The JSON key file of the service account used for GCS + access when google.cloud.auth.service.account.enable is true. + + +``` + +Tested with Druid 0.17.0, Hadoop 2.8.5 and gcs-connector jar 2.0.0-hadoop2. + +## Reading data from HDFS or Cloud Storage + +### Native batch ingestion + +The [HDFS input source](../../ingestion/native-batch.md#hdfs-input-source) is supported by the [Parallel task](../../ingestion/native-batch.md#parallel-task) +to read files directly from the HDFS Storage. You may be able to read objects from cloud storage +with the HDFS input source, but we highly recommend to use a proper +[Input Source](../../ingestion/native-batch.md#input-sources) instead if possible because +it is simple to set up. For now, only the [S3 input source](../../ingestion/native-batch.md#s3-input-source) +and the [Google Cloud Storage input source](../../ingestion/native-batch.md#google-cloud-storage-input-source) +are supported for cloud storage types, and so you may still want to use the HDFS input source +to read from cloud storage other than those two. + +### Hadoop-based ingestion + +If you use the [Hadoop ingestion](../../ingestion/hadoop.md), you can read data from HDFS +by specifying the paths in your [`inputSpec`](../../ingestion/hadoop.md#inputspec). +See the [Static](../../ingestion/hadoop.md#static) inputSpec for details. diff --git a/development/extensions-core/kafka-extraction-namespace.md b/development/extensions-core/kafka-extraction-namespace.md new file mode 100644 index 0000000..b72472a --- /dev/null +++ b/development/extensions-core/kafka-extraction-namespace.md @@ -0,0 +1,66 @@ +--- +id: kafka-extraction-namespace +title: "Apache Kafka Lookups" +--- + + + +> Lookups are an [experimental](../experimental.md) feature. + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-lookups-cached-global` and `druid-kafka-extraction-namespace` as an extension. + +If you need updates to populate as promptly as possible, it is possible to plug into a Kafka topic whose key is the old value and message is the desired new value (both in UTF-8) as a LookupExtractorFactory. + +```json +{ + "type":"kafka", + "kafkaTopic":"testTopic", + "kafkaProperties":{"zookeeper.connect":"somehost:2181/kafka"} +} +``` + +|Parameter|Description|Required|Default| +|---------|-----------|--------|-------| +|`kafkaTopic`|The Kafka topic to read the data from|Yes|| +|`kafkaProperties`|Kafka consumer properties. At least"zookeeper.connect" must be specified. Only the zookeeper connector is supported|Yes|| +|`connectTimeout`|How long to wait for an initial connection|No|`0` (do not wait)| +|`isOneToOne`|The map is a one-to-one (see [Lookup DimensionSpecs](../../querying/dimensionspecs.md))|No|`false`| + +The extension `kafka-extraction-namespace` enables reading from a Kafka feed which has name/key pairs to allow renaming of dimension values. An example use case would be to rename an ID to a human readable format. + +The consumer properties `group.id` and `auto.offset.reset` CANNOT be set in `kafkaProperties` as they are set by the extension as `UUID.randomUUID().toString()` and `smallest` respectively. + +See [lookups](../../querying/lookups.md) for how to configure and use lookups. + +## Limitations + +Currently the Kafka lookup extractor feeds the entire Kafka stream into a local cache. If you are using on-heap caching, this can easily clobber your java heap if the Kafka stream spews a lot of unique keys. +off-heap caching should alleviate these concerns, but there is still a limit to the quantity of data that can be stored. +There is currently no eviction policy. + +## Testing the Kafka rename functionality + +To test this setup, you can send key/value pairs to a Kafka stream via the following producer console: + +``` +./bin/kafka-console-producer.sh --property parse.key=true --property key.separator="->" --broker-list localhost:9092 --topic testTopic +``` + +Renames can then be published as `OLD_VAL->NEW_VAL` followed by newline (enter or return) diff --git a/development/extensions-core/kafka-ingestion.md b/development/extensions-core/kafka-ingestion.md new file mode 100644 index 0000000..3eb3499 --- /dev/null +++ b/development/extensions-core/kafka-ingestion.md @@ -0,0 +1,417 @@ +--- +id: kafka-ingestion +title: "Apache Kafka ingestion" +sidebar_label: "Apache Kafka" +--- + + + + +The Kafka indexing service enables the configuration of *supervisors* on the Overlord, which facilitate ingestion from +Kafka by managing the creation and lifetime of Kafka indexing tasks. These indexing tasks read events using Kafka's own +partition and offset mechanism and are therefore able to provide guarantees of exactly-once ingestion. +The supervisor oversees the state of the indexing tasks to coordinate handoffs, +manage failures, and ensure that the scalability and replication requirements are maintained. + +This service is provided in the `druid-kafka-indexing-service` core Apache Druid extension (see +[Including Extensions](../../development/extensions.md#loading-extensions)). + +> The Kafka indexing service supports transactional topics which were introduced in Kafka 0.11.x. These changes make the +> Kafka consumer that Druid uses incompatible with older brokers. Ensure that your Kafka brokers are version 0.11.x or +> better before using this functionality. Refer [Kafka upgrade guide](https://kafka.apache.org/documentation/#upgrade) +> if you are using older version of Kafka brokers. + +## Tutorial + +This page contains reference documentation for Apache Kafka-based ingestion. +For a walk-through instead, check out the [Loading from Apache Kafka](../../tutorials/tutorial-kafka.md) tutorial. + +## Submitting a Supervisor Spec + +The Kafka indexing service requires that the `druid-kafka-indexing-service` extension be loaded on both the Overlord and the +MiddleManagers. A supervisor for a dataSource is started by submitting a supervisor spec via HTTP POST to +`http://:/druid/indexer/v1/supervisor`, for example: + +``` +curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8090/druid/indexer/v1/supervisor +``` + +A sample supervisor spec is shown below: + +```json +{ + "type": "kafka", + "dataSchema": { + "dataSource": "metrics-kafka", + "timestampSpec": { + "column": "timestamp", + "format": "auto" + }, + "dimensionsSpec": { + "dimensions": [], + "dimensionExclusions": [ + "timestamp", + "value" + ] + }, + "metricsSpec": [ + { + "name": "count", + "type": "count" + }, + { + "name": "value_sum", + "fieldName": "value", + "type": "doubleSum" + }, + { + "name": "value_min", + "fieldName": "value", + "type": "doubleMin" + }, + { + "name": "value_max", + "fieldName": "value", + "type": "doubleMax" + } + ], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": "NONE" + } + }, + "ioConfig": { + "topic": "metrics", + "inputFormat": { + "type": "json" + }, + "consumerProperties": { + "bootstrap.servers": "localhost:9092" + }, + "taskCount": 1, + "replicas": 1, + "taskDuration": "PT1H" + }, + "tuningConfig": { + "type": "kafka", + "maxRowsPerSegment": 5000000 + } +} +``` + +## Supervisor Configuration + +|Field|Description|Required| +|--------|-----------|---------| +|`type`|The supervisor type, this should always be `kafka`.|yes| +|`dataSchema`|The schema that will be used by the Kafka indexing task during ingestion. See [`dataSchema`](../../ingestion/index.md#dataschema) for details.|yes| +|`ioConfig`|A KafkaSupervisorIOConfig object for configuring Kafka connection and I/O-related settings for the supervisor and indexing task. See [KafkaSupervisorIOConfig](#kafkasupervisorioconfig) below.|yes| +|`tuningConfig`|A KafkaSupervisorTuningConfig object for configuring performance-related settings for the supervisor and indexing tasks. See [KafkaSupervisorTuningConfig](#kafkasupervisortuningconfig) below.|no| + +### KafkaSupervisorIOConfig + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`topic`|String|The Kafka topic to read from. This must be a specific topic as topic patterns are not supported.|yes| +|`inputFormat`|Object|[`inputFormat`](../../ingestion/data-formats.md#input-format) to specify how to parse input data. See [the below section](#specifying-data-format) for details about specifying the input format.|yes| +|`consumerProperties`|Map|A map of properties to be passed to the Kafka consumer. This must contain a property `bootstrap.servers` with a list of Kafka brokers in the form: `:,:,...`. For SSL connections, the `keystore`, `truststore` and `key` passwords can be provided as a [Password Provider](../../operations/password-provider.md) or String password.|yes| +|`pollTimeout`|Long|The length of time to wait for the Kafka consumer to poll records, in milliseconds|no (default == 100)| +|`replicas`|Integer|The number of replica sets, where 1 means a single set of tasks (no replication). Replica tasks will always be assigned to different workers to provide resiliency against process failure.|no (default == 1)| +|`taskCount`|Integer|The maximum number of *reading* tasks in a *replica set*. This means that the maximum number of reading tasks will be `taskCount * replicas` and the total number of tasks (*reading* + *publishing*) will be higher than this. See [Capacity Planning](#capacity-planning) below for more details. The number of reading tasks will be less than `taskCount` if `taskCount > {numKafkaPartitions}`.|no (default == 1)| +|`taskDuration`|ISO8601 Period|The length of time before tasks stop reading and begin publishing their segment.|no (default == PT1H)| +|`startDelay`|ISO8601 Period|The period to wait before the supervisor starts managing tasks.|no (default == PT5S)| +|`period`|ISO8601 Period|How often the supervisor will execute its management logic. Note that the supervisor will also run in response to certain events (such as tasks succeeding, failing, and reaching their taskDuration) so this value specifies the maximum time between iterations.|no (default == PT30S)| +|`useEarliestOffset`|Boolean|If a supervisor is managing a dataSource for the first time, it will obtain a set of starting offsets from Kafka. This flag determines whether it retrieves the earliest or latest offsets in Kafka. Under normal circumstances, subsequent tasks will start from where the previous segments ended so this flag will only be used on first run.|no (default == false)| +|`completionTimeout`|ISO8601 Period|The length of time to wait before declaring a publishing task as failed and terminating it. If this is set too low, your tasks may never publish. The publishing clock for a task begins roughly after `taskDuration` elapses.|no (default == PT30M)| +|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to reject messages with timestamps earlier than this date time; for example if this is set to `2016-01-01T11:00Z` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| +|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). Please note that only one of `lateMessageRejectionPeriod` or `lateMessageRejectionStartDateTime` can be specified.|no (default == none)| +|`earlyMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps later than this period after the task reached its taskDuration; for example if this is set to `PT1H`, the taskDuration is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps later than *2016-01-01T14:00Z* will be dropped. **Note:** Tasks sometimes run past their task duration, for example, in cases of supervisor failover. Setting earlyMessageRejectionPeriod too low may cause messages to be dropped unexpectedly whenever a task runs past its originally configured task duration.|no (default == none)| + +#### Specifying data format + +Kafka indexing service supports both [`inputFormat`](../../ingestion/data-formats.md#input-format) and [`parser`](../../ingestion/data-formats.md#parser) to specify the data format. +The `inputFormat` is a new and recommended way to specify the data format for Kafka indexing service, +but unfortunately, it doesn't support all data formats supported by the legacy `parser`. +(They will be supported in the future.) + +The supported `inputFormat`s include [`csv`](../../ingestion/data-formats.md#csv), +[`delimited`](../../ingestion/data-formats.md#tsv-delimited), and [`json`](../../ingestion/data-formats.md#json). +You can also read [`avro_stream`](../../ingestion/data-formats.md#avro-stream-parser), +[`protobuf`](../../ingestion/data-formats.md#protobuf-parser), +and [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. + + + +### KafkaSupervisorTuningConfig + +The tuningConfig is optional and default parameters will be used if no tuningConfig is specified. + +| Field | Type | Description | Required | +|-----------------------------------|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------| +| `type` | String | The indexing task type, this should always be `kafka`. | yes | +| `maxRowsInMemory` | Integer | The number of rows to aggregate before persisting. This number is the post-aggregation rows, so it is not equivalent to the number of input events, but the number of aggregated rows that those events result in. This is used to manage the required JVM heap size. Maximum heap memory usage for indexing scales with maxRowsInMemory * (2 + maxPendingPersists). Normally user does not need to set this, but depending on the nature of data, if rows are short in terms of bytes, user may not want to store a million rows in memory and this value should be set. | no (default == 1000000) | +| `maxBytesInMemory` | Long | The number of bytes to aggregate in heap memory before persisting. This is based on a rough estimate of memory usage and not actual usage. Normally this is computed internally and user does not need to set it. The maximum heap memory usage for indexing is maxBytesInMemory * (2 + maxPendingPersists). | no (default == One-sixth of max JVM memory) | +| `maxRowsPerSegment` | Integer | The number of rows to aggregate into a segment; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == 5000000) | +| `maxTotalRows` | Long | The number of rows to aggregate across all segments; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == unlimited) | +| `intermediatePersistPeriod` | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M) | +| `maxPendingPersists` | Integer | Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with maxRowsInMemory * (2 + maxPendingPersists). | no (default == 0, meaning one persist can be running concurrently with ingestion, and none can be queued up) | +| `indexSpec` | Object | Tune how data is indexed. See [IndexSpec](#indexspec) for more information. | no | +| `indexSpecForIntermediatePersists`| | Defines segment storage format options to be used at indexing time for intermediate persisted temporary segments. This can be used to disable dimension/metric compression on intermediate segments to reduce memory required for final merging. However, disabling compression on intermediate segments might increase page cache use while they are used before getting merged into final segment published, see [IndexSpec](#indexspec) for possible values. | no (default = same as indexSpec) | +| `reportParseExceptions` | Boolean | *DEPRECATED*. If true, exceptions encountered during parsing will be thrown and will halt ingestion; if false, unparseable rows and fields will be skipped. Setting `reportParseExceptions` to true will override existing configurations for `maxParseExceptions` and `maxSavedParseExceptions`, setting `maxParseExceptions` to 0 and limiting `maxSavedParseExceptions` to no more than 1. | no (default == false) | +| `handoffConditionTimeout` | Long | Milliseconds to wait for segment handoff. It must be >= 0, where 0 means to wait forever. | no (default == 0) | +| `resetOffsetAutomatically` | Boolean | Controls behavior when Druid needs to read Kafka messages that are no longer available (i.e. when OffsetOutOfRangeException is encountered).

If false, the exception will bubble up, which will cause your tasks to fail and ingestion to halt. If this occurs, manual intervention is required to correct the situation; potentially using the [Reset Supervisor API](../../operations/api-reference.html#supervisors). This mode is useful for production, since it will make you aware of issues with ingestion.

If true, Druid will automatically reset to the earlier or latest offset available in Kafka, based on the value of the `useEarliestOffset` property (earliest if true, latest if false). Please note that this can lead to data being _DROPPED_ (if `useEarliestOffset` is false) or _DUPLICATED_ (if `useEarliestOffset` is true) without your knowledge. Messages will be logged indicating that a reset has occurred, but ingestion will continue. This mode is useful for non-production situations, since it will make Druid attempt to recover from problems automatically, even if they lead to quiet dropping or duplicating of data.

This feature behaves similarly to the Kafka `auto.offset.reset` consumer property. | no (default == false) | +| `workerThreads` | Integer | The number of threads that the supervisor uses to handle requests/responses for worker tasks, along with any other internal asynchronous operation. | no (default == min(10, taskCount)) | +| `chatThreads` | Integer | The number of threads that will be used for communicating with indexing tasks. | no (default == min(10, taskCount * replicas)) | +| `chatRetries` | Integer | The number of times HTTP requests to indexing tasks will be retried before considering tasks unresponsive. | no (default == 8) | +| `httpTimeout` | ISO8601 Period | How long to wait for a HTTP response from an indexing task. | no (default == PT10S) | +| `shutdownTimeout` | ISO8601 Period | How long to wait for the supervisor to attempt a graceful shutdown of tasks before exiting. | no (default == PT80S) | +| `offsetFetchPeriod` | ISO8601 Period | How often the supervisor queries Kafka and the indexing tasks to fetch current offsets and calculate lag. | no (default == PT30S, min == PT5S) | +| `segmentWriteOutMediumFactory` | Object | Segment write-out medium to use when creating segments. See below for more information. | no (not specified by default, the value from `druid.peon.defaultSegmentWriteOutMediumFactory.type` is used) | +| `intermediateHandoffPeriod` | ISO8601 Period | How often the tasks should hand off segments. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == P2147483647D) | +| `logParseExceptions` | Boolean | If true, log an error message when a parsing exception occurs, containing information about the row where the error occurred. | no, default == false | +| `maxParseExceptions` | Integer | The maximum number of parse exceptions that can occur before the task halts ingestion and fails. Overridden if `reportParseExceptions` is set. | no, unlimited default | +| `maxSavedParseExceptions` | Integer | When a parse exception occurs, Druid can keep track of the most recent parse exceptions. "maxSavedParseExceptions" limits how many exception instances will be saved. These saved exceptions will be made available after the task finishes in the [task completion report](../../ingestion/tasks.md#reports). Overridden if `reportParseExceptions` is set. | no, default == 0 | + +#### IndexSpec + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|bitmap|Object|Compression format for bitmap indexes. Should be a JSON object. See [Bitmap types](#bitmap-types) below for options.|no (defaults to Roaring)| +|dimensionCompression|String|Compression format for dimension columns. Choose from `LZ4`, `LZF`, or `uncompressed`.|no (default == `LZ4`)| +|metricCompression|String|Compression format for primitive type metric columns. Choose from `LZ4`, `LZF`, `uncompressed`, or `none`.|no (default == `LZ4`)| +|longEncoding|String|Encoding format for metric and dimension columns with type long. Choose from `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as is with 8 bytes each.|no (default == `longs`)| + +##### Bitmap types + +For Roaring bitmaps: + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|Must be `roaring`.|yes| +|`compressRunOnSerialization`|Boolean|Use a run-length encoding where it is estimated as more space efficient.|no (default == `true`)| + +For Concise bitmaps: + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|Must be `concise`.|yes| + +#### SegmentWriteOutMediumFactory + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|See [Additional Peon Configuration: SegmentWriteOutMediumFactory](../../configuration/index.html#segmentwriteoutmediumfactory) for explanation and available options.|yes| + +## Operations + +This section gives descriptions of how some supervisor APIs work specifically in Kafka Indexing Service. +For all supervisor APIs, please check [Supervisor APIs](../../operations/api-reference.html#supervisors). + +### Getting Supervisor Status Report + +`GET /druid/indexer/v1/supervisor//status` returns a snapshot report of the current state of the tasks managed by the given supervisor. This includes the latest +offsets as reported by Kafka, the consumer lag per partition, as well as the aggregate lag of all partitions. The +consumer lag per partition may be reported as negative values if the supervisor has not received a recent latest offset +response from Kafka. The aggregate lag value will always be >= 0. + +The status report also contains the supervisor's state and a list of recently thrown exceptions (reported as +`recentErrors`, whose max size can be controlled using the `druid.supervisor.maxStoredExceptionEvents` configuration). +There are two fields related to the supervisor's state - `state` and `detailedState`. The `state` field will always be +one of a small number of generic states that are applicable to any type of supervisor, while the `detailedState` field +will contain a more descriptive, implementation-specific state that may provide more insight into the supervisor's +activities than the generic `state` field. + +The list of possible `state` values are: [`PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`, `UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`] + +The list of `detailedState` values and their corresponding `state` mapping is as follows: + +|Detailed State|Corresponding State|Description| +|--------------|-------------------|-----------| +|UNHEALTHY_SUPERVISOR|UNHEALTHY_SUPERVISOR|The supervisor has encountered errors on the past `druid.supervisor.unhealthinessThreshold` iterations| +|UNHEALTHY_TASKS|UNHEALTHY_TASKS|The last `druid.supervisor.taskUnhealthinessThreshold` tasks have all failed| +|UNABLE_TO_CONNECT_TO_STREAM|UNHEALTHY_SUPERVISOR|The supervisor is encountering connectivity issues with Kafka and has not successfully connected in the past| +|LOST_CONTACT_WITH_STREAM|UNHEALTHY_SUPERVISOR|The supervisor is encountering connectivity issues with Kafka but has successfully connected in the past| +|PENDING (first iteration only)|PENDING|The supervisor has been initialized and hasn't started connecting to the stream| +|CONNECTING_TO_STREAM (first iteration only)|RUNNING|The supervisor is trying to connect to the stream and update partition data| +|DISCOVERING_INITIAL_TASKS (first iteration only)|RUNNING|The supervisor is discovering already-running tasks| +|CREATING_TASKS (first iteration only)|RUNNING|The supervisor is creating tasks and discovering state| +|RUNNING|RUNNING|The supervisor has started tasks and is waiting for taskDuration to elapse| +|SUSPENDED|SUSPENDED|The supervisor has been suspended| +|STOPPING|STOPPING|The supervisor is stopping| + +On each iteration of the supervisor's run loop, the supervisor completes the following tasks in sequence: + 1) Fetch the list of partitions from Kafka and determine the starting offset for each partition (either based on the + last processed offset if continuing, or starting from the beginning or ending of the stream if this is a new topic). + 2) Discover any running indexing tasks that are writing to the supervisor's datasource and adopt them if they match + the supervisor's configuration, else signal them to stop. + 3) Send a status request to each supervised task to update our view of the state of the tasks under our supervision. + 4) Handle tasks that have exceeded `taskDuration` and should transition from the reading to publishing state. + 5) Handle tasks that have finished publishing and signal redundant replica tasks to stop. + 6) Handle tasks that have failed and clean up the supervisor's internal state. + 7) Compare the list of healthy tasks to the requested `taskCount` and `replicas` configurations and create additional tasks if required. + +The `detailedState` field will show additional values (those marked with "first iteration only") the first time the +supervisor executes this run loop after startup or after resuming from a suspension. This is intended to surface +initialization-type issues, where the supervisor is unable to reach a stable state (perhaps because it can't connect to +Kafka, it can't read from the Kafka topic, or it can't communicate with existing tasks). Once the supervisor is stable - +that is, once it has completed a full execution without encountering any issues - `detailedState` will show a `RUNNING` +state until it is stopped, suspended, or hits a failure threshold and transitions to an unhealthy state. + +### Getting Supervisor Ingestion Stats Report + +`GET /druid/indexer/v1/supervisor//stats` returns a snapshot of the current ingestion row counters for each task being managed by the supervisor, along with moving averages for the row counters. + +See [Task Reports: Row Stats](../../ingestion/tasks.md#row-stats) for more information. + +### Supervisor Health Check + +`GET /druid/indexer/v1/supervisor//health` returns `200 OK` if the supervisor is healthy and +`503 Service Unavailable` if it is unhealthy. Healthiness is determined by the supervisor's `state` (as returned by the +`/status` endpoint) and the `druid.supervisor.*` Overlord configuration thresholds. + +### Updating Existing Supervisors + +`POST /druid/indexer/v1/supervisor` can be used to update existing supervisor spec. +Calling this endpoint when there is already an existing supervisor for the same dataSource will cause: + +- The running supervisor to signal its managed tasks to stop reading and begin publishing. +- The running supervisor to exit. +- A new supervisor to be created using the configuration provided in the request body. This supervisor will retain the +existing publishing tasks and will create new tasks starting at the offsets the publishing tasks ended on. + +Seamless schema migrations can thus be achieved by simply submitting the new schema using this endpoint. + +### Suspending and Resuming Supervisors + +You can suspend and resume a supervisor using `POST /druid/indexer/v1/supervisor//suspend` and `POST /druid/indexer/v1/supervisor//resume`, respectively. + +Note that the supervisor itself will still be operating and emitting logs and metrics, +it will just ensure that no indexing tasks are running until the supervisor is resumed. + +### Resetting Supervisors + +The `POST /druid/indexer/v1/supervisor//reset` operation clears stored +offsets, causing the supervisor to start reading offsets from either the earliest or latest +offsets in Kafka (depending on the value of `useEarliestOffset`). After clearing stored +offsets, the supervisor kills and recreates any active tasks, so that tasks begin reading +from valid offsets. + +Use care when using this operation! Resetting the supervisor may cause Kafka messages +to be skipped or read twice, resulting in missing or duplicate data. + +The reason for using this operation is to recover from a state in which the supervisor +ceases operating due to missing offsets. The indexing service keeps track of the latest +persisted Kafka offsets in order to provide exactly-once ingestion guarantees across +tasks. Subsequent tasks must start reading from where the previous task completed in +order for the generated segments to be accepted. If the messages at the expected +starting offsets are no longer available in Kafka (typically because the message retention +period has elapsed or the topic was removed and re-created) the supervisor will refuse +to start and in flight tasks will fail. This operation enables you to recover from this condition. + +Note that the supervisor must be running for this endpoint to be available. + +### Terminating Supervisors + +The `POST /druid/indexer/v1/supervisor//terminate` operation terminates a supervisor and causes all +associated indexing tasks managed by this supervisor to immediately stop and begin +publishing their segments. This supervisor will still exist in the metadata store and it's history may be retrieved +with the supervisor history API, but will not be listed in the 'get supervisors' API response nor can it's configuration +or status report be retrieved. The only way this supervisor can start again is by submitting a functioning supervisor +spec to the create API. + +### Capacity Planning + +Kafka indexing tasks run on MiddleManagers and are thus limited by the resources available in the MiddleManager +cluster. In particular, you should make sure that you have sufficient worker capacity (configured using the +`druid.worker.capacity` property) to handle the configuration in the supervisor spec. Note that worker capacity is +shared across all types of indexing tasks, so you should plan your worker capacity to handle your total indexing load +(e.g. batch processing, realtime tasks, merging tasks, etc.). If your workers run out of capacity, Kafka indexing tasks +will queue and wait for the next available worker. This may cause queries to return partial results but will not result +in data loss (assuming the tasks run before Kafka purges those offsets). + +A running task will normally be in one of two states: *reading* or *publishing*. A task will remain in reading state for +`taskDuration`, at which point it will transition to publishing state. A task will remain in publishing state for as long +as it takes to generate segments, push segments to deep storage, and have them be loaded and served by a Historical process +(or until `completionTimeout` elapses). + +The number of reading tasks is controlled by `replicas` and `taskCount`. In general, there will be `replicas * taskCount` +reading tasks, the exception being if taskCount > {numKafkaPartitions} in which case {numKafkaPartitions} tasks will +be used instead. When `taskDuration` elapses, these tasks will transition to publishing state and `replicas * taskCount` +new reading tasks will be created. Therefore to allow for reading tasks and publishing tasks to run concurrently, there +should be a minimum capacity of: + +``` +workerCapacity = 2 * replicas * taskCount +``` + +This value is for the ideal situation in which there is at most one set of tasks publishing while another set is reading. +In some circumstances, it is possible to have multiple sets of tasks publishing simultaneously. This would happen if the +time-to-publish (generate segment, push to deep storage, loaded on Historical) > `taskDuration`. This is a valid +scenario (correctness-wise) but requires additional worker capacity to support. In general, it is a good idea to have +`taskDuration` be large enough that the previous set of tasks finishes publishing before the current set begins. + +### Supervisor Persistence + +When a supervisor spec is submitted via the `POST /druid/indexer/v1/supervisor` endpoint, it is persisted in the +configured metadata database. There can only be a single supervisor per dataSource, and submitting a second spec for +the same dataSource will overwrite the previous one. + +When an Overlord gains leadership, either by being started or as a result of another Overlord failing, it will spawn +a supervisor for each supervisor spec in the metadata database. The supervisor will then discover running Kafka indexing +tasks and will attempt to adopt them if they are compatible with the supervisor's configuration. If they are not +compatible because they have a different ingestion spec or partition allocation, the tasks will be killed and the +supervisor will create a new set of tasks. In this way, the supervisors are persistent across Overlord restarts and +fail-overs. + +A supervisor is stopped via the `POST /druid/indexer/v1/supervisor//terminate` endpoint. This places a +tombstone marker in the database (to prevent the supervisor from being reloaded on a restart) and then gracefully +shuts down the currently running supervisor. When a supervisor is shut down in this way, it will instruct its +managed tasks to stop reading and begin publishing their segments immediately. The call to the shutdown endpoint will +return after all tasks have been signaled to stop but before the tasks finish publishing their segments. + +### Schema/Configuration Changes + +Schema and configuration changes are handled by submitting the new supervisor spec via the same +`POST /druid/indexer/v1/supervisor` endpoint used to initially create the supervisor. The Overlord will initiate a +graceful shutdown of the existing supervisor which will cause the tasks being managed by that supervisor to stop reading +and begin publishing their segments. A new supervisor will then be started which will create a new set of tasks that +will start reading from the offsets where the previous now-publishing tasks left off, but using the updated schema. +In this way, configuration changes can be applied without requiring any pause in ingestion. + +### Deployment Notes + +#### On the Subject of Segments + +Each Kafka Indexing Task puts events consumed from Kafka partitions assigned to it in a single segment for each segment +granular interval until maxRowsPerSegment, maxTotalRows or intermediateHandoffPeriod limit is reached, at this point a new partition +for this segment granularity is created for further events. Kafka Indexing Task also does incremental hand-offs which +means that all the segments created by a task will not be held up till the task duration is over. As soon as maxRowsPerSegment, +maxTotalRows or intermediateHandoffPeriod limit is hit, all the segments held by the task at that point in time will be handed-off +and new set of segments will be created for further events. This means that the task can run for longer durations of time +without accumulating old segments locally on Middle Manager processes and it is encouraged to do so. + +Kafka Indexing Service may still produce some small segments. Lets say the task duration is 4 hours, segment granularity +is set to an HOUR and Supervisor was started at 9:10 then after 4 hours at 13:10, new set of tasks will be started and +events for the interval 13:00 - 14:00 may be split across previous and new set of tasks. If you see it becoming a problem then +one can schedule re-indexing tasks be run to merge segments together into new segments of an ideal size (in the range of ~500-700 MB per segment). +Details on how to optimize the segment size can be found on [Segment size optimization](../../operations/segment-optimization.md). +There is also ongoing work to support automatic segment compaction of sharded segments as well as compaction not requiring +Hadoop (see [here](https://github.com/apache/druid/pull/5102)). diff --git a/development/extensions-core/kinesis-ingestion.md b/development/extensions-core/kinesis-ingestion.md new file mode 100644 index 0000000..2551d6e --- /dev/null +++ b/development/extensions-core/kinesis-ingestion.md @@ -0,0 +1,473 @@ +--- +id: kinesis-ingestion +title: "Amazon Kinesis ingestion" +sidebar_label: "Amazon Kinesis" +--- + + + + +Similar to the [Kafka indexing service](./kafka-ingestion.md), the Kinesis indexing service enables the configuration of *supervisors* on the Overlord, which facilitate ingestion from +Kinesis by managing the creation and lifetime of Kinesis indexing tasks. These indexing tasks read events using Kinesis's own +Shards and Sequence Number mechanism and are therefore able to provide guarantees of exactly-once ingestion. +The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, +and ensure that the scalability and replication requirements are maintained. + +The Kinesis indexing service is provided as the `druid-kinesis-indexing-service` core Apache Druid extension (see +[Including Extensions](../../development/extensions.md#loading-extensions)). Please note that this is +currently designated as an *experimental feature* and is subject to the usual +[experimental caveats](../experimental.md). + +## Submitting a Supervisor Spec + +The Kinesis indexing service requires that the `druid-kinesis-indexing-service` extension be loaded on both the Overlord +and the MiddleManagers. A supervisor for a dataSource is started by submitting a supervisor spec via HTTP POST to +`http://:/druid/indexer/v1/supervisor`, for example: + +``` +curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8090/druid/indexer/v1/supervisor +``` + +A sample supervisor spec is shown below: + +```json +{ + "type": "kinesis", + "dataSchema": { + "dataSource": "metrics-kinesis", + "timestampSpec": { + "column": "timestamp", + "format": "auto" + }, + "dimensionsSpec": { + "dimensions": [], + "dimensionExclusions": [ + "timestamp", + "value" + ] + }, + "metricsSpec": [ + { + "name": "count", + "type": "count" + }, + { + "name": "value_sum", + "fieldName": "value", + "type": "doubleSum" + }, + { + "name": "value_min", + "fieldName": "value", + "type": "doubleMin" + }, + { + "name": "value_max", + "fieldName": "value", + "type": "doubleMax" + } + ], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": "NONE" + } + }, + "ioConfig": { + "stream": "metrics", + "inputFormat": { + "type": "json" + }, + "endpoint": "kinesis.us-east-1.amazonaws.com", + "taskCount": 1, + "replicas": 1, + "taskDuration": "PT1H", + "recordsPerFetch": 2000, + "fetchDelayMillis": 1000 + }, + "tuningConfig": { + "type": "kinesis", + "maxRowsPerSegment": 5000000 + } +} +``` + + +## Supervisor Spec + +|Field|Description|Required| +|--------|-----------|---------| +|`type`|The supervisor type, this should always be `kinesis`.|yes| +|`dataSchema`|The schema that will be used by the Kinesis indexing task during ingestion. See [`dataSchema`](../../ingestion/index.md#dataschema).|yes| +|`ioConfig`|A KinesisSupervisorIOConfig object for configuring Kafka connection and I/O-related settings for the supervisor and indexing task. See [KinesisSupervisorIOConfig](#kinesissupervisorioconfig) below.|yes| +|`tuningConfig`|A KinesisSupervisorTuningConfig object for configuring performance-related settings for the supervisor and indexing tasks. See [KinesisSupervisorTuningConfig](#kinesissupervisortuningconfig) below.|no| + + +### KinesisSupervisorIOConfig + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`stream`|String|The Kinesis stream to read.|yes| +|`inputFormat`|Object|[`inputFormat`](../../ingestion/data-formats.md#input-format) to specify how to parse input data. See [the below section](#specifying-data-format) for details about specifying the input format.|yes| +|`endpoint`|String|The AWS Kinesis stream endpoint for a region. You can find a list of endpoints [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region).|no (default == kinesis.us-east-1.amazonaws.com)| +|`replicas`|Integer|The number of replica sets, where 1 means a single set of tasks (no replication). Replica tasks will always be assigned to different workers to provide resiliency against process failure.|no (default == 1)| +|`taskCount`|Integer|The maximum number of *reading* tasks in a *replica set*. This means that the maximum number of reading tasks will be `taskCount * replicas` and the total number of tasks (*reading* + *publishing*) will be higher than this. See [Capacity Planning](#capacity-planning) below for more details. The number of reading tasks will be less than `taskCount` if `taskCount > {numKinesisShards}`.|no (default == 1)| +|`taskDuration`|ISO8601 Period|The length of time before tasks stop reading and begin publishing their segment.|no (default == PT1H)| +|`startDelay`|ISO8601 Period|The period to wait before the supervisor starts managing tasks.|no (default == PT5S)| +|`period`|ISO8601 Period|How often the supervisor will execute its management logic. Note that the supervisor will also run in response to certain events (such as tasks succeeding, failing, and reaching their taskDuration) so this value specifies the maximum time between iterations.|no (default == PT30S)| +|`useEarliestSequenceNumber`|Boolean|If a supervisor is managing a dataSource for the first time, it will obtain a set of starting sequence numbers from Kinesis. This flag determines whether it retrieves the earliest or latest sequence numbers in Kinesis. Under normal circumstances, subsequent tasks will start from where the previous segments ended so this flag will only be used on first run.|no (default == false)| +|`completionTimeout`|ISO8601 Period|The length of time to wait before declaring a publishing task as failed and terminating it. If this is set too low, your tasks may never publish. The publishing clock for a task begins roughly after `taskDuration` elapses.|no (default == PT6H)| +|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| +|`earlyMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps later than this period after the task reached its taskDuration; for example if this is set to `PT1H`, the taskDuration is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps later than *2016-01-01T14:00Z* will be dropped. **Note:** Tasks sometimes run past their task duration, for example, in cases of supervisor failover. Setting earlyMessageRejectionPeriod too low may cause messages to be dropped unexpectedly whenever a task runs past its originally configured task duration.|no (default == none)| +|`recordsPerFetch`|Integer|The number of records to request per GetRecords call to Kinesis. See 'Determining Fetch Settings' below.|no (default == 2000)| +|`fetchDelayMillis`|Integer|Time in milliseconds to wait between subsequent GetRecords calls to Kinesis. See 'Determining Fetch Settings' below.|no (default == 1000)| +|`awsAssumedRoleArn`|String|The AWS assumed role to use for additional permissions.|no| +|`awsExternalId`|String|The AWS external id to use for additional permissions.|no| +|`deaggregate`|Boolean|Whether to use the de-aggregate function of the KCL. See below for details.|no| + +#### Specifying data format + +Kinesis indexing service supports both [`inputFormat`](../../ingestion/data-formats.md#input-format) and [`parser`](../../ingestion/data-formats.md#parser) to specify the data format. +The `inputFormat` is a new and recommended way to specify the data format for Kinesis indexing service, +but unfortunately, it doesn't support all data formats supported by the legacy `parser`. +(They will be supported in the future.) + +The supported `inputFormat`s include [`csv`](../../ingestion/data-formats.md#csv), +[`delimited`](../../ingestion/data-formats.md#tsv-delimited), and [`json`](../../ingestion/data-formats.md#json). +You can also read [`avro_stream`](../../ingestion/data-formats.md#avro-stream-parser), +[`protobuf`](../../ingestion/data-formats.md#protobuf-parser), +and [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. + + + +### KinesisSupervisorTuningConfig + +The tuningConfig is optional and default parameters will be used if no tuningConfig is specified. + +| Field | Type | Description | Required | +|---------------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------| +| `type` | String | The indexing task type, this should always be `kinesis`. | yes | +| `maxRowsInMemory` | Integer | The number of rows to aggregate before persisting. This number is the post-aggregation rows, so it is not equivalent to the number of input events, but the number of aggregated rows that those events result in. This is used to manage the required JVM heap size. Maximum heap memory usage for indexing scales with maxRowsInMemory * (2 + maxPendingPersists). | no (default == 100000) | +| `maxBytesInMemory` | Long | The number of bytes to aggregate in heap memory before persisting. This is based on a rough estimate of memory usage and not actual usage. Normally this is computed internally and user does not need to set it. The maximum heap memory usage for indexing is maxBytesInMemory * (2 + maxPendingPersists). | no (default == One-sixth of max JVM memory) | +| `maxRowsPerSegment` | Integer | The number of rows to aggregate into a segment; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == 5000000) | +| `maxTotalRows` | Long | The number of rows to aggregate across all segments; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == unlimited) | +| `intermediatePersistPeriod` | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M) | +| `maxPendingPersists` | Integer | Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with maxRowsInMemory * (2 + maxPendingPersists). | no (default == 0, meaning one persist can be running concurrently with ingestion, and none can be queued up) | +| `indexSpec` | Object | Tune how data is indexed. See [IndexSpec](#indexspec) for more information. | no | +| `indexSpecForIntermediatePersists` | | Defines segment storage format options to be used at indexing time for intermediate persisted temporary segments. This can be used to disable dimension/metric compression on intermediate segments to reduce memory required for final merging. However, disabling compression on intermediate segments might increase page cache use while they are used before getting merged into final segment published, see [IndexSpec](#indexspec) for possible values. | no (default = same as indexSpec) | +| `reportParseExceptions` | Boolean | If true, exceptions encountered during parsing will be thrown and will halt ingestion; if false, unparseable rows and fields will be skipped. | no (default == false) | +| `handoffConditionTimeout` | Long | Milliseconds to wait for segment handoff. It must be >= 0, where 0 means to wait forever. | no (default == 0) | +| `resetOffsetAutomatically` | Boolean | Controls behavior when Druid needs to read Kinesis messages that are no longer available.

If false, the exception will bubble up, which will cause your tasks to fail and ingestion to halt. If this occurs, manual intervention is required to correct the situation; potentially using the [Reset Supervisor API](../../operations/api-reference.html#supervisors). This mode is useful for production, since it will make you aware of issues with ingestion.

If true, Druid will automatically reset to the earlier or latest sequence number available in Kinesis, based on the value of the `useEarliestSequenceNumber` property (earliest if true, latest if false). Please note that this can lead to data being _DROPPED_ (if `useEarliestSequenceNumber` is false) or _DUPLICATED_ (if `useEarliestSequenceNumber` is true) without your knowledge. Messages will be logged indicating that a reset has occurred, but ingestion will continue. This mode is useful for non-production situations, since it will make Druid attempt to recover from problems automatically, even if they lead to quiet dropping or duplicating of data. | no (default == false) | +| `skipSequenceNumberAvailabilityCheck` | Boolean | Whether to enable checking if the current sequence number is still available in a particular Kinesis shard. If set to false, the indexing task will attempt to reset the current sequence number (or not), depending on the value of `resetOffsetAutomatically`. | no (default == false) | +| `workerThreads` | Integer | The number of threads that the supervisor uses to handle requests/responses for worker tasks, along with any other internal asynchronous operation. | no (default == min(10, taskCount)) | +| `chatThreads` | Integer | The number of threads that will be used for communicating with indexing tasks. | no (default == min(10, taskCount * replicas)) | +| `chatRetries` | Integer | The number of times HTTP requests to indexing tasks will be retried before considering tasks unresponsive. | no (default == 8) | +| `httpTimeout` | ISO8601 Period | How long to wait for a HTTP response from an indexing task. | no (default == PT10S) | +| `shutdownTimeout` | ISO8601 Period | How long to wait for the supervisor to attempt a graceful shutdown of tasks before exiting. | no (default == PT80S) | +| `recordBufferSize` | Integer | Size of the buffer (number of events) used between the Kinesis fetch threads and the main ingestion thread. | no (default == 10000) | +| `recordBufferOfferTimeout` | Integer | Length of time in milliseconds to wait for space to become available in the buffer before timing out. | no (default == 5000) | +| `recordBufferFullWait` | Integer | Length of time in milliseconds to wait for the buffer to drain before attempting to fetch records from Kinesis again. | no (default == 5000) | +| `fetchSequenceNumberTimeout` | Integer | Length of time in milliseconds to wait for Kinesis to return the earliest or latest sequence number for a shard. Kinesis will not return the latest sequence number if no data is actively being written to that shard. In this case, this fetch call will repeatedly timeout and retry until fresh data is written to the stream. | no (default == 60000) | +| `fetchThreads` | Integer | Size of the pool of threads fetching data from Kinesis. There is no benefit in having more threads than Kinesis shards. | no (default == procs * 2, where "procs" is the number of processors on the server that the task is running on) | +| `segmentWriteOutMediumFactory` | Object | Segment write-out medium to use when creating segments. See below for more information. | no (not specified by default, the value from `druid.peon.defaultSegmentWriteOutMediumFactory.type` is used) | +| `intermediateHandoffPeriod` | ISO8601 Period | How often the tasks should hand off segments. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == P2147483647D) | +| `logParseExceptions` | Boolean | If true, log an error message when a parsing exception occurs, containing information about the row where the error occurred. | no, default == false | +| `maxParseExceptions` | Integer | The maximum number of parse exceptions that can occur before the task halts ingestion and fails. Overridden if `reportParseExceptions` is set. | no, unlimited default | +| `maxSavedParseExceptions` | Integer | When a parse exception occurs, Druid can keep track of the most recent parse exceptions. "maxSavedParseExceptions" limits how many exception instances will be saved. These saved exceptions will be made available after the task finishes in the [task completion report](../../ingestion/tasks.md#reports). Overridden if `reportParseExceptions` is set. | no, default == 0 | +| `maxRecordsPerPoll` | Integer | The maximum number of records/events to be fetched from buffer per poll. The actual maximum will be `Max(maxRecordsPerPoll, Max(bufferSize, 1))` | no, default == 100 | +| `repartitionTransitionDuration` | ISO8601 Period | When shards are split or merged, the supervisor will recompute shard -> task group mappings, and signal any running tasks created under the old mappings to stop early at (current time + `repartitionTransitionDuration`). Stopping the tasks early allows Druid to begin reading from the new shards more quickly. The repartition transition wait time controlled by this property gives the stream additional time to write records to the new shards after the split/merge, which helps avoid the issues with empty shard handling described at https://github.com/apache/druid/issues/7600. | no, (default == PT2M) | + +#### IndexSpec + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|bitmap|Object|Compression format for bitmap indexes. Should be a JSON object. See [Bitmap types](#bitmap-types) below for options.|no (defaults to Roaring)| +|dimensionCompression|String|Compression format for dimension columns. Choose from `LZ4`, `LZF`, or `uncompressed`.|no (default == `LZ4`)| +|metricCompression|String|Compression format for primitive type metric columns. Choose from `LZ4`, `LZF`, `uncompressed`, or `none`.|no (default == `LZ4`)| +|longEncoding|String|Encoding format for metric and dimension columns with type long. Choose from `auto` or `longs`. `auto` encodes the values using sequence number or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as is with 8 bytes each.|no (default == `longs`)| + +##### Bitmap types + +For Roaring bitmaps: + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|Must be `roaring`.|yes| +|`compressRunOnSerialization`|Boolean|Use a run-length encoding where it is estimated as more space efficient.|no (default == `true`)| + +For Concise bitmaps: + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|Must be `concise`.|yes| + +#### SegmentWriteOutMediumFactory + +|Field|Type|Description|Required| +|-----|----|-----------|--------| +|`type`|String|See [Additional Peon Configuration: SegmentWriteOutMediumFactory](../../configuration/index.html#segmentwriteoutmediumfactory) for explanation and available options.|yes| + +## Operations + +This section gives descriptions of how some supervisor APIs work specifically in Kinesis Indexing Service. +For all supervisor APIs, please check [Supervisor APIs](../../operations/api-reference.html#supervisors). + +### AWS Authentication +To authenticate with AWS, you must provide your AWS access key and AWS secret key via runtime.properties, for example: +``` +-Ddruid.kinesis.accessKey=123 -Ddruid.kinesis.secretKey=456 +``` +The AWS access key ID and secret access key are used for Kinesis API requests. If this is not provided, the service will +look for credentials set in environment variables, in the default profile configuration file, and from the EC2 instance +profile provider (in this order). + +### Getting Supervisor Status Report + +`GET /druid/indexer/v1/supervisor//status` returns a snapshot report of the current state of the tasks +managed by the given supervisor. This includes the latest sequence numbers as reported by Kinesis. Unlike the Kafka +Indexing Service, stats about lag are not yet supported. + +The status report also contains the supervisor's state and a list of recently thrown exceptions (reported as +`recentErrors`, whose max size can be controlled using the `druid.supervisor.maxStoredExceptionEvents` configuration). +There are two fields related to the supervisor's state - `state` and `detailedState`. The `state` field will always be +one of a small number of generic states that are applicable to any type of supervisor, while the `detailedState` field +will contain a more descriptive, implementation-specific state that may provide more insight into the supervisor's +activities than the generic `state` field. + +The list of possible `state` values are: [`PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`, `UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`] + +The list of `detailedState` values and their corresponding `state` mapping is as follows: + +|Detailed State|Corresponding State|Description| +|--------------|-------------------|-----------| +|UNHEALTHY_SUPERVISOR|UNHEALTHY_SUPERVISOR|The supervisor has encountered errors on the past `druid.supervisor.unhealthinessThreshold` iterations| +|UNHEALTHY_TASKS|UNHEALTHY_TASKS|The last `druid.supervisor.taskUnhealthinessThreshold` tasks have all failed| +|UNABLE_TO_CONNECT_TO_STREAM|UNHEALTHY_SUPERVISOR|The supervisor is encountering connectivity issues with Kinesis and has not successfully connected in the past| +|LOST_CONTACT_WITH_STREAM|UNHEALTHY_SUPERVISOR|The supervisor is encountering connectivity issues with Kinesis but has successfully connected in the past| +|PENDING (first iteration only)|PENDING|The supervisor has been initialized and hasn't started connecting to the stream| +|CONNECTING_TO_STREAM (first iteration only)|RUNNING|The supervisor is trying to connect to the stream and update partition data| +|DISCOVERING_INITIAL_TASKS (first iteration only)|RUNNING|The supervisor is discovering already-running tasks| +|CREATING_TASKS (first iteration only)|RUNNING|The supervisor is creating tasks and discovering state| +|RUNNING|RUNNING|The supervisor has started tasks and is waiting for taskDuration to elapse| +|SUSPENDED|SUSPENDED|The supervisor has been suspended| +|STOPPING|STOPPING|The supervisor is stopping| + +On each iteration of the supervisor's run loop, the supervisor completes the following tasks in sequence: + 1) Fetch the list of shards from Kinesis and determine the starting sequence number for each shard (either based on the + last processed sequence number if continuing, or starting from the beginning or ending of the stream if this is a new stream). + 2) Discover any running indexing tasks that are writing to the supervisor's datasource and adopt them if they match + the supervisor's configuration, else signal them to stop. + 3) Send a status request to each supervised task to update our view of the state of the tasks under our supervision. + 4) Handle tasks that have exceeded `taskDuration` and should transition from the reading to publishing state. + 5) Handle tasks that have finished publishing and signal redundant replica tasks to stop. + 6) Handle tasks that have failed and clean up the supervisor's internal state. + 7) Compare the list of healthy tasks to the requested `taskCount` and `replicas` configurations and create additional tasks if required. + +The `detailedState` field will show additional values (those marked with "first iteration only") the first time the +supervisor executes this run loop after startup or after resuming from a suspension. This is intended to surface +initialization-type issues, where the supervisor is unable to reach a stable state (perhaps because it can't connect to +Kinesis, it can't read from the stream, or it can't communicate with existing tasks). Once the supervisor is stable - +that is, once it has completed a full execution without encountering any issues - `detailedState` will show a `RUNNING` +state until it is stopped, suspended, or hits a failure threshold and transitions to an unhealthy state. + +### Updating Existing Supervisors + +`POST /druid/indexer/v1/supervisor` can be used to update existing supervisor spec. +Calling this endpoint when there is already an existing supervisor for the same dataSource will cause: + +- The running supervisor to signal its managed tasks to stop reading and begin publishing. +- The running supervisor to exit. +- A new supervisor to be created using the configuration provided in the request body. This supervisor will retain the +existing publishing tasks and will create new tasks starting at the sequence numbers the publishing tasks ended on. + +Seamless schema migrations can thus be achieved by simply submitting the new schema using this endpoint. + +### Suspending and Resuming Supervisors + +You can suspend and resume a supervisor using `POST /druid/indexer/v1/supervisor//suspend` and `POST /druid/indexer/v1/supervisor//resume`, respectively. + +Note that the supervisor itself will still be operating and emitting logs and metrics, +it will just ensure that no indexing tasks are running until the supervisor is resumed. + +### Resetting Supervisors + +The `POST /druid/indexer/v1/supervisor//reset` operation clears stored +sequence numbers, causing the supervisor to start reading from either the earliest or +latest sequence numbers in Kinesis (depending on the value of `useEarliestSequenceNumber`). +After clearing stored sequence numbers, the supervisor kills and recreates active tasks, +so that tasks begin reading from valid sequence numbers. + +Use care when using this operation! Resetting the supervisor may cause Kinesis messages +to be skipped or read twice, resulting in missing or duplicate data. + +The reason for using this operation is to recover from a state in which the supervisor +ceases operating due to missing sequence numbers. The indexing service keeps track of the latest +persisted sequence number in order to provide exactly-once ingestion guarantees across +tasks. + +Subsequent tasks must start reading from where the previous task completed in +order for the generated segments to be accepted. If the messages at the expected starting sequence numbers are +no longer available in Kinesis (typically because the message retention period has elapsed or the topic was +removed and re-created) the supervisor will refuse to start and in-flight tasks will fail. This operation +enables you to recover from this condition. + +Note that the supervisor must be running for this endpoint to be available. + +### Terminating Supervisors + +The `POST /druid/indexer/v1/supervisor//terminate` operation terminates a supervisor and causes +all associated indexing tasks managed by this supervisor to immediately stop and begin +publishing their segments. This supervisor will still exist in the metadata store and its history may be retrieved +with the supervisor history API, but will not be listed in the 'get supervisors' API response nor can its configuration +or status report be retrieved. The only way this supervisor can start again is by submitting a functioning supervisor +spec to the create API. + +### Capacity Planning + +Kinesis indexing tasks run on MiddleManagers and are thus limited by the resources available in the MiddleManager +cluster. In particular, you should make sure that you have sufficient worker capacity (configured using the +`druid.worker.capacity` property) to handle the configuration in the supervisor spec. Note that worker capacity is +shared across all types of indexing tasks, so you should plan your worker capacity to handle your total indexing load +(e.g. batch processing, realtime tasks, merging tasks, etc.). If your workers run out of capacity, Kinesis indexing tasks +will queue and wait for the next available worker. This may cause queries to return partial results but will not result +in data loss (assuming the tasks run before Kinesis purges those sequence numbers). + +A running task will normally be in one of two states: *reading* or *publishing*. A task will remain in reading state for +`taskDuration`, at which point it will transition to publishing state. A task will remain in publishing state for as long +as it takes to generate segments, push segments to deep storage, and have them be loaded and served by a Historical process +(or until `completionTimeout` elapses). + +The number of reading tasks is controlled by `replicas` and `taskCount`. In general, there will be `replicas * taskCount` +reading tasks, the exception being if taskCount > {numKinesisShards} in which case {numKinesisShards} tasks will +be used instead. When `taskDuration` elapses, these tasks will transition to publishing state and `replicas * taskCount` +new reading tasks will be created. Therefore to allow for reading tasks and publishing tasks to run concurrently, there +should be a minimum capacity of: + +``` +workerCapacity = 2 * replicas * taskCount +``` + +This value is for the ideal situation in which there is at most one set of tasks publishing while another set is reading. +In some circumstances, it is possible to have multiple sets of tasks publishing simultaneously. This would happen if the +time-to-publish (generate segment, push to deep storage, loaded on Historical) > `taskDuration`. This is a valid +scenario (correctness-wise) but requires additional worker capacity to support. In general, it is a good idea to have +`taskDuration` be large enough that the previous set of tasks finishes publishing before the current set begins. + +### Supervisor Persistence + +When a supervisor spec is submitted via the `POST /druid/indexer/v1/supervisor` endpoint, it is persisted in the +configured metadata database. There can only be a single supervisor per dataSource, and submitting a second spec for +the same dataSource will overwrite the previous one. + +When an Overlord gains leadership, either by being started or as a result of another Overlord failing, it will spawn +a supervisor for each supervisor spec in the metadata database. The supervisor will then discover running Kinesis indexing +tasks and will attempt to adopt them if they are compatible with the supervisor's configuration. If they are not +compatible because they have a different ingestion spec or shard allocation, the tasks will be killed and the +supervisor will create a new set of tasks. In this way, the supervisors are persistent across Overlord restarts and +fail-overs. + +A supervisor is stopped via the `POST /druid/indexer/v1/supervisor//terminate` endpoint. This places a +tombstone marker in the database (to prevent the supervisor from being reloaded on a restart) and then gracefully +shuts down the currently running supervisor. When a supervisor is shut down in this way, it will instruct its +managed tasks to stop reading and begin publishing their segments immediately. The call to the shutdown endpoint will +return after all tasks have been signalled to stop but before the tasks finish publishing their segments. + +### Schema/Configuration Changes + +Schema and configuration changes are handled by submitting the new supervisor spec via the same +`POST /druid/indexer/v1/supervisor` endpoint used to initially create the supervisor. The Overlord will initiate a +graceful shutdown of the existing supervisor which will cause the tasks being managed by that supervisor to stop reading +and begin publishing their segments. A new supervisor will then be started which will create a new set of tasks that +will start reading from the sequence numbers where the previous now-publishing tasks left off, but using the updated schema. +In this way, configuration changes can be applied without requiring any pause in ingestion. + +### Deployment Notes + +#### On the Subject of Segments + +Each Kinesis Indexing Task puts events consumed from Kinesis Shards assigned to it in a single segment for each segment +granular interval until maxRowsPerSegment, maxTotalRows or intermediateHandoffPeriod limit is reached, at this point a new shard +for this segment granularity is created for further events. Kinesis Indexing Task also does incremental hand-offs which +means that all the segments created by a task will not be held up till the task duration is over. As soon as maxRowsPerSegment, +maxTotalRows or intermediateHandoffPeriod limit is hit, all the segments held by the task at that point in time will be handed-off +and new set of segments will be created for further events. This means that the task can run for longer durations of time +without accumulating old segments locally on Middle Manager processes and it is encouraged to do so. + +Kinesis Indexing Service may still produce some small segments. Lets say the task duration is 4 hours, segment granularity +is set to an HOUR and Supervisor was started at 9:10 then after 4 hours at 13:10, new set of tasks will be started and +events for the interval 13:00 - 14:00 may be split across previous and new set of tasks. If you see it becoming a problem then +one can schedule re-indexing tasks be run to merge segments together into new segments of an ideal size (in the range of ~500-700 MB per segment). +Details on how to optimize the segment size can be found on [Segment size optimization](../../operations/segment-optimization.md). +There is also ongoing work to support automatic segment compaction of sharded segments as well as compaction not requiring +Hadoop (see [here](https://github.com/apache/druid/pull/5102)). + +### Determining Fetch Settings +Internally, the Kinesis Indexing Service uses the Kinesis Record Supplier abstraction for fetching Kinesis data records and storing the records +locally. The way the Kinesis Record Supplier fetches records is to have a separate thread run the fetching operation per each Kinesis Shard, the +max number of threads is determined by `fetchThreads`. For example, a Kinesis stream with 3 shards will have 3 threads, each fetching from a shard separately. +There is a delay between each fetching operation, which is controlled by `fetchDelayMillis`. The maximum number of records to be fetched per thread per +operation is controlled by `recordsPerFetch`. Note that this is not the same as `maxRecordsPerPoll`. + +The records fetched by each thread will be pushed to a queue in the order that they are fetched. The records are stored in this queue until `poll()` is called +by either the supervisor or the indexing task. `poll()` will attempt to drain the internal buffer queue up to a limit of `max(maxRecordsPerPoll, q.size())`. +Here `maxRecordsPerPoll` controls the theoretical maximum records to drain out of the buffer queue, so setting this parameter to a reasonable value is essential +in preventing the queue from overflowing or memory exceeding heap size. + +Kinesis places the following restrictions on calls to fetch records: + +- Each data record can be up to 1 MB in size. +- Each shard can support up to 5 transactions per second for reads. +- Each shard can read up to 2 MB per second. +- The maximum size of data that GetRecords can return is 10 MB. + +Values for `recordsPerFetch` and `fetchDelayMillis` should be chosen to maximize throughput under the above constraints. +The values that you choose will depend on the average size of a record and the number of consumers you have reading from +a given shard (which will be `replicas` unless you have other consumers also reading from this Kinesis stream). + +If the above limits are violated, AWS will throw ProvisionedThroughputExceededException errors on subsequent calls to +read data. When this happens, the Kinesis indexing service will pause by `fetchDelayMillis` and then attempt the call +again. + +Internally, each indexing task maintains a buffer that stores the fetched but not yet processed record. `recordsPerFetch` and `fetchDelayMillis` +control this behavior. The number of records that the indexing task fetch from the buffer is controlled by `maxRecordsPerPoll`, which +determines the number of records to be processed per each ingestion loop in the task. + +## Deaggregation +See [issue](https://github.com/apache/druid/issues/6714) + +The Kinesis indexing service supports de-aggregation of multiple rows packed into a single record by the Kinesis +Producer Library's aggregate method for more efficient data transfer. Currently, enabling the de-aggregate functionality +requires the user to manually provide the Kinesis Client Library on the classpath, since this library has a license not +compatible with Apache projects. + +To enable this feature, add the `amazon-kinesis-client` (tested on version `1.9.2`) jar file ([link](https://mvnrepository.com/artifact/com.amazonaws/amazon-kinesis-client/1.9.2)) under `dist/druid/extensions/druid-kinesis-indexing-service/`. +Then when submitting a supervisor-spec, set `deaggregate` to true. + +## Resharding + +When changing the shard count for a Kinesis stream, there will be a window of time around the resharding operation with early shutdown of Kinesis ingestion tasks and possible task failures. + +The early shutdowns and task failures are expected, and they occur because the supervisor will update the shard -> task group mappings as shards are closed and fully read, to ensure that tasks are not running +with an assignment of closed shards that have been fully read and to ensure a balanced distribution of active shards across tasks. + +This window with early task shutdowns and possible task failures will conclude when: +- All closed shards have been fully read and the Kinesis ingestion tasks have published the data from those shards, committing the "closed" state to metadata storage +- Any remaining tasks that had inactive shards in the assignment have been shutdown (these tasks would have been created before the closed shards were completely drained) + diff --git a/development/extensions-core/lookups-cached-global.md b/development/extensions-core/lookups-cached-global.md new file mode 100644 index 0000000..9ce6997 --- /dev/null +++ b/development/extensions-core/lookups-cached-global.md @@ -0,0 +1,378 @@ +--- +id: lookups-cached-global +title: "Globally Cached Lookups" +--- + + + + +> Lookups are an [experimental](../experimental.md) feature. + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-lookups-cached-global` as an extension. + +## Configuration +> Static configuration is no longer supported. Lookups can be configured through +> [dynamic configuration](../../querying/lookups.md#configuration). + +Globally cached lookups are appropriate for lookups which are not possible to pass at query time due to their size, +or are not desired to be passed at query time because the data is to reside in and be handled by the Druid servers, +and are small enough to reasonably populate in-memory. This usually means tens to tens of thousands of entries per lookup. + +Globally cached lookups all draw from the same cache pool, allowing each process to have a fixed cache pool that can be used by cached lookups. + +Globally cached lookups can be specified as part of the [cluster wide config for lookups](../../querying/lookups.md) as a type of `cachedNamespace` + + ```json + { + "type": "cachedNamespace", + "extractionNamespace": { + "type": "uri", + "uri": "file:/tmp/prefix/", + "namespaceParseSpec": { + "format": "csv", + "columns": [ + "key", + "value" + ] + }, + "pollPeriod": "PT5M" + }, + "firstCacheTimeout": 0 + } + ``` + + ```json +{ + "type": "cachedNamespace", + "extractionNamespace": { + "type": "jdbc", + "connectorConfig": { + "createTables": true, + "connectURI": "jdbc:mysql:\/\/localhost:3306\/druid", + "user": "druid", + "password": "diurd" + }, + "table": "lookupTable", + "keyColumn": "mykeyColumn", + "valueColumn": "myValueColumn", + "filter" : "myFilterSQL (Where clause statement e.g LOOKUPTYPE=1)", + "tsColumn": "timeColumn" + }, + "firstCacheTimeout": 120000, + "injective":true +} + ``` + +The parameters are as follows + +|Property|Description|Required|Default| +|--------|-----------|--------|-------| +|`extractionNamespace`|Specifies how to populate the local cache. See below|Yes|-| +|`firstCacheTimeout`|How long to wait (in ms) for the first run of the cache to populate. 0 indicates to not wait|No|`0` (do not wait)| +|`injective`|If the underlying map is [injective](../../querying/lookups.html#query-execution) (keys and values are unique) then optimizations can occur internally by setting this to `true`|No|`false`| + +If `firstCacheTimeout` is set to a non-zero value, it should be less than `druid.manager.lookups.hostUpdateTimeout`. If `firstCacheTimeout` is NOT set, then management is essentially asynchronous and does not know if a lookup succeeded or failed in starting. In such a case logs from the processes using lookups should be monitored for repeated failures. + +Proper functionality of globally cached lookups requires the following extension to be loaded on the Broker, Peon, and Historical processes: +`druid-lookups-cached-global` + +## Example configuration + +In a simple case where only one [tier](../../querying/lookups.html#dynamic-configuration) exists (`realtime_customer2`) with one `cachedNamespace` lookup called `country_code`, the resulting configuration JSON looks similar to the following: + +```json +{ + "realtime_customer2": { + "country_code": { + "version": "v0", + "lookupExtractorFactory": { + "type": "cachedNamespace", + "extractionNamespace": { + "type": "jdbc", + "connectorConfig": { + "createTables": true, + "connectURI": "jdbc:mysql:\/\/localhost:3306\/druid", + "user": "druid", + "password": "diurd" + }, + "table": "lookupValues", + "keyColumn": "value_id", + "valueColumn": "value_text", + "filter": "value_type='country'", + "tsColumn": "timeColumn" + }, + "firstCacheTimeout": 120000, + "injective": true + } + } + } +} +``` + +Where the Coordinator endpoint `/druid/coordinator/v1/lookups/realtime_customer2/country_code` should return + +```json +{ + "version": "v0", + "lookupExtractorFactory": { + "type": "cachedNamespace", + "extractionNamespace": { + "type": "jdbc", + "connectorConfig": { + "createTables": true, + "connectURI": "jdbc:mysql://localhost:3306/druid", + "user": "druid", + "password": "diurd" + }, + "table": "lookupValues", + "keyColumn": "value_id", + "valueColumn": "value_text", + "filter": "value_type='country'", + "tsColumn": "timeColumn" + }, + "firstCacheTimeout": 120000, + "injective": true + } +} +``` + +## Cache Settings + +Lookups are cached locally on Historical processes. The following are settings used by the processes which service queries when +setting namespaces (Broker, Peon, Historical) + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.lookup.namespace.cache.type`|Specifies the type of caching to be used by the namespaces. May be one of [`offHeap`, `onHeap`]. `offHeap` uses a temporary file for off-heap storage of the namespace (memory mapped files). `onHeap` stores all cache on the heap in standard java map types.|`onHeap`| +|`druid.lookup.namespace.numExtractionThreads`|The number of threads in the thread pool dedicated for lookup extraction and updates. This number may need to be scaled up, if you have a lot of lookups and they take long time to extract, to avoid timeouts.|2| +|`druid.lookup.namespace.numBufferedEntries`|If using off-heap caching, the number of records to be stored on an on-heap buffer.|100,000| + +The cache is populated in different ways depending on the settings below. In general, most namespaces employ +a `pollPeriod` at the end of which time they poll the remote resource of interest for updates. + +`onHeap` uses `ConcurrentMap`s in the java heap, and thus affects garbage collection and heap sizing. +`offHeap` uses an on-heap buffer and MapDB using memory-mapped files in the java temporary directory. +So if total number of entries in the `cachedNamespace` is in excess of the buffer's configured capacity, the extra will be kept in memory as page cache, and paged in and out by general OS tunings. +It's highly recommended that `druid.lookup.namespace.numBufferedEntries` is set when using `offHeap`, the value should be chosen from the range between 10% and 50% of the number of entries in the lookup. + +## Supported lookups + +For additional lookups, please see our [extensions list](../extensions.md). + +### URI lookup + +The remapping values for each globally cached lookup can be specified by a JSON object as per the following examples: + +```json +{ + "type":"uri", + "uri": "s3://bucket/some/key/prefix/renames-0003.gz", + "namespaceParseSpec":{ + "format":"csv", + "columns":["key","value"] + }, + "pollPeriod":"PT5M" +} +``` + +```json +{ + "type":"uri", + "uriPrefix": "s3://bucket/some/key/prefix/", + "fileRegex":"renames-[0-9]*\\.gz", + "namespaceParseSpec":{ + "format":"csv", + "columns":["key","value"] + }, + "pollPeriod":"PT5M" +} +``` + +|Property|Description|Required|Default| +|--------|-----------|--------|-------| +|`pollPeriod`|Period between polling for updates|No|0 (only once)| +|`uri`|URI for the file of interest, specified as a file, hdfs, or s3 path|No|Use `uriPrefix`| +|`uriPrefix`|A URI that specifies a directory (or other searchable resource) in which to search for files|No|Use `uri`| +|`fileRegex`|Optional regex for matching the file name under `uriPrefix`. Only used if `uriPrefix` is used|No|`".*"`| +|`namespaceParseSpec`|How to interpret the data at the URI|Yes|| + +One of either `uri` or `uriPrefix` must be specified, as either a local file system (file://), HDFS (hdfs://), or S3 (s3://) location. HTTP location is not currently supported. + +The `pollPeriod` value specifies the period in ISO 8601 format between checks for replacement data for the lookup. If the source of the lookup is capable of providing a timestamp, the lookup will only be updated if it has changed since the prior tick of `pollPeriod`. A value of 0, an absent parameter, or `null` all mean populate once and do not attempt to look for new data later. Whenever an poll occurs, the updating system will look for a file with the most recent timestamp and assume that one with the most recent data set, replacing the local cache of the lookup data. + +The `namespaceParseSpec` can be one of a number of values. Each of the examples below would rename foo to bar, baz to bat, and buck to truck. All parseSpec types assumes each input is delimited by a new line. See below for the types of parseSpec supported. + +Only ONE file which matches the search will be used. For most implementations, the discriminator for choosing the URIs is by whichever one reports the most recent timestamp for its modification time. + +#### csv lookupParseSpec +|Parameter|Description|Required|Default| +|---------|-----------|--------|-------| +|`columns`|The list of columns in the csv file|no if `hasHeaderRow` is set|`null`| +|`keyColumn`|The name of the column containing the key|no|The first column| +|`valueColumn`|The name of the column containing the value|no|The second column| +|`hasHeaderRow`|A flag to indicate that column information can be extracted from the input files' header row|no|false| +|`skipHeaderRows`|Number of header rows to be skipped|no|0| + +If both `skipHeaderRows` and `hasHeaderRow` options are set, `skipHeaderRows` is first applied. For example, if you set +`skipHeaderRows` to 2 and `hasHeaderRow` to true, Druid will skip the first two lines and then extract column information +from the third line. + +*example input* + +``` +bar,something,foo +bat,something2,baz +truck,something3,buck +``` + +*example namespaceParseSpec* + +```json +"namespaceParseSpec": { + "format": "csv", + "columns": ["value","somethingElse","key"], + "keyColumn": "key", + "valueColumn": "value" +} +``` + +#### tsv lookupParseSpec +|Parameter|Description|Required|Default| +|---------|-----------|--------|-------| +|`columns`|The list of columns in the tsv file|yes|`null`| +|`keyColumn`|The name of the column containing the key|no|The first column| +|`valueColumn`|The name of the column containing the value|no|The second column| +|`delimiter`|The delimiter in the file|no|tab (`\t`)| +|`listDelimiter`|The list delimiter in the file|no| (`\u0001`)| +|`hasHeaderRow`|A flag to indicate that column information can be extracted from the input files' header row|no|false| +|`skipHeaderRows`|Number of header rows to be skipped|no|0| + +If both `skipHeaderRows` and `hasHeaderRow` options are set, `skipHeaderRows` is first applied. For example, if you set +`skipHeaderRows` to 2 and `hasHeaderRow` to true, Druid will skip the first two lines and then extract column information +from the third line. + +*example input* + +``` +bar|something,1|foo +bat|something,2|baz +truck|something,3|buck +``` + +*example namespaceParseSpec* + +```json +"namespaceParseSpec": { + "format": "tsv", + "columns": ["value","somethingElse","key"], + "keyColumn": "key", + "valueColumn": "value", + "delimiter": "|" +} +``` + +#### customJson lookupParseSpec + +|Parameter|Description|Required|Default| +|---------|-----------|--------|-------| +|`keyFieldName`|The field name of the key|yes|null| +|`valueFieldName`|The field name of the value|yes|null| + +*example input* + +```json +{"key": "foo", "value": "bar", "somethingElse" : "something"} +{"key": "baz", "value": "bat", "somethingElse" : "something"} +{"key": "buck", "somethingElse": "something", "value": "truck"} +``` + +*example namespaceParseSpec* + +```json +"namespaceParseSpec": { + "format": "customJson", + "keyFieldName": "key", + "valueFieldName": "value" +} +``` + +With customJson parsing, if the value field for a particular row is missing or null then that line will be skipped, and +will not be included in the lookup. + +#### simpleJson lookupParseSpec +The `simpleJson` lookupParseSpec does not take any parameters. It is simply a line delimited JSON file where the field is the key, and the field's value is the value. + +*example input* + +```json +{"foo": "bar"} +{"baz": "bat"} +{"buck": "truck"} +``` + +*example namespaceParseSpec* + +```json +"namespaceParseSpec":{ + "format": "simpleJson" +} +``` + +### JDBC lookup + +The JDBC lookups will poll a database to populate its local cache. If the `tsColumn` is set it must be able to accept comparisons in the format `'2015-01-01 00:00:00'`. For example, the following must be valid SQL for the table `SELECT * FROM some_lookup_table WHERE timestamp_column > '2015-01-01 00:00:00'`. If `tsColumn` is set, the caching service will attempt to only poll values that were written *after* the last sync. If `tsColumn` is not set, the entire table is pulled every time. + +|Parameter|Description|Required|Default| +|---------|-----------|--------|-------| +|`namespace`|The namespace to define|Yes|| +|`connectorConfig`|The connector config to use|Yes|| +|`table`|The table which contains the key value pairs|Yes|| +|`keyColumn`|The column in `table` which contains the keys|Yes|| +|`valueColumn`|The column in `table` which contains the values|Yes|| +|`filter`|The filter to use when selecting lookups, this is used to create a where clause on lookup population|No|No Filter| +|`tsColumn`| The column in `table` which contains when the key was updated|No|Not used| +|`pollPeriod`|How often to poll the DB|No|0 (only once)| + +```json +{ + "type":"jdbc", + "namespace":"some_lookup", + "connectorConfig":{ + "createTables":true, + "connectURI":"jdbc:mysql://localhost:3306/druid", + "user":"druid", + "password":"diurd" + }, + "table":"some_lookup_table", + "keyColumn":"the_old_dim_value", + "valueColumn":"the_new_dim_value", + "tsColumn":"timestamp_column", + "pollPeriod":600000 +} +``` + +> If using JDBC, you will need to add your database's client JAR files to the extension's directory. +> For Postgres, the connector JAR is already included. +> For MySQL, you can get it from https://dev.mysql.com/downloads/connector/j/. +> Copy or symlink the downloaded file to `extensions/druid-lookups-cached-global` under the distribution root directory. + +## Introspection + +Globally cached lookups have introspection points at `/keys` and `/values` which return a complete set of the keys and values (respectively) in the lookup. Introspection to `/` returns the entire map. Introspection to `/version` returns the version indicator for the lookup. diff --git a/development/extensions-core/mysql.md b/development/extensions-core/mysql.md new file mode 100644 index 0000000..f39e93c --- /dev/null +++ b/development/extensions-core/mysql.md @@ -0,0 +1,173 @@ +--- +id: mysql +title: "MySQL Metadata Store" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `mysql-metadata-storage` as an extension. + +> The MySQL extension requires the MySQL Connector/J library which is not included in the Druid distribution. +> Refer to the following section for instructions on how to install this library. + +## Installing the MySQL connector library + +This extension uses Oracle's MySQL JDBC driver which is not included in the Druid distribution and must be +installed separately. There are a few ways to obtain this library: + +- It can be downloaded from the MySQL site at: https://dev.mysql.com/downloads/connector/j/ +- It can be fetched from Maven Central at: https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.48/mysql-connector-java-5.1.48.jar +- It may be available through your package manager, e.g. as `libmysql-java` on APT for a Debian-based OS + +This should fetch a JAR file named similar to 'mysql-connector-java-x.x.xx.jar'. + +Copy or symlink this file to `extensions/mysql-metadata-storage` under the distribution root directory. + +## Setting up MySQL + +1. Install MySQL + + Use your favorite package manager to install mysql, e.g.: + - on Ubuntu/Debian using apt `apt-get install mysql-server` + - on OS X, using [Homebrew](http://brew.sh/) `brew install mysql` + + Alternatively, download and follow installation instructions for MySQL + Community Server here: + [http://dev.mysql.com/downloads/mysql/](http://dev.mysql.com/downloads/mysql/) + +2. Create a druid database and user + + Connect to MySQL from the machine where it is installed. + + ```bash + > mysql -u root + ``` + + Paste the following snippet into the mysql prompt: + + ```sql + -- create a druid database, make sure to use utf8mb4 as encoding + CREATE DATABASE druid DEFAULT CHARACTER SET utf8mb4; + + -- create a druid user + CREATE USER 'druid'@'localhost' IDENTIFIED BY 'diurd'; + + -- grant the user all the permissions on the database we just created + GRANT ALL PRIVILEGES ON druid.* TO 'druid'@'localhost'; + ``` + +3. Configure your Druid metadata storage extension: + + Add the following parameters to your Druid configuration, replacing `` + with the location (host name and port) of the database. + + ```properties + druid.extensions.loadList=["mysql-metadata-storage"] + druid.metadata.storage.type=mysql + druid.metadata.storage.connector.connectURI=jdbc:mysql:///druid + druid.metadata.storage.connector.user=druid + druid.metadata.storage.connector.password=diurd + ``` + +## Encrypting MySQL connections + This extension provides support for encrypting MySQL connections. To get more information about encrypting MySQL connections using TLS/SSL in general, please refer to this [guide](https://dev.mysql.com/doc/refman/5.7/en/using-encrypted-connections.html). + +## Configuration + +|Property|Description|Default|Required| +|--------|-----------|-------|--------| +|`druid.metadata.mysql.ssl.useSSL`|Enable SSL|`false`|no| +|`druid.metadata.mysql.ssl.clientCertificateKeyStoreUrl`|The file path URL to the client certificate key store.|none|no| +|`druid.metadata.mysql.ssl.clientCertificateKeyStoreType`|The type of the key store where the client certificate is stored.|none|no| +|`druid.metadata.mysql.ssl.clientCertificateKeyStorePassword`|The [Password Provider](../../operations/password-provider.md) or String password for the client key store.|none|no| +|`druid.metadata.mysql.ssl.verifyServerCertificate`|Enables server certificate verification.|false|no| +|`druid.metadata.mysql.ssl.trustCertificateKeyStoreUrl`|The file path to the trusted root certificate key store.|Default trust store provided by MySQL|yes if `verifyServerCertificate` is set to true and a custom trust store is used| +|`druid.metadata.mysql.ssl.trustCertificateKeyStoreType`|The type of the key store where trusted root certificates are stored.|JKS|yes if `verifyServerCertificate` is set to true and keystore type is not JKS| +|`druid.metadata.mysql.ssl.trustCertificateKeyStorePassword`|The [Password Provider](../../operations/password-provider.md) or String password for the trust store.|none|yes if `verifyServerCertificate` is set to true and password is not null| +|`druid.metadata.mysql.ssl.enabledSSLCipherSuites`|Overrides the existing cipher suites with these cipher suites.|none|no| +|`druid.metadata.mysql.ssl.enabledTLSProtocols`|Overrides the TLS protocols with these protocols.|none|no| + + +### MySQL Firehose + +The MySQL extension provides an implementation of an [SqlFirehose](../../ingestion/native-batch.md#firehoses-deprecated) which can be used to ingest data into Druid from a MySQL database. + +```json +{ + "type": "index_parallel", + "spec": { + "dataSchema": { + "dataSource": "some_datasource", + "parser": { + "parseSpec": { + "format": "timeAndDims", + "dimensionsSpec": { + "dimensionExclusions": [], + "dimensions": [ + "dim1", + "dim2", + "dim3" + ] + }, + "timestampSpec": { + "format": "auto", + "column": "ts" + } + } + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "DAY", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": null + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "type": "index_parallel", + "firehose": { + "type": "sql", + "database": { + "type": "mysql", + "connectorConfig": { + "connectURI": "jdbc:mysql://some-rds-host.us-west-1.rds.amazonaws.com:3306/druid", + "user": "admin", + "password": "secret" + } + }, + "sqls": [ + "SELECT * FROM some_table" + ] + } + }, + "tuningconfig": { + "type": "index_parallel" + } + } +} +``` diff --git a/development/extensions-core/orc.md b/development/extensions-core/orc.md new file mode 100644 index 0000000..6856baf --- /dev/null +++ b/development/extensions-core/orc.md @@ -0,0 +1,84 @@ +--- +id: orc +title: "ORC Extension" +--- + + + +## ORC extension + +This Apache Druid extension enables Druid to ingest and understand the Apache ORC data format. + +The extension provides the [ORC input format](../../ingestion/data-formats.md#orc) and the [ORC Hadoop parser](../../ingestion/data-formats.md#orc-hadoop-parser) +for [native batch ingestion](../../ingestion/native-batch.md) and [Hadoop batch ingestion](../../ingestion/hadoop.md), respectively. +Please see corresponding docs for details. + +To use this extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-orc-extensions`. + +### Migration from 'contrib' extension +This extension, first available in version 0.15.0, replaces the previous 'contrib' extension which was available until +0.14.0-incubating. While this extension can index any data the 'contrib' extension could, the JSON spec for the +ingestion task is *incompatible*, and will need modified to work with the newer 'core' extension. + +To migrate to 0.15.0+: + +* In `inputSpec` of `ioConfig`, `inputFormat` must be changed from `"org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat"` to +`"org.apache.orc.mapreduce.OrcInputFormat"` +* The 'contrib' extension supported a `typeString` property, which provided the schema of the +ORC file, of which was essentially required to have the types correct, but notably _not_ the column names, which +facilitated column renaming. In the 'core' extension, column renaming can be achieved with +[`flattenSpec`](../../ingestion/index.md#flattenspec). For example, `"typeString":"struct"` +with the actual schema `struct<_col0:string,_col1:string>`, to preserve Druid schema would need replaced with: + +```json +"flattenSpec": { + "fields": [ + { + "type": "path", + "name": "time", + "expr": "$._col0" + }, + { + "type": "path", + "name": "name", + "expr": "$._col1" + } + ] + ... +} +``` + +* The 'contrib' extension supported a `mapFieldNameFormat` property, which provided a way to specify a dimension to + flatten `OrcMap` columns with primitive types. This functionality has also been replaced with + [`flattenSpec`](../../ingestion/index.md#flattenspec). For example: `"mapFieldNameFormat": "_"` + for a dimension `nestedData_dim1`, to preserve Druid schema could be replaced with + + ```json +"flattenSpec": { + "fields": [ + { + "type": "path", + "name": "nestedData_dim1", + "expr": "$.nestedData.dim1" + } + ] + ... +} +``` diff --git a/development/extensions-core/parquet.md b/development/extensions-core/parquet.md new file mode 100644 index 0000000..614e5dc --- /dev/null +++ b/development/extensions-core/parquet.md @@ -0,0 +1,36 @@ +--- +id: parquet +title: "Apache Parquet Extension" +--- + + + + +This Apache Druid module extends [Druid Hadoop based indexing](../../ingestion/hadoop.md) to ingest data directly from offline +Apache Parquet files. + +Note: If using the `parquet-avro` parser for Apache Hadoop based indexing, `druid-parquet-extensions` depends on the `druid-avro-extensions` module, so be sure to + [include both](../../development/extensions.md#loading-extensions). + +The `druid-parquet-extensions` provides the [Parquet input format](../../ingestion/data-formats.md#parquet), the [Parquet Hadoop parser](../../ingestion/data-formats.md#parquet-hadoop-parser), +and the [Parquet Avro Hadoop Parser](../../ingestion/data-formats.md#parquet-avro-hadoop-parser) with `druid-avro-extensions`. +The Parquet input format is available for [native batch ingestion](../../ingestion/native-batch.md) +and the other 2 parsers are for [Hadoop batch ingestion](../../ingestion/hadoop.md). +Please see corresponding docs for details. diff --git a/development/extensions-core/postgresql.md b/development/extensions-core/postgresql.md new file mode 100644 index 0000000..1fc2e20 --- /dev/null +++ b/development/extensions-core/postgresql.md @@ -0,0 +1,152 @@ +--- +id: postgresql +title: "PostgreSQL Metadata Store" +--- + + + + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `postgresql-metadata-storage` as an extension. + +## Setting up PostgreSQL + +1. Install PostgreSQL + + Use your favorite package manager to install PostgreSQL, e.g.: + - on Ubuntu/Debian using apt `apt-get install postgresql` + - on OS X, using [Homebrew](http://brew.sh/) `brew install postgresql` + +2. Create a druid database and user + + On the machine where PostgreSQL is installed, using an account with proper + postgresql permissions: + + Create a druid user, enter `diurd` when prompted for the password. + + ```bash + createuser druid -P + ``` + + Create a druid database owned by the user we just created + + ```bash + createdb druid -O druid + ``` + + *Note:* On Ubuntu / Debian you may have to prefix the `createuser` and + `createdb` commands with `sudo -u postgres` in order to gain proper + permissions. + +3. Configure your Druid metadata storage extension: + + Add the following parameters to your Druid configuration, replacing `` + with the location (host name and port) of the database. + + ```properties + druid.extensions.loadList=["postgresql-metadata-storage"] + druid.metadata.storage.type=postgresql + druid.metadata.storage.connector.connectURI=jdbc:postgresql:///druid + druid.metadata.storage.connector.user=druid + druid.metadata.storage.connector.password=diurd + ``` + +## Configuration + +In most cases, the configuration options map directly to the [postgres JDBC connection options](https://jdbc.postgresql.org/documentation/head/connect.html). + +|Property|Description|Default|Required| +|--------|-----------|-------|--------| +| `druid.metadata.postgres.ssl.useSSL` | Enables SSL | `false` | no | +| `druid.metadata.postgres.ssl.sslPassword` | The [Password Provider](../../operations/password-provider.md) or String password for the client's key. | none | no | +| `druid.metadata.postgres.ssl.sslFactory` | The class name to use as the `SSLSocketFactory` | none | no | +| `druid.metadata.postgres.ssl.sslFactoryArg` | An optional argument passed to the sslFactory's constructor | none | no | +| `druid.metadata.postgres.ssl.sslMode` | The sslMode. Possible values are "disable", "require", "verify-ca", "verify-full", "allow" and "prefer"| none | no | +| `druid.metadata.postgres.ssl.sslCert` | The full path to the certificate file. | none | no | +| `druid.metadata.postgres.ssl.sslKey` | The full path to the key file. | none | no | +| `druid.metadata.postgres.ssl.sslRootCert` | The full path to the root certificate. | none | no | +| `druid.metadata.postgres.ssl.sslHostNameVerifier` | The classname of the hostname verifier. | none | no | +| `druid.metadata.postgres.ssl.sslPasswordCallback` | The classname of the SSL password provider. | none | no | +| `druid.metadata.postgres.dbTableSchema` | druid meta table schema | `public` | no | + +### PostgreSQL Firehose + +The PostgreSQL extension provides an implementation of an [SqlFirehose](../../ingestion/native-batch.md#firehoses-deprecated) which can be used to ingest data into Druid from a PostgreSQL database. + +```json +{ + "type": "index_parallel", + "spec": { + "dataSchema": { + "dataSource": "some_datasource", + "parser": { + "parseSpec": { + "format": "timeAndDims", + "dimensionsSpec": { + "dimensionExclusions": [], + "dimensions": [ + "dim1", + "dim2", + "dim3" + ] + }, + "timestampSpec": { + "format": "auto", + "column": "ts" + } + } + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "DAY", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": null + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "type": "index_parallel", + "firehose": { + "type": "sql", + "database": { + "type": "postgresql", + "connectorConfig": { + "connectURI": "jdbc:postgresql://some-rds-host.us-west-1.rds.amazonaws.com:5432/druid", + "user": "admin", + "password": "secret" + } + }, + "sqls": [ + "SELECT * FROM some_table" + ] + } + }, + "tuningconfig": { + "type": "index_parallel" + } + } +} +``` \ No newline at end of file diff --git a/development/extensions-core/protobuf.md b/development/extensions-core/protobuf.md new file mode 100644 index 0000000..7515e50 --- /dev/null +++ b/development/extensions-core/protobuf.md @@ -0,0 +1,239 @@ +--- +id: protobuf +title: "Protobuf" +--- + + + + +This Apache Druid extension enables Druid to ingest and understand the Protobuf data format. Make sure to [include](../../development/extensions.md#loading-extensions) `druid-protobuf-extensions` as an extension. + +The `druid-protobuf-extensions` provides the [Protobuf Parser](../../ingestion/data-formats.md#protobuf-parser) +for [stream ingestion](../../ingestion/index.md#streaming). See corresponding docs for details. + +## Example: Load Protobuf messages from Kafka + +This example demonstrates how to load Protobuf messages from Kafka. Please read the [Load from Kafka tutorial](../../tutorials/tutorial-kafka.md) first, and see [Kafka Indexing Service](./kafka-ingestion.md) documentation for more details. + +The files used in this example are found at [`./examples/quickstart/protobuf` in your Druid directory](https://github.com/apache/druid/tree/master/examples/quickstart/protobuf). + +For this example: +- Kafka broker host is `localhost:9092` +- Kafka topic is `metrics_pb` +- Datasource name is `metrics-protobuf` + +Here is a JSON example of the 'metrics' data schema used in the example. + +```json +{ + "unit": "milliseconds", + "http_method": "GET", + "value": 44, + "timestamp": "2017-04-06T02:36:22Z", + "http_code": "200", + "page": "/", + "metricType": "request/latency", + "server": "www1.example.com" +} +``` + +### Proto file + +The corresponding proto file for our 'metrics' dataset looks like this. + +``` +syntax = "proto3"; +message Metrics { + string unit = 1; + string http_method = 2; + int32 value = 3; + string timestamp = 4; + string http_code = 5; + string page = 6; + string metricType = 7; + string server = 8; +} +``` + +### Descriptor file + +Next, we use the `protoc` Protobuf compiler to generate the descriptor file and save it as `metrics.desc`. The descriptor file must be either in the classpath or reachable by URL. In this example the descriptor file was saved at `/tmp/metrics.desc`, however this file is also available in the example files. From your Druid install directory: + +``` +protoc -o /tmp/metrics.desc ./quickstart/protobuf/metrics.proto +``` + +## Create Kafka Supervisor + +Below is the complete Supervisor spec JSON to be submitted to the Overlord. +Make sure these keys are properly configured for successful ingestion. + +Important supervisor properties +- `descriptor` for the descriptor file URL +- `protoMessageType` from the proto definition +- `parser` should have `type` set to `protobuf`, but note that the `format` of the `parseSpec` must be `json` + +```json +{ + "type": "kafka", + "dataSchema": { + "dataSource": "metrics-protobuf", + "parser": { + "type": "protobuf", + "descriptor": "file:///tmp/metrics.desc", + "protoMessageType": "Metrics", + "parseSpec": { + "format": "json", + "timestampSpec": { + "column": "timestamp", + "format": "auto" + }, + "dimensionsSpec": { + "dimensions": [ + "unit", + "http_method", + "http_code", + "page", + "metricType", + "server" + ], + "dimensionExclusions": [ + "timestamp", + "value" + ] + } + } + }, + "metricsSpec": [ + { + "name": "count", + "type": "count" + }, + { + "name": "value_sum", + "fieldName": "value", + "type": "doubleSum" + }, + { + "name": "value_min", + "fieldName": "value", + "type": "doubleMin" + }, + { + "name": "value_max", + "fieldName": "value", + "type": "doubleMax" + } + ], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": "NONE" + } + }, + "tuningConfig": { + "type": "kafka", + "maxRowsPerSegment": 5000000 + }, + "ioConfig": { + "topic": "metrics_pb", + "consumerProperties": { + "bootstrap.servers": "localhost:9092" + }, + "taskCount": 1, + "replicas": 1, + "taskDuration": "PT1H" + } +} +``` + +## Adding Protobuf messages to Kafka + +If necessary, from your Kafka installation directory run the following command to create the Kafka topic + +``` +./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic metrics_pb +``` + +This example script requires `protobuf` and `kafka-python` modules. With the topic in place, messages can be inserted running the following command from your Druid installation directory + +``` +./bin/generate-example-metrics | ./quickstart/protobuf/pb_publisher.py +``` + +You can confirm that data has been inserted to your Kafka topic using the following command from your Kafka installation directory + +``` +./bin/kafka-console-consumer --zookeeper localhost --topic metrics_pb +``` + +which should print messages like this + +``` +millisecondsGETR"2017-04-06T03:23:56Z*2002/list:request/latencyBwww1.example.com +``` + +If your supervisor created in the previous step is running, the indexing tasks should begin producing the messages and the data will soon be available for querying in Druid. + +## Generating the example files + +The files provided in the example quickstart can be generated in the following manner starting with only `metrics.proto`. + +### `metrics.desc` + +The descriptor file is generated using `protoc` Protobuf compiler. Given a `.proto` file, a `.desc` file can be generated like so. + +``` +protoc -o metrics.desc metrics.proto +``` + +### `metrics_pb2.py` +`metrics_pb2.py` is also generated with `protoc` + +``` + protoc -o metrics.desc metrics.proto --python_out=. +``` + +### `pb_publisher.py` +After `metrics_pb2.py` is generated, another script can be constructed to parse JSON data, convert it to Protobuf, and produce to a Kafka topic + +```python +#!/usr/bin/env python + +import sys +import json + +from kafka import KafkaProducer +from metrics_pb2 import Metrics + + +producer = KafkaProducer(bootstrap_servers='localhost:9092') +topic = 'metrics_pb' + +for row in iter(sys.stdin): + d = json.loads(row) + metrics = Metrics() + for k, v in d.items(): + setattr(metrics, k, v) + pb = metrics.SerializeToString() + producer.send(topic, pb) + +producer.flush() +``` diff --git a/development/extensions-core/s3.md b/development/extensions-core/s3.md new file mode 100644 index 0000000..b30ba4c --- /dev/null +++ b/development/extensions-core/s3.md @@ -0,0 +1,126 @@ +--- +id: s3 +title: "S3-compatible" +--- + + + +## S3 extension + +This extension allows you to do 2 things: +* [Ingest data](#reading-data-from-s3) from files stored in S3. +* Write segments to [deep storage](#deep-storage) in S3. + +To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `druid-s3-extensions` as an extension. + +### Reading data from S3 + +The [S3 input source](../../ingestion/native-batch.md#s3-input-source) is supported by the [Parallel task](../../ingestion/native-batch.md#parallel-task) +to read objects directly from S3. If you use the [Hadoop task](../../ingestion/hadoop.md), +you can read data from S3 by specifying the S3 paths in your [`inputSpec`](../../ingestion/hadoop.md#inputspec). + +To configure the extension to read objects from S3 you need to configure how to [connect to S3](#configuration). + +### Deep Storage + +S3-compatible deep storage means either AWS S3 or a compatible service like Google Storage which exposes the same API as S3. + +S3 deep storage needs to be explicitly enabled by setting `druid.storage.type=s3`. **Only after setting the storage type to S3 will any of the settings below take effect.** + +To correctly configure this extension for deep storage in S3, first configure how to [connect to S3](#configuration). +In addition to this you need to set additional configuration, specific for [deep storage](#deep-storage-specific-configuration) + +#### Deep storage specific configuration + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.storage.bucket`|Bucket to store in.|Must be set.| +|`druid.storage.baseKey`|A prefix string that will be prepended to the object names for the segments published to S3 deep storage|Must be set.| +|`druid.storage.type`|Global deep storage provider. Must be set to `s3` to make use of this extension.|Must be set (likely `s3`).| +|`druid.storage.archiveBucket`|S3 bucket name for archiving when running the *archive task*.|none| +|`druid.storage.archiveBaseKey`|S3 object key prefix for archiving.|none| +|`druid.storage.disableAcl`|Boolean flag to disable ACL. If this is set to `false`, the full control would be granted to the bucket owner. This may require to set additional permissions. See [S3 permissions settings](#s3-permissions-settings).|false| +|`druid.storage.useS3aSchema`|If true, use the "s3a" filesystem when using Hadoop-based ingestion. If false, the "s3n" filesystem will be used. Only affects Hadoop-based ingestion.|false| + +## Configuration + +### S3 authentication methods + +Druid uses the following credentials provider chain to connect to your S3 bucket (whether a deep storage bucket or source bucket). +**Note :** *You can override the default credentials provider chain for connecting to source bucket by specifying an access key and secret key using [Properties Object](../../ingestion/native-batch.md#s3-input-source) parameters in the ingestionSpec.* + +|order|type|details| +|--------|-----------|-------| +|1|Druid config file|Based on your runtime.properties if it contains values `druid.s3.accessKey` and `druid.s3.secretKey` | +|2|Custom properties file| Based on custom properties file where you can supply `sessionToken`, `accessKey` and `secretKey` values. This file is provided to Druid through `druid.s3.fileSessionCredentials` properties| +|3|Environment variables|Based on environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`| +|4|Java system properties|Based on JVM properties `aws.accessKeyId` and `aws.secretKey` | +|5|Profile information|Based on credentials you may have on your druid instance (generally in `~/.aws/credentials`)| +|6|ECS container credentials|Based on environment variables available on AWS ECS (AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI) as described in the [EC2ContainerCredentialsProviderWrapper documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/EC2ContainerCredentialsProviderWrapper.html)| +|7|Instance profile information|Based on the instance profile you may have attached to your druid instance| + +You can find more information about authentication method [here](https://docs.aws.amazon.com/fr_fr/sdk-for-java/v1/developer-guide/credentials.html)
+**Note :** *Order is important here as it indicates the precedence of authentication methods.
+So if you are trying to use Instance profile information, you **must not** set `druid.s3.accessKey` and `druid.s3.secretKey` in your Druid runtime.properties* + + +### S3 permissions settings + +`s3:GetObject` and `s3:PutObject` are basically required for pushing/loading segments to/from S3. +If `druid.storage.disableAcl` is set to `false`, then `s3:GetBucketAcl` and `s3:PutObjectAcl` are additionally required to set ACL for objects. + +### AWS region + +The AWS SDK requires that the target region be specified. Two ways of doing this are by using the JVM system property `aws.region` or the environment variable `AWS_REGION`. + +As an example, to set the region to 'us-east-1' through system properties: + +- Add `-Daws.region=us-east-1` to the jvm.config file for all Druid services. +- Add `-Daws.region=us-east-1` to `druid.indexer.runner.javaOpts` in [Middle Manager configuration](../../configuration/index.md#middlemanager-configuration) so that the property will be passed to Peon (worker) processes. + +### Connecting to S3 configuration + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.s3.accessKey`|S3 access key. See [S3 authentication methods](#s3-authentication-methods) for more details|Can be omitted according to authentication methods chosen.| +|`druid.s3.secretKey`|S3 secret key. See [S3 authentication methods](#s3-authentication-methods) for more details|Can be omitted according to authentication methods chosen.| +|`druid.s3.fileSessionCredentials`|Path to properties file containing `sessionToken`, `accessKey` and `secretKey` value. One key/value pair per line (format `key=value`). See [S3 authentication methods](#s3-authentication-methods) for more details |Can be omitted according to authentication methods chosen.| +|`druid.s3.protocol`|Communication protocol type to use when sending requests to AWS. `http` or `https` can be used. This configuration would be ignored if `druid.s3.endpoint.url` is filled with a URL with a different protocol.|`https`| +|`druid.s3.disableChunkedEncoding`|Disables chunked encoding. See [AWS document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#disableChunkedEncoding--) for details.|false| +|`druid.s3.enablePathStyleAccess`|Enables path style access. See [AWS document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#enablePathStyleAccess--) for details.|false| +|`druid.s3.forceGlobalBucketAccessEnabled`|Enables global bucket access. See [AWS document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#setForceGlobalBucketAccessEnabled-java.lang.Boolean-) for details.|false| +|`druid.s3.endpoint.url`|Service endpoint either with or without the protocol.|None| +|`druid.s3.endpoint.signingRegion`|Region to use for SigV4 signing of requests (e.g. us-west-1).|None| +|`druid.s3.proxy.host`|Proxy host to connect through.|None| +|`druid.s3.proxy.port`|Port on the proxy host to connect through.|None| +|`druid.s3.proxy.username`|User name to use when connecting through a proxy.|None| +|`druid.s3.proxy.password`|Password to use when connecting through a proxy.|None| +|`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| +|`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| +|`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| + +## Server-side encryption + +You can enable [server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) by setting +`druid.storage.sse.type` to a supported type of server-side encryption. The current supported types are: + +- s3: [Server-side encryption with S3-managed encryption keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) +- kms: [Server-side encryption with AWS KMS–Managed Keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) +- custom: [Server-side encryption with Customer-Provided Encryption Keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html) diff --git a/development/extensions-core/simple-client-sslcontext.md b/development/extensions-core/simple-client-sslcontext.md new file mode 100644 index 0000000..7452a7b --- /dev/null +++ b/development/extensions-core/simple-client-sslcontext.md @@ -0,0 +1,52 @@ +--- +id: simple-client-sslcontext +title: "Simple SSLContext Provider Module" +--- + + + + +This Apache Druid module contains a simple implementation of [SSLContext](http://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLContext.html) +that will be injected to be used with HttpClient that Druid processes use internally to communicate with each other. To learn more about +Java's SSL support, please refer to [this](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) guide. + + +|Property|Description|Default|Required| +|--------|-----------|-------|--------| +|`druid.client.https.protocol`|SSL protocol to use.|`TLSv1.2`|no| +|`druid.client.https.trustStoreType`|The type of the key store where trusted root certificates are stored.|`java.security.KeyStore.getDefaultType()`|no| +|`druid.client.https.trustStorePath`|The file path or URL of the TLS/SSL Key store where trusted root certificates are stored.|none|yes| +|`druid.client.https.trustStoreAlgorithm`|Algorithm to be used by TrustManager to validate certificate chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|no| +|`druid.client.https.trustStorePassword`|The [Password Provider](../../operations/password-provider.md) or String password for the Trust Store.|none|yes| + +The following table contains optional parameters for supporting client certificate authentication: + +|Property|Description|Default|Required| +|--------|-----------|-------|--------| +|`druid.client.https.keyStorePath`|The file path or URL of the TLS/SSL Key store containing the client certificate that Druid will use when communicating with other Druid services. If this is null, the other properties in this table are ignored.|none|yes| +|`druid.client.https.keyStoreType`|The type of the key store.|none|yes| +|`druid.client.https.certAlias`|Alias of TLS client certificate in the keystore.|none|yes| +|`druid.client.https.keyStorePassword`|The [Password Provider](../../operations/password-provider.md) or String password for the Key Store.|none|no| +|`druid.client.https.keyManagerFactoryAlgorithm`|Algorithm to use for creating KeyManager, more details [here](https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#KeyManager).|`javax.net.ssl.KeyManagerFactory.getDefaultAlgorithm()`|no| +|`druid.client.https.keyManagerPassword`|The [Password Provider](../../operations/password-provider.md) or String password for the Key Manager.|none|no| +|`druid.client.https.validateHostnames`|Validate the hostname of the server. This should not be disabled unless you are using [custom TLS certificate checks](../../operations/tls-support.md) and know that standard hostname validation is not needed.|true|no| + +This [document](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html) lists all the possible +values for the above mentioned configs among others provided by Java implementation. diff --git a/development/extensions-core/stats.md b/development/extensions-core/stats.md new file mode 100644 index 0000000..61cb698 --- /dev/null +++ b/development/extensions-core/stats.md @@ -0,0 +1,171 @@ +--- +id: stats +title: "Stats aggregator" +--- + + + + +This Apache Druid extension includes stat-related aggregators, including variance and standard deviations, etc. Make sure to [include](../../development/extensions.md#loading-extensions) `druid-stats` as an extension. + +## Variance aggregator + +Algorithm of the aggregator is the same with that of apache hive. This is the description in GenericUDAFVariance in hive. + +Evaluate the variance using the algorithm described by Chan, Golub, and LeVeque in +"Algorithms for computing the sample variance: analysis and recommendations" +The American Statistician, 37 (1983) pp. 242--247. + +variance = variance1 + variance2 + n/(m*(m+n)) * pow(((m/n)*t1 - t2),2) + +where: - variance is sum(x-avg^2) (this is actually n times the variance) +and is updated at every step. - n is the count of elements in chunk1 - m is +the count of elements in chunk2 - t1 = sum of elements in chunk1, t2 = +sum of elements in chunk2. + +This algorithm was proven to be numerically stable by J.L. Barlow in +"Error analysis of a pairwise summation algorithm to compute sample variance" +Numer. Math, 58 (1991) pp. 583--590 + +### Pre-aggregating variance at ingestion time + +To use this feature, an "variance" aggregator must be included at indexing time. +The ingestion aggregator can only apply to numeric values. If you use "variance" +then any input rows missing the value will be considered to have a value of 0. + +User can specify expected input type as one of "float", "double", "long", "variance" for ingestion, which is by default "float". + +```json +{ + "type" : "variance", + "name" : , + "fieldName" : , + "inputType" : , + "estimator" : +} +``` + +To query for results, "variance" aggregator with "variance" input type or simply a "varianceFold" aggregator must be included in the query. + +```json +{ + "type" : "varianceFold", + "name" : , + "fieldName" : , + "estimator" : +} +``` + +|Property |Description |Default | +|-------------------------|------------------------------|----------------------------------| +|`estimator`|Set "population" to get variance_pop rather than variance_sample, which is default.|null| + + +### Standard deviation post-aggregator + +To acquire standard deviation from variance, user can use "stddev" post aggregator. + +```json +{ + "type": "stddev", + "name": "", + "fieldName": "", + "estimator": +} +``` + +## Query examples: + +### Timeseries query + +```json +{ + "queryType": "timeseries", + "dataSource": "testing", + "granularity": "day", + "aggregations": [ + { + "type": "variance", + "name": "index_var", + "fieldName": "index_var" + } + ], + "intervals": [ + "2016-03-01T00:00:00.000/2013-03-20T00:00:00.000" + ] +} +``` + +### TopN query + +```json +{ + "queryType": "topN", + "dataSource": "testing", + "dimensions": ["alias"], + "threshold": 5, + "granularity": "all", + "aggregations": [ + { + "type": "variance", + "name": "index_var", + "fieldName": "index" + } + ], + "postAggregations": [ + { + "type": "stddev", + "name": "index_stddev", + "fieldName": "index_var" + } + ], + "intervals": [ + "2016-03-06T00:00:00/2016-03-06T23:59:59" + ] +} +``` + +### GroupBy query + +```json +{ + "queryType": "groupBy", + "dataSource": "testing", + "dimensions": ["alias"], + "granularity": "all", + "aggregations": [ + { + "type": "variance", + "name": "index_var", + "fieldName": "index" + } + ], + "postAggregations": [ + { + "type": "stddev", + "name": "index_stddev", + "fieldName": "index_var" + } + ], + "intervals": [ + "2016-03-06T00:00:00/2016-03-06T23:59:59" + ] +} +``` diff --git a/development/extensions-core/test-stats.md b/development/extensions-core/test-stats.md new file mode 100644 index 0000000..9f069d8 --- /dev/null +++ b/development/extensions-core/test-stats.md @@ -0,0 +1,117 @@ +--- +id: test-stats +title: "Test Stats Aggregators" +--- + + + + +This Apache Druid extension incorporates test statistics related aggregators, including z-score and p-value. Please refer to [https://www.paypal-engineering.com/2017/06/29/democratizing-experimentation-data-for-product-innovations/](https://www.paypal-engineering.com/2017/06/29/democratizing-experimentation-data-for-product-innovations/) for math background and details. + +Make sure to include `druid-stats` extension in order to use these aggregators. + +## Z-Score for two sample ztests post aggregator + +Please refer to [https://www.isixsigma.com/tools-templates/hypothesis-testing/making-sense-two-proportions-test/](https://www.isixsigma.com/tools-templates/hypothesis-testing/making-sense-two-proportions-test/) and [http://www.ucs.louisiana.edu/~jcb0773/Berry_statbook/Berry_statbook_chpt6.pdf](http://www.ucs.louisiana.edu/~jcb0773/Berry_statbook/Berry_statbook_chpt6.pdf) for more details. + +z = (p1 - p2) / S.E. (assuming null hypothesis is true) + +Please see below for p1 and p2. +Please note S.E. stands for standard error where + +S.E. = sqrt{ p1 * ( 1 - p1 )/n1 + p2 * (1 - p2)/n2) } + +(p1 – p2) is the observed difference between two sample proportions. + +### zscore2sample post aggregator +* **`zscore2sample`**: calculate the z-score using two-sample z-test while converting binary variables (***e.g.*** success or not) to continuous variables (***e.g.*** conversion rate). + +```json +{ + "type": "zscore2sample", + "name": "", + "successCount1": success count of sample 1, + "sample1Size": sample 1 size, + "successCount2": success count of sample 2, + "sample2Size" : sample 2 size +} +``` + +Please note the post aggregator will be converting binary variables to continuous variables for two population proportions. Specifically + +p1 = (successCount1) / (sample size 1) + +p2 = (successCount2) / (sample size 2) + +### pvalue2tailedZtest post aggregator + +* **`pvalue2tailedZtest`**: calculate p-value of two-sided z-test from zscore + - ***pvalue2tailedZtest(zscore)*** - the input is a z-score which can be calculated using the zscore2sample post aggregator + + +```json +{ + "type": "pvalue2tailedZtest", + "name": "", + "zScore": +} +``` + +## Example Usage + +In this example, we use zscore2sample post aggregator to calculate z-score, and then feed the z-score to pvalue2tailedZtest post aggregator to calculate p-value. + +A JSON query example can be as follows: + +```json +{ + ... + "postAggregations" : { + "type" : "pvalue2tailedZtest", + "name" : "pvalue", + "zScore" : + { + "type" : "zscore2sample", + "name" : "zscore", + "successCount1" : + { "type" : "constant", + "name" : "successCountFromPopulation1Sample", + "value" : 300 + }, + "sample1Size" : + { "type" : "constant", + "name" : "sampleSizeOfPopulation1", + "value" : 500 + }, + "successCount2": + { "type" : "constant", + "name" : "successCountFromPopulation2Sample", + "value" : 450 + }, + "sample2Size" : + { "type" : "constant", + "name" : "sampleSizeOfPopulation2", + "value" : 600 + } + } + } +} + +``` diff --git a/Development/extensions.md b/development/extensions.md similarity index 100% rename from Development/extensions.md rename to development/extensions.md diff --git a/development/index.md b/development/index.md new file mode 100644 index 0000000..874eccc --- /dev/null +++ b/development/index.md @@ -0,0 +1,3 @@ +# Druid 开发指南 + +[开发概述](overview.md ':include') \ No newline at end of file diff --git a/development/modules.md b/development/modules.md new file mode 100644 index 0000000..8b34fc3 --- /dev/null +++ b/development/modules.md @@ -0,0 +1,339 @@ +# 创建 Druid 扩展 + +Druid 使用一个定义的 模块（module）系统来允许在运行时（runtime）使用扩展来（extensions）扩充功能。 + +## Writing your own extensions + +Druid's extensions leverage Guice in order to add things at runtime. +Basically, Guice is a framework for Dependency Injection, but we use it to hold the expected object graph of the Druid process. +Extensions can make any changes they want/need to the object graph via adding Guice bindings. +While the extensions actually give you the capability to change almost anything however you want, in general, we expect people to want to extend one of the things listed below. +This means that we honor our [versioning strategy](./versioning.md) for changes that affect the interfaces called out on this page, but other interfaces are deemed "internal" and can be changed in an incompatible manner even between patch releases. + +1. Add a new deep storage implementation by extending the `org.apache.druid.segment.loading.DataSegment*` and + `org.apache.druid.tasklogs.TaskLog*` classes. +1. Add a new input source by extending `org.apache.druid.data.input.InputSource`. +1. Add a new input entity by extending `org.apache.druid.data.input.InputEntity`. +1. Add a new input source reader if necessary by extending `org.apache.druid.data.input.InputSourceReader`. You can use `org.apache.druid.data.input.impl.InputEntityIteratingReader` in most cases. +1. Add a new input format by extending `org.apache.druid.data.input.InputFormat`. +1. Add a new input entity reader by extending `org.apache.druid.data.input.TextReader` for text formats or `org.apache.druid.data.input.IntermediateRowParsingReader` for binary formats. +1. Add Aggregators by extending `org.apache.druid.query.aggregation.AggregatorFactory`, `org.apache.druid.query.aggregation.Aggregator`, + and `org.apache.druid.query.aggregation.BufferAggregator`. +1. Add PostAggregators by extending `org.apache.druid.query.aggregation.PostAggregator`. +1. Add ExtractionFns by extending `org.apache.druid.query.extraction.ExtractionFn`. +1. Add Complex metrics by extending `org.apache.druid.segment.serde.ComplexMetricSerde`. +1. Add new Query types by extending `org.apache.druid.query.QueryRunnerFactory`, `org.apache.druid.query.QueryToolChest`, and + `org.apache.druid.query.Query`. +1. Add new Jersey resources by calling `Jerseys.addResource(binder, clazz)`. +1. Add new Jetty filters by extending `org.apache.druid.server.initialization.jetty.ServletFilterHolder`. +1. Add new secret providers by extending `org.apache.druid.metadata.PasswordProvider`. +1. Add new ingest transform by implementing the `org.apache.druid.segment.transform.Transform` interface from the `druid-processing` package. +1. Bundle your extension with all the other Druid extensions + +Extensions are added to the system via an implementation of `org.apache.druid.initialization.DruidModule`. + +### Creating a Druid Module + +The DruidModule class is has two methods + +1. A `configure(Binder)` method +2. A `getJacksonModules()` method + +The `configure(Binder)` method is the same method that a normal Guice module would have. + +The `getJacksonModules()` method provides a list of Jackson modules that are used to help initialize the Jackson ObjectMapper instances used by Druid. This is how you add extensions that are instantiated via Jackson (like AggregatorFactory and InputSource objects) to Druid. + +### Registering your Druid Module + +Once you have your DruidModule created, you will need to package an extra file in the `META-INF/services` directory of your jar. This is easiest to accomplish with a maven project by creating files in the `src/main/resources` directory. There are examples of this in the Druid code under the `cassandra-storage`, `hdfs-storage` and `s3-extensions` modules, for examples. + +The file that should exist in your jar is + +`META-INF/services/org.apache.druid.initialization.DruidModule` + +It should be a text file with a new-line delimited list of package-qualified classes that implement DruidModule like + +``` +org.apache.druid.storage.cassandra.CassandraDruidModule +``` + +If your jar has this file, then when it is added to the classpath or as an extension, Druid will notice the file and will instantiate instances of the Module. Your Module should have a default constructor, but if you need access to runtime configuration properties, it can have a method with @Inject on it to get a Properties object injected into it from Guice. + +### Adding a new deep storage implementation + +Check the `azure-storage`, `google-storage`, `cassandra-storage`, `hdfs-storage` and `s3-extensions` modules for examples of how to do this. + +The basic idea behind the extension is that you need to add bindings for your DataSegmentPusher and DataSegmentPuller objects. The way to add them is something like (taken from HdfsStorageDruidModule) + +``` java +Binders.dataSegmentPullerBinder(binder) + .addBinding("hdfs") + .to(HdfsDataSegmentPuller.class).in(LazySingleton.class); + +Binders.dataSegmentPusherBinder(binder) + .addBinding("hdfs") + .to(HdfsDataSegmentPusher.class).in(LazySingleton.class); +``` + +`Binders.dataSegment*Binder()` is a call provided by the druid-core jar which sets up a Guice multibind "MapBinder". If that doesn't make sense, don't worry about it, just think of it as a magical incantation. + +`addBinding("hdfs")` for the Puller binder creates a new handler for loadSpec objects of type "hdfs". For the Pusher binder it creates a new type value that you can specify for the `druid.storage.type` parameter. + +`to(...).in(...);` is normal Guice stuff. + +In addition to DataSegmentPusher and DataSegmentPuller, you can also bind: + +* DataSegmentKiller: Removes segments, used as part of the Kill Task to delete unused segments, i.e. perform garbage collection of segments that are either superseded by newer versions or that have been dropped from the cluster. +* DataSegmentMover: Allow migrating segments from one place to another, currently this is only used as part of the MoveTask to move unused segments to a different S3 bucket or prefix, typically to reduce storage costs of unused data (e.g. move to glacier or cheaper storage) +* DataSegmentArchiver: Just a wrapper around Mover, but comes with a pre-configured target bucket/path, so it doesn't have to be specified at runtime as part of the ArchiveTask. + +### Validating your deep storage implementation + +**WARNING!** This is not a formal procedure, but a collection of hints to validate if your new deep storage implementation is able do push, pull and kill segments. + +It's recommended to use batch ingestion tasks to validate your implementation. +The segment will be automatically rolled up to Historical note after ~20 seconds. +In this way, you can validate both push (at realtime process) and pull (at Historical process) segments. + +* DataSegmentPusher + +Wherever your data storage (cloud storage service, distributed file system, etc.) is, you should be able to see one new file: `index.zip` (`partitionNum_index.zip` for HDFS data storage) after your ingestion task ends. + +* DataSegmentPuller + +After ~20 secs your ingestion task ends, you should be able to see your Historical process trying to load the new segment. + +The following example was retrieved from a Historical process configured to use Azure for deep storage: + +``` +2015-04-14T02:42:33,450 INFO [ZkCoordinator-0] org.apache.druid.server.coordination.ZkCoordinator - New request[LOAD: dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00 +.000Z_2015-04-14T02:41:09.484Z] with zNode[/druid/dev/loadQueue/192.168.33.104:8081/dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_2015-04-14T02:41:09. +484Z]. +2015-04-14T02:42:33,451 INFO [ZkCoordinator-0] org.apache.druid.server.coordination.ZkCoordinator - Loading segment dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.0 +00Z_2015-04-14T02:41:09.484Z +2015-04-14T02:42:33,463 INFO [ZkCoordinator-0] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.storage.azure.AzureAccountConfig] from props[drui +d.azure.] as [org.apache.druid.storage.azure.AzureAccountConfig@759c9ad9] +2015-04-14T02:49:08,275 INFO [ZkCoordinator-0] org.apache.druid.utils.CompressionUtils - Unzipping file[/opt/druid/tmp/compressionUtilZipCache1263964429587449785.z +ip] to [/opt/druid/zk_druid/dde/2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z/2015-04-14T02:41:09.484Z/0] +2015-04-14T02:49:08,276 INFO [ZkCoordinator-0] org.apache.druid.storage.azure.AzureDataSegmentPuller - Loaded 1196 bytes from [dde/2015-01-02T00:00:00.000Z_2015-01-03 +T00:00:00.000Z/2015-04-14T02:41:09.484Z/0/index.zip] to [/opt/druid/zk_druid/dde/2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z/2015-04-14T02:41:09.484Z/0] +2015-04-14T02:49:08,277 WARN [ZkCoordinator-0] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Segment [dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_2015-04-14T02:41:09.484Z] is different than expected size. Expected [0] found [1196] +2015-04-14T02:49:08,282 INFO [ZkCoordinator-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_2015-04-14T02:41:09.484Z] at path[/druid/dev/segments/192.168.33.104:8081/192.168.33.104:8081_historical__default_tier_2015-04-14T02:49:08.282Z_7bb87230ebf940188511dd4a53ffd7351] +2015-04-14T02:49:08,292 INFO [ZkCoordinator-0] org.apache.druid.server.coordination.ZkCoordinator - Completed request [LOAD: dde_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_2015-04-14T02:41:09.484Z] +``` + +* DataSegmentKiller + +The easiest way of testing the segment killing is marking a segment as not used and then starting a killing task through the old Coordinator console. + +To mark a segment as not used, you need to connect to your metadata storage and update the `used` column to `false` on the segment table rows. + +To start a segment killing task, you need to access the old Coordinator console `http://: getJacksonModules() +{ + return ImmutableList.of( + new SimpleModule().registerSubtypes(new NamedType(S3InputSource.class, "s3")) + ); +} +``` + +This is registering the InputSource with Jackson's polymorphic serialization/deserialization layer. More concretely, having this will mean that if you specify a `"inputSource": { "type": "s3", ... }` in your IO config, then the system will load this InputSource for your `InputSource` implementation. + +Note that inside of Druid, we have made the `@JacksonInject` annotation for Jackson deserialized objects actually use the base Guice injector to resolve the object to be injected. So, if your InputSource needs access to some object, you can add a `@JacksonInject` annotation on a setter and it will get set on instantiation. + +### Adding support for a new data format + +Adding support for a new data format requires implementing two interfaces, i.e., `InputFormat` and `InputEntityReader`. +`InputFormat` is to define how your data is formatted. `InputEntityReader` is to define how to parse your data and convert into Druid `InputRow`. + +There is an example in the `druid-orc-extensions` module with the `OrcInputFormat` and `OrcReader`. + +Adding an InputFormat is very similar to adding an InputSource. They operate purely through Jackson and thus should just be additions to the Jackson modules returned by your DruidModule. + +### Adding Aggregators + +Adding AggregatorFactory objects is very similar to InputSource objects. They operate purely through Jackson and thus should just be additions to the Jackson modules returned by your DruidModule. + +### Adding Complex Metrics + +Adding ComplexMetrics is a little ugly in the current version. The method of getting at complex metrics is through registration with the `ComplexMetrics.registerSerde()` method. There is no special Guice stuff to get this working, just in your `configure(Binder)` method register the serialization/deserialization. + +### Adding new Query types + +Adding a new Query type requires the implementation of three interfaces. + +1. `org.apache.druid.query.Query` +1. `org.apache.druid.query.QueryToolChest` +1. `org.apache.druid.query.QueryRunnerFactory` + +Registering these uses the same general strategy as a deep storage mechanism does. You do something like + +``` java +DruidBinders.queryToolChestBinder(binder) + .addBinding(SegmentMetadataQuery.class) + .to(SegmentMetadataQueryQueryToolChest.class); + +DruidBinders.queryRunnerFactoryBinder(binder) + .addBinding(SegmentMetadataQuery.class) + .to(SegmentMetadataQueryRunnerFactory.class); +``` + +The first one binds the SegmentMetadataQueryQueryToolChest for usage when a SegmentMetadataQuery is used. The second one does the same thing but for the QueryRunnerFactory instead. + +### Adding new Jersey resources + +Adding new Jersey resources to a module requires calling the following code to bind the resource in the module: + +```java +Jerseys.addResource(binder, NewResource.class); +``` + +### Adding a new Password Provider implementation + +You will need to implement `org.apache.druid.metadata.PasswordProvider` interface. For every place where Druid uses PasswordProvider, a new instance of the implementation will be created, +thus make sure all the necessary information required for fetching each password is supplied during object instantiation. +In your implementation of `org.apache.druid.initialization.DruidModule`, `getJacksonModules` should look something like this - + +``` java + return ImmutableList.of( + new SimpleModule("SomePasswordProviderModule") + .registerSubtypes( + new NamedType(SomePasswordProvider.class, "some") + ) + ); +``` + +where `SomePasswordProvider` is the implementation of `PasswordProvider` interface, you can have a look at `org.apache.druid.metadata.EnvironmentVariablePasswordProvider` for example. + +### Adding a Transform Extension + +To create a transform extension implement the `org.apache.druid.segment.transform.Transform` interface. You'll need to install the `druid-processing` package to import `org.apache.druid.segment.transform`. + +```java +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import org.apache.druid.segment.transform.RowFunction; +import org.apache.druid.segment.transform.Transform; + +public class MyTransform implements Transform { + private final String name; + + @JsonCreator + public MyTransform( + @JsonProperty("name") final String name + ) { + this.name = name; + } + + @JsonProperty + @Override + public String getName() { + return name; + } + + @Override + public RowFunction getRowFunction() { + return new MyRowFunction(); + } + + static class MyRowFunction implements RowFunction { + @Override + public Object eval(Row row) { + return "transformed-value"; + } + } +} +``` + +Then register your transform as a Jackson module. + +```java +import com.fasterxml.jackson.databind.Module; +import com.fasterxml.jackson.databind.jsontype.NamedModule; +import com.fasterxml.jackson.databind.module.SimpleModule; +import com.google.inject.Binder; +import com.google.common.collect.ImmutableList; +import org.apache.druid.initialization.DruidModule; + +public class MyTransformModule implements DruidModule { + @Override + public List getJacksonModules() { + return return ImmutableList.of( + new SimpleModule("MyTransformModule").registerSubtypes( + new NamedType(MyTransform.class, "my-transform") + ) + ): + } + + @Override + public void configure(Binder binder) { + } +} +``` + +### Bundle your extension with all the other Druid extensions + +When you do `mvn install`, Druid extensions will be packaged within the Druid tarball and `extensions` directory, which are both underneath `distribution/target/`. + +If you want your extension to be included, you can add your extension's maven coordinate as an argument at +[distribution/pom.xml](https://github.com/apache/druid/blob/master/distribution/pom.xml#L95) + +During `mvn install`, maven will install your extension to the local maven repository, and then call [pull-deps](../operations/pull-deps.md) to pull your extension from +there. In the end, you should see your extension underneath `distribution/target/extensions` and within Druid tarball. + +### 管理依赖 + +针对扩展的依赖和常用依赖冲突的管理可能让人非常头疼。Managing library collisions can be daunting for extensions which draw in commonly used libraries. +针对下面的库的 group IDs 我们建议在 Maven 的 scope 使用 `provided` 来避免与 Druid 中使用的相同包的依赖产生冲突： + +``` +"org.apache.druid", +"com.metamx.druid", +"asm", +"org.ow2.asm", +"org.jboss.netty", +"com.google.guava", +"com.google.code.findbugs", +"com.google.protobuf", +"com.esotericsoftware.minlog", +"log4j", +"org.slf4j", +"commons-logging", +"org.eclipse.jetty", +"org.mortbay.jetty", +"com.sun.jersey", +"com.sun.jersey.contribs", +"common-beanutils", +"commons-codec", +"commons-lang", +"commons-cli", +"commons-io", +"javax.activation", +"org.apache.httpcomponents", +"org.apache.zookeeper", +"org.codehaus.jackson", +"com.fasterxml.jackson", +"com.fasterxml.jackson.core", +"com.fasterxml.jackson.dataformat", +"com.fasterxml.jackson.datatype", +"org.roaringbitmap", +"net.java.dev.jets3t" +``` +请查看源代码 `org.apache.druid.cli.PullDependencies` 中的注释文档来获得更多的信息。 + diff --git a/Development/orc-extensions.md b/development/orc-extensions.md similarity index 100% rename from Development/orc-extensions.md rename to development/orc-extensions.md diff --git a/development/overview.md b/development/overview.md new file mode 100644 index 0000000..8d9bdb1 --- /dev/null +++ b/development/overview.md @@ -0,0 +1,157 @@ +--- +id: extensions +title: "Extensions" +--- + + + + +Druid implements an extension system that allows for adding functionality at runtime. Extensions +are commonly used to add support for deep storages (like HDFS and S3), metadata stores (like MySQL +and PostgreSQL), new aggregators, new input formats, and so on. + +Production clusters will generally use at least two extensions; one for deep storage and one for a +metadata store. Many clusters will also use additional extensions. + +## Core extensions + +Core extensions are maintained by Druid committers. + +|Name|Description|Docs| +|----|-----------|----| +|druid-avro-extensions|Support for data in Apache Avro data format.|[link](../development/extensions-core/avro.md)| +|druid-azure-extensions|Microsoft Azure deep storage.|[link](../development/extensions-core/azure.md)| +|druid-basic-security|Support for Basic HTTP authentication and role-based access control.|[link](../development/extensions-core/druid-basic-security.md)| +|druid-bloom-filter|Support for providing Bloom filters in druid queries.|[link](../development/extensions-core/bloom-filter.md)| +|druid-datasketches|Support for approximate counts and set operations with [Apache DataSketches](https://datasketches.apache.org/).|[link](../development/extensions-core/datasketches-extension.md)| +|druid-google-extensions|Google Cloud Storage deep storage.|[link](../development/extensions-core/google.md)| +|druid-hdfs-storage|HDFS deep storage.|[link](../development/extensions-core/hdfs.md)| +|druid-histogram|Approximate histograms and quantiles aggregator. Deprecated, please use the [DataSketches quantiles aggregator](../development/extensions-core/datasketches-quantiles.md) from the `druid-datasketches` extension instead.|[link](../development/extensions-core/approximate-histograms.md)| +|druid-kafka-extraction-namespace|Apache Kafka-based namespaced lookup. Requires namespace lookup extension.|[link](../development/extensions-core/kafka-extraction-namespace.md)| +|druid-kafka-indexing-service|Supervised exactly-once Apache Kafka ingestion for the indexing service.|[link](../development/extensions-core/kafka-ingestion.md)| +|druid-kinesis-indexing-service|Supervised exactly-once Kinesis ingestion for the indexing service.|[link](../development/extensions-core/kinesis-ingestion.md)| +|druid-kerberos|Kerberos authentication for druid processes.|[link](../development/extensions-core/druid-kerberos.md)| +|druid-lookups-cached-global|A module for [lookups](../querying/lookups.md) providing a jvm-global eager caching for lookups. It provides JDBC and URI implementations for fetching lookup data.|[link](../development/extensions-core/lookups-cached-global.md)| +|druid-lookups-cached-single| Per lookup caching module to support the use cases where a lookup need to be isolated from the global pool of lookups |[link](../development/extensions-core/druid-lookups.md)| +|druid-orc-extensions|Support for data in Apache ORC data format.|[link](../development/extensions-core/orc.md)| +|druid-parquet-extensions|Support for data in Apache Parquet data format. Requires druid-avro-extensions to be loaded.|[link](../development/extensions-core/parquet.md)| +|druid-protobuf-extensions| Support for data in Protobuf data format.|[link](../development/extensions-core/protobuf.md)| +|druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-core/druid-ranger-security.md)| +|druid-s3-extensions|Interfacing with data in AWS S3, and using S3 as deep storage.|[link](../development/extensions-core/s3.md)| +|druid-ec2-extensions|Interfacing with AWS EC2 for autoscaling middle managers|UNDOCUMENTED| +|druid-stats|Statistics related module including variance and standard deviation.|[link](../development/extensions-core/stats.md)| +|mysql-metadata-storage|MySQL metadata store.|[link](../development/extensions-core/mysql.md)| +|postgresql-metadata-storage|PostgreSQL metadata store.|[link](../development/extensions-core/postgresql.md)| +|simple-client-sslcontext|Simple SSLContext provider module to be used by Druid's internal HttpClient when talking to other Druid processes over HTTPS.|[link](../development/extensions-core/simple-client-sslcontext.md)| +|druid-pac4j|OpenID Connect authentication for druid processes.|[link](../development/extensions-core/druid-pac4j.md)| + +## Community extensions + +> Community extensions are not maintained by Druid committers, although we accept patches from community members using these extensions. They may not have been as extensively tested as the core extensions. + +A number of community members have contributed their own extensions to Druid that are not packaged with the default Druid tarball. +If you'd like to take on maintenance for a community extension, please post on [dev@druid.apache.org](https://lists.apache.org/list.html?dev@druid.apache.org) to let us know! + +All of these community extensions can be downloaded using [pull-deps](../operations/pull-deps.md) while specifying a `-c` coordinate option to pull `org.apache.druid.extensions.contrib:{EXTENSION_NAME}:{DRUID_VERSION}`. + +|Name|Description|Docs| +|----|-----------|----| +|aliyun-oss-extensions|Aliyun OSS deep storage |[link](../development/extensions-contrib/aliyun-oss-extensions.md)| +|ambari-metrics-emitter|Ambari Metrics Emitter |[link](../development/extensions-contrib/ambari-metrics-emitter.md)| +|druid-cassandra-storage|Apache Cassandra deep storage.|[link](../development/extensions-contrib/cassandra.md)| +|druid-cloudfiles-extensions|Rackspace Cloudfiles deep storage and firehose.|[link](../development/extensions-contrib/cloudfiles.md)| +|druid-distinctcount|DistinctCount aggregator|[link](../development/extensions-contrib/distinctcount.md)| +|druid-redis-cache|A cache implementation for Druid based on Redis.|[link](../development/extensions-contrib/redis-cache.md)| +|druid-time-min-max|Min/Max aggregator for timestamp.|[link](../development/extensions-contrib/time-min-max.md)| +|sqlserver-metadata-storage|Microsoft SQLServer deep storage.|[link](../development/extensions-contrib/sqlserver.md)| +|graphite-emitter|Graphite metrics emitter|[link](../development/extensions-contrib/graphite.md)| +|statsd-emitter|StatsD metrics emitter|[link](../development/extensions-contrib/statsd.md)| +|kafka-emitter|Kafka metrics emitter|[link](../development/extensions-contrib/kafka-emitter.md)| +|druid-thrift-extensions|Support thrift ingestion |[link](../development/extensions-contrib/thrift.md)| +|druid-opentsdb-emitter|OpenTSDB metrics emitter |[link](../development/extensions-contrib/opentsdb-emitter.md)| +|materialized-view-selection, materialized-view-maintenance|Materialized View|[link](../development/extensions-contrib/materialized-view.md)| +|druid-moving-average-query|Support for [Moving Average](https://en.wikipedia.org/wiki/Moving_average) and other Aggregate [Window Functions](https://en.wikibooks.org/wiki/Structured_Query_Language/Window_functions) in Druid queries.|[link](../development/extensions-contrib/moving-average-query.md)| +|druid-influxdb-emitter|InfluxDB metrics emitter|[link](../development/extensions-contrib/influxdb-emitter.md)| +|druid-momentsketch|Support for approximate quantile queries using the [momentsketch](https://github.com/stanford-futuredata/momentsketch) library|[link](../development/extensions-contrib/momentsketch-quantiles.md)| +|druid-tdigestsketch|Support for approximate sketch aggregators based on [T-Digest](https://github.com/tdunning/t-digest)|[link](../development/extensions-contrib/tdigestsketch-quantiles.md)| +|gce-extensions|GCE Extensions|[link](../development/extensions-contrib/gce-extensions.md)| + +## Promoting community extensions to core extensions + +Please post on [dev@druid.apache.org](https://lists.apache.org/list.html?dev@druid.apache.org) if you'd like an extension to be promoted to core. +If we see a community extension actively supported by the community, we can promote it to core based on community feedback. + + +For information how to create your own extension, please see [here](../development/modules.md). + +## Loading extensions + +### Loading core extensions + +Apache Druid bundles all [core extensions](../development/extensions.md#core-extensions) out of the box. +See the [list of extensions](../development/extensions.md#core-extensions) for your options. You +can load bundled extensions by adding their names to your common.runtime.properties +`druid.extensions.loadList` property. For example, to load the *postgresql-metadata-storage* and +*druid-hdfs-storage* extensions, use the configuration: + +``` +druid.extensions.loadList=["postgresql-metadata-storage", "druid-hdfs-storage"] +``` + +These extensions are located in the `extensions` directory of the distribution. + +> Druid bundles two sets of configurations: one for the [quickstart](../tutorials/index.md) and +> one for a [clustered configuration](../tutorials/cluster.md). Make sure you are updating the correct +> common.runtime.properties for your setup. + +> Because of licensing, the mysql-metadata-storage extension does not include the required MySQL JDBC driver. For instructions +> on how to install this library, see the [MySQL extension page](../development/extensions-core/mysql.md). + +### Loading community extensions + +You can also load community and third-party extensions not already bundled with Druid. To do this, first download the extension and +then install it into your `extensions` directory. You can download extensions from their distributors directly, or +if they are available from Maven, the included [pull-deps](../operations/pull-deps.md) can download them for you. To use *pull-deps*, +specify the full Maven coordinate of the extension in the form `groupId:artifactId:version`. For example, +for the (hypothetical) extension *com.example:druid-example-extension:1.0.0*, run: + +``` +java \ + -cp "lib/*" \ + -Ddruid.extensions.directory="extensions" \ + -Ddruid.extensions.hadoopDependenciesDir="hadoop-dependencies" \ + org.apache.druid.cli.Main tools pull-deps \ + --no-default-hadoop \ + -c "com.example:druid-example-extension:1.0.0" +``` + +You only have to install the extension once. Then, add `"druid-example-extension"` to +`druid.extensions.loadList` in common.runtime.properties to instruct Druid to load the extension. + +> Please make sure all the Extensions related configuration properties listed [here](../configuration/index.md#extensions) are set correctly. + +> The Maven groupId for almost every [community extension](../development/extensions.md#community-extensions) is org.apache.druid.extensions.contrib. The artifactId is the name +> of the extension, and the version is the latest Druid stable version. + +### Loading extensions from the classpath + +If you add your extension jar to the classpath at runtime, Druid will also load it into the system. This mechanism is relatively easy to reason about, +but it also means that you have to ensure that all dependency jars on the classpath are compatible. That is, Druid makes no provisions while using +this method to maintain class loader isolation so you must make sure that the jars on your classpath are mutually compatible. diff --git a/Development/parquet-extensions.md b/development/parquet-extensions.md similarity index 100% rename from Development/parquet-extensions.md rename to development/parquet-extensions.md diff --git a/Development/protobuf-extensions.md b/development/protobuf-extensions.md similarity index 100% rename from Development/protobuf-extensions.md rename to development/protobuf-extensions.md diff --git a/Development/rackspacecloudfiles.md b/development/rackspacecloudfiles.md similarity index 100% rename from Development/rackspacecloudfiles.md rename to development/rackspacecloudfiles.md diff --git a/misc/index.md b/misc/index.md index bc0f418..d1e39c7 100644 --- a/misc/index.md +++ b/misc/index.md @@ -23,44 +23,46 @@ clickhouse 是俄罗斯的搜索引擎（Yandex）公司在 2016 年开源的，一款针对大数据实时分析的高性能分布式数据库，与之对应的有 hadoop 生态 hive，Vertica 和 palo。 ## 源代码与进阶 -* [Apache Druid源码导读--Google Guice DI框架](https://blog.csdn.net/yueguanghaidao/article/details/102531570) - 在大数据应用组件中，有两款OLAP引擎应用广泛，一款是偏离线处理的Kylin，另一个是偏实时的Druid。Kylin是一款国人开源的优秀离线OLAP引擎，基本上是Hadoop领域离线OLAP事实标准，在离线报表，指标分析领域应用广泛。而Apache Druid则在实时OLAP领域独领风骚，优异的性能、高可用、易扩展。 +* [Apache Druid 源码导读--Google Guice DI框架](https://blog.csdn.net/yueguanghaidao/article/details/102531570) - + 在大数据应用组件中，有两款OLAP引擎应用广泛，一款是偏离线处理的Kylin，另一个是偏实时的Druid。 + Kylin是一款国人开源的优秀离线OLAP引擎，基本上是Hadoop领域离线OLAP事实标准，在离线报表，指标分析领域应用广泛。 + 而Apache Druid则在实时OLAP领域独领风骚，优异的性能、高可用、易扩展。 -* [Apache Druid源码解析的一个合集](https://blog.csdn.net/mytobaby00/category_7561069.html) +* [Apache Druid 源码解析的一个合集](https://blog.csdn.net/mytobaby00/category_7561069.html) -* [Druid中的Extension在启动时是如何加载的](https://blog.csdn.net/mytobaby00/article/details/79857681) +* [Druid 中的 Extension 在启动时是如何加载的](https://blog.csdn.net/mytobaby00/article/details/79857681) -* [Druid解析之管理用的接口大全](https://blog.csdn.net/mytobaby00/article/details/80088795) +* [Druid 解析之管理用的接口大全](https://blog.csdn.net/mytobaby00/article/details/80088795) -* [Druid原理分析之内存池管理](https://blog.csdn.net/mytobaby00/article/details/80071101) +* [Druid 原理分析之内存池管理](https://blog.csdn.net/mytobaby00/article/details/80071101) -* [Druid源码解析之Segment](Druid源码解析之Segment) +* [Druid 源码解析之 Segment](https://blog.csdn.net/mytobaby00/article/details/80059820) -* [Druid源码解析之Column](https://blog.csdn.net/mytobaby00/article/details/80056826) +* [Druid 源码解析之 Column](https://blog.csdn.net/mytobaby00/article/details/80056826) -* [Druid源码解析之HDFS存储](https://blog.csdn.net/mytobaby00/article/details/80045662) +* [Druid 源码解析之 HDFS 存储](https://blog.csdn.net/mytobaby00/article/details/80045662) -* [Druid源码解析之Coordinator](https://blog.csdn.net/mytobaby00/article/details/80041970) +* [Druid 源码解析之 Coordinator](https://blog.csdn.net/mytobaby00/article/details/80041970) -* [让Druid实现事件设备数留存数的精准计算](https://blog.csdn.net/mytobaby00/article/details/79804685) +* [让 Druid 实现事件设备数留存数的精准计算](https://blog.csdn.net/mytobaby00/article/details/79804685) -* [在Druid中定制自己的扩展【Extension】](https://blog.csdn.net/mytobaby00/article/details/79803605) +* [在 Druid 中定制自己的扩展【Extension】](https://blog.csdn.net/mytobaby00/article/details/79803605) -* [Druid原理分析之“批”任务数据流转过程](https://blog.csdn.net/mytobaby00/article/details/79802776) +* [Druid 原理分析之“批”任务数据流转过程](https://blog.csdn.net/mytobaby00/article/details/79802776) -* [Druid原理分析之“流”任务数据流转过程](https://blog.csdn.net/mytobaby00/article/details/79801614) +* [Druid 原理分析之“流”任务数据流转过程](https://blog.csdn.net/mytobaby00/article/details/79801614) -* [Druid原理分析之Segment的存储结构](https://blog.csdn.net/mytobaby00/article/details/79801425) +* [Druid 原理分析之 Segment 的存储结构](https://blog.csdn.net/mytobaby00/article/details/79801425) -* [Druid索引与查询原理简析](https://blog.csdn.net/mytobaby00/article/details/79800553) +* [Druid 索引与查询原理简析](https://blog.csdn.net/mytobaby00/article/details/79800553) -* [Druid中的负载均衡策略分析](https://blog.csdn.net/mytobaby00/article/details/79860836) +* [Druid 中的负载均衡策略分析](https://blog.csdn.net/mytobaby00/article/details/79860836) -* [Druid中的Kafka Indexing Service源码分析](https://blog.csdn.net/mytobaby00/article/details/79858403) +* [Druid 中的 Kafka Indexing Service 源码分析](https://blog.csdn.net/mytobaby00/article/details/79858403) -* [Druid源码分析之Query -- Sequence与Yielder](https://blog.csdn.net/mytobaby00/article/details/80103230) +* [Druid 源码分析之 Query -- Sequence 与 Yielder](https://blog.csdn.net/mytobaby00/article/details/80103230) -* [Druid原理分析之Segment的存储结构](https://blog.csdn.net/mytobaby00/article/details/79801425) +* [Druid 原理分析之 Segment 的存储结构](https://blog.csdn.net/mytobaby00/article/details/79801425) ## 优化与实践 @@ -73,15 +75,19 @@ 爱奇艺大数据服务团队评估了市面上主流的OLAP引擎，最终选择Apache Druid时序数据库来满足业务的实时分析需求。 本文将介绍Druid在爱奇艺的实践情况、优化经验以及平台化建设的一些思考。 -* [熵简技术谈 | 实时OLAP引擎之Apache Druid：架构、原理和应用实践](https://zhuanlan.zhihu.com/p/178572172) - 本文以实时 OLAP 引擎的优秀代表 Druid 为研究对象，详细介绍 Druid 的架构思想和核心特性。在此基础上，我们介绍了熵简科技在数据智能分析场景下，针对私有化部署与实时响应优化的实践经验。 +* [实时 OLAP 引擎之 Apache Druid：架构、原理和应用实践](https://www.ossez.com/t/olap-apache-druid/13581) - + 本文以实时 OLAP 引擎的优秀代表 Druid 为研究对象，详细介绍 Druid 的架构思想和核心特性。 + 在此基础上，我们介绍了熵简科技在数据智能分析场景下，针对私有化部署与实时响应优化的实践经验。 -* [Apache Druid性能测评-云栖社区-阿里云](https://developer.aliyun.com/article/712725) +* [Apache Druid 性能测评](https://www.ossez.com/t/apache-druid/13580) - + 模拟生产由 5000 个 agent、5000 个 URL 和 2 类请求方式做为聚合字段的1亿条明细数据，来测试Druid集群在配置不同 TaksCount数时，Druid聚合任务的执行时长。 -* [Druid在有赞的实践](https://www.cnblogs.com/oldtrafford/p/10301581.html) - 有赞作为一家 SaaS 公司，有很多的业务的场景和非常大量的实时数据和离线数据。在没有是使用 Druid 之前，一些 OLAP 场景的场景分析，开发的同学都是使用 SparkStreaming 或者 Storm 做的。用这类方案会除了需要写实时任务之外，还需要为了查询精心设计存储。带来问题是：开发的周期长；初期的存储设计很难满足需求的迭代发展；不可扩展。 +* [Druid 在有赞的实践](https://www.ossez.com/t/druid/13579) - + 有赞作为一家 SaaS 公司，有很多的业务的场景和非常大量的实时数据和离线数据。 + 在没有是使用 Druid 之前，一些 OLAP 场景的场景分析，开发的同学都是使用 SparkStreaming 或者 Storm 做的。 + 用这类方案会除了需要写实时任务之外，还需要为了查询精心设计存储。带来问题是：开发的周期长；初期的存储设计很难满足需求的迭代发展；不可扩展。 -* [Druid 在小米公司的技术实践](https://zhuanlan.zhihu.com/p/25593670) - +* [Druid 在小米公司的技术实践](https://www.ossez.com/t/druid/13578) - Druid 作为一款开源的实时大数据分析软件，自诞生以来，凭借着自己优秀的特质，逐渐在技术圈收获了越来越多的知名度与口碑， 并陆续成为了很多技术团队解决方案中的关键一环，从而真正在很多公司的技术栈中赢得了一席之地。 diff --git a/tutorials/cluster.md b/tutorials/cluster.md new file mode 100644 index 0000000..bc64495 --- /dev/null +++ b/tutorials/cluster.md @@ -0,0 +1,474 @@ +--- +id: cluster +title: "Clustered deployment" +--- + + + + +Apache Druid is designed to be deployed as a scalable, fault-tolerant cluster. + +In this document, we'll set up a simple cluster and discuss how it can be further configured to meet +your needs. + +This simple cluster will feature: + + - A Master server to host the Coordinator and Overlord processes + - Two scalable, fault-tolerant Data servers running Historical and MiddleManager processes + - A query server, hosting the Druid Broker and Router processes + +In production, we recommend deploying multiple Master servers and multiple Query servers in a fault-tolerant configuration based on your specific fault-tolerance needs, but you can get started quickly with one Master and one Query server and add more servers later. + +## Select hardware + +### Fresh Deployment + +If you do not have an existing Druid cluster, and wish to start running Druid in a clustered deployment, this guide provides an example clustered deployment with pre-made configurations. + +#### Master server + +The Coordinator and Overlord processes are responsible for handling the metadata and coordination needs of your cluster. They can be colocated together on the same server. + +In this example, we will be deploying the equivalent of one AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instance. + +This hardware offers: + +- 8 vCPUs +- 31 GB RAM + +Example Master server configurations that have been sized for this hardware can be found under `conf/druid/cluster/master`. + +#### Data server + +Historicals and MiddleManagers can be colocated on the same server to handle the actual data in your cluster. These servers benefit greatly from CPU, RAM, +and SSDs. + +In this example, we will be deploying the equivalent of two AWS [i3.4xlarge](https://aws.amazon.com/ec2/instance-types/i3/) instances. + +This hardware offers: + +- 16 vCPUs +- 122 GB RAM +- 2 * 1.9TB SSD storage + +Example Data server configurations that have been sized for this hardware can be found under `conf/druid/cluster/data`. + +#### Query server + +Druid Brokers accept queries and farm them out to the rest of the cluster. They also optionally maintain an +in-memory query cache. These servers benefit greatly from CPU and RAM. + +In this example, we will be deploying the equivalent of one AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instance. + +This hardware offers: + +- 8 vCPUs +- 31 GB RAM + +You can consider co-locating any open source UIs or query libraries on the same server that the Broker is running on. + +Example Query server configurations that have been sized for this hardware can be found under `conf/druid/cluster/query`. + +#### Other Hardware Sizes + +The example cluster above is chosen as a single example out of many possible ways to size a Druid cluster. + +You can choose smaller/larger hardware or less/more servers for your specific needs and constraints. + +If your use case has complex scaling requirements, you can also choose to not co-locate Druid processes (e.g., standalone Historical servers). + +The information in the [basic cluster tuning guide](../operations/basic-cluster-tuning.md) can help with your decision-making process and with sizing your configurations. + +### Migrating from a single-server deployment + +If you have an existing single-server deployment, such as the ones from the [single-server deployment examples](../operations/single-server.md), and you wish to migrate to a clustered deployment of similar scale, the following section contains guidelines for choosing equivalent hardware using the Master/Data/Query server organization. + +#### Master server + +The main considerations for the Master server are available CPUs and RAM for the Coordinator and Overlord heaps. + +Sum up the allocated heap sizes for your Coordinator and Overlord from the single-server deployment, and choose Master server hardware with enough RAM for the combined heaps, with some extra RAM for other processes on the machine. + +For CPU cores, you can choose hardware with approximately 1/4th of the cores of the single-server deployment. + +#### Data server + +When choosing Data server hardware for the cluster, the main considerations are available CPUs and RAM, and using SSD storage if feasible. + +In a clustered deployment, having multiple Data servers is a good idea for fault-tolerance purposes. + +When choosing the Data server hardware, you can choose a split factor `N`, divide the original CPU/RAM of the single-server deployment by `N`, and deploy `N` Data servers of reduced size in the new cluster. + +Instructions for adjusting the Historical/MiddleManager configs for the split are described in a later section in this guide. + +#### Query server + +The main considerations for the Query server are available CPUs and RAM for the Broker heap + direct memory, and Router heap. + +Sum up the allocated memory sizes for your Broker and Router from the single-server deployment, and choose Query server hardware with enough RAM to cover the Broker/Router, with some extra RAM for other processes on the machine. + +For CPU cores, you can choose hardware with approximately 1/4th of the cores of the single-server deployment. + +The [basic cluster tuning guide](../operations/basic-cluster-tuning.md) has information on how to calculate Broker/Router memory usage. + +## Select OS + +We recommend running your favorite Linux distribution. You will also need: + + * **Java 8 or later** + +> **Warning:** Druid only officially supports Java 8. Any Java version later than 8 is still experimental. +> +> If needed, you can specify where to find Java using the environment variables `DRUID_JAVA_HOME` or `JAVA_HOME`. For more details run the verify-java script. + +Your OS package manager should be able to help for both Java. If your Ubuntu-based OS +does not have a recent enough version of Java, WebUpd8 offers [packages for those +OSes](http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html). + +## Download the distribution + +First, download and unpack the release archive. It's best to do this on a single machine at first, +since you will be editing the configurations and then copying the modified distribution out to all +of your servers. + +[Download](https://www.apache.org/dyn/closer.cgi?path=/druid/{{DRUIDVERSION}}/apache-druid-{{DRUIDVERSION}}-bin.tar.gz) +the {{DRUIDVERSION}} release. + +Extract Druid by running the following commands in your terminal: + +```bash +tar -xzf apache-druid-{{DRUIDVERSION}}-bin.tar.gz +cd apache-druid-{{DRUIDVERSION}} +``` + +In the package, you should find: + +* `LICENSE` and `NOTICE` files +* `bin/*` - scripts related to the [single-machine quickstart](index.html) +* `conf/druid/cluster/*` - template configurations for a clustered setup +* `extensions/*` - core Druid extensions +* `hadoop-dependencies/*` - Druid Hadoop dependencies +* `lib/*` - libraries and dependencies for core Druid +* `quickstart/*` - files related to the [single-machine quickstart](index.html) + +We'll be editing the files in `conf/druid/cluster/` in order to get things running. + +### Migrating from Single-Server Deployments + +In the following sections we will be editing the configs under `conf/druid/cluster`. + +If you have an existing single-server deployment, please copy your existing configs to `conf/druid/cluster` to preserve any config changes you have made. + +## Configure metadata storage and deep storage + +### Migrating from Single-Server Deployments + +If you have an existing single-server deployment and you wish to preserve your data across the migration, please follow the instructions at [metadata migration](../operations/metadata-migration.md) and [deep storage migration](../operations/deep-storage-migration.md) before updating your metadata/deep storage configs. + +These guides are targeted at single-server deployments that use the Derby metadata store and local deep storage. If you are already using a non-Derby metadata store in your single-server cluster, you can reuse the existing metadata store for the new cluster. + +These guides also provide information on migrating segments from local deep storage. A clustered deployment requires distributed deep storage like S3 or HDFS. If your single-server deployment was already using distributed deep storage, you can reuse the existing deep storage for the new cluster. + +### Metadata storage + +In `conf/druid/cluster/_common/common.runtime.properties`, replace +"metadata.storage.*" with the address of the machine that you will use as your metadata store: + +- `druid.metadata.storage.connector.connectURI` +- `druid.metadata.storage.connector.host` + +In a production deployment, we recommend running a dedicated metadata store such as MySQL or PostgreSQL with replication, deployed separately from the Druid servers. + +The [MySQL extension](../development/extensions-core/mysql.md) and [PostgreSQL extension](../development/extensions-core/postgresql.md) docs have instructions for extension configuration and initial database setup. + +### Deep storage + +Druid relies on a distributed filesystem or large object (blob) store for data storage. The most +commonly used deep storage implementations are S3 (popular for those on AWS) and HDFS (popular if +you already have a Hadoop deployment). + +#### S3 + +In `conf/druid/cluster/_common/common.runtime.properties`, + +- Add "druid-s3-extensions" to `druid.extensions.loadList`. + +- Comment out the configurations for local storage under "Deep Storage" and "Indexing service logs". + +- Uncomment and configure appropriate values in the "For S3" sections of "Deep Storage" and +"Indexing service logs". + +After this, you should have made the following changes: + +``` +druid.extensions.loadList=["druid-s3-extensions"] + +#druid.storage.type=local +#druid.storage.storageDirectory=var/druid/segments + +druid.storage.type=s3 +druid.storage.bucket=your-bucket +druid.storage.baseKey=druid/segments +druid.s3.accessKey=... +druid.s3.secretKey=... + +#druid.indexer.logs.type=file +#druid.indexer.logs.directory=var/druid/indexing-logs + +druid.indexer.logs.type=s3 +druid.indexer.logs.s3Bucket=your-bucket +druid.indexer.logs.s3Prefix=druid/indexing-logs +``` + +Please see the [S3 extension](../development/extensions-core/s3.md) documentation for more info. + +#### HDFS + +In `conf/druid/cluster/_common/common.runtime.properties`, + +- Add "druid-hdfs-storage" to `druid.extensions.loadList`. + +- Comment out the configurations for local storage under "Deep Storage" and "Indexing service logs". + +- Uncomment and configure appropriate values in the "For HDFS" sections of "Deep Storage" and +"Indexing service logs". + +After this, you should have made the following changes: + +``` +druid.extensions.loadList=["druid-hdfs-storage"] + +#druid.storage.type=local +#druid.storage.storageDirectory=var/druid/segments + +druid.storage.type=hdfs +druid.storage.storageDirectory=/druid/segments + +#druid.indexer.logs.type=file +#druid.indexer.logs.directory=var/druid/indexing-logs + +druid.indexer.logs.type=hdfs +druid.indexer.logs.directory=/druid/indexing-logs +``` + +Also, + +- Place your Hadoop configuration XMLs (core-site.xml, hdfs-site.xml, yarn-site.xml, +mapred-site.xml) on the classpath of your Druid processes. You can do this by copying them into +`conf/druid/cluster/_common/`. + +Please see the [HDFS extension](../development/extensions-core/hdfs.md) documentation for more info. + + + +## Configure for connecting to Hadoop (optional) + +If you will be loading data from a Hadoop cluster, then at this point you should configure Druid to be aware +of your cluster: + +- Update `druid.indexer.task.hadoopWorkingPath` in `conf/druid/cluster/middleManager/runtime.properties` to +a path on HDFS that you'd like to use for temporary files required during the indexing process. +`druid.indexer.task.hadoopWorkingPath=/tmp/druid-indexing` is a common choice. + +- Place your Hadoop configuration XMLs (core-site.xml, hdfs-site.xml, yarn-site.xml, +mapred-site.xml) on the classpath of your Druid processes. You can do this by copying them into +`conf/druid/cluster/_common/core-site.xml`, `conf/druid/cluster/_common/hdfs-site.xml`, and so on. + +Note that you don't need to use HDFS deep storage in order to load data from Hadoop. For example, if +your cluster is running on Amazon Web Services, we recommend using S3 for deep storage even if you +are loading data using Hadoop or Elastic MapReduce. + +For more info, please see the [Hadoop-based ingestion](../ingestion/hadoop.md) page. + +## Configure Zookeeper connection + +In a production cluster, we recommend using a dedicated ZK cluster in a quorum, deployed separately from the Druid servers. + +In `conf/druid/cluster/_common/common.runtime.properties`, set +`druid.zk.service.host` to a [connection string](https://zookeeper.apache.org/doc/current/zookeeperProgrammers.html) +containing a comma separated list of host:port pairs, each corresponding to a ZooKeeper server in your ZK quorum. +(e.g. "127.0.0.1:4545" or "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002") + +You can also choose to run ZK on the Master servers instead of having a dedicated ZK cluster. If doing so, we recommend deploying 3 Master servers so that you have a ZK quorum. + +## Configuration Tuning + +### Migrating from a Single-Server Deployment + +#### Master + +If you are using an example configuration from [single-server deployment examples](../operations/single-server.md), these examples combine the Coordinator and Overlord processes into one combined process. + +The example configs under `conf/druid/cluster/master/coordinator-overlord` also combine the Coordinator and Overlord processes. + +You can copy your existing `coordinator-overlord` configs from the single-server deployment to `conf/druid/cluster/master/coordinator-overlord`. + +#### Data + +Suppose we are migrating from a single-server deployment that had 32 CPU and 256GB RAM. In the old deployment, the following configurations for Historicals and MiddleManagers were applied: + +Historical (Single-server) + +``` +druid.processing.buffer.sizeBytes=500000000 +druid.processing.numMergeBuffers=8 +druid.processing.numThreads=31 +``` + +MiddleManager (Single-server) + +``` +druid.worker.capacity=8 +druid.indexer.fork.property.druid.processing.numMergeBuffers=2 +druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000 +druid.indexer.fork.property.druid.processing.numThreads=1 +``` + +In the clustered deployment, we can choose a split factor (2 in this example), and deploy 2 Data servers with 16CPU and 128GB RAM each. The areas to scale are the following: + +Historical + +- `druid.processing.numThreads`: Set to `(num_cores - 1)` based on the new hardware +- `druid.processing.numMergeBuffers`: Divide the old value from the single-server deployment by the split factor +- `druid.processing.buffer.sizeBytes`: Keep this unchanged + +MiddleManager: + +- `druid.worker.capacity`: Divide the old value from the single-server deployment by the split factor +- `druid.indexer.fork.property.druid.processing.numMergeBuffers`: Keep this unchanged +- `druid.indexer.fork.property.druid.processing.buffer.sizeBytes`: Keep this unchanged +- `druid.indexer.fork.property.druid.processing.numThreads`: Keep this unchanged + +The resulting configs after the split: + +New Historical (on 2 Data servers) + +``` + druid.processing.buffer.sizeBytes=500000000 + druid.processing.numMergeBuffers=8 + druid.processing.numThreads=31 +``` + +New MiddleManager (on 2 Data servers) + +``` +druid.worker.capacity=4 +druid.indexer.fork.property.druid.processing.numMergeBuffers=2 +druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000 +druid.indexer.fork.property.druid.processing.numThreads=1 +``` + +#### Query + +You can copy your existing Broker and Router configs to the directories under `conf/druid/cluster/query`, no modifications are needed, as long as the new hardware is sized accordingly. + +### Fresh deployment + +If you are using the example cluster described above: +- 1 Master server (m5.2xlarge) +- 2 Data servers (i3.4xlarge) +- 1 Query server (m5.2xlarge) + +The configurations under `conf/druid/cluster` have already been sized for this hardware and you do not need to make further modifications for general use cases. + +If you have chosen different hardware, the [basic cluster tuning guide](../operations/basic-cluster-tuning.md) can help you size your configurations. + +## Open ports (if using a firewall) + +If you're using a firewall or some other system that only allows traffic on specific ports, allow +inbound connections on the following: + +### Master Server +- 1527 (Derby metadata store; not needed if you are using a separate metadata store like MySQL or PostgreSQL) +- 2181 (ZooKeeper; not needed if you are using a separate ZooKeeper cluster) +- 8081 (Coordinator) +- 8090 (Overlord) + +### Data Server +- 8083 (Historical) +- 8091, 8100–8199 (Druid Middle Manager; you may need higher than port 8199 if you have a very high `druid.worker.capacity`) + +### Query Server +- 8082 (Broker) +- 8088 (Router, if used) + +> In production, we recommend deploying ZooKeeper and your metadata store on their own dedicated hardware, +> rather than on the Master server. + +## Start Master Server + +Copy the Druid distribution and your edited configurations to your Master server. + +If you have been editing the configurations on your local machine, you can use *rsync* to copy them: + +```bash +rsync -az apache-druid-{{DRUIDVERSION}}/ MASTER_SERVER:apache-druid-{{DRUIDVERSION}}/ +``` + +### No Zookeeper on Master + +From the distribution root, run the following command to start the Master server: + +``` +bin/start-cluster-master-no-zk-server +``` + +### With Zookeeper on Master + +If you plan to run ZK on Master servers, first update `conf/zoo.cfg` to reflect how you plan to run ZK. Then, you +can start the Master server processes together with ZK using: + +``` +bin/start-cluster-master-with-zk-server +``` + +> In production, we also recommend running a ZooKeeper cluster on its own dedicated hardware. + +## Start Data Server + +Copy the Druid distribution and your edited configurations to your Data servers. + +From the distribution root, run the following command to start the Data server: + +``` +bin/start-cluster-data-server +``` + +You can add more Data servers as needed. + +> For clusters with complex resource allocation needs, you can break apart Historicals and MiddleManagers and scale the components individually. +> This also allows you take advantage of Druid's built-in MiddleManager autoscaling facility. + +## Start Query Server + +Copy the Druid distribution and your edited configurations to your Query servers. + +From the distribution root, run the following command to start the Query server: + +``` +bin/start-cluster-query-server +``` + +You can add more Query servers as needed based on query load. If you increase the number of Query servers, be sure to adjust the connection pools on your Historicals and Tasks as described in the [basic cluster tuning guide](../operations/basic-cluster-tuning.md). + +## Loading data + +Congratulations, you now have a Druid cluster! The next step is to learn about recommended ways to load data into +Druid based on your use case. Read more about [loading data](../ingestion/index.md). diff --git a/GettingStarted/Docker.md b/tutorials/docker.md similarity index 77% rename from GettingStarted/Docker.md rename to tutorials/docker.md index 9285881..4269ea3 100644 --- a/GettingStarted/Docker.md +++ b/tutorials/docker.md @@ -1,21 +1,8 @@ - - - - - - -## Docker +# Docker 在这个部分中，我们将从 [Docker Hub](https://hub.docker.com/r/apache/druid) 下载Apache Druid镜像，并使用 [Docker](https://www.docker.com/get-started) 和 [Docker Compose](https://docs.docker.com/compose/) 在一台机器上安装它。完成此初始设置后，集群将准备好加载数据。 -在开始快速启动之前，阅读 [Druid概述](chapter-1.md) 和 [摄取概述](../DataIngestion/ingestion.md) 是很有帮助的，因为教程将参考这些页面上讨论的概念。此外，建议熟悉Docker。 +在开始快速启动之前，阅读 [Druid概述](../GettingStarted/chapter-1.md) 和 [摄取概述](../DataIngestion/ingestion.md) 是很有帮助的，因为教程将参考这些页面上讨论的概念。此外，建议熟悉Docker。 ### 前提条件 @@ -54,10 +41,10 @@ Druid `docker-compose.yml` 示例使用单个环境文件来指定完整的Drui 运行 `docker-compose up` 启动附加shell的集群，或运行 `docker-compose up -d` 在后台运行集群。如果直接使用示例文件，这个命令应该从Druid安装目录中的 `distribution/docker/` 运行。 -启动集群后，可以导航到 [http://localhost:8888](http://localhost/) 。服务于 [Druid控制台](../Operations/druid-console.md) 的 [Druid路由进程](../Design/Router.md) 位于这个地址。 +启动集群后，可以导航到 [http://localhost:8888](http://localhost/) 。服务于 [Druid控制台](../Operations/druid-console.md) 的 [Druid路由进程](../design/Router.md) 位于这个地址。 ![](img/tutorial-quickstart-01.png) 所有Druid进程需要几秒钟才能完全启动。如果在启动服务后立即打开控制台，可能会看到一些可以安全忽略的错误。 -从这里你可以跟着 [标准教程](chapter-2.md)，或者详细说明你的 `docker-compose.yml` 根据需要添加任何其他外部服务依赖项。 \ No newline at end of file +从这里你可以跟着 [标准教程](../GettingStarted/chapter-2.md)，或者详细说明你的 `docker-compose.yml` 根据需要添加任何其他外部服务依赖项。 \ No newline at end of file diff --git a/GettingStarted/chapter-1.md b/tutorials/img/chapter-1.md similarity index 100% rename from GettingStarted/chapter-1.md rename to tutorials/img/chapter-1.md diff --git a/GettingStarted/chapter-2.md b/tutorials/img/chapter-2.md similarity index 84% rename from GettingStarted/chapter-2.md rename to tutorials/img/chapter-2.md index f082f25..51d7164 100644 --- a/GettingStarted/chapter-2.md +++ b/tutorials/img/chapter-2.md @@ -14,7 +14,7 @@ 在本快速入门教程中，我们将下载Druid并将其安装在一台服务器上，完成初始安装后，向集群中加载数据。 -在开始快速入门之前，阅读[Druid概述](./chapter-1.md)和[数据摄取概述](../DataIngestion/index.md)会很有帮助，因为当前教程会引用这些页面上讨论的概念。 +在开始快速入门之前，阅读[Druid概述](chapter-1.md)和[数据摄取概述](../DataIngestion/index.md)会很有帮助，因为当前教程会引用这些页面上讨论的概念。 #### 预备条件 ##### 软件 @@ -26,7 +26,7 @@ ##### 硬件 -Druid安装包提供了几个[单服务器配置](./chapter-3.md)的示例，以及使用这些配置启动Druid进程的脚本。 +Druid安装包提供了几个[单服务器配置](chapter-3.md)的示例，以及使用这些配置启动Druid进程的脚本。 如果您正在使用便携式等小型计算机上运行服务，则配置为4CPU/16GB RAM环境的`micro-quickstart`配置是一个不错的选择。 @@ -80,7 +80,7 @@ $ ./bin/start-micro-quickstart 集群启动后，可以访问[http://localhost:8888](http://localhost:8888)来Druid控制台，控制台由Druid Router进程启动。 -![tutorial-quickstart](img/tutorial-quickstart-01.png) +![tutorial-quickstart](tutorial-quickstart-01.png) 所有Druid进程完全启动需要花费几秒钟。 如果在启动服务后立即打开控制台，则可能会看到一些可以安全忽略的错误。 @@ -141,10 +141,10 @@ $ ./bin/start-micro-quickstart 以下教程演示了将数据加载到Druid的各种方法，包括批处理和流处理用例。 所有教程均假定您使用的是上面提到的`micro-quickstart`单机配置。 -* [加载本地文件](../tutorials/tutorial-batch.md) - 本教程演示了如何使用Druid的本地批处理摄取来执行批文件加载 -* [从Kafka加载流数据](../tutorials/chapter-2.md) - 本教程演示了如何从Kafka主题加载流数据 -* [从Hadoop加载数据](../tutorials/chapter-3.md) - 本教程演示了如何使用远程Hadoop集群执行批处理文件加载 -* [编写一个自己的数据摄取规范](../tutorials/chapter-10.md) - 本教程演示了如何编写新的数据摄取规范并使用它来加载数据 +* [加载本地文件](../tutorial-batch.md) - 本教程演示了如何使用Druid的本地批处理摄取来执行批文件加载 +* [从Kafka加载流数据](../chapter-2.md) - 本教程演示了如何从Kafka主题加载流数据 +* [从Hadoop加载数据](../chapter-3.md) - 本教程演示了如何使用远程Hadoop集群执行批处理文件加载 +* [编写一个自己的数据摄取规范](../chapter-10.md) - 本教程演示了如何编写新的数据摄取规范并使用它来加载数据 ##### 重置集群状态 @@ -154,7 +154,7 @@ $ ./bin/start-micro-quickstart ##### 重置Kafka -如果您完成了[教程：从Kafka加载流数据](../tutorials/chapter-2.md)并希望重置集群状态，则还应该清除所有Kafka状态。 +如果您完成了[教程：从Kafka加载流数据](../chapter-2.md)并希望重置集群状态，则还应该清除所有Kafka状态。 在停止ZooKeeper和Druid服务之前，使用`CTRL-C`关闭`Kafka Broker`，然后删除`/tmp/kafka-logs`中的Kafka日志目录： diff --git a/GettingStarted/chapter-3.md b/tutorials/img/chapter-3.md similarity index 90% rename from GettingStarted/chapter-3.md rename to tutorials/img/chapter-3.md index a232c9e..076cf1a 100644 --- a/GettingStarted/chapter-3.md +++ b/tutorials/img/chapter-3.md @@ -31,7 +31,7 @@ Druid包括一组参考配置和用于单机部署的启动脚本： 这些示例配置的启动脚本与Druid服务一起运行单个ZK实例,您也可以选择单独部署ZK。 -通过[Coordinator配置文档](../Configuration/configuration.md#Coordinator)中描述的可选配置`druid.coordinator.asOverlord.enabled = true`可以在单个进程中同时运行Druid Coordinator和Overlord。 +通过[Coordinator配置文档](../../Configuration/configuration.md#Coordinator)中描述的可选配置`druid.coordinator.asOverlord.enabled = true`可以在单个进程中同时运行Druid Coordinator和Overlord。 虽然为大型单台计算机提供了示例配置，但在更高规模下，我们建议在集群部署中运行Druid，以实现容错和减少资源争用。 diff --git a/GettingStarted/chapter-4.md b/tutorials/img/chapter-4.md similarity index 91% rename from GettingStarted/chapter-4.md rename to tutorials/img/chapter-4.md index 597c3b5..e7e873f 100644 --- a/GettingStarted/chapter-4.md +++ b/tutorials/img/chapter-4.md @@ -77,11 +77,11 @@ Druid Broker服务接收查询请求，并将其转发到集群中的其他部 如果您的使用场景具有复杂的扩展要求，则还可以选择不将Druid服务混合部署（例如，独立的Historical Server）。 -[基本集群调整指南](../Operations/basicClusterTuning.md)中的信息可以帮助您进行决策，并可以调整配置大小。 +[基本集群调整指南](../../Operations/basicClusterTuning.md)中的信息可以帮助您进行决策，并可以调整配置大小。 #### 从单服务器环境迁移部署 -如果您现在已有单服务器部署的环境，例如[单服务器部署示例](./chapter-3.md)中的部署，并且希望迁移到类似规模的集群部署，则以下部分包含一些选择Master/Data/Query服务等效硬件的准则。 +如果您现在已有单服务器部署的环境，例如[单服务器部署示例](chapter-3.md)中的部署，并且希望迁移到类似规模的集群部署，则以下部分包含一些选择Master/Data/Query服务等效硬件的准则。 ##### Master服务 @@ -107,7 +107,7 @@ Query服务的硬件选择主要考虑可用的CPU、Broker服务的堆内和堆 对于CPU，可以选择接近于单服务器环境核数1/4的硬件。 -[基本集群调优指南](../Operations/basicClusterTuning.md)包含有关如何计算Broker和Router服务内存使用量的信息。 +[基本集群调优指南](../../Operations/basicClusterTuning.md)包含有关如何计算Broker和Router服务内存使用量的信息。 ### 选择操作系统 @@ -139,7 +139,7 @@ cd apache-druid-0.17.0 * `extensions/*` - Druid核心扩展 * `hadoop-dependencies/*` - Druid Hadoop依赖 * `lib/*` - Druid核心库和依赖 -* `quickstart/*` - 与[快速入门](./chapter-2.md)相关的文件 +* `quickstart/*` - 与[快速入门](chapter-2.md)相关的文件 我们主要是编辑`conf/druid/cluster/`中的文件。 @@ -152,7 +152,7 @@ cd apache-druid-0.17.0 ### 配置元数据存储和深度存储 #### 从单服务器环境迁移部署 -如果您已经有一个单服务器部署，并且希望在整个迁移过程中保留数据，请在更新元数据/深层存储配置之前，按照[元数据迁移](../Operations/metadataMigration.md)和[深层存储迁移](../Operations/DeepstorageMigration.md)中的说明进行操作。 +如果您已经有一个单服务器部署，并且希望在整个迁移过程中保留数据，请在更新元数据/深层存储配置之前，按照[元数据迁移](../../Operations/metadataMigration.md)和[深层存储迁移](../../Operations/DeepstorageMigration.md)中的说明进行操作。 这些指南针对使用Derby元数据存储和本地深度存储的单服务器部署。 如果您已经在单服务器集群中使用了非Derby元数据存储，则可以在新集群中可以继续使用当前的元数据存储。 @@ -167,7 +167,7 @@ cd apache-druid-0.17.0 在生产部署中，我们建议运行专用的元数据存储，例如具有复制功能的MySQL或PostgreSQL，与Druid服务器分开部署。 -[MySQL扩展](../Configuration/core-ext/mysql.md)和[PostgreSQL](../Configuration/core-ext/postgresql.md)扩展文档包含有关扩展配置和初始数据库安装的说明。 +[MySQL扩展](../../Configuration/core-ext/mysql.md)和[PostgreSQL](../../Configuration/core-ext/postgresql.md)扩展文档包含有关扩展配置和初始数据库安装的说明。 #### 深度存储 @@ -202,7 +202,7 @@ druid.indexer.logs.type=s3 druid.indexer.logs.s3Bucket=your-bucket druid.indexer.logs.s3Prefix=druid/indexing-logs ``` -更多信息可以看[S3扩展](../Configuration/core-ext/s3.md)部分的文档。 +更多信息可以看[S3扩展](../../Configuration/core-ext/s3.md)部分的文档。 ##### HDFS @@ -234,7 +234,7 @@ druid.indexer.logs.directory=/druid/indexing-logs * 需要将Hadoop的配置文件（core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml）放置在Druid进程的classpath中，可以将他们拷贝到`conf/druid/cluster/_common`目录中 -更多信息可以看[HDFS扩展](../Configuration/core-ext/hdfs.md)部分的文档。 +更多信息可以看[HDFS扩展](../../Configuration/core-ext/hdfs.md)部分的文档。 ### Hadoop连接配置 @@ -245,7 +245,7 @@ druid.indexer.logs.directory=/druid/indexing-logs 请注意，您无需为了可以从Hadoop加载数据而使用HDFS深度存储。例如，如果您的集群在Amazon Web Services上运行，即使您使用Hadoop或Elastic MapReduce加载数据，我们也建议使用S3进行深度存储。 -更多信息可以看[基于Hadoop的数据摄取](../DataIngestion/hadoopbased.md)部分的文档。 +更多信息可以看[基于Hadoop的数据摄取](../../DataIngestion/hadoopbased.md)部分的文档。 ### Zookeeper连接配置 @@ -259,7 +259,7 @@ druid.indexer.logs.directory=/druid/indexing-logs #### 从单服务器环境迁移部署 ##### Master服务 -如果您使用的是[单服务器部署示例](./chapter-3.md)中的示例配置，则这些示例中将Coordinator和Overlord进程合并为一个合并的进程。 +如果您使用的是[单服务器部署示例](chapter-3.md)中的示例配置，则这些示例中将Coordinator和Overlord进程合并为一个合并的进程。 `conf/druid/cluster/master/coordinator-overlord` 下的示例配置同样合并了Coordinator和Overlord进程。 @@ -334,7 +334,7 @@ druid.indexer.fork.property.druid.processing.numThreads=1 `conf/druid/cluster`下的配置已经为此硬件确定了，一般情况下您无需做进一步的修改。 -如果您选择了其他硬件，则[基本的集群调整指南](../Operations/basicClusterTuning.md)可以帮助您调整配置大小。 +如果您选择了其他硬件，则[基本的集群调整指南](../../Operations/basicClusterTuning.md)可以帮助您调整配置大小。 ### 开启端口(如果使用了防火墙) @@ -410,7 +410,7 @@ bin/start-cluster-data-server bin/start-cluster-query-server ``` -您可以根据查询负载添加更多查询服务器。 如果增加了查询服务器的数量，请确保按照[基本集群调优指南](../Operations/basicClusterTuning.md)中的说明调整Historical和Task上的连接池。 +您可以根据查询负载添加更多查询服务器。 如果增加了查询服务器的数量，请确保按照[基本集群调优指南](../../Operations/basicClusterTuning.md)中的说明调整Historical和Task上的连接池。 ### 加载数据 diff --git a/GettingStarted/img/tutorial-quickstart-01.png b/tutorials/img/tutorial-quickstart-01.png similarity index 100% rename from GettingStarted/img/tutorial-quickstart-01.png rename to tutorials/img/tutorial-quickstart-01.png diff --git a/tutorials/index.md b/tutorials/index.md new file mode 100644 index 0000000..5c8bdd4 --- /dev/null +++ b/tutorials/index.md @@ -0,0 +1,268 @@ +--- +id: index +title: "Quickstart" +--- + + + + +This quickstart gets you started with Apache Druid and introduces you to some of its basic features. +Following these steps, you will install Druid and load sample +data using its native batch ingestion feature. + +Before starting, you may want to read the [general Druid overview](../design/index.md) and +[ingestion overview](../ingestion/index.md), as the tutorials refer to concepts discussed on those pages. + +## Requirements + +You can follow these steps on a relatively small machine, such as a laptop with around 4 CPU and 16 GB of RAM. + +Druid comes with several startup configuration profiles for a range of machine sizes. +The `micro-quickstart`configuration profile shown here is suitable for evaluating Druid. If you want to +try out Druid's performance or scaling capabilities, you'll need a larger machine and configuration profile. + +The configuration profiles included with Druid range from the even smaller _Nano-Quickstart_ configuration (1 CPU, 4GB RAM) +to the _X-Large_ configuration (64 CPU, 512GB RAM). For more information, see +[Single server deployment](../operations/single-server.md). Alternatively, see [Clustered deployment](./cluster.md) for +information on deploying Druid services across clustered machines. + +The software requirements for the installation machine are: + +* Linux, Mac OS X, or other Unix-like OS (Windows is not supported) +* Java 8, Update 92 or later (8u92+) + +> Druid officially supports Java 8 only. Support for later major versions of Java is currently in experimental status. + +> Druid relies on the environment variables `JAVA_HOME` or `DRUID_JAVA_HOME` to find Java on the machine. You can set +`DRUID_JAVA_HOME` if there is more than one instance of Java. To verify Java requirements for your environment, run the +`bin/verify-java` script. + +Before installing a production Druid instance, be sure to consider the user account on the operating system under +which Druid will run. This is important because any Druid console user will have, effectively, the same permissions as +that user. So, for example, the file browser UI will show console users the files that the underlying user can +access. In general, avoid running Druid as root user. Consider creating a dedicated user account for running Druid. + +## Step 1. Install Druid + +After confirming the [requirements](#requirements), follow these steps: + +1. Download +the [{{DRUIDVERSION}} release](https://www.apache.org/dyn/closer.cgi?path=/druid/{{DRUIDVERSION}}/apache-druid-{{DRUIDVERSION}}-bin.tar.gz). +2. In your terminal, extract Druid and change directories to the distribution directory: + + ```bash + tar -xzf apache-druid-{{DRUIDVERSION}}-bin.tar.gz + cd apache-druid-{{DRUIDVERSION}} + ``` +In the directory, you'll find `LICENSE` and `NOTICE` files and subdirectories for executable files, configuration files, sample data and more. + +## Step 2: Start up Druid services + +Start up Druid services using the `micro-quickstart` single-machine configuration. + +From the apache-druid-{{DRUIDVERSION}} package root, run the following command: + +```bash +./bin/start-micro-quickstart +``` + +This brings up instances of ZooKeeper and the Druid services: + +```bash +$ ./bin/start-micro-quickstart +[Fri May 3 11:40:50 2019] Running command[zk], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/zk.log]: bin/run-zk conf +[Fri May 3 11:40:50 2019] Running command[coordinator-overlord], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/coordinator-overlord.log]: bin/run-druid coordinator-overlord conf/druid/single-server/micro-quickstart +[Fri May 3 11:40:50 2019] Running command[broker], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/broker.log]: bin/run-druid broker conf/druid/single-server/micro-quickstart +[Fri May 3 11:40:50 2019] Running command[router], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/router.log]: bin/run-druid router conf/druid/single-server/micro-quickstart +[Fri May 3 11:40:50 2019] Running command[historical], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/historical.log]: bin/run-druid historical conf/druid/single-server/micro-quickstart +[Fri May 3 11:40:50 2019] Running command[middleManager], logging to[/apache-druid-{{DRUIDVERSION}}/var/sv/middleManager.log]: bin/run-druid middleManager conf/druid/single-server/micro-quickstart +``` + +All persistent state, such as the cluster metadata store and segments for the services, are kept in the `var` directory under +the Druid root directory, apache-druid-{{DRUIDVERSION}}. Each service writes to a log file under `var/sv`, as noted in the startup script output above. + +At any time, you can revert Druid to its original, post-installation state by deleting the entire `var` directory. You may +want to do this, for example, between Druid tutorials or after experimentation, to start with a fresh instance. + +To stop Druid at any time, use CTRL-C in the terminal. This exits the `bin/start-micro-quickstart` script and +terminates all Druid processes. + + +## Step 3. Open the Druid console + +After the Druid services finish startup, open the [Druid console](../operations/druid-console.md) at [http://localhost:8888](http://localhost:8888). + +![Druid console](../assets/tutorial-quickstart-01.png "Druid console") + +It may take a few seconds for all Druid services to finish starting, including the [Druid router](../design/router.md), which serves the console. If you attempt to open the Druid console before startup is complete, you may see errors in the browser. Wait a few moments and try again. + + +## Step 4. Load data + + +Ingestion specs define the schema of the data Druid reads and stores. You can write ingestion specs by hand or using the _data loader_, +as we'll do here to perform batch file loading with Druid's native batch ingestion. + +The Druid distribution bundles sample data we can use. The sample data located in `quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz` +in the Druid root directory represents Wikipedia page edits for a given day. + +1. Click **Load data** from the Druid console header (![Load data](../assets/tutorial-batch-data-loader-00.png)). + +2. Select the **Local disk** tile and then click **Connect data**. + + ![Data loader init](../assets/tutorial-batch-data-loader-01.png "Data loader init") + +3. Enter the following values: + + - **Base directory**: `quickstart/tutorial/` + + - **File filter**: `wikiticker-2015-09-12-sampled.json.gz` + + ![Data location](../assets/tutorial-batch-data-loader-015.png "Data location") + + Entering the base directory and [wildcard file filter](https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/filefilter/WildcardFileFilter.html) separately, as afforded by the UI, allows you to specify multiple files for ingestion at once. + +4. Click **Apply**. + + The data loader displays the raw data, giving you a chance to verify that the data + appears as expected. + + ![Data loader sample](../assets/tutorial-batch-data-loader-02.png "Data loader sample") + + Notice that your position in the sequence of steps to load data, **Connect** in our case, appears at the top of the console, as shown below. + You can click other steps to move forward or backward in the sequence at any time. + + ![Load data](../assets/tutorial-batch-data-loader-12.png) + + +5. Click **Next: Parse data**. + + The data loader tries to determine the parser appropriate for the data format automatically. In this case + it identifies the data format as `json`, as shown in the **Input format** field at the bottom right. + + ![Data loader parse data](../assets/tutorial-batch-data-loader-03.png "Data loader parse data") + + Feel free to select other **Input format** options to get a sense of their configuration settings + and how Druid parses other types of data. + +6. With the JSON parser selected, click **Next: Parse time**. The **Parse time** settings are where you view and adjust the + primary timestamp column for the data. + + ![Data loader parse time](../assets/tutorial-batch-data-loader-04.png "Data loader parse time") + + Druid requires data to have a primary timestamp column (internally stored in a column called `__time`). + If you do not have a timestamp in your data, select `Constant value`. In our example, the data loader + determines that the `time` column is the only candidate that can be used as the primary time column. + +7. Click **Next: Transform**, **Next: Filter**, and then **Next: Configure schema**, skipping a few steps. + + You do not need to adjust transformation or filtering settings, as applying ingestion time transforms and + filters are out of scope for this tutorial. + +8. The Configure schema settings are where you configure what [dimensions](../ingestion/index.md#dimensions) + and [metrics](../ingestion/index.md#metrics) are ingested. The outcome of this configuration represents exactly how the + data will appear in Druid after ingestion. + + Since our dataset is very small, you can turn off [rollup](../ingestion/index.md#rollup) + by unsetting the **Rollup** switch and confirming the change when prompted. + + ![Data loader schema](../assets/tutorial-batch-data-loader-05.png "Data loader schema") + + +10. Click **Next: Partition** to configure how the data will be split into segments. In this case, choose `DAY` as + the **Segment granularity**. + + ![Data loader partition](../assets/tutorial-batch-data-loader-06.png "Data loader partition") + + Since this is a small dataset, we can have just a single segment, which is what selecting `DAY` as the + segment granularity gives us. + +11. Click **Next: Tune** and **Next: Publish**. + +12. The Publish settings are where you specify the datasource name in Druid. Let's change the default name from +`wikiticker-2015-09-12-sampled` to `wikipedia`. + + ![Data loader publish](../assets/tutorial-batch-data-loader-07.png "Data loader publish") + + +13. Click **Next: Edit spec** to review the ingestion spec we've constructed with the data loader. + + ![Data loader spec](../assets/tutorial-batch-data-loader-08.png "Data loader spec") + + Feel free to go back and change settings from previous steps to see how doing so updates the spec. + Similarly, you can edit the spec directly and see it reflected in the previous steps. + + > For other ways to load ingestion specs in Druid, see [Tutorial: Loading a file](./tutorial-batch.md). + +14. Once you are satisfied with the spec, click **Submit**. + + The new task for our wikipedia datasource now appears in the Ingestion view. + + ![Tasks view](../assets/tutorial-batch-data-loader-09.png "Tasks view") + + The task may take a minute or two to complete. When done, the task status should be "SUCCESS", with + the duration of the task indicated. Note that the view is set to automatically + refresh, so you do not need to refresh the browser to see the status change. + + A successful task means that one or more segments have been built and are now picked up by our data servers. + + +## Step 5. Query the data + +You can now see the data as a datasource in the console and try out a query, as follows: + +1. Click **Datasources** from the console header. + + If the wikipedia datasource doesn't appear, wait a few moments for the segment to finish loading. A datasource is + queryable once it is shown to be "Fully available" in the **Availability** column. + +2. When the datasource is available, open the Actions menu (![Actions](../assets/datasources-action-button.png)) for that + datasource and choose **Query with SQL**. + + ![Datasource view](../assets/tutorial-batch-data-loader-10.png "Datasource view") + + > Notice the other actions you can perform for a datasource, including configuring retention rules, compaction, and more. + +3. Run the prepopulated query, `SELECT * FROM "wikipedia"` to see the results. + + ![Query view](../assets/tutorial-batch-data-loader-11.png "Query view") + +Congratulations! You've gone from downloading Druid to querying data in just one quickstart. See the following +section for what to do next. + + +## Next steps + +After finishing the quickstart, check out the [query tutorial](../tutorials/tutorial-query.md) to further explore +Query features in the Druid console. + +Alternatively, learn about other ways to ingest data in one of these tutorials: + +- [Loading stream data from Apache Kafka](./tutorial-kafka.md) – How to load streaming data from a Kafka topic. +- [Loading a file using Apache Hadoop](./tutorial-batch-hadoop.md) – How to perform a batch file load, using a remote Hadoop cluster. +- [Writing your own ingestion spec](./tutorial-ingestion-spec.md) – How to write a new ingestion spec and use it to load data. + + +Remember that after stopping Druid services, you can start clean next time by deleting the `var` directory from the Druid root directory and +running the `bin/start-micro-quickstart` script again. You will likely want to do this before taking other data ingestion tutorials, +since in them you will create the same wikipedia datasource. + + + diff --git a/tutorials/tutorial-batch-hadoop.md b/tutorials/tutorial-batch-hadoop.md new file mode 100644 index 0000000..38abbfa --- /dev/null +++ b/tutorials/tutorial-batch-hadoop.md @@ -0,0 +1,260 @@ +--- +id: tutorial-batch-hadoop +title: "Tutorial: Load batch data using Apache Hadoop" +sidebar_label: "Load from Apache Hadoop" +--- + + + + +This tutorial shows you how to load data files into Apache Druid using a remote Hadoop cluster. + +For this tutorial, we'll assume that you've already completed the previous +[batch ingestion tutorial](tutorial-batch.html) using Druid's native batch ingestion system and are using the +`micro-quickstart` single-machine configuration as described in the [quickstart](index.html). + +## Install Docker + +This tutorial requires [Docker](https://docs.docker.com/install/) to be installed on the tutorial machine. + +Once the Docker install is complete, please proceed to the next steps in the tutorial. + +## Build the Hadoop docker image + +For this tutorial, we've provided a Dockerfile for a Hadoop 2.8.5 cluster, which we'll use to run the batch indexing task. + +This Dockerfile and related files are located at `quickstart/tutorial/hadoop/docker`. + +From the apache-druid-{{DRUIDVERSION}} package root, run the following commands to build a Docker image named "druid-hadoop-demo" with version tag "2.8.5": + +```bash +cd quickstart/tutorial/hadoop/docker +docker build -t druid-hadoop-demo:2.8.5 . +``` + +This will start building the Hadoop image. Once the image build is done, you should see the message `Successfully tagged druid-hadoop-demo:2.8.5` printed to the console. + +## Setup the Hadoop docker cluster + +### Create temporary shared directory + +We'll need a shared folder between the host and the Hadoop container for transferring some files. + +Let's create some folders under `/tmp`, we will use these later when starting the Hadoop container: + +```bash +mkdir -p /tmp/shared +mkdir -p /tmp/shared/hadoop_xml +``` + +### Configure /etc/hosts + +On the host machine, add the following entry to `/etc/hosts`: + +``` +127.0.0.1 druid-hadoop-demo +``` + +### Start the Hadoop container + +Once the `/tmp/shared` folder has been created and the `etc/hosts` entry has been added, run the following command to start the Hadoop container. + +```bash +docker run -it -h druid-hadoop-demo --name druid-hadoop-demo -p 2049:2049 -p 2122:2122 -p 8020:8020 -p 8021:8021 -p 8030:8030 -p 8031:8031 -p 8032:8032 -p 8033:8033 -p 8040:8040 -p 8042:8042 -p 8088:8088 -p 8443:8443 -p 9000:9000 -p 10020:10020 -p 19888:19888 -p 34455:34455 -p 49707:49707 -p 50010:50010 -p 50020:50020 -p 50030:50030 -p 50060:50060 -p 50070:50070 -p 50075:50075 -p 50090:50090 -p 51111:51111 -v /tmp/shared:/shared druid-hadoop-demo:2.8.5 /etc/bootstrap.sh -bash +``` + +Once the container is started, your terminal will attach to a bash shell running inside the container: + +```bash +Starting sshd: [ OK ] +18/07/26 17:27:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable +Starting namenodes on [druid-hadoop-demo] +druid-hadoop-demo: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-druid-hadoop-demo.out +localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-druid-hadoop-demo.out +Starting secondary namenodes [0.0.0.0] +0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-druid-hadoop-demo.out +18/07/26 17:27:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable +starting yarn daemons +starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-druid-hadoop-demo.out +localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-druid-hadoop-demo.out +starting historyserver, logging to /usr/local/hadoop/logs/mapred--historyserver-druid-hadoop-demo.out +bash-4.1# +``` + +The `Unable to load native-hadoop library for your platform... using builtin-java classes where applicable` warning messages can be safely ignored. + +#### Accessing the Hadoop container shell + +To open another shell to the Hadoop container, run the following command: + +``` +docker exec -it druid-hadoop-demo bash +``` + +### Copy input data to the Hadoop container + +From the apache-druid-{{DRUIDVERSION}} package root on the host, copy the `quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz` sample data to the shared folder: + +```bash +cp quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz /tmp/shared/wikiticker-2015-09-12-sampled.json.gz +``` + +### Setup HDFS directories + +In the Hadoop container's shell, run the following commands to setup the HDFS directories needed by this tutorial and copy the input data to HDFS. + +```bash +cd /usr/local/hadoop/bin +./hdfs dfs -mkdir /druid +./hdfs dfs -mkdir /druid/segments +./hdfs dfs -mkdir /quickstart +./hdfs dfs -chmod 777 /druid +./hdfs dfs -chmod 777 /druid/segments +./hdfs dfs -chmod 777 /quickstart +./hdfs dfs -chmod -R 777 /tmp +./hdfs dfs -chmod -R 777 /user +./hdfs dfs -put /shared/wikiticker-2015-09-12-sampled.json.gz /quickstart/wikiticker-2015-09-12-sampled.json.gz +``` + +If you encounter namenode errors when running this command, the Hadoop container may not be finished initializing. When this occurs, wait a couple of minutes and retry the commands. + +## Configure Druid to use Hadoop + +Some additional steps are needed to configure the Druid cluster for Hadoop batch indexing. + +### Copy Hadoop configuration to Druid classpath + +From the Hadoop container's shell, run the following command to copy the Hadoop .xml configuration files to the shared folder: + +```bash +cp /usr/local/hadoop/etc/hadoop/*.xml /shared/hadoop_xml +``` + +From the host machine, run the following, where {PATH_TO_DRUID} is replaced by the path to the Druid package. + +```bash +mkdir -p {PATH_TO_DRUID}/conf/druid/single-server/micro-quickstart/_common/hadoop-xml +cp /tmp/shared/hadoop_xml/*.xml {PATH_TO_DRUID}/conf/druid/single-server/micro-quickstart/_common/hadoop-xml/ +``` + +### Update Druid segment and log storage + +In your favorite text editor, open `conf/druid/single-server/micro-quickstart/_common/common.runtime.properties`, and make the following edits: + +#### Disable local deep storage and enable HDFS deep storage + +``` +# +# Deep storage +# + +# For local disk (only viable in a cluster if this is a network mount): +#druid.storage.type=local +#druid.storage.storageDirectory=var/druid/segments + +# For HDFS: +druid.storage.type=hdfs +druid.storage.storageDirectory=/druid/segments +``` + + +#### Disable local log storage and enable HDFS log storage + +``` +# +# Indexing service logs +# + +# For local disk (only viable in a cluster if this is a network mount): +#druid.indexer.logs.type=file +#druid.indexer.logs.directory=var/druid/indexing-logs + +# For HDFS: +druid.indexer.logs.type=hdfs +druid.indexer.logs.directory=/druid/indexing-logs + +``` + +### Restart Druid cluster + +Once the Hadoop .xml files have been copied to the Druid cluster and the segment/log storage configuration has been updated to use HDFS, the Druid cluster needs to be restarted for the new configurations to take effect. + +If the cluster is still running, CTRL-C to terminate the `bin/start-micro-quickstart` script, and re-run it to bring the Druid services back up. + +## Load batch data + +We've included a sample of Wikipedia edits from September 12, 2015 to get you started. + +To load this data into Druid, you can submit an *ingestion task* pointing to the file. We've included +a task that loads the `wikiticker-2015-09-12-sampled.json.gz` file included in the archive. + +Let's submit the `wikipedia-index-hadoop-.json` task: + +```bash +bin/post-index-task --file quickstart/tutorial/wikipedia-index-hadoop.json --url http://localhost:8081 +``` + +## Querying your data + +After the data load is complete, please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data. + +## Cleanup + +This tutorial is only meant to be used together with the [query tutorial](../tutorials/tutorial-query.md). + +If you wish to go through any of the other tutorials, you will need to: +* Shut down the cluster and reset the cluster state by removing the contents of the `var` directory under the druid package. +* Revert the deep storage and task storage config back to local types in `conf/druid/single-server/micro-quickstart/_common/common.runtime.properties` +* Restart the cluster + +This is necessary because the other ingestion tutorials will write to the same "wikipedia" datasource, and later tutorials expect the cluster to use local deep storage. + +Example reverted config: + +``` +# +# Deep storage +# + +# For local disk (only viable in a cluster if this is a network mount): +druid.storage.type=local +druid.storage.storageDirectory=var/druid/segments + +# For HDFS: +#druid.storage.type=hdfs +#druid.storage.storageDirectory=/druid/segments + +# +# Indexing service logs +# + +# For local disk (only viable in a cluster if this is a network mount): +druid.indexer.logs.type=file +druid.indexer.logs.directory=var/druid/indexing-logs + +# For HDFS: +#druid.indexer.logs.type=hdfs +#druid.indexer.logs.directory=/druid/indexing-logs + +``` + +## Further reading + +For more information on loading batch data with Hadoop, please see [the Hadoop batch ingestion documentation](../ingestion/hadoop.md). diff --git a/tutorials/tutorial-compaction.md b/tutorials/tutorial-compaction.md new file mode 100644 index 0000000..9805217 --- /dev/null +++ b/tutorials/tutorial-compaction.md @@ -0,0 +1,173 @@ +--- +id: tutorial-compaction +title: "Tutorial: Compacting segments" +sidebar_label: "Compacting segments" +--- + + + + +This tutorial demonstrates how to compact existing segments into fewer but larger segments. + +Because there is some per-segment memory and processing overhead, it can sometimes be beneficial to reduce the total number of segments. +Please check [Segment size optimization](../operations/segment-optimization.md) for details. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md). + +## Load the initial data + +For this tutorial, we'll be using the Wikipedia edits sample data, with an ingestion task spec that will create 1-3 segments per hour in the input data. + +The ingestion spec can be found at `quickstart/tutorial/compaction-init-index.json`. Let's submit that spec, which will create a datasource called `compaction-tutorial`: + +```bash +bin/post-index-task --file quickstart/tutorial/compaction-init-index.json --url http://localhost:8081 +``` + +> Please note that `maxRowsPerSegment` in the ingestion spec is set to 1000. This is to generate multiple segments per hour and _NOT_ recommended in production. +> It's 5000000 by default and may need to be adjusted to make your segments optimized. + +After the ingestion completes, go to [http://localhost:8888/unified-console.html#datasources](http://localhost:8888/unified-console.html#datasources) in a browser to see the new datasource in the Druid Console. + +![compaction-tutorial datasource](../assets/tutorial-compaction-01.png "compaction-tutorial datasource") + +Click the `51 segments` link next to "Fully Available" for the `compaction-tutorial` datasource to view information about the datasource's segments: + +There will be 51 segments for this datasource, 1-3 segments per hour in the input data: + +![Original segments](../assets/tutorial-compaction-02.png "Original segments") + +Running a COUNT(*) query on this datasource shows that there are 39,244 rows: + +```bash +dsql> select count(*) from "compaction-tutorial"; +┌────────┐ +│ EXPR$0 │ +├────────┤ +│ 39244 │ +└────────┘ +Retrieved 1 row in 1.38s. +``` + +## Compact the data + +Let's now compact these 51 small segments. + +We have included a compaction task spec for this tutorial datasource at `quickstart/tutorial/compaction-keep-granularity.json`: + +```json +{ + "type": "compact", + "dataSource": "compaction-tutorial", + "interval": "2015-09-12/2015-09-13", + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000, + "maxRowsInMemory" : 25000 + } +} +``` + +This will compact all segments for the interval `2015-09-12/2015-09-13` in the `compaction-tutorial` datasource. + +The parameters in the `tuningConfig` control how many segments will be present in the compacted set of segments. + +In this tutorial example, only one compacted segment will be created per hour, as each hour has less rows than the 5000000 `maxRowsPerSegment` (note that the total number of rows is 39244). + +Let's submit this task now: + +```bash +bin/post-index-task --file quickstart/tutorial/compaction-keep-granularity.json --url http://localhost:8081 +``` + +After the task finishes, refresh the [segments view](http://localhost:8888/unified-console.html#segments). + +The original 51 segments will eventually be marked as "unused" by the Coordinator and removed, with the new compacted segments remaining. + +By default, the Druid Coordinator will not mark segments as unused until the Coordinator process has been up for at least 15 minutes, so you may see the old segment set and the new compacted set at the same time in the Druid Console, with 75 total segments: + +![Compacted segments intermediate state 1](../assets/tutorial-compaction-03.png "Compacted segments intermediate state 1") + +![Compacted segments intermediate state 2](../assets/tutorial-compaction-04.png "Compacted segments intermediate state 2") + +The new compacted segments have a more recent version than the original segments, so even when both sets of segments are shown in the Druid Console, queries will only read from the new compacted segments. + +Let's try running a COUNT(*) on `compaction-tutorial` again, where the row count should still be 39,244: + +```bash +dsql> select count(*) from "compaction-tutorial"; +┌────────┐ +│ EXPR$0 │ +├────────┤ +│ 39244 │ +└────────┘ +Retrieved 1 row in 1.30s. +``` + +After the Coordinator has been running for at least 15 minutes, the [segments view](http://localhost:8888/unified-console.html#segments) should show there are 24 segments, one per hour: + +![Compacted segments hourly granularity 1](../assets/tutorial-compaction-05.png "Compacted segments hourly granularity 1") + +![Compacted segments hourly granularity 2](../assets/tutorial-compaction-06.png "Compacted segments hourly granularity 2") + +## Compact the data with new segment granularity + +The compaction task can also produce compacted segments with a granularity different from the granularity of the input segments. + +We have included a compaction task spec that will create DAY granularity segments at `quickstart/tutorial/compaction-day-granularity.json`: + +```json +{ + "type": "compact", + "dataSource": "compaction-tutorial", + "interval": "2015-09-12/2015-09-13", + "segmentGranularity": "DAY", + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000, + "maxRowsInMemory" : 25000, + "forceExtendableShardSpecs" : true + } +} +``` + +Note that `segmentGranularity` is set to `DAY` in this compaction task spec. + +Let's submit this task now: + +```bash +bin/post-index-task --file quickstart/tutorial/compaction-day-granularity.json --url http://localhost:8081 +``` + +It will take a bit of time before the Coordinator marks the old input segments as unused, so you may see an intermediate state with 25 total segments. Eventually, there will only be one DAY granularity segment: + +![Compacted segments day granularity 1](../assets/tutorial-compaction-07.png "Compacted segments day granularity 1") + +![Compacted segments day granularity 2](../assets/tutorial-compaction-08.png "Compacted segments day granularity 2") + + +## Further reading + +[Task documentation](../ingestion/tasks.md) + +[Segment optimization](../operations/segment-optimization.md) diff --git a/tutorials/tutorial-delete-data.md b/tutorials/tutorial-delete-data.md new file mode 100644 index 0000000..4f08b0e --- /dev/null +++ b/tutorials/tutorial-delete-data.md @@ -0,0 +1,180 @@ +--- +id: tutorial-delete-data +title: "Tutorial: Deleting data" +sidebar_label: "Deleting data" +--- + + + + +This tutorial demonstrates how to delete existing data. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +## Load initial data + +In this tutorial, we will use the Wikipedia edits data, with an indexing spec that creates hourly segments. This spec is located at `quickstart/tutorial/deletion-index.json`, and it creates a datasource called `deletion-tutorial`. + +Let's load this initial data: + +```bash +bin/post-index-task --file quickstart/tutorial/deletion-index.json --url http://localhost:8081 +``` + +When the load finishes, open [http://localhost:8888/unified-console.html#datasources](http://localhost:8888/unified-console.html#datasources) in a browser. + +## How to permanently delete data + +Permanent deletion of a Druid segment has two steps: + +1. The segment must first be marked as "unused". This occurs when a user manually disables a segment through the Coordinator API. +2. After segments have been marked as "unused", a Kill Task will delete any "unused" segments from Druid's metadata store as well as deep storage. + +Let's drop some segments now, by using the coordinator API to drop data by interval and segmentIds. + +## Disable segments by interval + +Let's disable segments in a specified interval. This will mark all segments in the interval as "unused", but not remove them from deep storage. +Let's disable segments in interval `2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z` i.e. between hour 18 and 20. + +```bash +curl -X 'POST' -H 'Content-Type:application/json' -d '{ "interval" : "2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z" }' http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused +``` + +After that command completes, you should see that the segment for hour 18 and 19 have been disabled: + +![Segments 2](../assets/tutorial-deletion-02.png "Segments 2") + +Note that the hour 18 and 19 segments are still present in deep storage: + +```bash +$ ls -l1 var/druid/segments/deletion-tutorial/ +2015-09-12T00:00:00.000Z_2015-09-12T01:00:00.000Z +2015-09-12T01:00:00.000Z_2015-09-12T02:00:00.000Z +2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z +2015-09-12T03:00:00.000Z_2015-09-12T04:00:00.000Z +2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z +2015-09-12T05:00:00.000Z_2015-09-12T06:00:00.000Z +2015-09-12T06:00:00.000Z_2015-09-12T07:00:00.000Z +2015-09-12T07:00:00.000Z_2015-09-12T08:00:00.000Z +2015-09-12T08:00:00.000Z_2015-09-12T09:00:00.000Z +2015-09-12T09:00:00.000Z_2015-09-12T10:00:00.000Z +2015-09-12T10:00:00.000Z_2015-09-12T11:00:00.000Z +2015-09-12T11:00:00.000Z_2015-09-12T12:00:00.000Z +2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z +2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z +2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z +2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z +2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z +2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z +2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z +2015-09-12T19:00:00.000Z_2015-09-12T20:00:00.000Z +2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z +2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z +2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z +2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z +``` + +## Disable segments by segment IDs + +Let's disable some segments by their segmentID. This will again mark the segments as "unused", but not remove them from deep storage. You can see the full segmentID for a segment from UI as explained below. + +In the [segments view](http://localhost:8888/unified-console.html#segments), click the arrow on the left side of one of the remaining segments to expand the segment entry: + +![Segments](../assets/tutorial-deletion-01.png "Segments") + +The top of the info box shows the full segment ID, e.g. `deletion-tutorial_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2019-02-28T01:11:51.606Z` for the segment of hour 14. + +Let's disable the hour 13 and 14 segments by sending a POST request to the Coordinator with this payload + +```json +{ + "segmentIds": + [ + "deletion-tutorial_2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z_2019-05-01T17:38:46.961Z", + "deletion-tutorial_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2019-05-01T17:38:46.961Z" + ] +} +``` + +This payload json has been provided at `quickstart/tutorial/deletion-disable-segments.json`. Submit the POST request to Coordinator like this: + +```bash +curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/deletion-disable-segments.json http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused +``` + +After that command completes, you should see that the segments for hour 13 and 14 have been disabled: + +![Segments 3](../assets/tutorial-deletion-03.png "Segments 3") + +Note that the hour 13 and 14 segments are still in deep storage: + +```bash +$ ls -l1 var/druid/segments/deletion-tutorial/ +2015-09-12T00:00:00.000Z_2015-09-12T01:00:00.000Z +2015-09-12T01:00:00.000Z_2015-09-12T02:00:00.000Z +2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z +2015-09-12T03:00:00.000Z_2015-09-12T04:00:00.000Z +2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z +2015-09-12T05:00:00.000Z_2015-09-12T06:00:00.000Z +2015-09-12T06:00:00.000Z_2015-09-12T07:00:00.000Z +2015-09-12T07:00:00.000Z_2015-09-12T08:00:00.000Z +2015-09-12T08:00:00.000Z_2015-09-12T09:00:00.000Z +2015-09-12T09:00:00.000Z_2015-09-12T10:00:00.000Z +2015-09-12T10:00:00.000Z_2015-09-12T11:00:00.000Z +2015-09-12T11:00:00.000Z_2015-09-12T12:00:00.000Z +2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z +2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z +2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z +2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z +2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z +2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z +2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z +2015-09-12T19:00:00.000Z_2015-09-12T20:00:00.000Z +2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z +2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z +2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z +2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z +``` + +## Run a kill task + +Now that we have disabled some segments, we can submit a Kill Task, which will delete the disabled segments from metadata and deep storage. + +A Kill Task spec has been provided at `quickstart/tutorial/deletion-kill.json`. Submit this task to the Overlord with the following command: + +```bash +curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/deletion-kill.json http://localhost:8081/druid/indexer/v1/task +``` + +After this task completes, you can see that the disabled segments have now been removed from deep storage: + +```bash +$ ls -l1 var/druid/segments/deletion-tutorial/ +2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z +2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z +2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z +2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z +2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z +2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z +2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z +2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z +``` diff --git a/tutorials/tutorial-ingestion-spec.md b/tutorials/tutorial-ingestion-spec.md new file mode 100644 index 0000000..773b920 --- /dev/null +++ b/tutorials/tutorial-ingestion-spec.md @@ -0,0 +1,606 @@ +--- +id: tutorial-ingestion-spec +title: "Tutorial: Writing an ingestion spec" +sidebar_label: "Writing an ingestion spec" +--- + + + + +This tutorial will guide the reader through the process of defining an ingestion spec, pointing out key considerations and guidelines. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md), [Tutorial: Querying data](../tutorials/tutorial-query.md), and [Tutorial: Rollup](../tutorials/tutorial-rollup.md). + +## Example data + +Suppose we have the following network flow data: + +* `srcIP`: IP address of sender +* `srcPort`: Port of sender +* `dstIP`: IP address of receiver +* `dstPort`: Port of receiver +* `protocol`: IP protocol number +* `packets`: number of packets transmitted +* `bytes`: number of bytes transmitted +* `cost`: the cost of sending the traffic + +```json +{"ts":"2018-01-01T01:01:35Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":2000, "dstPort":3000, "protocol": 6, "packets":10, "bytes":1000, "cost": 1.4} +{"ts":"2018-01-01T01:01:51Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":2000, "dstPort":3000, "protocol": 6, "packets":20, "bytes":2000, "cost": 3.1} +{"ts":"2018-01-01T01:01:59Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":2000, "dstPort":3000, "protocol": 6, "packets":30, "bytes":3000, "cost": 0.4} +{"ts":"2018-01-01T01:02:14Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":5000, "dstPort":7000, "protocol": 6, "packets":40, "bytes":4000, "cost": 7.9} +{"ts":"2018-01-01T01:02:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":5000, "dstPort":7000, "protocol": 6, "packets":50, "bytes":5000, "cost": 10.2} +{"ts":"2018-01-01T01:03:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":5000, "dstPort":7000, "protocol": 6, "packets":60, "bytes":6000, "cost": 4.3} +{"ts":"2018-01-01T02:33:14Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8", "srcPort":4000, "dstPort":5000, "protocol": 17, "packets":100, "bytes":10000, "cost": 22.4} +{"ts":"2018-01-01T02:33:45Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8", "srcPort":4000, "dstPort":5000, "protocol": 17, "packets":200, "bytes":20000, "cost": 34.5} +{"ts":"2018-01-01T02:35:45Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8", "srcPort":4000, "dstPort":5000, "protocol": 17, "packets":300, "bytes":30000, "cost": 46.3} +``` + +Save the JSON contents above into a file called `ingestion-tutorial-data.json` in `quickstart/`. + +Let's walk through the process of defining an ingestion spec that can load this data. + +For this tutorial, we will be using the native batch indexing task. When using other task types, some aspects of the ingestion spec will differ, and this tutorial will point out such areas. + +## Defining the schema + +The core element of a Druid ingestion spec is the `dataSchema`. The `dataSchema` defines how to parse input data into a set of columns that will be stored in Druid. + +Let's start with an empty `dataSchema` and add fields to it as we progress through the tutorial. + +Create a new file called `ingestion-tutorial-index.json` in `quickstart/` with the following contents: + +```json +"dataSchema" : {} +``` + +We will be making successive edits to this ingestion spec as we progress through the tutorial. + +### Datasource name + +The datasource name is specified by the `dataSource` parameter in the `dataSchema`. + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", +} +``` + +Let's call the tutorial datasource `ingestion-tutorial`. + +### Time column + +The `dataSchema` needs to know how to extract the main timestamp field from the input data. + +The timestamp column in our input data is named "ts", containing ISO 8601 timestamps, so let's add a `timestampSpec` with that information to the `dataSchema`: + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + } +} +``` + +### Column types + +Now that we've defined the time column, let's look at definitions for other columns. + +Druid supports the following column types: String, Long, Float, Double. We will see how these are used in the following sections. + +Before we move on to how we define our other non-time columns, let's discuss `rollup` first. + +### Rollup + +When ingesting data, we must consider whether we wish to use rollup or not. + +* If rollup is enabled, we will need to separate the input columns into two categories, "dimensions" and "metrics". "Dimensions" are the grouping columns for rollup, while "metrics" are the columns that will be aggregated. + +* If rollup is disabled, then all columns are treated as "dimensions" and no pre-aggregation occurs. + +For this tutorial, let's enable rollup. This is specified with a `granularitySpec` on the `dataSchema`. + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "granularitySpec" : { + "rollup" : true + } +} +``` + +#### Choosing dimensions and metrics + +For this example dataset, the following is a sensible split for "dimensions" and "metrics": + +* Dimensions: srcIP, srcPort, dstIP, dstPort, protocol +* Metrics: packets, bytes, cost + +The dimensions here are a group of properties that identify a unidirectional flow of IP traffic, while the metrics represent facts about the IP traffic flow specified by a dimension grouping. + +Let's look at how to define these dimensions and metrics within the ingestion spec. + +#### Dimensions + +Dimensions are specified with a `dimensionsSpec` inside the `dataSchema`. + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "granularitySpec" : { + "rollup" : true + } +} +``` + +Each dimension has a `name` and a `type`, where `type` can be "long", "float", "double", or "string". + +Note that `srcIP` is a "string" dimension; for string dimensions, it is enough to specify just a dimension name, since "string" is the default dimension type. + +Also note that `protocol` is a numeric value in the input data, but we are ingesting it as a "string" column; Druid will coerce the input longs to strings during ingestion. + +##### Strings vs. Numerics + +Should a numeric input be ingested as a numeric dimension or as a string dimension? + +Numeric dimensions have the following pros/cons relative to String dimensions: +* Pros: Numeric representation can result in smaller column sizes on disk and lower processing overhead when reading values from the column +* Cons: Numeric dimensions do not have indices, so filtering on them will often be slower than filtering on an equivalent String dimension (which has bitmap indices) + +#### Metrics + +Metrics are specified with a `metricsSpec` inside the `dataSchema`: + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "rollup" : true + } +} +``` + +When defining a metric, it is necessary to specify what type of aggregation should be performed on that column during rollup. + +Here we have defined long sum aggregations on the two long metric columns, `packets` and `bytes`, and a double sum aggregation for the `cost` column. + +Note that the `metricsSpec` is on a different nesting level than `dimensionSpec` or `parseSpec`; it belongs on the same nesting level as `parser` within the `dataSchema`. + +Note that we have also defined a `count` aggregator. The count aggregator will track how many rows in the original input data contributed to a "rolled up" row in the final ingested data. + +### No rollup + +If we were not using rollup, all columns would be specified in the `dimensionsSpec`, e.g.: + +```json + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" }, + { "name" : "packets", "type" : "long" }, + { "name" : "bytes", "type" : "long" }, + { "name" : "srcPort", "type" : "double" } + ] + }, +``` + + +### Define granularities + +At this point, we are done defining the `parser` and `metricsSpec` within the `dataSchema` and we are almost done writing the ingestion spec. + +There are some additional properties we need to set in the `granularitySpec`: +* Type of granularitySpec: `uniform` and `arbitrary` are the two supported types. For this tutorial, we will use a `uniform` granularity spec, where all segments have uniform interval sizes (for example, all segments cover an hour's worth of data). +* The segment granularity: what size of time interval should a single segment contain data for? e.g., `DAY`, `WEEK` +* The bucketing granularity of the timestamps in the time column (referred to as `queryGranularity`) + +#### Segment granularity + +Segment granularity is configured by the `segmentGranularity` property in the `granularitySpec`. For this tutorial, we'll create hourly segments: + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "rollup" : true + } +} +``` + +Our input data has events from two separate hours, so this task will generate two segments. + +#### Query granularity + +The query granularity is configured by the `queryGranularity` property in the `granularitySpec`. For this tutorial, let's use minute granularity: + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "queryGranularity" : "MINUTE", + "rollup" : true + } +} +``` + +To see the effect of the query granularity, let's look at this row from the raw input data: + +```json +{"ts":"2018-01-01T01:03:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":5000, "dstPort":7000, "protocol": 6, "packets":60, "bytes":6000, "cost": 4.3} +``` + +When this row is ingested with minute queryGranularity, Druid will floor the row's timestamp to minute buckets: + +```json +{"ts":"2018-01-01T01:03:00Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2", "srcPort":5000, "dstPort":7000, "protocol": 6, "packets":60, "bytes":6000, "cost": 4.3} +``` + +#### Define an interval (batch only) + +For batch tasks, it is necessary to define a time interval. Input rows with timestamps outside of the time interval will not be ingested. + +The interval is also specified in the `granularitySpec`: + +```json +"dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "queryGranularity" : "MINUTE", + "intervals" : ["2018-01-01/2018-01-02"], + "rollup" : true + } +} +``` + +## Define the task type + +We've now finished defining our `dataSchema`. The remaining steps are to place the `dataSchema` we created into an ingestion task spec, and specify the input source. + +The `dataSchema` is shared across all task types, but each task type has its own specification format. For this tutorial, we will use the native batch ingestion task: + +```json +{ + "type" : "index_parallel", + "spec" : { + "dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "queryGranularity" : "MINUTE", + "intervals" : ["2018-01-01/2018-01-02"], + "rollup" : true + } + } + } +} +``` + +## Define the input source + +Now let's define our input source, which is specified in an `ioConfig` object. Each task type has its own type of `ioConfig`. To read input data, we need to specify an `inputSource`. The example netflow data we saved earlier needs to be read from a local file, which is configured below: + + +```json + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/", + "filter" : "ingestion-tutorial-data.json" + } + } +``` + + +### Define the format of the data + +Since our input data is represented as JSON strings, we'll use a `inputFormat` to `json` format: + +```json + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/", + "filter" : "ingestion-tutorial-data.json" + }, + "inputFormat" : { + "type" : "json" + } + } +``` + +```json +{ + "type" : "index_parallel", + "spec" : { + "dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "queryGranularity" : "MINUTE", + "intervals" : ["2018-01-01/2018-01-02"], + "rollup" : true + } + }, + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/", + "filter" : "ingestion-tutorial-data.json" + }, + "inputFormat" : { + "type" : "json" + } + } + } +} +``` + +## Additional tuning + +Each ingestion task has a `tuningConfig` section that allows users to tune various ingestion parameters. + +As an example, let's add a `tuningConfig` that sets a target segment size for the native batch ingestion task: + +```json + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000 + } +``` + +Note that each ingestion task has its own type of `tuningConfig`. + +## Final spec + +We've finished defining the ingestion spec, it should now look like the following: + +```json +{ + "type" : "index_parallel", + "spec" : { + "dataSchema" : { + "dataSource" : "ingestion-tutorial", + "timestampSpec" : { + "format" : "iso", + "column" : "ts" + }, + "dimensionsSpec" : { + "dimensions": [ + "srcIP", + { "name" : "srcPort", "type" : "long" }, + { "name" : "dstIP", "type" : "string" }, + { "name" : "dstPort", "type" : "long" }, + { "name" : "protocol", "type" : "string" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" }, + { "type" : "doubleSum", "name" : "cost", "fieldName" : "cost" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "HOUR", + "queryGranularity" : "MINUTE", + "intervals" : ["2018-01-01/2018-01-02"], + "rollup" : true + } + }, + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/", + "filter" : "ingestion-tutorial-data.json" + }, + "inputFormat" : { + "type" : "json" + } + }, + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000 + } + } +} +``` + +## Submit the task and query the data + +From the apache-druid-{{DRUIDVERSION}} package root, run the following command: + +```bash +bin/post-index-task --file quickstart/ingestion-tutorial-index.json --url http://localhost:8081 +``` + +After the script completes, we will query the data. + +Let's run `bin/dsql` and issue a `select * from "ingestion-tutorial";` query to see what data was ingested. + +```bash +$ bin/dsql +Welcome to dsql, the command-line client for Druid SQL. +Type "\h" for help. +dsql> select * from "ingestion-tutorial"; + +┌──────────────────────────┬───────┬──────┬───────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┐ +│ __time │ bytes │ cost │ count │ dstIP │ dstPort │ packets │ protocol │ srcIP │ srcPort │ +├──────────────────────────┼───────┼──────┼───────┼─────────┼─────────┼─────────┼──────────┼─────────┼─────────┤ +│ 2018-01-01T01:01:00.000Z │ 6000 │ 4.9 │ 3 │ 2.2.2.2 │ 3000 │ 60 │ 6 │ 1.1.1.1 │ 2000 │ +│ 2018-01-01T01:02:00.000Z │ 9000 │ 18.1 │ 2 │ 2.2.2.2 │ 7000 │ 90 │ 6 │ 1.1.1.1 │ 5000 │ +│ 2018-01-01T01:03:00.000Z │ 6000 │ 4.3 │ 1 │ 2.2.2.2 │ 7000 │ 60 │ 6 │ 1.1.1.1 │ 5000 │ +│ 2018-01-01T02:33:00.000Z │ 30000 │ 56.9 │ 2 │ 8.8.8.8 │ 5000 │ 300 │ 17 │ 7.7.7.7 │ 4000 │ +│ 2018-01-01T02:35:00.000Z │ 30000 │ 46.3 │ 1 │ 8.8.8.8 │ 5000 │ 300 │ 17 │ 7.7.7.7 │ 4000 │ +└──────────────────────────┴───────┴──────┴───────┴─────────┴─────────┴─────────┴──────────┴─────────┴─────────┘ +Retrieved 5 rows in 0.12s. + +dsql> +``` diff --git a/tutorials/tutorial-kafka.md b/tutorials/tutorial-kafka.md new file mode 100644 index 0000000..1174ad8 --- /dev/null +++ b/tutorials/tutorial-kafka.md @@ -0,0 +1,278 @@ +--- +id: tutorial-kafka +title: "Tutorial: Load streaming data from Apache Kafka" +sidebar_label: "Load from Apache Kafka" +--- + + + + +## Getting started + +This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service. + +For this tutorial, we'll assume you've already downloaded Druid as described in +the [quickstart](index.html) using the `micro-quickstart` single-machine configuration and have it +running on your local machine. You don't need to have loaded any data yet. + +## Download and start Kafka + +[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with +Druid. For this tutorial, we will use Kafka 2.1.0. To download Kafka, issue the following +commands in your terminal: + +```bash +curl -O https://archive.apache.org/dist/kafka/2.1.0/kafka_2.12-2.1.0.tgz +tar -xzf kafka_2.12-2.1.0.tgz +cd kafka_2.12-2.1.0 +``` + +Start a Kafka broker by running the following command in a new terminal: + +```bash +./bin/kafka-server-start.sh config/server.properties +``` + +Run this command to create a Kafka topic called *wikipedia*, to which we'll send data: + +```bash +./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic wikipedia +``` + +## Load data into Kafka + +Let's launch a producer for our topic and send some data! + +In your Druid directory, run the following command: + +```bash +cd quickstart/tutorial +gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json +``` + +In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory: + +```bash +export KAFKA_OPTS="-Dfile.encoding=UTF-8" +./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json +``` + +The previous command posted sample events to the *wikipedia* Kafka topic. +Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic. + +## Loading data with the data loader + +Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header. + +![Data loader init](../assets/tutorial-kafka-data-loader-01.png "Data loader init") + +Select `Apache Kafka` and click `Connect data`. + +![Data loader sample](../assets/tutorial-kafka-data-loader-02.png "Data loader sample") + +Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic. + +Click `Apply` and make sure that the data you are seeing is correct. + +Once the data is located, you can click "Next: Parse data" to go to the next step. + +![Data loader parse data](../assets/tutorial-kafka-data-loader-03.png "Data loader parse data") + +The data loader will try to automatically determine the correct parser for the data. +In this case it will successfully determine `json`. +Feel free to play around with different parser options to get a preview of how Druid will parse your data. + +With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column. + +![Data loader parse time](../assets/tutorial-kafka-data-loader-04.png "Data loader parse time") + +Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`). +If you do not have a timestamp in your data, select `Constant value`. +In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column. + +Click `Next: ...` twice to go past the `Transform` and `Filter` steps. +You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial. + +![Data loader schema](../assets/tutorial-kafka-data-loader-05.png "Data loader schema") + +In the `Configure schema` step, you can configure which [dimensions](../ingestion/index.md#dimensions) and [metrics](../ingestion/index.md#metrics) will be ingested into Druid. +This is exactly what the data will appear like in Druid once it is ingested. +Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/index.md#rollup) by clicking on the switch and confirming the change. + +Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments. + +![Data loader partition](../assets/tutorial-kafka-data-loader-06.png "Data loader partition") + +Here, you can adjust how the data will be split up into segments in Druid. +Since this is a small dataset, there are no adjustments that need to be made in this step. + +Click `Next: Tune` to go to the tuning step. + +![Data loader tune](../assets/tutorial-kafka-data-loader-07.png "Data loader tune") + +In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream. +There are no other changes that need to be made hear, so click `Next: Publish` to go to the `Publish` step. + +![Data loader publish](../assets/tutorial-kafka-data-loader-08.png "Data loader publish") + +Let's name this datasource `wikipedia-kafka`. + +Finally, click `Next` to review your spec. + +![Data loader spec](../assets/tutorial-kafka-data-loader-09.png "Data loader spec") + +This is the spec you have constructed. +Feel free to go back and make changes in previous steps to see how changes will update the spec. +Similarly, you can also edit the spec directly and see it reflected in the previous steps. + +Once you are satisfied with the spec, click `Submit` and an ingestion task will be created. + +![Tasks view](../assets/tutorial-kafka-data-loader-10.png "Tasks view") + +You will be taken to the task view with the focus on the newly created supervisor. + +The task view is set to auto refresh, wait until your supervisor launches a task. + +When a tasks starts running, it will also start serving the data that it is ingesting. + +Navigate to the `Datasources` view from the header. + +![Datasource view](../assets/tutorial-kafka-data-loader-11.png "Datasource view") + +When the `wikipedia-kafka` datasource appears here it can be queried. + +*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step). + +At this point, you can go to the `Query` view to run SQL queries against the datasource. + +Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results. + +![Query view](../assets/tutorial-kafka-data-loader-12.png "Query view") + +Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data. + + +### Submit a supervisor via the console + +In the console, click `Submit supervisor` to open the submit supervisor dialog. + +![Submit supervisor](../assets/tutorial-kafka-submit-supervisor-01.png "Submit supervisor") + +Paste in this spec and click `Submit`. + +```json +{ + "type": "kafka", + "spec" : { + "dataSchema": { + "dataSource": "wikipedia", + "timestampSpec": { + "column": "time", + "format": "auto" + }, + "dimensionsSpec": { + "dimensions": [ + "channel", + "cityName", + "comment", + "countryIsoCode", + "countryName", + "isAnonymous", + "isMinor", + "isNew", + "isRobot", + "isUnpatrolled", + "metroCode", + "namespace", + "page", + "regionIsoCode", + "regionName", + "user", + { "name": "added", "type": "long" }, + { "name": "deleted", "type": "long" }, + { "name": "delta", "type": "long" } + ] + }, + "metricsSpec" : [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "DAY", + "queryGranularity": "NONE", + "rollup": false + } + }, + "tuningConfig": { + "type": "kafka", + "reportParseExceptions": false + }, + "ioConfig": { + "topic": "wikipedia", + "inputFormat": { + "type": "json" + }, + "replicas": 2, + "taskDuration": "PT10M", + "completionTimeout": "PT20M", + "consumerProperties": { + "bootstrap.servers": "localhost:9092" + } + } + } +} +``` + +This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data. + +### Submit a supervisor directly + +To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root: + +```bash +curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor +``` + + +If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`. + +For more details about what's going on here, check out the +[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md). + +You can view the current supervisors and tasks in the Druid Console: [http://localhost:8888/unified-console.html#tasks](http://localhost:8888/unified-console.html#tasks). + +## Querying your data + +After data is sent to the Kafka stream, it is immediately available for querying. + +Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data. + +## Cleanup + +To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource. + +You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`: + +```bash +rm -rf /tmp/kafka-logs +``` + + +## Further reading + +For more information on loading data from Kafka streams, please see the [Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md). diff --git a/tutorials/tutorial-kerberos-hadoop.md b/tutorials/tutorial-kerberos-hadoop.md new file mode 100644 index 0000000..254e23f --- /dev/null +++ b/tutorials/tutorial-kerberos-hadoop.md @@ -0,0 +1,123 @@ +--- +id: tutorial-kerberos-hadoop +title: "Configuring Apache Druid to use Kerberized Apache Hadoop as deep storage" +sidebar_label: "Kerberized HDFS deep storage" +--- + + + + +## Hadoop Setup + +Following are the configurations files required to be copied over to Druid conf folders: + +1. For HDFS as a deep storage, hdfs-site.xml, core-site.xml +2. For ingestion, mapred-site.xml, yarn-site.xml + +### HDFS Folders and permissions + +1. Choose any folder name for the druid deep storage, for example 'druid' +2. Create the folder in hdfs under the required parent folder. For example, +`hdfs dfs -mkdir /druid` +OR +`hdfs dfs -mkdir /apps/druid` + +3. Give druid processes appropriate permissions for the druid processes to access this folder. This would ensure that druid is able to create necessary folders like data and indexing_log in HDFS. +For example, if druid processes run as user 'root', then + + `hdfs dfs -chown root:root /apps/druid` + + OR + + `hdfs dfs -chmod 777 /apps/druid` + +Druid creates necessary sub-folders to store data and index under this newly created folder. + +## Druid Setup + +Edit common.runtime.properties at conf/druid/_common/common.runtime.properties to include the HDFS properties. Folders used for the location are same as the ones used for example above. + +### common.runtime.properties + +```properties +# Deep storage +# +# For HDFS: +druid.storage.type=hdfs +druid.storage.storageDirectory=/druid/segments +# OR +# druid.storage.storageDirectory=/apps/druid/segments + +# +# Indexing service logs +# + +# For HDFS: +druid.indexer.logs.type=hdfs +druid.indexer.logs.directory=/druid/indexing-logs +# OR +# druid.storage.storageDirectory=/apps/druid/indexing-logs +``` + +Note: Comment out Local storage and S3 Storage parameters in the file + +Also include hdfs-storage core extension to `conf/druid/_common/common.runtime.properties` + +```properties +# +# Extensions +# + +druid.extensions.directory=dist/druid/extensions +druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies +druid.extensions.loadList=["mysql-metadata-storage", "druid-hdfs-storage", "druid-kerberos"] +``` + +### Hadoop Jars + +Ensure that Druid has necessary jars to support the Hadoop version. + +Find the hadoop version using command, `hadoop version` + +In case there is other software used with hadoop, like `WanDisco`, ensure that +1. the necessary libraries are available +2. add the requisite extensions to `druid.extensions.loadlist` in `conf/druid/_common/common.runtime.properties` + +### Kerberos setup + +Create a headless keytab which would have access to the druid data and index. + +Edit conf/druid/_common/common.runtime.properties and add the following properties: + +```properties +druid.hadoop.security.kerberos.principal +druid.hadoop.security.kerberos.keytab +``` + +For example + +```properties +druid.hadoop.security.kerberos.principal=hdfs-test@EXAMPLE.IO +druid.hadoop.security.kerberos.keytab=/etc/security/keytabs/hdfs.headless.keytab +``` + +### Restart Druid Services + +With the above changes, restart Druid. This would ensure that Druid works with Kerberized Hadoop diff --git a/tutorials/tutorial-query.md b/tutorials/tutorial-query.md new file mode 100644 index 0000000..19461ac --- /dev/null +++ b/tutorials/tutorial-query.md @@ -0,0 +1,278 @@ +--- +id: tutorial-query +title: "Tutorial: Querying data" +sidebar_label: "Querying data" +--- + + + + +This tutorial demonstrates how to query data in Apache Druid using SQL. + +It assumes that you've completed the [Quickstart](../tutorials/index.md) +or one of the following tutorials, since we'll query datasources that you would have created +by following one of them: + +* [Tutorial: Loading a file](../tutorials/tutorial-batch.md) +* [Tutorial: Loading stream data from Kafka](../tutorials/tutorial-kafka.md) +* [Tutorial: Loading a file using Hadoop](../tutorials/tutorial-batch-hadoop.md) + +There are various ways to run Druid SQL queries: from the Druid console, using a command line utility +and by posting the query by HTTP. We'll look at each of these. + + +## Query SQL from the Druid console + +The Druid console includes a view that makes it easier to build and test queries, and +view their results. + +1. Start up the Druid cluster, if it's not already running, and open the Druid console in your web +browser. + +2. Click **Query** from the header to open the Query view: + + ![Query view](../assets/tutorial-query-01.png "Query view") + + You can always write queries directly in the edit pane, but the Query view also provides + facilities to help you construct SQL queries, which we will use to generate a starter query. + +3. Expand the wikipedia datasource tree in the left pane. We'll +create a query for the page dimension. + +4. Click `page` and then **Show:page** from the menu: + + ![Query select page](../assets/tutorial-query-02.png "Query select page") + + A SELECT query appears in the query edit pane and immediately runs. However, in this case, the query + returns no data, since by default the query filters for data from the last day, while our data is considerably + older than that. Let's remove the filter. + +5. In the datasource tree, click `__time` and **Remove Filter**. + + ![Clear WHERE filter](../assets/tutorial-query-03.png "Clear WHERE filter") + +6. Click **Run** to run the query. + + You should now see two columns of data, a page name and the count: + + ![Query results](../assets/tutorial-query-04.png "Query results") + + Notice that the results are limited in the console to about a hundred, by default, due to the **Smart query limit** + feature. This helps users avoid inadvertently running queries that return an excessive amount of data, possibly + overwhelming their system. + +7. Let's edit the query directly and take a look at a few more query building features in the editor. + Click in the query edit pane and make the following changes: + + 1. Add a line after the first column, `"page"` and Start typing the name of a new column, `"countryName"`. Notice that the autocomplete menu suggests column names, functions, keywords, and more. Choose "countryName" and +add the new column to the GROUP BY clause as well, either by name or by reference to its position, `2`. + + 2. For readability, replace `Count` column name with `Edits`, since the `COUNT()` function actually +returns the number of edits for the page. Make the same column name change in the ORDER BY clause as well. + + The `COUNT()` function is one of many functions available for use in Druid SQL queries. You can mouse over a function name + in the autocomplete menu to see a brief description of a function. Also, you can find more information in the Druid + documentation; for example, the `COUNT()` function is documented in + [Aggregation functions](../querying/sql.md#aggregation-functions). + + The query should now be: + + ```sql + SELECT + "page", + "countryName", + COUNT(*) AS "Edits" + FROM "wikipedia" + GROUP BY 1, 2 + ORDER BY "Edits" DESC + ``` + + When you run the query again, notice that we're getting the new dimension,`countryName`, but for most of the rows, its value + is null. Let's + show only rows with a `countryName` value. + +8. Click the countryName dimension in the left pane and choose the first filtering option. It's not exactly what we want, but +we'll edit it by hand. The new WHERE clause should appear in your query. + +8. Modify the WHERE clause to exclude results that do not have a value for countryName: + + ```sql + WHERE "countryName" IS NOT NULL + ``` + Run the query again. You should now see the top edits by country: + + ![Finished query](../assets/tutorial-query-035.png "Finished query") + +9. Under the covers, every Druid SQL query is translated into a query in the JSON-based _Druid native query_ format before it runs + on data nodes. You can view the native query for this query by clicking `...` and **Explain SQL Query**. + + While you can use Druid SQL for most purposes, familiarity with native query is useful for composing complex queries and for troubleshooting +performance issues. For more information, see [Native queries](../querying/querying.md). + + ![Explain query](../assets/tutorial-query-06.png "Explain query") + + > Another way to view the explain plan is by adding EXPLAIN PLAN FOR to the front of your query, as follows: + > + >```sql + >EXPLAIN PLAN FOR + >SELECT + > "page", + > "countryName", + > COUNT(*) AS "Edits" + >FROM "wikipedia" + >WHERE "countryName" IS NOT NULL + >GROUP BY 1, 2 + >ORDER BY "Edits" DESC + >``` + >This is particularly useful when running queries + from the command line or over HTTP. + + +9. Finally, click `...` and **Edit context** to see how you can add additional parameters controlling the execution of the query execution. In the field, enter query context options as JSON key-value pairs, as described in [Context flags](../querying/query-context.md). + +That's it! We've built a simple query using some of the query builder features built into the Druid Console. The following +sections provide a few more example queries you can try. Also, see [Other ways to invoke SQL queries](#other-ways-to-invoke-sql-queries) to learn how +to run Druid SQL from the command line or over HTTP. + +## More Druid SQL examples + +Here is a collection of queries to try out: + +### Query over time + +```sql +SELECT FLOOR(__time to HOUR) AS HourTime, SUM(deleted) AS LinesDeleted +FROM wikipedia WHERE "__time" BETWEEN TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' +GROUP BY 1 +``` + +![Query example](../assets/tutorial-query-07.png "Query example") + +### General group by + +```sql +SELECT channel, page, SUM(added) +FROM wikipedia WHERE "__time" BETWEEN TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' +GROUP BY channel, page +ORDER BY SUM(added) DESC +``` + +![Query example](../assets/tutorial-query-08.png "Query example") + + +## Other ways to invoke SQL queries + +### Query SQL via dsql + +For convenience, the Druid package includes a SQL command-line client, located at `bin/dsql` in the Druid package root. + +Let's now run `bin/dsql`; you should see the following prompt: + +```bash +Welcome to dsql, the command-line client for Druid SQL. +Type "\h" for help. +dsql> +``` + +To submit the query, paste it to the `dsql` prompt and press enter: + +```bash +dsql> SELECT page, COUNT(*) AS Edits FROM wikipedia WHERE "__time" BETWEEN TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' GROUP BY page ORDER BY Edits DESC LIMIT 10; +┌──────────────────────────────────────────────────────────┬───────┐ +│ page │ Edits │ +├──────────────────────────────────────────────────────────┼───────┤ +│ Wikipedia:Vandalismusmeldung │ 33 │ +│ User:Cyde/List of candidates for speedy deletion/Subpage │ 28 │ +│ Jeremy Corbyn │ 27 │ +│ Wikipedia:Administrators' noticeboard/Incidents │ 21 │ +│ Flavia Pennetta │ 20 │ +│ Total Drama Presents: The Ridonculous Race │ 18 │ +│ User talk:Dudeperson176123 │ 18 │ +│ Wikipédia:Le Bistro/12 septembre 2015 │ 18 │ +│ Wikipedia:In the news/Candidates │ 17 │ +│ Wikipedia:Requests for page protection │ 17 │ +└──────────────────────────────────────────────────────────┴───────┘ +Retrieved 10 rows in 0.06s. +``` + + +### Query SQL over HTTP + + +You can submit queries directly to the Druid Broker over HTTP. + +The tutorial package includes an example file that contains the SQL query shown above at `quickstart/tutorial/wikipedia-top-pages-sql.json`. Let's submit that query to the Druid Broker: + +```bash +curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/wikipedia-top-pages-sql.json http://localhost:8888/druid/v2/sql +``` + +The following results should be returned: + +```json +[ + { + "page": "Wikipedia:Vandalismusmeldung", + "Edits": 33 + }, + { + "page": "User:Cyde/List of candidates for speedy deletion/Subpage", + "Edits": 28 + }, + { + "page": "Jeremy Corbyn", + "Edits": 27 + }, + { + "page": "Wikipedia:Administrators' noticeboard/Incidents", + "Edits": 21 + }, + { + "page": "Flavia Pennetta", + "Edits": 20 + }, + { + "page": "Total Drama Presents: The Ridonculous Race", + "Edits": 18 + }, + { + "page": "User talk:Dudeperson176123", + "Edits": 18 + }, + { + "page": "Wikipédia:Le Bistro/12 septembre 2015", + "Edits": 18 + }, + { + "page": "Wikipedia:In the news/Candidates", + "Edits": 17 + }, + { + "page": "Wikipedia:Requests for page protection", + "Edits": 17 + } +] +``` + +## Further reading + +See the [Druid SQL documentation](../querying/sql.md) for more information on using Druid SQL queries. + +See the [Queries documentation](../querying/querying.md) for more information on Druid native queries. diff --git a/tutorials/tutorial-retention.md b/tutorials/tutorial-retention.md new file mode 100644 index 0000000..e4ff428 --- /dev/null +++ b/tutorials/tutorial-retention.md @@ -0,0 +1,115 @@ +--- +id: tutorial-retention +title: "Tutorial: Configuring data retention" +sidebar_label: "Configuring data retention" +--- + + + + +This tutorial demonstrates how to configure retention rules on a datasource to set the time intervals of data that will be retained or dropped. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md). + +## Load the example data + +For this tutorial, we'll be using the Wikipedia edits sample data, with an ingestion task spec that will create a separate segment for each hour in the input data. + +The ingestion spec can be found at `quickstart/tutorial/retention-index.json`. Let's submit that spec, which will create a datasource called `retention-tutorial`: + +```bash +bin/post-index-task --file quickstart/tutorial/retention-index.json --url http://localhost:8081 +``` + +After the ingestion completes, go to [http://localhost:8888/unified-console.html#datasources](http://localhost:8888/unified-console.html#datasources) in a browser to access the Druid Console's datasource view. + +This view shows the available datasources and a summary of the retention rules for each datasource: + +![Summary](../assets/tutorial-retention-01.png "Summary") + +Currently there are no rules set for the `retention-tutorial` datasource. Note that there are default rules for the cluster: load forever with 2 replicas in `_default_tier`. + +This means that all data will be loaded regardless of timestamp, and each segment will be replicated to two Historical processes in the default tier. + +In this tutorial, we will ignore the tiering and redundancy concepts for now. + +Let's view the segments for the `retention-tutorial` datasource by clicking the "24 Segments" link next to "Fully Available". + +The segments view ([http://localhost:8888/unified-console.html#segments](http://localhost:8888/unified-console.html#segments)) provides information about what segments a datasource contains. The page shows that there are 24 segments, each one containing data for a specific hour of 2015-09-12: + +![Original segments](../assets/tutorial-retention-02.png "Original segments") + +## Set retention rules + +Suppose we want to drop data for the first 12 hours of 2015-09-12 and keep data for the later 12 hours of 2015-09-12. + +Go to the [datasources view](http://localhost:8888/unified-console.html#datasources) and click the blue pencil icon next to `Cluster default: loadForever` for the `retention-tutorial` datasource. + +A rule configuration window will appear: + +![Rule configuration](../assets/tutorial-retention-03.png "Rule configuration") + +Now click the `+ New rule` button twice. + +In the upper rule box, select `Load` and `by interval`, and then enter `2015-09-12T12:00:00.000Z/2015-09-13T00:00:00.000Z` in field next to `by interval`. Replicas can remain at 2 in the `_default_tier`. + +In the lower rule box, select `Drop` and `forever`. + +The rules should look like this: + +![Set rules](../assets/tutorial-retention-04.png "Set rules") + +Now click `Next`. The rule configuration process will ask for a user name and comment, for change logging purposes. You can enter `tutorial` for both. + +Now click `Save`. You can see the new rules in the datasources view: + +![New rules](../assets/tutorial-retention-05.png "New rules") + +Give the cluster a few minutes to apply the rule change, and go to the [segments view](http://localhost:8888/unified-console.html#segments) in the Druid Console. +The segments for the first 12 hours of 2015-09-12 are now gone: + +![New segments](../assets/tutorial-retention-06.png "New segments") + +The resulting retention rule chain is the following: + +1. loadByInterval 2015-09-12T12/2015-09-13 (12 hours) + +2. dropForever + +3. loadForever (default rule) + +The rule chain is evaluated from top to bottom, with the default rule chain always added at the bottom. + +The tutorial rule chain we just created loads data if it is within the specified 12 hour interval. + +If data is not within the 12 hour interval, the rule chain evaluates `dropForever` next, which will drop any data. + +The `dropForever` terminates the rule chain, effectively overriding the default `loadForever` rule, which will never be reached in this rule chain. + +Note that in this tutorial we defined a load rule on a specific interval. + +If instead you want to retain data based on how old it is (e.g., retain data that ranges from 3 months in the past to the present time), you would define a Period load rule instead. + +## Further reading + +* [Load rules](../operations/rule-configuration.md) diff --git a/tutorials/tutorial-rollup.md b/tutorials/tutorial-rollup.md new file mode 100644 index 0000000..7927617 --- /dev/null +++ b/tutorials/tutorial-rollup.md @@ -0,0 +1,196 @@ +--- +id: tutorial-rollup +title: "Tutorial: Roll-up" +sidebar_label: "Roll-up" +--- + + + + +Apache Druid can summarize raw data at ingestion time using a process we refer to as "roll-up". Roll-up is a first-level aggregation operation over a selected set of columns that reduces the size of stored data. + +This tutorial will demonstrate the effects of roll-up on an example dataset. + +For this tutorial, we'll assume you've already downloaded Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md). + +## Example data + +For this tutorial, we'll use a small sample of network flow event data, representing packet and byte counts for traffic from a source to a destination IP address that occurred within a particular second. + +```json +{"timestamp":"2018-01-01T01:01:35Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":20,"bytes":9024} +{"timestamp":"2018-01-01T01:01:51Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":255,"bytes":21133} +{"timestamp":"2018-01-01T01:01:59Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":11,"bytes":5780} +{"timestamp":"2018-01-01T01:02:14Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":38,"bytes":6289} +{"timestamp":"2018-01-01T01:02:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":377,"bytes":359971} +{"timestamp":"2018-01-01T01:03:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":49,"bytes":10204} +{"timestamp":"2018-01-02T21:33:14Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8","packets":38,"bytes":6289} +{"timestamp":"2018-01-02T21:33:45Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8","packets":123,"bytes":93999} +{"timestamp":"2018-01-02T21:35:45Z","srcIP":"7.7.7.7", "dstIP":"8.8.8.8","packets":12,"bytes":2818} +``` + +A file containing this sample input data is located at `quickstart/tutorial/rollup-data.json`. + +We'll ingest this data using the following ingestion task spec, located at `quickstart/tutorial/rollup-index.json`. + +```json +{ + "type" : "index_parallel", + "spec" : { + "dataSchema" : { + "dataSource" : "rollup-tutorial", + "dimensionsSpec" : { + "dimensions" : [ + "srcIP", + "dstIP" + ] + }, + "timestampSpec": { + "column": "timestamp", + "format": "iso" + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "packets", "fieldName" : "packets" }, + { "type" : "longSum", "name" : "bytes", "fieldName" : "bytes" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "week", + "queryGranularity" : "minute", + "intervals" : ["2018-01-01/2018-01-03"], + "rollup" : true + } + }, + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/tutorial", + "filter" : "rollup-data.json" + }, + "inputFormat" : { + "type" : "json" + }, + "appendToExisting" : false + }, + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000, + "maxRowsInMemory" : 25000 + } + } +} +``` + +Roll-up has been enabled by setting `"rollup" : true` in the `granularitySpec`. + +Note that we have `srcIP` and `dstIP` defined as dimensions, a longSum metric is defined for the `packets` and `bytes` columns, and the `queryGranularity` has been defined as `minute`. + +We will see how these definitions are used after we load this data. + +## Load the example data + +From the apache-druid-{{DRUIDVERSION}} package root, run the following command: + +```bash +bin/post-index-task --file quickstart/tutorial/rollup-index.json --url http://localhost:8081 +``` + +After the script completes, we will query the data. + +## Query the example data + +Let's run `bin/dsql` and issue a `select * from "rollup-tutorial";` query to see what data was ingested. + +```bash +$ bin/dsql +Welcome to dsql, the command-line client for Druid SQL. +Type "\h" for help. +dsql> select * from "rollup-tutorial"; +┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐ +│ __time │ bytes │ count │ dstIP │ packets │ srcIP │ +├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤ +│ 2018-01-01T01:01:00.000Z │ 35937 │ 3 │ 2.2.2.2 │ 286 │ 1.1.1.1 │ +│ 2018-01-01T01:02:00.000Z │ 366260 │ 2 │ 2.2.2.2 │ 415 │ 1.1.1.1 │ +│ 2018-01-01T01:03:00.000Z │ 10204 │ 1 │ 2.2.2.2 │ 49 │ 1.1.1.1 │ +│ 2018-01-02T21:33:00.000Z │ 100288 │ 2 │ 8.8.8.8 │ 161 │ 7.7.7.7 │ +│ 2018-01-02T21:35:00.000Z │ 2818 │ 1 │ 8.8.8.8 │ 12 │ 7.7.7.7 │ +└──────────────────────────┴────────┴───────┴─────────┴─────────┴─────────┘ +Retrieved 5 rows in 1.18s. + +dsql> +``` + +Let's look at the three events in the original input data that occurred during `2018-01-01T01:01`: + +```json +{"timestamp":"2018-01-01T01:01:35Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":20,"bytes":9024} +{"timestamp":"2018-01-01T01:01:51Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":255,"bytes":21133} +{"timestamp":"2018-01-01T01:01:59Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":11,"bytes":5780} +``` + +These three rows have been "rolled up" into the following row: + +```bash +┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐ +│ __time │ bytes │ count │ dstIP │ packets │ srcIP │ +├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤ +│ 2018-01-01T01:01:00.000Z │ 35937 │ 3 │ 2.2.2.2 │ 286 │ 1.1.1.1 │ +└──────────────────────────┴────────┴───────┴─────────┴─────────┴─────────┘ +``` + +The input rows have been grouped by the timestamp and dimension columns `{timestamp, srcIP, dstIP}` with sum aggregations on the metric columns `packets` and `bytes`. + +Before the grouping occurs, the timestamps of the original input data are bucketed/floored by minute, due to the `"queryGranularity":"minute"` setting in the ingestion spec. + +Likewise, these two events that occurred during `2018-01-01T01:02` have been rolled up: + +```json +{"timestamp":"2018-01-01T01:02:14Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":38,"bytes":6289} +{"timestamp":"2018-01-01T01:02:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":377,"bytes":359971} +``` + +```bash +┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐ +│ __time │ bytes │ count │ dstIP │ packets │ srcIP │ +├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤ +│ 2018-01-01T01:02:00.000Z │ 366260 │ 2 │ 2.2.2.2 │ 415 │ 1.1.1.1 │ +└──────────────────────────┴────────┴───────┴─────────┴─────────┴─────────┘ +``` + +For the last event recording traffic between 1.1.1.1 and 2.2.2.2, no roll-up took place, because this was the only event that occurred during `2018-01-01T01:03`: + +```json +{"timestamp":"2018-01-01T01:03:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":49,"bytes":10204} +``` + +```bash +┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐ +│ __time │ bytes │ count │ dstIP │ packets │ srcIP │ +├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤ +│ 2018-01-01T01:03:00.000Z │ 10204 │ 1 │ 2.2.2.2 │ 49 │ 1.1.1.1 │ +└──────────────────────────┴────────┴───────┴─────────┴─────────┴─────────┘ +``` + +Note that the `count` metric shows how many rows in the original input data contributed to the final "rolled up" row. diff --git a/tutorials/tutorial-transform-spec.md b/tutorials/tutorial-transform-spec.md new file mode 100644 index 0000000..35695de --- /dev/null +++ b/tutorials/tutorial-transform-spec.md @@ -0,0 +1,154 @@ +--- +id: tutorial-transform-spec +title: "Tutorial: Transforming input data" +sidebar_label: "Transforming input data" +--- + + + + +This tutorial will demonstrate how to use transform specs to filter and transform input data during ingestion. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md) and [Tutorial: Querying data](../tutorials/tutorial-query.md). + +## Sample data + +We've included sample data for this tutorial at `quickstart/tutorial/transform-data.json`, reproduced here for convenience: + +```json +{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100} +{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200} +{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300} +{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300} +``` + +## Load data with transform specs + +We will ingest the sample data using the following spec, which demonstrates the use of transform specs: + +```json +{ + "type" : "index_parallel", + "spec" : { + "dataSchema" : { + "dataSource" : "transform-tutorial", + "timestampSpec": { + "column": "timestamp", + "format": "iso" + }, + "dimensionsSpec" : { + "dimensions" : [ + "animal", + { "name": "location", "type": "long" } + ] + }, + "metricsSpec" : [ + { "type" : "count", "name" : "count" }, + { "type" : "longSum", "name" : "number", "fieldName" : "number" }, + { "type" : "longSum", "name" : "triple-number", "fieldName" : "triple-number" } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "week", + "queryGranularity" : "minute", + "intervals" : ["2018-01-01/2018-01-03"], + "rollup" : true + }, + "transformSpec": { + "transforms": [ + { + "type": "expression", + "name": "animal", + "expression": "concat('super-', animal)" + }, + { + "type": "expression", + "name": "triple-number", + "expression": "number * 3" + } + ], + "filter": { + "type":"or", + "fields": [ + { "type": "selector", "dimension": "animal", "value": "super-mongoose" }, + { "type": "selector", "dimension": "triple-number", "value": "300" }, + { "type": "selector", "dimension": "location", "value": "3" } + ] + } + } + }, + "ioConfig" : { + "type" : "index_parallel", + "inputSource" : { + "type" : "local", + "baseDir" : "quickstart/tutorial", + "filter" : "transform-data.json" + }, + "inputFormat" : { + "type" :"json" + }, + "appendToExisting" : false + }, + "tuningConfig" : { + "type" : "index_parallel", + "maxRowsPerSegment" : 5000000, + "maxRowsInMemory" : 25000 + } + } +} +``` + +In the transform spec, we have two expression transforms: +* `super-animal`: prepends "super-" to the values in the `animal` column. This will override the `animal` column with the transformed version, since the transform's name is `animal`. +* `triple-number`: multiplies the `number` column by 3. This will create a new `triple-number` column. Note that we are ingesting both the original and the transformed column. + +Additionally, we have an OR filter with three clauses: +* `super-animal` values that match "super-mongoose" +* `triple-number` values that match 300 +* `location` values that match 3 + +This filter selects the first 3 rows, and it will exclude the final "lion" row in the input data. Note that the filter is applied after the transformation. + +Let's submit this task now, which has been included at `quickstart/tutorial/transform-index.json`: + +```bash +bin/post-index-task --file quickstart/tutorial/transform-index.json --url http://localhost:8081 +``` + +## Query the transformed data + +Let's run `bin/dsql` and issue a `select * from "transform-tutorial";` query to see what was ingested: + +```bash +dsql> select * from "transform-tutorial"; +┌──────────────────────────┬────────────────┬───────┬──────────┬────────┬───────────────┐ +│ __time │ animal │ count │ location │ number │ triple-number │ +├──────────────────────────┼────────────────┼───────┼──────────┼────────┼───────────────┤ +│ 2018-01-01T05:01:00.000Z │ super-mongoose │ 1 │ 2 │ 200 │ 600 │ +│ 2018-01-01T06:01:00.000Z │ super-snake │ 1 │ 3 │ 300 │ 900 │ +│ 2018-01-01T07:01:00.000Z │ super-octopus │ 1 │ 1 │ 100 │ 300 │ +└──────────────────────────┴────────────────┴───────┴──────────┴────────┴───────────────┘ +Retrieved 3 rows in 0.03s. +``` + +The "lion" row has been discarded, the `animal` column has been transformed, and we have both the original and transformed `number` column. diff --git a/tutorials/tutorial-update-data.md b/tutorials/tutorial-update-data.md new file mode 100644 index 0000000..8043850 --- /dev/null +++ b/tutorials/tutorial-update-data.md @@ -0,0 +1,169 @@ +--- +id: tutorial-update-data +title: "Tutorial: Updating existing data" +sidebar_label: "Updating existing data" +--- + + + + +This tutorial demonstrates how to update existing data, showing both overwrites and appends. + +For this tutorial, we'll assume you've already downloaded Apache Druid as described in +the [single-machine quickstart](index.html) and have it running on your local machine. + +It will also be helpful to have finished [Tutorial: Loading a file](../tutorials/tutorial-batch.md), [Tutorial: Querying data](../tutorials/tutorial-query.md), and [Tutorial: Rollup](../tutorials/tutorial-rollup.md). + +## Overwrite + +This section of the tutorial will cover how to overwrite an existing interval of data. + +### Load initial data + +Let's load an initial data set which we will overwrite and append to. + +The spec we'll use for this tutorial is located at `quickstart/tutorial/updates-init-index.json`. This spec creates a datasource called `updates-tutorial` from the `quickstart/tutorial/updates-data.json` input file. + +Let's submit that task: + +```bash +bin/post-index-task --file quickstart/tutorial/updates-init-index.json --url http://localhost:8081 +``` + +We have three initial rows containing an "animal" dimension and "number" metric: + +```bash +dsql> select * from "updates-tutorial"; +┌──────────────────────────┬──────────┬───────┬────────┐ +│ __time │ animal │ count │ number │ +├──────────────────────────┼──────────┼───────┼────────┤ +│ 2018-01-01T01:01:00.000Z │ tiger │ 1 │ 100 │ +│ 2018-01-01T03:01:00.000Z │ aardvark │ 1 │ 42 │ +│ 2018-01-01T03:01:00.000Z │ giraffe │ 1 │ 14124 │ +└──────────────────────────┴──────────┴───────┴────────┘ +Retrieved 3 rows in 1.42s. +``` + +### Overwrite the initial data + +To overwrite this data, we can submit another task for the same interval, but with different input data. + +The `quickstart/tutorial/updates-overwrite-index.json` spec will perform an overwrite on the `updates-tutorial` datasource. + +Note that this task reads input from `quickstart/tutorial/updates-data2.json`, and `appendToExisting` is set to `false` (indicating this is an overwrite). + +Let's submit that task: + +```bash +bin/post-index-task --file quickstart/tutorial/updates-overwrite-index.json --url http://localhost:8081 +``` + +When Druid finishes loading the new segment from this overwrite task, the "tiger" row now has the value "lion", the "aardvark" row has a different number, and the "giraffe" row has been replaced. It may take a couple of minutes for the changes to take effect: + +```bash +dsql> select * from "updates-tutorial"; +┌──────────────────────────┬──────────┬───────┬────────┐ +│ __time │ animal │ count │ number │ +├──────────────────────────┼──────────┼───────┼────────┤ +│ 2018-01-01T01:01:00.000Z │ lion │ 1 │ 100 │ +│ 2018-01-01T03:01:00.000Z │ aardvark │ 1 │ 9999 │ +│ 2018-01-01T04:01:00.000Z │ bear │ 1 │ 111 │ +└──────────────────────────┴──────────┴───────┴────────┘ +Retrieved 3 rows in 0.02s. +``` + +## Combine old data with new data and overwrite + +Let's try appending some new data to the `updates-tutorial` datasource now. We will add the data from `quickstart/tutorial/updates-data3.json`. + +The `quickstart/tutorial/updates-append-index.json` task spec has been configured to read from the existing `updates-tutorial` datasource and the `quickstart/tutorial/updates-data3.json` file. The task will combine data from the two input sources, and then overwrite the original data with the new combined data. + +Let's submit that task: + +```bash +bin/post-index-task --file quickstart/tutorial/updates-append-index.json --url http://localhost:8081 +``` + +When Druid finishes loading the new segment from this overwrite task, the new rows will have been added to the datasource. Note that roll-up occurred for the "lion" row: + +```bash +dsql> select * from "updates-tutorial"; +┌──────────────────────────┬──────────┬───────┬────────┐ +│ __time │ animal │ count │ number │ +├──────────────────────────┼──────────┼───────┼────────┤ +│ 2018-01-01T01:01:00.000Z │ lion │ 2 │ 400 │ +│ 2018-01-01T03:01:00.000Z │ aardvark │ 1 │ 9999 │ +│ 2018-01-01T04:01:00.000Z │ bear │ 1 │ 111 │ +│ 2018-01-01T05:01:00.000Z │ mongoose │ 1 │ 737 │ +│ 2018-01-01T06:01:00.000Z │ snake │ 1 │ 1234 │ +│ 2018-01-01T07:01:00.000Z │ octopus │ 1 │ 115 │ +└──────────────────────────┴──────────┴───────┴────────┘ +Retrieved 6 rows in 0.02s. +``` + +## Append to the data + +Let's try another way of appending data. + +The `quickstart/tutorial/updates-append-index2.json` task spec reads input from `quickstart/tutorial/updates-data4.json` and will append its data to the `updates-tutorial` datasource. Note that `appendToExisting` is set to `true` in this spec. + +Let's submit that task: + +```bash +bin/post-index-task --file quickstart/tutorial/updates-append-index2.json --url http://localhost:8081 +``` + +When the new data is loaded, we can see two additional rows after "octopus". Note that the new "bear" row with number 222 has not been rolled up with the existing bear-111 row, because the new data is held in a separate segment. + +```bash +dsql> select * from "updates-tutorial"; +┌──────────────────────────┬──────────┬───────┬────────┐ +│ __time │ animal │ count │ number │ +├──────────────────────────┼──────────┼───────┼────────┤ +│ 2018-01-01T01:01:00.000Z │ lion │ 2 │ 400 │ +│ 2018-01-01T03:01:00.000Z │ aardvark │ 1 │ 9999 │ +│ 2018-01-01T04:01:00.000Z │ bear │ 1 │ 111 │ +│ 2018-01-01T05:01:00.000Z │ mongoose │ 1 │ 737 │ +│ 2018-01-01T06:01:00.000Z │ snake │ 1 │ 1234 │ +│ 2018-01-01T07:01:00.000Z │ octopus │ 1 │ 115 │ +│ 2018-01-01T04:01:00.000Z │ bear │ 1 │ 222 │ +│ 2018-01-01T09:01:00.000Z │ falcon │ 1 │ 1241 │ +└──────────────────────────┴──────────┴───────┴────────┘ +Retrieved 8 rows in 0.02s. + +``` + +If we run a GroupBy query instead of a `select *`, we can see that the "bear" rows will group together at query time: + +```bash +dsql> select __time, animal, SUM("count"), SUM("number") from "updates-tutorial" group by __time, animal; +┌──────────────────────────┬──────────┬────────┬────────┐ +│ __time │ animal │ EXPR$2 │ EXPR$3 │ +├──────────────────────────┼──────────┼────────┼────────┤ +│ 2018-01-01T01:01:00.000Z │ lion │ 2 │ 400 │ +│ 2018-01-01T03:01:00.000Z │ aardvark │ 1 │ 9999 │ +│ 2018-01-01T04:01:00.000Z │ bear │ 2 │ 333 │ +│ 2018-01-01T05:01:00.000Z │ mongoose │ 1 │ 737 │ +│ 2018-01-01T06:01:00.000Z │ snake │ 1 │ 1234 │ +│ 2018-01-01T07:01:00.000Z │ octopus │ 1 │ 115 │ +│ 2018-01-01T09:01:00.000Z │ falcon │ 1 │ 1241 │ +└──────────────────────────┴──────────┴────────┴────────┘ +Retrieved 7 rows in 0.23s. +```