对 Kafka 数据载入 - 格式化内容,删除不需要的字段

This commit is contained in:
YuCheng Hu 2021-08-09 10:42:10 -04:00
parent 90a8927c37
commit 122bdda012
1 changed files with 0 additions and 83 deletions

View File

@ -393,89 +393,6 @@ Hadoop (see [here](https://github.com/apache/druid/pull/5102)).
## Apache Kafka 摄取数据
### 教程
本页包含基于Apache Kafka的摄取的参考文档。同样您可以查看 [Apache Kafka教程](../tutorials/chapter-2.md) 中的加载。
### 提交一个supervisor规范
Kafka索引服务需要同时在Overlord和MiddleManagers中加载 `druid-kafka-indexing-service` 扩展。 用于一个数据源的supervisor通过向 `http://<OVERLORD_IP>:<OVERLORD_PORT>/druid/indexer/v1/supervisor` 发送一个HTTP POST请求来启动例如
```json
curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8090/druid/indexer/v1/supervisor
```
一个示例supervisor规范如下
```json
{
"type": "kafka",
"dataSchema": {
"dataSource": "metrics-kafka",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [],
"dimensionExclusions": [
"timestamp",
"value"
]
},
"metricsSpec": [
{
"name": "count",
"type": "count"
},
{
"name": "value_sum",
"fieldName": "value",
"type": "doubleSum"
},
{
"name": "value_min",
"fieldName": "value",
"type": "doubleMin"
},
{
"name": "value_max",
"fieldName": "value",
"type": "doubleMax"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE"
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 5000000
},
"ioConfig": {
"topic": "metrics",
"inputFormat": {
"type": "json"
},
"consumerProperties": {
"bootstrap.servers": "localhost:9092"
},
"taskCount": 1,
"replicas": 1,
"taskDuration": "PT1H"
}
}
```
### supervisor配置
| 字段 | 描述 | 是否必须 |
|-|-|-|-|
| `type` | supervisor类型 总是 `kafka` | 是 |
| `dataSchema` | Kafka索引服务在摄取时使用的schema。详情见 [dataSchema](ingestion.md#dataschema) | 是 |
| `ioConfig` | 用于配置supervisor和索引任务的KafkaSupervisorIOConfig详情见以下 | 是 |
| `tuningConfig` | 用于配置supervisor和索引任务的KafkaSupervisorTuningConfig详情见以下 | 是 |
#### KafkaSupervisorTuningConfig