对 Kafka 数据载入 - 格式化内容,删除不需要的字段
This commit is contained in:
parent
90a8927c37
commit
122bdda012
|
@ -393,89 +393,6 @@ Hadoop (see [here](https://github.com/apache/druid/pull/5102)).
|
|||
## Apache Kafka 摄取数据
|
||||
|
||||
|
||||
### 教程
|
||||
|
||||
本页包含基于Apache Kafka的摄取的参考文档。同样,您可以查看 [Apache Kafka教程](../tutorials/chapter-2.md) 中的加载。
|
||||
|
||||
### 提交一个supervisor规范
|
||||
|
||||
Kafka索引服务需要同时在Overlord和MiddleManagers中加载 `druid-kafka-indexing-service` 扩展。 用于一个数据源的supervisor通过向 `http://<OVERLORD_IP>:<OVERLORD_PORT>/druid/indexer/v1/supervisor` 发送一个HTTP POST请求来启动,例如:
|
||||
|
||||
```json
|
||||
curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8090/druid/indexer/v1/supervisor
|
||||
```
|
||||
|
||||
一个示例supervisor规范如下:
|
||||
```json
|
||||
{
|
||||
"type": "kafka",
|
||||
"dataSchema": {
|
||||
"dataSource": "metrics-kafka",
|
||||
"timestampSpec": {
|
||||
"column": "timestamp",
|
||||
"format": "auto"
|
||||
},
|
||||
"dimensionsSpec": {
|
||||
"dimensions": [],
|
||||
"dimensionExclusions": [
|
||||
"timestamp",
|
||||
"value"
|
||||
]
|
||||
},
|
||||
"metricsSpec": [
|
||||
{
|
||||
"name": "count",
|
||||
"type": "count"
|
||||
},
|
||||
{
|
||||
"name": "value_sum",
|
||||
"fieldName": "value",
|
||||
"type": "doubleSum"
|
||||
},
|
||||
{
|
||||
"name": "value_min",
|
||||
"fieldName": "value",
|
||||
"type": "doubleMin"
|
||||
},
|
||||
{
|
||||
"name": "value_max",
|
||||
"fieldName": "value",
|
||||
"type": "doubleMax"
|
||||
}
|
||||
],
|
||||
"granularitySpec": {
|
||||
"type": "uniform",
|
||||
"segmentGranularity": "HOUR",
|
||||
"queryGranularity": "NONE"
|
||||
}
|
||||
},
|
||||
"tuningConfig": {
|
||||
"type": "kafka",
|
||||
"maxRowsPerSegment": 5000000
|
||||
},
|
||||
"ioConfig": {
|
||||
"topic": "metrics",
|
||||
"inputFormat": {
|
||||
"type": "json"
|
||||
},
|
||||
"consumerProperties": {
|
||||
"bootstrap.servers": "localhost:9092"
|
||||
},
|
||||
"taskCount": 1,
|
||||
"replicas": 1,
|
||||
"taskDuration": "PT1H"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### supervisor配置
|
||||
|
||||
| 字段 | 描述 | 是否必须 |
|
||||
|-|-|-|-|
|
||||
| `type` | supervisor类型, 总是 `kafka` | 是 |
|
||||
| `dataSchema` | Kafka索引服务在摄取时使用的schema。详情见 [dataSchema](ingestion.md#dataschema) | 是 |
|
||||
| `ioConfig` | 用于配置supervisor和索引任务的KafkaSupervisorIOConfig,详情见以下 | 是 |
|
||||
| `tuningConfig` | 用于配置supervisor和索引任务的KafkaSupervisorTuningConfig,详情见以下 | 是 |
|
||||
|
||||
#### KafkaSupervisorTuningConfig
|
||||
|
||||
|
|
Loading…
Reference in New Issue