mirror of https://github.com/apache/druid.git
66 lines
4.9 KiB
Markdown
66 lines
4.9 KiB
Markdown
---
|
|
id: kafka-emitter
|
|
title: "Kafka Emitter"
|
|
---
|
|
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
|
|
To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `kafka-emitter` in the extensions load list.
|
|
|
|
## Introduction
|
|
|
|
This extension emits Druid metrics to [Apache Kafka](https://kafka.apache.org) directly with JSON format.<br />
|
|
Currently, Kafka has not only their nice ecosystem but also consumer API readily available.
|
|
So, If you currently use Kafka, It's easy to integrate various tool or UI
|
|
to monitor the status of your Druid cluster with this extension.
|
|
|
|
## Configuration
|
|
|
|
All the configuration parameters for the Kafka emitter are under `druid.emitter.kafka`.
|
|
|
|
| Property | Description | Required | Default |
|
|
|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------|
|
|
| `druid.emitter.kafka.bootstrap.servers` | Comma-separated Kafka broker. (`[hostname:port],[hostname:port]...`) | yes | none |
|
|
| `druid.emitter.kafka.event.types` | Comma-separated event types. <br/>Supported types are `alerts`, `metrics`, `requests`, and `segment_metadata`. | no | `["metrics", "alerts"]` |
|
|
| `druid.emitter.kafka.metric.topic` | Kafka topic name for emitter's target to emit service metrics. If `event.types` contains `metrics`, this field cannot be empty. | no | none |
|
|
| `druid.emitter.kafka.alert.topic` | Kafka topic name for emitter's target to emit alerts. If `event.types` contains `alerts`, this field cannot empty. | no | none |
|
|
| `druid.emitter.kafka.request.topic` | Kafka topic name for emitter's target to emit request logs. If `event.types` contains `requests`, this field cannot be empty. | no | none |
|
|
| `druid.emitter.kafka.segmentMetadata.topic` | Kafka topic name for emitter's target to emit segment metadata. If `event.types` contains `segment_metadata`, this field cannot be empty. | no | none |
|
|
| `druid.emitter.kafka.producer.config` | JSON configuration to set additional properties to Kafka producer. | no | none |
|
|
| `druid.emitter.kafka.clusterName` | Optional value to specify the name of your Druid cluster. It can help make groups in your monitoring environment. | no | none |
|
|
| `druid.emitter.kafka.extra.dimensions` | Optional JSON configuration to specify a map of extra string dimensions for the events emitted. These can help make groups in your monitoring environment. | no | none |
|
|
| `druid.emitter.kafka.producer.hiddenProperties` | JSON configuration to specify sensitive Kafka producer properties such as username and password. This property accepts a [DynamicConfigProvider](../../operations/dynamic-config-provider.md) implementation. | no | none |
|
|
|
|
### Example
|
|
|
|
```
|
|
druid.emitter.kafka.bootstrap.servers=hostname1:9092,hostname2:9092
|
|
druid.emitter.kafka.event.types=["metrics", alerts", "requests", "segment_metadata"]
|
|
druid.emitter.kafka.metric.topic=druid-metric
|
|
druid.emitter.kafka.alert.topic=druid-alert
|
|
druid.emitter.kafka.request.topic=druid-request-logs
|
|
druid.emitter.kafka.segmentMetadata.topic=druid-segment-metadata
|
|
druid.emitter.kafka.producer.config={"max.block.ms":10000}
|
|
druid.emitter.kafka.extra.dimensions={"region":"us-east-1","environment":"preProd"}
|
|
druid.emitter.kafka.producer.hiddenProperties={"config":{"sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\\"KV...NI\\" password=\\"gA3...n6a/\\";"}}
|
|
```
|
|
|