mirror of https://github.com/apache/druid.git
118b50195e
Today Kafka message support in streaming indexing tasks is limited to message values, and does not provide a way to expose Kafka headers, timestamps, or keys, which may be of interest to more specialized Druid input formats. For instance, Kafka headers may be used to indicate payload format/encoding or additional metadata, and timestamps are often omitted from values in Kafka streams applications, since they are included in the record. This change proposes to introduce KafkaRecordEntity as InputEntity, which would give input formats full access to the underlying Kafka record, including headers, key, timestamps. It would also open access to low-level information such as topic, partition, offset if needed. KafkaEntity is a subclass of ByteEntity for backwards compatibility with existing input formats, and to avoid introducing unnecessary complexity for Kinesis indexing tasks. |
||
|---|---|---|
| .. | ||
| avro-extensions | ||
| azure-extensions | ||
| datasketches | ||
| druid-aws-rds-extensions | ||
| druid-basic-security | ||
| druid-bloom-filter | ||
| druid-kerberos | ||
| druid-pac4j | ||
| druid-ranger-security | ||
| ec2-extensions | ||
| google-extensions | ||
| hdfs-storage | ||
| histogram | ||
| kafka-extraction-namespace | ||
| kafka-indexing-service | ||
| kinesis-indexing-service | ||
| kubernetes-extensions | ||
| lookups-cached-global | ||
| lookups-cached-single | ||
| mysql-metadata-storage | ||
| orc-extensions | ||
| parquet-extensions | ||
| postgresql-metadata-storage | ||
| protobuf-extensions | ||
| s3-extensions | ||
| simple-client-sslcontext | ||
| stats | ||