mirror of https://github.com/apache/druid.git
ad6609a606
### Description Today we ingest a number of high cardinality metrics into Druid across dimensions. These metrics are rolled up on a per minute basis, and are very useful when looking at metrics on a partition or client basis. Events is another class of data that provides useful information about a particular incident/scenario inside a Kafka cluster. Events themselves are carried inside kafka payload, but nonetheless there are some very useful metadata that is carried in kafka headers that can serve as useful dimension for aggregation and in turn bringing better insights. PR(https://github.com/apache/druid/pull/10730) introduced support of Kafka headers in InputFormats. We still need an input format to parse out the headers and translate those into relevant columns in Druid. Until that’s implemented, none of the information available in the Kafka message headers would be exposed. So first there is a need to write an input format that can parse headers in any given format(provided we support the format) like we parse payloads today. Apart from headers there is also some useful information present in the key portion of the kafka record. We also need a way to expose the data present in the key as druid columns. We need a generic way to express at configuration time what attributes from headers, key and payload need to be ingested into druid. We need to keep the design generic enough so that users can specify different parsers for headers, key and payload. This PR is designed to solve the above by providing wrapper around any existing input formats and merging the data into a single unified Druid row. Lets look at a sample input format from the above discussion "inputFormat": { "type": "kafka", // New input format type "headerLabelPrefix": "kafka.header.", // Label prefix for header columns, this will avoid collusions while merging columns "recordTimestampLabelPrefix": "kafka.", // Kafka record's timestamp is made available in case payload does not carry timestamp "headerFormat": // Header parser specifying that values are of type string { "type": "string" }, "valueFormat": // Value parser from json parsing { "type": "json", "flattenSpec": { "useFieldDiscovery": true, "fields": [...] } }, "keyFormat": // Key parser also from json parsing { "type": "json" } } Since we have independent sections for header, key and payload, it will enable parsing each section with its own parser, eg., headers coming in as string and payload as json. KafkaInputFormat will be the uber class extending inputFormat interface and will be responsible for creating individual parsers for header, key and payload, blend the data resolving conflicts in columns and generating a single unified InputRow for Druid ingestion. "headerFormat" will allow users to plug parser type for the header values and will add default header prefix as "kafka.header."(can be overridden) for attributes to avoid collision while merging attributes with payload. Kafka payload parser will be responsible for parsing the Value portion of the Kafka record. This is where most of the data will come from and we should be able to plugin existing parser. One thing to note here is that if batching is performed, then the code is augmenting header and key values to every record in the batch. Kafka key parser will handle parsing Key portion of the Kafka record and will ingest the Key with dimension name as "kafka.key". ## KafkaInputFormat Class: This is the class that orchestrates sending the consumerRecord to each parser, retrieve rows, merge the columns into one final row for Druid consumption. KafkaInputformat should make sure to release the resources that gets allocated as a part of reader in CloseableIterator<InputRow> during normal and exception cases. During conflicts in dimension/metrics names, the code will prefer dimension names from payload and ignore the dimension either from headers/key. This is done so that existing input formats can be easily migrated to this new format without worrying about losing information. |
||
|---|---|---|
| .. | ||
| core | ||
| i18n | ||
| pages/en | ||
| script | ||
| scss | ||
| static | ||
| .gitignore | ||
| .spelling | ||
| README.md | ||
| package-lock.json | ||
| package.json | ||
| pom.xml | ||
| redirects.json | ||
| sidebars.json | ||
| siteConfig.js | ||
README.md
Druid doc builder
This website was created with Docusaurus.
To view documentation run:
npm install
Then run:
npm start
The current version of the web site appears in your browser. Edit pages with your favorite editor. Refresh the web page after each edit to review your changes.
Dependencies
- NodeJS. Use the version Docusaurus specifies, not a newer one. (For example, if 12.x is requested, don't install 16.x.) Docusaurus may require a version newer than that available in your Linux package repository, but older than the latest version. See this page to find the version required by Docusaurus.
- The Yarn dependency from Docusaurus is optional. (This Yarn is not the Hadoop resource manager, it is a package manager for Node.js).
- Docusaurus. Installed automatically
as part of the the above
npmcommands.
Variables
Documentation pages can refer to a number of special variables using the
{{var}} syntax:
DRUIDVERSION- the version of Druid in which the page appears. Allows creating links to files of the same version on GitHub.
The variables are not replaced when running the web site locally using the
start command above.
Spellcheck
Please run a spellcheck before issuing a pull request to avoid a build failure due to spelling issues. Run:
npm run link-lint
npm run spellcheck
If you introduce new (correctly spelled) project names or technical terms, add
them to the dictionary in the .spelling file in this directory. Also, terms
enclosed in backticks are not spell checked. Example: `symbolName`