0f29ece6a9
Merging the work so far. @ektravel , @vogievetsky if there are additional improvements, let's track them & make another pr. * Refactor streaming ingestion docs * Update property definition * Update after review * Update known issues * Move kinesis and kafka topics to ingestion, add redirects * Saving changes * Saving * Add input format text * Update after review * Minor text edit * Update example syntax * Revert back to colon * Fix merge conflicts * Fix broken links * Fix spelling error
1.9 KiB
| id | title |
|---|---|
| streaming | Streaming ingestion |
Apache Druid can consume data streams from the following external streaming sources:
- Apache Kafka through the bundled Kafka indexing service extension.
- Amazon Kinesis through the bundled Kinesis indexing service extension.
Each indexing service provides real-time data ingestion with exactly-once stream processing guarantee. To use either of the streaming ingestion methods, you must first load the associated extension on both the Overlord and the MiddleManager. See Loading extensions for more information.
Streaming ingestion is controlled by a continuously running supervisor. The supervisor oversees the state of indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained. You start a supervisor by submitting a JSON specification, often referred to as the supervisor spec, either though the Druid web console or using the Supervisor API.