druid/docs/content/ingestion/tasks.md

layout
doc_page

Tasks Overview

Tasks are run on middle managers and always operate on a single data source.

Tasks are submitted using POST requests to the Overlord. Please see Overlord Task API for API details.

There are several different types of tasks.

Segment Creation Tasks

Hadoop Index Task

See batch ingestion.

Native Index Tasks

Druid provides a native index task which doesn't need any dependencies on other systems. See native index tasks for more details.

Kafka Indexing Tasks

Kafka Indexing tasks are automatically created by a Kafka Supervisor and are responsible for pulling data from Kafka streams. These tasks are not meant to be created/submitted directly by users. See Kafka Indexing Service for more details.

Stream Push Tasks (Tranquility)

Tranquility Server automatically creates "realtime" tasks that receive events over HTTP using an EventReceiverFirehose. These tasks are not meant to be created/submitted directly by users. See Tranquility Stream Push for more info.

Compaction Tasks

Compaction tasks merge all segments of the given interval. Please see Compaction for details.

Segment Merging Tasks

The documentation for the Append Task, Merge Task, and Same Interval Merge Task has been moved to Miscellaneous Tasks.

Kill Task

Kill tasks delete all information about a segment and removes it from deep storage.

Please see Deleting Data for details.

Misc. Tasks

Please see Miscellaneous Tasks.

Task Locking and Priority

Please see Task Locking and Priority.