4.9 KiB
| layout |
|---|
| doc_page |
Indexing Service
For Indexing Service Configuration, see Indexing Service Configuration.
The indexing service is a highly-available, distributed service that runs indexing related tasks. Indexing service tasks create (and sometimes destroy) Druid segments. The indexing service has a master/slave like architecture.
The indexing service is composed of three main components: a peon component that can run a single task, a Middle Manager component that manages peons, and an overlord component that manages task distribution to middle managers. Overlords and middle managers may run on the same node or across multiple nodes while middle managers and Peons always run on the same node.
Indexing Service Overview
Overlord Node
The overlord node is responsible for accepting tasks, coordinating task distribution, creating locks around tasks, and returning statuses to callers. Overlord can be configured to run in one of two modes - local or remote (local being default). In local mode overlord is also responsible for creating peons for executing tasks. When running the overlord in local mode, all middle manager and peon configurations must be provided as well. Local mode is typically used for simple workflows. In remote mode, the overlord and middle manager are run in separate processes and you can run each on a different server. This mode is recommended if you intend to use the indexing service as the single endpoint for all Druid indexing.
Submitting Tasks and Querying Task Status
Tasks are submitted to the overlord node in the form of JSON objects. Tasks can be submitted via POST requests to:
http://<OVERLORD_IP>:<port>/druid/indexer/v1/task
this will return the taskId of the submitted task.
Tasks can be shut down via POST requests to:
http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/shutdown
Task statuses can be retrieved via GET requests to:
http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/status
Task segments can be retrieved via GET requests to:
http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/segments
Overlord Console
The overlord console can be used to view pending tasks, running tasks, available workers, and recent worker creation and termination. The console can be accessed at:
http://<OVERLORD_IP>:<port>/console.html
Autoscaling
The Autoscaling mechanisms currently in place are tightly coupled with our deployment infrastructure but the framework should be in place for other implementations. We are highly open to new implementations or extensions of the existing mechanisms. In our own deployments, middle manager nodes are Amazon AWS EC2 nodes and they are provisioned to register themselves in a galaxy environment.
If autoscaling is enabled, new middle managers may be added when a task has been in pending state for too long. Middle managers may be terminated if they have not run any tasks for a period of time.
Middle Managers
See Middle Manager.
Peons
See Peon.
Tasks
See Tasks.
HTTP Endpoints
GET
/status
Returns the Druid version, loaded extensions, memory used, total memory and other useful information about the node.