Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
druid/docs/api-reference/dynamic-configuration-api.md

15 KiB
Raw Blame History

id title sidebar_label
dynamic-configuration-api Dynamic configuration API Dynamic configuration

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

This document describes the API endpoints to retrieve and manage dynamic configurations for the Coordinator and Overlord in Apache Druid.

In this topic, http://ROUTER_IP:ROUTER_PORT is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use http://localhost:8888 for quickstart deployments.

Coordinator dynamic configuration

The Coordinator has dynamic configurations to tune certain behavior on the fly, without requiring a service restart. For information on the supported properties, see Coordinator dynamic configuration.

Get dynamic configuration

Retrieves the current Coordinator dynamic configuration. Returns a JSON object with the dynamic configuration properties.

URL

GET /druid/coordinator/v1/config

Responses

Successfully retrieved dynamic configuration

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config"

GET /druid/coordinator/v1/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

View the response 
{
    "millisToWaitBeforeDeleting": 900000,
    "mergeBytesLimit": 524288000,
    "mergeSegmentsLimit": 100,
    "maxSegmentsToMove": 100,
    "replicantLifetime": 15,
    "replicationThrottleLimit": 500,
    "balancerComputeThreads": 1,
    "killDataSourceWhitelist": [],
    "killPendingSegmentsSkipList": [],
    "maxSegmentsInNodeLoadingQueue": 500,
    "decommissioningNodes": [],
    "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
    "pauseCoordination": false,
    "replicateAfterLoadTimeout": false,
    "maxNonPrimaryReplicantsToLoad": 2147483647,
    "useRoundRobinSegmentAssignment": true,
    "smartSegmentLoading": true,
    "debugDimensions": null
}

Update dynamic configuration

Submits a JSON-based dynamic configuration spec to the Coordinator. For information on the supported properties, see Dynamic configuration.

URL

POST /druid/coordinator/v1/config

Header parameters

The endpoint supports a set of optional header parameters to populate the author and comment fields in the configuration history.

  • X-Druid-Author
    • Type: String
    • Author of the configuration change.
  • X-Druid-Comment
    • Type: String
    • Description for the update.

Responses

Successfully updated dynamic configuration

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" \
--header 'Content-Type: application/json' \
--data '{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 5,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": true,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 10,
  "balancerComputeThreads": 1,
  "emitBalancingStats": true,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 100,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647,
  "useRoundRobinSegmentAssignment": true
}'

POST /druid/coordinator/v1/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 683

{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 5,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": true,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 10,
  "balancerComputeThreads": 1,
  "emitBalancingStats": true,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 100,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647,
  "useRoundRobinSegmentAssignment": true
}

Sample response

A successful request returns an HTTP 200 OK message code and an empty response body.

Get dynamic configuration history

Retrieves the history of changes to Coordinator dynamic configuration over an interval of time. Returns an empty array if there are no history records available.

URL

GET /druid/coordinator/v1/config/history

Query parameters

The endpoint supports a set of optional query parameters to filter results.

  • interval

    • Type: String
    • Limit the results to the specified time interval in ISO 8601 format delimited with /. For example, 2023-07-13/2023-07-19. The default interval is one week. You can change this period by setting druid.audit.manager.auditHistoryMillis in the runtime.properties file for the Coordinator.

  • count

    • Type: Integer
    • Limit the number of results to the last n entries.

Responses

Successfully retrieved history

Sample request

The following example retrieves the dynamic configuration history between 2022-07-13 and 2024-07-19. The response is limited to 10 entries.

curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/history?interval=2022-07-13%2F2024-07-19&count=10"

GET /druid/coordinator/v1/config/history?interval=2022-07-13/2024-07-19&count=10 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

View the response 
[
    {
        "key": "coordinator.config",
        "type": "coordinator.config",
        "auditInfo": {
            "author": "",
            "comment": "",
            "ip": "127.0.0.1"
        },
        "payload": "{\"millisToWaitBeforeDeleting\":900000,\"mergeBytesLimit\":524288000,\"mergeSegmentsLimit\":100,\"maxSegmentsToMove\":5,\"replicantLifetime\":15,\"replicationThrottleLimit\":10,\"balancerComputeThreads\":1,\"killDataSourceWhitelist\":[],\"killPendingSegmentsSkipList\":[],\"maxSegmentsInNodeLoadingQueue\":100,\"decommissioningNodes\":[],\"decommissioningMaxPercentOfMaxSegmentsToMove\":70,\"pauseCoordination\":false,\"replicateAfterLoadTimeout\":false,\"maxNonPrimaryReplicantsToLoad\":2147483647,\"useRoundRobinSegmentAssignment\":true,\"smartSegmentLoading\":true,\"debugDimensions\":null}",
        "auditTime": "2023-10-03T20:59:51.622Z"
    }
]

Overlord dynamic configuration

The Overlord has dynamic configurations to tune how Druid assigns tasks to workers. For information on the supported properties, see Overlord dynamic configuration.

Get dynamic configuration

Retrieves the current Overlord dynamic configuration. Returns a JSON object with the dynamic configuration properties. Returns an empty response body if there is no current Overlord dynamic configuration.

URL

GET /druid/indexer/v1/worker

Responses

Successfully retrieved dynamic configuration

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker"

GET /druid/indexer/v1/worker HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

View the response 
{
    "type": "default",
    "selectStrategy": {
        "type": "fillCapacityWithCategorySpec",
        "workerCategorySpec": {
            "categoryMap": {},
            "strong": true
        }
    },
    "autoScaler": null
}

Update dynamic configuration

Submits a JSON-based dynamic configuration spec to the Overlord. For information on the supported properties, see Overlord dynamic configuration.

URL

POST /druid/indexer/v1/worker

Header parameters

The endpoint supports a set of optional header parameters to populate the author and comment fields in the configuration history.

  • X-Druid-Author
    • Type: String
    • Author of the configuration change.
  • X-Druid-Comment
    • Type: String
    • Description for the update.

Responses

Successfully updated dynamic configuration

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" \
--header 'Content-Type: application/json' \
--data '{
  "type": "default",
  "selectStrategy": {
    "type": "fillCapacityWithCategorySpec",
    "workerCategorySpec": {
      "categoryMap": {},
      "strong": true
    }
  },
  "autoScaler": null
}'

POST /druid/indexer/v1/worker HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 196

{
  "type": "default",
  "selectStrategy": {
    "type": "fillCapacityWithCategorySpec",
    "workerCategorySpec": {
      "categoryMap": {},
      "strong": true
    }
  },
  "autoScaler": null
}

Sample response

A successful request returns an HTTP 200 OK message code and an empty response body.

Get dynamic configuration history

Retrieves the history of changes to Overlord dynamic configuration over an interval of time. Returns an empty array if there are no history records available.

URL

GET /druid/indexer/v1/worker/history

Query parameters

The endpoint supports a set of optional query parameters to filter results.

  • interval

    • Type: String
    • Limit the results to the specified time interval in ISO 8601 format delimited with /. For example, 2023-07-13/2023-07-19. The default interval is one week. You can change this period by setting druid.audit.manager.auditHistoryMillis in the runtime.properties file for the Overlord.

  • count

    • Type: Integer
    • Limit the number of results to the last n entries.

Responses

Successfully retrieved history

Sample request

The following example retrieves the dynamic configuration history between 2022-07-13 and 2024-07-19. The response is limited to 10 entries.

curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10"

GET /druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

View the response 
[
    {
        "key": "worker.config",
        "type": "worker.config",
        "auditInfo": {
            "author": "",
            "comment": "",
            "ip": "127.0.0.1"
        },
        "payload": "{\"type\":\"default\",\"selectStrategy\":{\"type\":\"fillCapacityWithCategorySpec\",\"workerCategorySpec\":{\"categoryMap\":{},\"strong\":true}},\"autoScaler\":null}",
        "auditTime": "2023-10-03T21:49:49.991Z"
    }
]

Get an array of worker nodes in the cluster

Returns an array of all the worker nodes in the cluster along with its corresponding metadata.

GET /druid/indexer/v1/workers

Responses

Successfully retrieved worker nodes

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/workers"

GET /druid/indexer/v1/workers HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

View the response 
[
    {
        "worker": {
            "scheme": "http",
            "host": "localhost:8091",
            "ip": "198.51.100.0",
            "capacity": 2,
            "version": "0",
            "category": "_default_worker_category"
        },
        "currCapacityUsed": 0,
        "currParallelIndexCapacityUsed": 0,
        "availabilityGroups": [],
        "runningTasks": [],
        "lastCompletedTaskTime": "2023-09-29T19:13:05.505Z",
        "blacklistedUntil": null
    }
]

Get scaling events

Returns Overlord scaling events if autoscaling runners are in use. Returns an empty response body if there are no Overlord scaling events.

URL

GET /druid/indexer/v1/scaling

Responses

Successfully retrieved scaling events

Sample request

curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/scaling"

GET /druid/indexer/v1/scaling HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

A successful request returns a 200 OK response and an array of scaling events.