Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Retention rules API documentation refactor (#14623)

This commit is contained in:
Nhi Pham 2023-08-15 13:44:44 -07:00 committed by GitHub
parent aeeeed3b35
commit a38579ab3c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 477 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

501
docs/api-reference/retention-rules-api.md
View File

@ -23,47 +23,500 @@ sidebar_label: Retention rules
~ under the License. ~ under the License.
--> -->
This document describes the API endpoints for managing retention rules in Apache Druid. This topic describes the API endpoints for managing retention rules in Apache Druid. You can configure retention rules in the Druid web console or API.
## Retention rules Druid uses retention rules to determine what data is retained in the cluster. Druid supports load, drop, and broadcast rules. For more information, see [Using rules to drop and retain data](../operations/rule-configuration.md).
Note that all _interval_ URL parameters are ISO 8601 strings delimited by a `_` instead of a `/` as in `2016-06-27_2016-06-28`. In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use `http://localhost:8888` for quickstart deployments.
`GET /druid/coordinator/v1/rules` ## Update retention rules for a datasource
Returns all rules as JSON objects for all datasources in the cluster including the default datasource. Updates one or more retention rules for a datasource. The request body takes an array of retention rule objects. For details on defining retention rules, see the following sources:
`GET /druid/coordinator/v1/rules/{dataSourceName}` * [Load rules](../operations/rule-configuration.md#load-rules)
* [Drop rules](../operations/rule-configuration.md#drop-rules)
* [Broadcast rules](../operations/rule-configuration.md#broadcast-rules)
Returns all rules for a specified datasource. This request overwrites any existing rules for the datasource.
Druid reads rules in the order in which they appear; for more information, see [rule structure](../operations/rule-configuration.md).
`GET /druid/coordinator/v1/rules/{dataSourceName}?full` Note that this endpoint returns an HTTP `200 OK` even if the datasource does not exist.
Returns all rules for a specified datasource and includes default datasource. ### URL
`GET /druid/coordinator/v1/rules/history?interval=<interval>` <code class="postAPI">POST</code> <code>/druid/coordinator/v1/rules/:dataSource</code>
Returns audit history of rules for all datasources. Default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Coordinator `runtime.properties`. ### Header parameters
`GET /druid/coordinator/v1/rules/history?count=<n>` The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the `auditInfo` property for audit history.
Returns last `n` entries of audit history of rules for all datasources. * `X-Druid-Author` (optional)
* Type: String
* A string representing the author making the configuration change.
* `X-Druid-Comment` (optional)
* Type: String
* A string describing the update.
`GET /druid/coordinator/v1/rules/{dataSourceName}/history?interval=<interval>` ### Responses
Returns audit history of rules for a specified datasource. Default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Coordinator `runtime.properties`. <!--DOCUSAURUS_CODE_TABS-->
`GET /druid/coordinator/v1/rules/{dataSourceName}/history?count=<n>` <!--200 SUCCESS-->
Returns last `n` entries of audit history of rules for a specified datasource. *Successfully updated retention rules for specified datasource*
`POST /druid/coordinator/v1/rules/{dataSourceName}` <!--END_DOCUSAURUS_CODE_TABS-->
POST with a list of rules in JSON form to update rules. ---
Optional Header Parameters for auditing the config change can also be specified. ### Sample request
|Header Param Name| Description | Default | The following example sets a set of broadcast, load, and drop retention rules for the `kttm1` datasource.
|----------|-------------|---------|
|`X-Druid-Author`| Author making the config change|`""`| <!--DOCUSAURUS_CODE_TABS-->
|`X-Druid-Comment`| Comment describing the change being done|`""`|
<!--cURL-->
```shell
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/kttm1" \
--header 'X-Druid-Author: doc intern' \
--header 'X-Druid-Comment: submitted via api' \
--header 'Content-Type: application/json' \
--data '[
{
"type": "broadcastForever"
},
{
"type": "loadForever",
"tieredReplicants": {
"_default_tier": 2
},
"useDefaultTierForNull": true
},
{
"type": "dropByPeriod",
"period": "P1M"
}
]'
```
<!--HTTP-->
```HTTP
POST /druid/coordinator/v1/rules/kttm1 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
X-Druid-Author: doc intern
X-Druid-Comment: submitted via api
Content-Type: application/json
Content-Length: 273
[
{
"type": "broadcastForever"
},
{
"type": "loadForever",
"tieredReplicants": {
"_default_tier": 1
},
"useDefaultTierForNull": true
},
{
"type": "dropByPeriod",
"period": "P1M"
}
]
```
<!--END_DOCUSAURUS_CODE_TABS-->
### Sample response
A successful request returns an HTTP `200 OK` message code and an empty response body.
## Update default retention rules for all datasources
Updates one or more default retention rules for all datasources. Submit retention rules as an array of objects in the request body. For details on defining retention rules, see the following sources:
* [Load rules](../operations/rule-configuration.md#load-rules)
* [Drop rules](../operations/rule-configuration.md#drop-rules)
* [Broadcast rules](../operations/rule-configuration.md#broadcast-rules)
This request overwrites any existing rules for all datasources. To remove default retention rules for all datasources, submit an empty rule array in the request body. Rules are read in the order in which they appear; for more information, see [rule structure](../operations/rule-configuration.md).
### URL
<code class="postAPI">POST</code> <code>/druid/coordinator/v1/rules/_default</code>
### Header parameters
The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the `auditInfo` property for audit history.
* `X-Druid-Author` (optional)
* Type: String
* A string representing the author making the configuration change.
* `X-Druid-Comment` (optional)
* Type: String
* A string describing the update.
### Responses
<!--DOCUSAURUS_CODE_TABS-->
<!--200 SUCCESS-->
*Successfully updated default retention rules*
<!--500 SERVER ERROR-->
*Error with request body*
<!--END_DOCUSAURUS_CODE_TABS-->
---
### Sample request
The following example updates the default retention rule for all datasources with a `loadByInterval` rule.
<!--DOCUSAURUS_CODE_TABS-->
<!--cURL-->
```shell
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/_default" \
--header 'Content-Type: application/json' \
--data '[
{
"type": "loadByInterval",
"tieredReplicants": {},
"useDefaultTierForNull": false,
"interval": "2010-01-01/2020-01-01"
}
]'
```
<!--HTTP-->
```HTTP
POST /druid/coordinator/v1/rules/_default HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 205
[
{
"type": "loadByInterval",
"tieredReplicants": {},
"useDefaultTierForNull": false,
"interval": "2010-01-01/2020-01-01"
}
]
```
<!--END_DOCUSAURUS_CODE_TABS-->
### Sample response
A successful request returns an HTTP `200 OK` message code and an empty response body.
## Get an array of all retention rules
Retrieves all current retention rules in the cluster including the default retention rule. Returns an array of objects for each datasource and their associated retention rules.
### URL
<code class="getAPI">GET</code> <code>/druid/coordinator/v1/rules</code>
### Responses
<!--DOCUSAURUS_CODE_TABS-->
<!--200 SUCCESS-->
*Successfully retrieved retention rules*
<!--END_DOCUSAURUS_CODE_TABS-->
---
### Sample request
<!--DOCUSAURUS_CODE_TABS-->
<!--cURL-->
```shell
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules"
```
<!--HTTP-->
```HTTP
GET /druid/coordinator/v1/rules HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
```
<!--END_DOCUSAURUS_CODE_TABS-->
### Sample response
<details>
<summary>Click to show sample response</summary>
```json
{
"_default": [
{
"tieredReplicants": {
"_default_tier": 2
},
"type": "loadForever"
}
],
"social_media": [
{
"interval": "2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z",
"type": "dropByInterval"
}
],
"wikipedia_api": [],
}
```
</details>
## Get an array of retention rules for a datasource
Retrieves an array of rule objects for a single datasource. Returns an empty array if there are no retention rules.
Note that this endpoint returns an HTTP `200 OK` message code even if the datasource does not exist.
### URL
<code class="getAPI">GET</code> <code>/druid/coordinator/v1/rules/:dataSource</code>
### Query parameters
* `full` (optional)
* Includes the default retention rule for the datasource in the response.
### Responses
<!--DOCUSAURUS_CODE_TABS-->
<!--200 SUCCESS-->
*Successfully retrieved retention rules*
<!--END_DOCUSAURUS_CODE_TABS-->
---
### Sample request
The following example retrieves the custom retention rules and default retention rules for datasource with the name `social_media`.
<!--DOCUSAURUS_CODE_TABS-->
<!--cURL-->
```shell
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/social_media?full=null"
```
<!--HTTP-->
```HTTP
GET /druid/coordinator/v1/rules/social_media?full=null HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
```
<!--END_DOCUSAURUS_CODE_TABS-->
### Sample response
<details>
<summary>Click to show sample response</summary>
```json
[
{
"interval": "2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z",
"type": "dropByInterval"
},
{
"interval": "2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z",
"tieredReplicants": {
"_default_tier": 2
},
"type": "loadByInterval"
},
{
"tieredReplicants": {
"_default_tier": 2
},
"type": "loadForever"
}
]
```
</details>
## Get audit history for all datasources
Retrieves the audit history of rules for all datasources over an interval of time. The default interval is 1 week. You can change this period by setting `druid.audit.manager.auditHistoryMillis` in the `runtime.properties` file for the Coordinator.
### URL
<code class="getAPI">GET</code> <code>/druid/coordinator/v1/rules/history</code>
### Query parameters
Note that the following query parameters cannot be chained.
* `interval` (optional)
* Type: ISO 8601.
* Limits the number of results to the specified time interval. Delimit with `/`. For example, `2023-07-13/2023-07-19`.
* `count` (optional)
* Type: Int
* Limits the number of results to the last `n` entries.
### Responses
<!--DOCUSAURUS_CODE_TABS-->
<!--200 SUCCESS-->
*Successfully retrieved audit history*
<!--400 BAD REQUEST-->
*Request in the incorrect format*
<!--404 NOT FOUND-->
*`count` query parameter too large*
<!--END_DOCUSAURUS_CODE_TABS-->
---
### Sample request
The following example retrieves the audit history for all datasources from `2023-07-13` to `2023-07-19`.
<!--DOCUSAURUS_CODE_TABS-->
<!--cURL-->
```shell
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/history?interval=2023-07-13%2F2023-07-19"
```
<!--HTTP-->
```HTTP
GET /druid/coordinator/v1/rules/history?interval=2023-07-13/2023-07-19 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
```
<!--END_DOCUSAURUS_CODE_TABS-->
### Sample response
<details>
<summary>Click to show sample response</summary>
```json
[
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"interval\":\"2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]",
"auditTime": "2023-07-13T18:05:33.066Z"
},
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[]",
"auditTime": "2023-07-18T18:10:21.203Z"
},
{
"key": "wikipedia_api",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]",
"auditTime": "2023-07-18T18:10:44.519Z"
},
{
"key": "wikipedia_api",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[]",
"auditTime": "2023-07-18T18:11:02.110Z"
},
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"interval\":\"2023-07-03T18:49:54.848Z/2023-07-03T18:49:55.861Z\",\"type\":\"dropByInterval\"}]",
"auditTime": "2023-07-18T18:32:50.060Z"
},
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]",
"auditTime": "2023-07-18T18:34:09.657Z"
},
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]",
"auditTime": "2023-07-18T18:38:37.223Z"
},
{
"key": "social_media",
"type": "rules",
"auditInfo": {
"author": "console",
"comment": "test",
"ip": "127.0.0.1"
},
"payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"interval\":\"2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z\",\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadByInterval\"}]",
"auditTime": "2023-07-18T18:49:43.964Z"
}
]
```
</details>