From 1f3b1f85f95394e2e5986247559c0ff34c3b93e3 Mon Sep 17 00:00:00 2001 From: zachjsh Date: Tue, 12 Nov 2024 14:50:55 -0500 Subject: [PATCH] Add documentation for Druids catalog extension (#17459) * SQL syntax error should target USER persona * * revert change to queryHandler and related tests, based on review comments * * add test * Add documentation for druid-catalog extension * * fix error * * fix error * Apply suggestions from code review Co-authored-by: Andreas Maechler * * fix spelling error * * fix spelling --------- Co-authored-by: Andreas Maechler --- docs/development/extensions-core/catalog.md | 453 ++++++++++++++++++++ website/.spelling | 7 + 2 files changed, 460 insertions(+) create mode 100644 docs/development/extensions-core/catalog.md diff --git a/docs/development/extensions-core/catalog.md b/docs/development/extensions-core/catalog.md new file mode 100644 index 00000000000..4590ef49591 --- /dev/null +++ b/docs/development/extensions-core/catalog.md @@ -0,0 +1,453 @@ + + + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Consider this an [EXPERIMENTAL](../experimental.md) feature mostly because it has not been tested yet on a wide variety of long running Druid clusters. + +This extension allows users to configure, update, retrieve, and manage metadata stored in Druid's catalog. At present, only metadata about tables is stored in the catalog. This extension only supports MSQ based ingestion. + +## Configuration + +To use this extension please make sure to [include](../../configuration/extensions.md#loading-extensions) `druid-catalog` in the extensions load list. + +# Catalog Metadata + +## Tables + +A user may define a table with a defined set of column names, and respective data types, along with other properties. When +ingesting data into a table defined in the catalog, the DML query is validated against the definition of the table +as defined in the catalog. This allows the user to omit the table's properties that are found in its definition, +allowing queries to be more concise, and simpler to write. This also allows the user to ensure that the type of data being +written into a defined column of the table is consistent with that columns definition, minimizing errors where unexpected +data is written into a particular column of the table. + +### API Objects + +#### TableSpec + +A tableSpec defines a table + +| Property | Type | Description | Required | Default | +|--------------|---------------------------------|---------------------------------------------------------------------------|----------|---------| +| `type` | String | the type of table. The only value supported at this time is `datasource` | yes | null | +| `properties` | Map | the table's defined properties. see [table properties](#table-properties) | no | null | +| `columns` | List<[ColumnSpec](#columnspec)> | the table's defined columns | no | null | + +#### Table Properties + +| PropertyKeyName | PropertyValueType | Description | Required | Default | +|----------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------| +| `segmentGranularity` | String | determines how time-based partitioning is done. See [Partitioning by time](../../multi-stage-query/concepts.md#partitioning-by-time). Can specify any of the values as permitted for [PARTITIONED BY](../../multi-stage-query/reference.md#partitioned-by). This property value may be overridden at query time, by specifying the PARTITIONED BY clause. | no | null | +| `sealed` | boolean | require all columns in the table schema to be fully declared before data is ingested. Setting this to true will cause failure when DML queries attempt to add undefined columns to the table. | no | false | + +#### ColumnSpec + +| Property | Type | Description | Required | Default | +|--------------|---------------------|------------------------------------------------------------------------------------------------------------------------|----------|---------| +| `name` | String | The name of the column | yes | null | +| `dataType` | String | The type of the column. Can be any column data type that is available to Druid. Depends on what extensions are loaded. | no | null | +| `properties` | Map | the column's defined properties. Non properties defined at this time. | no | null | + +### APIs + +#### Create or update a table + +Update or create a new table containing the given table specification. + +##### URL + +`POST` `/druid/coordinator/v1/catalog/schemas/{schema}/tables/{name}` + +##### Request body + +The request object for this request is a [TableSpec](#tablespec) + +##### Query parameters + +The endpoint supports a set of optional query parameters to enforce optimistic locking, and to specify that a request +is meant to update a table rather than create a new one. In the default case, with no query parameters set, this request +will return an error if a table of the same name already exists in the schema specified. + +| Parameter | Type | Description | +|-------------|---------|-------------------------------------------------------------------------------------------------------------------------------| +| `version` | Long | the expected version of an existing table. The version must match. If not (or if the table does not exist), returns an error. | +| `overwrite` | boolean | if true, then overwrites any existing table. Otherwise, the operation fails if the table already exists. | + +##### Responses + + + + + +*Successfully submitted table spec. Returns an object that includes the version of the table created or updated:* + +```json +{ + "version": 12345687 +} +``` + + + + +*Error thrown due to bad request. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +*Error thrown due to unexpected conditions. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +##### Sample request + +The following example shows how to create a sealed table with several defined columns, and a defined segment granularity of `"P1D"` + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/catalog/schemas/druid/tables/test_table" \ +-X 'POST' \ +--header 'Content-Type: application/json' \ +--data '{ + "type": "datasource", + "columns": [ + { + "name": "__time", + "dataType": "long" + }, + { + "name": "double_col", + "dataType": "double" + }, + { + "name": "float_col", + "dataType": "float" + }, + { + "name": "long_col", + "dataType": "long" + }, + { + "name": "string_col", + "dataType": "string" + } + ], + "properties": { + "segmentGranularity": "P1D", + "sealed": true + } +}' +``` + +##### Sample response + +```json +{ + "version": 1730965026295 +} +``` + +#### Retrieve a table + +Retrieve a table + +##### URL + +`GET` `/druid/coordinator/v1/catalog/schemas/{schema}/tables/{name} + +##### Responses + + + + + +*Successfully retrieved corresponding table's [TableSpec](#tablespec)* + + + + +*Error thrown due to bad request. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + +*Error thrown due to unexpected conditions. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +##### Sample request + +The following example shows how to retrieve a table named 'test_table' in schema 'druid' + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/catalog/schemas/druid/tables/test_table" +``` + +##### Sample response + +
+ View the response + +```json +{ + "id": { + "schema": "druid", + "name": "test_table" + }, + "creationTime": 1730965026295, + "updateTime": 1730965026295, + "state": "ACTIVE", + "spec": { + "type": "datasource", + "properties": { + "segmentGranularity": "P1D", + "sealed": true + }, + "columns": [ + { + "name": "__time", + "dataType": "long" + }, + { + "name": "double_col", + "dataType": "double" + }, + { + "name": "float_col", + "dataType": "float" + }, + { + "name": "long_col", + "dataType": "long" + }, + { + "name": "string_col", + "dataType": "string" + } + ] + } +} +``` +
+ +#### Delete a table + +Delete a table + +##### URL + +`DELETE` `/druid/coordinator/v1/catalog/schemas/{schema}/tables/{name} + +##### Responses + + + + + +*No response body* + + + + +*Error thrown due to bad request. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + +*Error thrown due to unexpected conditions. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +##### Sample request + +The following example shows how to delete the a table named `test_table` in schema `druid` + +```shell +curl -X 'DELETE' "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/catalog/schemas/druid/tables/test_table" +``` + +##### Sample response + +No response body + +#### Retrieve list of schema names + +retrieve list of schema names + +##### URL + +`GET` `/druid/coordinator/v1/catalog/schemas + +##### Responses + + + + + +*Successfully retrieved list of schema names* + + + + +*Error thrown due to bad request. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + +*Error thrown due to unexpected conditions. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +##### Sample request + +The following example shows how to retrieve the list of schema names. + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/catalog/schemas" +``` + +##### Sample response + +```json +[ + "INFORMATION_SCHEMA", + "druid", + "ext", + "lookups", + "sys", + "view" +] +``` + +#### Retrieve list of table names in schema + +Retrieve a list of table names in the schema. + +##### URL + +`GET` `/druid/coordinator/v1/catalog/schemas/{schema}/table + +##### Responses + + + + + +*Successfully retrieved list of table names belonging to schema* + + + + +*Error thrown due to bad request. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + +*Error thrown due to unexpected conditions. Returns a JSON object detailing the error with the following format:* + +```json +{ + "error": "A well-defined error code.", + "errorMessage": "A message with additional details about the error." +} +``` + + + + +##### Sample request + +The following example shows how to retrieve all of the table names of tables belonging to the `druid` schema. + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/catalog/schemas/druid/tables" +``` + +##### Sample response + +```json +[ + "test_table" +] +``` \ No newline at end of file diff --git a/website/.spelling b/website/.spelling index ed2389c5c9e..75ce996c8d9 100644 --- a/website/.spelling +++ b/website/.spelling @@ -2421,3 +2421,10 @@ quantilesFromDDSketch quantileFromDDSketch collapsingLowestDense snapshotVersion + +- ../docs/development/extensions-core/catalog.md +ColumnSpec +PropertyKeyName +PropertyValueType +tableSpec +TableSpec