From cf3ba3219ae0fe2aed5728cb64e34258d3701d34 Mon Sep 17 00:00:00 2001 From: Chloe Zhang Date: Mon, 24 May 2021 16:50:29 -0700 Subject: [PATCH 01/32] Updated SQL and PPL doc Signed-off-by: chloe-zh --- docs/ppl/commands.md | 35 +------- docs/ppl/endpoint.md | 6 +- docs/ppl/index.md | 12 +-- docs/ppl/protocol.md | 4 +- docs/ppl/settings.md | 24 ++++-- docs/sql/basic.md | 122 +++++++++++++------------- docs/sql/complex.md | 8 +- docs/sql/datatypes.md | 2 +- docs/sql/delete.md | 13 +++ docs/sql/endpoints.md | 16 ++-- docs/sql/index.md | 14 +-- docs/sql/limitation.md | 2 +- docs/sql/monitoring.md | 2 +- docs/sql/protocol.md | 177 +++++++++++++++++++------------------- docs/sql/settings.md | 26 +++--- docs/sql/sql-full-text.md | 2 +- docs/sql/troubleshoot.md | 35 +------- docs/sql/workbench.md | 24 +++--- 18 files changed, 250 insertions(+), 274 deletions(-) diff --git a/docs/ppl/commands.md b/docs/ppl/commands.md index 1f9e2586..6060567e 100644 --- a/docs/ppl/commands.md +++ b/docs/ppl/commands.md @@ -46,7 +46,7 @@ search source=accounts; ``` | account_number | firstname | address | balance | gender | city | employer | state | age | email | lastname | -:--- | :--- | +:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | 1 | Amber | 880 Holmes Lane | 39225 | M | Brogan | Pyrami | IL | 32 | amberduke@pyrami.com | Duke | 6 | Hattie | 671 Bristol Street | 5686 | M | Dante | Netagy | TN | 36 | hattiebond@netagy.com | Bond | 13 | Nanette | 789 Madison Street | 32838 | F | Nogal | Quility | VA | 28 | null | Bates @@ -61,7 +61,7 @@ search source=accounts account_number=1 or gender="F"; ``` | account_number | firstname | address | balance | gender | city | employer | state | age | email | lastname | -:--- | :--- | +:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | 1 | Amber | 880 Holmes Lane | 39225 | M | Brogan | Pyrami | IL | 32 | amberduke@pyrami.com | Duke | | 13 | Nanette | 789 Madison Street | 32838 | F | Nogal | Quility | VA | 28 | null | Bates | @@ -513,13 +513,11 @@ Use the `head` command to return the first N number of results in a specified se ### Syntax ```sql -head [keeplast = (true | false)] [while "("")"] [N] +head [N] ``` Field | Description | Required | Default :--- | :--- |:--- -`keeplast` | Use along with the `while` argument to check if the last result in the result set is retained. The last result is what caused the `while` condition to evaluate to false or NULL. Set `keeplast` to true to retain the last result and false to discard it. | No | True -`while` | An expression that evaluates to either true or false. You cannot use statistical functions in this expression. | No | False `N` | Specify the number of results to return. | No | 10 *Example 1*: Get the first 10 results @@ -549,31 +547,6 @@ search source=accounts | fields firstname, age | head 2; | Amber | 32 | Hattie | 36 -*Example 3*: Get the first N results that match a while condition - -To get the first 3 results from all accounts with age less than 30: - -```sql -search source=accounts | fields firstname, age | sort age | head while(age < 30) 3; -``` - -| firstname | age -:--- | :--- | -| Nanette | 28 -| Amber | 32 - -*Example 4*: Get the first N results with a while condition with the last result that failed the condition - -To get the first 3 results from all accounts with age less than 30 and include the last failed condition: - -```sql -search source=accounts | fields firstname, age | sort age | head keeplast=false while(age < 30) 3; -``` - -| firstname | age -:--- | :--- | -| Nanette | 28 - ## rare Use the `rare` command to find the least common values of all fields in a field list. @@ -617,7 +590,7 @@ search source=accounts | rare age by gender; | M | 32 | M | 33 -## top {#top-command} +## top Use the `top` command to find the most common values of all fields in the field list. diff --git a/docs/ppl/endpoint.md b/docs/ppl/endpoint.md index 4ea6919a..bee36a1a 100644 --- a/docs/ppl/endpoint.md +++ b/docs/ppl/endpoint.md @@ -10,13 +10,13 @@ nav_order: 1 To send a query request to PPL plugin, use the HTTP POST request. We recommend a POST request because it doesn't have any length limit and it allows you to pass other parameters to the plugin for other functionality. -Use the explain endpoint for query translation and troubleshooting. +Use the `_explain` endpoint for query translation and troubleshooting. ## Request Format -To use the PPL plugin with your own applications, send requests to `_opensearch/_ppl`, with your query in the request body: +To use the PPL plugin with your own applications, send requests to `_plugins/_ppl`, with your query in the request body: ```json -curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_ppl \ +curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_ppl \ ... -d '{"query" : "source=accounts | fields firstname, lastname"}' ``` diff --git a/docs/ppl/index.md b/docs/ppl/index.md index ec68cee3..d6980704 100644 --- a/docs/ppl/index.md +++ b/docs/ppl/index.md @@ -48,11 +48,11 @@ search source=accounts #### Sample Response -| id | firstname | lastname | -:--- | :--- | :--- | -| 0 | Amber | Duke -| 1 | Hattie | Bond -| 2 | Nanette | Bates -| 3 | Dale | Adams +firstname | lastname | +:--- | :--- | +Amber | Duke +Hattie | Bond +Nanette | Bates +Dale | Adams ![PPL query workbench](../images/ppl.png) diff --git a/docs/ppl/protocol.md b/docs/ppl/protocol.md index 747a265c..3636801f 100644 --- a/docs/ppl/protocol.md +++ b/docs/ppl/protocol.md @@ -14,7 +14,7 @@ The PPL plugin provides responses in JDBC format. The JDBC format is widely used The body of HTTP POST request can take a few more additional fields with the PPL query: ```json -curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_ppl \ +curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_ppl \ ... -d '{"query" : "source=accounts | fields firstname, lastname"}' ``` @@ -58,7 +58,7 @@ The following example shows a normal response where the schema includes a field If any error occurred, error message and the cause will be returned instead: ```json -curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_ppl \ +curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_ppl \ ... -d '{"query" : "source=unknown | fields firstname, lastname"}' { "error": { diff --git a/docs/ppl/settings.md b/docs/ppl/settings.md index 742d3e2b..8dd7476b 100644 --- a/docs/ppl/settings.md +++ b/docs/ppl/settings.md @@ -15,7 +15,7 @@ You can update these settings like any other cluster setting: PUT _cluster/settings { "transient": { - "opensearch": { + "plugins": { "ppl": { "enabled": "false" } @@ -24,12 +24,26 @@ PUT _cluster/settings } ``` -Requests to `_opensearch/_ppl` include index names in the request body, so they have the same access policy considerations as the `bulk`, `mget`, and `msearch` operations. If you set the `rest.action.multi.allow_explicit_index` parameter to `false`, the PPL plugin is disabled. +Similarly, you can also update the settings by sending request to the plugin setting endpoint `_plugins/_query/settings` : +```json +PUT _plugins/_query/settings +{ + "transient": { + "plugins": { + "ppl": { + "enabled": "false" + } + } + } +} +``` + +Requests to `_plugins/_ppl` include index names in the request body, so they have the same access policy considerations as the `bulk`, `mget`, and `msearch` operations. If you set the `rest.action.multi.allow_explicit_index` parameter to `false`, the PPL plugin is disabled. You can specify the settings shown in the following table: Setting | Description | Default :--- | :--- | :--- -`opensearch.ppl.enabled` | Change to `false` to disable the plugin. | True -`opensearch.ppl.query.memory_limit` | Set heap memory usage limit. If a query crosses this limit, it's terminated. | 85% -`opensearch.query.size_limit` | Set the maximum number of results that you want to see. This impacts the accuracy of aggregation operations. For example, if you have 1000 documents in an index, by default, only 200 documents are extracted from the index for aggregation. | 200 +`plugins.ppl.enabled` | Change to `false` to disable the PPL component. | True +`plugins.query.memory_limit` | Set heap memory usage limit. If a query crosses this limit, it's terminated. | 85% +`plugins.query.size_limit` | Set the maximum number of results that you want to see. This impacts the accuracy of aggregation operations. For example, if you have 1000 documents in an index, by default, only 200 documents are extracted from the index for aggregation. | 200 diff --git a/docs/sql/basic.md b/docs/sql/basic.md index c074ece9..a32bcebe 100644 --- a/docs/sql/basic.md +++ b/docs/sql/basic.md @@ -81,12 +81,12 @@ SELECT * FROM accounts ``` -| id | account_number | firstname | gender | city | balance | employer | state | email | address | lastname | age -:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- -0 | 1 | Amber | M | Brogan | 39225 | Pyrami | IL | amberduke@pyrami.com | 880 Holmes Lane | Duke | 32 -1 | 16 | Hattie | M | Dante | 5686 | Netagy | TN | hattiebond@netagy.com | 671 Bristol Street | Bond | 36 -2 | 13 | Nanette | F | Nogal | 32838 | Quility | VA | nanettebates@quility.com | 789 Madison Street | Bates | 28 -3 | 18 | Dale | M | Orick | 4180 | | MD | daleadams@boink.com | 467 Hutchinson Court | Adams | 33 +| account_number | firstname | gender | city | balance | employer | state | email | address | lastname | age +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- +| 1 | Amber | M | Brogan | 39225 | Pyrami | IL | amberduke@pyrami.com | 880 Holmes Lane | Duke | 32 +| 16 | Hattie | M | Dante | 5686 | Netagy | TN | hattiebond@netagy.com | 671 Bristol Street | Bond | 36 +| 13 | Nanette | F | Nogal | 32838 | Quility | VA | nanettebates@quility.com | 789 Madison Street | Bates | 28 +| 18 | Dale | M | Orick | 4180 | | MD | daleadams@boink.com | 467 Hutchinson Court | Adams | 33 *Example 2*: Use field name(s) to retrieve only specific fields: @@ -95,12 +95,12 @@ SELECT firstname, lastname FROM accounts ``` -| id | firstname | lastname -:--- | :--- | :--- -0 | Amber | Duke -1 | Hattie | Bond -2 | Nanette | Bates -3 | Dale | Adams +| firstname | lastname +| :--- | :--- +| Amber | Duke +| Hattie | Bond +| Nanette | Bates +| Dale | Adams *Example 3*: Use field aliases instead of field names. Field aliases are used to make field names more readable: @@ -109,12 +109,12 @@ SELECT account_number AS num FROM accounts ``` -| id | num -:--- | :--- -0 | 1 -1 | 6 -2 | 13 -3 | 18 +| num +:--- +| 1 +| 6 +| 13 +| 18 *Example 4*: Use the `DISTINCT` clause to get back only unique field values. You can specify one or more field names: @@ -123,12 +123,12 @@ SELECT DISTINCT age FROM accounts ``` -| id | age -:--- | :--- -0 | 28 -1 | 32 -2 | 33 -3 | 36 +| age +:--- +| 28 +| 32 +| 33 +| 36 ## From @@ -156,12 +156,12 @@ SELECT account_number, acc.age FROM accounts acc ``` -| id | account_number | age -:--- | :--- | :--- -0 | 1 | 32 -1 | 6 | 36 -2 | 13 | 28 -3 | 18 | 33 +| account_number | age +| :--- | :--- +| 1 | 32 +| 6 | 36 +| 13 | 28 +| 18 | 33 *Example 2*: Use index patterns to query indices that match a specific pattern: @@ -170,12 +170,12 @@ SELECT account_number FROM account* ``` -| id | account_number -:--- | :--- -0 | 1 -1 | 6 -2 | 13 -3 | 18 +| account_number +:--- +| 1 +| 6 +| 13 +| 18 ## Where @@ -205,11 +205,11 @@ FROM accounts WHERE account_number = 1 ``` -| id | account_number -:--- | :--- -0 | 1 +| account_number +| :--- +| 1 -*Example 2*: OpenSearch allows for flexible schema so documents in an index may have different fields. Use `IS NULL` or `IS NOT NULL` to retrieve only missing fields or existing fields. We do not differentiate between missing fields and fields explicitly set to `NULL`: +*Example 2*: OpenSearch allows for flexible schema， so documents in an index may have different fields. Use `IS NULL` or `IS NOT NULL` to retrieve only missing fields or existing fields. We do not differentiate between missing fields and fields explicitly set to `NULL`: ```sql SELECT account_number, employer @@ -217,9 +217,9 @@ FROM accounts WHERE employer IS NULL ``` -| id | account_number | employer -:--- | :--- | :--- -0 | 18 | +| account_number | employer +| :--- | :--- +| 18 | *Example 3*: Deletes a document that satisfies the predicates in the `WHERE` clause: @@ -307,12 +307,12 @@ FROM accounts ORDER BY account_number DESC ``` -| id | account_number -:--- | :--- -0 | 18 -1 | 13 -2 | 6 -3 | 1 +| account_number +| :--- +| 18 +| 13 +| 6 +| 1 *Example 2*: Specify if documents with missing fields are to be put at the beginning or at the end of the results. The default behavior of OpenSearch is to return nulls or missing fields at the end. To push them before non-nulls, use the `IS NOT NULL` operator: @@ -322,12 +322,12 @@ FROM accounts ORDER BY employer IS NOT NULL ``` -| id | employer -:--- | :--- -0 | -1 | Netagy -2 | Pyrami -3 | Quility +| employer +| :--- +|| +| Netagy +| Pyrami +| Quility ## Limit @@ -341,9 +341,9 @@ FROM accounts ORDER BY account_number LIMIT 1 ``` -| id | account_number -:--- | :--- -0 | 1 +| account_number +| :--- +| 1 *Example 2*: If you pass in two arguments, the first is mapped to the `from` parameter and the second to the `size` parameter in OpenSearch. You can use this for simple pagination for small indices, as it's inefficient for large indices. Use `ORDER BY` to ensure the same order between pages: @@ -354,6 +354,6 @@ FROM accounts ORDER BY account_number LIMIT 1, 1 ``` -| id | account_number -:--- | :--- -0 | 6 +| account_number +| :--- +| 6 diff --git a/docs/sql/complex.md b/docs/sql/complex.md index 4222aedc..6bb5544f 100644 --- a/docs/sql/complex.md +++ b/docs/sql/complex.md @@ -142,7 +142,7 @@ The `explain` output is complicated, because a `JOIN` clause is associated with Result set: | a.account_number | a.firstname | a.lastname | e.id | e.name -:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- +:--- | :--- | :--- | :--- | :--- 6 | Hattie | Bond | 6 | Jane Smith ### Example 2: Cross join @@ -167,7 +167,7 @@ JOIN employees_nested e Result set: | a.account_number | a.firstname | a.lastname | e.id | e.name -:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- +:--- | :--- | :--- | :--- | :--- 1 | Amber | Duke | 3 | Bob Smith 1 | Amber | Duke | 4 | Susan Smith 1 | Amber | Duke | 6 | Jane Smith @@ -199,7 +199,7 @@ LEFT JOIN employees_nested e Result set: | a.account_number | a.firstname | a.lastname | e.id | e.name -:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- +:--- | :--- | :--- | :--- | :--- 1 | Amber | Duke | null | null 6 | Hattie | Bond | 6 | Jane Smith 13 | Nanette | Bates | null | null @@ -350,7 +350,7 @@ Explain: Result set: | a1.firstname | a1.lastname | a1.balance -:--- | :--- | :--- | :--- | :--- | :--- +:--- | :--- | :--- Amber | Duke | 39225 Nanette | Bates | 32838 diff --git a/docs/sql/datatypes.md b/docs/sql/datatypes.md index 36a6f4e8..06202bd2 100644 --- a/docs/sql/datatypes.md +++ b/docs/sql/datatypes.md @@ -79,7 +79,7 @@ timestamp | `yyyy-MM-dd hh:mm:ss[.fraction]` | `0001-01-01 00:00:01.000000` UTC The `interval` type represents a temporal duration or a period. | Type | Syntax -:--- | :--- | :--- +:--- | :--- interval | `INTERVAL expr unit` The `expr` unit is any expression that eventually iterates to a quantity value. It represents a unit for interpreting the quantity, including `MICROSECOND`, `SECOND`, `MINUTE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `QUARTER`, and `YEAR`. The `INTERVAL` keyword and the unit specifier are not case sensitive. diff --git a/docs/sql/delete.md b/docs/sql/delete.md index 90e534d1..ac58d59d 100644 --- a/docs/sql/delete.md +++ b/docs/sql/delete.md @@ -11,6 +11,19 @@ nav_order: 12 The `DELETE` statement deletes documents that satisfy the predicates in the `WHERE` clause. If you don't specify the `WHERE` clause, all documents are deleted. +### Setting + +The `DELETE` statement is disabled by default. To enable the `DELETE` functionality in SQL, you need to update the configuration by sending the following request: + +```json +PUT _plugins/_query/settings +{ + "transient": { + "plugins.sql.delete.enabled": "true" + } +} +``` + ### Syntax Rule `singleDeleteStatement`: diff --git a/docs/sql/endpoints.md b/docs/sql/endpoints.md index 2514234b..97526375 100644 --- a/docs/sql/endpoints.md +++ b/docs/sql/endpoints.md @@ -26,7 +26,7 @@ You can send HTTP GET request with your query embedded in URL parameter. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X GET localhost:9200/_opensearch/_sql?sql=SELECT * FROM accounts +>> curl -H 'Content-Type: application/json' -X GET localhost:9200/_plugins/_sql?sql=SELECT * FROM accounts ``` ## POST @@ -40,7 +40,7 @@ You can also send HTTP POST request with your query in request body. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ "query" : "SELECT * FROM accounts" }' ``` @@ -59,7 +59,7 @@ directly. Explain query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql/_explain -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql/_explain -d '{ "query" : "SELECT firstname, lastname FROM accounts WHERE age > 20" }' ``` @@ -119,7 +119,7 @@ The `fetch_size` parameter is only supported for the JDBC response format. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ "fetch_size" : 5, "query" : "SELECT firstname, lastname FROM accounts WHERE age > 20 ORDER BY state ASC" }' @@ -171,7 +171,7 @@ Result set: To fetch subsequent pages, use the `cursor` from last response: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMiLCJsIjo5NTF9" }' ``` @@ -182,7 +182,7 @@ The `datarows` can have more than the `fetch_size` number of records in case the ```json { - "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMabcde12345", + "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMabcde12345", "datarows": [ [ "Abbey", @@ -209,10 +209,10 @@ The `datarows` can have more than the `fetch_size` number of records in case the ``` The `cursor` context is automatically cleared on the last page. -To explicitly clear cursor context, use the `_opensearch/_sql/close endpoint` operation. +To explicitly clear cursor context, use the `_plugins/_sql/close endpoint` operation. ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql/close -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql/close -d '{ "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMiLCJsIjo5NTF9" }' ``` diff --git a/docs/sql/index.md b/docs/sql/index.md index 5e41060f..e6758d84 100644 --- a/docs/sql/index.md +++ b/docs/sql/index.md @@ -13,17 +13,17 @@ OpenSearch SQL lets you write queries in SQL rather than the [OpenSearch query d ## Workbench -The easiest way to get familiar with the SQL plugin is to use **SQL Workbench** in OpenSearch Dashboards to test various queries. To learn more, see [Workbench](workbench/). +The easiest way to get familiar with the SQL plugin is to use **Query Workbench** in OpenSearch Dashboards to test various queries. To learn more, see [Workbench](workbench/). ![OpenSearch Dashboards SQL UI plugin](../images/sql.png) ## REST API -To use the SQL plugin with your own applications, send requests to `_opensearch/_sql`: +To use the SQL plugin with your own applications, send requests to `_plugins/_sql`: ```json -POST _opensearch/_sql +POST _plugins/_sql { "query": "SELECT * FROM my-index LIMIT 50" } @@ -40,12 +40,12 @@ Column | Field You can query multiple indices by listing them or using wildcards: ```json -POST _opensearch/_sql +POST _plugins/_sql { "query": "SELECT * FROM my-index1,myindex2,myindex3 LIMIT 50" } -POST _opensearch/_sql +POST _plugins/_sql { "query": "SELECT * FROM my-index* LIMIT 50" } @@ -54,13 +54,13 @@ POST _opensearch/_sql For a sample [curl](https://curl.haxx.se/) command, try: ```bash -curl -XPOST https://localhost:9200/_opensearch/_sql -u 'admin:admin' -k -H 'Content-Type: application/json' -d '{"query": "SELECT * FROM opensearch_dashboards_sample_data_flights LIMIT 10"}' +curl -XPOST https://localhost:9200/_plugins/_sql -u 'admin:admin' -k -H 'Content-Type: application/json' -d '{"query": "SELECT * FROM opensearch_dashboards_sample_data_flights LIMIT 10"}' ``` By default, queries return data in JDBC format, but you can also return data in standard OpenSearch JSON, CSV, or raw formats: ```json -POST _opensearch/_sql?format=json|csv|raw +POST _plugins/_sql?format=json|csv|raw { "query": "SELECT * FROM my-index LIMIT 50" } diff --git a/docs/sql/limitation.md b/docs/sql/limitation.md index 0fa0eb6e..c7bebb39 100644 --- a/docs/sql/limitation.md +++ b/docs/sql/limitation.md @@ -84,7 +84,7 @@ The pagination query enables you to get back paginated responses. Currently, the pagination only supports basic queries. For example, the following query returns the data with cursor id. ```json -POST _opensearch/_sql/ +POST _plugins/_sql/ { "fetch_size" : 5, "query" : "SELECT OriginCountry, DestCountry FROM opensearch_dashboards_sample_data_flights ORDER BY OriginCountry ASC" diff --git a/docs/sql/monitoring.md b/docs/sql/monitoring.md index 6992170d..b8f6478a 100644 --- a/docs/sql/monitoring.md +++ b/docs/sql/monitoring.md @@ -33,7 +33,7 @@ The meaning of fields in the response is as follows: SQL query: ```console ->> curl -H 'Content-Type: application/json' -X GET localhost:9200/_opensearch/_sql/stats +>> curl -H 'Content-Type: application/json' -X GET localhost:9200/_plugins/_sql/stats ``` Result set: diff --git a/docs/sql/protocol.md b/docs/sql/protocol.md index 6d9316bf..5ed1994f 100644 --- a/docs/sql/protocol.md +++ b/docs/sql/protocol.md @@ -28,7 +28,7 @@ OpenSearch DSL directly. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ "query" : "SELECT firstname, lastname, balance FROM accounts", "filter" : { "range" : { @@ -88,7 +88,7 @@ in prepared SQL query. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ "query": "SELECT * FROM accounts WHERE age = ?", "parameters": [{ "type": "integer", @@ -124,13 +124,96 @@ Explain: } } } + +``` +## JDBC Format + +### Description + +By default, the plugin returns the JDBC standard format. This format +is provided for JDBC driver and clients that need both schema and +result set well formatted. + +### Example 1 + +Here is an example for normal response. The +`schema` includes field name and its type +and `datarows` includes the result set. + +SQL query: + +```console +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ + "query" : "SELECT firstname, lastname, age FROM accounts ORDER BY age LIMIT 2" +}' +``` + +Result set: + +```json +{ + "schema": [{ + "name": "firstname", + "type": "text" + }, + { + "name": "lastname", + "type": "text" + }, + { + "name": "age", + "type": "long" + } + ], + "total": 4, + "datarows": [ + [ + "Nanette", + "Bates", + 28 + ], + [ + "Amber", + "Duke", + 32 + ] + ], + "size": 2, + "status": 200 +} +``` + +### Example 2 + +If any error occurred, error message and the cause will be returned +instead. + +SQL query: + +```console +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{ + "query" : "SELECT unknown FROM accounts" +}' +``` + +Result set: + +```json +{ + "error": { + "reason": "Invalid SQL query", + "details": "Field [unknown] cannot be found or used here.", + "type": "SemanticAnalysisException" + }, + "status": 400 +} ``` ## OpenSearch DSL ### Description -By default the plugin returns original response from OpenSearch in +The `json` format returns original response from OpenSearch in JSON. Because this is the native response from OpenSearch, extra efforts are needed to parse and interpret it. @@ -139,7 +222,7 @@ efforts are needed to parse and interpret it. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql?format=json -d '{ "query" : "SELECT firstname, lastname, age FROM accounts ORDER BY age LIMIT 2" }' ``` @@ -195,88 +278,6 @@ Result set: } ``` -## JDBC Format - -### Description - -JDBC format is provided for JDBC driver and client side that needs both -schema and result set well formatted. - -### Example 1 - -Here is an example for normal response. The -`schema` includes field name and its type -and `datarows` includes the result set. - -SQL query: - -```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql?format=jdbc -d '{ - "query" : "SELECT firstname, lastname, age FROM accounts ORDER BY age LIMIT 2" -}' -``` - -Result set: - -```json -{ - "schema": [{ - "name": "firstname", - "type": "text" - }, - { - "name": "lastname", - "type": "text" - }, - { - "name": "age", - "type": "long" - } - ], - "total": 4, - "datarows": [ - [ - "Nanette", - "Bates", - 28 - ], - [ - "Amber", - "Duke", - 32 - ] - ], - "size": 2, - "status": 200 -} -``` - -### Example 2 - -If any error occurred, error message and the cause will be returned -instead. - -SQL query: - -```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql?format=jdbc -d '{ - "query" : "SELECT unknown FROM accounts" -}' -``` - -Result set: - -```json -{ - "error": { - "reason": "Invalid SQL query", - "details": "Field [unknown] cannot be found or used here.", - "type": "SemanticAnalysisException" - }, - "status": 400 -} -``` - ## CSV Format ### Description @@ -288,7 +289,7 @@ You can also use CSV format to download result set as CSV. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql?format=csv -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql?format=csv -d '{ "query" : "SELECT firstname, lastname, age FROM accounts ORDER BY age" }' ``` @@ -315,7 +316,7 @@ line tool for post processing. SQL query: ```console ->> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opensearch/_sql?format=raw -d '{ +>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql?format=raw -d '{ "query" : "SELECT firstname, lastname, age FROM accounts ORDER BY age" }' ``` diff --git a/docs/sql/settings.md b/docs/sql/settings.md index 873b18b9..020ad988 100644 --- a/docs/sql/settings.md +++ b/docs/sql/settings.md @@ -15,19 +15,25 @@ You can update these settings like any other cluster setting: PUT _cluster/settings { "transient" : { - "opendistro.sql.enabled" : false + "plugins.sql.enabled" : false + } +} +``` + +Similarly, you can also update the settings by sending the request to the plugin setting endpoing `_plugins/_query/setting`: +```json +PUT _plugins/_query/settings +{ + "transient" : { + "plugins.sql.enabled" : false } } ``` Setting | Default | Description :--- | :--- | :--- -`opendistro.sql.enabled` | True | Change to `false` to disable the plugin. -`opendistro.sql.query.slowlog` | 2 seconds | Configure the time limit (in seconds) for slow queries. The plugin logs slow queries as `Slow query: elapsed=xxx (ms)` in `opensearch.log`. -`opendistro.sql.query.analysis.enabled` | True | Enables or disables the query analyzer. Changing this setting to `false` lets you bypass strict syntactic and semantic analysis. -`opendistro.sql.query.analysis.semantic.suggestion` | False | If enabled, the query analyzer suggests correct field names for quick fixes. -`opendistro.sql.query.analysis.semantic.threshold` | 200 | Because query analysis needs to build semantic context in memory, indices with a large number of fields are be skipped. You can update this setting to apply analysis to smaller or larger indices as needed. -`opendistro.sql.query.response.format` | JDBC | Sets the default response format for queries. The supported formats are JDBC, JSON, CSV, raw, and table. -`opendistro.sql.cursor.enabled` | False | You can enable or disable pagination for all queries that are supported. -`opendistro.sql.cursor.fetch_size` | 1,000 | You can set the default `fetch_size` for all queries that are supported by pagination. An explicit `fetch_size` passed in request overrides this value. -`opendistro.sql.cursor.keep_alive` | 1 minute | This value configures how long the cursor context is kept open. Cursor contexts are resource heavy, so we recommend a low value. +`plugins.sql.enabled` | True | Change to `false` to disable the plugin. +`plugins.sql.slowlog` | 2 seconds | Configure the time limit (in seconds) for slow queries. The plugin logs slow queries as `Slow query: elapsed=xxx (ms)` in `opensearch.log`. +`plugins.sql.cursor.keep_alive` | 1 minute | This value configures how long the cursor context is kept open. Cursor contexts are resource heavy, so we recommend a low value. +`plugins.query.memory_limit` | 85% | This setting configures the heap memory usage limit for the circuit breaker of the query engine. +`plugins.query.size_limit` | 200 | The setting sets the default size of index that the query engine fetches from OpenSearch. \ No newline at end of file diff --git a/docs/sql/sql-full-text.md b/docs/sql/sql-full-text.md index 809edea6..0baec86e 100644 --- a/docs/sql/sql-full-text.md +++ b/docs/sql/sql-full-text.md @@ -113,7 +113,7 @@ ORDER BY _score | account_number | address | score -:--- | :--- +:--- | :--- | :--- 1 | 880 Holmes Lane | 0.5 6 | 671 Bristol Street | 100 13 | 789 Madison Street | 100 diff --git a/docs/sql/troubleshoot.md b/docs/sql/troubleshoot.md index 6f289b52..d2d16f63 100644 --- a/docs/sql/troubleshoot.md +++ b/docs/sql/troubleshoot.md @@ -12,7 +12,7 @@ The SQL plugin is stateless, so troubleshooting is mostly focused on why a parti The most common error is the dreaded null pointer exception, which can occur during parsing errors or when using the wrong HTTP method (POST vs. GET and vice versa). The POST method and HTTP request body offer the most consistent results: ```json -POST _opensearch/_sql +POST _plugins/_sql { "query": "SELECT * FROM my-index WHERE ['name.firstname']='saanvi' LIMIT 5" } @@ -23,7 +23,7 @@ If a query isn't behaving the way you expect, use the `_explain` API to see the #### Sample request ```json -POST _opensearch/_sql/_explain +POST _plugins/_sql/_explain { "query": "SELECT * FROM my-index LIMIT 50" } @@ -39,37 +39,6 @@ POST _opensearch/_sql/_explain } ``` -## Syntax analysis exception - -You might receive the following error if the plugin can't parse your query: - -```json -{ - "reason": "Invalid SQL query", - "details": "Failed to parse query due to offending symbol [:] at: 'SELECT * FROM xxx WHERE xxx:' <--- HERE... - More details: Expecting tokens in {, 'AND', 'BETWEEN', 'GROUP', 'HAVING', 'IN', 'IS', 'LIKE', 'LIMIT', - 'NOT', 'OR', 'ORDER', 'REGEXP', '*', '/', '%', '+', '-', 'DIV', 'MOD', '=', '>', '<', '!', - '|', '&', '^', '.', DOT_ID}", - "type": "SyntaxAnalysisException" -} -``` - -To resolve this error: - -1. Check if your syntax follows the [MySQL grammar](https://dev.mysql.com/doc/refman/8.0/en/). -2. If your syntax is correct, disable strict query analysis: - - ```json - PUT _cluster/settings - { - "persistent" : { - "opendistro.sql.query.analysis.enabled" : false - } - } - ``` - -3. Run the query again to see if it works. - ## Index mapping verification exception If you see the following verification exception: diff --git a/docs/sql/workbench.md b/docs/sql/workbench.md index 579cd3b8..f2f557ab 100644 --- a/docs/sql/workbench.md +++ b/docs/sql/workbench.md @@ -1,14 +1,14 @@ --- layout: default -title: Workbench +title: Query Workbench parent: SQL nav_order: 1 --- -# Workbench +# Query Workbench -Use the SQL workbench to easily run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. +Use the Query workbench to easily run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. ## Quick start @@ -38,9 +38,9 @@ To list all your indices: SHOW TABLES LIKE % ``` -| id | TABLE_NAME -:--- | :--- -0 | accounts +| TABLE_NAME +| :--- +| accounts ### Read data @@ -53,9 +53,9 @@ FROM accounts WHERE _id = 1 ``` -| id | account_number | firstname | gender | city | balance | employer | state | email | address | lastname | age -:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- -0 | 1 | Amber | M | Brogan | 39225 | Pyrami | IL | amberduke@pyrami.com | 880 Holmes Lane | Duke | 32 +| account_number | firstname | gender | city | balance | employer | state | email | address | lastname | age +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- +| 1 | Amber | M | Brogan | 39225 | Pyrami | IL | amberduke@pyrami.com | 880 Holmes Lane | Duke | 32 ### Delete data @@ -68,6 +68,6 @@ FROM accounts WHERE _id = 0 ``` -| id | deleted_rows -:--- | :--- -0 | 1 +| deleted_rows +| :--- +| 1 From 2f0577869da4d3a14a905c78735354e429c5f38f Mon Sep 17 00:00:00 2001 From: Sruti Parthiban Date: Tue, 25 May 2021 11:18:35 -0700 Subject: [PATCH 02/32] Update PA docs Signed-off-by: Sruti Parthiban --- docs/opensearch/install/docker.md | 8 ++++---- docs/opensearch/install/plugins.md | 2 +- docs/opensearch/install/tar.md | 8 ++++---- docs/pa/api.md | 6 +++--- docs/pa/index.md | 12 ++++++------ docs/pa/rca/api.md | 4 ++-- docs/pa/rca/reference.md | 2 +- 7 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/opensearch/install/docker.md b/docs/opensearch/install/docker.md index 3c3672c9..effdf768 100644 --- a/docs/opensearch/install/docker.md +++ b/docs/opensearch/install/docker.md @@ -192,25 +192,25 @@ You can also configure `docker-compose.yml` and `opensearch.yml` [to take your o 1. Enable the Performance Analyzer plugin: ```bash - curl -XPOST localhost:9200/_opensearch/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' + curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' ``` If you receive the `curl: (52) Empty reply from server` error, you are likely protecting your cluster with the security plugin and you need to provide credentials. Modify the following command to use your username and password: ```bash - curl -XPOST https://localhost:9200/_opensearch/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k + curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k ``` 1. Enable the Root Cause Analyzer (RCA) framework ```bash - curl -XPOST localhost:9200/_opensearch/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' + curl -XPOST localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' ``` Similar to step 1, if you run into `curl: (52) Empty reply from server`, run the command below to enable RCA ```bash - curl -XPOST https://localhost:9200/_opensearch/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k + curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k ``` 1. By default, Performance Analyzer's endpoints are not accessible from outside the Docker container. diff --git a/docs/opensearch/install/plugins.md b/docs/opensearch/install/plugins.md index ca5d451d..352dabcc 100644 --- a/docs/opensearch/install/plugins.md +++ b/docs/opensearch/install/plugins.md @@ -227,7 +227,7 @@ Performance Analyzer requires some manual configuration after installing the plu 1. Send a test request: ```bash - curl -XGET "localhost:9600/_opensearch/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all" + curl -XGET "localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all" ``` diff --git a/docs/opensearch/install/tar.md b/docs/opensearch/install/tar.md index c362a1dd..1d65839f 100644 --- a/docs/opensearch/install/tar.md +++ b/docs/opensearch/install/tar.md @@ -118,25 +118,25 @@ In a tarball installation, Performance Analyzer collects data when it is enabled 1. In a separate window, enable the Performance Analyzer plugin: ```bash - curl -XPOST localhost:9200/_opensearch/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' + curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' ``` If you receive the `curl: (52) Empty reply from server` error, you are likely protecting your cluster with the security plugin and you need to provide credentials. Modify the following command to use your username and password: ```bash - curl -XPOST https://localhost:9200/_opensearch/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k + curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k ``` 1. Finally, enable the Root Cause Analyzer (RCA) framework ```bash - curl -XPOST localhost:9200/_opensearch/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' + curl -XPOST localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' ``` Similar to step 4, if you run into `curl: (52) Empty reply from server`, run the command below to enable RCA ```bash - curl -XPOST https://localhost:9200/_opensearch/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k + curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k ``` {% comment %} diff --git a/docs/pa/api.md b/docs/pa/api.md index 784d1c6b..9e391b9d 100644 --- a/docs/pa/api.md +++ b/docs/pa/api.md @@ -10,7 +10,7 @@ nav_order: 1 Performance Analyzer uses a single HTTP method and URI for most requests: ``` -GET :9600/_opensearch/_performanceanalyzer/metrics +GET :9600/_plugins/_performanceanalyzer/metrics ``` Note the use of port 9600. Provide parameters for metrics, aggregations, dimensions, and nodes (optional): @@ -25,7 +25,7 @@ For a full list of metrics, see [Metrics reference](../reference/). Performance #### Sample request ``` -GET localhost:9600/_opensearch/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all +GET localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all ``` @@ -104,7 +104,7 @@ Performance Analyzer has one additional URI that returns the unit for each metri #### Sample request ``` -GET localhost:9600/_opensearch/_performanceanalyzer/metrics/units +GET localhost:9600/_plugins/_performanceanalyzer/metrics/units ``` diff --git a/docs/pa/index.md b/docs/pa/index.md index f4ea41b9..daa3abc7 100644 --- a/docs/pa/index.md +++ b/docs/pa/index.md @@ -25,25 +25,25 @@ npm install -g @aws/opensearch-perftop The basic syntax is: ```bash -./perf-top- --dashboard .json --endpoint +./opensearch-perf-top- --dashboard .json --endpoint ``` If you're using npm, the syntax is similar: ```bash -perf-top --dashboard --endpoint +opensearch-perf-top --dashboard --endpoint ``` If you're running PerfTop from a node (i.e. locally), specify port 9600: ```bash -./perf-top-linux --dashboard dashboards/.json --endpoint localhost:9600 +./opensearch-perf-top-linux --dashboard dashboards/.json --endpoint localhost:9600 ``` Otherwise, just specify the OpenSearch endpoint: ```bash -./perf-top-macos --dashboard dashboards/.json --endpoint my-cluster.my-domain.com +./opensearch-perf-top-macos --dashboard dashboards/.json --endpoint my-cluster.my-domain.com ``` PerfTop has four pre-built dashboards in the `dashboards` directory, but you can also [create your own](dashboards/). @@ -83,10 +83,10 @@ mount -o remount /dev/shm ### Security -Performance Analyzer supports encryption in transit for requests. It currently does *not* support client or server authentication for requests. To enable encryption in transit, edit `performance-analyzer.properties` in your `$ES_HOME` directory: +Performance Analyzer supports encryption in transit for requests. It currently does *not* support client or server authentication for requests. To enable encryption in transit, edit `performance-analyzer.properties` in your `$OPENSEARCH_HOME` directory: ```bash -vi $ES_HOME/plugins/opensearch_performance_analyzer/pa_config/performance-analyzer.properties +vi $OPENSEARCH_HOME/plugins/opensearch-performance-analyzer/pa_config/performance-analyzer.properties ``` Change the following lines to configure encryption in transit. Note that `certificate-file-path` must be a certificate for the server, not a root CA: diff --git a/docs/pa/rca/api.md b/docs/pa/rca/api.md index d43ae681..2d3aeb3e 100644 --- a/docs/pa/rca/api.md +++ b/docs/pa/rca/api.md @@ -12,10 +12,10 @@ nav_order: 1 ``` # Request all available RCAs -GET localhost:9600/_opensearch/_performanceanalyzer/rca +GET localhost:9600/_plugins/_performanceanalyzer/rca # Request a specific RCA -GET localhost:9600/_opensearch/_performanceanalyzer/rca?name=HighHeapUsageClusterRca +GET localhost:9600/_plugins/_performanceanalyzer/rca?name=HighHeapUsageClusterRca ``` diff --git a/docs/pa/rca/reference.md b/docs/pa/rca/reference.md index 8a853e3a..c6642785 100644 --- a/docs/pa/rca/reference.md +++ b/docs/pa/rca/reference.md @@ -8,4 +8,4 @@ nav_order: 3 # RCA reference -You can find a reference of available RCAs and their purposes on [Github](https://github.com/opensearch-project/performance-analyzer-rca/tree/master/docs). +You can find a reference of available RCAs and their purposes on [Github](https://github.com/opensearch-project/performance-analyzer-rca/tree/main/docs). From b8fa4ae9db7902ad932a3e5cc4b3958fe2b0228c Mon Sep 17 00:00:00 2001 From: chloe-zh Date: Tue, 25 May 2021 12:05:54 -0700 Subject: [PATCH 03/32] Addressed comments Signed-off-by: chloe-zh --- docs/ppl/commands.md | 2 +- docs/sql/workbench.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ppl/commands.md b/docs/ppl/commands.md index 6060567e..fa472d06 100644 --- a/docs/ppl/commands.md +++ b/docs/ppl/commands.md @@ -590,7 +590,7 @@ search source=accounts | rare age by gender; | M | 32 | M | 33 -## top +## top {#top-command} Use the `top` command to find the most common values of all fields in the field list. diff --git a/docs/sql/workbench.md b/docs/sql/workbench.md index f2f557ab..7f735617 100644 --- a/docs/sql/workbench.md +++ b/docs/sql/workbench.md @@ -8,7 +8,7 @@ nav_order: 1 # Query Workbench -Use the Query workbench to easily run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. +Use the Query Workbench to easily run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. ## Quick start From 58613ec94f2626b985a67fbdeba2612095f7d27d Mon Sep 17 00:00:00 2001 From: keithhc2 Date: Fri, 28 May 2021 16:09:25 -0700 Subject: [PATCH 04/32] Added new section on transform jobs --- docs/im/index-transforms/index.md | 178 +++++ docs/im/index-transforms/transforms-apis.md | 714 ++++++++++++++++++++ docs/im/refresh-analyzer/index.md | 2 +- 3 files changed, 893 insertions(+), 1 deletion(-) create mode 100644 docs/im/index-transforms/index.md create mode 100644 docs/im/index-transforms/transforms-apis.md diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md new file mode 100644 index 00000000..82e6b42c --- /dev/null +++ b/docs/im/index-transforms/index.md @@ -0,0 +1,178 @@ +--- +layout: default +title: Index Transforms +nav_order: 40 +parent: Index management +has_children: true +has_toc: false +--- + +# Index transforms + +Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indices, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways. + +For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories. + +You can use transform jobs in two ways: + +1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then, select the fields you want to transform, aggregations to use in the transformations, and define a schedule for your job to follow. +2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow. + +OpenSearch Dashboards provides a nice view of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards. + +Your use cases will help you decide which method to use to create transform jobs. + +## Create a transform job + +If you don't have any data in your cluster, you can add the sample data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**. + +### Step 1: Choose indices + +1. In the **Job name and description** section, specify a name and an optional description for your job. +2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Add Data Filter** and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL](../../opensearch/query-dsl/). +3. Choose **Next**. + +### Step 2: Select fields to transform + +After specifying the indices, you can select the fields you want to transform, as well as whether to transform them using groupings or aggregations. + +You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to see how many people logged in to a system separated into days of the week, you can group the `day_of_week` field into a target field of `day`, and your transformed index will sort your data into separate groups. + +On the other hand, aggregations let you perform simple calculations. For example, you can include an aggregation in your transform job to define a new field of `sum_of_total_ticket_price` that calculates the sum of all airplane tickets, and then analyze the newly summer data within your transformed index. + +1. In the data table, select the fields you want to transform and expand the drop-down menu to choose the grouping or aggregation you want to use. + + Currently, transform jobs support histogram, date_histogram, and terms groupings. For more information about groupings, see [Bucket Aggregations](../../opensearch/bucket-agg/). In terms of aggregations, you can select from `sum`, `avg`, `max`, `min`, `value_count`, `percentiles`, and `scripted_metric`. For more information about aggregations, see [Metric Aggregations](../../opensearch/metric-agg/). +2. Repeat step 1 for any other fields that you want to transform. +3. After selecting the fields that you want to transform and verifying the transformation, choose **Next**. + +### Step 3: Specify a schedule + +You can configure transform jobs to run once or multiple times on a schedule. Transform jobs are enabled by default. + +1. For **transformation execution frequency**, select either **Define by fixed interval** and specify a **transform interval**. +2. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur. +3. Choose **Next**. + +### Step 4: Review and confirm details + +After confirming your transform job’s details are correct, choose **Create Transform Job**. If you want to edit any part of the job, choose **Edit** of the section you want to change, and make the necessary changes. You can’t change aggregations or groupings after creating a job. + +### Step 5: Search through the transformed index. + +Once the transform job finishes, you can use the _search API operation to search the target index. + +```json +GET target_index/_search +``` + +For example, after running a transform job that transforms the source index based on a `customer_gender field`, you can run the following request that returns all of the fields that have a value of `FEMALE`. + +**Sample Request** + +```json +GET kc_test_job/_search +{ + "query": { + "match": { + "customer_gender_terms" : "FEMALE" + } + } +} +``` + +**Sample Response** + +```json +{ + "took" : 9, + "timed_out" : false, + "_shards" : { + "total" : 5, + "successful" : 5, + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 6, + "relation" : "eq" + }, + "max_score" : 0.9808291, + "hits" : [ + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "w5oN7gnx5nw2PJaS6zDlCw", + "_score" : 0.9808291, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 23, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Men's Shoes" + } + }, + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "r4xZ81gHD1hZG-5HJpcNaw", + "_score" : 0.9808291, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 19, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Men's Accessories" + } + }, + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "sJLR4dGCws_jsUKjNj3u-Q", + "_score" : 0.2876821, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 61, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Men's Clothing" + } + }, + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "nrkzdJOQU7p1BB4fPUT6Ag", + "_score" : 0.13353139, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 730, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Women's Accessories" + } + }, + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "1raYCebejLhqCvA81fytpg", + "_score" : 0.13353139, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 1903, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Women's Clothing" + } + }, + { + "_index" : "kc_test_job", + "_type" : "_doc", + "_id" : "X1fLVivgCgLKmctQid3L5A", + "_score" : 0.13353139, + "_source" : { + "transform._id" : "kc_test_job", + "transform._doc_count" : 1136, + "customer_gender_terms" : "FEMALE", + "category.keyword_terms" : "Women's Shoes" + } + } + ] + } +} +``` diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md new file mode 100644 index 00000000..6f957e32 --- /dev/null +++ b/docs/im/index-transforms/transforms-apis.md @@ -0,0 +1,714 @@ +--- +layout: default +title: Transforms APIs +nav_order: 45 +parent: Index Transforms +grand_parent: Index management +has_toc: true +--- + +# Transforms APIs + +Aside from using OpenSearch Dashboards, you can also use the REST API to create, start, stop, and complete other operations relative to transform jobs. + +#### Table of contents +- TOC +{:toc} + +## Create a transform job + +Creates a transform job. + +**Sample Request** + +```json +PUT _opendistro/_transform/ + +{ + "transform": { + "enabled": true, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "Sample transform job", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +**Sample Response** + +```json +{ + "_id": "sample", + "_version": 7, + "_seq_no": 13, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1621467964243, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, + "description": "Sample transform job", + "source_index": "sample_index", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` +You can specify the following options in the HTTP request body: + +Option | Data Type | Description | Required +:--- | :--- | :--- | :--- +enabled | Boolean | If true, the transform job is enabled at creation. | No +schedule | JSON | The schedule the transform job runs on. | Yes +start_time | Integer | The Unix epoch time of the transform job's start time. | Yes +description | String | Describes the transform job. | No +metadata_id | String | Any metadata to be associated with the transform job. | No +source_index | String | The source index whose data to transform. | Yes +target_index | String | The target index the newly transformed data is added into. You can create a new index or update an existing one. | Yes +data_selection_query | JSON | The query DSL to use to filter a subset of the source index for the transform job. See [query DSL](../../../opensearch/query-dsl) for more information. | Yes +page_size | Integer | The number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1) | Yes +groups | Array | Specifies the grouping(s) to use in the transform job. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations](../../../opensearch/bucket-agg). | Yes +source_field | String | The field(s) to transform | Yes +aggregations | JSON | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations](../../../opensearch/metric-agg). | No + +## Update a transform job + +Updates a transform job if `transform_id` already exists. + +**Sample Request** + +```json +PUT _opendistro/_transform/ + +{ + "transform": { + "enabled": true, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "Sample transform job", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +**Sample Response** + +```json +{ + "_id": "sample", + "_version": 2, + "_seq_no": 14, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1602100553, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": null, + "updated_at": 1621889843874, + "enabled": true, + "enabled_at": 1621889843874, + "description": "Sample transform job", + "source_index": "sample_index", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +The `Update` operation supports the following URL parameters: + +Parameter | Description | Required +:---| :--- | :--- +`if_seq_no` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence number. | No +`if_primary_term` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence term. | No + +## Get a transform job's details + +Returns a transform job's details. + +**Sample Request** + +```json +GET _opendistro/_transform/ +``` + +**Sample Response** + +```json +{ + "_id": "sample", + "_version": 7, + "_seq_no": 13, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1621467964243, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, + "description": "Sample transform job", + "source_index": "sample_index", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +You can also get details of all transform jobs by omitting `transform_id`. + +**Sample Request** + +```json +GET _opendistro/_transform/ +``` + +**Sample Response** + +```json +{ + "total_transforms": 1, + "transforms": [ + { + "_id": "sample", + "_seq_no": 13, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1621467964243, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, + "description": "Sample transform job", + "source_index": "sample_index", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + } + ] +} +``` + +You can specify these options as the `GET` API operation’s URL parameters to filter results: + +Parameter | Description | Required +:--- | :--- | :--- +from | The starting index to search from. (Default: 0) | No +size | Specifies the amount of results to return (Default: 10) | No +search |The search term to use to filter results. | No +sortField | The field to sort results with. | No +sortDirection | Specifies the direction to sort results in. (Default: ASC) | No + +For example, this request returns two results starting from the eighth index. + +**Sample Request** + +```json +GET /_opendistro/_transform/?size=2&from=8 +``` + +**Sample Response** + +```json +{ + "total_transforms": 18, + "transforms": [ + { + "_id": "sample8", + "_seq_no": 93, + "_primary_term": 1, + "transform": { + "transform_id": "sample8", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1622063596812, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA", + "updated_at": 1622063657233, + "enabled": false, + "enabled_at": null, + "description": "Sample transform job", + "source_index": "sample_index3", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target3", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + }, + { + "_id": "sample9", + "_seq_no": 98, + "_primary_term": 1, + "transform": { + "transform_id": "sample9", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1622063598065, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": "x8tCIiYMTE3veSbIJkit5A", + "updated_at": 1622063658388, + "enabled": false, + "enabled_at": null, + "description": "Sample transform job", + "source_index": "sample_index4", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target4", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + } + ] +} +``` + +## Start a transform job + +Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the `start` API operation. + +**Sample Request** + +```json +POST _opendistro//_start +``` + +**Sample Response** + +```json +{ + "acknowledged": true +} +``` + +## Stop a transform job + +Stops/disables a transform job. + +**Sample Request** + +```json +POST _opendistro//_stop +``` + +**Sample Response** + +```json +{ + "acknowledged": true +} +``` + +## Get the status of a transform job + +Returns the status and metadata of a transform job. + +**Sample Request** + +```json +GET _opendistro//_explain +``` + +**Sample Response** + +```json +{ + "sample": { + "metadata_id": "PzmjweME5xbgkenl9UpsYw", + "transform_metadata": { + "transform_id": "sample", + "last_updated_at": 1621883525873, + "status": "finished", + "failure_reason": "null", + "stats": { + "pages_processed": 0, + "documents_processed": 0, + "documents_indexed": 0, + "index_time_in_millis": 0, + "search_time_in_millis": 0 + } + } + } +} +``` + +## Preview a transform job's results + +Returns a preview of what a transformed index would look like. + +**Sample Request** + +```json +POST _opendistro/_transform/_preview + +{ + "transform": { + "enabled": false, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "test transform", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 10, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +**Sample Response** + +```json +{ + "documents" : [ + { + "quantity" : 862.0, + "gender" : "FEMALE", + "day" : "Friday" + }, + { + "quantity" : 682.0, + "gender" : "FEMALE", + "day" : "Monday" + }, + { + "quantity" : 772.0, + "gender" : "FEMALE", + "day" : "Saturday" + }, + { + "quantity" : 669.0, + "gender" : "FEMALE", + "day" : "Sunday" + }, + { + "quantity" : 887.0, + "gender" : "FEMALE", + "day" : "Thursday" + } + ] +} +``` + +## Delete a transform job + +Deletes a transform job. This operation does not delete the source or target indices. + +**Sample Request** + +```json +DELETE _opendistro/_transform/ +``` + +**Sample Response** + +```json +{ + "took": 205, + "errors": false, + "items": [ + { + "delete": { + "_index": ".opendistro-ism-config", + "_type": "_doc", + "_id": "sample", + "_version": 4, + "result": "deleted", + "forced_refresh": true, + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 6, + "_primary_term": 1, + "status": 200 + } + } + ] +} +``` diff --git a/docs/im/refresh-analyzer/index.md b/docs/im/refresh-analyzer/index.md index 4bd5685a..00673fed 100644 --- a/docs/im/refresh-analyzer/index.md +++ b/docs/im/refresh-analyzer/index.md @@ -1,7 +1,7 @@ --- layout: default title: Refresh Search Analyzer -nav_order: 40 +nav_order: 50 parent: Index management has_children: false redirect_from: /docs/ism/refresh-analyzer/ From 6133598b4e2f7cef66a4f9a4ce154ed5f69a18f0 Mon Sep 17 00:00:00 2001 From: keithhc2 Date: Fri, 28 May 2021 16:11:37 -0700 Subject: [PATCH 05/32] Tweaked some names --- docs/im/index-transforms/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md index 82e6b42c..b6e4aaca 100644 --- a/docs/im/index-transforms/index.md +++ b/docs/im/index-transforms/index.md @@ -71,7 +71,7 @@ For example, after running a transform job that transforms the source index base **Sample Request** ```json -GET kc_test_job/_search +GET sample_target_index/_search { "query": { "match": { @@ -101,7 +101,7 @@ GET kc_test_job/_search "max_score" : 0.9808291, "hits" : [ { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "w5oN7gnx5nw2PJaS6zDlCw", "_score" : 0.9808291, From c2104dcdf7c62d4d3eb3a48320beb120a776aa9d Mon Sep 17 00:00:00 2001 From: keithhc2 Date: Fri, 28 May 2021 16:16:56 -0700 Subject: [PATCH 06/32] More tweaking of names --- docs/im/index-transforms/index.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md index b6e4aaca..86d0d0b8 100644 --- a/docs/im/index-transforms/index.md +++ b/docs/im/index-transforms/index.md @@ -106,67 +106,67 @@ GET sample_target_index/_search "_id" : "w5oN7gnx5nw2PJaS6zDlCw", "_score" : 0.9808291, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 23, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Men's Shoes" } }, { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "r4xZ81gHD1hZG-5HJpcNaw", "_score" : 0.9808291, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 19, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Men's Accessories" } }, { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "sJLR4dGCws_jsUKjNj3u-Q", "_score" : 0.2876821, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 61, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Men's Clothing" } }, { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "nrkzdJOQU7p1BB4fPUT6Ag", "_score" : 0.13353139, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 730, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Women's Accessories" } }, { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "1raYCebejLhqCvA81fytpg", "_score" : 0.13353139, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 1903, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Women's Clothing" } }, { - "_index" : "kc_test_job", + "_index" : "sample_target_index", "_type" : "_doc", "_id" : "X1fLVivgCgLKmctQid3L5A", "_score" : 0.13353139, "_source" : { - "transform._id" : "kc_test_job", + "transform._id" : "sample_target_index", "transform._doc_count" : 1136, "customer_gender_terms" : "FEMALE", "category.keyword_terms" : "Women's Shoes" From 0a14f1bfb161c10afc8812b7ae47b189c42896af Mon Sep 17 00:00:00 2001 From: keithhc2 Date: Fri, 28 May 2021 16:34:58 -0700 Subject: [PATCH 07/32] Adjusted spacing --- docs/im/index-transforms/transforms-apis.md | 1018 +++++++++---------- 1 file changed, 509 insertions(+), 509 deletions(-) diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md index 6f957e32..9750145c 100644 --- a/docs/im/index-transforms/transforms-apis.md +++ b/docs/im/index-transforms/transforms-apis.md @@ -26,20 +26,77 @@ PUT _opendistro/_transform/ { "transform": { - "enabled": true, + "enabled": true, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "Sample transform job", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +**Sample Response** + +```json +{ + "_id": "sample", + "_version": 7, + "_seq_no": 13, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, "schedule": { "interval": { + "start_time": 1621467964243, "period": 1, - "unit": "Minutes", - "start_time": 1602100553 + "unit": "Minutes" } }, + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, "description": "Sample transform job", "source_index": "sample_index", - "target_index": "sample_target", "data_selection_query": { - "match_all": {} + "match_all": { + "boost": 1.0 + } }, + "target_index": "sample_target", + "roles": [], "page_size": 1, "groups": [ { @@ -65,63 +122,6 @@ PUT _opendistro/_transform/ } } ``` - -**Sample Response** - -```json -{ - "_id": "sample", - "_version": 7, - "_seq_no": 13, - "_primary_term": 1, - "transform": { - "transform_id": "sample", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1621467964243, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": null, - "updated_at": 1621467964243, - "enabled": true, - "enabled_at": 1621467964243, - "description": "Sample transform job", - "source_index": "sample_index", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } -} -``` You can specify the following options in the HTTP request body: Option | Data Type | Description | Required @@ -150,20 +150,77 @@ PUT _opendistro/_transform/ { "transform": { - "enabled": true, + "enabled": true, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "Sample transform job", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + +**Sample Response** + +```json +{ + "_id": "sample", + "_version": 2, + "_seq_no": 14, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, "schedule": { "interval": { + "start_time": 1602100553, "period": 1, - "unit": "Minutes", - "start_time": 1602100553 + "unit": "Minutes" } }, + "metadata_id": null, + "updated_at": 1621889843874, + "enabled": true, + "enabled_at": 1621889843874, "description": "Sample transform job", "source_index": "sample_index", - "target_index": "sample_target", "data_selection_query": { - "match_all": {} + "match_all": { + "boost": 1.0 + } }, + "target_index": "sample_target", + "roles": [], "page_size": 1, "groups": [ { @@ -190,63 +247,6 @@ PUT _opendistro/_transform/ } ``` -**Sample Response** - -```json -{ - "_id": "sample", - "_version": 2, - "_seq_no": 14, - "_primary_term": 1, - "transform": { - "transform_id": "sample", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1602100553, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": null, - "updated_at": 1621889843874, - "enabled": true, - "enabled_at": 1621889843874, - "description": "Sample transform job", - "source_index": "sample_index", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } -} -``` - The `Update` operation supports the following URL parameters: Parameter | Description | Required @@ -268,353 +268,34 @@ GET _opendistro/_transform/ ```json { - "_id": "sample", - "_version": 7, - "_seq_no": 13, - "_primary_term": 1, - "transform": { - "transform_id": "sample", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1621467964243, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": null, - "updated_at": 1621467964243, - "enabled": true, - "enabled_at": 1621467964243, - "description": "Sample transform job", - "source_index": "sample_index", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } -} -``` - -You can also get details of all transform jobs by omitting `transform_id`. - -**Sample Request** - -```json -GET _opendistro/_transform/ -``` - -**Sample Response** - -```json -{ - "total_transforms": 1, - "transforms": [ - { - "_id": "sample", - "_seq_no": 13, - "_primary_term": 1, - "transform": { - "transform_id": "sample", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1621467964243, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": null, - "updated_at": 1621467964243, - "enabled": true, - "enabled_at": 1621467964243, - "description": "Sample transform job", - "source_index": "sample_index", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } - } - ] -} -``` - -You can specify these options as the `GET` API operation’s URL parameters to filter results: - -Parameter | Description | Required -:--- | :--- | :--- -from | The starting index to search from. (Default: 0) | No -size | Specifies the amount of results to return (Default: 10) | No -search |The search term to use to filter results. | No -sortField | The field to sort results with. | No -sortDirection | Specifies the direction to sort results in. (Default: ASC) | No - -For example, this request returns two results starting from the eighth index. - -**Sample Request** - -```json -GET /_opendistro/_transform/?size=2&from=8 -``` - -**Sample Response** - -```json -{ - "total_transforms": 18, - "transforms": [ - { - "_id": "sample8", - "_seq_no": 93, - "_primary_term": 1, - "transform": { - "transform_id": "sample8", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1622063596812, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA", - "updated_at": 1622063657233, - "enabled": false, - "enabled_at": null, - "description": "Sample transform job", - "source_index": "sample_index3", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target3", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } - }, - { - "_id": "sample9", - "_seq_no": 98, - "_primary_term": 1, - "transform": { - "transform_id": "sample9", - "schema_version": 7, - "schedule": { - "interval": { - "start_time": 1622063598065, - "period": 1, - "unit": "Minutes" - } - }, - "metadata_id": "x8tCIiYMTE3veSbIJkit5A", - "updated_at": 1622063658388, - "enabled": false, - "enabled_at": null, - "description": "Sample transform job", - "source_index": "sample_index4", - "data_selection_query": { - "match_all": { - "boost": 1.0 - } - }, - "target_index": "sample_target4", - "roles": [], - "page_size": 1, - "groups": [ - { - "terms": { - "source_field": "customer_gender", - "target_field": "gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day" - } - } - ], - "aggregations": { - "quantity": { - "sum": { - "field": "total_quantity" - } - } - } - } - } - ] -} -``` - -## Start a transform job - -Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the `start` API operation. - -**Sample Request** - -```json -POST _opendistro//_start -``` - -**Sample Response** - -```json -{ - "acknowledged": true -} -``` - -## Stop a transform job - -Stops/disables a transform job. - -**Sample Request** - -```json -POST _opendistro//_stop -``` - -**Sample Response** - -```json -{ - "acknowledged": true -} -``` - -## Get the status of a transform job - -Returns the status and metadata of a transform job. - -**Sample Request** - -```json -GET _opendistro//_explain -``` - -**Sample Response** - -```json -{ - "sample": { - "metadata_id": "PzmjweME5xbgkenl9UpsYw", - "transform_metadata": { - "transform_id": "sample", - "last_updated_at": 1621883525873, - "status": "finished", - "failure_reason": "null", - "stats": { - "pages_processed": 0, - "documents_processed": 0, - "documents_indexed": 0, - "index_time_in_millis": 0, - "search_time_in_millis": 0 - } - } - } -} -``` - -## Preview a transform job's results - -Returns a preview of what a transformed index would look like. - -**Sample Request** - -```json -POST _opendistro/_transform/_preview - -{ + "_id": "sample", + "_version": 7, + "_seq_no": 13, + "_primary_term": 1, "transform": { - "enabled": false, + "transform_id": "sample", + "schema_version": 7, "schedule": { "interval": { + "start_time": 1621467964243, "period": 1, - "unit": "Minutes", - "start_time": 1602100553 + "unit": "Minutes" } }, - "description": "test transform", + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, + "description": "Sample transform job", "source_index": "sample_index", - "target_index": "sample_target", "data_selection_query": { - "match_all": {} + "match_all": { + "boost": 1.0 + } }, - "page_size": 10, + "target_index": "sample_target", + "roles": [], + "page_size": 1, "groups": [ { "terms": { @@ -640,36 +321,355 @@ POST _opendistro/_transform/_preview } ``` +You can also get details of all transform jobs by omitting `transform_id`. + +**Sample Request** + +```json +GET _opendistro/_transform/ +``` + +**Sample Response** + +```json +{ + "total_transforms": 1, + "transforms": [ + { + "_id": "sample", + "_seq_no": 13, + "_primary_term": 1, + "transform": { + "transform_id": "sample", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1621467964243, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": null, + "updated_at": 1621467964243, + "enabled": true, + "enabled_at": 1621467964243, + "description": "Sample transform job", + "source_index": "sample_index", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + } + ] +} +``` + +You can specify these options as the `GET` API operation’s URL parameters to filter results: + +Parameter | Description | Required +:--- | :--- | :--- +from | The starting index to search from. (Default: 0) | No +size | Specifies the amount of results to return (Default: 10) | No +search |The search term to use to filter results. | No +sortField | The field to sort results with. | No +sortDirection | Specifies the direction to sort results in. (Default: ASC) | No + +For example, this request returns two results starting from the eighth index. + +**Sample Request** + +```json +GET /_opendistro/_transform/?size=2&from=8 +``` + +**Sample Response** + +```json +{ + "total_transforms": 18, + "transforms": [ + { + "_id": "sample8", + "_seq_no": 93, + "_primary_term": 1, + "transform": { + "transform_id": "sample8", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1622063596812, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA", + "updated_at": 1622063657233, + "enabled": false, + "enabled_at": null, + "description": "Sample transform job", + "source_index": "sample_index3", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target3", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + }, + { + "_id": "sample9", + "_seq_no": 98, + "_primary_term": 1, + "transform": { + "transform_id": "sample9", + "schema_version": 7, + "schedule": { + "interval": { + "start_time": 1622063598065, + "period": 1, + "unit": "Minutes" + } + }, + "metadata_id": "x8tCIiYMTE3veSbIJkit5A", + "updated_at": 1622063658388, + "enabled": false, + "enabled_at": null, + "description": "Sample transform job", + "source_index": "sample_index4", + "data_selection_query": { + "match_all": { + "boost": 1.0 + } + }, + "target_index": "sample_target4", + "roles": [], + "page_size": 1, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } + } + ] +} +``` + +## Start a transform job + +Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the `start` API operation. + +**Sample Request** + +```json +POST _opendistro//_start +``` + +**Sample Response** + +```json +{ + "acknowledged": true +} +``` + +## Stop a transform job + +Stops/disables a transform job. + +**Sample Request** + +```json +POST _opendistro//_stop +``` + +**Sample Response** + +```json +{ + "acknowledged": true +} +``` + +## Get the status of a transform job + +Returns the status and metadata of a transform job. + +**Sample Request** + +```json +GET _opendistro//_explain +``` + +**Sample Response** + +```json +{ + "sample": { + "metadata_id": "PzmjweME5xbgkenl9UpsYw", + "transform_metadata": { + "transform_id": "sample", + "last_updated_at": 1621883525873, + "status": "finished", + "failure_reason": "null", + "stats": { + "pages_processed": 0, + "documents_processed": 0, + "documents_indexed": 0, + "index_time_in_millis": 0, + "search_time_in_millis": 0 + } + } + } +} +``` + +## Preview a transform job's results + +Returns a preview of what a transformed index would look like. + +**Sample Request** + +```json +POST _opendistro/_transform/_preview + +{ + "transform": { + "enabled": false, + "schedule": { + "interval": { + "period": 1, + "unit": "Minutes", + "start_time": 1602100553 + } + }, + "description": "test transform", + "source_index": "sample_index", + "target_index": "sample_target", + "data_selection_query": { + "match_all": {} + }, + "page_size": 10, + "groups": [ + { + "terms": { + "source_field": "customer_gender", + "target_field": "gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day" + } + } + ], + "aggregations": { + "quantity": { + "sum": { + "field": "total_quantity" + } + } + } + } +} +``` + **Sample Response** ```json { "documents" : [ - { - "quantity" : 862.0, - "gender" : "FEMALE", - "day" : "Friday" - }, - { - "quantity" : 682.0, - "gender" : "FEMALE", - "day" : "Monday" - }, - { - "quantity" : 772.0, - "gender" : "FEMALE", - "day" : "Saturday" - }, - { - "quantity" : 669.0, - "gender" : "FEMALE", - "day" : "Sunday" - }, - { - "quantity" : 887.0, - "gender" : "FEMALE", - "day" : "Thursday" - } + { + "quantity" : 862.0, + "gender" : "FEMALE", + "day" : "Friday" + }, + { + "quantity" : 682.0, + "gender" : "FEMALE", + "day" : "Monday" + }, + { + "quantity" : 772.0, + "gender" : "FEMALE", + "day" : "Saturday" + }, + { + "quantity" : 669.0, + "gender" : "FEMALE", + "day" : "Sunday" + }, + { + "quantity" : 887.0, + "gender" : "FEMALE", + "day" : "Thursday" + } ] } ``` @@ -688,27 +688,27 @@ DELETE _opendistro/_transform/ ```json { - "took": 205, - "errors": false, - "items": [ - { - "delete": { - "_index": ".opendistro-ism-config", - "_type": "_doc", - "_id": "sample", - "_version": 4, - "result": "deleted", - "forced_refresh": true, - "_shards": { - "total": 2, - "successful": 1, - "failed": 0 - }, - "_seq_no": 6, - "_primary_term": 1, - "status": 200 - } - } - ] + "took": 205, + "errors": false, + "items": [ + { + "delete": { + "_index": ".opendistro-ism-config", + "_type": "_doc", + "_id": "sample", + "_version": 4, + "result": "deleted", + "forced_refresh": true, + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 6, + "_primary_term": 1, + "status": 200 + } + } + ] } ``` From f4bd2a96ddd4d6864d193943c3a725be18cd4d7c Mon Sep 17 00:00:00 2001 From: keithhc2 Date: Tue, 1 Jun 2021 12:04:16 -0700 Subject: [PATCH 08/32] Minor language tweaks --- docs/im/index-transforms/index.md | 11 ++++++----- docs/im/index-transforms/transforms-apis.md | 4 ++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md index 86d0d0b8..5b36fad8 100644 --- a/docs/im/index-transforms/index.md +++ b/docs/im/index-transforms/index.md @@ -24,25 +24,26 @@ Your use cases will help you decide which method to use to create transform jobs ## Create a transform job -If you don't have any data in your cluster, you can add the sample data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**. +If you don't have any data in your cluster, you can use the sample data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**. ### Step 1: Choose indices 1. In the **Job name and description** section, specify a name and an optional description for your job. -2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Add Data Filter** and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL](../../opensearch/query-dsl/). +2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Add Data Filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL](../../opensearch/query-dsl/). 3. Choose **Next**. ### Step 2: Select fields to transform -After specifying the indices, you can select the fields you want to transform, as well as whether to transform them using groupings or aggregations. +After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations. You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to see how many people logged in to a system separated into days of the week, you can group the `day_of_week` field into a target field of `day`, and your transformed index will sort your data into separate groups. On the other hand, aggregations let you perform simple calculations. For example, you can include an aggregation in your transform job to define a new field of `sum_of_total_ticket_price` that calculates the sum of all airplane tickets, and then analyze the newly summer data within your transformed index. -1. In the data table, select the fields you want to transform and expand the drop-down menu to choose the grouping or aggregation you want to use. +1. In the data table, select the fields you want to transform and expand the drop-down menu within the column header to choose the grouping or aggregation you want to use. Currently, transform jobs support histogram, date_histogram, and terms groupings. For more information about groupings, see [Bucket Aggregations](../../opensearch/bucket-agg/). In terms of aggregations, you can select from `sum`, `avg`, `max`, `min`, `value_count`, `percentiles`, and `scripted_metric`. For more information about aggregations, see [Metric Aggregations](../../opensearch/metric-agg/). + 2. Repeat step 1 for any other fields that you want to transform. 3. After selecting the fields that you want to transform and verifying the transformation, choose **Next**. @@ -60,7 +61,7 @@ After confirming your transform job’s details are correct, choose **Create Tra ### Step 5: Search through the transformed index. -Once the transform job finishes, you can use the _search API operation to search the target index. +Once the transform job finishes, you can use the `_search` API operation to search the target index. ```json GET target_index/_search diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md index 9750145c..eb77da6a 100644 --- a/docs/im/index-transforms/transforms-apis.md +++ b/docs/im/index-transforms/transforms-apis.md @@ -135,9 +135,9 @@ source_index | String | The source index whose data to transform. | Yes target_index | String | The target index the newly transformed data is added into. You can create a new index or update an existing one. | Yes data_selection_query | JSON | The query DSL to use to filter a subset of the source index for the transform job. See [query DSL](../../../opensearch/query-dsl) for more information. | Yes page_size | Integer | The number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1) | Yes -groups | Array | Specifies the grouping(s) to use in the transform job. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations](../../../opensearch/bucket-agg). | Yes +groups | Array | Specifies the grouping(s) to use in the transform job. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations](../../../opensearch/bucket-agg). | Yes if not using aggregations source_field | String | The field(s) to transform | Yes -aggregations | JSON | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations](../../../opensearch/metric-agg). | No +aggregations | JSON | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations](../../../opensearch/metric-agg). | Yes if not using groups ## Update a transform job From 78a72f5d22a9f7594409c40d4f7b03d03390b2e6 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 15:34:24 -0400 Subject: [PATCH 09/32] Adds Agensts and Ingestion Tools page with compatibility matrices --- docs/agents-and-ingestion-tools/index.md | 64 ++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/agents-and-ingestion-tools/index.md diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md new file mode 100644 index 00000000..51b04902 --- /dev/null +++ b/docs/agents-and-ingestion-tools/index.md @@ -0,0 +1,64 @@ +--- +layout: default +title: Agents and Ingestion Tools +nav_order: 100 +has_children: false +has_toc: false +--- + +# Agents and Ingestion Tools + +Historically there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier a [set version YML configuration](https://github.com/opensearch-project/OpenSearch/issues/693) is being added so that you can set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. This will enable it to connect with tools that do version checking while need OpenSource compatibilities are added to your tools. + + +## Compatibility Matrices + +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** + +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](). + +### Compatibility Matrix for Logstash + +| |Logstash OSS 7.x to 7.11.x* |Logstash OSS 7.12.x** |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin***^ | +|--- |--- |--- |--- |--- | +|Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |*No* |*Yes* | +|Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |*No* |*Yes* | +|ODFE OSS v1.x to 1.12 |*Yes* |*Yes* |*No* |*Yes* | +|ODFE 1.13^ |*Yes* |*Yes* |*No* |*Yes* | +|Amazon Elasticsearch Service 7.x to 7.9 |Yes |Yes |*No* |*Yes* | +|Amazon Elasticsearch Service 7.10^ |Yes |Yes |*No* |*Yes* | +|Amazon Elasticsearch Service 7.x to 7.9 with IAM |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | +|Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | +|OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | + +*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) +**Most current compatible version with Elasticsearch OSS +***Planning to build +^ Most current version of software +*Italicized cells are based on documentation or release notes but are still pending testing to validate* + +### Compatibility Matrix for Beats + +| |Beats OSS 7.x to 7.11.x* |Beats OSS 7.12.x** |Beats 7.13.x^ | +|--- |--- |--- |--- | +|Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |No | +|Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |No | +|ODFE OSS v1.x to 1.12 |*Yes* |*Yes* |No | +|ODFE 1.13^ |*Yes* |*Yes* |No | +|Amazon Elasticsearch Service 7.x to 7.9 |*Yes* |*Yes* |No | +|Amazon Elasticsearch Service 7.10^ |*Yes* |*Yes* |No | +|OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |No | +|Logstash OSS 7.x to 7.11.x* |*Yes* |*Yes* |*Yes* | +|Logstash OSS 7.12.x** |*Yes* |*Yes* |*Yes* | +|Logstash 7.13.x with OpenSearch output plugin |*Yes* |*Yes* |*Yes* | + +*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) +**Most current compatible version with Elasticsearch OSS +^ Most current version of software +*Italicized cells are based on documentation or release notes but are still pending testing to validate* + +### Credits and Thank You for Contributing and Testing + +* [VijayanB](https://github.com/VijayanB) +* [vamshin](https://github.com/vamshin) +* [anirudha](https://github.com/anirudha) From 0775042a7ded893a67a774fc16d2c0afe3f7e514 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 15:56:31 -0400 Subject: [PATCH 10/32] fixes missing link to OpenSearch output plugin issue --- docs/agents-and-ingestion-tools/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index 51b04902..ae277514 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -15,7 +15,7 @@ Historically there have been multiple popular agents and ingestion tools that wo So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** -Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](). +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). ### Compatibility Matrix for Logstash From 7190744ae79aa9d2fd82c12f74662199d3e921a5 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:01:23 -0400 Subject: [PATCH 11/32] Fixes formatting of content below tables --- docs/agents-and-ingestion-tools/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index ae277514..58218f40 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -31,10 +31,10 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | |OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | -*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) -**Most current compatible version with Elasticsearch OSS -***Planning to build -^ Most current version of software +\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ +\*\*Most current compatible version with Elasticsearch OSS\ +\*\*\*Planning to build\ +^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate* ### Compatibility Matrix for Beats @@ -52,9 +52,9 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Logstash OSS 7.12.x** |*Yes* |*Yes* |*Yes* | |Logstash 7.13.x with OpenSearch output plugin |*Yes* |*Yes* |*Yes* | -*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) -**Most current compatible version with Elasticsearch OSS -^ Most current version of software +\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ +\*\*Most current compatible version with Elasticsearch OSS\ +^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate* ### Credits and Thank You for Contributing and Testing From e066d01f6da7de4eca0790a3fdaacaae8bb7103b Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:39:26 -0400 Subject: [PATCH 12/32] Fixes typo --- docs/agents-and-ingestion-tools/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index 58218f40..b5d25c04 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -13,7 +13,7 @@ Historically there have been multiple popular agents and ingestion tools that wo ## Compatibility Matrices -So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). From be083bb5db2f9fa6a55860c94bf2bac7a7e7564a Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:54:25 -0400 Subject: [PATCH 13/32] Fixes typos and tries to simplify some sentences --- docs/agents-and-ingestion-tools/index.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index b5d25c04..cac4bb97 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -8,18 +8,18 @@ has_toc: false # Agents and Ingestion Tools -Historically there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier a [set version YML configuration](https://github.com/opensearch-project/OpenSearch/issues/693) is being added so that you can set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. This will enable it to connect with tools that do version checking while need OpenSource compatibilities are added to your tools. +Historically, there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier, a [version value](https://github.com/opensearch-project/OpenSearch/issues/693) is being added to the OpenSearch YML. This will let you set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. By reporting 7.10.2, the cluster will be able to connect with tools that do version checking. This is intended to be an intermediate solution while OpenSearch support is added to more tools. ## Compatibility Matrices -So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) and add yourself to the "Credits and Thanks you" section at the bottom of the page. We greatly appreciate the help!** -Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [proposed OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). ### Compatibility Matrix for Logstash -| |Logstash OSS 7.x to 7.11.x* |Logstash OSS 7.12.x** |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin***^ | +| |Logstash OSS 7.x to 7.11.x |Logstash OSS 7.12.x* |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin**^ | |--- |--- |--- |--- |--- | |Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |*No* |*Yes* | |Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |*No* |*Yes* | @@ -31,9 +31,8 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | |OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | -\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ -\*\*Most current compatible version with Elasticsearch OSS\ -\*\*\*Planning to build\ +\*Most current compatible version with Elasticsearch OSS\ +\*\*Planning to build\ ^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate* From 37ac04ee8ee2b271b725dc071fd71d892d518c42 Mon Sep 17 00:00:00 2001 From: ashwinkumar12345 Date: Sun, 6 Jun 2021 00:22:43 -0700 Subject: [PATCH 14/32] minor edits --- docs/opensearch/aggregations.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/opensearch/aggregations.md b/docs/opensearch/aggregations.md index 160ec4d6..1f572402 100644 --- a/docs/opensearch/aggregations.md +++ b/docs/opensearch/aggregations.md @@ -16,13 +16,13 @@ OpenSearch can perform aggregations on massive datasets in milliseconds. Compare ## Aggregations on text fields -By default, OpenSearch doesn't support aggregations on a text field. -Because text fields are tokenized, an aggregation on a text field has to reverse the tokenization process back to its original string and then formulate an aggregation based on that. Such an operation consumes significant memory and degrades cluster performance. +By default, OpenSearch doesn't support aggregations on a text field. Because text fields are tokenized, an aggregation on a text field has to reverse the tokenization process back to its original string and then formulate an aggregation based on that. This kind of an operation consumes significant memory and degrades cluster performance. While you can enable aggregations on text fields by setting the `fielddata` parameter to `true` in the mapping, the aggregations are still based on the tokenized words and not on the raw text. We recommend keeping a raw version of the text field as a `keyword` field that you can aggregate on. -In this case, you can perform aggregations on the `title.raw` field, instead of the `title` field: + +In this case, you can perform aggregations on the `title.raw` field, instead of on the `title` field: ```json PUT movies @@ -61,15 +61,13 @@ GET _search If you’re only interested in the aggregation result and not in the results of the query, set `size` to 0. -In the `aggs` property (you can use `aggregations` if you want), you can define any number of aggregations. -Each aggregation is defined by its name and one of the types of aggregations that OpenSearch supports. +In the `aggs` property (you can use `aggregations` if you want), you can define any number of aggregations. Each aggregation is defined by its name and one of the types of aggregations that OpenSearch supports. -The name of the aggregation helps you to distinguish between different aggregations in the response. -The `AGG_TYPE` property is where you specify the type of aggregation. +The name of the aggregation helps you to distinguish between different aggregations in the response. The `AGG_TYPE` property is where you specify the type of aggregation. ## Sample aggregation -This section uses the OpenSearch Dashboards sample e-commerce data and web log data. To add the sample data, log in to OpenSearch Dashboards, choose **Home** and **Try our sample data**. For **Sample eCommerce orders** and **Sample web logs**, choose **Add data**. +This section uses the OpenSearch Dashboards sample ecommerce data and web log data. To add the sample data, log in to OpenSearch Dashboards, choose **Home**, and then choose **Try our sample data**. For **Sample eCommerce orders** and **Sample web logs**, choose **Add data**. ### avg @@ -129,7 +127,7 @@ There are three main types of aggregations: ## Nested aggregations -Aggregations within aggregations are called nested or sub aggregations. +Aggregations within aggregations are called nested or subaggregations. Metric aggregations produce simple results and can't contain nested aggregations. @@ -158,4 +156,4 @@ Bucket aggregations produce buckets of documents that you can nest in other aggr The inner `aggs` keyword begins a new nested aggregation. The syntax of the parent aggregation and the nested aggregation is the same. Nested aggregations run in the context of the preceding parent aggregations. -You can also pair your aggregations with search queries to narrow down things you’re trying to analyze before aggregating. If you don't add a query, OpenSearch implicitly uses the `match_all` query. +You can also pair your aggregations with search queries to narrow down things you’re trying to analyze before aggregating. If you don't add a query, Elasticsearch implicitly uses the `match_all` query. From 59d63291b9387289da705d8a9fcedf547e8fd771 Mon Sep 17 00:00:00 2001 From: ashwinkumar12345 Date: Sun, 6 Jun 2021 00:27:01 -0700 Subject: [PATCH 15/32] minor fix --- docs/opensearch/aggregations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/opensearch/aggregations.md b/docs/opensearch/aggregations.md index 1f572402..3310a931 100644 --- a/docs/opensearch/aggregations.md +++ b/docs/opensearch/aggregations.md @@ -156,4 +156,4 @@ Bucket aggregations produce buckets of documents that you can nest in other aggr The inner `aggs` keyword begins a new nested aggregation. The syntax of the parent aggregation and the nested aggregation is the same. Nested aggregations run in the context of the preceding parent aggregations. -You can also pair your aggregations with search queries to narrow down things you’re trying to analyze before aggregating. If you don't add a query, Elasticsearch implicitly uses the `match_all` query. +You can also pair your aggregations with search queries to narrow down things you’re trying to analyze before aggregating. If you don't add a query, OpenSearch implicitly uses the `match_all` query. From 157114350020d559ea09f508ef9d609ff97204d1 Mon Sep 17 00:00:00 2001 From: aetter Date: Sun, 6 Jun 2021 19:19:20 -0700 Subject: [PATCH 16/32] Minor changes for RC1 release --- _config.yml | 2 +- _layouts/default.html | 2 +- docs/opensearch-dashboards/install/plugins.md | 20 ++++++++++++++++-- docs/opensearch/install/plugins.md | 21 +++++++++++++++++-- 4 files changed, 39 insertions(+), 6 deletions(-) diff --git a/_config.yml b/_config.yml index b33172dd..1a22a3d1 100644 --- a/_config.yml +++ b/_config.yml @@ -20,7 +20,7 @@ baseurl: "" # the subpath of your site, e.g. /blog url: "https://docs-beta.opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com permalink: pretty -opensearch_version: 1.0.0-beta1 +opensearch_version: 1.0.0-rc1 opensearch_major_minor_version: 1.0 # Build settings diff --git a/_layouts/default.html b/_layouts/default.html index 14d95c9a..0c77dfcd 100755 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -142,7 +142,7 @@ layout: table_wrappers {% if site.heading_anchors != false %} {% include vendor/anchor_headings.html html=content beforeHeading="true" anchorBody="" anchorClass="anchor-heading" anchorAttrs="aria-labelledby=\"%html_id%\"" %} {% else %} -

Like OpenSearch itself, this documentation is a beta. It has content gaps and might contain bugs.

+

This documentation remains in a beta state. It has content gaps and might contain bugs.

{{ content }} {% endif %} diff --git a/docs/opensearch-dashboards/install/plugins.md b/docs/opensearch-dashboards/install/plugins.md index 1e568937..5255dd02 100644 --- a/docs/opensearch-dashboards/install/plugins.md +++ b/docs/opensearch-dashboards/install/plugins.md @@ -29,11 +29,25 @@ If you don't want to use the all-in-one installation options, you can install th + + 1.0.0-rc1 + + 
alertingDashboards          1.0.0.0-rc1
+anomalyDetectionDashboards  1.0.0.0-rc1
+ganttChartDashboards        1.0.0.0-rc1
+indexManagementDashboards   1.0.0.0-rc1
+notebooksDashboards         1.0.0.0-rc1
+queryWorkbenchDashboards    1.0.0.0-rc1
+reportsDashboards           1.0.0.0-rc1
+securityDashboards          1.0.0.0-rc1
+traceAnalyticsDashboards    1.0.0.0-rc1
+
+ + 1.0.0-beta1 - 
-alertingDashboards          1.0.0.0-beta1
+      
alertingDashboards          1.0.0.0-beta1
 anomalyDetectionDashboards  1.0.0.0-beta1
 ganttChartDashboards        1.0.0.0-beta1
 indexManagementDashboards   1.0.0.0-beta1
@@ -60,6 +74,7 @@ traceAnalyticsDashboards    1.0.0.0-beta1
 
 Navigate to the OpenSearch Dashboards home directory (likely `/usr/share/opensearch-dashboards`) and run the install command for each plugin.
 
+{% comment %}
 
 #### Security OpenSearch Dashboards
 
@@ -146,6 +161,7 @@ sudo bin/opensearch-dashboards-plugin install https://d3g5vo6xdbdb9a.cloudfront.
 
 This plugin adds a new Gantt chart visualization.
 
+{% endcomment %}
 
 ## List installed plugins
 
diff --git a/docs/opensearch/install/plugins.md b/docs/opensearch/install/plugins.md
index ca5d451d..ef9a845d 100644
--- a/docs/opensearch/install/plugins.md
+++ b/docs/opensearch/install/plugins.md
@@ -30,6 +30,23 @@ If you don't want to use the all-in-one OpenSearch installation options, you can
     
   
   
+  
+    1.0.0-rc1
+    
+      
opensearch-alerting             1.0.0.0-rc1
+opensearch-anomaly-detection    1.0.0.0-rc1
+opensearch-asynchronous-search  1.0.0.0-rc1
+opensearch-index-management     1.0.0.0-rc1
+opensearch-job-scheduler        1.0.0.0-rc1
+opensearch-knn                  1.0.0.0-rc1
+opensearch-notebooks            1.0.0.0-rc1
+opensearch-performance-analyzer 1.0.0.0-rc1
+opensearch-reports-scheduler    1.0.0.0-rc1
+opensearch-security             1.0.0.0-rc1
+opensearch-sql                  1.0.0.0-rc1
+

+    
+  
   
     1.0.0-beta1
     
@@ -65,7 +82,7 @@ Then you can specify the version that you need:
 sudo yum install opensearch-oss-6.7.1
 ```
 
-{% endcomment %}
+
 
 
 ## Install plugins
@@ -229,7 +246,7 @@ Performance Analyzer requires some manual configuration after installing the plu
    ```bash
    curl -XGET "localhost:9600/_opensearch/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all"
    ```
-
+{% endcomment %}
 
 ## List installed plugins
 

From 2236490581a034a5ac4909cf3c41fc0d88dd1720 Mon Sep 17 00:00:00 2001
From: aetter 
Date: Mon, 7 Jun 2021 08:36:50 -0700
Subject: [PATCH 17/32] Update version-history.md

---
 version-history.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/version-history.md b/version-history.md
index 211cb58c..54aa2de0 100644
--- a/version-history.md
+++ b/version-history.md
@@ -9,4 +9,5 @@ permalink: /version-history/
 
 OpenSearch version | Release highlights | Release date
 :--- | :--- | :--- | :---
+[1.0.0-rc1](https://github.com/opensearch-project/opensearch-build/tree/main/release-notes/opensearch-release-notes-1.0.0-rc1.md) | First release candidate. | 7 June 2021
 [1.0.0-beta1](https://github.com/opensearch-project/opensearch-build/tree/main/release-notes/opensearch-release-notes-1.0.0-beta1.md) | Initial beta release. Refactors plugins to work with OpenSearch. | 13 May 2021

From 937b0e06b5b174a17e5b651244dfa516ddb72f34 Mon Sep 17 00:00:00 2001
From: aetter 
Date: Mon, 7 Jun 2021 09:37:35 -0700
Subject: [PATCH 18/32] Update index.md

---
 docs/agents-and-ingestion-tools/index.md | 78 +++++++++++-------------
 1 file changed, 37 insertions(+), 41 deletions(-)

diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md
index cac4bb97..c41b475d 100644
--- a/docs/agents-and-ingestion-tools/index.md
+++ b/docs/agents-and-ingestion-tools/index.md
@@ -1,63 +1,59 @@
 ---
 layout: default
-title: Agents and Ingestion Tools
+title: Agents and ingestion tools
 nav_order: 100
 has_children: false
 has_toc: false
 ---
 
-# Agents and Ingestion Tools
+# Agents and ingestion tools
 
-Historically, there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier, a [version value](https://github.com/opensearch-project/OpenSearch/issues/693) is being added to the OpenSearch YML. This will let you set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. By reporting 7.10.2, the cluster will be able to connect with tools that do version checking. This is intended to be an intermediate solution while OpenSearch support is added to more tools.
+Historically, many multiple popular agents and ingestion tools have worked with Elasticsearch OSS, such as Beats, Logstash, Fluentd, FluentBit, and OpenTelemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility.
+
+As an intermediate solution, we are adding a [version value](https://github.com/opensearch-project/OpenSearch/issues/693) to `opensearch.yml`. This change will let you set OpenSearch 1.x clusters to report version 7.10.2 (or any other arbitrary value). By reporting 7.10.2, the cluster will be able to connect with tools that check for a particular version number.
+
+For a longer term solution, we plan to create an OpenSearch output plugin for Logstash. This plugin *does not exist yet*, but we've included it in the compatibility matrices below based on its expected behavior.
 
 
 ## Compatibility Matrices
 
-So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) and add yourself to the "Credits and Thanks you" section at the bottom of the page. We greatly appreciate the help!** 
+*Italicized* cells are untested, but indicate what a value theoretically what it should be based on existing information.
 
-Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [proposed OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820).
 
 ### Compatibility Matrix for Logstash
 
-|	|Logstash OSS 7.x to 7.11.x	|Logstash OSS 7.12.x*	|Logstash 7.13.x without OpenSearch output plugin^	|Logstash 7.13.x with OpenSearch output plugin**^	|
-|---	|---	|---	|---	|---	|
-|Elasticsearch OSS v7.x to v7.9.x	|*Yes*	|*Yes*	|*No*	|*Yes*	|
-|Elasticsearch OSS v7.10.2^	|*Yes*	|*Yes*	|*No*	|*Yes*	|
-|ODFE OSS v1.x to 1.12	|*Yes*	|*Yes*	|*No*	|*Yes*	|
-|ODFE 1.13^	|*Yes*	|*Yes*	|*No*	|*Yes*	|
-|Amazon Elasticsearch Service 7.x to 7.9	|Yes	|Yes	|*No*	|*Yes*	|
-|Amazon Elasticsearch Service 7.10^	|Yes	|Yes	|*No*	|*Yes*	|
-|Amazon Elasticsearch Service 7.x to 7.9 with IAM	|[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html)	|[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html)	|No	|*Yes*	|
-|Amazon Elasticsearch Service 7.10 with IAM^	|[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html)	|[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html)	|No	|*Yes*	|
-|OpenSearch 1.0^	|[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693)	|[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693)	|*No*	|*Yes*	|
+| | Logstash OSS 7.x to 7.11.x | Logstash OSS 7.12.x\* | Logstash 7.13.x without OpenSearch output plugin | Logstash 7.13.x with OpenSearch output plugin\*\* |
+| :---| :--- | :--- | :--- | :--- |
+| Elasticsearch OSS v7.x to v7.9.x | *Yes* | *Yes* | *No* | *Yes* |
+| Elasticsearch OSS v7.10.2 | *Yes* | *Yes* | *No* | *Yes* |
+| ODFE OSS v1.x to 1.12 | *Yes* | *Yes* | *No* | *Yes* |
+| ODFE 1.13 | *Yes* | *Yes* | *No* | *Yes* |
+| Amazon Elasticsearch Service 7.x to 7.9 | Yes | Yes | *No* | *Yes* |
+| Amazon Elasticsearch Service 7.10 | Yes | Yes | *No* | *Yes* |
+| Amazon Elasticsearch Service 7.x to 7.9 with IAM | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | No | *Yes* |
+| Amazon Elasticsearch Service 7.10 with IAM | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | No | *Yes* |
+| OpenSearch 1.0 | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | *No* | *Yes* |
+
+\* Most current compatible version with Elasticsearch OSS.
+
+\*\* Planning to build.
 
-\*Most current compatible version with Elasticsearch OSS\
-\*\*Planning to build\
-^ Most current version of software\
-*Italicized cells are based on documentation or release notes but are still pending testing to validate*
 
 ### Compatibility Matrix for Beats
 
-|	|Beats OSS 7.x to 7.11.x*	|Beats OSS 7.12.x**	|Beats 7.13.x^	|
-|---	|---	|---	|---	|
-|Elasticsearch OSS v7.x to v7.9.x	|*Yes*	|*Yes*	|No	|
-|Elasticsearch OSS v7.10.2^	|*Yes*	|*Yes*	|No	|
-|ODFE OSS v1.x to 1.12	|*Yes*	|*Yes*	|No	|
-|ODFE 1.13^	|*Yes*	|*Yes*	|No	|
-|Amazon Elasticsearch Service 7.x to 7.9	|*Yes*	|*Yes*	|No	|
-|Amazon Elasticsearch Service 7.10^	|*Yes*	|*Yes*	|No	|
-|OpenSearch 1.0^	|[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693)	|[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693)	|No	|
-|Logstash OSS 7.x to 7.11.x*	|*Yes*	|*Yes*	|*Yes*	|
-|Logstash OSS 7.12.x**	|*Yes*	|*Yes*	|*Yes*	|
-|Logstash 7.13.x with OpenSearch output plugin	|*Yes*	|*Yes*	|*Yes*	|
+| | Beats OSS 7.x to 7.11.x\*\* | Beats OSS 7.12.x\* | Beats 7.13.x |
+| :--- | :--- | :--- | :--- |
+| Elasticsearch OSS v7.x to v7.9.x | *Yes* | *Yes* | No |
+| Elasticsearch OSS v7.10.2 | *Yes* | *Yes* | No |
+| ODFE OSS v1.x to 1.12 | *Yes* | *Yes* | No |
+| ODFE 1.13 | *Yes* | *Yes* | No |
+| Amazon Elasticsearch Service 7.x to 7.9 | *Yes* | *Yes* | No |
+| Amazon Elasticsearch Service 7.10 | *Yes* | *Yes* | No |
+| OpenSearch 1.0 | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | No |
+| Logstash OSS 7.x to 7.11.x* | *Yes* | *Yes* | *Yes* |
+| Logstash OSS 7.12.x** | *Yes* | *Yes* | *Yes* |
+| Logstash 7.13.x with OpenSearch output plugin | *Yes* | *Yes* | *Yes* |
 
-\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\
-\*\*Most current compatible version with Elasticsearch OSS\
-^ Most current version of software\
-*Italicized cells are based on documentation or release notes but are still pending testing to validate*
+\* Most current compatible version with Elasticsearch OSS.
 
-### Credits and Thank You for Contributing and Testing
-
-* [VijayanB](https://github.com/VijayanB)
-* [vamshin](https://github.com/vamshin)
-* [anirudha](https://github.com/anirudha)
+\*\* Beats OSS includes all Apache 2.0 Beats agents (i.e. Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat).

From ffc720bfe98bfdf48737475a7e186337a1b66e1a Mon Sep 17 00:00:00 2001
From: aetter 
Date: Mon, 7 Jun 2021 09:43:35 -0700
Subject: [PATCH 19/32] Update index.md

---
 docs/agents-and-ingestion-tools/index.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md
index c41b475d..5ac789f8 100644
--- a/docs/agents-and-ingestion-tools/index.md
+++ b/docs/agents-and-ingestion-tools/index.md
@@ -50,8 +50,8 @@ For a longer term solution, we plan to create an OpenSearch output plugin for Lo
 | Amazon Elasticsearch Service 7.x to 7.9 | *Yes* | *Yes* | No |
 | Amazon Elasticsearch Service 7.10 | *Yes* | *Yes* | No |
 | OpenSearch 1.0 | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | No |
-| Logstash OSS 7.x to 7.11.x* | *Yes* | *Yes* | *Yes* |
-| Logstash OSS 7.12.x** | *Yes* | *Yes* | *Yes* |
+| Logstash OSS 7.x to 7.11.x | *Yes* | *Yes* | *Yes* |
+| Logstash OSS 7.12.x\* | *Yes* | *Yes* | *Yes* |
 | Logstash 7.13.x with OpenSearch output plugin | *Yes* | *Yes* | *Yes* |
 
 \* Most current compatible version with Elasticsearch OSS.

From b4b215c219055a9be01209632f34c67451a38ce8 Mon Sep 17 00:00:00 2001
From: Liz Snyder 
Date: Mon, 7 Jun 2021 10:48:43 -0700
Subject: [PATCH 20/32] Add cluster health to API reference

---
 docs/opensearch/rest-api/cluster-health.md | 71 ++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 docs/opensearch/rest-api/cluster-health.md

diff --git a/docs/opensearch/rest-api/cluster-health.md b/docs/opensearch/rest-api/cluster-health.md
new file mode 100644
index 00000000..244ad9a7
--- /dev/null
+++ b/docs/opensearch/rest-api/cluster-health.md
@@ -0,0 +1,71 @@
+---
+layout: default
+title: Cluster health
+parent: REST API reference
+grand_parent: OpenSearch
+nav_order: 45
+---
+
+# Cluster health
+
+The most basic cluster health request returns a simple status of the health of your cluster. OpenSearch expresses cluster health in three colors: green, yellow, and red. A green status means all primary shards and their replicas are allocated to nodes. A yellow status means all primary shards are allocated to nodes, but some replicas aren't. A red status means at least one primary shard is not allocated to any node.
+
+To get the status of a specific index, provide the index name.
+
+## Example
+
+This request waits 50 seconds for the cluster to reach the yellow status or better:
+
+```
+GET /_cluster/health?wait_for_status=yellow&timeout=50s
+```
+
+If the cluster health becomes yellow or green before 50 seconds elapse, it returns a response immediately. Otherwise it returns a response as soon as it exceeds the timeout.
+
+## Path and HTTP methods
+
+```
+GET /_cluster/health
+GET /_cluster/health/
+```
+
+## URL parameters
+
+All cluster health parameters are optional.
+
+Parameter | Type | Description
+:--- | :--- | :---
+expand_wildcards | enum | Expands wildcard expressions to concrete indices. Combine multiple values with commas. Supported values are `all`, `open`, `closed`, `hidden`, and `none`. Default is `open`. 
+level | enum | The level of detail for returned health information. Supported values are `cluster`, `indices`, and `shards`. Default is `cluster`.
+local | boolean | Whether to return information from the local node only instead of from the master node. Default is false.
+master_timeout | time | The amount of time to wait for a connection to the master node. Default is 30 seconds.
+timeout | time | The amount of time to wait for a response. If the timeout expires, the request fails. Default is 30 seconds.
+wait_for_active_shards | string | Wait until the specified number of shards is active before returning a response. `all` for all shards. Default is `0`. 
+wait_for_nodes | string | Wait until the specified number of nodes is available. Also supports operators <=, >=, <, and >. 
+wait_for_events | enum | Wait until all currently queued events with the given priority are processed. Supported values are `immediate`, `urgent`, `high`, `normal`, `low`, and `languid`.
+wait_for_no_relocating_shards | boolean | Whether to wait until there are no relocating shards in the cluster. Default is false.
+wait_for_no_initializing_shards | boolean | Whether to wait until there are no initializing shards in the cluster. Default is false.
+wait_for_status | enum | Wait until the cluster is in a specific state or better. Supported values are `green`, `yellow`, and `red`.
+
+
+## Response
+
+```json
+{
+  "cluster_name" : "opensearch-cluster",
+  "status" : "green",
+  "timed_out" : false,
+  "number_of_nodes" : 2,
+  "number_of_data_nodes" : 2,
+  "active_primary_shards" : 6,
+  "active_shards" : 12,
+  "relocating_shards" : 0,
+  "initializing_shards" : 0,
+  "unassigned_shards" : 0,
+  "delayed_unassigned_shards" : 0,
+  "number_of_pending_tasks" : 0,
+  "number_of_in_flight_fetch" : 0,
+  "task_max_waiting_in_queue_millis" : 0,
+  "active_shards_percent_as_number" : 100.0
+}
+```

From ab2132111e864b440a63b03c33899544ad82efd7 Mon Sep 17 00:00:00 2001
From: keithhc2 
Date: Mon, 7 Jun 2021 17:28:59 -0700
Subject: [PATCH 21/32] Addressed some comments

---
 docs/im/index-transforms/index.md           | 105 ++++++++------------
 docs/im/index-transforms/transforms-apis.md |   2 +-
 2 files changed, 42 insertions(+), 65 deletions(-)

diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md
index 5b36fad8..5545d310 100644
--- a/docs/im/index-transforms/index.md
+++ b/docs/im/index-transforms/index.md
@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Index Transforms
+title: Index transforms
 nav_order: 40
 parent: Index management
 has_children: true
@@ -15,16 +15,16 @@ For example, suppose that you have airline data that’s scattered across multip
 
 You can use transform jobs in two ways:
 
-1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then, select the fields you want to transform, aggregations to use in the transformations, and define a schedule for your job to follow.
+1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
 2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
 
-OpenSearch Dashboards provides a nice view of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
+OpenSearch Dashboards provides a summarized view of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
 
 Your use cases will help you decide which method to use to create transform jobs.
 
 ## Create a transform job
 
-If you don't have any data in your cluster, you can use the sample data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**.
+If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**.
 
 ### Step 1: Choose indices
 
@@ -36,7 +36,7 @@ If you don't have any data in your cluster, you can use the sample data within O
 
 After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations.
 
-You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to see how many people logged in to a system separated into days of the week, you can group the `day_of_week` field into a target field of `day`, and your transformed index will sort your data into separate groups.
+You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes.
 
 On the other hand, aggregations let you perform simple calculations. For example, you can include an aggregation in your transform job to define a new field of `sum_of_total_ticket_price` that calculates the sum of all airplane tickets, and then analyze the newly summer data within your transformed index.
 
@@ -64,19 +64,19 @@ After confirming your transform job’s details are correct, choose **Create Tra
 Once the transform job finishes, you can use the `_search` API operation to search the target index.
 
 ```json
-GET target_index/_search
+GET /_search
 ```
 
-For example, after running a transform job that transforms the source index based on a `customer_gender field`, you can run the following request that returns all of the fields that have a value of `FEMALE`.
+For example, after running a transform job that transforms the flight data based on a `DestAirportID` field, you can run the following request that returns all of the fields that have a value of `SFO`.
 
 **Sample Request**
 
 ```json
-GET sample_target_index/_search
+GET finished_flight_job/_search
 {
   "query": {
     "match": {
-      "customer_gender_terms" : "FEMALE"
+      "DestAirportID_terms" : "SFO"
     }
   }
 }
@@ -86,7 +86,7 @@ GET sample_target_index/_search
 
 ```json
 {
-  "took" : 9,
+  "took" : 3,
   "timed_out" : false,
   "_shards" : {
     "total" : 5,
@@ -96,84 +96,61 @@ GET sample_target_index/_search
   },
   "hits" : {
     "total" : {
-      "value" : 6,
+      "value" : 4,
       "relation" : "eq"
     },
-    "max_score" : 0.9808291,
+    "max_score" : 3.845883,
     "hits" : [
       {
-        "_index" : "sample_target_index",
+        "_index" : "finished_flight_job",
         "_type" : "_doc",
-        "_id" : "w5oN7gnx5nw2PJaS6zDlCw",
-        "_score" : 0.9808291,
+        "_id" : "dSNKGb8U3OJOmC4RqVCi1Q",
+        "_score" : 3.845883,
         "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 23,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Men's Shoes"
+          "transform._id" : "sample_flight_job",
+          "transform._doc_count" : 14,
+          "Carrier_terms" : "Kibana Airlines",
+          "DestAirportID_terms" : "SFO"
         }
       },
       {
-        "_index" : "sample_target_index",
+        "_index" : "finished_flight_job",
         "_type" : "_doc",
-        "_id" : "r4xZ81gHD1hZG-5HJpcNaw",
-        "_score" : 0.9808291,
+        "_id" : "_D7oqOy7drx9E-MG96U5RA",
+        "_score" : 3.845883,
         "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 19,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Men's Accessories"
+          "transform._id" : "sample_flight_job",
+          "transform._doc_count" : 14,
+          "Carrier_terms" : "Logstash Airways",
+          "DestAirportID_terms" : "SFO"
         }
       },
       {
-        "_index" : "sample_target_index",
+        "_index" : "finished_flight_job",
         "_type" : "_doc",
-        "_id" : "sJLR4dGCws_jsUKjNj3u-Q",
-        "_score" : 0.2876821,
+        "_id" : "YuZ8tOt1OsBA54e84WuAEw",
+        "_score" : 3.6988301,
         "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 61,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Men's Clothing"
+          "transform._id" : "sample_flight_job",
+          "transform._doc_count" : 11,
+          "Carrier_terms" : "ES-Air",
+          "DestAirportID_terms" : "SFO"
         }
       },
       {
-        "_index" : "sample_target_index",
+        "_index" : "finished_flight_job",
         "_type" : "_doc",
-        "_id" : "nrkzdJOQU7p1BB4fPUT6Ag",
-        "_score" : 0.13353139,
+        "_id" : "W_-e7bVmH6eu8veJeK8ZxQ",
+        "_score" : 3.6988301,
         "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 730,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Women's Accessories"
-        }
-      },
-      {
-        "_index" : "sample_target_index",
-        "_type" : "_doc",
-        "_id" : "1raYCebejLhqCvA81fytpg",
-        "_score" : 0.13353139,
-        "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 1903,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Women's Clothing"
-        }
-      },
-      {
-        "_index" : "sample_target_index",
-        "_type" : "_doc",
-        "_id" : "X1fLVivgCgLKmctQid3L5A",
-        "_score" : 0.13353139,
-        "_source" : {
-          "transform._id" : "sample_target_index",
-          "transform._doc_count" : 1136,
-          "customer_gender_terms" : "FEMALE",
-          "category.keyword_terms" : "Women's Shoes"
+          "transform._id" : "sample_flight_job",
+          "transform._doc_count" : 10,
+          "Carrier_terms" : "JetBeats",
+          "DestAirportID_terms" : "SFO"
         }
       }
     ]
   }
 }
+
 ```
diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md
index eb77da6a..f7065f23 100644
--- a/docs/im/index-transforms/transforms-apis.md
+++ b/docs/im/index-transforms/transforms-apis.md
@@ -398,7 +398,7 @@ from | The starting index to search from. (Default: 0) | No
 size | Specifies the amount of results to return (Default: 10) | No
 search |The search term to use to filter results. | No
 sortField | The field to sort results with. | No
-sortDirection | Specifies the direction to sort results in. (Default: ASC) | No
+sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. (Default: ASC) | No
 
 For example, this request returns two results starting from the eighth index.
 

From ccbb4c8c4dcc728eff576f66aabb5ab9da468629 Mon Sep 17 00:00:00 2001
From: Keith Chan <12404772+keithhc2@users.noreply.github.com>
Date: Mon, 7 Jun 2021 22:53:13 -0700
Subject: [PATCH 22/32] Fixed some wording

---
 docs/im/index-transforms/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md
index 5545d310..f1f2ddb2 100644
--- a/docs/im/index-transforms/index.md
+++ b/docs/im/index-transforms/index.md
@@ -18,7 +18,7 @@ You can use transform jobs in two ways:
 1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
 2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
 
-OpenSearch Dashboards provides a summarized view of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
+OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
 
 Your use cases will help you decide which method to use to create transform jobs.
 

From cd6f4177c9e75ae980e951774127f1cd56e1b7f3 Mon Sep 17 00:00:00 2001
From: Eli Fisher 
Date: Tue, 8 Jun 2021 12:31:47 -0400
Subject: [PATCH 23/32] Removes Amazon ES from table

Removes Amazon ES from table so only OSS ES, ODFE, and OpenSearch are in the table.
---
 docs/agents-and-ingestion-tools/index.md | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md
index 5ac789f8..d627eb6b 100644
--- a/docs/agents-and-ingestion-tools/index.md
+++ b/docs/agents-and-ingestion-tools/index.md
@@ -28,10 +28,6 @@ For a longer term solution, we plan to create an OpenSearch output plugin for Lo
 | Elasticsearch OSS v7.10.2 | *Yes* | *Yes* | *No* | *Yes* |
 | ODFE OSS v1.x to 1.12 | *Yes* | *Yes* | *No* | *Yes* |
 | ODFE 1.13 | *Yes* | *Yes* | *No* | *Yes* |
-| Amazon Elasticsearch Service 7.x to 7.9 | Yes | Yes | *No* | *Yes* |
-| Amazon Elasticsearch Service 7.10 | Yes | Yes | *No* | *Yes* |
-| Amazon Elasticsearch Service 7.x to 7.9 with IAM | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | No | *Yes* |
-| Amazon Elasticsearch Service 7.10 with IAM | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | [Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) | No | *Yes* |
 | OpenSearch 1.0 | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | *No* | *Yes* |
 
 \* Most current compatible version with Elasticsearch OSS.
@@ -47,8 +43,6 @@ For a longer term solution, we plan to create an OpenSearch output plugin for Lo
 | Elasticsearch OSS v7.10.2 | *Yes* | *Yes* | No |
 | ODFE OSS v1.x to 1.12 | *Yes* | *Yes* | No |
 | ODFE 1.13 | *Yes* | *Yes* | No |
-| Amazon Elasticsearch Service 7.x to 7.9 | *Yes* | *Yes* | No |
-| Amazon Elasticsearch Service 7.10 | *Yes* | *Yes* | No |
 | OpenSearch 1.0 | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | [Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) | No |
 | Logstash OSS 7.x to 7.11.x | *Yes* | *Yes* | *Yes* |
 | Logstash OSS 7.12.x\* | *Yes* | *Yes* | *Yes* |

From d790eb6e14e6fd7db79cff59f08bb5572ff6a74c Mon Sep 17 00:00:00 2001
From: Keith Chan <12404772+keithhc2@users.noreply.github.com>
Date: Tue, 8 Jun 2021 14:22:35 -0700
Subject: [PATCH 24/32] Fixed parent metadata

---
 docs/im/index-transforms/transforms-apis.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md
index f7065f23..c15d8bd3 100644
--- a/docs/im/index-transforms/transforms-apis.md
+++ b/docs/im/index-transforms/transforms-apis.md
@@ -2,7 +2,7 @@
 layout: default
 title: Transforms APIs
 nav_order: 45
-parent: Index Transforms
+parent: Index transforms
 grand_parent: Index management
 has_toc: true
 ---

From 86202b8df008caa6982f078e2f5009a7229b1588 Mon Sep 17 00:00:00 2001
From: keithhc2 
Date: Tue, 8 Jun 2021 14:48:50 -0700
Subject: [PATCH 25/32] Fixed API endpoints

---
 docs/im/index-transforms/transforms-apis.md | 24 ++++++++++-----------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md
index f7065f23..d7b3b3ac 100644
--- a/docs/im/index-transforms/transforms-apis.md
+++ b/docs/im/index-transforms/transforms-apis.md
@@ -2,7 +2,7 @@
 layout: default
 title: Transforms APIs
 nav_order: 45
-parent: Index Transforms
+parent: Index transforms
 grand_parent: Index management
 has_toc: true
 ---
@@ -22,7 +22,7 @@ Creates a transform job.
 **Sample Request**
 
 ```json
-PUT _opendistro/_transform/
+PUT _plugins/_transform/
 
 {
   "transform": {
@@ -146,7 +146,7 @@ Updates a transform job if `transform_id` already exists.
 **Sample Request**
 
 ```json
-PUT _opendistro/_transform/
+PUT _plugins/_transform/
 
 {
   "transform": {
@@ -261,7 +261,7 @@ Returns a transform job's details.
 **Sample Request**
 
 ```json
-GET _opendistro/_transform/
+GET _plugins/_transform/
 ```
 
 **Sample Response**
@@ -326,7 +326,7 @@ You can also get details of all transform jobs by omitting `transform_id`.
 **Sample Request**
 
 ```json
-GET _opendistro/_transform/
+GET _plugins/_transform/
 ```
 
 **Sample Response**
@@ -405,7 +405,7 @@ For example, this request returns two results starting from the eighth index.
 **Sample Request**
 
 ```json
-GET /_opendistro/_transform/?size=2&from=8
+GET _plugins/_transform?size=2&from=8
 ```
 
 **Sample Response**
@@ -527,7 +527,7 @@ Transform jobs created using the API are automatically enabled, but if you ever
 **Sample Request**
 
 ```json
-POST _opendistro//_start
+POST _plugins/_transform//_start
 ```
 
 **Sample Response**
@@ -545,7 +545,7 @@ Stops/disables a transform job.
 **Sample Request**
 
 ```json
-POST _opendistro//_stop
+POST _plugins/_transform//_stop
 ```
 
 **Sample Response**
@@ -563,7 +563,7 @@ Returns the status and metadata of a transform job.
 **Sample Request**
 
 ```json
-GET _opendistro//_explain
+GET _plugins/_transform//_explain
 ```
 
 **Sample Response**
@@ -596,7 +596,7 @@ Returns a preview of what a transformed index would look like.
 **Sample Request**
 
 ```json
-POST _opendistro/_transform/_preview
+POST _plugins/_transform/_preview
 
 {
   "transform": {
@@ -681,7 +681,7 @@ Deletes a transform job. This operation does not delete the source or target ind
 **Sample Request**
 
 ```json
-DELETE _opendistro/_transform/
+DELETE _plugins/_transform/
 ```
 
 **Sample Response**
@@ -693,7 +693,7 @@ DELETE _opendistro/_transform/
   "items": [
     {
       "delete": {
-        "_index": ".opendistro-ism-config",
+        "_index": ".opensearch-ism-config",
         "_type": "_doc",
         "_id": "sample",
         "_version": 4,

From bdaf73283428f4201a83a6ff024bb87057861973 Mon Sep 17 00:00:00 2001
From: Keith Chan <12404772+keithhc2@users.noreply.github.com>
Date: Tue, 8 Jun 2021 14:50:25 -0700
Subject: [PATCH 26/32] Fixed a typo

---
 docs/im/index-transforms/transforms-apis.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/im/index-transforms/transforms-apis.md b/docs/im/index-transforms/transforms-apis.md
index d7b3b3ac..074d92e4 100644
--- a/docs/im/index-transforms/transforms-apis.md
+++ b/docs/im/index-transforms/transforms-apis.md
@@ -326,7 +326,7 @@ You can also get details of all transform jobs by omitting `transform_id`.
 **Sample Request**
 
 ```json
-GET _plugins/_transform/
+GET _plugins/_transform/
 ```
 
 **Sample Response**

From c4cf6838bf8bfd6264a914d07fa71cb8a6c0a855 Mon Sep 17 00:00:00 2001
From: aetter 
Date: Tue, 8 Jun 2021 15:35:12 -0700
Subject: [PATCH 27/32] Updates security settings

---
 docs/alerting/api.md                          |  2 +-
 docs/alerting/monitors.md                     |  4 +-
 docs/alerting/security.md                     |  2 +-
 docs/im/index-rollups/index.md                |  2 +-
 docs/im/index-rollups/rollup-api.md           |  4 +-
 docs/im/ism/api.md                            |  5 +-
 docs/opensearch/install/docker-security.md    | 48 ++++++------
 docs/opensearch/install/docker.md             |  6 +-
 docs/security/access-control/api.md           | 12 +--
 docs/security/access-control/field-masking.md |  4 +-
 docs/security/access-control/impersonation.md |  4 +-
 docs/security/access-control/multi-tenancy.md | 20 ++---
 docs/security/access-control/users-roles.md   |  4 +-
 docs/security/audit-logs/field-reference.md   |  2 +-
 docs/security/audit-logs/index.md             | 40 +++++-----
 docs/security/audit-logs/storage-types.md     | 52 ++++++-------
 docs/security/configuration/client-auth.md    |  2 +-
 docs/security/configuration/configuration.md  | 12 +--
 docs/security/configuration/disable.md        |  2 +-
 .../configuration/generate-certificates.md    |  6 +-
 docs/security/configuration/ldap.md           |  4 +-
 docs/security/configuration/openid-connect.md | 24 +++---
 docs/security/configuration/proxy.md          |  6 +-
 docs/security/configuration/saml.md           |  2 +-
 docs/security/configuration/security-admin.md |  2 +-
 docs/security/configuration/system-indices.md |  6 +-
 docs/security/configuration/tls.md            | 78 +++++++++----------
 docs/troubleshoot/openid-connect.md           | 14 ++--
 docs/troubleshoot/tls.md                      |  8 +-
 29 files changed, 188 insertions(+), 189 deletions(-)

diff --git a/docs/alerting/api.md b/docs/alerting/api.md
index eb0e8364..867ecf18 100644
--- a/docs/alerting/api.md
+++ b/docs/alerting/api.md
@@ -460,7 +460,7 @@ GET _plugins/_alerting//stats/
     "failed": 0
   },
   "cluster_name": "475300751431:alerting65-dont-delete",
-  "opensearch.scheduled_jobs.enabled": true,
+  "plugins.scheduled_jobs.enabled": true,
   "scheduled_job_index_exists": true,
   "scheduled_job_index_status": "green",
   "nodes_on_schedule": 9,
diff --git a/docs/alerting/monitors.md b/docs/alerting/monitors.md
index 66d483b8..0b24bf23 100644
--- a/docs/alerting/monitors.md
+++ b/docs/alerting/monitors.md
@@ -78,8 +78,8 @@ You can enter individual email addresses or an email group in the **Recipients**
 If your email provider requires SSL or TLS, you must authenticate each sender account before you can send an email. Enter these credentials in the OpenSearch keystore using the CLI. Run the following commands (in your OpenSearch directory) to enter your username and password. The `` is the name you entered for **Sender** earlier.
 
 ```bash
-./bin/opensearch-keystore add opendistro.alerting.destination.email..username
-./bin/opensearch-keystore add opendistro.alerting.destination.email..password
+./bin/opensearch-keystore add plugins.alerting.destination.email..username
+./bin/opensearch-keystore add plugins.alerting.destination.email..password
 ```
 
 **Note**: Keystore settings are node-specific. You must run these commands on each node.
diff --git a/docs/alerting/security.md b/docs/alerting/security.md
index b046ec31..3e04e321 100644
--- a/docs/alerting/security.md
+++ b/docs/alerting/security.md
@@ -44,7 +44,7 @@ Next, enable the following setting:
 PUT _cluster/settings
 {
   "transient": {
-    "opendistro.alerting.filter_by_backend_roles": "true"
+    "plugins.alerting.filter_by_backend_roles": "true"
   }
 }
 ```
diff --git a/docs/im/index-rollups/index.md b/docs/im/index-rollups/index.md
index e83fe5a4..d4545243 100644
--- a/docs/im/index-rollups/index.md
+++ b/docs/im/index-rollups/index.md
@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Index Rollups
+title: Index rollups
 nav_order: 35
 parent: Index management
 has_children: true
diff --git a/docs/im/index-rollups/rollup-api.md b/docs/im/index-rollups/rollup-api.md
index a13106cc..cd781d56 100644
--- a/docs/im/index-rollups/rollup-api.md
+++ b/docs/im/index-rollups/rollup-api.md
@@ -1,7 +1,7 @@
 ---
 layout: default
-title: Index Rollups API
-parent: Index Rollups
+title: Index rollups API
+parent: Index rollups
 grand_parent: Index management
 redirect_from: /docs/ism/rollup-api/
 nav_order: 9
diff --git a/docs/im/ism/api.md b/docs/im/ism/api.md
index cfab714f..71f31124 100644
--- a/docs/im/ism/api.md
+++ b/docs/im/ism/api.md
@@ -452,13 +452,12 @@ GET _plugins/_ism/explain/index_1
 ```json
 {
   "index_1": {
-    "index.opendistro.index_state_management.policy_id": "policy_1"
+    "index.plugins.index_state_management.policy_id": "policy_1"
   }
 }
 ```
 
-The `opendistro.index_state_management.policy_id` setting is deprecated starting from version 1.13.0.
-We retain this field in the response API for consistency. 
+The `plugins.index_state_management.policy_id` setting is deprecated starting from ODFE version 1.13.0. We retain this field in the response API for consistency. 
 
 ---
 
diff --git a/docs/opensearch/install/docker-security.md b/docs/opensearch/install/docker-security.md
index 88a3f18d..dd61e320 100644
--- a/docs/opensearch/install/docker-security.md
+++ b/docs/opensearch/install/docker-security.md
@@ -112,26 +112,26 @@ networks:
 Then make your changes to `opensearch.yml`. For a full list of settings, see [Security](../../../security/configuration/). This example adds (extremely) verbose audit logging:
 
 ```yml
-opensearch_security.ssl.transport.pemcert_filepath: node.pem
-opensearch_security.ssl.transport.pemkey_filepath: node-key.pem
-opensearch_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
-opensearch_security.ssl.transport.enforce_hostname_verification: false
-opensearch_security.ssl.http.enabled: true
-opensearch_security.ssl.http.pemcert_filepath: node.pem
-opensearch_security.ssl.http.pemkey_filepath: node-key.pem
-opensearch_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
-opensearch_security.allow_default_init_securityindex: true
-opensearch_security.authcz.admin_dn:
+plugins.security.ssl.transport.pemcert_filepath: node.pem
+plugins.security.ssl.transport.pemkey_filepath: node-key.pem
+plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
+plugins.security.ssl.transport.enforce_hostname_verification: false
+plugins.security.ssl.http.enabled: true
+plugins.security.ssl.http.pemcert_filepath: node.pem
+plugins.security.ssl.http.pemkey_filepath: node-key.pem
+plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
+plugins.security.allow_default_init_securityindex: true
+plugins.security.authcz.admin_dn:
   - CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=N,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
-opensearch_security.audit.type: internal_opensearch
-opensearch_security.enable_snapshot_restore_privilege: true
-opensearch_security.check_snapshot_restore_write_privileges: true
-opensearch_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
+plugins.security.audit.type: internal_opensearch
+plugins.security.enable_snapshot_restore_privilege: true
+plugins.security.check_snapshot_restore_write_privileges: true
+plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
 cluster.routing.allocation.disk.threshold_enabled: false
-opensearch_security.audit.config.disabled_rest_categories: NONE
-opensearch_security.audit.config.disabled_transport_categories: NONE
+plugins.security.audit.config.disabled_rest_categories: NONE
+plugins.security.audit.config.disabled_transport_categories: NONE
 ```
 
 Use this same override process to specify new [authentication settings](../../../security/configuration/configuration/) in `/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml`, as well as new default [internal users, roles, mappings, action groups, and tenants](../../../security/configuration/yaml/).
@@ -166,13 +166,13 @@ volumes:
 Remember that the certificates you specify in your Docker Compose file must be the same as the certificates listed in your custom `opensearch.yml` file. At a minimum, you should replace the root, admin, and node certificates with your own. For more information about adding and using certificates, see [Configure TLS certificates](../security/configuration/tls.md).
 
 ```yml
-opensearch_security.ssl.transport.pemcert_filepath: new-node-cert.pem
-opensearch_security.ssl.transport.pemkey_filepath: new-node-cert-key.pem
-opensearch_security.ssl.transport.pemtrustedcas_filepath: new-root-ca.pem
-opensearch_security.ssl.http.pemcert_filepath: new-node-cert.pem
-opensearch_security.ssl.http.pemkey_filepath: new-node-cert-key.pem
-opensearch_security.ssl.http.pemtrustedcas_filepath: new-root-ca.pem
-opensearch_security.authcz.admin_dn:
+plugins.security.ssl.transport.pemcert_filepath: new-node-cert.pem
+plugins.security.ssl.transport.pemkey_filepath: new-node-cert-key.pem
+plugins.security.ssl.transport.pemtrustedcas_filepath: new-root-ca.pem
+plugins.security.ssl.http.pemcert_filepath: new-node-cert.pem
+plugins.security.ssl.http.pemkey_filepath: new-node-cert-key.pem
+plugins.security.ssl.http.pemtrustedcas_filepath: new-root-ca.pem
+plugins.security.authcz.admin_dn:
   - CN=admin,OU=SSL,O=Test,L=Test,C=DE
 ```
 
diff --git a/docs/opensearch/install/docker.md b/docs/opensearch/install/docker.md
index effdf768..38c511d0 100644
--- a/docs/opensearch/install/docker.md
+++ b/docs/opensearch/install/docker.md
@@ -299,7 +299,7 @@ docker build --tag=opensearch-custom-plugin .
 docker run -p 9200:9200 -p 9600:9600 -v /usr/share/opensearch/data opensearch-custom-plugin
 ```
 
-You can also use a `Dockerfile` to pass your own certificates for use with the [Security](../../../security/) plugin, similar to the `-v` argument in [Configure OpenSearch](#configure-opensearch):
+You can also use a `Dockerfile` to pass your own certificates for use with the [security](../../../security/) plugin, similar to the `-v` argument in [Configure OpenSearch](#configure-opensearch):
 
 ```
 FROM opensearchproject/opensearch:{{site.opensearch_version}}
@@ -313,11 +313,11 @@ Alternately, you might want to remove a plugin. This `Dockerfile` removes the se
 
 ```
 FROM opensearchproject/opensearch:{{site.opensearch_version}}
-RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch_security
+RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security
 COPY --chown=opensearch:opensearch opensearch.yml /usr/share/opensearch/config/
 ```
 
-In this case, `opensearch.yml` is a "vanilla" version of the file with no OpenSearch entries. It might look like this:
+In this case, `opensearch.yml` is a "vanilla" version of the file with no plugin entries. It might look like this:
 
 ```yml
 cluster.name: "docker-cluster"
diff --git a/docs/security/access-control/api.md b/docs/security/access-control/api.md
index 91402708..4a1314c5 100644
--- a/docs/security/access-control/api.md
+++ b/docs/security/access-control/api.md
@@ -24,13 +24,13 @@ The security plugin REST API lets you programmatically create and manage users,
 Just like OpenSearch permissions, you control access to the security plugin REST API using roles. Specify roles in `opensearch.yml`:
 
 ```yml
-opensearch_security.restapi.roles_enabled: ["", ...]
+plugins.security.restapi.roles_enabled: ["", ...]
 ```
 
 These roles can now access all APIs. To prevent access to certain APIs:
 
 ```yml
-opensearch_security.restapi.endpoints_disabled..: ["", ...]
+plugins.security.restapi.endpoints_disabled..: ["", ...]
 ```
 
 Possible values for `endpoint` are:
@@ -55,15 +55,15 @@ Possible values for `method` are:
 For example, the following configuration grants three roles access to the REST API, but then prevents `test-role` from making PUT, POST, DELETE, or PATCH requests to `_opensearch/_security/api/roles` or `_opensearch/_security/api/internalusers`:
 
 ```yml
-opensearch_security.restapi.roles_enabled: ["all_access", "security_rest_api_access", "test-role"]
-opensearch_security.restapi.endpoints_disabled.test-role.ROLES: ["PUT", "POST", "DELETE", "PATCH"]
-opensearch_security.restapi.endpoints_disabled.test-role.INTERNALUSERS: ["PUT", "POST", "DELETE", "PATCH"]
+plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access", "test-role"]
+plugins.security.restapi.endpoints_disabled.test-role.ROLES: ["PUT", "POST", "DELETE", "PATCH"]
+plugins.security.restapi.endpoints_disabled.test-role.INTERNALUSERS: ["PUT", "POST", "DELETE", "PATCH"]
 ```
 
 To use the PUT and PATCH methods for the [configuration APIs](#configuration), add the following line to `opensearch.yml`:
 
 ```yml
-opensearch_security.unsupported.restapi.allow_securityconfig_modification: true
+plugins.security.unsupported.restapi.allow_securityconfig_modification: true
 ```
 
 
diff --git a/docs/security/access-control/field-masking.md b/docs/security/access-control/field-masking.md
index 84095953..a6e6c3dd 100644
--- a/docs/security/access-control/field-masking.md
+++ b/docs/security/access-control/field-masking.md
@@ -32,12 +32,12 @@ Field masking works alongside field-level security on the same per-role, per-ind
 You set the salt (a random string used to hash your data) in `opensearch.yml`:
 
 ```yml
-opensearch_security.compliance.salt: abcdefghijklmnopqrstuvqxyz1234567890
+plugins.security.compliance.salt: abcdefghijklmnopqrstuvqxyz1234567890
 ```
 
 Property | Description
 :--- | :---
-`opensearch_security.compliance.salt` | The salt to use when generating the hash value. Must be at least 32 characters. Only ASCII characters are allowed. Optional.
+`plugins.security.compliance.salt` | The salt to use when generating the hash value. Must be at least 32 characters. Only ASCII characters are allowed. Optional.
 
 Setting the salt is optional, but we highly recommend it.
 
diff --git a/docs/security/access-control/impersonation.md b/docs/security/access-control/impersonation.md
index 1fcfc70a..458ba9e6 100644
--- a/docs/security/access-control/impersonation.md
+++ b/docs/security/access-control/impersonation.md
@@ -20,7 +20,7 @@ Impersonation can occur on either the REST interface or at the transport layer.
 To allow one user to impersonate another, add the following to `opensearch.yml`:
 
 ```yml
-opensearch_security.authcz.rest_impersonation_user:
+plugins.security.authcz.rest_impersonation_user:
   :
     - 
     - 
@@ -34,7 +34,7 @@ The impersonated user field supports wildcards. Setting it to `*` allows `AUTHEN
 In a similar fashion, add the following to enable transport layer impersonation:
 
 ```yml
-opensearch_security.authcz.impersonation_dn:
+plugins.security.authcz.impersonation_dn:
   "CN=spock,OU=client,O=client,L=Test,C=DE":
     - worf
 ```
diff --git a/docs/security/access-control/multi-tenancy.md b/docs/security/access-control/multi-tenancy.md
index 4434535c..ff92c951 100644
--- a/docs/security/access-control/multi-tenancy.md
+++ b/docs/security/access-control/multi-tenancy.md
@@ -48,21 +48,21 @@ Setting | Description
 opensearch.username: kibanaserver
 opensearch.password: kibanaserver
 opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
-opensearch_security.multitenancy.enabled: true
-opensearch_security.multitenancy.tenants.enable_global: true
-opensearch_security.multitenancy.tenants.enable_private: true
-opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
-opensearch_security.multitenancy.enable_filter: false
+plugins.security.multitenancy.enabled: true
+plugins.security.multitenancy.tenants.enable_global: true
+plugins.security.multitenancy.tenants.enable_private: true
+plugins.security.multitenancy.tenants.preferred: ["Private", "Global"]
+plugins.security.multitenancy.enable_filter: false
 ```
 
 Setting | Description
 :--- | :---
 `opensearch.requestHeadersWhitelist` | OpenSearch Dashboards requires that you whitelist all HTTP headers that it passes to OpenSearch. Multi-tenancy uses a specific header, `securitytenant`, that must be present with the standard `Authorization` header. If the `securitytenant` header is not whitelisted, OpenSearch Dashboards starts with a red status.
-`opensearch_security.multitenancy.enabled` | Enables or disables multi-tenancy in OpenSearch Dashboards. Default is true.
-`opensearch_security.multitenancy.tenants.enable_global` | Enables or disables the global tenant. Default is true.
-`opensearch_security.multitenancy.tenants.enable_private` | Enables or disables the private tenant. Default is true.
-`opensearch_security.multitenancy.tenants.preferred` | Lets you change ordering in the **Tenants** tab of OpenSearch Dashboards. By default, the list starts with global and private (if enabled) and then proceeds alphabetically. You can add tenants here to move them to the top of the list.
-`opensearch_security.multitenancy.enable_filter` | If you have many tenants, you can add a search bar to the top of the list. Default is false.
+`plugins.security.multitenancy.enabled` | Enables or disables multi-tenancy in OpenSearch Dashboards. Default is true.
+`plugins.security.multitenancy.tenants.enable_global` | Enables or disables the global tenant. Default is true.
+`plugins.security.multitenancy.tenants.enable_private` | Enables or disables the private tenant. Default is true.
+`plugins.security.multitenancy.tenants.preferred` | Lets you change ordering in the **Tenants** tab of OpenSearch Dashboards. By default, the list starts with global and private (if enabled) and then proceeds alphabetically. You can add tenants here to move them to the top of the list.
+`plugins.security.multitenancy.enable_filter` | If you have many tenants, you can add a search bar to the top of the list. Default is false.
 
 
 ## Add tenants
diff --git a/docs/security/access-control/users-roles.md b/docs/security/access-control/users-roles.md
index daab6bf2..a6288c14 100644
--- a/docs/security/access-control/users-roles.md
+++ b/docs/security/access-control/users-roles.md
@@ -110,13 +110,13 @@ Role | Description
 `anomaly_full_access` | Grants full permissions to all anomaly detection actions.
 `anomaly_read_access` | Grants permissions to view detectors, but not create, modify, or delete detectors.
 `all_access` | Grants full access to the cluster: all cluster-wide operations, write to all indices, write to all tenants.
-`kibana_read_only` | A special role that prevents users from making changes to visualizations, dashboards, and other OpenSearch Dashboards objects. See `opensearch_security.readonly_mode.roles` in `opensearch_dashboards.yml`. Pair with the `kibana_user` role.
+`kibana_read_only` | A special role that prevents users from making changes to visualizations, dashboards, and other OpenSearch Dashboards objects. See `plugins.security.readonly_mode.roles` in `opensearch_dashboards.yml`. Pair with the `kibana_user` role.
 `kibana_user` | Grants permissions to use OpenSearch Dashboards: cluster-wide searches, index monitoring, and write to various OpenSearch Dashboards indices.
 `logstash` | Grants permissions for Logstash to interact with the cluster: cluster-wide searches, cluster monitoring, and write to the various Logstash indices.
 `manage_snapshots` | Grants permissions to manage snapshot repositories, take snapshots, and restore snapshots.
 `readall` | Grants permissions for cluster-wide searches like `msearch` and search permissions for all indices.
 `readall_and_monitor` | Same as `readall`, but with added cluster monitoring permissions.
-`security_rest_api_access` | A special role that allows access to the REST API. See `opensearch_security.restapi.roles_enabled` in `opensearch.yml` and [Access control for the API](../api/#access-control-for-the-api).
+`security_rest_api_access` | A special role that allows access to the REST API. See `plugins.security.restapi.roles_enabled` in `opensearch.yml` and [Access control for the API](../api/#access-control-for-the-api).
 `reports_read_access` | Grants permissions to generate on-demand reports, download existing reports, and view report definitions, but not to create report definitions.
 `reports_instances_read_access` | Grants permissions to generate on-demand reports and download existing reports, but not to view or create report definitions.
 `reports_full_access` | Grants full permissions to reports.
diff --git a/docs/security/audit-logs/field-reference.md b/docs/security/audit-logs/field-reference.md
index 9694cf28..541f9fbd 100644
--- a/docs/security/audit-logs/field-reference.md
+++ b/docs/security/audit-logs/field-reference.md
@@ -18,7 +18,7 @@ The following attributes are logged for all event categories, independent of the
 Name | Description
 :--- | :---
 `audit_format_version` | The audit log message format version.
-`audit_category` | The audit log category, one of FAILED_LOGIN, MISSING_PRIVILEGES, BAD_HEADERS, SSL_EXCEPTION, opensearch_SECURITY_INDEX_ATTEMPT, AUTHENTICATED or GRANTED_PRIVILEGES.
+`audit_category` | The audit log category. FAILED_LOGIN, MISSING_PRIVILEGES, BAD_HEADERS, SSL_EXCEPTION, OPENSEARCH_SECURITY_INDEX_ATTEMPT, AUTHENTICATED, or GRANTED_PRIVILEGES.
 `audit_node_id ` | The ID of the node where the event was generated.
 `audit_node_name` | The name of the node where the event was generated.
 `audit_node_host_address` | The host address of the node where the event was generated.
diff --git a/docs/security/audit-logs/index.md b/docs/security/audit-logs/index.md
index b0d5c072..97b4948b 100644
--- a/docs/security/audit-logs/index.md
+++ b/docs/security/audit-logs/index.md
@@ -16,7 +16,7 @@ To enable audit logging:
 1. Add the following line to `opensearch.yml` on each node:
 
    ```yml
-   opensearch_security.audit.type: internal_opensearch
+   plugins.security.audit.type: internal_opensearch
    ```
 
    This setting stores audit logs on the current cluster. For other storage options, see [Audit Log Storage Types](storage-types/).
@@ -57,22 +57,22 @@ These default log settings work well for most use cases, but you can change sett
 To exclude categories, set:
 
 ```yml
-opensearch_security.audit.config.disabled_rest_categories: 
-opensearch_security.audit.config.disabled_transport_categories: 
+plugins.security.audit.config.disabled_rest_categories: 
+plugins.security.audit.config.disabled_transport_categories: 
 ```
 
 For example:
 
 ```yml
-opensearch_security.audit.config.disabled_rest_categories: AUTHENTICATED, opensearch_SECURITY_INDEX_ATTEMPT
-opensearch_security.audit.config.disabled_transport_categories: GRANTED_PRIVILEGES
+plugins.security.audit.config.disabled_rest_categories: AUTHENTICATED, opensearch_SECURITY_INDEX_ATTEMPT
+plugins.security.audit.config.disabled_transport_categories: GRANTED_PRIVILEGES
 ```
 
 If you want to log events in all categories, use `NONE`:
 
 ```yml
-opensearch_security.audit.config.disabled_rest_categories: NONE
-opensearch_security.audit.config.disabled_transport_categories: NONE
+plugins.security.audit.config.disabled_rest_categories: NONE
+plugins.security.audit.config.disabled_transport_categories: NONE
 ```
 
 
@@ -81,8 +81,8 @@ opensearch_security.audit.config.disabled_transport_categories: NONE
 By default, the security plugin logs events on both REST and the transport layer. You can disable either type:
 
 ```yml
-opensearch_security.audit.enable_rest: false
-opensearch_security.audit.enable_transport: false
+plugins.security.audit.enable_rest: false
+plugins.security.audit.enable_transport: false
 ```
 
 
@@ -91,7 +91,7 @@ opensearch_security.audit.enable_transport: false
 By default, the security plugin includes the body of the request (if available) for both REST and the transport layer. If you do not want or need the request body, you can disable it:
 
 ```yml
-opensearch_security.audit.log_request_body: false
+plugins.security.audit.log_request_body: false
 ```
 
 
@@ -113,10 +113,10 @@ audit_trace_resolved_indices: [
 You can disable this feature by setting:
 
 ```yml
-opensearch_security.audit.resolve_indices: false
+plugins.security.audit.resolve_indices: false
 ```
 
-Disabling this feature only takes effect if `opensearch_security.audit.log_request_body` is also set to `false`.
+Disabling this feature only takes effect if `plugins.security.audit.log_request_body` is also set to `false`.
 {: .note }
 
 
@@ -127,7 +127,7 @@ Bulk requests can contain many indexing operations. By default, the security plu
 The security plugin can be configured to log each indexing operation as a separate event:
 
 ```yml
-opensearch_security.audit.resolve_bulk_requests: true
+plugins.security.audit.resolve_bulk_requests: true
 ```
 
 This change can create a massive number of events in the audit logs, so we don't recommend enabling this setting if you make heavy use of the `_bulk` API.
@@ -138,7 +138,7 @@ This change can create a massive number of events in the audit logs, so we don't
 You can exclude certain requests from being logged completely, by either configuring actions (for transport requests) and/or HTTP request paths (REST):
 
 ```yml
-opensearch_security.audit.ignore_requests: ["indices:data/read/*", "SearchRequest"]
+plugins.security.audit.ignore_requests: ["indices:data/read/*", "SearchRequest"]
 ```
 
 
@@ -147,7 +147,7 @@ opensearch_security.audit.ignore_requests: ["indices:data/read/*", "SearchReques
 By default, the security plugin logs events from all users, but excludes the internal OpenSearch Dashboards server user `kibanaserver`. You can exclude other users:
 
 ```yml
-opensearch_security.audit.ignore_users:
+plugins.security.audit.ignore_users:
   - kibanaserver
   - admin
 ```
@@ -155,7 +155,7 @@ opensearch_security.audit.ignore_users:
 If requests from all users should be logged, use `NONE`:
 
 ```yml
-opensearch_security.audit.ignore_users: NONE
+plugins.security.audit.ignore_users: NONE
 ```
 
 
@@ -164,13 +164,13 @@ opensearch_security.audit.ignore_users: NONE
 By default, the security plugin stores audit events in a daily rolling index named `auditlog-YYYY.MM.dd`. You can configure the name of the index in `opensearch.yml`:
 
 ```yml
-opensearch_security.audit.config.index: myauditlogindex
+plugins.security.audit.config.index: myauditlogindex
 ```
 
 Use a date pattern in the index name to configure daily, weekly, or monthly rolling indices:
 
 ```yml
-opensearch_security.audit.config.index: "'auditlog-'YYYY.MM.dd"
+plugins.security.audit.config.index: "'auditlog-'YYYY.MM.dd"
 ```
 
 For a reference on the date pattern format, see the [Joda DateTimeFormat documentation](http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html).
@@ -181,11 +181,11 @@ For a reference on the date pattern format, see the [Joda DateTimeFormat documen
 The Search plugin logs events asynchronously, which keeps performance impact on your cluster minimal. The plugin uses a fixed thread pool to log events. You can define the number of threads in the pool in `opensearch.yml`:
 
 ```yml
-opensearch_security.audit.threadpool.size: 
+plugins.security.audit.threadpool.size: 
 ```
 
 The default setting is `10`. Setting this value to `0` disables the thread pool, which means the plugin logs events synchronously. To set the maximum queue length per thread:
 
 ```yml
-opensearch_security.audit.threadpool.max_queue_len: 100000
+plugins.security.audit.threadpool.max_queue_len: 100000
 ```
diff --git a/docs/security/audit-logs/storage-types.md b/docs/security/audit-logs/storage-types.md
index 84fc3c32..b64adce8 100644
--- a/docs/security/audit-logs/storage-types.md
+++ b/docs/security/audit-logs/storage-types.md
@@ -21,7 +21,7 @@ log4j | Writes the events to a Log4j logger. You can use any Log4j [appender](ht
 You configure the output location in `opensearch.yml`:
 
 ```
-opensearch_security.audit.type: 
+plugins.security.audit.type: 
 ```
 
 `external_opensearch`, `webhook`, and `log4j` all have additional configuration options. Details follow.
@@ -32,16 +32,16 @@ opensearch_security.audit.type: ]
-opensearch_security.audit.config.index: 
-opensearch_security.audit.config.type: _doc
+plugins.security.audit.type: external_opensearch
+plugins.security.audit.config.http_endpoints: []
+plugins.security.audit.config.index: 
+plugins.security.audit.config.type: _doc
 ```
 
-The security plugin uses the OpenSearch REST API to send events, just like any other indexing request. For `opensearch_security.audit.config.http_endpoints`, use a comma-separated list of hosts/IP addresses and the REST port (default 9200).
+The security plugin uses the OpenSearch REST API to send events, just like any other indexing request. For `plugins.security.audit.config.http_endpoints`, use a comma-separated list of hosts/IP addresses and the REST port (default 9200).
 
 ```
-opensearch_security.audit.config.http_endpoints: [192.168.178.1:9200,192.168.178.2:9200]
+plugins.security.audit.config.http_endpoints: [192.168.178.1:9200,192.168.178.2:9200]
 ```
 
 If you use `external_opensearch` and the remote cluster also uses the security plugin, you must supply some additional parameters for authentication. These parameters depend on which authentication type you configured for the remote cluster.
@@ -51,16 +51,16 @@ If you use `external_opensearch` and the remote cluster also uses the security p
 
 Name | Data Type | Description
 :--- | :--- | :---
-`opensearch_security.audit.config.enable_ssl` | Boolean | If you enabled SSL/TLS on the receiving cluster, set to true. The default is false.
-`opensearch_security.audit.config.verify_hostnames` |  Boolean | Whether to verify the hostname of the SSL/TLS certificate of the receiving cluster. Default is true.
-`opensearch_security.audit.config.pemtrustedcas_filepath` | String | The trusted root certificate of the external OpenSearch cluster, relative to the `config` directory.
-`opensearch_security.audit.config.pemtrustedcas_content` | String | Instead of specifying the path (`opensearch_security.audit.config.pemtrustedcas_filepath`), you can configure the Base64-encoded certificate content directly.
-`opensearch_security.audit.config.enable_ssl_client_auth` | Boolean | Whether to enable SSL/TLS client authentication. If you set this to true, the audit log module sends the node's certificate along with the request. The receiving cluster can use this certificate to verify the identity of the caller.
-`opensearch_security.audit.config.pemcert_filepath` | String | The path to the TLS certificate to send to the external OpenSearch cluster, relative to the `config` directory.
-`opensearch_security.audit.config.pemcert_content` | String | Instead of specifying the path (`opensearch_security.audit.config.pemcert_filepath`), you can configure the Base64-encoded certificate content directly.
-`opensearch_security.audit.config.pemkey_filepath` | String | The path to the private key of the TLS certificate to send to the external OpenSearch cluster, relative to the `config` directory.
-`opensearch_security.audit.config.pemkey_content` | String | Instead of specifying the path (`opensearch_security.audit.config.pemkey_filepath`), you can configure the Base64-encoded certificate content directly.
-`opensearch_security.audit.config.pemkey_password` | String | The password of the private key.
+`plugins.security.audit.config.enable_ssl` | Boolean | If you enabled SSL/TLS on the receiving cluster, set to true. The default is false.
+`plugins.security.audit.config.verify_hostnames` |  Boolean | Whether to verify the hostname of the SSL/TLS certificate of the receiving cluster. Default is true.
+`plugins.security.audit.config.pemtrustedcas_filepath` | String | The trusted root certificate of the external OpenSearch cluster, relative to the `config` directory.
+`plugins.security.audit.config.pemtrustedcas_content` | String | Instead of specifying the path (`plugins.security.audit.config.pemtrustedcas_filepath`), you can configure the Base64-encoded certificate content directly.
+`plugins.security.audit.config.enable_ssl_client_auth` | Boolean | Whether to enable SSL/TLS client authentication. If you set this to true, the audit log module sends the node's certificate along with the request. The receiving cluster can use this certificate to verify the identity of the caller.
+`plugins.security.audit.config.pemcert_filepath` | String | The path to the TLS certificate to send to the external OpenSearch cluster, relative to the `config` directory.
+`plugins.security.audit.config.pemcert_content` | String | Instead of specifying the path (`plugins.security.audit.config.pemcert_filepath`), you can configure the Base64-encoded certificate content directly.
+`plugins.security.audit.config.pemkey_filepath` | String | The path to the private key of the TLS certificate to send to the external OpenSearch cluster, relative to the `config` directory.
+`plugins.security.audit.config.pemkey_content` | String | Instead of specifying the path (`plugins.security.audit.config.pemkey_filepath`), you can configure the Base64-encoded certificate content directly.
+`plugins.security.audit.config.pemkey_password` | String | The password of the private key.
 
 
 ### Basic auth settings
@@ -68,8 +68,8 @@ Name | Data Type | Description
 If you enabled HTTP basic authentication on the receiving cluster, use these settings to specify the username and password:
 
 ```yml
-opensearch_security.audit.config.username: 
-opensearch_security.audit.config.password: 
+plugins.security.audit.config.username: 
+plugins.security.audit.config.password: 
 ```
 
 
@@ -79,11 +79,11 @@ Use the following keys to configure the `webhook` storage type.
 
 Name | Data Type | Description
 :--- | :--- | :---
-`opensearch_security.audit.config.webhook.url` | String | The HTTP or HTTPS URL to send the logs to.
-`opensearch_security.audit.config.webhook.ssl.verify` | Boolean | If true, the TLS certificate provided by the endpoint (if any) will be verified. If set to false, no verification is performed. You can disable this check if you use self-signed certificates.
-`opensearch_security.audit.config.webhook.ssl.pemtrustedcas_filepath` | String | The path to the trusted certificate against which the webhook's TLS certificate is validated.
-`opensearch_security.audit.config.webhook.ssl.pemtrustedcas_content` | String | Same as `opensearch_security.audit.config.webhook.ssl.pemtrustedcas_content`, but you can configure the base 64 encoded certificate content directly.
-`opensearch_security.audit.config.webhook.format` | String | The format in which the audit log message is logged, can be one of `URL_PARAMETER_GET`, `URL_PARAMETER_POST`, `TEXT`, `JSON`, `SLACK`. See [Formats](#formats).
+`plugins.security.audit.config.webhook.url` | String | The HTTP or HTTPS URL to send the logs to.
+`plugins.security.audit.config.webhook.ssl.verify` | Boolean | If true, the TLS certificate provided by the endpoint (if any) will be verified. If set to false, no verification is performed. You can disable this check if you use self-signed certificates.
+`plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath` | String | The path to the trusted certificate against which the webhook's TLS certificate is validated.
+`plugins.security.audit.config.webhook.ssl.pemtrustedcas_content` | String | Same as `plugins.security.audit.config.webhook.ssl.pemtrustedcas_content`, but you can configure the base 64 encoded certificate content directly.
+`plugins.security.audit.config.webhook.format` | String | The format in which the audit log message is logged, can be one of `URL_PARAMETER_GET`, `URL_PARAMETER_POST`, `TEXT`, `JSON`, `SLACK`. See [Formats](#formats).
 
 
 ### Formats
@@ -102,8 +102,8 @@ Format | Description
 The `log4j` storage type lets you specify the name of the logger and log level.
 
 ```yml
-opensearch_security.audit.config.log4j.logger_name: audit
-opensearch_security.audit.config.log4j.level: INFO
+plugins.security.audit.config.log4j.logger_name: audit
+plugins.security.audit.config.log4j.level: INFO
 ```
 
 By default, the security plugin uses the logger name `audit` and logs the events on `INFO` level. Audit events are stored in JSON format.
diff --git a/docs/security/configuration/client-auth.md b/docs/security/configuration/client-auth.md
index 96229b18..85ec8b0b 100644
--- a/docs/security/configuration/client-auth.md
+++ b/docs/security/configuration/client-auth.md
@@ -19,7 +19,7 @@ Another benefit of client certificate authentication is you can use it along wit
 To enable client certificate authentication, you must first set `clientauth_mode` in `opensearch.yml` to either `OPTIONAL` or `REQUIRE`:
 
 ```yml
-opensearch_security.ssl.http.clientauth_mode: OPTIONAL
+plugins.security.ssl.http.clientauth_mode: OPTIONAL
 ```
 
 Next, enable client certificate authentication in the `client_auth_domain` section of `config.yml`.
diff --git a/docs/security/configuration/configuration.md b/docs/security/configuration/configuration.md
index 5cd2582d..debc6496 100755
--- a/docs/security/configuration/configuration.md
+++ b/docs/security/configuration/configuration.md
@@ -151,15 +151,15 @@ Due to the nature of Kerberos, you must define some settings in `opensearch.yml`
 In `opensearch.yml`, define the following:
 
 ```yml
-opensearch_security.kerberos.krb5_filepath: '/etc/krb5.conf'
-opensearch_security.kerberos.acceptor_keytab_filepath: 'eskeytab.tab'
+plugins.security.kerberos.krb5_filepath: '/etc/krb5.conf'
+plugins.security.kerberos.acceptor_keytab_filepath: 'eskeytab.tab'
 ```
 
-`opensearch_security.kerberos.krb5_filepath` defines the path to your Kerberos configuration file. This file contains various settings regarding your Kerberos installation, for example, the realm names, hostnames, and ports of the Kerberos key distribution center (KDC).
+`plugins.security.kerberos.krb5_filepath` defines the path to your Kerberos configuration file. This file contains various settings regarding your Kerberos installation, for example, the realm names, hostnames, and ports of the Kerberos key distribution center (KDC).
 
-`opensearch_security.kerberos.acceptor_keytab_filepath` defines the path to the keytab file, which contains the principal that the security plugin uses to issue requests against Kerberos.
+`plugins.security.kerberos.acceptor_keytab_filepath` defines the path to the keytab file, which contains the principal that the security plugin uses to issue requests against Kerberos.
 
-`opensearch_security.kerberos.acceptor_principal: 'HTTP/localhost'` defines the principal that the security plugin uses to issue requests against Kerberos. This value must be present in the keytab file.
+`plugins.security.kerberos.acceptor_principal: 'HTTP/localhost'` defines the principal that the security plugin uses to issue requests against Kerberos. This value must be present in the keytab file.
 
 Due to security restrictions, the keytab file must be placed in `config` or a subdirectory, and the path in `opensearch.yml` must be relative, not absolute.
 {: .warning }
@@ -273,7 +273,7 @@ eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb2dnZWRJbkFzIjoiYWRtaW4iLCJpYXQiOjE0MjI
 
 ### Configure JSON web tokens
 
-If JSON web tokens are the only authentication method that you use, disable the user cache by setting `opensearch_security.cache.ttl_minutes: 0`.
+If JSON web tokens are the only authentication method that you use, disable the user cache by setting `plugins.security.cache.ttl_minutes: 0`.
 {: .warning }
 
 Set up an authentication domain and choose `jwt` as the HTTP authentication type. Because the tokens already contain all required information to verify the request, `challenge` must be set to `false` and `authentication_backend` to `noop`.
diff --git a/docs/security/configuration/disable.md b/docs/security/configuration/disable.md
index 7278f48c..27397bd9 100755
--- a/docs/security/configuration/disable.md
+++ b/docs/security/configuration/disable.md
@@ -11,7 +11,7 @@ nav_order: 99
 You might want to temporarily disable the security plugin to make testing or internal usage more straightforward. To disable the plugin, add the following line in `opensearch.yml`:
 
 ```yml
-opensearch_security.disabled: true
+plugins.security.disabled: true
 ```
 
 A more permanent option is to remove the security plugin entirely. Delete the `plugins/opensearch-security` folder on all nodes, and delete the `opensearch_security` configuration entries from `opensearch.yml`.
diff --git a/docs/security/configuration/generate-certificates.md b/docs/security/configuration/generate-certificates.md
index cfee2c4a..c7b160d8 100755
--- a/docs/security/configuration/generate-certificates.md
+++ b/docs/security/configuration/generate-certificates.md
@@ -89,7 +89,7 @@ Just like the root certificate, use the `-days` option to specify an expiration
 
 Follow the steps in [Generate an admin certificate](#generate-an-admin-certificate) with new file names to generate a new certificate for each node and as many client certificates as you need. Each certificate should use its own private key.
 
-If you generate node certificates and have `opensearch_security.ssl.transport.enforce_hostname_verification` set to `true` (default), be sure to specify a common name (CN) for the certificate that matches the hostname of the intended node. If you want to use the same node certificate on all nodes (not recommended), set hostname verification to `false`. For more information, see [Configure TLS certificates](../tls/#advanced-hostname-verification-and-dns-lookup).
+If you generate node certificates and have `plugins.security.ssl.transport.enforce_hostname_verification` set to `true` (default), be sure to specify a common name (CN) for the certificate that matches the hostname of the intended node. If you want to use the same node certificate on all nodes (not recommended), set hostname verification to `false`. For more information, see [Configure TLS certificates](../tls/#advanced-hostname-verification-and-dns-lookup).
 
 
 ### Sample script
@@ -134,9 +134,9 @@ openssl req -new -key node-key.pem -subj "/C=CA/ST=ONTARIO/L=TORONTO/O=ORG/OU=UN
 If you created admin and node certificates, you must specify their distinguished names (DNs) in `opensearch.yml` on all nodes:
 
 ```yml
-opensearch_security.authcz.admin_dn:
+plugins.security.authcz.admin_dn:
   - 'CN=ADMIN,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=node1.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
   - 'CN=node2.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
 ```
diff --git a/docs/security/configuration/ldap.md b/docs/security/configuration/ldap.md
index c73a1c37..a71c22d7 100755
--- a/docs/security/configuration/ldap.md
+++ b/docs/security/configuration/ldap.md
@@ -154,8 +154,8 @@ Name | Description
 By default, the security plugin validates the TLS certificate of the LDAP servers against the root CA configured in `opensearch.yml`, either as a PEM certificate or a truststore:
 
 ```
-opensearch_security.ssl.transport.pemtrustedcas_filepath: ...
-opensearch_security.ssl.http.truststore_filepath: ...
+plugins.security.ssl.transport.pemtrustedcas_filepath: ...
+plugins.security.ssl.http.truststore_filepath: ...
 ```
 
 If your server uses a certificate signed by a different CA, import this CA into your truststore or add it to your trusted CA file on each node.
diff --git a/docs/security/configuration/openid-connect.md b/docs/security/configuration/openid-connect.md
index 83950966..ddb1dbcb 100755
--- a/docs/security/configuration/openid-connect.md
+++ b/docs/security/configuration/openid-connect.md
@@ -244,7 +244,7 @@ Name | Description
 Activate OpenID Connect by adding the following to `opensearch_dashboards.yml`:
 
 ```
-opensearch_security.auth.type: "openid"
+plugins.security.auth.type: "openid"
 ```
 
 
@@ -266,29 +266,29 @@ OpenID Connect providers usually publish their configuration in JSON format unde
 
 Name | Description
 :--- | :---
-`opensearch_security.openid.connect_url` | The URL where the IdP publishes the OpenID metadata. Required.
-`opensearch_security.openid.client_id` | The ID of the OpenID Connect client configured in your IdP. Required.
-`opensearch_security.openid.client_secret` | The client secret of the OpenID Connect client configured in your IdP. Required.
-`opensearch_security.openid.scope` | The [scope of the identity token](https://auth0.com/docs/scopes/current) issued by the IdP. Optional. Default is `openid profile email address phone`.
-`opensearch_security.openid.header` | HTTP header name of the JWT token. Optional. Default is `Authorization`.
-`opensearch_security.openid.logout_url` | The logout URL of your IdP. Optional. Only necessary if your IdP does not publish the logout URL in its metadata.
-`opensearch_security.openid.base_redirect_url` | The base of the redirect URL that will be sent to your IdP. Optional. Only necessary when OpenSearch Dashboards is behind a reverse proxy, in which case it should be different than `server.host` and `server.port` in `opensearch_dashboards.yml`.
+`plugins.security.openid.connect_url` | The URL where the IdP publishes the OpenID metadata. Required.
+`plugins.security.openid.client_id` | The ID of the OpenID Connect client configured in your IdP. Required.
+`plugins.security.openid.client_secret` | The client secret of the OpenID Connect client configured in your IdP. Required.
+`plugins.security.openid.scope` | The [scope of the identity token](https://auth0.com/docs/scopes/current) issued by the IdP. Optional. Default is `openid profile email address phone`.
+`plugins.security.openid.header` | HTTP header name of the JWT token. Optional. Default is `Authorization`.
+`plugins.security.openid.logout_url` | The logout URL of your IdP. Optional. Only necessary if your IdP does not publish the logout URL in its metadata.
+`plugins.security.openid.base_redirect_url` | The base of the redirect URL that will be sent to your IdP. Optional. Only necessary when OpenSearch Dashboards is behind a reverse proxy, in which case it should be different than `server.host` and `server.port` in `opensearch_dashboards.yml`.
 
 
 ### Configuration example
 
 ```yml
 # Enable OpenID authentication
-opensearch_security.auth.type: "openid"
+plugins.security.auth.type: "openid"
 
 # The IdP metadata endpoint
-opensearch_security.openid.connect_url: "http://keycloak.example.com:8080/auth/realms/master/.well-known/openid-configuration"
+plugins.security.openid.connect_url: "http://keycloak.example.com:8080/auth/realms/master/.well-known/openid-configuration"
 
 # The ID of the OpenID Connect client in your IdP
-opensearch_security.openid.client_id: "opensearch-dashboards-sso"
+plugins.security.openid.client_id: "opensearch-dashboards-sso"
 
 # The client secret of the OpenID Connect client
-opensearch_security.openid.client_secret: "a59c51f5-f052-4740-a3b0-e14ba355b520"
+plugins.security.openid.client_secret: "a59c51f5-f052-4740-a3b0-e14ba355b520"
 
 # Use HTTPS instead of HTTP
 opensearch.url: "https://.com:"
diff --git a/docs/security/configuration/proxy.md b/docs/security/configuration/proxy.md
index e3e6d296..2c4fdf91 100644
--- a/docs/security/configuration/proxy.md
+++ b/docs/security/configuration/proxy.md
@@ -202,7 +202,7 @@ opensearch.requestHeadersWhitelist: ["securitytenant","Authorization","x-forward
 You must also enable the authentication type in `opensearch_dashboards.yml`:
 
 ```yml
-opensearch_security.auth.type: "proxy"
-opensearch_security.proxycache.user_header: "x-proxy-user"
-opensearch_security.proxycache.roles_header: "x-proxy-roles"
+plugins.security.auth.type: "proxy"
+plugins.security.proxycache.user_header: "x-proxy-user"
+plugins.security.proxycache.roles_header: "x-proxy-roles"
 ```
diff --git a/docs/security/configuration/saml.md b/docs/security/configuration/saml.md
index e81d3545..a8c64797 100755
--- a/docs/security/configuration/saml.md
+++ b/docs/security/configuration/saml.md
@@ -302,7 +302,7 @@ authc:
 Because most of the SAML-specific configuration is done in the security plugin, just activate SAML in your `opensearch_dashboards.yml` by adding the following:
 
 ```
-opensearch_security.auth.type: "saml"
+plugins.security.auth.type: "saml"
 ```
 
 In addition, the OpenSearch Dashboards endpoint for validating the SAML assertions must be whitelisted:
diff --git a/docs/security/configuration/security-admin.md b/docs/security/configuration/security-admin.md
index d692ac0c..01caaa6a 100755
--- a/docs/security/configuration/security-admin.md
+++ b/docs/security/configuration/security-admin.md
@@ -20,7 +20,7 @@ After the `.opensearch_security` index is initialized, you can use OpenSearch Da
 You can configure all certificates that should have admin privileges in `opensearch.yml` by specifying respective distinguished names (DNs). If you use the demo certificates, for example, you can use the `kirk` certificate:
 
 ```yml
-opensearch_security.authcz.admin_dn:
+plugins.security.authcz.admin_dn:
   - CN=kirk,OU=client,O=client,L=test,C=DE
 ```
 
diff --git a/docs/security/configuration/system-indices.md b/docs/security/configuration/system-indices.md
index 95611552..d65190f7 100644
--- a/docs/security/configuration/system-indices.md
+++ b/docs/security/configuration/system-indices.md
@@ -13,8 +13,8 @@ By default, OpenSearch has a protected system index, `.opensearch_security`, whi
 You can add additional system indices in in `opensearch.yml`. In addition to automatically creating `.opensearch_security`, the demo configuration adds several indices for the various OpenSearch plugins that integrate with the security plugin:
 
 ```yml
-opendistro_security.system_indices.enabled: true
-opendistro_security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
+plugins.security.system_indices.enabled: true
+plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
 ```
 
 To access these indices, you must authenticate with an [admin certificate](../tls/#configure-admin-certificates):
@@ -23,4 +23,4 @@ To access these indices, you must authenticate with an [admin certificate](../tl
 curl -k --cert ./kirk.pem --key ./kirk-key.pem -XGET 'https://localhost:9200/.opensearch_security/_search'
 ```
 
-The alternative is to remove indices from the `opensearch_security.system_indices.indices` list on each node and restart OpenSearch.
+The alternative is to remove indices from the `plugins.security.system_indices.indices` list on each node and restart OpenSearch.
diff --git a/docs/security/configuration/tls.md b/docs/security/configuration/tls.md
index 36aca4a9..56205e2f 100755
--- a/docs/security/configuration/tls.md
+++ b/docs/security/configuration/tls.md
@@ -23,20 +23,20 @@ The following tables contain the settings you can use to configure the location
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.transport.pemkey_filepath` | Path to the certificate's key file (PKCS \#8), which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.transport.pemkey_password` | Key password. Omit this setting if the key has no password. Optional.
-`opensearch_security.ssl.transport.pemcert_filepath` | Path to the X.509 node certificate chain (PEM format), which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.transport.pemtrustedcas_filepath` | Path to the root CAs (PEM format), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.transport.pemkey_filepath` | Path to the certificate's key file (PKCS \#8), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.transport.pemkey_password` | Key password. Omit this setting if the key has no password. Optional.
+`plugins.security.ssl.transport.pemcert_filepath` | Path to the X.509 node certificate chain (PEM format), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.transport.pemtrustedcas_filepath` | Path to the root CAs (PEM format), which must be under the `config` directory, specified using a relative path. Required.
 
 
 ### REST layer TLS
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.http.pemkey_filepath` | Path to the certificate's key file (PKCS \#8), which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.http.pemkey_password` | Key password. Omit this setting if the key has no password. Optional.
-`opensearch_security.ssl.http.pemcert_filepath` | Path to the X.509 node certificate chain (PEM format), which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.http.pemtrustedcas_filepath` | Path to the root CAs (PEM format), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.http.pemkey_filepath` | Path to the certificate's key file (PKCS \#8), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.http.pemkey_password` | Key password. Omit this setting if the key has no password. Optional.
+`plugins.security.ssl.http.pemcert_filepath` | Path to the X.509 node certificate chain (PEM format), which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.http.pemtrustedcas_filepath` | Path to the root CAs (PEM format), which must be under the `config` directory, specified using a relative path. Required.
 
 
 ## Keystore and truststore files
@@ -50,29 +50,29 @@ The following settings configure the location and password of your keystore and
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.transport.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
-`opensearch_security.ssl.transport.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.transport.keystore_alias: my_alias` | Alias name. Optional. Default is the first alias.
-`opensearch_security.ssl.transport.keystore_password` | Keystore password. Default is `changeit`.
-`opensearch_security.ssl.transport.truststore_type` | The type of the truststore file, JKS or PKCS12/PFX. Default is JKS.
-`opensearch_security.ssl.transport.truststore_filepath` | Path to the truststore file, which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.transport.truststore_alias` | Alias name. Optional. Default is all certificates.
-`opensearch_security.ssl.transport.truststore_password` | Truststore password. Default is `changeit`.
+`plugins.security.ssl.transport.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
+`plugins.security.ssl.transport.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.transport.keystore_alias: my_alias` | Alias name. Optional. Default is the first alias.
+`plugins.security.ssl.transport.keystore_password` | Keystore password. Default is `changeit`.
+`plugins.security.ssl.transport.truststore_type` | The type of the truststore file, JKS or PKCS12/PFX. Default is JKS.
+`plugins.security.ssl.transport.truststore_filepath` | Path to the truststore file, which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.transport.truststore_alias` | Alias name. Optional. Default is all certificates.
+`plugins.security.ssl.transport.truststore_password` | Truststore password. Default is `changeit`.
 
 
 ### REST layer TLS
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.http.enabled` | Whether to enable TLS on the REST layer. If enabled, only HTTPS is allowed. Optional. Default is false.
-`opensearch_security.ssl.http.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
-`opensearch_security.ssl.http.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.http.keystore_alias` | Alias name. Optional. Default is the first alias.
-`opensearch_security.ssl.http.keystore_password` | Keystore password. Default is `changeit`.
-`opensearch_security.ssl.http.truststore_type` | The type of the truststore file, JKS or PKCS12/PFX. Default is JKS.
-`opensearch_security.ssl.http.truststore_filepath` | Path to the truststore file, which must be under the `config` directory, specified using a relative path. Required.
-`opensearch_security.ssl.http.truststore_alias` | Alias name. Optional. Default is all certificates.
-`opensearch_security.ssl.http.truststore_password` | Truststore password. Default is `changeit`.
+`plugins.security.ssl.http.enabled` | Whether to enable TLS on the REST layer. If enabled, only HTTPS is allowed. Optional. Default is false.
+`plugins.security.ssl.http.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
+`plugins.security.ssl.http.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.http.keystore_alias` | Alias name. Optional. Default is the first alias.
+`plugins.security.ssl.http.keystore_password` | Keystore password. Default is `changeit`.
+`plugins.security.ssl.http.truststore_type` | The type of the truststore file, JKS or PKCS12/PFX. Default is JKS.
+`plugins.security.ssl.http.truststore_filepath` | Path to the truststore file, which must be under the `config` directory, specified using a relative path. Required.
+`plugins.security.ssl.http.truststore_alias` | Alias name. Optional. Default is all certificates.
+`plugins.security.ssl.http.truststore_password` | Truststore password. Default is `changeit`.
 
 
 ## Configure node certificates
@@ -80,7 +80,7 @@ Name | Description
 The security plugin needs to identify inter-cluster requests (i.e. requests between the nodes). The simplest way of configuring node certificates is to list the Distinguished Names (DNs) of these certificates in `opensearch.yml`. All DNs must be included in `opensearch.yml` on all nodes. The security plugin supports wildcards and regular expressions:
 
 ```yml
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=node.other.com,OU=SSL,O=Test,L=Test,C=DE'
   - 'CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE'
   - 'CN=elk-devcluster*'
@@ -95,7 +95,7 @@ If your node certificates have an Object ID (OID) identifier in the SAN section,
 Admin certificates are regular client certificates that have elevated rights to perform administrative tasks. You need an admin certificate to change the the security plugin configuration using `plugins/opensearch-security/tools/securityadmin.sh` or the REST API. Admin certificates are configured in `opensearch.yml` by stating their DN(s):
 
 ```yml
-opensearch_security.authcz.admin_dn:
+plugins.security.authcz.admin_dn:
   - CN=admin,OU=SSL,O=Test,L=Test,C=DE
 ```
 
@@ -112,8 +112,8 @@ If OpenSSL is enabled, but for one reason or another the installation does not w
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.transport.enable_openssl_if_available` | Enable OpenSSL on the transport layer if available. Optional. Default is true.
-`opensearch_security.ssl.http.enable_openssl_if_available` | Enable OpenSSL on the REST layer if available. Optional. Default is true.
+`plugins.security.ssl.transport.enable_openssl_if_available` | Enable OpenSSL on the transport layer if available. Optional. Default is true.
+`plugins.security.ssl.http.enable_openssl_if_available` | Enable OpenSSL on the REST layer if available. Optional. Default is true.
 
 
 {% comment %}
@@ -144,8 +144,8 @@ In addition, when `resolve_hostnames` is enabled, the security plugin resolves t
 
 Name | Description
 :--- | :---
-`opensearch_security.ssl.transport.enforce_hostname_verification` | Whether to verify hostnames on the transport layer. Optional. Default is true.
-`opensearch_security.ssl.transport.resolve_hostname` | Whether to resolve hostnames against DNS on the transport layer. Optional. Default is true. Only works if hostname verification is also enabled.
+`plugins.security.ssl.transport.enforce_hostname_verification` | Whether to verify hostnames on the transport layer. Optional. Default is true.
+`plugins.security.ssl.transport.resolve_hostname` | Whether to resolve hostnames against DNS on the transport layer. Optional. Default is true. Only works if hostname verification is also enabled.
 
 
 ## (Advanced) Client authentication
@@ -168,7 +168,7 @@ You can configure the client authentication mode by using the following setting:
 
 Name | Description
 :--- | :---
-opensearch_security.ssl.http.clientauth_mode | The TLS client authentication mode to use. Can be one of `NONE`, `OPTIONAL` (default) or `REQUIRE`. Optional.
+plugins.security.ssl.http.clientauth_mode | The TLS client authentication mode to use. Can be one of `NONE`, `OPTIONAL` (default) or `REQUIRE`. Optional.
 
 
 ## (Advanced) Enabled ciphers and protocols
@@ -179,18 +179,18 @@ If this setting is not enabled, the ciphers and TLS versions are negotiated betw
 
 Name | Data Type | Description
 :--- | :--- | :---
-`opensearch_security.ssl.http.enabled_ciphers` | Array | Enabled TLS cipher suites for the REST layer. Only Java format is supported.
-`opensearch_security.ssl.http.enabled_protocols` | Array | Enabled TLS protocols for the REST layer. Only Java format is supported.
-`opensearch_security.ssl.transport.enabled_ciphers` | Array | Enabled TLS cipher suites for the transport layer. Only Java format is supported.
-`opensearch_security.ssl.transport.enabled_protocols` | Array | Enabled TLS protocols for the transport layer. Only Java format is supported.
+`plugins.security.ssl.http.enabled_ciphers` | Array | Enabled TLS cipher suites for the REST layer. Only Java format is supported.
+`plugins.security.ssl.http.enabled_protocols` | Array | Enabled TLS protocols for the REST layer. Only Java format is supported.
+`plugins.security.ssl.transport.enabled_ciphers` | Array | Enabled TLS cipher suites for the transport layer. Only Java format is supported.
+`plugins.security.ssl.transport.enabled_protocols` | Array | Enabled TLS protocols for the transport layer. Only Java format is supported.
 
 ### Example settings
 
 ```yml
-opensearch_security.ssl.http.enabled_ciphers:
+plugins.security.ssl.http.enabled_ciphers:
   - "TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
   - "TLS_DHE_DSS_WITH_AES_128_CBC_SHA256"
-opensearch_security.ssl.http.enabled_protocols:
+plugins.security.ssl.http.enabled_protocols:
   - "TLSv1.1"
   - "TLSv1.2"
 ```
@@ -198,7 +198,7 @@ opensearch_security.ssl.http.enabled_protocols:
 Because it is insecure, the security plugin disables `TLSv1` by default. If you need to use `TLSv1` and accept the risks, you can still enable it:
 
 ```yml
-opensearch_security.ssl.http.enabled_protocols:
+plugins.security.ssl.http.enabled_protocols:
   - "TLSv1"
   - "TLSv1.1"
   - "TLSv1.2"
diff --git a/docs/troubleshoot/openid-connect.md b/docs/troubleshoot/openid-connect.md
index 96f175e5..4a920835 100644
--- a/docs/troubleshoot/openid-connect.md
+++ b/docs/troubleshoot/openid-connect.md
@@ -24,8 +24,8 @@ This page includes troubleshooting steps for using OpenID Connect with the secur
 To help troubleshoot OpenID Connect, set the log level to `debug` on OpenSearch. Add the following lines in `config/log4j2.properties` and restart the node:
 
 ```
-logger.opensearch_security.name = com.amazon.dlic.auth.http.jwt
-logger.opensearch_security.level = trace
+logger.plugins.security.name = com.amazon.dlic.auth.http.jwt
+logger.plugins.security.level = trace
 ```
 
 This setting prints a lot of helpful information to your log file. If this information isn't sufficient, you can also set the log level to `trace`.
@@ -36,7 +36,7 @@ This setting prints a lot of helpful information to your log file. If this infor
 This error indicates that the security plugin can't reach the metadata endpoint of your IdP. In `opensearch_dashboards.yml`, check the following setting:
 
 ```
-opensearch_security.openid.connect_url: "http://keycloak.example.com:8080/auth/realms/master/.well-known/openid-configuration"
+plugins.security.openid.connect_url: "http://keycloak.example.com:8080/auth/realms/master/.well-known/openid-configuration"
 ```
 
 If this error occurs on OpenSearch, check the following setting in `config.yml`:
@@ -60,9 +60,9 @@ This indicates that one or more of the OpenSearch Dashboards configuration setti
 Check `opensearch_dashboards.yml` and make sure you have set the following minimal configuration:
 
 ```yml
-opensearch_security.openid.connect_url: "..."
-opensearch_security.openid.client_id: "..."
-opensearch_security.openid.client_secret: "..."
+plugins.security.openid.connect_url: "..."
+plugins.security.openid.client_id: "..."
+plugins.security.openid.client_secret: "..."
 ```
 
 
@@ -81,7 +81,7 @@ Please delete all cached browser data, or try again in a private browser window.
 To trade the access token for an identity token, most IdPs require you to provide a client secret. Check if the client secret in `opensearch_dashboards.yml` matches the client secret of your IdP configuration:
 
 ```
-opensearch_security.openid.client_secret: "..."
+plugins.security.openid.client_secret: "..."
 ```
 
 
diff --git a/docs/troubleshoot/tls.md b/docs/troubleshoot/tls.md
index 3db3b0d7..7768fd02 100644
--- a/docs/troubleshoot/tls.md
+++ b/docs/troubleshoot/tls.md
@@ -49,7 +49,7 @@ The security plugin uses the [string representation of Distinguished Names (RFC1
 If parts of your DN contain special characters (e.g. a comma), make sure you escape it in your configuration:
 
 ```yml
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'
 ```
 
@@ -58,14 +58,14 @@ You can have whitespace within a field, but not between fields.
 #### Bad configuration
 
 ```yml
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=node-0.example.com, OU=SSL,O=My\, Test, L=Test, C=DE'
 ```
 
 #### Good configuration
 
 ```yml
-opensearch_security.nodes_dn:
+plugins.security.nodes_dn:
   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'
 ```
 
@@ -197,7 +197,7 @@ ExtendedKeyUsages [
 The security plugin disables TLS version 1.0 by default; it is outdated, insecure, and vulnerable. If you need to use `TLSv1` and accept the risks, you can enable it in `opensearch.yml`:
 
 ```yml
-opensearch_security.ssl.http.enabled_protocols:
+plugins.security.ssl.http.enabled_protocols:
   - "TLSv1"
   - "TLSv1.1"
   - "TLSv1.2"

From 39d9c3037aad3e1ad9875e15e5739cb85453b6d1 Mon Sep 17 00:00:00 2001
From: Keith Chan <12404772+keithhc2@users.noreply.github.com>
Date: Tue, 8 Jun 2021 17:32:38 -0700
Subject: [PATCH 28/32] Update index.md

---
 docs/im/index-transforms/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md
index f1f2ddb2..0d0fdad2 100644
--- a/docs/im/index-transforms/index.md
+++ b/docs/im/index-transforms/index.md
@@ -109,7 +109,7 @@ GET finished_flight_job/_search
         "_source" : {
           "transform._id" : "sample_flight_job",
           "transform._doc_count" : 14,
-          "Carrier_terms" : "Kibana Airlines",
+          "Carrier_terms" : "Dashboards Airlines",
           "DestAirportID_terms" : "SFO"
         }
       },

From fd78ed448e46f46e8decd9c6dea77c3adb1b2792 Mon Sep 17 00:00:00 2001
From: Liz Snyder 
Date: Tue, 8 Jun 2021 20:35:36 -0700
Subject: [PATCH 29/32] Comment out busted prop

---
 docs/opensearch/rest-api/cluster-health.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/docs/opensearch/rest-api/cluster-health.md b/docs/opensearch/rest-api/cluster-health.md
index 244ad9a7..eb0f6473 100644
--- a/docs/opensearch/rest-api/cluster-health.md
+++ b/docs/opensearch/rest-api/cluster-health.md
@@ -41,12 +41,13 @@ local | boolean | Whether to return information from the local node only instead
 master_timeout | time | The amount of time to wait for a connection to the master node. Default is 30 seconds.
 timeout | time | The amount of time to wait for a response. If the timeout expires, the request fails. Default is 30 seconds.
 wait_for_active_shards | string | Wait until the specified number of shards is active before returning a response. `all` for all shards. Default is `0`. 
-wait_for_nodes | string | Wait until the specified number of nodes is available. Also supports operators <=, >=, <, and >. 
 wait_for_events | enum | Wait until all currently queued events with the given priority are processed. Supported values are `immediate`, `urgent`, `high`, `normal`, `low`, and `languid`.
 wait_for_no_relocating_shards | boolean | Whether to wait until there are no relocating shards in the cluster. Default is false.
 wait_for_no_initializing_shards | boolean | Whether to wait until there are no initializing shards in the cluster. Default is false.
 wait_for_status | enum | Wait until the cluster is in a specific state or better. Supported values are `green`, `yellow`, and `red`.
 
+
 
 ## Response
 
@@ -68,4 +69,4 @@ wait_for_status | enum | Wait until the cluster is in a specific state or better
   "task_max_waiting_in_queue_millis" : 0,
   "active_shards_percent_as_number" : 100.0
 }
-```
+```
\ No newline at end of file

From 0cf268f91ab9c2fdfe8de0595126a3a48fd87630 Mon Sep 17 00:00:00 2001
From: dblock 
Date: Wed, 9 Jun 2021 07:04:09 -0400
Subject: [PATCH 30/32] Fix names of entries to delete from opensearch.yml when
 uninstalling the security plugin.

Signed-off-by: dblock 
---
 docs/security/configuration/disable.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/security/configuration/disable.md b/docs/security/configuration/disable.md
index 27397bd9..fbb56d23 100755
--- a/docs/security/configuration/disable.md
+++ b/docs/security/configuration/disable.md
@@ -14,7 +14,7 @@ You might want to temporarily disable the security plugin to make testing or int
 plugins.security.disabled: true
 ```
 
-A more permanent option is to remove the security plugin entirely. Delete the `plugins/opensearch-security` folder on all nodes, and delete the `opensearch_security` configuration entries from `opensearch.yml`.
+A more permanent option is to remove the security plugin entirely. Delete the `plugins/opensearch-security` folder on all nodes, and delete the `plugins.security.*` configuration entries from `opensearch.yml`.
 
 To perform these steps on the Docker image, see [Customize the Docker image](../../../opensearch/install/docker/#customize-the-docker-image).
 

From a9489c0f491d764c1f77cd08af7f2974f4729e77 Mon Sep 17 00:00:00 2001
From: Keith Chan <12404772+keithhc2@users.noreply.github.com>
Date: Wed, 9 Jun 2021 09:16:56 -0700
Subject: [PATCH 31/32] Fixed a typo

---
 docs/im/index-transforms/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/im/index-transforms/index.md b/docs/im/index-transforms/index.md
index 0d0fdad2..7fab9063 100644
--- a/docs/im/index-transforms/index.md
+++ b/docs/im/index-transforms/index.md
@@ -51,7 +51,7 @@ On the other hand, aggregations let you perform simple calculations. For example
 
 You can configure transform jobs to run once or multiple times on a schedule. Transform jobs are enabled by default.
 
-1. For **transformation execution frequency**, select either **Define by fixed interval** and specify a **transform interval**.
+1. For **transformation execution frequency**, select **Define by fixed interval** and specify a **transform interval**.
 2. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur.
 3. Choose **Next**.
 

From 6022b9361a6d260530b535500925b8a3f97590b3 Mon Sep 17 00:00:00 2001
From: aetter 
Date: Wed, 9 Jun 2021 11:33:10 -0700
Subject: [PATCH 32/32] Typo.

---
 docs/agents-and-ingestion-tools/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md
index d627eb6b..b6d6e695 100644
--- a/docs/agents-and-ingestion-tools/index.md
+++ b/docs/agents-and-ingestion-tools/index.md
@@ -17,7 +17,7 @@ For a longer term solution, we plan to create an OpenSearch output plugin for Lo
 
 ## Compatibility Matrices
 
-*Italicized* cells are untested, but indicate what a value theoretically what it should be based on existing information.
+*Italicized* cells are untested, but indicate what a value theoretically should be based on existing information.
 
 
 ### Compatibility Matrix for Logstash