diff --git a/_dashboards/query-workbench.md b/_dashboards/query-workbench.md new file mode 100644 index 00000000..8fe41afc --- /dev/null +++ b/_dashboards/query-workbench.md @@ -0,0 +1,118 @@ +--- +layout: default +title: Query Workbench +nav_order: 125 +redirect_from: + - /search-plugins/sql/workbench/ +--- + +# Query Workbench + +Query Workbench is a tool within OpenSearch Dashboards. You can use Query Workbench to run on-demand [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql/index/) and [PPL]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) queries, translate queries into their equivalent REST API calls, and view and save results in different [response formats]({{site.url}}{{site.baseurl}}/search-plugins/sql/response-formats/). + +A view of the Query Workbench interface within OpenSearch Dashboards is shown in the following image. + +Query Workbench interface within OpenSearch Dashboards + +## Prerequisites + +Before getting started, make sure you have [indexed your data]({{site.url}}{{site.baseurl}}/im-plugin/index/). + +For this tutorial, you can index the following sample documents. Alternatively, you can use the [OpenSearch Playground](https://playground.opensearch.org/app/opensearch-query-workbench#/), which has preloaded indexes that you can use to try out Query Workbench. + +To index sample documents, send the following [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) request: + +```json +PUT accounts/_bulk?refresh +{"index":{"_id":"1"}} +{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"} +{"index":{"_id":"6"}} +{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"} +{"index":{"_id":"13"}} +{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","email":"nanettebates@quility.com","city":"Nogal","state":"VA"} +{"index":{"_id":"18"}} +{"account_number":18,"balance":4180,"firstname":"Dale","lastname":"Adams","age":33,"gender":"M","address":"467 Hutchinson Court","email":"daleadams@boink.com","city":"Orick","state":"MD"} +``` +{% include copy-curl.html %} + +## Running SQL queries within Query Workbench + +Follow these steps to learn how to run SQL queries against your OpenSearch data using Query Workbench: + +1. Access Query Workbench. + - To access Query Workbench, go to OpenSearch Dashboards and choose **OpenSearch Plugins** > **Query Workbench** from the main menu. + +2. Run a query. + - Select the **SQL** button. In the query editor, type a SQL expression and then select the **Run** button to run the query. + + The following example query retrieves the first name, last name, and balance from the `accounts` index for accounts with a balance greater than 10,000 and sorts by balance in descending order: + + ```sql + SELECT + firstname, + lastname, + balance + FROM + accounts + WHERE + balance > 10000 + ORDER BY + balance DESC; + ``` + {% include copy.html %} + +3. View the results. + - View the results in the **Results** pane, which presents the query output in tabular format. You can filter and download the results as needed. + + The following image shows the query editor pane and results pane for the preceding SQL query: + + Query Workbench SQL query input and results output panes + +4. Clear the query editor. + - Select the **Clear** button to clear the query editor and run a new query. + +5. Examine how the query is processed. + - Select the **Explain** button to examine how OpenSearch processes the query, including the steps involved and order of operations. + + The following image shows the explanation of the SQL query that was run in step 2. + + Query Workbench SQL query explanation pane + +## Running PPL queries within Query Workbench + +Follow these steps to learn how to run PPL queries against your OpenSearch data using Query Workbench: + +1. Access Query Workbench. + - To access Query Workbench, go to OpenSearch Dashboards and choose **OpenSearch Plugins** > **Query Workbench** from the main menu. + +2. Run a query. + - Select the **PPL** button. In the query editor, type a PPL query and then select the **Run** button to run the query. + + The following is an example query that retrieves the `firstname` and `lastname` fields for documents in the `accounts` index with age greater than `18`: + + ```sql + search source=accounts + | where age > 18 + | fields firstname, lastname + ``` + {% include copy.html %} + +3. View the results. + - View the results in the **Results** pane, which presents the query output in tabular format. + + The following image shows the query editor pane and results pane for the PPL query that was run in step 2: + + Query Workbench PPL query input and results output panes + +4. Clear the query editor. + - Select the **Clear** button to clear the query editor and run a new query. + +5. Examine how the query is processed. + - Select the **Explain** button to examine how OpenSearch processes the query, including the steps involved and order of operations. + + The following image shows the explanation of the PPL query that was run in step 2. + + Query Workbench PPL query explanation pane + +Query Workbench does not support delete or update operations through SQL or PPL. Access to data is read-only. +{: .important} \ No newline at end of file diff --git a/_search-plugins/sql/index.md b/_search-plugins/sql/index.md index 2d4c98f0..f1e884f1 100644 --- a/_search-plugins/sql/index.md +++ b/_search-plugins/sql/index.md @@ -10,8 +10,12 @@ redirect_from: # SQL and PPL -OpenSearch SQL lets you write queries in SQL rather than the [OpenSearch query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text). If you're already familiar with SQL and don't want to learn the query DSL, this feature is a great option. +OpenSearch SQL lets you write queries in SQL rather than the [OpenSearch query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/). -## Contributing +## Next steps -To get involved and help us improve the SQL plugin, see the [development guide](https://github.com/opensearch-project/sql/blob/main/DEVELOPER_GUIDE.rst) for instructions on setting up your development environment and building the project. +- Learn about the [SQL and PPL API]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql-ppl-api/). +- Learn about [using SQL within OpenSearch]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql/index/). +- Learn about [using PPL within OpenSearch]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/). +- Learn about [using Query Workbench for SQL and PPL queries within OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/query-workbench/). +- Learn more about OpenSearch SQL in the [Developer Guide](https://github.com/opensearch-project/sql/blob/main/DEVELOPER_GUIDE.rst). \ No newline at end of file diff --git a/_search-plugins/sql/ppl/index.md b/_search-plugins/sql/ppl/index.md index a0a45414..1573f251 100644 --- a/_search-plugins/sql/ppl/index.md +++ b/_search-plugins/sql/ppl/index.md @@ -1,6 +1,6 @@ --- layout: default -title: PPL – Piped Processing Language +title: PPL parent: SQL and PPL nav_order: 5 has_children: true @@ -14,53 +14,34 @@ redirect_from: - /search-plugins/ppl/protocol/ --- -# PPL – Piped Processing Language +# PPL -Piped Processing Language (PPL) is a query language that lets you use pipe (`|`) syntax to explore, discover, and query data stored in OpenSearch. +Piped Processing Language (PPL) is a query language that focuses on processing data in a sequential, step-by-step manner. PPL uses the pipe (`|`) operator to combine commands to find and retrieve data. It is the primary language used with observability in OpenSearch and supports multi-data queries. -To quickly get up and running with PPL, use **Query Workbench** in OpenSearch Dashboards. To learn more, see [Workbench]({{site.url}}{{site.baseurl}}/search-plugins/sql/workbench/). +## PPL syntax -The PPL syntax consists of commands delimited by the pipe character (`|`) where data flows from left to right through each pipeline. +The following example shows the basic PPL syntax: ```sql -search command | command 1 | command 2 ... +search source= | | | ... | ``` +{% include copy.html %} -You can only use read-only commands like `search`, `where`, `fields`, `rename`, `dedup`, `stats`, `sort`, `eval`, `head`, `top`, and `rare`. +See [Syntax]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/syntax/) for specific PPL syntax examples. -## Quick start +## PPL commands -To get started with PPL, choose **Dev Tools** in OpenSearch Dashboards and use the `bulk` operation to index some sample data: +PPL filters, transforms, and aggregates data using a series of commands. See [Commands](/search-plugins/sql/ppl/functions/) for a description and an example of each command. -```json -PUT accounts/_bulk?refresh -{"index":{"_id":"1"}} -{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"} -{"index":{"_id":"6"}} -{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"} -{"index":{"_id":"13"}} -{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","city":"Nogal","state":"VA"} -{"index":{"_id":"18"}} -{"account_number":18,"balance":4180,"firstname":"Dale","lastname":"Adams","age":33,"gender":"M","address":"467 Hutchinson Court","email":"daleadams@boink.com","city":"Orick","state":"MD"} -``` +## Using PPL within OpenSearch -Go to **Query Workbench** and select **PPL**. +To use PPL, you must have installed OpenSearch Dashboards. PPL is available within the [Query Workbench tool](https://playground.opensearch.org/app/opensearch-query-workbench#/). See the [Query Workbench]({{site.url}}{{site.baseurl}}/dashboards/query-workbench/) documentation for a tutorial on using PPL within OpenSearch. -The following example returns `firstname` and `lastname` fields for documents in an `accounts` index with `age` greater than 18: +## Developer documentation -```sql -search source=accounts -| where age > 18 -| fields firstname, lastname -``` +Developers can find information in the following resources: -#### Example response - -firstname | lastname | -:--- | :--- | -Amber | Duke -Hattie | Bond -Nanette | Bates -Dale | Adams - -![PPL query workbench]({{site.url}}{{site.baseurl}}/images/ppl.png) +- [Piped Processing Language](https://github.com/opensearch-project/piped-processing-language) specification +- [OpenSearch PPL Reference Manual](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) +- [Observability](https://github.com/opensearch-project/dashboards-observability/) using [PPL-based visualizations](https://github.com/opensearch-project/dashboards-observability#event-analytics) +- PPL [Data Types](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/general/datatypes.rst) diff --git a/_search-plugins/sql/sql-ppl-api.md b/_search-plugins/sql/sql-ppl-api.md index 42625e4f..fefd612c 100644 --- a/_search-plugins/sql/sql-ppl-api.md +++ b/_search-plugins/sql/sql-ppl-api.md @@ -1,28 +1,16 @@ --- layout: default -title: SQL/PPL API +title: SQL and PPL API parent: SQL and PPL nav_order: 1 --- -# SQL/PPL API +# SQL and PPL API Use the SQL and PPL API to send queries to the SQL plugin. Use the `_sql` endpoint to send queries in SQL, and the `_ppl` endpoint to send queries in PPL. For both of these, you can also use the `_explain` endpoint to translate your query into [OpenSearch domain-specific language]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/) (DSL) or to troubleshoot errors. ---- - -#### Table of contents -- TOC -{:toc} - - ---- - ## Query API -Introduced 1.0 -{: .label .label-purple } - Sends an SQL/PPL query to the SQL plugin. You can pass the format for the response as a query parameter. ### Query parameters diff --git a/_search-plugins/sql/sql/index.md b/_search-plugins/sql/sql/index.md index 340c40aa..9a466902 100644 --- a/_search-plugins/sql/sql/index.md +++ b/_search-plugins/sql/sql/index.md @@ -12,11 +12,7 @@ redirect_from: # SQL -## Workbench - -The easiest way to get familiar with the SQL plugin is to use **Query Workbench** in OpenSearch Dashboards to test various queries. To learn more, see [Workbench]({{site.url}}{{site.baseurl}}/search-plugins/sql/workbench/). - -![OpenSearch Dashboards SQL UI plugin]({{site.url}}{{site.baseurl}}/images/sql.png) +SQL in OpenSearch bridges the gap between traditional relational database concepts and the flexibility of OpenSearch's document-oriented data storage. This integration gives you the ability to use your SQL knowledge to query, analyze, and extract insights from your OpenSearch data. ## SQL and OpenSearch terminology @@ -30,7 +26,7 @@ Column | Field ## REST API -For a complete REST API reference for the SQL plugin, see [SQL/PPL API]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql-ppl-api). +For a complete REST API reference for the SQL plugin, see [SQL/PPL API]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql-ppl-api/). To use the SQL plugin with your own applications, send requests to the `_plugins/_sql` endpoint: @@ -40,6 +36,7 @@ POST _plugins/_sql "query": "SELECT * FROM my-index LIMIT 50" } ``` +{% include copy-curl.html %} You can query multiple indexes by using a comma-separated list: @@ -49,8 +46,9 @@ POST _plugins/_sql "query": "SELECT * FROM my-index1,myindex2,myindex3 LIMIT 50" } ``` +{% include copy-curl.html %} -You can also specify an index pattern with a wildcard expression: +You can specify an index pattern with a wildcard expression: ```json POST _plugins/_sql @@ -58,14 +56,16 @@ POST _plugins/_sql "query": "SELECT * FROM my-index* LIMIT 50" } ``` +{% include copy-curl.html %} -To run the above query in the command line, use the [curl](https://curl.haxx.se/) command: +To run the preceding query in the command line, use the [curl](https://curl.haxx.se/) command: ```bash curl -XPOST https://localhost:9200/_plugins/_sql -u 'admin:admin' -k -H 'Content-Type: application/json' -d '{"query": "SELECT * FROM my-index* LIMIT 50"}' ``` +{% include copy.html %} -You can specify the [response format]({{site.url}}{{site.baseurl}}/search-plugins/sql/response-formats) as JDBC, standard OpenSearch JSON, CSV, or raw. By default, queries return data in JDBC format. The following query sets the format to JSON: +You can specify the [response format]({{site.url}}{{site.baseurl}}/search-plugins/sql/response-formats/) as JDBC, standard OpenSearch JSON, CSV, or raw. By default, queries return data in JDBC format. The following query sets the format to JSON: ```json POST _plugins/_sql?format=json @@ -73,5 +73,6 @@ POST _plugins/_sql?format=json "query": "SELECT * FROM my-index LIMIT 50" } ``` +{% include copy-curl.html %} -See the rest of this guide for more information about request parameters, settings, supported operations, and tools. \ No newline at end of file +For more information about request parameters, settings, supported operations, and tools, see the related topics under [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql/index/). diff --git a/_search-plugins/sql/workbench.md b/_search-plugins/sql/workbench.md deleted file mode 100644 index 71ece5f5..00000000 --- a/_search-plugins/sql/workbench.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -layout: default -title: Query Workbench -parent: SQL -grand_parent: SQL and PPL -nav_order: 1 -redirect_from: - - /search-plugins/sql/workbench/ - ---- - -# Query Workbench - -Use the Query Workbench to easily run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. - - -## Quick start - -To get started with SQL Workbench, choose **Dev Tools** in OpenSearch Dashboards and use the `bulk` operation to index some sample data: - -```json -PUT accounts/_bulk?refresh -{"index":{"_id":"1"}} -{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"} -{"index":{"_id":"6"}} -{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"} -{"index":{"_id":"13"}} -{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","email":"nanettebates@quility.com","city":"Nogal","state":"VA"} -{"index":{"_id":"18"}} -{"account_number":18,"balance":4180,"firstname":"Dale","lastname":"Adams","age":33,"gender":"M","address":"467 Hutchinson Court","email":"daleadams@boink.com","city":"Orick","state":"MD"} -``` - -Then return to SQL Workbench. - - -### List indexes - -To list all your indexes: - -```sql -SHOW TABLES LIKE % -``` - -| TABLE_NAME -| :--- -| accounts - - -### Read data - -After you index a document, retrieve it using the following SQL expression: - -```sql -SELECT * -FROM accounts -WHERE _id = 1 -``` - -| account_number | firstname | gender | city | balance | employer | state | email | address | lastname | age -| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- -| 1 | Amber | M | Brogan | 39225 | Pyrami | IL | amberduke@pyrami.com | 880 Holmes Lane | Duke | 32 - - -### Delete data - -To delete a document from an index, use the `DELETE` clause: - -```sql -DELETE -FROM accounts -WHERE _id = 0 -``` - -| deleted_rows -| :--- -| 1 diff --git a/images/dashboards/query-PPL-explain.png b/images/dashboards/query-PPL-explain.png new file mode 100644 index 00000000..33ef117a Binary files /dev/null and b/images/dashboards/query-PPL-explain.png differ diff --git a/images/dashboards/query-explain.png b/images/dashboards/query-explain.png new file mode 100644 index 00000000..0d2c5774 Binary files /dev/null and b/images/dashboards/query-explain.png differ diff --git a/images/dashboards/query-list.png b/images/dashboards/query-list.png new file mode 100644 index 00000000..5b368e37 Binary files /dev/null and b/images/dashboards/query-list.png differ diff --git a/images/dashboards/query-workbench-ppl.png b/images/dashboards/query-workbench-ppl.png new file mode 100644 index 00000000..e2b4c82b Binary files /dev/null and b/images/dashboards/query-workbench-ppl.png differ diff --git a/images/dashboards/query-workbench-query-step2.png b/images/dashboards/query-workbench-query-step2.png new file mode 100644 index 00000000..a3f75301 Binary files /dev/null and b/images/dashboards/query-workbench-query-step2.png differ diff --git a/images/dashboards/query-workbench-ui.png b/images/dashboards/query-workbench-ui.png new file mode 100644 index 00000000..8b8538d9 Binary files /dev/null and b/images/dashboards/query-workbench-ui.png differ diff --git a/images/query-workbench-UI.png b/images/query-workbench-UI.png new file mode 100644 index 00000000..6ebdbbf7 Binary files /dev/null and b/images/query-workbench-UI.png differ diff --git a/images/sql.png b/images/sql.png deleted file mode 100644 index e3ebd656..00000000 Binary files a/images/sql.png and /dev/null differ