Add Query Insights documentation (#6261)

* Query Insights initial documentation

Signed-off-by: Chenyang Ji <cyji@amazon.com>

* Doc review

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* update endpoints and remove exporters

Signed-off-by: Chenyang Ji <cyji@amazon.com>

* Moved query insights to additional plugins section

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Added installation instructions

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* editorial comments

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Resolve merge conflicts

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* merge conflicts and link fix

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Fix link

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

---------

Signed-off-by: Chenyang Ji <cyji@amazon.com>
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Fanit Kolchina <kolchfa@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>

.github/vale/styles/Vocab/OpenSearch/Plugins/accept.txt

@@ -20,6 +20,7 @@ ML Commons plugin
 Neural Search plugin
 Observability plugin
 Performance Analyzer plugin
+Query Insights plugin
 Query Workbench plugin
 Search Relevance plugin
 Security plugin

_install-and-configure/configuring-opensearch/plugin-settings.md

@@ -83,6 +83,10 @@ The Notifications plugin supports the following settings. All settings in this l
 
 - `opensearch.notifications.general.filter_by_backend_roles` (Boolean): Enables filtering by backend roles (role-based access control for the notification channels). Default is `false`.
 
+## Query Insights plugin settings
+
+For information about Query Insights plugin settings, see [Query insights settings]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/index#query-insights-settings).
+
 ## Security plugin settings
 
 For information about the Security plugin settings, see [Security settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/security-settings/).

_install-and-configure/plugins.md

@@ -65,7 +65,7 @@ You can also list installed plugins by using the [CAT API]({{site.url}}{{site.ba
 GET _cat/plugins
 ```
 
-#### Sample response
+#### Example response
 
 ```bash
 opensearch-node1 opensearch-alerting                  2.0.1.0
@@ -250,7 +250,7 @@ bin/opensearch-plugin install --batch <plugin-name>
 Major, minor, and patch plugin versions must match OpenSearch major, minor, and patch versions in order to be compatible. For example, plugins versions 2.3.0.x work only with OpenSearch 2.3.0.
 {: .warning}
 
-### Bundled Plugins
+### Bundled plugins
 
 The following plugins are bundled with all OpenSearch distributions except for minimum distribution packages.
 
@@ -285,7 +285,7 @@ _<sup>2</sup>Performance Analyzer is not available on Windows._
 
 Members of the OpenSearch community have built countless plugins for the service. Although it isn't possible to build an exhaustive list of every plugin, since many plugins are not maintained within the OpenSearch GitHub repository, the following list of plugins are available to be installed by name using `bin/opensearch-plugin install <plugin-name>`.
 
-| Plugin Name | Earliest Available Version |
+| Plugin name | Earliest available version |
 | :--- | :--- |
 | analysis-icu | 1.0.0 |
 | analysis-kuromoji | 1.0.0 |
@@ -301,6 +301,7 @@ Members of the OpenSearch community have built countless plugins for the service
 | mapper-annotated-text | 1.0.0 |
 | mapper-murmur3 | 1.0.0 |
 | mapper-size | 1.0.0 |
+| query-insights | 2.12.0 |
 | repository-azure | 1.0.0 |
 | repository-gcs | 1.0.0 |
 | repository-hdfs | 1.0.0 |

_observing-your-data/query-insights/index.md

@@ -0,0 +1,38 @@
+---
+layout: default
+title: Query insights
+nav_order: 40
+has_children: true
+has_toc: false
+---
+
+# Query insights
+
+To monitor and analyze the search queries within your OpenSearch clusterQuery information, you can obtain query insights. With minimal performance impact, query insights features aim to provide comprehensive insights into search query execution, enabling you to better understand search query characteristics, patterns, and system behavior during query execution stages. Query insights facilitate enhanced detection, diagnosis, and prevention of query performance issues, ultimately improving query processing performance, user experience, and overall system resilience.
+
+Typical use cases for query insights features include the following:
+
+- Identifying top queries by latency within specific time frames
+- Debugging slow search queries and latency spikes
+
+Query insights features are supported by the Query Insights plugin. At a high level, query insights features comprise the following components:
+
+* _Collectors_: Gather performance-related data points at various stages of search query execution.
+* _Processors_: Perform lightweight aggregation and processing on data collected by the collectors.
+* _Exporters_: Export the data into different sinks.
+
+
+## Installing the Query Insights plugin
+
+You need to install the `query-insights` plugin to enable query insights features. To install the plugin, run the following command:
+
+```bash
+bin/opensearch-plugin install query-insights
+```
+For information about installing plugins, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).
+
+## Query insights settings
+
+Query insights features support the following settings:
+
+- [Top n queries]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/top-n-queries/)

_observing-your-data/query-insights/top-n-queries.md

@@ -0,0 +1,82 @@
+---
+layout: default
+title: Top n queries
+parent: Query insights
+nav_order: 65
+---
+
+# Top n queries
+
+Monitoring the top N queries in query insights features can help you gain real-time insights into the top queries with high latency within a certain time frame (for example, the last hour). 
+
+## Getting started
+
+To enable monitoring of the top N queries, configure the following [dynamic settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#dynamic-settings):
+
+- `search.insights.top_queries.latency.enabled`: Set to `true` to [enable monitoring of the top N queries](#enabling-the-top-n-queries-feature).
+- `search.insights.top_queries.latency.window_size`: [Configure the window size](#configuring-window-size). 
+- `search.insights.top_queries.latency.top_n_size`: [Specify the value of n](#configuring-the-value-of-n).
+
+It's important to exercise caution when enabling this feature because it can consume system resources.
+{: .important}
+
+
+For detailed information about enabling and configuring this feature, see the following sections.
+
+## Enabling the top N queries feature 
+
+After installing the `query-insights` plugin, you can enable the top N queries feature (which is disabled by default) by using the following dynamic setting. This setting enables the corresponding collectors and aggregators in the running cluster:
+
+```json
+PUT _cluster/settings
+{
+  "persistent" : {
+    "search.insights.top_queries.latency.enabled" : true
+  }
+}
+```
+{% include copy-curl.html %}
+
+## Configuring window size
+
+You can configure the window size for the top N queries by latency with `search.insights.top_queries.latency.window_size`. For example, a cluster with the following configuration will collect top N queries in a 60-minute window:
+
+```json
+PUT _cluster/settings
+{
+  "persistent" : {
+    "search.insights.top_queries.latency.window_size" : "60m"
+  }
+}
+```
+{% include copy-curl.html %}
+
+## Configuring the value of N 
+
+You can configure the value of N in the `search.insights.top_queries.latency.top_n_size` parameter. For example, a cluster with the following configuration will collect the top 10 queries in the specified window size:
+
+```
+PUT _cluster/settings
+{
+  "persistent" : {
+    "search.insights.top_queries.latency.top_n_size" : 10
+  }
+}
+```
+{% include copy-curl.html %}
+
+## Monitoring the top N queries 
+
+You can use the Insights API endpoint to obtain top N queries by latency:
+
+```json
+GET /_insights/top_queries
+```
+{% include copy-curl.html %}
+
+Specify a metric type to filter the response by metric type (latency is the only supported type as of 2.12):
+
+```json
+GET /_insights/top_queries?type=latency
+```
+{% include copy-curl.html %}