Task consumer integration (#866)

* added resource_stats object to sample responses

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Added information about log files

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Removed resource_stats from existing sample responses that lack the detailed flag

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Added example with detailed parameter

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Changed phrasing to use active voice

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Removed extra comma from sample JSON and rephrased feature description

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Replaced query string in detailed flag example with placeholder

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

* Small wording change

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

Signed-off-by: jeffhuss <jeffhuss@amazon.com>

_opensearch/logs.md

@@ -168,7 +168,36 @@ node1 | [2019-10-24T19:48:51,012][WARN][i.i.s.index] [node1] [some-index/i86iF5k
 
 Slow logs can consume considerable disk space if you set thresholds or levels too low. Consider enabling them temporarily for troubleshooting or performance tuning. To disable slow logs, return all thresholds to `-1`.
 
+## Task logs
+
+OpenSearch can log CPU time and memory utilization for the top N memory expensive search tasks when task resource consumers are enabled. By default, task resource consumers will log the top 10 search tasks at 60 second intervals. These values can be configured in `opensearch.yml`.
+
+Task logging is enabled dynamically through the cluster settings API:
+
+```bash
+PUT _cluster/settings
+{
+  "persistent" : {
+    "task_resource_consumers.enabled" : "true"
+  }
+}
+```
+
+Enabling task resource consumers can have an impact on search latency.
+{:.tip}
+
+Once enabled, logs will be written to `logs/opensearch_task_detailslog.json` and `logs/opensearch_task_detailslog.log`.
+
+To configure the logging interval and the number of search tasks logged, add the following lines to `opensearch.yml`:
+
+```bash
+# Number of expensive search tasks to log
+cluster.task.consumers.top_n.size:100
+
+# Logging interval
+cluster.task.consumers.top_n.frequency:30s
+```
 
 ## Deprecation logs
 
-Deprecation logs record when clients make deprecated API calls to your cluster. These logs can help you identify and fix issues prior to upgrading to a new major version. By default, OpenSearch logs deprecated API calls at the WARN level, which works well for almost all use cases. If desired, configure `logger.deprecation.level` using `_cluster/settings`, `opensearch.yml`, or `log4j2.properties`.
+Deprecation logs record when clients make deprecated API calls to your cluster. These logs can help you identify and fix issues prior to upgrading to a new major version. By default, OpenSearch logs deprecated API calls at the WARN level, which works well for almost all use cases. If desired, configure `logger.deprecation.level` using `_cluster/settings`, `opensearch.yml`, or `log4j2.properties`.

_opensearch/rest-api/tasks.md

@@ -50,6 +50,7 @@ Note that if a task finishes running, it won't be returned as part of your reque
           "running_time_in_nanos": 994000,
           "cancellable": false,
           "headers": {}
+          }
         },
         "Mgqdm0r9SEGClWxp_RbnaQ:17413": {
           "node": "Mgqdm0r9SEGClWxp_RbnaQ",
@@ -142,6 +143,60 @@ GET /_tasks?nodes=opensearch-node1
 }
 ```
 
+The following request will return detailed information about active search tasks:
+
+**Sample Request**
+
+```bash
+curl -XGET "localhost:9200/_tasks?actions=*search&detailed
+```
+
+**Sample Response**
+
+```json
+{
+  "nodes" : {
+    "CRqNwnEeRXOjeTSYYktw-A" : {
+      "name" : "runTask-0",
+      "transport_address" : "127.0.0.1:9300",
+      "host" : "127.0.0.1",
+      "ip" : "127.0.0.1:9300",
+      "roles" : [
+        "cluster_manager",
+        "data",
+        "ingest",
+        "remote_cluster_client"
+      ],
+      "attributes" : {
+        "testattr" : "test",
+        "shard_indexing_pressure_enabled" : "true"
+      },
+      "tasks" : {
+        "CRqNwnEeRXOjeTSYYktw-A:677" : {
+          "node" : "CRqNwnEeRXOjeTSYYktw-A",
+          "id" : 677,
+          "type" : "transport",
+          "action" : "indices:data/read/search",
+          "description" : "indices[], search_type[QUERY_THEN_FETCH], source[{\"query\":{\"query_string\":<QUERY_STRING>}}]",
+          "start_time_in_millis" : 1660106254525,
+          "running_time_in_nanos" : 1354236,
+          "cancellable" : true,
+          "cancelled" : false,
+          "headers" : { },
+          "resource_stats" : {
+            "total" : {
+              "cpu_time_in_nanos" : 0,
+              "memory_in_bytes" : 0
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+```
+
 ## Task canceling
 
 After getting a list of tasks, you can cancel all cancelable tasks with the following request:
@@ -230,4 +285,4 @@ This operation supports the same parameters as the `tasks` operation. The follow
 
 ```bash
 curl -i -H "X-Opaque-Id: 123456" "https://localhost:9200/_tasks?nodes=opensearch-node1" -u 'admin:admin' --insecure
-```
+```