Update docs for node specifications (#30468)

Expands and clarifies exactly what is and isn't allowed when specifying a
subset of the nodes as targets of a cluster API, and adds missing links to this
from the hot threads and cluster stats API docs.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yu <yu.liu003@gmail.com>

docs/reference/cluster.asciidoc

@@ -6,23 +6,70 @@
 ["float",id="cluster-nodes"]
 == Node specification
 
-Most cluster level APIs allow to specify which nodes to execute on (for
+Some cluster-level APIs may operate on a subset of the nodes which can be
-example, getting the node stats for a node). Nodes can be identified in
+specified with _node filters_. For example, the <<tasks,Task Management>>,
-the APIs either using their internal node id, the node name, address,
+<<cluster-nodes-stats,Nodes Stats>>, and <<cluster-nodes-info,Nodes Info>> APIs
-custom attributes, or just the `_local` node receiving the request. For
+can all report results from a filtered set of nodes rather than from all nodes.
-example, here are some sample executions of nodes info:
+
+_Node filters_ are written as a comma-separated list of individual filters,
+each of which adds or removes nodes from the chosen subset. Each filter can be
+one of the following:
+
+* `_all`, to add all nodes to the subset.
+* `_local`, to add the local node to the subset.
+* `_master`, to add the currently-elected master node to the subset.
+* a node id or name, to add this node to the subset.
+* an IP address or hostname, to add all matching nodes to the subset.
+* a pattern, using `*` wildcards, which adds all nodes to the subset
+  whose name, address or hostname matches the pattern.
+* `master:true`, `data:true`, `ingest:true` or `coordinating_only:true`, which
+  respectively add to the subset all master-eligible nodes, all data nodes,
+  all ingest nodes, and all coordinating-only nodes.
+* `master:false`, `data:false`, `ingest:false` or `coordinating_only:false`,
+  which respectively remove from the subset all master-eligible nodes, all data
+  nodes, all ingest nodes, and all coordinating-only nodes.
+* a pair of patterns, using `*` wildcards, of the form `attrname:attrvalue`,
+  which adds to the subset all nodes with a custom node attribute whose name
+  and value match the respective patterns. Custom node attributes are
+  configured by setting properties in the configuration file of the form
+  `node.attr.attrname: attrvalue`.
+
+NOTE: node filters run in the order in which they are given, which is important
+if using filters that remove nodes from the set. For example
+`_all,master:false` means all the nodes except the master-eligible ones, but
+`master:false,_all` means the same as `_all` because the `_all` filter runs
+after the `master:false` filter.
+
+NOTE: if no filters are given, the default is to select all nodes. However, if
+any filters are given then they run starting with an empty chosen subset. This
+means that filters such as `master:false` which remove nodes from the chosen
+subset are only useful if they come after some other filters. When used on its
+own, `master:false` selects no nodes.
+
+Here are some examples of the use of node filters with the
+<<cluster-nodes-info,Nodes Info>> APIs.
 
 [source,js]
 --------------------------------------------------
-# Local
+# If no filters are given, the default is to select all nodes
+GET /_nodes
+# Explicitly select all nodes
+GET /_nodes/_all
+# Select just the local node
 GET /_nodes/_local
-# Address
+# Select the elected master node
-GET /_nodes/10.0.0.3,10.0.0.4
+GET /_nodes/_master
-GET /_nodes/10.0.0.*
+# Select nodes by name, which can include wildcards
-# Names
 GET /_nodes/node_name_goes_here
 GET /_nodes/node_name_goes_*
-# Attributes (set something like node.attr.rack: 2 in the config)
+# Select nodes by address, which can include wildcards
+GET /_nodes/10.0.0.3,10.0.0.4
+GET /_nodes/10.0.0.*
+# Select nodes by role
+GET /_nodes/_all,master:false
+GET /_nodes/data:true,ingest:true
+GET /_nodes/coordinating_only:true
+# Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file)
 GET /_nodes/rack:2
 GET /_nodes/ra*:2
 GET /_nodes/ra*:2*

docs/reference/cluster/nodes-hot-threads.asciidoc

@@ -1,12 +1,23 @@
 [[cluster-nodes-hot-threads]]
 == Nodes hot_threads
 
-An API allowing to get the current hot threads on each node in the
+This API yields a breakdown of the hot threads on each selected node in the
-cluster. Endpoints are `/_nodes/hot_threads`, and
+cluster. Its endpoints are `/_nodes/hot_threads` and
-`/_nodes/{nodesIds}/hot_threads`.
+`/_nodes/{nodes}/hot_threads`:
 
-The output is plain text with a breakdown of each node's top hot
+[source,js]
-threads. Parameters allowed are:
+--------------------------------------------------
+GET /_nodes/hot_threads
+GET /_nodes/nodeId1,nodeId2/hot_threads
+--------------------------------------------------
+// CONSOLE
+
+The first command gets the hot threads of all the nodes in the cluster. The
+second command gets the hot threads of only `nodeId1` and `nodeId2`. Nodes can
+be selected using <<cluster-nodes,node filters>>.
+
+The output is plain text with a breakdown of each node's top hot threads.  The
+allowed parameters are:
 
 [horizontal]
 `threads`:: 	number of hot threads to provide, defaults to 3.

docs/reference/cluster/stats.asciidoc

@@ -213,3 +213,12 @@ Will return, for example:
 // 3. All of the numbers and strings on the right hand side of *every* field in
 //    the response are ignored. So we're really only asserting things about the
 //    the shape of this response, not the values in it.
+
+This API can be restricted to a subset of the nodes using the `?nodeId`
+parameter, which accepts <<cluster-nodes,node filters>>:
+
+[source,js]
+--------------------------------------------------
+GET /_cluster/stats?nodeId=node1,node*,master:false
+--------------------------------------------------
+// CONSOLE