[DOCS] add details on version compatibility and remote gateway selection (#40056)

This commit clarifies how the gateway selection works when configuring
remote clusters for CCR or CCS. Specifically, it clarifies compatibility
between different versions which is a very common question.

docs/reference/modules/cross-cluster-search.asciidoc

@@ -4,7 +4,8 @@
 The _{ccs}_ feature allows any node to act as a federated client across
 multiple clusters. A {ccs} node won't join the remote cluster, instead
 it connects to a remote cluster in a light fashion in order to execute
-federated search requests.
+federated search requests. For details on communication and compatibility
+between different clusters, see <<modules-remote-clusters>>.
 
 [float]
 === Using {ccs}
@@ -43,7 +44,8 @@ PUT _cluster/settings
 // TEST[s/127.0.0.1:9300/\${transport_host}/]
 
 To search the `twitter` index on remote cluster `cluster_one` the index name
-must be prefixed with the cluster alias separated by a `:` character:
+must be prefixed with the alias of the remote cluster followed by the `:`
+character:
 
 [source,js]
 --------------------------------------------------
@@ -104,7 +106,7 @@ GET /cluster_one:twitter/_search
 // TESTRESPONSE[s/"_score": 1/"_score": "$body.hits.hits.0._score"/]
 
 
-Indices can also be searched with the same name on different clusters:
+Indices with the same name on different clusters can also be searched:
 
 [source,js]
 --------------------------------------------------
@@ -120,10 +122,10 @@ GET /cluster_one:twitter,twitter/_search
 // CONSOLE
 // TEST[continued]
 
-Search results are disambiguated the same way as the indices are disambiguated in the request. Even if index names are
+Search results are disambiguated the same way as the indices are disambiguated in the request.
-identical these indices will be treated as different indices when results are merged. All results retrieved from a
+Indices with same names are treated as different indices when results are merged. All results
-remote index
+retrieved from an index located in a remote cluster are prefixed with their corresponding
-will be prefixed with their remote cluster name:
+cluster alias:
 
 [source,js]
 --------------------------------------------------
@@ -185,10 +187,11 @@ will be prefixed with their remote cluster name:
 [float]
 === Skipping disconnected clusters
 
-By default all remote clusters that are searched via {ccs} need to be available when
+By default, all remote clusters that are searched via {ccs} need to be
-the search request is executed, otherwise the whole request fails and no search results are returned
+available when the search request is executed. Otherwise, the whole request
-despite some of the clusters are available. Remote clusters can be made optional through the
+fails; even if some of the clusters are available, no search results are
-boolean `skip_unavailable` setting, set to `false` by default.
+returned. You can use the boolean `skip_unavailable` setting to make remote
+clusters optional. By default, it is set to `false`.
 
 [source,js]
 --------------------------------
@@ -312,6 +315,3 @@ The <<search-request-body, search API>> supports the `ccs_minimize_roundtrips`
 parameter, which defaults to `true` and can be set to `false` in case
 minimizing network round-trips is not desirable.
 
-Note that all the communication between the nodes, regardless of which cluster
-they belong to and the selected reduce mode, happens through the
-<<modules-transport,transport layer>>.

docs/reference/modules/remote-clusters.asciidoc

@@ -14,16 +14,46 @@ connections to a remote cluster. This functionality is used in
 endif::[]
 
 Remote cluster connections work by configuring a remote cluster and connecting
-only to a limited number of nodes in the remote cluster. Each remote cluster is
+only to a limited number of nodes in that remote cluster. Each remote cluster
-referenced by a name and a list of seed nodes. When a remote cluster is
+is referenced by a name and a list of seed nodes. When a remote cluster is
-registered, its cluster state is retrieved from one of the seed nodes so that by
+registered, its cluster state is retrieved from one of the seed nodes and up
-default up to three _gateway nodes_ are selected to be connected to as part of
+to three _gateway nodes_ are selected to be connected to as part of remote
-remote cluster requests.  Remote cluster connections consist of uni-directional
+cluster requests. All the communication required between different clusters
-connections from the coordinating node to the previously selected remote nodes
+goes through the <<modules-transport,transport layer>>. Remote cluster
-only. You can tag which nodes should be selected by using node attributes (see <<remote-cluster-settings>>).
+connections consist of uni-directional connections from the coordinating
+node to the selected remote _gateway nodes_ only.
 
-Each node in a cluster that has remote clusters configured connects to one or
+[float]
-more _gateway nodes_ and uses them to federate requests to the remote cluster.
+[[gateway-nodes-selection]]
+=== Gateway nodes selection
+
+The _gateway nodes_ selection depends on the following criteria:
+
+- *version*: Remote nodes must be compatible with the cluster they are
+registered to. This is subject to the same rules as <<rolling-upgrades>>.
+Any node can communicate with any other node on the same major version (e.g.
+6.0 can talk to any 6.x node). Only nodes on the last minor version of a
+certain major version can communicate with nodes on the following major
+version (e.g. 6.7 can communicate with 7.0, as well as any 7.x node, while
+ 6.6 or earlier cannot talk to any 7.x node). Note that version compatibility
+ is symmetric, meaning that if 6.7 can communicate with 7.0, 7.0 can also
+ communicate with 6.7. The matrix below summarizes compatibility as described
+ above.
+
+[cols="^,^,^,^,^,^"]
+|====
+| Compatibility | 5.0->5.5 | 5.6 | 6.0->6.6 | 6.7 | 7.x
+| 5.0->5.5      |    Yes   | Yes |    No    | No  | No
+| 5.6           |    Yes   | Yes |    Yes   | Yes | No
+| 6.0->6.6      |    No    | Yes |    Yes   | Yes | No
+| 6.7           |    No    | Yes |    Yes   | Yes | Yes
+| 7.x           |    No    | No  |    No    | Yes | Yes
+|====
+
+- *role*: Dedicated master nodes never get selected.
+- *attributes*: You can tag which nodes should be selected
+(see <<remote-cluster-settings>>), though such tagged nodes still have
+to satisfy the two above requirements.
 
 [float]
 [[configuring-remote-clusters]]
@@ -31,7 +61,8 @@ more _gateway nodes_ and uses them to federate requests to the remote cluster.
 
 You can configure remote clusters globally by using
 <<cluster-update-settings,cluster settings>>, which you can update dynamically.
-Alternatively, you can configure them locally on individual nodes by using the `elasticsearch.yml` file.
+Alternatively, you can configure them locally on individual nodes by using the
+ `elasticsearch.yml` file.
 
 If you specify the settings in `elasticsearch.yml` files, only the nodes with
 those settings can connect to the remote cluster. In other words, functionality
@@ -59,7 +90,8 @@ between local and remote indices.
 For more information about the optional transport settings, see
 <<modules-transport>>. 
 
-If you use <<cluster-update-settings,cluster settings>>, the remote clusters are available on every node in the cluster. For example:
+If you use <<cluster-update-settings,cluster settings>>, the remote clusters
+are available on every node in the cluster. For example:
 
 [source,js]
 --------------------------------