diff --git a/docs/reference/modules.asciidoc b/docs/reference/modules.asciidoc index 2346fdb4c2b..80643febc3b 100644 --- a/docs/reference/modules.asciidoc +++ b/docs/reference/modules.asciidoc @@ -73,6 +73,8 @@ The modules in this section are: Configure the transport networking layer, used internally by Elasticsearch to communicate between nodes. +<>:: + Remote clusters are used in features that work by connecting across clusters on the transport layer. <>:: @@ -106,4 +108,6 @@ include::modules/threadpool.asciidoc[] include::modules/transport.asciidoc[] +include::modules/remote-clusters.asciidoc[] + include::modules/cross-cluster-search.asciidoc[] diff --git a/docs/reference/modules/cross-cluster-search.asciidoc b/docs/reference/modules/cross-cluster-search.asciidoc index d6c65eaff01..2db2494d2f3 100644 --- a/docs/reference/modules/cross-cluster-search.asciidoc +++ b/docs/reference/modules/cross-cluster-search.asciidoc @@ -6,52 +6,10 @@ multiple clusters. A cross cluster search node won't join the remote cluster, in it connects to a remote cluster in a light fashion in order to execute federated search requests. -Cross cluster search works by configuring a remote cluster in the cluster state and connecting only to a -limited number of nodes in the remote cluster. Each remote cluster is referenced by a name and a list of seed nodes. -When a remote cluster is registered, its cluster state is retrieved from one of the seed nodes so that up to 3 -_gateway nodes_ are selected to be connected to as part of upcoming cross cluster search requests. -Cross cluster search requests consist of uni-directional connections from the coordinating node to the previously -selected remote nodes only. It is possible to tag which nodes should be selected through -node attributes (see <>). - -Each node in a cluster that has remote clusters configured connects to one or more _gateway nodes_ and uses -them to federate search requests to the remote cluster. - [float] -=== Configuring Cross Cluster Search +=== Using cross cluster search -Remote clusters can be specified globally using <> -(which can be updated dynamically), or local to individual nodes using the -`elasticsearch.yml` file. - -If a remote cluster is configured via `elasticsearch.yml` only the nodes with -that configuration will be able to connect to the remote cluster. In other -words, federated search requests will have to be sent specifically to those -nodes. Remote clusters set via the <> -will be available on every node in the cluster. - -[WARNING] -This feature was added as Beta in Elasticsearch `v5.3` with further improvements made in 5.4 and 5.5. It requires gateway eligible nodes to be on `v5.5` onwards. - -The `elasticsearch.yml` config file for a _cross cluster search_ node just needs to list the -remote clusters that should be connected to, for instance: - -[source,yaml] --------------------------------- -cluster: - remote: - cluster_one: <1> - seeds: 127.0.0.1:9300 - cluster_two: <1> - seeds: 127.0.0.1:9301 - --------------------------------- -<1> `cluster_one` and `cluster_two` are arbitrary cluster aliases representing the connection to each cluster. -These names are subsequently used to distinguish between local and remote indices. - -The equivalent example using the <> -to add remote clusters to all nodes in the cluster would look like the -following: +Cross-cluster search requires <>. [source,js] -------------------------------- @@ -84,85 +42,6 @@ PUT _cluster/settings // TEST[setup:host] // TEST[s/127.0.0.1:9300/\${transport_host}/] -////////////////////////// - -We want to be sure that settings have been updated, -because we'll use them later. - -[source,js] --------------------------------------------------- -{ - "acknowledged" : true, - "persistent": { - "cluster": { - "remote": { - "cluster_one": { - "seeds": [ - "127.0.0.1:9300" - ] - }, - "cluster_two": { - "seeds": [ - "127.0.0.1:9301" - ] - }, - "cluster_three": { - "seeds": [ - "127.0.0.1:9302" - ] - } - } - } - }, - "transient" : {} -} --------------------------------------------------- -// TESTRESPONSE[s/127.0.0.1:9300/\${transport_host}/] - -////////////////////////// - - -A remote cluster can be deleted from the cluster settings by setting its seeds to `null`: - -[source,js] --------------------------------- -PUT _cluster/settings -{ - "persistent": { - "cluster": { - "remote": { - "cluster_three": { - "seeds": null <1> - } - } - } - } -} --------------------------------- -// CONSOLE -// TEST[continued] -<1> `cluster_three` would be removed from the cluster settings, leaving `cluster_one` and `cluster_two` intact. - -////////////////////////// - -We want to be sure that settings have been updated, -because we'll use them later. - -[source,js] --------------------------------------------------- -{ - "acknowledged" : true, - "persistent" : {}, - "transient" : {} -} --------------------------------------------------- -// TESTRESPONSE - -////////////////////////// - -[float] -=== Using cross cluster search - To search the `twitter` index on remote cluster `cluster_one` the index name must be prefixed with the cluster alias separated by a `:` character: @@ -385,47 +264,3 @@ GET /cluster_one:twitter,cluster_two:twitter,twitter/_search <1> // TESTRESPONSE[s/"_score": 1/"_score": "$body.hits.hits.0._score"/] // TESTRESPONSE[s/"_score": 2/"_score": "$body.hits.hits.1._score"/] <1> The `clusters` section indicates that one cluster was unavailable and got skipped - - -[float] -[[cross-cluster-search-settings]] -=== Cross cluster search settings - -`cluster.remote.connections_per_cluster`:: - - The number of nodes to connect to per remote cluster. The default is `3`. - -`cluster.remote.initial_connect_timeout`:: - - The time to wait for remote connections to be established when the node starts. The default is `30s`. - -`cluster.remote.node.attr`:: - - A node attribute to filter out nodes that are eligible as a gateway node in - the remote cluster. For instance a node can have a node attribute - `node.attr.gateway: true` such that only nodes with this attribute will be - connected to if `cluster.remote.node.attr` is set to `gateway`. - -`cluster.remote.connect`:: - - By default, any node in the cluster can act as a cross-cluster client and - connect to remote clusters. The `cluster.remote.connect` setting can be set - to `false` (defaults to `true`) to prevent certain nodes from connecting to - remote clusters. Cross-cluster search requests must be sent to a node that - is allowed to act as a cross-cluster client. - -`cluster.remote.${cluster_alias}.skip_unavailable`:: - - Per cluster boolean setting that allows to skip specific clusters when no - nodes belonging to them are available and they are searched as part of a - cross cluster search request. Default is `false`, meaning that all clusters - are mandatory by default, but they can selectively be made optional by - setting this setting to `true`. - -[float] -[[retrieve-remote-clusters-info]] -=== Retrieving remote clusters info - -The <> allows to retrieve -information about the configured remote clusters, as well as the remote -nodes that the Cross Cluster Search node is connected to. diff --git a/docs/reference/modules/remote-clusters.asciidoc b/docs/reference/modules/remote-clusters.asciidoc new file mode 100644 index 00000000000..b2cf46524dc --- /dev/null +++ b/docs/reference/modules/remote-clusters.asciidoc @@ -0,0 +1,161 @@ +[[modules-remote-clusters]] +== Remote clusters + +ifndef::include-xpack[] +The _remote clusters_ module allows establishing uni-directional connections to +a remote cluster. This functionality is used in +<>. +endif::[] +ifdef::include-xpack[] +The _remote clusters_ module allows establishing uni-directional connections to +a remote cluster. This functionality is used in cross-cluster replication, +<>. +endif::[] + +Remote cluster connections work by configuring a remote cluster and connecting +only to a limited number of nodes in the remote cluster. Each remote cluster is +referenced by a name and a list of seed nodes. When a remote cluster is +registered, its cluster state is retrieved from one of the seed nodes so that by +default up to three _gateway nodes_ are selected to be connected to as part of +remote cluster requests. Remote cluster connections consist of uni-directional +connections from the coordinating node to the previously selected remote nodes +only. It is possible to tag which nodes should be selected through node +attributes (see <>). + +Each node in a cluster that has remote clusters configured connects to one or +more _gateway nodes_ and uses them to federate requests to the remote cluster. + +[float] +[[configuring-remote-clusters]] +=== Configuring Remote Clusters + +Remote clusters can be specified globally using +<> (which can be updated dynamically), +or local to individual nodes using the `elasticsearch.yml` file. + +If a remote cluster is configured via `elasticsearch.yml` only the nodes with +that configuration will be able to connect to the remote cluster. In other +words, functionality that relies on remote cluster requests will have to be +driven specifically from those nodes. Remote clusters set via the +<> will be available on every node +in the cluster. + +The `elasticsearch.yml` config file for a node that connects to remote clusters +needs to list the remote clusters that should be connected to, for instance: + +[source,yaml] +-------------------------------- +cluster: + remote: + cluster_one: <1> + seeds: 127.0.0.1:9300 + cluster_two: <1> + seeds: 127.0.0.1:9301 + +-------------------------------- +<1> `cluster_one` and `cluster_two` are arbitrary _cluster aliases_ representing +the connection to each cluster. These names are subsequently used to distinguish +between local and remote indices. + +The equivalent example using the <> to add remote clusters to all nodes in the cluster would look like the +following: + +[source,js] +-------------------------------- +PUT _cluster/settings +{ + "persistent": { + "cluster": { + "remote": { + "cluster_one": { + "seeds": [ + "127.0.0.1:9300" + ] + }, + "cluster_two": { + "seeds": [ + "127.0.0.1:9301" + ] + }, + "cluster_three": { + "seeds": [ + "127.0.0.1:9302" + ] + } + } + } + } +} +-------------------------------- +// CONSOLE +// TEST[setup:host] +// TEST[s/127.0.0.1:9300/\${transport_host}/] + +A remote cluster can be deleted from the cluster settings by setting its seeds +to `null`: + +[source,js] +-------------------------------- +PUT _cluster/settings +{ + "persistent": { + "cluster": { + "remote": { + "cluster_three": { + "seeds": null <1> + } + } + } + } +} +-------------------------------- +// CONSOLE +// TEST[continued] +<1> `cluster_three` would be removed from the cluster settings, leaving +`cluster_one` and `cluster_two` intact. + +[float] +[[remote-cluster-settings]] +=== Remote cluster settings + +`cluster.remote.connections_per_cluster`:: + + The number of gateway nodes to connect to per remote cluster. The default is + `3`. + +`cluster.remote.initial_connect_timeout`:: + + The time to wait for remote connections to be established when the node + starts. The default is `30s`. + +`cluster.remote.node.attr`:: + + A node attribute to filter out nodes that are eligible as a gateway node in + the remote cluster. For instance a node can have a node attribute + `node.attr.gateway: true` such that only nodes with this attribute will be + connected to if `cluster.remote.node.attr` is set to `gateway`. + +`cluster.remote.connect`:: + + By default, any node in the cluster can act as a cross-cluster client and + connect to remote clusters. The `cluster.remote.connect` setting can be set to + `false` (defaults to `true`) to prevent certain nodes from connecting to + remote clusters. Remote cluster requests must be sent to a node that is + allowed to act as a cross-cluster client. + +`cluster.remote.${cluster_alias}.skip_unavailable`:: + + Per cluster boolean setting that allows to skip specific clusters when no + nodes belonging to them are available and they are the targetof a remote + cluster request. Default is `false`, meaning that all clusters are mandatory + by default, but they can selectively be made optional by setting this setting + to `true`. + +[float] +[[retrieve-remote-clusters-info]] +=== Retrieving remote clusters info + +The <> allows to retrieve +information about the configured remote clusters, as well as the remote nodes +that the node is connected to.