2013-08-28 19:24:34 -04:00
|
|
|
[[cluster-reroute]]
|
|
|
|
== Cluster Reroute
|
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
The reroute command allows for manual changes to the allocation of individual
|
|
|
|
shards in the cluster. For example, a shard can be moved from one node to
|
|
|
|
another explicitly, an allocation can be cancelled, and an unassigned shard can
|
|
|
|
be explicitly allocated to a specific node.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
Here is a short example of a simple reroute API call:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
2017-02-07 16:24:05 -05:00
|
|
|
POST /_cluster/reroute
|
|
|
|
{
|
|
|
|
"commands" : [
|
|
|
|
{
|
|
|
|
"move" : {
|
|
|
|
"index" : "test", "shard" : 0,
|
|
|
|
"from_node" : "node1", "to_node" : "node2"
|
2013-08-28 19:24:34 -04:00
|
|
|
}
|
|
|
|
},
|
|
|
|
{
|
2016-01-13 10:59:39 -05:00
|
|
|
"allocate_replica" : {
|
2017-02-07 16:24:05 -05:00
|
|
|
"index" : "test", "shard" : 1,
|
|
|
|
"node" : "node3"
|
2013-08-28 19:24:34 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
]
|
2017-02-07 16:24:05 -05:00
|
|
|
}
|
2013-08-28 19:24:34 -04:00
|
|
|
--------------------------------------------------
|
2017-02-07 16:24:05 -05:00
|
|
|
// CONSOLE
|
|
|
|
// TEST[skip:doc tests run with only a single node]
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
It is important to note that that after processing any reroute commands
|
|
|
|
Elasticsearch will perform rebalancing as normal (respecting the values of
|
|
|
|
settings such as `cluster.routing.rebalance.enable`) in order to remain in a
|
|
|
|
balanced state. For example, if the requested allocation includes moving a
|
|
|
|
shard from `node1` to `node2` then this may cause a shard to be moved from
|
|
|
|
`node2` back to `node1` to even things out.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
The cluster can be set to disable allocations using the
|
|
|
|
`cluster.routing.allocation.enable` setting. If allocations are disabled then
|
|
|
|
the only allocations that will be performed are explicit ones given using the
|
|
|
|
`reroute` command, and consequent allocations due to rebalancing.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
It is possible to run `reroute` commands in "dry run" mode by using the
|
|
|
|
`?dry_run` URI query parameter, or by passing `"dry_run": true` in the request
|
|
|
|
body. This will calculate the result of applying the commands to the current
|
|
|
|
cluster state, and return the resulting cluster state after the commands (and
|
|
|
|
re-balancing) has been applied, but will not actually perform the requested
|
|
|
|
changes.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
If the `?explain` URI query parameter is included then a detailed explanation
|
|
|
|
of why the commands could or could not be executed is included in the response.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
Add `explain` flag support to the reroute API
By specifying the `explain` flag, an explanation for the reason a
command can or cannot be executed is returned. No allocation commands
are actually performed.
Returns a response similar to:
{
"state": {...cluster state...},
"acknowledged": true,
"explanations" : [ {
"command" : "cancel",
"parameters" : {
"index" : "decide",
"shard" : 0,
"node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"allow_primary" : false
},
"decisions" : [ {
"decider" : "cancel_allocation_command",
"decision" : "YES",
"explanation" : "..."
} ]
}, {
"command" : "move",
"parameters" : {
"index" : "decide",
"shard" : 0,
"from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
},
"decisions" : [ {
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
},
etc
]
}]
}
also removes AllocationExplanation from cluster state
Closes #2483
Closes #5169
2014-01-31 18:50:32 -05:00
|
|
|
The commands supported are:
|
|
|
|
|
|
|
|
`move`::
|
2013-08-28 19:24:34 -04:00
|
|
|
Move a started shard from one node to another node. Accepts
|
|
|
|
`index` and `shard` for index name and shard number, `from_node` for the
|
2018-04-30 08:09:03 -04:00
|
|
|
node to move the shard from, and `to_node` for the node to move the
|
Add `explain` flag support to the reroute API
By specifying the `explain` flag, an explanation for the reason a
command can or cannot be executed is returned. No allocation commands
are actually performed.
Returns a response similar to:
{
"state": {...cluster state...},
"acknowledged": true,
"explanations" : [ {
"command" : "cancel",
"parameters" : {
"index" : "decide",
"shard" : 0,
"node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"allow_primary" : false
},
"decisions" : [ {
"decider" : "cancel_allocation_command",
"decision" : "YES",
"explanation" : "..."
} ]
}, {
"command" : "move",
"parameters" : {
"index" : "decide",
"shard" : 0,
"from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
},
"decisions" : [ {
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
},
etc
]
}]
}
also removes AllocationExplanation from cluster state
Closes #2483
Closes #5169
2014-01-31 18:50:32 -05:00
|
|
|
shard to.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
Add `explain` flag support to the reroute API
By specifying the `explain` flag, an explanation for the reason a
command can or cannot be executed is returned. No allocation commands
are actually performed.
Returns a response similar to:
{
"state": {...cluster state...},
"acknowledged": true,
"explanations" : [ {
"command" : "cancel",
"parameters" : {
"index" : "decide",
"shard" : 0,
"node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"allow_primary" : false
},
"decisions" : [ {
"decider" : "cancel_allocation_command",
"decision" : "YES",
"explanation" : "..."
} ]
}, {
"command" : "move",
"parameters" : {
"index" : "decide",
"shard" : 0,
"from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
},
"decisions" : [ {
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
},
etc
]
}]
}
also removes AllocationExplanation from cluster state
Closes #2483
Closes #5169
2014-01-31 18:50:32 -05:00
|
|
|
`cancel`::
|
2018-04-30 08:09:03 -04:00
|
|
|
Cancel allocation of a shard (or recovery). Accepts `index` and `shard` for
|
|
|
|
index name and shard number, and `node` for the node to cancel the shard
|
|
|
|
allocation on. This can be used to force resynchronization of existing
|
|
|
|
replicas from the primary shard by cancelling them and allowing them to be
|
|
|
|
reinitialized through the standard recovery process. By default only
|
|
|
|
replica shard allocations can be cancelled. If it is necessary to cancel
|
|
|
|
the allocation of a primary shard then the `allow_primary` flag must also
|
|
|
|
be included in the request.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2016-01-13 10:59:39 -05:00
|
|
|
`allocate_replica`::
|
2018-04-30 08:09:03 -04:00
|
|
|
Allocate an unassigned replica shard to a node. Accepts `index` and `shard`
|
|
|
|
for index name and shard number, and `node` to allocate the shard to. Takes
|
|
|
|
<<modules-cluster,allocation deciders>> into account.
|
2016-01-13 10:59:39 -05:00
|
|
|
|
2018-03-20 12:53:48 -04:00
|
|
|
[float]
|
2018-04-30 08:09:03 -04:00
|
|
|
=== Retrying failed allocations
|
2018-03-20 12:53:48 -04:00
|
|
|
|
|
|
|
The cluster will attempt to allocate a shard a maximum of
|
|
|
|
`index.allocation.max_retries` times in a row (defaults to `5`), before giving
|
|
|
|
up and leaving the shard unallocated. This scenario can be caused by
|
|
|
|
structural problems such as having an analyzer which refers to a stopwords
|
|
|
|
file which doesn't exist on all nodes.
|
|
|
|
|
|
|
|
Once the problem has been corrected, allocation can be manually retried by
|
2018-04-30 08:09:03 -04:00
|
|
|
calling the <<cluster-reroute,`reroute`>> API with the `?retry_failed` URI
|
|
|
|
query parameter, which will attempt a single retry round for these shards.
|
2018-03-20 12:53:48 -04:00
|
|
|
|
|
|
|
[float]
|
|
|
|
=== Forced allocation on unrecoverable errors
|
|
|
|
|
2018-04-30 08:09:03 -04:00
|
|
|
Two more commands are available that allow the allocation of a primary shard to
|
|
|
|
a node. These commands should however be used with extreme care, as primary
|
|
|
|
shard allocation is usually fully automatically handled by Elasticsearch.
|
|
|
|
Reasons why a primary shard cannot be automatically allocated include the
|
|
|
|
following:
|
|
|
|
|
|
|
|
- A new index was created but there is no node which satisfies the allocation
|
|
|
|
deciders.
|
|
|
|
- An up-to-date shard copy of the data cannot be found on the current data
|
|
|
|
nodes in the cluster. To prevent data loss, the system does not automatically
|
|
|
|
promote a stale shard copy to primary.
|
|
|
|
|
2018-03-20 12:53:48 -04:00
|
|
|
The following two commands are dangerous and may result in data loss. They are
|
2018-04-30 08:09:03 -04:00
|
|
|
meant to be used in cases where the original data can not be recovered and the
|
|
|
|
cluster administrator accepts the loss. If you have suffered a temporary issue
|
|
|
|
that can be fixed, please see the `retry_failed` flag described above. To
|
|
|
|
emphasise: if these commands are performed and then a node joins the cluster
|
|
|
|
that holds a copy of the affected shard then the copy on the newly-joined node
|
|
|
|
will be deleted or overwritten.
|
2016-01-13 10:59:39 -05:00
|
|
|
|
|
|
|
`allocate_stale_primary`::
|
|
|
|
Allocate a primary shard to a node that holds a stale copy. Accepts the
|
2018-04-30 08:09:03 -04:00
|
|
|
`index` and `shard` for index name and shard number, and `node` to allocate
|
|
|
|
the shard to. Using this command may lead to data loss for the provided
|
|
|
|
shard id. If a node which has the good copy of the data rejoins the cluster
|
|
|
|
later on, that data will be deleted or overwritten with the data of the
|
|
|
|
stale copy that was forcefully allocated with this command. To ensure that
|
|
|
|
these implications are well-understood, this command requires the flag
|
|
|
|
`accept_data_loss` to be explicitly set to `true`.
|
2016-01-13 10:59:39 -05:00
|
|
|
|
|
|
|
`allocate_empty_primary`::
|
2018-04-30 08:09:03 -04:00
|
|
|
Allocate an empty primary shard to a node. Accepts the `index` and `shard`
|
|
|
|
for index name and shard number, and `node` to allocate the shard to. Using
|
|
|
|
this command leads to a complete loss of all data that was indexed into
|
|
|
|
this shard, if it was previously started. If a node which has a copy of the
|
|
|
|
data rejoins the cluster later on, that data will be deleted. To ensure
|
|
|
|
that these implications are well-understood, this command requires the flag
|
|
|
|
`accept_data_loss` to be explicitly set to `true`.
|
2016-05-20 14:37:45 -04:00
|
|
|
|