232 lines
7.7 KiB
Plaintext
232 lines
7.7 KiB
Plaintext
[[rolling-upgrades]]
|
|
== Rolling upgrades
|
|
|
|
A rolling upgrade allows an {es} cluster to be upgraded one node at
|
|
a time so upgrading does not interrupt service. Running multiple versions of
|
|
{es} in the same cluster beyond the duration of an upgrade is
|
|
not supported, as shards cannot be replicated from upgraded nodes to nodes
|
|
running the older version.
|
|
|
|
We strongly recommend that when you upgrade you divide your cluster's nodes
|
|
into the following two groups and upgrade the groups in this order:
|
|
|
|
. Nodes that are not <<master-node,master-eligible>>. You can retrieve a list
|
|
of these nodes with `GET /_nodes/_all,master:false` or by finding all the nodes
|
|
configured with `node.master: false`.
|
|
|
|
. Master-eligible nodes, which are the remaining nodes. You can retrieve a list
|
|
of these nodes with `GET /_nodes/master:true`.
|
|
|
|
You may upgrade the nodes within each of these groups in any order.
|
|
|
|
Upgrading the nodes in this order ensures that the master-ineligible nodes are
|
|
always running a version at least as new as the master-eligible nodes. Newer
|
|
nodes can always join a cluster with an older master, but older nodes cannot
|
|
always join a cluster with a newer master. By upgrading the master-eligible
|
|
nodes last you ensure that all the master-ineligible nodes will be able to join
|
|
the cluster whether the master-eligible nodes have been upgraded or not. If you
|
|
upgrade any master-eligible nodes before the master-ineligible nodes then there
|
|
is a risk that the older nodes will leave the cluster and will not be able to
|
|
rejoin until they have been upgraded.
|
|
|
|
Rolling upgrades are supported:
|
|
|
|
* Between minor versions
|
|
* {stack-ref-68}/upgrading-elastic-stack.html[From 5.6 to 6.8]
|
|
* From 6.8 to {version}
|
|
|
|
Upgrading directly to {version} from 6.7 or earlier requires a
|
|
<<restart-upgrade, full cluster restart>>.
|
|
|
|
include::preparing_to_upgrade.asciidoc[]
|
|
|
|
[float]
|
|
=== Upgrading your cluster
|
|
|
|
To perform a rolling upgrade to {version}:
|
|
|
|
. *Disable shard allocation*.
|
|
+
|
|
--
|
|
include::disable-shard-alloc.asciidoc[]
|
|
--
|
|
|
|
. *Stop non-essential indexing and perform a synced flush.* (Optional)
|
|
+
|
|
--
|
|
While you can continue indexing during the upgrade, shard recovery
|
|
is much faster if you temporarily stop non-essential indexing and perform a
|
|
<<indices-synced-flush-api, synced-flush>>.
|
|
|
|
include::synced-flush.asciidoc[]
|
|
|
|
--
|
|
|
|
. *Temporarily stop the tasks associated with active {ml} jobs and {dfeeds}.* (Optional)
|
|
+
|
|
--
|
|
include::close-ml.asciidoc[]
|
|
--
|
|
|
|
. [[upgrade-node]] *Shut down a single node*.
|
|
+
|
|
--
|
|
include::shut-down-node.asciidoc[]
|
|
--
|
|
|
|
. *Upgrade the node you shut down.*
|
|
+
|
|
--
|
|
include::upgrade-node.asciidoc[]
|
|
include::set-paths-tip.asciidoc[]
|
|
|
|
[[rolling-upgrades-bootstrapping]]
|
|
NOTE: You should leave `cluster.initial_master_nodes` unset while performing a
|
|
rolling upgrade. Each upgraded node is joining an existing cluster so there is
|
|
no need for <<modules-discovery-bootstrap-cluster,cluster bootstrapping>>. You
|
|
must configure <<built-in-hosts-providers,either `discovery.seed_hosts` or
|
|
`discovery.seed_providers`>> on every node.
|
|
--
|
|
|
|
. *Upgrade any plugins.*
|
|
+
|
|
Use the `elasticsearch-plugin` script to install the upgraded version of each
|
|
installed {es} plugin. All plugins must be upgraded when you upgrade
|
|
a node.
|
|
|
|
. If you use {es} {security-features} to define realms, verify that your realm
|
|
settings are up-to-date. The format of realm settings changed in version 7.0, in
|
|
particular, the placement of the realm type changed. See
|
|
<<realm-settings,Realm settings>>.
|
|
|
|
. *Start the upgraded node.*
|
|
+
|
|
--
|
|
|
|
Start the newly-upgraded node and confirm that it joins the cluster by checking
|
|
the log file or by submitting a `_cat/nodes` request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET _cat/nodes
|
|
--------------------------------------------------
|
|
--
|
|
|
|
. *Reenable shard allocation.*
|
|
+
|
|
--
|
|
|
|
Once the node has joined the cluster, remove the `cluster.routing.allocation.enable`
|
|
setting to enable shard allocation and start using the node:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _cluster/settings
|
|
{
|
|
"persistent": {
|
|
"cluster.routing.allocation.enable": null
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
--
|
|
|
|
. *Wait for the node to recover.*
|
|
+
|
|
--
|
|
|
|
Before upgrading the next node, wait for the cluster to finish shard allocation.
|
|
You can check progress by submitting a <<cat-health,`_cat/health`>> request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET _cat/health?v
|
|
--------------------------------------------------
|
|
|
|
Wait for the `status` column to switch from `yellow` to `green`. Once the
|
|
node is `green`, all primary and replica shards have been allocated.
|
|
|
|
[IMPORTANT]
|
|
====================================================
|
|
During a rolling upgrade, primary shards assigned to a node running the new
|
|
version cannot have their replicas assigned to a node with the old
|
|
version. The new version might have a different data format that is
|
|
not understood by the old version.
|
|
|
|
If it is not possible to assign the replica shards to another node
|
|
(there is only one upgraded node in the cluster), the replica
|
|
shards remain unassigned and status stays `yellow`.
|
|
|
|
In this case, you can proceed once there are no initializing or relocating shards
|
|
(check the `init` and `relo` columns).
|
|
|
|
As soon as another node is upgraded, the replicas can be assigned and the
|
|
status will change to `green`.
|
|
====================================================
|
|
|
|
Shards that were not <<indices-synced-flush-api,sync-flushed>> might take longer to
|
|
recover. You can monitor the recovery status of individual shards by
|
|
submitting a <<cat-recovery,`_cat/recovery`>> request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET _cat/recovery
|
|
--------------------------------------------------
|
|
|
|
If you stopped indexing, it is safe to resume indexing as soon as
|
|
recovery completes.
|
|
--
|
|
|
|
. *Repeat*
|
|
+
|
|
--
|
|
|
|
When the node has recovered and the cluster is stable, repeat these steps
|
|
for each node that needs to be updated. You can monitor the health of the cluster
|
|
with a <<cat-health,`_cat/health`>> request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_cat/health?v
|
|
--------------------------------------------------
|
|
|
|
And check which nodes have been upgraded with a <<cat-nodes,`_cat/nodes`>> request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_cat/nodes?h=ip,name,version&v
|
|
--------------------------------------------------
|
|
|
|
--
|
|
|
|
. *Restart machine learning jobs.*
|
|
+
|
|
--
|
|
include::open-ml.asciidoc[]
|
|
--
|
|
|
|
|
|
[IMPORTANT]
|
|
====================================================
|
|
|
|
During a rolling upgrade, the cluster continues to operate normally. However,
|
|
any new functionality is disabled or operates in a backward compatible mode
|
|
until all nodes in the cluster are upgraded. New functionality becomes
|
|
operational once the upgrade is complete and all nodes are running the new
|
|
version. Once that has happened, there's no way to return to operating in a
|
|
backward compatible mode. Nodes running the previous major version will not be
|
|
allowed to join the fully-updated cluster.
|
|
|
|
In the unlikely case of a network malfunction during the upgrade process that
|
|
isolates all remaining old nodes from the cluster, you must take the old nodes
|
|
offline and upgrade them to enable them to join the cluster.
|
|
|
|
If you stop half or more of the master-eligible nodes all at once during the
|
|
upgrade then the cluster will become unavailable, meaning that the upgrade is
|
|
no longer a _rolling_ upgrade. If this happens, you should upgrade and restart
|
|
all of the stopped master-eligible nodes to allow the cluster to form again, as
|
|
if performing a <<restart-upgrade,full-cluster restart upgrade>>. It may also
|
|
be necessary to upgrade all of the remaining old nodes before they can join the
|
|
cluster after it re-forms.
|
|
|
|
====================================================
|