[[rolling-upgrades]] == Rolling upgrades A rolling upgrade allows an {es} cluster to be upgraded one node at a time so upgrading does not interrupt service. Running multiple versions of {es} in the same cluster beyond the duration of an upgrade is not supported, as shards cannot be replicated from upgraded nodes to nodes running the older version. We strongly recommend that when you upgrade you divide your cluster's nodes into the following two groups and upgrade the groups in this order: . Nodes that are not <>. You can retrieve a list of these nodes with `GET /_nodes/_all,master:false` or by finding all the nodes configured with `node.master: false`. . Master-eligible nodes, which are the remaining nodes. You can retrieve a list of these nodes with `GET /_nodes/master:true`. You may upgrade the nodes within each of these groups in any order. Upgrading the nodes in this order ensures that the master-ineligible nodes are always running a version at least as new as the master-eligible nodes. Newer nodes can always join a cluster with an older master, but older nodes cannot always join a cluster with a newer master. By upgrading the master-eligible nodes last you ensure that all the master-ineligible nodes will be able to join the cluster whether the master-eligible nodes have been upgraded or not. If you upgrade any master-eligible nodes before the master-ineligible nodes then there is a risk that the older nodes will leave the cluster and will not be able to rejoin until they have been upgraded. Rolling upgrades are supported: * Between minor versions * {stack-ref-68}/upgrading-elastic-stack.html[From 5.6 to 6.8] * From 6.8 to {version} ifeval::[ "{version}" != "{minor-version}.0" ] * From any version since {minor-version}.0 to {version} endif::[] Upgrading directly to {version} from 6.7 or earlier requires a <>. include::preparing_to_upgrade.asciidoc[] [float] === Upgrading your cluster To perform a rolling upgrade to {version}: . *Disable shard allocation*. + -- include::disable-shard-alloc.asciidoc[] -- . *Stop non-essential indexing and perform a synced flush.* (Optional) + -- While you can continue indexing during the upgrade, shard recovery is much faster if you temporarily stop non-essential indexing and perform a <>. include::synced-flush.asciidoc[] -- . *Temporarily stop the tasks associated with active {ml} jobs and {dfeeds}.* (Optional) + -- include::close-ml.asciidoc[] -- . [[upgrade-node]] *Shut down a single node*. + -- include::shut-down-node.asciidoc[] -- . *Upgrade the node you shut down.* + -- include::upgrade-node.asciidoc[] include::set-paths-tip.asciidoc[] [[rolling-upgrades-bootstrapping]] NOTE: You should leave `cluster.initial_master_nodes` unset while performing a rolling upgrade. Each upgraded node is joining an existing cluster so there is no need for <>. You must configure <> on every node. -- . *Upgrade any plugins.* + Use the `elasticsearch-plugin` script to install the upgraded version of each installed {es} plugin. All plugins must be upgraded when you upgrade a node. . If you use {es} {security-features} to define realms, verify that your realm settings are up-to-date. The format of realm settings changed in version 7.0, in particular, the placement of the realm type changed. See <>. . *Start the upgraded node.* + -- Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a `_cat/nodes` request: [source,console] -------------------------------------------------- GET _cat/nodes -------------------------------------------------- -- . *Reenable shard allocation.* + -- Once the node has joined the cluster, remove the `cluster.routing.allocation.enable` setting to enable shard allocation and start using the node: [source,console] -------------------------------------------------- PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": null } } -------------------------------------------------- -- . *Wait for the node to recover.* + -- Before upgrading the next node, wait for the cluster to finish shard allocation. You can check progress by submitting a <> request: [source,console] -------------------------------------------------- GET _cat/health?v -------------------------------------------------- Wait for the `status` column to switch to `green`. Once the node is `green`, all primary and replica shards have been allocated. [IMPORTANT] ==================================================== During a rolling upgrade, primary shards assigned to a node running the new version cannot have their replicas assigned to a node with the old version. The new version might have a different data format that is not understood by the old version. If it is not possible to assign the replica shards to another node (there is only one upgraded node in the cluster), the replica shards remain unassigned and status stays `yellow`. In this case, you can proceed once there are no initializing or relocating shards (check the `init` and `relo` columns). As soon as another node is upgraded, the replicas can be assigned and the status will change to `green`. ==================================================== Shards that were not <> might take longer to recover. You can monitor the recovery status of individual shards by submitting a <> request: [source,console] -------------------------------------------------- GET _cat/recovery -------------------------------------------------- If you stopped indexing, it is safe to resume indexing as soon as recovery completes. -- . *Repeat* + -- When the node has recovered and the cluster is stable, repeat these steps for each node that needs to be updated. You can monitor the health of the cluster with a <> request: [source,console] -------------------------------------------------- GET /_cat/health?v -------------------------------------------------- And check which nodes have been upgraded with a <> request: [source,console] -------------------------------------------------- GET /_cat/nodes?h=ip,name,version&v -------------------------------------------------- -- . *Restart machine learning jobs.* + -- include::open-ml.asciidoc[] -- [IMPORTANT] ==================================================== During a rolling upgrade, the cluster continues to operate normally. However, any new functionality is disabled or operates in a backward compatible mode until all nodes in the cluster are upgraded. New functionality becomes operational once the upgrade is complete and all nodes are running the new version. Once that has happened, there's no way to return to operating in a backward compatible mode. Nodes running the previous major version will not be allowed to join the fully-updated cluster. In the unlikely case of a network malfunction during the upgrade process that isolates all remaining old nodes from the cluster, you must take the old nodes offline and upgrade them to enable them to join the cluster. If you stop half or more of the master-eligible nodes all at once during the upgrade then the cluster will become unavailable, meaning that the upgrade is no longer a _rolling_ upgrade. If this happens, you should upgrade and restart all of the stopped master-eligible nodes to allow the cluster to form again, as if performing a <>. It may also be necessary to upgrade all of the remaining old nodes before they can join the cluster after it re-forms. ====================================================