[[restart-upgrade]]
== Full cluster restart upgrade

To upgrade directly to {es} {version} from versions 6.0-6.7, you must shut down
all nodes in the cluster, upgrade each node to {version}, and restart the cluster.

NOTE: If you are running a version prior to 6.0,
{stack-ref-68}/upgrading-elastic-stack.html[upgrade to 6.8]
and reindex your old indices or bring up a new {version} cluster and
<<reindex-upgrade-remote, reindex from remote>>.

include::preparing_to_upgrade.asciidoc[]

[discrete]
=== Upgrading your cluster

To perform a full cluster restart upgrade to {version}:

. *Disable shard allocation.*
+
--
include::disable-shard-alloc.asciidoc[]
--

. *Stop indexing and perform a synced flush.*
+
--
Performing a <<indices-synced-flush-api, synced-flush>> speeds up shard
recovery.

include::synced-flush.asciidoc[]
--

. *Temporarily stop the tasks associated with active {ml} jobs and {dfeeds}.* (Optional)
+
--
include::close-ml.asciidoc[]
--

. *Shutdown all nodes.*
+
--
include::shut-down-node.asciidoc[]
--

. *Upgrade all nodes.*
+
--
include::remove-xpack.asciidoc[]
--
+
--
include::upgrade-node.asciidoc[]
--
+
--
include::set-paths-tip.asciidoc[]
--

If upgrading from a 6.x cluster, you must also
<<modules-discovery-bootstrap-cluster,configure cluster bootstrapping>> by
setting the <<initial_master_nodes,`cluster.initial_master_nodes` setting>> on
the master-eligible nodes.

. *Upgrade any plugins.*
+
Use the `elasticsearch-plugin` script to install the upgraded version of each
installed {es} plugin. All plugins must be upgraded when you upgrade
a node.

. If you use {es} {security-features} to define realms, verify that your realm
settings are up-to-date. The format of realm settings changed in version 7.0, in
particular, the placement of the realm type changed. See
<<realm-settings,Realm settings>>. 

. *Start each upgraded node.*
+
--
If you have dedicated master nodes, start them first and wait for them to
form a cluster and elect a master before proceeding with your data nodes.
You can check progress by looking at the logs.

As soon as enough master-eligible nodes have discovered each other, they form a
cluster and elect a master. At that point, you can use
<<cat-health,`_cat/health`>> and <<cat-nodes,`_cat/nodes`>> to monitor nodes
joining the cluster:

[source,console]
--------------------------------------------------
GET _cat/health

GET _cat/nodes
--------------------------------------------------

The `status` column returned by `_cat/health` shows the health of each node
in the cluster: `red`, `yellow`, or `green`.
--

. *Wait for all nodes to join the cluster and report a status of yellow.*
+
--
When a node joins the cluster, it begins to recover any primary shards that
are stored locally. The <<cat-health,`_cat/health`>> API initially reports
a `status` of `red`, indicating that not all primary shards have been allocated.

Once a node recovers its local shards, the cluster `status` switches to `yellow`,
indicating that all primary shards have been recovered, but not all replica
shards are allocated. This is to be expected because you have not yet
reenabled allocation. Delaying the allocation of replicas until all nodes
are `yellow` allows the master to allocate replicas to nodes that
already have local shard copies.
--

. *Reenable allocation.*
+
--
When all nodes have joined the cluster and recovered their primary shards,
reenable allocation by restoring `cluster.routing.allocation.enable` to its
default:

[source,console]
------------------------------------------------------
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}
------------------------------------------------------

Once allocation is reenabled, the cluster starts allocating replica shards to
the data nodes. At this point it is safe to resume indexing and searching,
but your cluster will recover more quickly if you can wait until all primary
and replica shards have been successfully allocated and the status of all nodes
is `green`.

You can monitor progress with the <<cat-health,`_cat/health`>> and
<<cat-recovery,`_cat/recovery`>> APIs:

[source,console]
--------------------------------------------------
GET _cat/health

GET _cat/recovery
--------------------------------------------------
--

. *Restart machine learning jobs.*
+
--
include::open-ml.asciidoc[]
--