Docs: Rewrote the upgrade section

This commit is contained in:
Clinton Gormley 2015-06-19 16:27:28 +02:00
parent 1ac728d22b
commit dd680669f5
4 changed files with 431 additions and 183 deletions

View File

@ -0,0 +1,86 @@
[[backup]]
=== Back Up Your Data!
Always back up your data before performing an upgrade. This will allow you to
roll back in the event of a problem. The upgrades sometimes include upgrades
to the Lucene libraries used by Elasticsearch to access the index files, and
after an index file has been updated to work with a new version of Lucene, it
may not be accessible to the versions of Lucene present in earlier
Elasticsearch releases.
[WARNING]
.Always back up your data before upgrading
=========================================
You cannot roll back to an earlier version unless you have a backup of your data.
=========================================
==== Backing up 1.0 and later
To back up a running 1.0 or later system, it is simplest to use the snapshot
feature. See the complete instructions for
<<modules-snapshots,backup and restore with snapshots>>.
==== Backing up 0.90.x and earlier
To back up a running 0.90.x system:
===== Step 1: Disable index flushing
This will prevent indices from being flushed to disk while the backup is in
process:
[source,json]
-----------------------------------
PUT /_all/_settings
{
"index": {
"translog.disable_flush": "true"
}
}
-----------------------------------
// AUTOSENSE
===== Step 2: Disable reallocation
This will prevent the cluster from moving data files from one node to another
while the backup is in process:
[source,json]
-----------------------------------
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.disable_allocation": "true"
}
}
-----------------------------------
// AUTOSENSE
===== Step 3: Backup your data
After reallocation and index flushing are disabled, initiate a backup of
Elasticsearch's data path using your favorite backup method (tar, storage
array snapshots, backup software).
===== Step 4: Reenable allocation and flushing
When the backup is complete and data no longer needs to be read from the
Elasticsearch data path, allocation and index flushing must be re-enabled:
[source,json]
-----------------------------------
PUT /_all/_settings
{
"index": {
"translog.disable_flush": "false"
}
}
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.disable_allocation": "false"
}
}
-----------------------------------
// AUTOSENSE

View File

@ -0,0 +1,149 @@
[[restart-upgrade]]
=== Full cluster restart upgrade
Elasticsearch requires a full cluster restart when upgrading across major
versions: from 0.x to 1.x or from 1.x to 2.x. Rolling upgrades are not
supported across major versions.
The process to perform an upgrade with a full cluster restart is as follows:
==== Step 1: Disable shard allocation
When you shut down a node, the allocation process will immediately try to
replicate the shards that were on that node to other nodes in the cluster,
causing a lot of wasted I/O. This can be avoided by disabling allocation
before shutting down a node:
[source,json]
--------------------------------------------------
PUT /_cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}
--------------------------------------------------
// AUTOSENSE
If upgrading from 0.90.x to 1.x, then use these settings instead:
[source,json]
--------------------------------------------------
PUT /_cluster/settings
{
"persistent": {
"cluster.routing.allocation.disable_allocation": false,
"cluster.routing.allocation.enable": "none"
}
}
--------------------------------------------------
// AUTOSENSE
==== Step 2: Perform a synced flush
added[1.6.0,Synced flush is only supported in Elasticsearch 1.6.0 and above]
Shard recovery will be much faster if you stop indexing and issue a
<<indices-synced-flush, synced-flush>> request:
[source,sh]
--------------------------------------------------
POST /_flush/synced
--------------------------------------------------
// AUTOSENSE
A synced flush request is a ``best effort'' operation. It will fail if there
are any pending indexing operations, but it is safe to reissue the request
multiple times if necessary.
==== Step 3: Shutdown and upgrade all nodes
Stop all Elasticsearch services on all nodes in the cluster. Each node can be
upgraded following the same procedure described in <<upgrade-node>>.
==== Step 4: Start the cluster
If you have dedicated master nodes -- nodes with `node.master` set to
`true`(the default) and `node.data` set to `false` -- then it is a good idea
to start them first. Wait for them to form a cluster and to elect a master
before proceding with the data nodes. You can check progress by looking at the
logs.
As soon as the <<master-election,minimum number of master-eligible nodes>>
have discovered each other, they will form a cluster and elect a master. From
that point on, the <<cat-health,`_cat/health`>> and <<cat-nodes,`_cat/nodes`>>
APIs can be used to monitor nodes joining the cluster:
[source,sh]
--------------------------------------------------
GET _cat/health
GET _cat/nodes
--------------------------------------------------
// AUTOSENSE
Use these APIs to check that all nodes have successfully joined the cluster.
==== Step 5: Wait for yellow
As soon as each node has joined the cluster, it will start to recover any
primary shards that are stored locally. Initially, the
<<cat-health,`_cat/health`>> request will report a `status` of `red`, meaning
that not all primary shards have been allocated.
Once each node has recovered its local shards, the `status` will become
`yellow`, meaning all primary shards have been recovered, but not all replica
shards are allocated. This is to be expected because allocation is still
disabled.
==== Step 6: Reenable allocation
Delaying the allocation of replicas until all nodes have joined the cluster
allows the master to allocate replicas to nodes which already have local shard
copies. At this point, with all the nodes in the cluster, it is safe to
reenable shard allocation:
[source,json]
------------------------------------------------------
PUT /_cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}
------------------------------------------------------
// AUTOSENSE
If upgrading from 0.90.x to 1.x, then use these settings instead:
[source,json]
--------------------------------------------------
PUT /_cluster/settings
{
"persistent": {
"cluster.routing.allocation.disable_allocation": true,
"cluster.routing.allocation.enable": "all"
}
}
--------------------------------------------------
// AUTOSENSE
The cluster will now start allocating replica shards to all data nodes. At this
point it is safe to resume indexing and searching, but your cluster will
recover more quickly if you can delay indexing and searching until all shards
have recovered.
You can monitor progress with the <<cat-health,`_cat/health`>> and
<<cat-recovery,`_cat/recovery`>> APIs:
[source,sh]
--------------------------------------------------
GET _cat/health
GET _cat/recovery
--------------------------------------------------
// AUTOSENSE
Once the `status` column in the `_cat/health` output has reached `green`, all
primary and replica shards have been successfully allocated.

View File

@ -0,0 +1,171 @@
[[rolling-upgrades]]
=== Rolling upgrades
A rolling upgrade allows the Elasticsearch cluster to be upgraded one node at
a time, with no downtime for end users. Running multiple versions of
Elasticsearch in the same cluster for any length of time beyond that required
for an upgrade is not supported, as shards will not be replicated from the
more recent version to the older version.
Consult this <<setup-upgrade,table>> to verify that rolling upgrades are
supported for your version of Elasticsearch.
To perform a rolling upgrade:
==== Step 1: Disable shard allocation
When you shut down a node, the allocation process will immediately try to
replicate the shards that were on that node to other nodes in the cluster,
causing a lot of wasted I/O. This can be avoided by disabling allocation
before shutting down a node:
[source,json]
--------------------------------------------------
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "none"
}
}
--------------------------------------------------
// AUTOSENSE
==== Step 2: Stop non-essential indexing and perform a synced flush (Optional)
added[1.6.0,Synced flush is only supported in Elasticsearch 1.6.0 and above]
You may happily continue indexing during the upgrade. However, shard recovery
will be much faster if you temporarily stop non-essential indexing and issue a
<<indices-synced-flush, synced-flush>> request:
[source,json]
--------------------------------------------------
POST /_flush/synced
--------------------------------------------------
// AUTOSENSE
A synced flush request is a ``best effort'' operation. It will fail if there
are any pending indexing operations, but it is safe to reissue the request
multiple times if necessary.
[[upgrade-node]]
==== Step 3: Stop and upgrade a single node
Shut down one of the nodes in the cluster *before* starting the upgrade.
[TIP]
================================================
When using the zip or tarball packages, the `config`, `data`, `logs` and
`plugins` directories are placed within the Elasticsearch home directory by
default.
It is a good idea to place these directories in a different location so that
there is no chance of deleting them when upgrading Elasticsearch. These
custom paths can be <<paths,configured>> with the `path.config` and
`path.data` settings.
The Debian and RPM packages place these directories in the
<<setup-dir-layout,appropriate place>> for each operating system.
================================================
To upgrade using a <<setup-repositories,Debian or RPM>> package:
* Use `rpm` or `dpkg` to install the new package. All files should be
placed in their proper locations, and config files should not be
overwritten.
To upgrade using a zip or compressed tarball:
* Extract the zip or tarball to a new directory, to be sure that you don't
overwrite the `config` or `data` directories.
* Either copy the files in the `config` directory from your old installation
to your new installation, or use the `--path.config` option on the command
line to point to an external config directory.
* Either copy the files in the `data` directory from your old installation
to your new installation, or configure the location of the data directory
in the `config/elasticsearch.yml` file, with the `path.data` setting.
==== Step 4: Start the upgraded node
Start the now upgraded node and confirm that it joins the cluster by checking
the log file or by checking the output of this request:
[source,sh]
--------------------------------------------------
GET _cat/nodes
--------------------------------------------------
// AUTOSENSE
==== Step 5: Reenable shard allocation
Once the node has joined the cluster, reenable shard allocation to start using
the node:
[source,json]
--------------------------------------------------
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}
--------------------------------------------------
// AUTOSENSE
==== Step 6: Wait for the node to recover
You should wait for the cluster to finish shard allocation before upgrading
the next node. You can check on progress with the <<cat-health,`_cat/health`>>
request:
[source,sh]
--------------------------------------------------
GET _cat/health
--------------------------------------------------
// AUTOSENSE
Wait for the `status` column to move from `yellow` to `green`. Status `green`
means that all primary and replica shards have been allocated.
[IMPORTANT]
====================================================
During a rolling upgrade, primary shards assigned to a node with the higher
version will never have their replicas assigned to a node with the lower
version, because the newer version may have a different data format which is
not understood by the older version.
If it is not possible to assign the replica shards to another node with the
higher version -- e.g. if there is only one node with the higher version in
the cluster -- then the replica shards will remain unassigned and the
cluster health will remain status `yellow`.
In this case, check that there are no initializing or relocating shards (the
`init` and `relo` columns) before proceding.
As soon as another node is upgraded, the replicas should be assigned and the
cluster health will reach status `green`.
====================================================
Shards that have not been <<synced-flush,sync-flushed>> may take some time to
recover. The recovery status of individual shards can be monitored with the
<<cat-recovery,`_cat/recovery`>> request:
[source,sh]
--------------------------------------------------
GET _cat/recovery
--------------------------------------------------
// AUTOSENSE
If you stopped indexing, then it is safe to resume indexing as soon as
recovery has completed.
==== Step 7: Repeat
When the cluster is stable and the node has recovered, repeat the above steps
for all remaining nodes.

View File

@ -1,194 +1,36 @@
[[setup-upgrade]]
== Upgrading
Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service. This section details how to perform both rolling and restart upgrades. To determine whether a rolling upgrade is supported for your release, please consult this table:
[cols="1,2,3",options="header",]
|=======================================================================
|Upgrade From |Upgrade To |Supported Upgrade Type
|0.90.x |1.x |Restart Upgrade
|< 0.90.7 |0.90.x |Restart Upgrade
|>= 0.90.7 |0.90.x |Rolling Upgrade
|1.x |1.x |Rolling Upgrade
|=======================================================================
TIP: Before upgrading Elasticsearch, it is a good idea to consult the
<<breaking-changes,breaking changes>> docs.
[float]
[[backup]]
=== Back Up Your Data!
Before performing an upgrade, it's a good idea to back up the data on your system. This will allow you to roll back in the event of a problem with the upgrade. The upgrades sometimes include upgrades to the Lucene libraries used by Elasticsearch to access the index files, and after an index file has been updated to work with a new version of Lucene, it may not be accessible to the versions of Lucene present in earlier Elasticsearch releases.
[float]
==== 0.90.x and earlier
To back up a running 0.90.x system, first disable index flushing. This will prevent indices from being flushed to disk while the backup is in process:
[source,sh]
-----------------------------------
$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
"index": {
"translog.disable_flush": "true"
}
}'
-----------------------------------
Then disable reallocation. This will prevent the cluster from moving data files from one node to another while the backup is in process:
[source,sh]
-----------------------------------
$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
"transient" : {
"cluster.routing.allocation.disable_allocation": "true"
}
}'
-----------------------------------
After reallocation and index flushing are disabled, initiate a backup of Elasticsearch's data path using your favorite backup method (tar, storage array snapshots, backup software). When the backup is complete and data no longer needs to be read from the Elasticsearch data path, reallocation and index flushing must be re-enabled:
[source,sh]
-----------------------------------
$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
"index": {
"translog.disable_flush": "false"
}
}'
$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
"transient" : {
"cluster.routing.allocation.disable_allocation": "false"
}
}'
-----------------------------------
[float]
==== 1.0 and later
To back up a running 1.0 or later system, it is simplest to use the snapshot feature. See the complete instructions for <<modules-snapshots,backup and restore with snapshots>>.
[float]
[[rolling-upgrades]]
=== Rolling upgrade process
A rolling upgrade allows the ES cluster to be upgraded one node at a time, with no observable downtime for end users. Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shard replication from the more recent version to the previous versions will not work.
Within minor or maintenance releases after release 1.0, rolling upgrades are supported. To perform a rolling upgrade:
* Disable shard reallocation (optional). This is done to allow for a faster startup after cluster shutdown. If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O. With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance. After startup is complete, reallocation will be turned back on.
This syntax applies to Elasticsearch 1.0 and later:
[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'
--------------------------------------------------
* There is no problem continuing to index while doing the upgrade. However, you can speed the process considerably
by *temporarily* stopping non-essential indexing and issuing a manual <<indices-synced-flush, synced flush>>.
A synced flush is special kind of flush which can seriously speed up recovery of shards. Elasticsearch automatically
uses it when an index has been inactive for a while (default is `30m`) but you can manually trigger it using the following command:
[source,sh]
--------------------------------------------------
curl -XPOST localhost:9200/_all/_flush/synced
--------------------------------------------------
Note that a synced flush call is a best effort operation. It will fail there are any pending indexing operations. It is safe to issue
it multiple times if needed.
* Shut down a single node within the cluster.
* Confirm that all shards are correctly reallocated to the remaining running nodes.
* Upgrade the stopped node. To upgrade using a zip or compressed tarball from elastic.co:
** Extract the zip or tarball to a new directory, usually in the same volume as the current Elasticsearch installation. Do not overwrite the existing installation, as the downloaded archive will contain a default elasticsearch.yml file and will overwrite your existing configuration.
** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory. Move data files from the old Elasticsesarch installation's data directory if necessary. If data files are not located within the tarball's extraction directory, they will not have to be moved.
** The simplest solution for moving from one version to another is to have a symbolic link for 'elasticsearch' that points to the currently running version. This link can be easily updated and will provide a stable access point to the most recent version. Update this symbolic link if it is being used.
* To upgrade using a `.deb` or `.rpm` package:
** Use `rpm` or `dpkg` to install the new package. All files should be placed in their proper locations, and config files should not be overwritten.
* Start the now upgraded node. Confirm that it joins the cluster.
* Re-enable shard reallocation:
[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
--------------------------------------------------
* Observe that all shards are properly allocated on all nodes. Balancing may take some time.
* Repeat this process for all remaining nodes.
[IMPORTANT]
====================================================
During a rolling upgrade, primary shards assigned to a node with the higher
version will never have their replicas assigned to a node with the lower
version, because the newer version may have a different data format which is
not understood by the older version.
===========================================
Before upgrading Elasticsearch:
If it is not possible to assign the replica shards to another node with the
higher version -- e.g. if there is only one node with the higher version in
the cluster -- then the replica shards will remain unassigned, i.e. the
cluster health will be status `yellow`. As soon as another node with the
higher version joins the cluster, the replicas should be assigned and the
cluster health will reach status `green`.
====================================================
* Consult the <<breaking-changes,breaking changes>> docs.
* Test upgrades in a dev environment before upgrading your production cluster.
* Always back up your data before upgrading. You cannot roll back to an earlier
version unless you have a <<backup,backup>> of your data.
===========================================
It may be possible to perform the upgrade by installing the new software while the service is running. This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded. This can be done by installing the new version in its own directory and using the symbolic link method outlined above. It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.
Elasticsearch can usually be upgraded using a rolling upgrade process,
resulting in no interruption of service. This section details how to perform
both rolling upgrades and upgrades with full cluster restarts.
[float]
[[restart-upgrade]]
=== Cluster restart upgrade process
To determine whether a rolling upgrade is supported for your release, please
consult this table:
Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible. In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required. In order to perform this upgrade:
[cols="1<m,1<m,3",options="header",]
|=======================================================================
|Upgrade From |Upgrade To |Supported Upgrade Type
|0.90.x |1.x, 2.x |<<restart-upgrade,Full cluster restart>>
|< 0.90.7 |0.90.x |<<restart-upgrade,Full cluster restart>>
|>= 0.90.7 |0.90.x |<<rolling-upgrades,Rolling upgrade>>
|1.0.0 - 1.3.1 |1.x |<<rolling-upgrades,Rolling upgrade>> (if <<recovery,`indices.recovery.compress`>> set to `false`)
|>= 1.3.2 |1.x |<<rolling-upgrades,Rolling upgrade>>
|1.x |2.x |<<restart-upgrade,Full cluster restart>>
|=======================================================================
* Disable shard reallocation (optional). This is done to allow for a faster startup after cluster shutdown. If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O. With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance. After startup is complete, reallocation will be turned back on.
include::backup.asciidoc[]
This syntax is from versions prior to 1.0:
include::rolling_upgrade.asciidoc[]
[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation" : true
}
}'
--------------------------------------------------
* Stop all Elasticsearch services on all nodes in the cluster.
* On the first node to be upgraded, extract the archive or install the new package as described above in the Rolling Upgrades section. Repeat for all nodes.
* After upgrading Elasticsearch on all nodes is complete, the cluster can be started by starting each node individually.
** Start master-eligible nodes first, one at a time. Verify that a quorum has been reached and a master has been elected before proceeding.
** Start data nodes and then client nodes one at a time, verifying that they successfully join the cluster.
* When the cluster is running and reaches a yellow state, shard reallocation can be enabled.
This syntax is from release 1.0 and later:
[source,sh]
------------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation": false,
"cluster.routing.allocation.enable" : "all"
}
}'
------------------------------------------------------
The cluster upgrade can be streamlined by installing the software before stopping cluster services. If this is done, testing must be performed to ensure that no production data or configuration files are overwritten prior to restart.
include::cluster_restart.asciidoc[]