Jeff Huss 9fd6cf9caf
Add steps for upgrading OpenSearch with Security enabled (#3052)
* Adding appendix that I want to use for reading aids or additional context for the user and reader

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Saving - trying to multitask and follow a migrations workshop so not a lot of progress should be expected today

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Working on splitting up the content

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Sanitized example commands

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added intro blurb to the appendix - I will revisit the wording after the doc grows a bit to see how much sense it makes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Redid headers and added cross-link

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Changed headings again to make them simpler

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Expanding on intro and starting to add steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Removed disclaimer about prerequisite skills since I backed-off with the specifics in this guide

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding more context to the overview

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added more details to the snapshot recommendation including links to the supported repository solutions

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Wording changes in appendix

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding process steps and testing the copy html button function - so far so good as long as indentation is correct

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Stashing changes because I need to get the script uploaded to the repo so I can make sure the permalink works

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Working out steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added one-liner for resetting the environment

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added one-liner for resetting the environment

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding more steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Still adding

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Reorganizing the page layout

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Improving landing page

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Breaking up the content for sustainability

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Phrasing

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Clip index page to keep it concise - I think Jekyll will automatically add a related articles section since it has_children is true

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Clean up environment

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Clean up environment

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Testing out codeblock labels

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Stashing changes while I pull down a remote branch for a PR review

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Continuing to build out process and added code block labels to improve readability

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Stashing to check another branch for PR review

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Continuing to add steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added headings to backups section so I can link the bullet list

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding steps and cleaning up sections for readability

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Completed some additional research on security and started adding steps with commentary

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Wrapping up security settings backup procedure

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Stashing changes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Upgrade steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding upgrade steps

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added final steps to the actual upgrade procedure

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added steps for upgrading OpenSearch Dashboards

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Almost forgot to re-enabled shard replica allocation

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Wrapped up first part of validation and working on next

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding more checks

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Finished adding steps which completes first draft now starting a first round review of the doc

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Fixed typo

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Reordering of intro and removal of a tip box in favor of just adding to the existing sentences

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Further trimmed intro and removed the other box

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Fixed wording in intro

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Fixed wording and ordering in set up the environment

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Revisions

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Gerunds

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Cleaning up phrasing

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added tip about curl

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Make wording more reader-friendly and informative

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Make wording more reader-friendly and informative

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Doc review edits

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Changed h1 and title to sentence case per other review comments

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Implementing review feedback on h2 header

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Making gerunds

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Editorial fixes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Final editorial changes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Final editorial changes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

---------

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>
2023-03-10 13:43:20 -08:00

13 KiB

layout title nav_order has_children redirect_from
default Upgrading OpenSearch 4 true
/upgrade-opensearch/index/

Upgrading OpenSearch

The OpenSearch Project releases regular updates that include new features, enhancements, and bug fixes. OpenSearch uses Semantic Versioning, which means that breaking changes are only introduced between major version releases. To learn about upcoming features and fixes, review the OpenSearch Project Roadmap on GitHub. To view a list of previous releases or to learn more about how OpenSearch uses versioning, see Release Schedule and Maintenance Policy.

We recognize that users are excited about upgrading OpenSearch in order to enjoy the latest features, and we will continue to expand on these upgrade and migration documents to cover additional topics, such as upgrading OpenSearch Dashboards and preserving custom configurations, such as for plugins. To see what's coming next or to make a request for future content, leave a comment on the upgrade and migration documentation meta issue in the OpenSearch Project on GitHub.

If you would like a specific process to be added or would like to contribute, create an issue on GitHub. See the Contributor Guidelines to learn how you can help. {: .tip}

Workflow considerations

Take time to plan the process before making any changes to your cluster. For example, consider the following questions:

  • How long will the upgrade process take?
  • If your cluster is being used in production, how impactful is downtime?
  • Do you have infrastructure in place to stand up the new cluster in a testing or development environment before you move it into production, or do you need to upgrade the production hosts directly?

The answers to questions like these will help you determine which upgrade path will work best in your environment.

At a minimum, you should be:

Stop any nonessential indexing before you begin the upgrade procedure to eliminate unnecessary resource demands on the cluster while you perform the upgrade. {: .tip}

Reviewing breaking changes

It's important to determine how the new version of OpenSearch will integreate with your environment. Review Breaking changes before beginning any upgrade procedures to determine whether you will need to make adjustments to your workflow. For example, upstream or downstream components might need to be modified to be compatible with an API change (see meta issue #2589).

Reviewing the OpenSearch tools compatibility matrices

If your OpenSearch cluster interacts with other services in your environment, like Logstash or Beats, then you should check the OpenSearch tools compatibility matrices to determine whether other components will need to be upgraded.

Reviewing plugin compatibility

Review the plugins you use to determine compatibility with the target version of OpenSearch. Official OpenSearch Project plugins can be found in the OpenSearch Project repository on GitHub. If you use any third-party plugins, then you should check the documentation for those plugins to determine whether they are compatible.

Go to Available plugins to see a reference table that highlights version compatibility for bundled OpenSearch plugins.

Major, minor, and patch plugin versions must match OpenSearch major, minor, and patch versions in order to be compatible. For example, plugin versions 2.3.0.x work only with OpenSearch 2.3.0. {: .important}

Backing up configuration files

Mitigate the risk of data loss by backing up any important files before you start an upgrade. Generally, these files will be located in either of two directories:

  • opensearch/config
  • opensearch-dashboards/config

Some examples include opensearch.yml, opensearch_dashboards.yml, plugin configuration files, and TLS certificates. Once you identify which files you want to back up, copy them to remote storage for safety.

If you use security features, make sure to read A word of caution for information about backing up and restoring your security settings.

Creating a snapshot

We recommend that you back up your cluster state and indexes using snapshots. Snapshots you take before an upgrade can be used as restore points if you need to roll back the cluster to its original version.

You can further reduce the risk of data loss by storing your snapshots on external storage, such as a mounted Network File System (NFS) or a cloud storage solution like those listed in the following table.

Snapshot repository location Required OpenSearch plugin
Amazon Simple Storage Service (Amazon S3) repository-s3
Google Cloud Storage (GCS) repository-gcs
Apache Hadoop Distributed File System (HDFS) repository-hdfs
Microsoft Azure Blob Storage repository-azure

Upgrade methods

Choose an appropriate method for upgrading your cluster to a new version of OpenSearch based on your requirements:

Upgrades spanning more than a single major version of OpenSearch will require additional effort due to the need for reindexing. For more information, refer to the Reindex API. See the Index compatibility reference table included later in this guide for help planning your data migration.

Rolling upgrade

A rolling upgrade is a great option if you want to keep your cluster operational throughout the process. Data may continue to be ingested, analyzed, and queried as nodes are individually stopped, upgraded, and restarted. A variation of the rolling upgrade referred to as "node replacement" follows exactly the same process except that hosts and containers are not reused for the new node. You might perform node replacement if you are upgrading the underlying host(s) as well.

OpenSearch nodes cannot join a cluster if the cluster manager is running a newer version of OpenSearch than the node requesting membership. To avoid this issue, upgrade the cluster-manager-eligible nodes last.

See Rolling Upgrade for more information about the process.

Cluster restart upgrade

OpenSearch administrators might choose to perform a cluster restart upgrade for several reasons, such as if the administrator doesn't want to perform maintenance on a running cluster or if the cluster is being migrated to a different environment.

Unlike a rolling upgrade, where only one node is offline at a time, a cluster restart upgrade requires you to stop OpenSearch and OpenSearch Dashboards on all nodes in the cluster before proceeding. After the nodes are stopped, a new version of OpenSearch is installed. Then OpenSearch is started and the cluster bootstraps to the new version.

Compatibility

OpenSearch nodes are compatible with other OpenSearch nodes running any other minor version within the same major version release. For example, 1.1.0 is compatible with 1.3.7 because they are part of the same major version (1.x). Additionally, OpenSearch nodes and indexes are backward compatible with the previous major version. That means, for example, that an index created by an OpenSearch node running any 1.x version can be restored from a snapshot to an OpenSearch cluster running any 2.x version.

OpenSearch 1.x nodes are compatible with nodes running Elasticsearch 7.x, but the longevity of a mixed-version environment should not extend beyond cluster upgrade activities. {: .tip}

Index compatibility is determined by the version of Apache Lucene that created the index. If an index was created by an OpenSearch cluster running version 1.0.0, then the index can be used by any other OpenSearch cluster running up to the latest 1.x or 2.x release. See the Index compatibility reference table for Lucene versions running in OpenSearch 1.0.0 and later and Elasticsearch 6.8 and later.

If your upgrade path spans more than a single major version and you want to retain any existing indexes, then you can use the Reindex API to make your indexes compatible with the target version of OpenSearch before upgrading. For example, if your cluster is currently running Elasticsearch 6.8 and you want to upgrade to OpenSearch 2.x, then you must first upgrade to OpenSearch 1.x, recreate your indexes using the Reindex API, and finally upgrade to 2.x. One alternative to reindexing is to reingest data from the origin, such as by replaying a data stream or ingesting data from a database.

Index compatibility reference

If you plan to retain old indexes after the OpenSearch version upgrade, then you might need to reindex or reingest the data. Refer to the following table for Lucene versions across recent OpenSearch and Elasticsearch releases.

Lucene Version OpenSearch Version Elasticsearch Version
9.4.2 2.5.0
2.4.1
8.6
9.4.1 2.4.0
9.4.0 8.5
9.3.0 2.3.0
2.2.x
8.4
9.2.0 2.1.0 8.3
9.1.0 2.0.x 8.2
9.0.0 8.1
8.0
8.11.1 7.17
8.10.1 1.3.x
1.2.x
7.16
8.9.0 1.1.0 7.15
7.14
8.8.2 1.0.0 7.13
8.8.0 7.12
8.7.0 7.11
7.10
8.6.2 7.9
8.5.1 7.8
7.7
8.4.0 7.6
8.3.0 7.5
8.2.0 7.4
8.1.0 7.3
8.0.0 7.2
7.1
7.7.3 6.8

A dash (—) indicates that there is no product version containing the specified version of Apache Lucene.