[DOC] Backup & Restore Security Configuration (#42970)

This commit documents the backup and restore of a cluster's
security configuration.

It is not possible to only backup (or only restore) security
configuration, independent to the rest of the cluster's conf,
so this describes how a full configuration backup&restore
will include security as well. Moreover, it explains how part
of the security conf data resides on the special .security
index and how to backup that using regular data snapshot API.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Tim Vernum <tim@adjective.org>

docs/reference/administering.asciidoc

@@ -9,17 +9,4 @@ cluster.
 
 --
 
-[[backup-cluster]]
-== Back up a cluster
-
-As with any software that stores data, it is important to routinely back up your
-data. {es} replicas provide high availability during runtime; they enable you to
-tolerate sporadic node loss without an interruption of service.
-
-Replicas do not provide protection from catastrophic failure, however. For that,
-you need a real backup of your cluster—a complete copy in case something goes
-wrong.
-
-To back up your cluster, you can use the <<modules-snapshots,snapshot API>>.
-
-include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro]
+include::administering/backup-cluster.asciidoc[]

docs/reference/administering/backup-and-restore-security-config.asciidoc

@@ -0,0 +1,281 @@
+[role="xpack"]
+[testenv="basic"]
+[[security-backup]]
+=== Back up a cluster's security configuration
+++++
+<titleabbrev>Back up the security configuration</titleabbrev>
+++++
+
+Security configuration information resides in two places:
+<<backup-security-file-based-configuration,files>> and
+<<backup-security-index-configuration,indices>>.
+
+[discrete]
+[[backup-security-file-based-configuration]]
+==== Back up file-based security configuration
+
+{es} {security-features} are configured using the <<security-settings,
+`xpack.security` namespace>> inside the `elasticsearch.yml` and
+`elasticsearch.keystore` files. In addition there are several other
+<<security-files, extra configuration files>> inside the same `ES_PATH_CONF`
+directory. These files define roles and role mappings and
+<<configuring-file-realm, configure the file realm>>. Some of the
+settings specify file paths to security-sensitive data, such as TLS keys and
+certificates for the HTTP client and inter-node communication and private key files for
+<<ref-saml-settings, SAML>>, <<ref-oidc-settings, OIDC>> and the
+<<ref-kerberos-settings, Kerberos>> realms. All these are also stored inside
+`ES_PATH_CONF`; the path settings are relative.
+
+IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos
+realms private key files require confidentiality. This is crucial when files
+are copied to the backup location, as this increases the surface for malicious
+snooping.
+
+To back up all this configuration you can use a <<backup-cluster-configuration,
+conventional file-based backup>>, as described in the previous section.
+
+[NOTE]
+====
+
+* File backups must run on every cluster node.
+* File backups will store non-security configuration as well. Backing-up
+only {security-features} configuration is not supported. A backup is a
+point in time record of state of the complete configuration.
+
+====
+
+[discrete]
+[[backup-security-index-configuration]]
+==== Back up index-based security configuration
+
+{es} {security-features} store system configuration data inside a
+dedicated index. This index is named `.security-6` in the {es} 6.x versions and
+`.security-7` in the 7.x releases. The `.security` alias always points to the
+appropriate index. This index contains the data which is not available in
+configuration files and *cannot* be reliably backed up using standard
+filesystem tools. This data describes:
+
+* the definition of users in the native realm (including hashed passwords)
+* role definitions (defined via the <<security-api-put-role,create roles API>>)
+* role mappings (defined via the
+  <<security-api-put-role-mapping,create role mappings API>>)
+* application privileges
+* API keys
+
+The `.security` index thus contains resources and definitions in addition to
+configuration information. All of that information is required in a complete
+{security-features} backup.
+
+Use the <<modules-snapshots, standard {es} snapshot functionality>> to backup
+`.security`, as you would for any <<backup-cluster-data, other data index>>.
+For convenience, here are the complete steps:
+
+. Create a repository that you can use to backup the `.security` index.
+It is preferable to have a <<backup-security-repos, dedicated repository>> for
+this special index. If you wish, you can also snapshot the system indices for other {stack} components to this repository. 
++
+--
+[source,js]
+-----------------------------------
+PUT /_snapshot/my_backup
+{
+  "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+-----------------------------------
+// CONSOLE
+
+The user calling this API must have the elevated `manage` cluster privilege to
+prevent non-administrators exfiltrating data.
+
+--
+
+. Create a user and assign it only the built-in `snapshot_user` role.
++
+--
+The following example creates a new user `snapshot_user` in the
+{stack-ov}/native-realm.html[native realm], but it is not important which
+realm the user is a member of:
+
+[source,js]
+--------------------------------------------------
+POST /_security/user/snapshot_user
+{
+  "password" : "secret",
+  "roles" : [ "snapshot_user" ]
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[skip:security is not enabled in this fixture]
+
+--
+
+. Create incremental snapshots authorized as `snapshot_user`.
++
+--
+The following example shows how to use the create snapshot API to backup
+the `.security` index to the `my_backup` repository:
+
+[source,js]
+--------------------------------------------------
+PUT /_snapshot/my_backup/snapshot_1
+{
+  "indices": ".security",
+  "include_global_state": true <1>
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+<1> This parameter value captures all the persistent settings stored in the
+global cluster metadata as well as other configurations such as aliases and
+stored scripts. Note that this includes non-security configuration and that it complements but does not replace the
+<<backup-cluster-configuration, filesystem configuration files backup>>.
+
+--
+
+IMPORTANT: The index format is only compatible within a single major version,
+and cannot be restored onto a version earlier than the version from which it
+originated. For example, you can restore a security snapshot from 6.6.0 into a
+6.7.0 cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0.
+
+[discrete]
+[[backup-security-repos]]
+===== Controlling access to the backup repository
+
+The snapshot of the security index will typically contain sensitive data such
+as user names and password hashes. Because passwords are stored using
+<<hashing-settings, cryptographic hashes>>, the disclosure of a snapshot would
+not automatically enable a third party to authenticate as one of your users or
+use API keys. However, it would disclose confidential information.
+
+It is also important that you protect the integrity of these backups in case
+you ever need to restore them. If a third party is able to modify the stored
+backups, they may be able to install a back door that would grant access if the
+snapshot is loaded into an {es} cluster.
+
+We recommend that you:
+
+* Snapshot the `.security` index in a dedicated repository, where read and write
+access is strictly restricted and audited.
+* If there are indications that the snapshot has been read, change the passwords
+of the users in the native realm and revoke API keys.
+* If there are indications that the snapshot has been tampered with, do not
+restore it. There is currently no option for the restore process to detect
+malicious tampering.
+
+[[restore-security-configuration]]
+=== Restore a cluster's security configuration
+++++
+<titleabbrev>Restore the security configuration</titleabbrev>
+++++
+
+NOTE: You can restore a snapshot of the `.security` index only if it was
+created in a previous minor version in the same major version. The last minor
+version of every major release can convert and read formats of the index for
+both its major version and the next one.
+
+When you restore security configuration you have the option of doing a complete
+restore of *all* configurations, including non-security ones, or to only restore
+the contents of the `.security` index. As described in
+<<backup-security-index-configuration>>, the second option comprises only
+resource-type configurations. The first option has the advantage of restoring
+a cluster to a clearly defined state from a past point in time. The second option
+touches only security configuration resources, but it does not completely restore
+the {security-features}.
+
+To restore your security configuration from a backup, first make sure that the
+repository holding `.security` snapshots is installed:
+
+[source,js]
+--------------------------------------------------
+GET /_snapshot/my_backup
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+[source,js]
+--------------------------------------------------
+GET /_snapshot/my_backup/snapshot_1
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+Then log into one of the node hosts, navigate to {es} installation directory,
+and follow these steps:
+
+. Add a new user with the `superuser` built-in role to the
+{stack-ov}/file-realm.html[file realm].
++
+--
+For example, create a user named `restore_user`:
+[source,shell]
+--------------------------------------------------
+bin/elasticsearch-users useradd restore_user -p password -r superuser
+--------------------------------------------------
+--
+
+. Using the previously created user, delete the existing `.security-6` or
+`.security-7` index.
++
+--
+[source,shell]
+--------------------------------------------------
+curl -u restore_user -X DELETE "localhost:9200/.security-*"
+--------------------------------------------------
+// NOTCONSOLE
+
+WARNING: After this step any authentication that relies on the `.security`
+index will not work. This means that all API calls that authenticate with
+native or reserved users will fail, as will any user that relies on a native role.
+The file realm user we created in the step above will continue to work
+because it is not stored in the `.security` index and uses the built-in
+`superuser` role.
+
+--
+
+. Using the same user, restore the `.security` index from the snapshot.
++
+--
+[source,shell]
+--------------------------------------------------
+ curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d'
+ {
+    "indices": ".security-*",
+    "include_global_state": true <1>
+ }
+ '
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The `include_global_state: true` is mandatory only for a complete restore.
+This will restore the global cluster metadata, which contains configuration
+information for the complete cluster. If you set this to `false`, it recovers
+only the contents of the `.security` index, such as usernames and password
+hashes, API keys, application privileges, role and role mapping definitions.
+--
+
+. Optionally, if you need to review and override the settings that were included
+in the snapshot (by the `include_global_state` flag), cherry-pick and
+<<cluster-update-settings,apply the persistent settings>> that you
+<<backup-cluster-configuration, have extracted>> with the
+`GET _cluster/settings` API.
+
+. If you pursue a complete point in time restore of the cluster, you also have
+to restore configuration files. Again, this will restore non-security settings as
+well.
++
+--
+This entails a straight-up filesystem copy of the backed up configuration files,
+overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This
+needs to be done on *every node*. Depending on the extent of the differences
+between your current cluster configuration and the restored configuration, you
+may not be able to perform a rolling restart. If you are performing a full
+restore of your configuration directory, we recommend a full cluster restart as
+the safest option. Alternatively, you may wish to restore your configuration
+files to a separate location on disk and use file comparison tools to review
+the differences between your existing configuration and the restored
+configuration.
+--

docs/reference/administering/backup-cluster-config.asciidoc

@@ -0,0 +1,62 @@
+[[backup-cluster-configuration]]
+=== Back up a cluster's configuration
+++++
+<titleabbrev>Back up the cluster configuration</titleabbrev>
+++++
+
+In addition to backing up the data in a cluster, it is important to back up its configuration--especially when the cluster becomes large and difficult to
+reconstruct.
+
+Configuration information resides in
+<<config-files-location, regular text files>> on every cluster node. Sensitive
+setting values such as passwords for the {watcher} notification servers, are
+specified inside a binary secure container, the
+<<secure-settings, elasticsearch.keystore>> file. Some setting values are
+file paths to the associated configuration data, such as the ingest geo ip
+database. All these files are contained inside the `ES_PATH_CONF` directory.
+
+NOTE: All changes to configuration files are done by manually editing the files
+or using command line utilities, but *not* through APIs. In practice, these
+changes are infrequent after the initial setup.
+
+We recommend that you take regular (ideally, daily) backups of your {es} config
+(`$ES_PATH_CONF`) directory using the file backup software of your choice.
+
+TIP: We recommend that you have a configuration management plan for these
+configuration files. You may wish to check them into version control, or
+provision them though your choice of configuration management tool.
+
+Some of these files may contain sensitive data such as passwords and TLS keys,
+therefore you should investigate whether your backup software and/or storage
+solution are able to encrypt this data.
+
+Some settings in configuration files might be overridden by
+<<cluster-update-settings,cluster settings>>. You can capture these settings in
+a *data* backup snapshot by specifying the `include_global_state: true` (default)
+parameter for the snapshot API. Alternatively, you can extract these
+configuration values in text format by using the
+<<cluster-get-settings, get settings API>>:
+
+[source,js]
+--------------------------------------------------
+GET _cluster/settings?pretty&flat_settings&filter_path=persistent
+--------------------------------------------------
+//CONSOLE
+//TEST
+
+You can store the output of this as a file together with the rest of
+configuration files.
+
+[NOTE]
+====
+
+* Transient settings are not considered for backup.
+* {es} {security-features} store configuration data such as role definitions and
+API keys inside a dedicate special index. This "system" data,
+complements the <<secure-settings, security settings>> configuration and should
+be <<backup-security-index-configuration, backed up as well>>.
+* Other {stack} components, like Kibana and {ml-cap}, store their configuration
+data inside other dedicated indices. From the {es} perspective these are just data
+so you can use the regular <<backup-cluster-data, data backup>> process.
+
+====

docs/reference/administering/backup-cluster-data.asciidoc

@@ -0,0 +1,35 @@
+[[backup-cluster-data]]
+=== Back up a cluster's data
+++++
+<titleabbrev>Back up the data</titleabbrev>
+++++
+
+As with any software that stores data, it is important to routinely back up your
+data. {es} replicas provide high availability during runtime; they enable you to
+tolerate sporadic node loss without an interruption of service.
+
+Replicas do not provide protection from catastrophic failure, however. For that,
+you need a real backup of your cluster—a complete copy in case something goes
+wrong.
+
+To back up your cluster's data, you can use the <<modules-snapshots,snapshot API>>.
+
+include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro]
+
+[TIP]
+====
+If your cluster has {es} {security-features} enabled, when you back up your data
+the snapshot API call must be authorized.
+
+The `snapshot_user` role is a reserved role that can be assigned to the user
+who is calling the snapshot endpoint. This is the only role necessary if all the user
+does is periodic snapshots as part of the backup procedure. This role includes
+the privileges to list all the existing snapshots (of any repository) as
+well as list and view settings of all indices, including the `.security` index.
+It does *not* grant privileges to create repositories, restore snapshots, or
+search within indices. Hence, the user can view and snapshot all indices, but cannot
+access or modify any data.
+
+For more information, see {stack-ov}/security-privileges.html[Security privileges]
+and {stack-ov}/built-in-roles.html[Built-in roles].
+====

docs/reference/administering/backup-cluster.asciidoc

@@ -0,0 +1,22 @@
+[[backup-cluster]]
+== Back up a cluster
+
+include::{es-repo-dir}/modules/snapshots.asciidoc[tag=backup-warning]
+
+To have a complete backup for your cluster:
+
+. <<backup-cluster-data,Back up the data>>
+. <<backup-cluster-configuration,Back up the cluster configuration>>
+. <<security-backup,Back up the security configuration>>
+
+To restore your cluster from a backup:
+
+. <<restore-cluster-data,Restore the data>>
+. <<restore-security-configuration,Restore the security configuration>>
+
+
+
+include::backup-cluster-data.asciidoc[]
+include::backup-cluster-config.asciidoc[]
+include::backup-and-restore-security-config.asciidoc[]
+include::restore-cluster-data.asciidoc[]

docs/reference/administering/restore-cluster-data.asciidoc

@@ -0,0 +1,15 @@
+[[restore-cluster-data]]
+=== Restore a cluster's data
+++++
+<titleabbrev>Restore the data</titleabbrev>
+++++
+
+include::{es-repo-dir}/modules/snapshots.asciidoc[tag=restore-intro]
+
+[TIP]
+====
+If your cluster has {es} {security-features} enabled, the restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only
+be granted to users in the "administrator" category. Specifically, it allows
+malicious users to exfiltrate data to a location of their choosing. Automated
+tools should not run as users with this privilege.
+====

docs/reference/modules/snapshots.asciidoc

@@ -11,12 +11,16 @@ Snapshots are taken incrementally. This means that when it creates a snapshot of
 an index, Elasticsearch avoids copying any data that is already stored in the
 repository as part of an earlier snapshot of the same index. Therefore it can be
 efficient to take snapshots of your cluster quite frequently.
+// end::snapshot-intro[]
 
+// tag::restore-intro[]
 You can restore snapshots into a running cluster via the
 <<restore-snapshot,restore API>>. When you restore an index, you can alter the
 name of the restored index as well as some of its settings. There is a great
 deal of flexibility in how the snapshot and restore functionality can be used.
+// end::restore-intro[]
 
+// tag::backup-warning[]
 WARNING: You cannot back up an Elasticsearch cluster by simply taking a copy of
 the data directories of all of its nodes. Elasticsearch may be making changes to
 the contents of its data directories while it is running; copying its data
@@ -26,7 +30,7 @@ corruption and/or missing files. Alternatively, it may appear to have succeeded
 though it silently lost some of its data. The only reliable way to back up a
 cluster is by using the snapshot and restore functionality.
 
-// end::snapshot-intro[]
+// end::backup-warning[]
 
 [float]
 === Version compatibility

x-pack/docs/en/security/configuring-es.asciidoc

@@ -162,7 +162,7 @@ include::authentication/configuring-saml-realm.asciidoc[]
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/authentication/configuring-kerberos-realm.asciidoc
 include::authentication/configuring-kerberos-realm.asciidoc[]
 
-:edit_url: 
-include::fips-140-compliance.asciidoc[]
+:edit_url:
 include::{es-repo-dir}/security/reference/files.asciidoc[]
+include::fips-140-compliance.asciidoc[]