diff --git a/docs/reference/indices.asciidoc b/docs/reference/indices.asciidoc index 2e8f9b36e43..a84a17c42c6 100644 --- a/docs/reference/indices.asciidoc +++ b/docs/reference/indices.asciidoc @@ -78,6 +78,14 @@ index settings, aliases, mappings, and index templates. * <> * <> +[float] +[[dangling-indices-api]] +=== Dangling indices: +* <> +* <> +* <> + + include::indices/add-alias.asciidoc[] @@ -162,3 +170,9 @@ include::indices/aliases.asciidoc[] include::indices/update-settings.asciidoc[] include::indices/resolve.asciidoc[] + +include::indices/dangling-indices-list.asciidoc[] + +include::indices/dangling-index-import.asciidoc[] + +include::indices/dangling-index-delete.asciidoc[] diff --git a/docs/reference/indices/dangling-index-delete.asciidoc b/docs/reference/indices/dangling-index-delete.asciidoc new file mode 100644 index 00000000000..3e3a74c456f --- /dev/null +++ b/docs/reference/indices/dangling-index-delete.asciidoc @@ -0,0 +1,44 @@ +[[dangling-index-delete]] +=== Delete dangling index API +++++ +Delete dangling index +++++ + +Deletes a dangling index. + +[[dangling-index-delete-api-request]] +==== {api-request-title} + +[source,console] +-------------------------------------------------- +DELETE /_dangling/?accept_data_loss=true +-------------------------------------------------- +// TEST[skip:Difficult to set up] + +[[dangling-index-delete-api-desc]] +==== {api-description-title} + +include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description] + + +Deletes a dangling index by referencing its UUID. Use the +<> to locate the UUID of an index. + + +[[dangling-index-delete-api-path-params]] +==== {api-path-parms-title} + +``:: +(Required, string) +UUID of the index to delete. You can find this using the +<>. + +[[dangling-index-delete-api-query-params]] +==== {api-query-parms-title} + +`accept_data_loss`:: +(Optional, boolean) +This field must be set to `true` in order to carry out the import, since it will +no longer be possible to recover the data from the dangling index. + +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout] diff --git a/docs/reference/indices/dangling-index-import.asciidoc b/docs/reference/indices/dangling-index-import.asciidoc new file mode 100644 index 00000000000..6129e484ede --- /dev/null +++ b/docs/reference/indices/dangling-index-import.asciidoc @@ -0,0 +1,65 @@ +[[dangling-index-import]] +=== Import dangling index API +++++ +Import dangling index +++++ + +Imports a dangling index. + +[[dangling-index-import-api-request]] +==== {api-request-title} + +[source,console] +-------------------------------------------------- +POST /_dangling/?accept_data_loss=true +-------------------------------------------------- +// TEST[skip:Difficult to set up] + +[[dangling-index-import-api-desc]] +==== {api-description-title} + +include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description] + +Import a single index into the cluster by referencing its UUID. Use the +<> to locate the UUID of an index. + + +[[dangling-index-import-api-path-params]] +==== {api-path-parms-title} + +``:: +(Required, string) +UUID of the index to import, which you can find using the +<>. + +[[dangling-index-import-api-query-params]] +==== {api-query-parms-title} + +`accept_data_loss`:: +(Required, boolean) +This field must be set to `true` to import a dangling index. Because {es} +cannot know where the dangling index data came from or determine which shard +copies are fresh and which are stale, it cannot guarantee that the imported data +represents the latest state of the index when it was last in the cluster. + +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout] + +[[dangling-index-import-api-example]] +==== {api-examples-title} + +The following example shows how to import a dangling index: + +[source,console] +-------------------------------------------------- +POST /_dangling/zmM4e0JtBkeUjiHD-MihPQ?accept_data_loss=true +-------------------------------------------------- +// TEST[skip:Difficult to set up] + +The API returns following response: + +[source,console-result] +-------------------------------------------------- +{ + "acknowledged" : true +} +-------------------------------------------------- diff --git a/docs/reference/indices/dangling-indices-list.asciidoc b/docs/reference/indices/dangling-indices-list.asciidoc new file mode 100644 index 00000000000..b4dafec4981 --- /dev/null +++ b/docs/reference/indices/dangling-indices-list.asciidoc @@ -0,0 +1,49 @@ +[[dangling-indices-list]] +=== List dangling indices API +++++ +List dangling indices +++++ + +Lists dangling indices. + +[[dangling-indices-list-api-request]] +==== {api-request-title} + +[source,console] +-------------------------------------------------- +GET /_dangling +-------------------------------------------------- +// TEST[skip:TBD] + +[[dangling-indices-list-api-desc]] +==== {api-description-title} + +// tag::dangling-index-description[] +If {es} encounters index data that is absent from the current cluster +state, those indices are considered to be dangling. For example, +this can happen if you delete more than +`cluster.indices.tombstones.size` indices while an {es} node is offline. +// end::dangling-index-description[] + +Use this API to list dangling indices, which you can then +<> or <>. + + +[[dangling-indices-list-api-example]] +==== {api-examples-title} + +The API returns the following response: + +[source,console-result] +-------------------------------------------------- +{ + "dangling_indices": [ + "index_name": "twitter", + "index_uuid": "zmM4e0JtBkeUjiHD-MihPQ", + "creation_date_millis": 1589414451372, + "node_ids": [ + "pL47UN3dAb2d5RCWP6lQ3e" + ] + ] +} +-------------------------------------------------- diff --git a/docs/reference/modules/cluster/misc.asciidoc b/docs/reference/modules/cluster/misc.asciidoc index 6986254fa1c..0b969a552c3 100644 --- a/docs/reference/modules/cluster/misc.asciidoc +++ b/docs/reference/modules/cluster/misc.asciidoc @@ -107,6 +107,10 @@ increase it if you expect nodes to be absent from the cluster and miss more than 500 deletes. We think that is rare, thus the default. Tombstones don't take up much space, but we also think that a number like 50,000 is probably too big. +include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description] +You can use the <> to manage +this situation. + [[cluster-logger]] ===== Logger diff --git a/docs/reference/modules/discovery.asciidoc b/docs/reference/modules/discovery.asciidoc index 3a7a86abb81..0fb42cc7edf 100644 --- a/docs/reference/modules/discovery.asciidoc +++ b/docs/reference/modules/discovery.asciidoc @@ -59,7 +59,7 @@ formation: <>:: There are settings that enable users to influence the discovery, cluster - formation, master election and fault detection processes. + formation, master election and fault detection processes. include::discovery/discovery.asciidoc[] diff --git a/docs/reference/modules/gateway.asciidoc b/docs/reference/modules/gateway.asciidoc index 4f2a74077f3..a5d5b3226bb 100644 --- a/docs/reference/modules/gateway.asciidoc +++ b/docs/reference/modules/gateway.asciidoc @@ -56,9 +56,29 @@ NOTE: These settings only take effect on a full cluster restart. [[dangling-indices]] ==== Dangling indices -When a node joins the cluster, any shards stored in its local data -directory which do not already exist in the cluster will be imported into the -cluster. This functionality is intended as a best effort to help users who -lose all master nodes. If a new master node is started which is unaware of -the other indices in the cluster, adding the old nodes will cause the old -indices to be imported, instead of being deleted. +When a node joins the cluster, if it finds any shards stored in its local data +directory that do not already exist in the cluster, it will consider those +shards to be "dangling". Importing dangling indices +into the cluster using `gateway.auto_import_dangling_indices` is not safe. +Instead, use the <>. Neither +mechanism provides any guarantees as to whether the imported data truly +represents the latest state of the data when the index was still part of +the cluster. + +`gateway.auto_import_dangling_indices`:: + + deprecated:[7.9.0, This setting will be removed in 8.0. You should use the dedicated dangling indices API instead.] + Whether to automatically import dangling indices into the cluster + state, provided no indices already exist with the same name. Defaults + to `false`. + +WARNING: The auto-import functionality was intended as a best effort to help users +who lose all master nodes. For example, if a new master node were to be +started which was unaware of the other indices in the cluster, adding the +old nodes would cause the old indices to be imported, instead of being +deleted. However there are several issues with automatic importing, and +its use is strongly discouraged in favour of the +<. + +WARNING: Losing all master nodes is a situation that should be avoided at +all costs, as it puts your cluster's metadata and data at risk.