mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-07 13:38:49 +00:00
a46eef9634
Using arrays of objects with embedded IDs is preferred for new APIs over using entity IDs as JSON keys. This commit changes the SLM stats API to use the preferred format.
413 lines
19 KiB
Plaintext
413 lines
19 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[snapshot-lifecycle-management-api]]
|
|
== Snapshot lifecycle management API
|
|
|
|
The Snapshot Lifecycle Management APIs are used to manage policies for the time
|
|
and frequency of automatic snapshots. Snapshot Lifecycle Management is related
|
|
to <<index-lifecycle-management,Index Lifecycle Management>>, however, instead
|
|
of managing a lifecycle of actions that are performed on a single index, SLM
|
|
allows configuring policies spanning multiple indices.
|
|
|
|
SLM policy management is split into three different CRUD APIs, a way to put or update
|
|
policies, a way to retrieve policies, and a way to delete unwanted policies, as
|
|
well as a separate API for immediately invoking a snapshot based on a policy.
|
|
|
|
Since SLM falls under the same category as ILM, it is stopped and started by
|
|
using the <<start-stop-ilm,start and stop>> ILM APIs. It is, however, managed
|
|
by a different enable setting. To disable SLM's functionality, set the cluster
|
|
setting `xpack.slm.enabled` to `false` in elasticsearch.yml.
|
|
|
|
[[slm-api-put]]
|
|
=== Put Snapshot Lifecycle Policy API
|
|
|
|
Creates or updates a snapshot policy. If the policy already exists, the version
|
|
is incremented. Only the latest version of a policy is stored.
|
|
|
|
When a policy is created it is immediately scheduled based on the schedule of
|
|
the policy, when a policy is updated its schedule changes are immediately
|
|
applied.
|
|
|
|
==== Path Parameters
|
|
|
|
`policy_id` (required)::
|
|
(string) Identifier (id) for the policy.
|
|
|
|
==== Request Parameters
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
|
|
|
|
==== Authorization
|
|
|
|
You must have the `manage_slm` cluster privilege to use this API. You must also
|
|
have the `manage` index privilege on all indices being managed by `policy`. All
|
|
operations executed by {slm} for a policy are executed as the user that put the
|
|
latest version of a policy. For more information, see
|
|
{stack-ov}/security-privileges.html[Security Privileges].
|
|
|
|
==== Example
|
|
|
|
The following creates a snapshot lifecycle policy with an id of
|
|
`daily-snapshots`:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_slm/policy/daily-snapshots
|
|
{
|
|
"schedule": "0 30 1 * * ?", <1>
|
|
"name": "<daily-snap-{now/d}>", <2>
|
|
"repository": "my_repository", <3>
|
|
"config": { <4>
|
|
"indices": ["data-*", "important"], <5>
|
|
"ignore_unavailable": false,
|
|
"include_global_state": false
|
|
},
|
|
"retention": {}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:setup-repository]
|
|
|
|
<1> When the snapshot should be taken, in this case, 1:30am daily
|
|
<2> The name each snapshot should be given
|
|
<3> Which repository to take the snapshot in
|
|
<4> Any extra snapshot configuration
|
|
<5> Which indices the snapshot should contain
|
|
|
|
The top-level keys that the policy supports are described below:
|
|
|
|
|==================
|
|
| Key | Description
|
|
|
|
| `schedule` | A periodic or absolute time schedule. Supports all values
|
|
supported by the cron scheduler:
|
|
{xpack-ref}/trigger-schedule.html#schedule-cron[Cron scheduler configuration]
|
|
|
|
| `name` | A name automatically given to each snapshot performed by this policy.
|
|
Supports the same <<date-math-index-names,date math>> supported in index
|
|
names. A UUID is automatically appended to the end of the name to prevent
|
|
conflicting snapshot names.
|
|
|
|
| `repository` | The snapshot repository that will contain snapshots created by
|
|
this policy. The repository must exist prior to the policy's creation and can
|
|
be created with the <<modules-snapshots,snapshot repository API>>.
|
|
|
|
| `config` | Configuration for each snapshot that will be created by this
|
|
policy. Any configuration is included with <<modules-snapshots,create snapshot
|
|
requests>> issued by this policy.
|
|
|==================
|
|
|
|
To update an existing policy, simply use the put snapshot lifecycle policy API
|
|
with the same policy id as an existing policy.
|
|
|
|
[[slm-api-get]]
|
|
=== Get Snapshot Lifecycle Policy API
|
|
|
|
Once a policy is in place, you can retrieve one or more of the policies using
|
|
the get snapshot lifecycle policy API. This also includes information about the
|
|
latest successful and failed invocation that the automatic snapshots have taken.
|
|
|
|
==== Path Parameters
|
|
|
|
`policy_ids` (optional)::
|
|
(string) Comma-separated ids of policies to retrieve.
|
|
|
|
==== Examples
|
|
|
|
To retrieve a policy, perform a `GET` with the policy's id
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_slm/policy/daily-snapshots?human
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
The output looks similar to the following:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"daily-snapshots" : {
|
|
"version": 1, <1>
|
|
"modified_date": "2019-04-23T01:30:00.000Z", <2>
|
|
"modified_date_millis": 1556048137314,
|
|
"policy" : {
|
|
"schedule": "0 30 1 * * ?",
|
|
"name": "<daily-snap-{now/d}>",
|
|
"repository": "my_repository",
|
|
"config": {
|
|
"indices": ["data-*", "important"],
|
|
"ignore_unavailable": false,
|
|
"include_global_state": false
|
|
},
|
|
"retention": {}
|
|
},
|
|
"stats": {
|
|
"policy": "daily-snapshots",
|
|
"snapshots_taken": 0,
|
|
"snapshots_failed": 0,
|
|
"snapshots_deleted": 0,
|
|
"snapshot_deletion_failures": 0
|
|
},
|
|
"next_execution": "2019-04-24T01:30:00.000Z", <3>
|
|
"next_execution_millis": 1556048160000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/"modified_date": "2019-04-23T01:30:00.000Z"/"modified_date": $body.daily-snapshots.modified_date/ s/"modified_date_millis": 1556048137314/"modified_date_millis": $body.daily-snapshots.modified_date_millis/ s/"next_execution": "2019-04-24T01:30:00.000Z"/"next_execution": $body.daily-snapshots.next_execution/ s/"next_execution_millis": 1556048160000/"next_execution_millis": $body.daily-snapshots.next_execution_millis/]
|
|
<1> The version of the snapshot policy, only the latest verison is stored and incremented when the policy is updated
|
|
<2> The last time this policy was modified
|
|
<3> The next time this policy will be executed
|
|
|
|
Or, to retrieve all policies:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_slm/policy
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
[[slm-api-execute]]
|
|
=== Execute Snapshot Lifecycle Policy API
|
|
|
|
Sometimes it can be useful to immediately execute a snapshot based on policy,
|
|
perhaps before an upgrade or before performing other maintenance on indices. The
|
|
execute snapshot policy API allows you to perform a snapshot immediately without
|
|
waiting for a policy's scheduled invocation.
|
|
|
|
==== Path Parameters
|
|
|
|
`policy_id` (required)::
|
|
(string) Id of the policy to execute
|
|
|
|
==== Example
|
|
|
|
To take an immediate snapshot using a policy, use the following
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_slm/policy/daily-snapshots/_execute
|
|
--------------------------------------------------
|
|
// TEST[skip:we can't easily handle snapshots from docs tests]
|
|
|
|
This API will immediately return with the generated snapshot name
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"snapshot_name": "daily-snap-2019.04.24-gwrqoo2xtea3q57vvg0uea"
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:we can't handle snapshots from docs tests]
|
|
|
|
The snapshot will be taken in the background, you can use the
|
|
<<modules-snapshots,snapshot APIs>> to monitor the status of the snapshot.
|
|
|
|
Once a snapshot has been kicked off, you can see the latest successful or failed
|
|
snapshot using the get snapshot lifecycle policy API:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_slm/policy/daily-snapshots?human
|
|
--------------------------------------------------
|
|
// TEST[skip:we already tested get policy above, the last_failure may not be present though]
|
|
|
|
Which, in this case shows an error because the index did not exist:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"daily-snapshots" : {
|
|
"version": 1,
|
|
"modified_date": "2019-04-23T01:30:00.000Z",
|
|
"modified_date_millis": 1556048137314,
|
|
"policy" : {
|
|
"schedule": "0 30 1 * * ?",
|
|
"name": "<daily-snap-{now/d}>",
|
|
"repository": "my_repository",
|
|
"config": {
|
|
"indices": ["data-*", "important"],
|
|
"ignore_unavailable": false,
|
|
"include_global_state": false
|
|
},
|
|
"retention": {}
|
|
},
|
|
"stats": {
|
|
"policy": "daily-snapshots",
|
|
"snapshots_taken": 0,
|
|
"snapshots_failed": 1,
|
|
"snapshots_deleted": 0,
|
|
"snapshot_deletion_failures": 0
|
|
}
|
|
"last_failure": { <1>
|
|
"snapshot_name": "daily-snap-2019.04.02-lohisb5ith2n8hxacaq3mw",
|
|
"time_string": "2019-04-02T01:30:00.000Z",
|
|
"time": 1556042030000,
|
|
"details": "{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [important]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"important\",\"index_uuid\":\"_na_\",\"index\":\"important\",\"stack_trace\":\"[important] IndexNotFoundException[no such index [important]]\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:762)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:714)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:670)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:163)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:142)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:102)\\n\\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:280)\\n\\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\\n\\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687)\\n\\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310)\\n\\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210)\\n\\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\\n\\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base/java.lang.Thread.run(Thread.java:834)\\n\"}"
|
|
} ,
|
|
"next_execution": "2019-04-24T01:30:00.000Z",
|
|
"next_execution_millis": 1556048160000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:the presence of last_failure is asynchronous and will be present for users, but is untestable]
|
|
|
|
<1> The last unsuccessfully initiated snapshot by this policy, along with the details of its failure
|
|
|
|
In this case, it failed due to the "important" index not existing and
|
|
`ignore_unavailable` setting being set to `false`.
|
|
|
|
Updating the policy to change the `ignore_unavailable` setting is done using the
|
|
same put snapshot lifecycle policy API:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_slm/policy/daily-snapshots
|
|
{
|
|
"schedule": "0 30 1 * * ?",
|
|
"name": "<daily-snap-{now/d}>",
|
|
"repository": "my_repository",
|
|
"config": {
|
|
"indices": ["data-*", "important"],
|
|
"ignore_unavailable": true,
|
|
"include_global_state": false
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
Another snapshot can immediately be executed to ensure the new policy works:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_slm/policy/daily-snapshots/_execute
|
|
--------------------------------------------------
|
|
// TEST[skip:we can't handle snapshots in docs tests]
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"snapshot_name": "daily-snap-2019.04.24-tmtnyjtrsxkhbrrdcgg18a"
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:we can't handle snapshots in docs tests]
|
|
|
|
Now retriving the policy shows that the policy has successfully been executed:
|
|
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_slm/policy/daily-snapshots?human
|
|
--------------------------------------------------
|
|
// TEST[skip:we already tested this above and the output may not be available yet]
|
|
|
|
Which now includes the successful snapshot information:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"daily-snapshots" : {
|
|
"version": 2, <1>
|
|
"modified_date": "2019-04-23T01:30:00.000Z",
|
|
"modified_date_millis": 1556048137314,
|
|
"policy" : {
|
|
"schedule": "0 30 1 * * ?",
|
|
"name": "<daily-snap-{now/d}>",
|
|
"repository": "my_repository",
|
|
"config": {
|
|
"indices": ["data-*", "important"],
|
|
"ignore_unavailable": true,
|
|
"include_global_state": false
|
|
},
|
|
"retention": {}
|
|
},
|
|
"stats": {
|
|
"policy": "daily-snapshots",
|
|
"snapshots_taken": 1,
|
|
"snapshots_failed": 1,
|
|
"snapshots_deleted": 0,
|
|
"snapshot_deletion_failures": 0
|
|
},
|
|
"last_success": { <2>
|
|
"snapshot_name": "daily-snap-2019.04.24-tmtnyjtrsxkhbrrdcgg18a",
|
|
"time_string": "2019-04-24T16:43:49.316Z",
|
|
"time": 1556124229316
|
|
} ,
|
|
"last_failure": {
|
|
"snapshot_name": "daily-snap-2019.04.02-lohisb5ith2n8hxacaq3mw",
|
|
"time_string": "2019-04-02T01:30:00.000Z",
|
|
"time": 1556042030000,
|
|
"details": "{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [important]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"important\",\"index_uuid\":\"_na_\",\"index\":\"important\",\"stack_trace\":\"[important] IndexNotFoundException[no such index [important]]\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:762)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:714)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:670)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:163)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:142)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:102)\\n\\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:280)\\n\\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\\n\\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687)\\n\\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310)\\n\\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210)\\n\\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\\n\\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base/java.lang.Thread.run(Thread.java:834)\\n\"}"
|
|
} ,
|
|
"next_execution": "2019-04-24T01:30:00.000Z",
|
|
"next_execution_millis": 1556048160000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:the presence of last_failure and last_success is asynchronous and will be present for users, but is untestable]
|
|
|
|
<1> The policy's version has been incremented because it was updated
|
|
<2> The last successfully initiated snapshot information
|
|
|
|
It is a good idea to test policies using the execute API to ensure they work.
|
|
|
|
[[slm-get-stats]]
|
|
=== Get Snapshot Lifecycle Stats API
|
|
|
|
SLM stores statistics on a global and per-policy level about actions taken. These stats can be
|
|
retrieved by using the following API:
|
|
|
|
==== Example
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_slm/stats
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
Which returns a response similar to:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"retention_runs": 13,
|
|
"retention_failed": 0,
|
|
"retention_timed_out": 0,
|
|
"retention_deletion_time": "1.4s",
|
|
"retention_deletion_time_millis": 1404,
|
|
"policy_metrics": [
|
|
{
|
|
"policy": "daily-snapshots",
|
|
"snapshots_taken": 1,
|
|
"snapshots_failed": 1,
|
|
"snapshots_deleted": 0,
|
|
"snapshot_deletion_failures": 0
|
|
}
|
|
],
|
|
"total_snapshots_taken": 1,
|
|
"total_snapshots_failed": 1,
|
|
"total_snapshots_deleted": 0,
|
|
"total_snapshot_deletion_failures": 0
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/runs": 13/runs": $body.retention_runs/ s/_failed": 0/_failed": $body.retention_failed/ s/_timed_out": 0/_timed_out": $body.retention_timed_out/ s/"1.4s"/$body.retention_deletion_time/ s/1404/$body.retention_deletion_time_millis/]
|
|
|
|
[[slm-api-delete]]
|
|
=== Delete Snapshot Lifecycle Policy API
|
|
|
|
A policy can be deleted by issuing a delete request with the policy id. Note
|
|
that this prevents any future snapshots from being taken, but does not cancel
|
|
any currently ongoing snapshots or remove any previously taken snapshots.
|
|
|
|
==== Path Parameters
|
|
|
|
`policy_id` (optional)::
|
|
(string) Id of the policy to remove.
|
|
|
|
==== Example
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE /_slm/policy/daily-snapshots
|
|
--------------------------------------------------
|
|
// TEST[continued]
|