670 lines
20 KiB
Plaintext
670 lines
20 KiB
Plaintext
[[docs-delete-by-query]]
|
|
=== Delete by query API
|
|
++++
|
|
<titleabbrev>Delete by query</titleabbrev>
|
|
++++
|
|
|
|
Deletes documents that match the specified query.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST /twitter/_delete_by_query
|
|
{
|
|
"query": {
|
|
"match": {
|
|
"message": "some message"
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:big_twitter]
|
|
|
|
[[docs-delete-by-query-api-request]]
|
|
==== {api-request-title}
|
|
|
|
`POST /<index>/_delete_by_query`
|
|
|
|
////
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"took" : 147,
|
|
"timed_out": false,
|
|
"deleted": 119,
|
|
"batches": 1,
|
|
"version_conflicts": 0,
|
|
"noops": 0,
|
|
"retries": {
|
|
"bulk": 0,
|
|
"search": 0
|
|
},
|
|
"throttled_millis": 0,
|
|
"requests_per_second": -1.0,
|
|
"throttled_until_millis": 0,
|
|
"total": 119,
|
|
"failures" : [ ]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]
|
|
////
|
|
|
|
[[docs-delete-by-query-api-desc]]
|
|
==== {api-description-title}
|
|
|
|
You can specify the query criteria in the request URI or the request body
|
|
using the same syntax as the <<search-search,Search API>>.
|
|
|
|
When you submit a delete by query request, {es} gets a snapshot of the index
|
|
when it begins processing the request and deletes matching documents using
|
|
`internal` versioning. If a document changes between the time that the
|
|
snapshot is taken and the delete operation is processed, it results in a version
|
|
conflict and the delete operation fails.
|
|
|
|
NOTE: Documents with a version equal to 0 cannot be deleted using delete by
|
|
query because `internal` versioning does not support 0 as a valid
|
|
version number.
|
|
|
|
While processing a delete by query request, {es} performs multiple search
|
|
requests sequentially to find all of the matching documents to delete. A bulk
|
|
delete request is performed for each batch of matching documents. If a
|
|
search or bulk request is rejected, the requests are retried up to 10 times, with
|
|
exponential back off. If the maximum retry limit is reached, processing halts
|
|
and all failed requests are returned in the response. Any delete requests that
|
|
completed successfully still stick, they are not rolled back.
|
|
|
|
You can opt to count version conflicts instead of halting and returning by
|
|
setting `conflicts` to `proceed`.
|
|
|
|
===== Refreshing shards
|
|
|
|
Specifying the `refresh` parameter refreshes all shards involved in the delete
|
|
by query once the request completes. This is different than the delete API's
|
|
`refresh` parameter, which causes just the shard that received the delete
|
|
request to be refreshed. Unlike the delete API, it does not support
|
|
`wait_for`.
|
|
|
|
[[docs-delete-by-query-task-api]]
|
|
===== Running delete by query asynchronously
|
|
|
|
If the request contains `wait_for_completion=false`, {es}
|
|
performs some preflight checks, launches the request, and returns a
|
|
<<docs-delete-by-query-task-api,`task`>>
|
|
you can use to cancel or get the status of the task. {es} creates a
|
|
record of this task as a document at `.tasks/task/${taskId}`. When you are
|
|
done with a task, you should delete the task document so {es} can reclaim the
|
|
space.
|
|
|
|
===== Waiting for active shards
|
|
|
|
`wait_for_active_shards` controls how many copies of a shard must be active
|
|
before proceeding with the request. See <<index-wait-for-active-shards>>
|
|
for details. `timeout` controls how long each write request waits for unavailable
|
|
shards to become available. Both work exactly the way they work in the
|
|
<<docs-bulk,Bulk API>>. Delete by query uses scrolled searches, so you can also
|
|
specify the `scroll` parameter to control how long it keeps the search context
|
|
alive, for example `?scroll=10m`. The default is 5 minutes.
|
|
|
|
===== Throttling delete requests
|
|
|
|
To control the rate at which delete by query issues batches of delete operations,
|
|
you can set `requests_per_second` to any positive decimal number. This pads each
|
|
batch with a wait time to throttle the rate. Set `requests_per_second` to `-1`
|
|
to disable throttling.
|
|
|
|
Throttling uses a wait time between batches so that the internal scroll requests
|
|
can be given a timeout that takes the request padding into account. The padding
|
|
time is the difference between the batch size divided by the
|
|
`requests_per_second` and the time spent writing. By default the batch size is
|
|
`1000`, so if `requests_per_second` is set to `500`:
|
|
|
|
[source,txt]
|
|
--------------------------------------------------
|
|
target_time = 1000 / 500 per second = 2 seconds
|
|
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
|
|
--------------------------------------------------
|
|
|
|
Since the batch is issued as a single `_bulk` request, large batch sizes
|
|
cause {es} to create many requests and wait before starting the next set.
|
|
This is "bursty" instead of "smooth".
|
|
|
|
[[docs-delete-by-query-slice]]
|
|
===== Slicing
|
|
|
|
Delete by query supports <<sliced-scroll, sliced scroll>> to parallelize the
|
|
delete process. This can improve efficiency and provide a
|
|
convenient way to break the request down into smaller parts.
|
|
|
|
Setting `slices` to `auto` chooses a reasonable number for most indices.
|
|
If you're slicing manually or otherwise tuning automatic slicing, keep in mind
|
|
that:
|
|
|
|
* Query performance is most efficient when the number of `slices` is equal to
|
|
the number of shards in the index. If that number is large (for example,
|
|
500), choose a lower number as too many `slices` hurts performance. Setting
|
|
`slices` higher than the number of shards generally does not improve efficiency
|
|
and adds overhead.
|
|
|
|
* Delete performance scales linearly across available resources with the
|
|
number of slices.
|
|
|
|
Whether query or delete performance dominates the runtime depends on the
|
|
documents being reindexed and cluster resources.
|
|
|
|
[[docs-delete-by-query-api-path-params]]
|
|
==== {api-path-parms-title}
|
|
|
|
`<index>`::
|
|
(Optional, string) A comma-separated list of index names to search. Use `_all`
|
|
or omit to search all indices.
|
|
|
|
[[docs-delete-by-query-api-query-params]]
|
|
==== {api-query-parms-title}
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=analyzer]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=analyze_wildcard]
|
|
|
|
`conflicts`::
|
|
(Optional, string) What to do if delete by query hits version conflicts:
|
|
`abort` or `proceed`. Defaults to `abort`.
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=default_operator]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=df]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
|
+
|
|
Defaults to `open`.
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=from]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=lenient]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=max_docs]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=preference]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=search-q]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=request_cache]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=refresh]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=requests_per_second]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=routing]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=scroll]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=scroll_size]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=search_type]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=search_timeout]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=slices]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=sort]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source_excludes]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source_includes]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=stats]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=terminate_after]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=timeout]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=version]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=timeout]
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
|
|
|
|
[[docs-delete-by-query-api-request-body]]
|
|
==== {api-request-body-title}
|
|
|
|
`query`::
|
|
(Optional, <<query-dsl,query object>>) Specifies the documents to delete
|
|
using the <<query-dsl,Query DSL>>.
|
|
|
|
|
|
[[docs-delete-by-quer-api-response-body]]
|
|
==== Response body
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST /twitter/_delete_by_query
|
|
{
|
|
"query": { <1>
|
|
"match": {
|
|
"message": "some message"
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:big_twitter]
|
|
|
|
//////////////////////////
|
|
|
|
The JSON response looks like this:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"took" : 147,
|
|
"timed_out": false,
|
|
"total": 119,
|
|
"deleted": 119,
|
|
"batches": 1,
|
|
"version_conflicts": 0,
|
|
"noops": 0,
|
|
"retries": {
|
|
"bulk": 0,
|
|
"search": 0
|
|
},
|
|
"throttled_millis": 0,
|
|
"requests_per_second": -1.0,
|
|
"throttled_until_millis": 0,
|
|
"failures" : [ ]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/: [0-9]+/: $body.$_path/]
|
|
|
|
`took`::
|
|
|
|
The number of milliseconds from start to end of the whole operation.
|
|
|
|
`timed_out`::
|
|
|
|
This flag is set to `true` if any of the requests executed during the
|
|
delete by query execution has timed out.
|
|
|
|
`total`::
|
|
|
|
The number of documents that were successfully processed.
|
|
|
|
`deleted`::
|
|
|
|
The number of documents that were successfully deleted.
|
|
|
|
`batches`::
|
|
|
|
The number of scroll responses pulled back by the delete by query.
|
|
|
|
`version_conflicts`::
|
|
|
|
The number of version conflicts that the delete by query hit.
|
|
|
|
`noops`::
|
|
|
|
This field is always equal to zero for delete by query. It only exists
|
|
so that delete by query, update by query, and reindex APIs return responses
|
|
with the same structure.
|
|
|
|
`retries`::
|
|
|
|
The number of retries attempted by delete by query. `bulk` is the number
|
|
of bulk actions retried, and `search` is the number of search actions retried.
|
|
|
|
`throttled_millis`::
|
|
|
|
Number of milliseconds the request slept to conform to `requests_per_second`.
|
|
|
|
`requests_per_second`::
|
|
|
|
The number of requests per second effectively executed during the delete by query.
|
|
|
|
`throttled_until_millis`::
|
|
|
|
This field should always be equal to zero in a `_delete_by_query` response. It only
|
|
has meaning when using the <<docs-delete-by-query-task-api, Task API>>, where it
|
|
indicates the next time (in milliseconds since epoch) a throttled request will be
|
|
executed again in order to conform to `requests_per_second`.
|
|
|
|
`failures`::
|
|
|
|
Array of failures if there were any unrecoverable errors during the process. If
|
|
this is non-empty then the request aborted because of those failures.
|
|
Delete by query is implemented using batches, and any failure causes the entire
|
|
process to abort but all failures in the current batch are collected into the
|
|
array. You can use the `conflicts` option to prevent reindex from aborting on
|
|
version conflicts.
|
|
|
|
[[docs-delete-by-query-api-example]]
|
|
==== {api-examples-title}
|
|
|
|
Delete all tweets from the `twitter` index:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST twitter/_delete_by_query?conflicts=proceed
|
|
{
|
|
"query": {
|
|
"match_all": {}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:twitter]
|
|
|
|
Delete documents from multiple indices:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST /twitter,blog/_delete_by_query
|
|
{
|
|
"query": {
|
|
"match_all": {}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[s/^/PUT twitter\nPUT blog\n/]
|
|
|
|
Limit the delete by query operation to shards that a particular routing
|
|
value:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST twitter/_delete_by_query?routing=1
|
|
{
|
|
"query": {
|
|
"range" : {
|
|
"age" : {
|
|
"gte" : 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:twitter]
|
|
|
|
By default `_delete_by_query` uses scroll batches of 1000. You can change the
|
|
batch size with the `scroll_size` URL parameter:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST twitter/_delete_by_query?scroll_size=5000
|
|
{
|
|
"query": {
|
|
"term": {
|
|
"user": "kimchy"
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:twitter]
|
|
|
|
[float]
|
|
[[docs-delete-by-query-manual-slice]]
|
|
===== Slice manually
|
|
|
|
Slice a delete by query manually by providing a slice id and total number of
|
|
slices:
|
|
|
|
[source,console]
|
|
----------------------------------------------------------------
|
|
POST twitter/_delete_by_query
|
|
{
|
|
"slice": {
|
|
"id": 0,
|
|
"max": 2
|
|
},
|
|
"query": {
|
|
"range": {
|
|
"likes": {
|
|
"lt": 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
POST twitter/_delete_by_query
|
|
{
|
|
"slice": {
|
|
"id": 1,
|
|
"max": 2
|
|
},
|
|
"query": {
|
|
"range": {
|
|
"likes": {
|
|
"lt": 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
// TEST[setup:big_twitter]
|
|
|
|
Which you can verify works with:
|
|
|
|
[source,console]
|
|
----------------------------------------------------------------
|
|
GET _refresh
|
|
POST twitter/_search?size=0&filter_path=hits.total
|
|
{
|
|
"query": {
|
|
"range": {
|
|
"likes": {
|
|
"lt": 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
Which results in a sensible `total` like this one:
|
|
|
|
[source,console-result]
|
|
----------------------------------------------------------------
|
|
{
|
|
"hits": {
|
|
"total" : {
|
|
"value": 0,
|
|
"relation": "eq"
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
|
|
[float]
|
|
[[docs-delete-by-query-automatic-slice]]
|
|
===== Use automatic slicing
|
|
|
|
You can also let delete-by-query automatically parallelize using
|
|
<<sliced-scroll, sliced scroll>> to slice on `_id`. Use `slices` to specify
|
|
the number of slices to use:
|
|
|
|
[source,console]
|
|
----------------------------------------------------------------
|
|
POST twitter/_delete_by_query?refresh&slices=5
|
|
{
|
|
"query": {
|
|
"range": {
|
|
"likes": {
|
|
"lt": 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
// TEST[setup:big_twitter]
|
|
|
|
Which you also can verify works with:
|
|
|
|
[source,console]
|
|
----------------------------------------------------------------
|
|
POST twitter/_search?size=0&filter_path=hits.total
|
|
{
|
|
"query": {
|
|
"range": {
|
|
"likes": {
|
|
"lt": 10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
Which results in a sensible `total` like this one:
|
|
|
|
[source,console-result]
|
|
----------------------------------------------------------------
|
|
{
|
|
"hits": {
|
|
"total" : {
|
|
"value": 0,
|
|
"relation": "eq"
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------
|
|
|
|
Setting `slices` to `auto` will let {es} choose the number of slices
|
|
to use. This setting will use one slice per shard, up to a certain limit. If
|
|
there are multiple source indices, it will choose the number of slices based
|
|
on the index with the smallest number of shards.
|
|
|
|
Adding `slices` to `_delete_by_query` just automates the manual process used in
|
|
the section above, creating sub-requests which means it has some quirks:
|
|
|
|
* You can see these requests in the
|
|
<<docs-delete-by-query-task-api,Tasks APIs>>. These sub-requests are "child"
|
|
tasks of the task for the request with `slices`.
|
|
* Fetching the status of the task for the request with `slices` only contains
|
|
the status of completed slices.
|
|
* These sub-requests are individually addressable for things like cancellation
|
|
and rethrottling.
|
|
* Rethrottling the request with `slices` will rethrottle the unfinished
|
|
sub-request proportionally.
|
|
* Canceling the request with `slices` will cancel each sub-request.
|
|
* Due to the nature of `slices` each sub-request won't get a perfectly even
|
|
portion of the documents. All documents will be addressed, but some slices may
|
|
be larger than others. Expect larger slices to have a more even distribution.
|
|
* Parameters like `requests_per_second` and `max_docs` on a request with
|
|
slices` are distributed proportionally to each sub-request. Combine that with
|
|
the point above about distribution being uneven and you should conclude that
|
|
using `max_docs` with `slices` might not result in exactly `max_docs` documents
|
|
being deleted.
|
|
* Each sub-request gets a slightly different snapshot of the source index
|
|
though these are all taken at approximately the same time.
|
|
|
|
[float]
|
|
[[docs-delete-by-query-rethrottle]]
|
|
===== Change throttling for a request
|
|
|
|
The value of `requests_per_second` can be changed on a running delete by query
|
|
using the `_rethrottle` API. Rethrottling that speeds up the
|
|
query takes effect immediately but rethrotting that slows down the query
|
|
takes effect after completing the current batch to prevent scroll
|
|
timeouts.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST _delete_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
|
|
--------------------------------------------------
|
|
|
|
Use the <<tasks,tasks API>> to get the task ID. Set `requests_per_second`
|
|
to any positive decimal value or `-1` to disable throttling.
|
|
|
|
===== Get the status of a delete by query operation
|
|
|
|
Use the <<tasks,tasks API>> to get the status of a delete by query
|
|
operation:
|
|
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET _tasks?detailed=true&actions=*/delete/byquery
|
|
--------------------------------------------------
|
|
// TEST[skip:No tasks to retrieve]
|
|
|
|
The response looks like:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"nodes" : {
|
|
"r1A2WoRbTwKZ516z6NEs5A" : {
|
|
"name" : "r1A2WoR",
|
|
"transport_address" : "127.0.0.1:9300",
|
|
"host" : "127.0.0.1",
|
|
"ip" : "127.0.0.1:9300",
|
|
"attributes" : {
|
|
"testattr" : "test",
|
|
"portsfile" : "true"
|
|
},
|
|
"tasks" : {
|
|
"r1A2WoRbTwKZ516z6NEs5A:36619" : {
|
|
"node" : "r1A2WoRbTwKZ516z6NEs5A",
|
|
"id" : 36619,
|
|
"type" : "transport",
|
|
"action" : "indices:data/write/delete/byquery",
|
|
"status" : { <1>
|
|
"total" : 6154,
|
|
"updated" : 0,
|
|
"created" : 0,
|
|
"deleted" : 3500,
|
|
"batches" : 36,
|
|
"version_conflicts" : 0,
|
|
"noops" : 0,
|
|
"retries": 0,
|
|
"throttled_millis": 0
|
|
},
|
|
"description" : ""
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
<1> This object contains the actual status. It is just like the response JSON
|
|
with the important addition of the `total` field. `total` is the total number
|
|
of operations that the reindex expects to perform. You can estimate the
|
|
progress by adding the `updated`, `created`, and `deleted` fields. The request
|
|
will finish when their sum is equal to the `total` field.
|
|
|
|
With the task id you can look up the task directly:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619
|
|
--------------------------------------------------
|
|
// TEST[catch:missing]
|
|
|
|
The advantage of this API is that it integrates with `wait_for_completion=false`
|
|
to transparently return the status of completed tasks. If the task is completed
|
|
and `wait_for_completion=false` was set on it then it'll come back with
|
|
`results` or an `error` field. The cost of this feature is the document that
|
|
`wait_for_completion=false` creates at `.tasks/task/${taskId}`. It is up to
|
|
you to delete that document.
|
|
|
|
|
|
[float]
|
|
[[docs-delete-by-query-cancel-task-api]]
|
|
==== Cancel a delete by query operation
|
|
|
|
Any delete by query can be canceled using the <<tasks,task cancel API>>:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
|
|
--------------------------------------------------
|
|
|
|
The task ID can be found using the <<tasks,tasks API>>.
|
|
|
|
Cancellation should happen quickly but might take a few seconds. The task status
|
|
API above will continue to list the delete by query task until this task checks that it
|
|
has been cancelled and terminates itself.
|