[7.x] [DOCS] Reformats the update by query API. (#46199) (#58700)

Co-authored-by: debadair <debadair@elastic.co>
This commit is contained in:
James Rodewig 2020-06-29 17:50:32 -04:00 committed by GitHub
parent 84513c7539
commit 8341ebc061
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 318 additions and 287 deletions

View File

@ -19,11 +19,6 @@ POST /twitter/_delete_by_query
-------------------------------------------------- --------------------------------------------------
// TEST[setup:big_twitter] // TEST[setup:big_twitter]
[[docs-delete-by-query-api-request]]
==== {api-request-title}
`POST /<index>/_delete_by_query`
//// ////
[source,console-result] [source,console-result]
@ -49,6 +44,11 @@ POST /twitter/_delete_by_query
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/] // TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]
//// ////
[[docs-delete-by-query-api-request]]
==== {api-request-title}
`POST /<index>/_delete_by_query`
[[docs-delete-by-query-api-desc]] [[docs-delete-by-query-api-desc]]
==== {api-description-title} ==== {api-description-title}
@ -89,8 +89,7 @@ request to be refreshed. Unlike the delete API, it does not support
If the request contains `wait_for_completion=false`, {es} If the request contains `wait_for_completion=false`, {es}
performs some preflight checks, launches the request, and returns a performs some preflight checks, launches the request, and returns a
<<docs-delete-by-query-task-api,`task`>> <<tasks,`task`>> you can use to cancel or get the status of the task. {es} creates a
you can use to cancel or get the status of the task. {es} creates a
record of this task as a document at `.tasks/task/${taskId}`. When you are record of this task as a document at `.tasks/task/${taskId}`. When you are
done with a task, you should delete the task document so {es} can reclaim the done with a task, you should delete the task document so {es} can reclaim the
space. space.
@ -227,9 +226,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=version] include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=version]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout] include::{docdir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
[[docs-delete-by-query-api-request-body]] [[docs-delete-by-query-api-request-body]]
==== {api-request-body-title} ==== {api-request-body-title}
@ -239,7 +236,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards
using the <<query-dsl,Query DSL>>. using the <<query-dsl,Query DSL>>.
[[docs-delete-by-quer-api-response-body]] [[docs-delete-by-query-api-response-body]]
==== Response body ==== Response body
////////////////////////// //////////////////////////
@ -330,7 +327,7 @@ The number of requests per second effectively executed during the delete by quer
`throttled_until_millis`:: `throttled_until_millis`::
This field should always be equal to zero in a `_delete_by_query` response. It only This field should always be equal to zero in a `_delete_by_query` response. It only
has meaning when using the <<docs-delete-by-query-task-api, Task API>>, where it has meaning when using the <<tasks, Task API>>, where it
indicates the next time (in milliseconds since epoch) a throttled request will be indicates the next time (in milliseconds since epoch) a throttled request will be
executed again in order to conform to `requests_per_second`. executed again in order to conform to `requests_per_second`.
@ -541,7 +538,7 @@ Adding `slices` to `_delete_by_query` just automates the manual process used in
the section above, creating sub-requests which means it has some quirks: the section above, creating sub-requests which means it has some quirks:
* You can see these requests in the * You can see these requests in the
<<docs-delete-by-query-task-api,Tasks APIs>>. These sub-requests are "child" <<tasks,Tasks APIs>>. These sub-requests are "child"
tasks of the task for the request with `slices`. tasks of the task for the request with `slices`.
* Fetching the status of the task for the request with `slices` only contains * Fetching the status of the task for the request with `slices` only contains
the status of completed slices. the status of completed slices.
@ -655,7 +652,7 @@ you to delete that document.
[float] [float]
[[docs-delete-by-query-cancel-task-api]] [[docs-delete-by-query-cancel-task-api]]
==== Cancel a delete by query operation ===== Cancel a delete by query operation
Any delete by query can be canceled using the <<tasks,task cancel API>>: Any delete by query can be canceled using the <<tasks,task cancel API>>:

View File

@ -1,10 +1,12 @@
[[docs-update-by-query]] [[docs-update-by-query]]
=== Update By Query API === Update By Query API
++++
<titleabbrev>Update by query</titleabbrev>
++++
The simplest usage of `_update_by_query` just performs an update on every Updates documents that match the specified query.
document in the index without changing the source. This is useful to If no query is specified, performs an update on every document in the index without
<<picking-up-a-new-property,pick up a new property>> or some other online modifying the source, which is useful for picking up mapping changes.
mapping change. Here is the API:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -12,7 +14,7 @@ POST twitter/_update_by_query?conflicts=proceed
-------------------------------------------------- --------------------------------------------------
// TEST[setup:big_twitter] // TEST[setup:big_twitter]
That will return something like this: ////
[source,console-result] [source,console-result]
-------------------------------------------------- --------------------------------------------------
@ -37,42 +39,262 @@ That will return something like this:
-------------------------------------------------- --------------------------------------------------
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/] // TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]
`_update_by_query` gets a snapshot of the index when it starts and indexes what ////
it finds using `internal` versioning. That means you'll get a version
conflict if the document changes between the time when the snapshot was taken
and when the index request is processed. When the versions match, the document
is updated and the version number is incremented.
NOTE: Since `internal` versioning does not support the value 0 as a valid [[docs-update-by-query-api-request]]
version number, documents with version equal to zero cannot be updated using ==== {api-request-title}
`_update_by_query` and will fail the request.
All update and query failures cause the `_update_by_query` to abort and are `POST /<index>/_update_by_query`
returned in the `failures` of the response. The updates that have been
performed still stick. In other words, the process is not rolled back, only
aborted. While the first failure causes the abort, all failures that are
returned by the failing bulk request are returned in the `failures` element; therefore
it's possible for there to be quite a few failed entities.
If you want to simply count version conflicts, and not cause the `_update_by_query` [[docs-update-by-query-api-desc]]
to abort, you can set `conflicts=proceed` on the url or `"conflicts": "proceed"` ==== {api-description-title}
in the request body. The first example does this because it is just trying to
pick up an online mapping change, and a version conflict simply means that the
conflicting document was updated between the start of the `_update_by_query`
and the time when it attempted to update the document. This is fine because
that update will have picked up the online mapping update.
Back to the API format, this will update tweets from the `twitter` index: You can specify the query criteria in the request URI or the request body
using the same syntax as the <<search-search,Search API>>.
[source,console] When you submit an update by query request, {es} gets a snapshot of the index
when it begins processing the request and updates matching documents using
`internal` versioning.
When the versions match, the document is updated and the version number is incremented.
If a document changes between the time that the snapshot is taken and
the update operation is processed, it results in a version conflict and the operation fails.
You can opt to count version conflicts instead of halting and returning by
setting `conflicts` to `proceed`.
NOTE: Documents with a version equal to 0 cannot be updated using update by
query because `internal` versioning does not support 0 as a valid
version number.
While processing an update by query request, {es} performs multiple search
requests sequentially to find all of the matching documents.
A bulk update request is performed for each batch of matching documents.
Any query or update failures cause the update by query request to fail and
the failures are shown in the response.
Any update requests that completed successfully still stick, they are not rolled back.
===== Refreshing shards
Specifying the `refresh` parameter refreshes all shards once the request completes.
This is different than the update API#8217;s `refresh` parameter, which causes just the shard
that received the request to be refreshed. Unlike the update API, it does not support
`wait_for`.
[[docs-update-by-query-task-api]]
===== Running update by query asynchronously
If the request contains `wait_for_completion=false`, {es}
performs some preflight checks, launches the request, and returns a
<<tasks,`task`>> you can use to cancel or get the status of the task.
{es} creates a record of this task as a document at `.tasks/task/${taskId}`.
When you are done with a task, you should delete the task document so
{es} can reclaim the space.
===== Waiting for active shards
`wait_for_active_shards` controls how many copies of a shard must be active
before proceeding with the request. See <<index-wait-for-active-shards>>
for details. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly the way they work in the
<<docs-bulk,Bulk API>>. Update by query uses scrolled searches, so you can also
specify the `scroll` parameter to control how long it keeps the search context
alive, for example `?scroll=10m`. The default is 5 minutes.
===== Throttling update requests
To control the rate at which update by query issues batches of update operations,
you can set `requests_per_second` to any positive decimal number. This pads each
batch with a wait time to throttle the rate. Set `requests_per_second` to `-1`
to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests
can be given a timeout that takes the request padding into account. The padding
time is the difference between the batch size divided by the
`requests_per_second` and the time spent writing. By default the batch size is
`1000`, so if `requests_per_second` is set to `500`:
[source,txt]
-------------------------------------------------- --------------------------------------------------
POST twitter/_update_by_query?conflicts=proceed target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
-------------------------------------------------- --------------------------------------------------
// TEST[setup:twitter]
You can also limit `_update_by_query` using the Since the batch is issued as a single `_bulk` request, large batch sizes
<<query-dsl,Query DSL>>. This will update all documents from the cause {es} to create many requests and wait before starting the next set.
`twitter` index for the user `kimchy`: This is "bursty" instead of "smooth".
[[docs-update-by-query-slice]]
===== Slicing
Update by query supports <<sliced-scroll, sliced scroll>> to parallelize the
update process. This can improve efficiency and provide a
convenient way to break the request down into smaller parts.
Setting `slices` to `auto` chooses a reasonable number for most indices.
If you're slicing manually or otherwise tuning automatic slicing, keep in mind
that:
* Query performance is most efficient when the number of `slices` is equal to
the number of shards in the index. If that number is large (for example,
500), choose a lower number as too many `slices` hurts performance. Setting
`slices` higher than the number of shards generally does not improve efficiency
and adds overhead.
* Update performance scales linearly across available resources with the
number of slices.
Whether query or update performance dominates the runtime depends on the
documents being reindexed and cluster resources.
[[docs-update-by-query-api-path-params]]
==== {api-path-parms-title}
`<index>`::
(Optional, string) A comma-separated list of index names to search. Use `_all`
or omit to search all indices.
[[docs-update-by-query-api-query-params]]
==== {api-query-parms-title}
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
+
Defaults to `true`.
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyzer]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyze_wildcard]
`conflicts`::
(Optional, string) What to do if delete by query hits version conflicts:
`abort` or `proceed`. Defaults to `abort`.
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=default_operator]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=df]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
+
Defaults to `open`.
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=from]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=lenient]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=max_docs]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pipeline]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=preference]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search-q]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=request_cache]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=refresh]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=requests_per_second]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=routing]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll_size]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_timeout]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=slices]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sort]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_excludes]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_includes]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=stats]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=terminate_after]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=version]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
[[docs-update-by-query-api-request-body]]
==== {api-request-body-title}
`query`::
(Optional, <<query-dsl,query object>>) Specifies the documents to update
using the <<query-dsl,Query DSL>>.
[[docs-update-by-query-api-response-body]]
==== Response body
`took`::
The number of milliseconds from start to end of the whole operation.
`timed_out`::
This flag is set to `true` if any of the requests executed during the
update by query execution has timed out.
`total`::
The number of documents that were successfully processed.
`updated`::
The number of documents that were successfully updated.
`deleted`::
The number of documents that were successfully deleted.
`batches`::
The number of scroll responses pulled back by the update by query.
`version_conflicts`::
The number of version conflicts that the update by query hit.
`noops`::
The number of documents that were ignored because the script used for
the update by query returned a `noop` value for `ctx.op`.
`retries`::
The number of retries attempted by update by query. `bulk` is the number of bulk
actions retried, and `search` is the number of search actions retried.
`throttled_millis`::
Number of milliseconds the request slept to conform to `requests_per_second`.
`requests_per_second`::
The number of requests per second effectively executed during the update by query.
`throttled_until_millis`::
This field should always be equal to zero in an `_update_by_query` response. It only
has meaning when using the <<docs-update-by-query-task-api, Task API>>, where it
indicates the next time (in milliseconds since epoch) a throttled request will be
executed again in order to conform to `requests_per_second`.
`failures`::
Array of failures if there were any unrecoverable errors during the process. If
this is non-empty then the request aborted because of those failures.
Update by query is implemented using batches. Any failure causes the entire
process to abort, but all failures in the current batch are collected into the
array. You can use the `conflicts` option to prevent reindex from aborting on
version conflicts.
[[docs-update-by-query-api-example]]
==== {api-examples-title}
The simplest usage of `_update_by_query` just performs an update on every
document in the index without changing the source. This is useful to
<<picking-up-a-new-property,pick up a new property>> or some other online
mapping change.
To update selected documents, specify a query in the request body:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -91,11 +313,36 @@ POST twitter/_update_by_query?conflicts=proceed
way as the <<search-search,Search API>>. You can also use the `q` way as the <<search-search,Search API>>. You can also use the `q`
parameter in the same way as the search API. parameter in the same way as the search API.
So far we've only been updating documents without changing their source. That Update documents in multiple indices:
is genuinely useful for things like
<<picking-up-a-new-property,picking up new properties>> but it's only half the [source,console]
fun. `_update_by_query` <<modules-scripting-using,supports scripts>> to update --------------------------------------------------
the document. This will increment the `likes` field on all of kimchy's tweets: POST twitter,blog/_update_by_query
--------------------------------------------------
// TEST[s/^/PUT twitter\nPUT blog\n/]
Limit the update by query operation to shards that a particular routing value:
[source,console]
--------------------------------------------------
POST twitter/_update_by_query?routing=1
--------------------------------------------------
// TEST[setup:twitter]
By default update by query uses scroll batches of 1000.
You can change the batch size with the `scroll_size` parameter:
[source,console]
--------------------------------------------------
POST twitter/_update_by_query?scroll_size=100
--------------------------------------------------
// TEST[setup:twitter]
[[docs-update-by-query-api-source]]
===== Update the document source
Update by query supports scripts to update the document source.
For example, the following request increments the likes field for all of kimchys tweets:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -114,62 +361,29 @@ POST twitter/_update_by_query
-------------------------------------------------- --------------------------------------------------
// TEST[setup:twitter] // TEST[setup:twitter]
Just as in <<docs-update,Update API>> you can set `ctx.op` to change the Note that `conflicts=proceed` is not specified in this example. In this case, a
operation that is executed: version conflict should halt the process so you can handle the failure.
As with the <<docs-update,Update API>>, you can set `ctx.op` to change the
operation that is performed:
[horizontal] [horizontal]
`noop`:: `noop`::
Set `ctx.op = "noop"` if your script decides that it doesn't have to make any changes.
Set `ctx.op = "noop"` if your script decides that it doesn't have to make any The update by query operation skips updating the document and increments the `noop` counter.
changes. That will cause `_update_by_query` to omit that document from its updates.
This no operation will be reported in the `noop` counter in the
<<docs-update-by-query-response-body, response body>>.
`delete`:: `delete`::
Set `ctx.op = "delete"` if your script decides that the document should be deleted.
The update by query operation deletes the document and increments the `deleted` counter.
Set `ctx.op = "delete"` if your script decides that the document must be Update by query only supports `update`, `noop`, and `delete`.
deleted. The deletion will be reported in the `deleted` counter in the Setting `ctx.op` to anything else is an error. Setting any other field in `ctx` is an error.
<<docs-update-by-query-response-body, response body>>. This API only enables you to modify the source of matching documents, you cannot move them.
Setting `ctx.op` to anything else is an error. Setting any [[docs-update-by-query-api-ingest-pipeline]]
other field in `ctx` is an error. ===== Update documents using an ingest pipeline
Note that we stopped specifying `conflicts=proceed`. In this case we want a Update by query can use the <<ingest>> feature by specifying a `pipeline`:
version conflict to abort the process so we can handle the failure.
This API doesn't allow you to move the documents it touches, just modify their
source. This is intentional! We've made no provisions for removing the document
from its original location.
It's also possible to do this whole thing on multiple indexes at once, just
like the search API:
[source,console]
--------------------------------------------------
POST twitter,blog/_update_by_query
--------------------------------------------------
// TEST[s/^/PUT twitter\nPUT blog\n/]
If you provide `routing` then the routing is copied to the scroll query,
limiting the process to the shards that match that routing value:
[source,console]
--------------------------------------------------
POST twitter/_update_by_query?routing=1
--------------------------------------------------
// TEST[setup:twitter]
By default `_update_by_query` uses scroll batches of 1000. You can change the
batch size with the `scroll_size` URL parameter:
[source,console]
--------------------------------------------------
POST twitter/_update_by_query?scroll_size=100
--------------------------------------------------
// TEST[setup:twitter]
`_update_by_query` can also use the <<ingest>> feature by
specifying a `pipeline` like this:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -187,162 +401,10 @@ POST twitter/_update_by_query?pipeline=set-foo
-------------------------------------------------- --------------------------------------------------
// TEST[setup:twitter] // TEST[setup:twitter]
[float]
==== URL parameters
In addition to the standard parameters like `pretty`, the Update By Query API
also supports `refresh`, `wait_for_completion`, `wait_for_active_shards`, `timeout`,
and `scroll`.
Sending the `refresh` will update all shards in the index being updated when
the request completes. This is different than the Update API's `refresh`
parameter, which causes just the shard that received the new data to be indexed.
Also unlike the Update API it does not support `wait_for`.
If the request contains `wait_for_completion=false` then Elasticsearch will
perform some preflight checks, launch the request, and then return a `task`
which can be used with <<docs-update-by-query-task-api,Tasks APIs>>
to cancel or get the status of the task. Elasticsearch will also create a
record of this task as a document at `.tasks/task/${taskId}`. This is yours
to keep or remove as you see fit. When you are done with it, delete it so
Elasticsearch can reclaim the space it uses.
`wait_for_active_shards` controls how many copies of a shard must be active
before proceeding with the request. See <<index-wait-for-active-shards,here>>
for details. `timeout` controls how long each write request waits for unavailable
shards to become available. Both work exactly how they work in the
<<docs-bulk,Bulk API>>. Because `_update_by_query` uses scroll search, you can also specify
the `scroll` parameter to control how long it keeps the "search context" alive,
e.g. `?scroll=10m`. By default it's 5 minutes.
`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
`1000`, etc.) and throttles the rate at which `_update_by_query` issues batches of
index operations by padding each batch with a wait time. The throttling can be
disabled by setting `requests_per_second` to `-1`.
The throttling is done by waiting between batches so that scroll that
`_update_by_query` uses internally can be given a timeout that takes into
account the padding. The padding time is the difference between the batch size
divided by the `requests_per_second` and the time spent writing. By default the
batch size is `1000`, so if the `requests_per_second` is set to `500`:
[source,txt]
--------------------------------------------------
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
--------------------------------------------------
Since the batch is issued as a single `_bulk` request, large batch sizes will
cause Elasticsearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth". The default is `-1`.
[float] [float]
[[docs-update-by-query-response-body]] [[docs-update-by-query-fetch-tasks]]
==== Response body ===== Get the status of update by query operations
//////////////////////////
[source,console]
--------------------------------------------------
POST /twitter/_update_by_query?conflicts=proceed
--------------------------------------------------
// TEST[setup:twitter]
//////////////////////////
The JSON response looks like this:
[source,console-result]
--------------------------------------------------
{
"took" : 147,
"timed_out": false,
"total": 5,
"updated": 5,
"deleted": 0,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures" : [ ]
}
--------------------------------------------------
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]
[horizontal]
`took`::
The number of milliseconds from start to end of the whole operation.
`timed_out`::
This flag is set to `true` if any of the requests executed during the
update by query execution has timed out.
`total`::
The number of documents that were successfully processed.
`updated`::
The number of documents that were successfully updated.
`deleted`::
The number of documents that were successfully deleted.
`batches`::
The number of scroll responses pulled back by the update by query.
`version_conflicts`::
The number of version conflicts that the update by query hit.
`noops`::
The number of documents that were ignored because the script used for
the update by query returned a `noop` value for `ctx.op`.
`retries`::
The number of retries attempted by update by query. `bulk` is the number of bulk
actions retried, and `search` is the number of search actions retried.
`throttled_millis`::
Number of milliseconds the request slept to conform to `requests_per_second`.
`requests_per_second`::
The number of requests per second effectively executed during the update by query.
`throttled_until_millis`::
This field should always be equal to zero in an `_update_by_query` response. It only
has meaning when using the <<docs-update-by-query-task-api, Task API>>, where it
indicates the next time (in milliseconds since epoch) a throttled request will be
executed again in order to conform to `requests_per_second`.
`failures`::
Array of failures if there were any unrecoverable errors during the process. If
this is non-empty then the request aborted because of those failures.
Update by query is implemented using batches. Any failure causes the entire
process to abort, but all failures in the current batch are collected into the
array. You can use the `conflicts` option to prevent reindex from aborting on
version conflicts.
[float]
[[docs-update-by-query-task-api]]
==== Works with the Task API
You can fetch the status of all running update by query requests with the You can fetch the status of all running update by query requests with the
<<tasks,Task API>>: <<tasks,Task API>>:
@ -421,7 +483,7 @@ you to delete that document.
[float] [float]
[[docs-update-by-query-cancel-task-api]] [[docs-update-by-query-cancel-task-api]]
==== Works with the Cancel Task API ===== Cancel an update by query operation
Any update by query can be cancelled using the <<tasks,Task Cancel API>>: Any update by query can be cancelled using the <<tasks,Task Cancel API>>:
@ -439,7 +501,7 @@ that it has been cancelled and terminates itself.
[float] [float]
[[docs-update-by-query-rethrottle]] [[docs-update-by-query-rethrottle]]
==== Rethrottling ===== Change throttling for a request
The value of `requests_per_second` can be changed on a running update by query The value of `requests_per_second` can be changed on a running update by query
using the `_rethrottle` API: using the `_rethrottle` API:
@ -458,17 +520,9 @@ query takes effect immediately, but rethrotting that slows down the query will
take effect after completing the current batch. This prevents scroll take effect after completing the current batch. This prevents scroll
timeouts. timeouts.
[float]
[[docs-update-by-query-slice]]
==== Slicing
Update by query supports <<sliced-scroll>> to parallelize the updating process.
This parallelization can improve efficiency and provide a convenient way to
break the request down into smaller parts.
[float] [float]
[[docs-update-by-query-manual-slice]] [[docs-update-by-query-manual-slice]]
===== Manual slicing ===== Slice manually
Slice an update by query manually by providing a slice id and total number of Slice an update by query manually by providing a slice id and total number of
slices to each request: slices to each request:
@ -522,7 +576,7 @@ Which results in a sensible `total` like this one:
[float] [float]
[[docs-update-by-query-automatic-slice]] [[docs-update-by-query-automatic-slice]]
===== Automatic slicing ===== Use automatic slicing
You can also let update by query automatically parallelize using You can also let update by query automatically parallelize using
<<sliced-scroll>> to slice on `_id`. Use `slices` to specify the number of <<sliced-scroll>> to slice on `_id`. Use `slices` to specify the number of
@ -590,29 +644,9 @@ being updated.
* Each sub-request gets a slightly different snapshot of the source index * Each sub-request gets a slightly different snapshot of the source index
though these are all taken at approximately the same time. though these are all taken at approximately the same time.
[float]
[[docs-update-by-query-picking-slices]]
====== Picking the number of slices
If slicing automatically, setting `slices` to `auto` will choose a reasonable
number for most indices. If you're slicing manually or otherwise tuning
automatic slicing, use these guidelines.
Query performance is most efficient when the number of `slices` is equal to the
number of shards in the index. If that number is large, (for example,
500) choose a lower number as too many `slices` will hurt performance. Setting
`slices` higher than the number of shards generally does not improve efficiency
and adds overhead.
Update performance scales linearly across available resources with the
number of slices.
Whether query or update performance dominates the runtime depends on the
documents being reindexed and cluster resources.
[float] [float]
[[picking-up-a-new-property]] [[picking-up-a-new-property]]
==== Pick up a new property ===== Pick up a new property
Say you created an index without dynamic mapping, filled it with data, and then Say you created an index without dynamic mapping, filled it with data, and then
added a mapping value to pick up more fields from the data: added a mapping value to pick up more fields from the data: