[[plugins-delete-by-query]] === Delete By Query Plugin The delete-by-query plugin adds support for deleting all of the documents (from one or more indices) which match the specified query. It is a replacement for the problematic _delete-by-query_ functionality which has been removed from Elasticsearch core. Internally, it uses the {ref}/search-request-scroll.html#scroll-scan[Scan/Scroll] and {ref}/docs-bulk.html[Bulk] APIs to delete documents in an efficient and safe manner. It is slower than the old _delete-by-query_ functionality, but fixes the problems with the previous implementation. To understand more about why we removed delete-by-query from core and about the semantics of the new implementation, see <>. [TIP] ============================================ Queries which match large numbers of documents may run for a long time, as every document has to be deleted individually. Don't use _delete-by-query_ to clean out all or most documents in an index. Rather create a new index and perhaps reindex the documents you want to keep. ============================================ [float] ==== Installation This plugin can be installed using the plugin manager: [source,sh] ---------------------------------------------------------------- sudo bin/plugin install delete-by-query ---------------------------------------------------------------- The plugin must be installed on every node in the cluster, and each node must be restarted after installation. [float] ==== Removal The plugin can be removed with the following command: [source,sh] ---------------------------------------------------------------- sudo bin/plugin remove delete-by-query ---------------------------------------------------------------- The node must be stopped before removing the plugin. [[delete-by-query-usage]] ==== Using Delete-by-Query The query can either be provided using a simple query string as a parameter: [source,shell] -------------------------------------------------- DELETE /twitter/tweet/_query?q=user:kimchy -------------------------------------------------- // AUTOSENSE or using the {ref}/query-dsl.html[Query DSL] defined within the request body: [source,js] -------------------------------------------------- DELETE /twitter/tweet/_query { "query": { <1> "term": { "user": "kimchy" } } } -------------------------------------------------- // AUTOSENSE <1> The query must be passed as a value to the `query` key, in the same way as the {ref}/search-search.html[search api]. Both of the above examples end up doing the same thing, which is to delete all tweets from the twitter index for the user `kimchy`. Delete-by-query supports deletion across {ref}/search-search.html#search-multi-index-type[multiple indices and multiple types]. [float] === Query-string parameters The following query string parameters are supported: `q`:: Instead of using the {ref}/query-dsl.html[Query DSL] to pass a `query` in the request body, you can use the `q` query string parameter to specify a query using {ref}/query-dsl-query-string-query.html#query-string-syntax[`query_string` syntax]. In this case, the following additional parameters are supported: `df`, `analyzer`, `default_operator`, `lowercase_expanded_terms`, `analyze_wildcard` and `lenient`. See {ref}/search-uri-request.html[URI search request] for details. `size`:: The number of hits returned *per shard* by the {ref}/search-request-scroll.html#scroll-scan[scan] request. Defaults to 10. May also be specified in the request body. `timeout`:: The maximum execution time of the delete by query process. Once expired, no more documents will be deleted. `routing`:: A comma separated list of routing values to control which shards the delete by query request should be executed on. When using the `q` parameter, the following additional parameters are supported (as explained in {ref}/search-uri-request.html[URI search request]): `df`, `analyzer`, `default_operator`. [float] === Response body The JSON response looks like this: [source,js] -------------------------------------------------- { "took" : 639, "timed_out" : false, "_indices" : { "_all" : { "found" : 5901, "deleted" : 5901, "missing" : 0, "failed" : 0 }, "twitter" : { "found" : 5901, "deleted" : 5901, "missing" : 0, "failed" : 0 } }, "failures" : [ ] } -------------------------------------------------- Internally, the query is used to execute an initial {ref}/search-request-scroll.html#scroll-scan[scroll/scan] request. As hits are pulled from the scroll API, they are passed to the {ref}/docs-bulk.html[Bulk API] for deletion. IMPORTANT: Delete by query will only delete the version of the document that was visible to search at the time the request was executed. Any documents that have been reindexed or updated during execution will not be deleted. Since documents can be updated or deleted by external operations during the _scan-scroll-bulk_ process, the plugin keeps track of different counters for each index, with the totals displayed under the `_all` index. The counters are as follows: `found`:: The number of documents matching the query for the given index. `deleted`:: The number of documents successfully deleted for the given index. `missing`:: The number of documents that were missing when the plugin tried to delete them. Missing documents were present when the original query was run, but have already been deleted by another process. `failed`:: The number of documents that failed to be deleted for the given index. A document may fail to be deleted if it has been updated to a new version by another process, or if the shard containing the document has gone missing due to hardware failure, for example. [[delete-by-query-plugin-reason]] ==== Why Delete-By-Query is a plugin The old delete-by-query API in Elasticsearch 1.x was fast but problematic. We decided to remove the feature from Elasticsearch for these reasons: Forward compatibility:: The old implementation wrote a delete-by-query request, including the query, to the transaction log. This meant that, when upgrading to a new version, old unsupported queries which cannot be executed might exist in the translog, thus causing data corruption. Consistency and correctness:: The old implementation executed the query and deleted all matching docs on the primary first. It then repeated this procedure on each replica shard. There was no guarantee that the queries on the primary and the replicas matched the same document, so it was quite possible to end up with different documents on each shard copy. Resiliency:: The old implementation could cause out-of-memory exceptions, merge storms, and dramatic slow downs if used incorrectly. [float] === New delete-by-query implementation The new implementation, provided by this plugin, is built internally using {ref}/search-request-scroll.html#scroll-scan[scan and scroll] to return the document IDs and versions of all the documents that need to be deleted. It then uses the {ref}/docs-bulk.html[`bulk` API] to do the actual deletion. This can have performance as well as visibility implications. Delete-by-query now has the following semantics: non-atomic:: A delete-by-query may fail at any time while some documents matching the query have already been deleted. try-once:: A delete-by-query may fail at any time and will not retry it's execution. All retry logic is left to the user. syntactic sugar:: A delete-by-query is equivalent to a scan/scroll search and corresponding bulk-deletes by ID. point-in-time:: A delete-by-query will only delete the documents that are visible at the point in time the delete-by-query was started, equivalent to the scan/scroll API. consistent:: A delete-by-query will yield consistent results across all replicas of a shard. forward-compatible:: A delete-by-query will only send IDs to the shards as deletes such that no queries are stored in the transaction logs that might not be supported in the future. visibility:: The effect of a delete-by-query request will not be visible to search until the user refreshes the index, or the index is refreshed automatically. The new implementation suffers from two issues, which is why we decided to move the functionality to a plugin instead of replacing the feautre in core: * It is not as fast as the previous implementation. For most use cases, this difference should not be noticeable but users running delete-by-query on many matching documents may be affected. * There is currently no way to monitor or cancel a running delete-by-query request, except for the `timeout` parameter. We have plans to solve both of these issues in a later version of Elasticsearch.