212 lines
10 KiB
Plaintext
212 lines
10 KiB
Plaintext
[[java-rest-high-document-bulk]]
|
|
=== Bulk API
|
|
|
|
NOTE: The Java High Level REST Client provides the <<java-rest-high-document-bulk-processor>> to assist with bulk requests
|
|
|
|
[[java-rest-high-document-bulk-request]]
|
|
==== Bulk Request
|
|
|
|
A `BulkRequest` can be used to execute multiple index, update and/or delete
|
|
operations using a single request.
|
|
|
|
It requires at least one operation to be added to the Bulk request:
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-request]
|
|
--------------------------------------------------
|
|
<1> Creates the `BulkRequest`
|
|
<2> Adds a first `IndexRequest` to the Bulk request. See <<java-rest-high-document-index>>
|
|
for more information on how to build `IndexRequest`.
|
|
<3> Adds a second `IndexRequest`
|
|
<4> Adds a third `IndexRequest`
|
|
|
|
WARNING: The Bulk API supports only documents encoded in JSON or SMILE. Providing documents
|
|
in any other format will result in an error.
|
|
|
|
And different operation types can be added to the same `BulkRequest`:
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-request-with-mixed-operations]
|
|
--------------------------------------------------
|
|
<1> Adds a `DeleteRequest` to the `BulkRequest`. See <<java-rest-high-document-delete>>
|
|
for more information on how to build `DeleteRequest`.
|
|
<2> Adds an `UpdateRequest` to the `BulkRequest`. See <<java-rest-high-document-update>>
|
|
for more information on how to build `UpdateRequest`.
|
|
<3> Adds an `IndexRequest` using the SMILE format
|
|
|
|
==== Optional arguments
|
|
The following arguments can optionally be provided:
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-request-timeout]
|
|
--------------------------------------------------
|
|
<1> Timeout to wait for the bulk request to be performed as a `TimeValue`
|
|
<2> Timeout to wait for the bulk request to be performed as a `String`
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-request-refresh]
|
|
--------------------------------------------------
|
|
<1> Refresh policy as a `WriteRequest.RefreshPolicy` instance
|
|
<2> Refresh policy as a `String`
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-request-active-shards]
|
|
--------------------------------------------------
|
|
<1> Sets the number of shard copies that must be active before proceeding with
|
|
the index/update/delete operations.
|
|
<2> Number of shard copies provided as a `ActiveShardCount`: can be `ActiveShardCount.ALL`,
|
|
`ActiveShardCount.ONE` or `ActiveShardCount.DEFAULT` (default)
|
|
|
|
|
|
[[java-rest-high-document-bulk-sync]]
|
|
==== Synchronous Execution
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-execute]
|
|
--------------------------------------------------
|
|
|
|
[[java-rest-high-document-bulk-async]]
|
|
==== Asynchronous Execution
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-execute-async]
|
|
--------------------------------------------------
|
|
<1> Called when the execution is successfully completed. The response is
|
|
provided as an argument and contains a list of individual results for each
|
|
operation that was executed. Note that one or more operations might have
|
|
failed while the others have been successfully executed.
|
|
<2> Called when the whole `BulkRequest` fails. In this case the raised
|
|
exception is provided as an argument and no operation has been executed.
|
|
|
|
[[java-rest-high-document-bulk-response]]
|
|
==== Bulk Response
|
|
|
|
The returned `BulkResponse` contains information about the executed operations and
|
|
allows to iterate over each result as follows:
|
|
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-response]
|
|
--------------------------------------------------
|
|
<1> Iterate over the results of all operations
|
|
<2> Retrieve the response of the operation (successful or not), can be `IndexResponse`,
|
|
`UpdateResponse` or `DeleteResponse` which can all be seen as `DocWriteResponse` instances
|
|
<3> Handle the response of an index operation
|
|
<4> Handle the response of a update operation
|
|
<5> Handle the response of a delete operation
|
|
|
|
The Bulk response provides a method to quickly check if one or more operation has failed:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-has-failures]
|
|
--------------------------------------------------
|
|
<1> This method returns `true` if at least one operation failed
|
|
|
|
In such situation it is necessary to iterate over all operation results in order to check
|
|
if the operation failed, and if so, retrieve the corresponding failure:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-errors]
|
|
--------------------------------------------------
|
|
<1> Indicate if a given operation failed
|
|
<2> Retrieve the failure of the failed operation
|
|
|
|
[[java-rest-high-document-bulk-processor]]
|
|
==== Bulk Processor
|
|
|
|
The `BulkProcessor` simplifies the usage of the Bulk API by providing
|
|
a utility class that allows index/update/delete operations to be
|
|
transparently executed as they are added to the processor.
|
|
|
|
In order to execute the requests, the `BulkProcessor` requires 3 components:
|
|
|
|
`RestHighLevelClient`:: This client is used to execute the `BulkRequest`
|
|
and to retrieve the `BulkResponse`
|
|
`BulkProcessor.Listener`:: This listener is called before and after
|
|
every `BulkRequest` execution or when a `BulkRequest` failed
|
|
`ThreadPool`:: The `BulkRequest` executions are done using threads from this
|
|
pool, allowing the `BulkProcessor` to work in a non-blocking manner and to
|
|
accept new index/update/delete requests while bulk requests are executing.
|
|
|
|
Then the `BulkProcessor.Builder` class can be used to build a new `BulkProcessor`:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-init]
|
|
--------------------------------------------------
|
|
<1> Create the `ThreadPool` using the given `Settings`
|
|
<2> Create the `BulkProcessor.Listener`
|
|
<3> This method is called before each execution of a `BulkRequest`
|
|
<4> This method is called after each execution of a `BulkRequest`
|
|
<5> This method is called when a `BulkRequest` failed
|
|
<6> Create the `BulkProcessor` by calling the `build()` method from
|
|
the `BulkProcessor.Builder`. The `RestHighLevelClient.bulkAsync()`
|
|
method will be used to execute the `BulkRequest` under the hood.
|
|
|
|
The `BulkProcessor.Builder` provides methods to configure how the `BulkProcessor`
|
|
should handle requests execution:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-options]
|
|
--------------------------------------------------
|
|
<1> Set when to flush a new bulk request based on the number of
|
|
actions currently added (defaults to 1000, use -1 to disable it)
|
|
<2> Set when to flush a new bulk request based on the size of
|
|
actions currently added (defaults to 5Mb, use -1 to disable it)
|
|
<3> Set the number of concurrent requests allowed to be executed
|
|
(default to 1, use 0 to only allow the execution of a single request)
|
|
<4> Set a flush interval flushing any `BulkRequest` pending if the
|
|
interval passes (defaults to not set)
|
|
<5> Set a constant back off policy that initially waits for 1 second
|
|
and retries up to 3 times. See `BackoffPolicy.noBackoff()`,
|
|
`BackoffPolicy.constantBackoff()` and `BackoffPolicy.exponentialBackoff()`
|
|
for more options.
|
|
|
|
Once the `BulkProcessor` is created requests can be added to it:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-add]
|
|
--------------------------------------------------
|
|
|
|
The requests will be executed by the `BulkProcessor`, which takes care of
|
|
calling the `BulkProcessor.Listener` for every bulk request.
|
|
|
|
The listener provides methods to access to the `BulkRequest` and the `BulkResponse`:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-listener]
|
|
--------------------------------------------------
|
|
<1> Called before each execution of a `BulkRequest`, this method allows
|
|
to know the number of operations that are going to be executed within the `BulkRequest`
|
|
<2> Called after each execution of a `BulkRequest`, this method allows
|
|
to know if the `BulkResponse` contains errors
|
|
<3> Called if the `BulkRequest` failed, this method allows to know
|
|
the failure
|
|
|
|
Once all requests have been added to the `BulkProcessor`, its instance needs to
|
|
be closed closed using one of the two available closing methods.
|
|
|
|
The `awaitClose()` method can be used to wait until all requests have been processed
|
|
or the specified waiting time elapses:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-await]
|
|
--------------------------------------------------
|
|
<1> The method returns `true` if all bulk requests completed and `false` if the
|
|
waiting time elapsed before all the bulk requests completed
|
|
|
|
The `close()` method can be used to immediately close the `BulkProcessor`:
|
|
["source","java",subs="attributes,callouts,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{doc-tests}/CRUDDocumentationIT.java[bulk-processor-close]
|
|
--------------------------------------------------
|
|
|
|
Both methods flush the requests added to the processor before closing the processor
|
|
and also forbid any new request to be added to it.
|