OpenSearch/x-pack
Alexander Reelsen 139128856a
Watcher: Use Bulkprocessor in HistoryStore/TriggeredWatchStore (#32490)
Currently a watch execution results in one bulk request, when the
triggered watches are written into the that index, that need to be
executed. However the update of the watch status, the creation of the
watch history entry as well as the deletion of the triggered watches
index are all single document operations.

This can have quite a negative impact, once you are executing a lot of
watches, as each execution results in 4 documents writes, three of them
being single document actions.

This commit switches to a bulk processor instead of a single document
action for writing watch history entries and deleting triggered watch
entries. However the defaults are to run synchronous as before because
the number of concurrent requests is set to 0. This also fixes a bug,
where the deletion of the triggered watch entry was done asynchronously.

However if you have a high number of watches being executed, you can
configure watcher to delete the triggered watches entries as well as
writing the watch history entries via bulk requests.

The triggered watches deletions should still happen in a timely manner,
where as the history entries might actually be bound by size as one
entry can easily have 20kb.

The following settings have been added:

- xpack.watcher.bulk.actions (default 1)
- xpack.watcher.bulk.concurrent_requests (default 0)
- xpack.watcher.bulk.flush_interval (default 1s)
- xpack.watcher.bulk.size (default 1mb)

The drawback of this is of course, that on a node outage you might end
up with watch history entries not being written or watches needing to be
executing again because they have not been deleted from the triggered
watches index. The window of these two cases increases configuring the bulk processor to wait to reach certain thresholds.
2018-09-18 10:25:16 +02:00
..
dev-tools Build: Merge xpack checkstyle config into core (#33399) 2018-09-05 09:17:02 -04:00
docs Docs: Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...' (#33744) 2018-09-17 15:35:54 -04:00
license-tools Build: Rework shadow plugin configuration (#32409) 2018-08-21 20:03:28 -04:00
plugin Watcher: Use Bulkprocessor in HistoryStore/TriggeredWatchStore (#32490) 2018-09-18 10:25:16 +02:00
protocol Test fix - Graph vertices could appear in different orders based on map insertion sequence (#33709) 2018-09-14 13:46:36 +01:00
qa Docs: Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...' (#33744) 2018-09-17 15:35:54 -04:00
test Build: Rework shadow plugin configuration (#32409) 2018-08-21 20:03:28 -04:00
transport-client Run forbidden api checks with runtimeJavaVersion (#32947) 2018-08-22 09:05:22 +03:00
NOTICE.txt Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
README.md Reorganize license files 2018-04-20 15:33:59 -07:00
build.gradle Build: Merge xpack checkstyle config into core (#33399) 2018-09-05 09:17:02 -04:00

README.md

Elastic License Functionality

This directory tree contains files subject to the Elastic License. The files subject to the Elastic License are grouped in this directory to clearly separate them from files licensed under the Apache License 2.0.