The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.
1. Running on shards and replicas
In order to run watches on the nodes with the watches index on its
primaries and replicas. To ensure that watches do not run twice, there is
a logic which checks the local shards, runs a murmurhash on the id and
runs modulo against the number of shards and replicas, this is the way to
find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
are basically just aliases for indexing actions, among them the
put/delete/get watch actions, the acknowledgement action, the de/activate
actions
3. Stats action moved to a broadcast node action, because we potentially
have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
the cluster state and then listeners acts on those. Because of this watches
can be running on two systems, if you those have different cluster state
versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
schedule engine most classes do not need a lot of resources while running.
However they have to run, because of the execute watch API, which can hit
any node - it does not make sense to find the right shard for this watch
and only then execute (as this also has to work with a watch, that has not
been stored before)
6. By using a indexing operation listener, each storing of a watch now
parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
different execution engines, how many watches are scheduled, so this is
not doing a search anymore
There will be follow up commits on this one, mainly to ensure BWC compatibility.
Original commit: elastic/x-pack-elasticsearch@0adb46e658
* [DOCS] Add ML limitations
* [DOCS] Address feedback about ML limitations
* [DOCS] Change ML limitations capitalization
Original commit: elastic/x-pack-elasticsearch@41682d8d93
When the active directory realm was refactored to add support for authenticating against multiple
domains, only the default authenticator respected the user_search.filter setting. This commit moves
this down to the base authenticator and also changes the UPN filter to not include sAMAccountName
in the filter.
Original commit: elastic/x-pack-elasticsearch@d2c19c9bee
* [DOCS] Edits for description of ML results resource
* [DOCS] Adds Authorization section to ML API pages
Original commit: elastic/x-pack-elasticsearch@797761e7ae
* [DOCS] Fix for prelertcategory
* [DOCS] _preview returns a page of data
* [DOCS] Added adv options e.g. background_persist_interval"
* [DOCS] Clarify meanings of model_snapshot params
* [DOCS] Format fixes
* [DOCS] Include _all keyword
* [DOCS] Explain retain.
* [DOCS] Further explanations for model size limits
* [DOCS] Format fixes in quick ref
* [DOCS] Update for exclude_interim
* [DOCS] Update for exclude_interim
* [DOCS] Update for exclude_interim
Original commit: elastic/x-pack-elasticsearch@cdd2fcefdd
* [DOCS] Add role info to ML getting started docs
* [DOCS] Getting started with sample ML data
* [DOCS] Getting started with server-metrics sample data
Original commit: elastic/x-pack-elasticsearch@2f268f87b4
Adds a gradle build for documentation testing, partially stolen from the xpack meta repo. Updated to make work with the current setup
index.asciidoc was updated/enabled as a test, and various pre-existing tests appear to be working. There are a large number of tests still ignored due to missing proper // CONSOLE tags
Original commit: elastic/x-pack-elasticsearch@1d596f0be3
* [DOCS] Review of close job and job stats
* [DOCS] Add force close
* [DOCS] Remove invalid params from get records
* [DOCS] Remove invalid params from get buckets
* [DOCS] Job resource corrections
Original commit: elastic/x-pack-elasticsearch@bc68d05097
* Adding limitations of Watch Edit page
* Removing image reference for now until image dir referencing is sorted out
* More end-user-friendly phrasing
* Language changes + bringing back image
Original commit: elastic/x-pack-elasticsearch@06bf93ee99
* Amending docs on security privileges required for Watcher
Previously, the watcher_user and watcher_admin roles did not exist so we documented the actual security privileges necessary. Now that these roles exist and encapsulate the security privileges, we update the documentation to refer to the roles instead.
* Breaking up sentences. Putting main content up front.
* Include triggered watches as well
* Emphasize read-only operations
Original commit: elastic/x-pack-elasticsearch@720d84557c
* [DOCS] First draft of ML getting started tutorial
* [DOCS] More ML getting started content
* [DOCS] Getting started content for data feeds
* [DOCS] Added ML getting started screenshot
Original commit: elastic/x-pack-elasticsearch@73174d27e8
Alexander Reelsen
debadair
Lisa Cawley
Clinton Gormley
Sophie Chang
Jay Modi
Dimitris Athanasiou
Zachary Tong
Shaunak Kashyap
Ali Beyad