c62f6f8177
The distribution of watches now happens on the node which holds the watches index, instead of on the master node. This requires several changes to the current implementation. 1. Running on shards and replicas In order to run watches on the nodes with the watches index on its primaries and replicas. To ensure that watches do not run twice, there is a logic which checks the local shards, runs a murmurhash on the id and runs modulo against the number of shards and replicas, this is the way to find out, if a watch should run local. Reloading happens 2. Several master node actions moved to a HandledTransportAction, as they are basically just aliases for indexing actions, among them the put/delete/get watch actions, the acknowledgement action, the de/activate actions 3. Stats action moved to a broadcast node action, because we potentially have to query every node to get watcher statistics 4. Starting/Stopping watcher now is a master node action, which updates the cluster state and then listeners acts on those. Because of this watches can be running on two systems, if you those have different cluster state versions, until the new watcher state is propagated 5. Watcher is started on all nodes now. With the exception of the ticker schedule engine most classes do not need a lot of resources while running. However they have to run, because of the execute watch API, which can hit any node - it does not make sense to find the right shard for this watch and only then execute (as this also has to work with a watch, that has not been stored before) 6. By using a indexing operation listener, each storing of a watch now parses the watch first and only stores on successful parsing 7. Execute watch API now uses the watcher threadpool for execution 8. Getting the number of watches for the stats now simply queries the different execution engines, how many watches are scheduled, so this is not doing a search anymore There will be follow up commits on this one, mainly to ensure BWC compatibility. Original commit: elastic/x-pack-elasticsearch@0adb46e658 |
||
|---|---|---|
| .github | ||
| buildSrc | ||
| dev-tools | ||
| docs | ||
| license-tools | ||
| migrate | ||
| plugin | ||
| qa | ||
| transport-client | ||
| .dir-locals.el | ||
| .gitignore | ||
| .projectile | ||
| GRADLE.CHEATSHEET.asciidoc | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| README.asciidoc | ||
| build.gradle | ||
| gradle.properties | ||
| migrate-issues.py | ||
| settings.gradle | ||
README.asciidoc
= Elasticsearch X-Pack A set of Elastic's commercial plugins for Elasticsearch: - License - Security - Watcher - Monitoring - Machine Learning - Graph = Setup You must checkout `x-pack-elasticsearch` and `elasticsearch` with a specific directory structure. The `elasticsearch` checkout will be used when building `x-pack-elasticsearch`. The structure is: - /path/to/elastic/elasticsearch - /path/to/elastic/elasticsearch-extra/x-pack-elasticsearch == Vault Secret The build requires a Vault Secret ID. You can use a GitHub token by following these steps: 1. Go to https://github.com/settings/tokens 2. Click *Generate new token* 3. Set permissions to `read:org` 4. Copy the token into `~/.elastic/github.token` 5. Set the token's file permissions to `600` ``` $ mkdir ~/.elastic $ vi ~/.elastic/github.token # Add your_token exactly as it is into the file and save it $ chmod 600 ~/.elastic/github.token ``` If you do not create the token, then you will see something along the lines of this as the failure when trying to build X-Pack: ``` * What went wrong: Missing ~/.elastic/github.token file or VAULT_SECRET_ID environment variable, needed to authenticate with vault for secrets ``` === Offline Mode When running the build in offline mode (`--offline`), it will not required to have the vault secret setup. == Native Code **This is mandatory as tests depend on it** Machine Learning requires platform specific binaries, build from https://github.com/elastic/machine-learning-cpp via CI servers. The native artifacts are stored in S3. To retrieve them infra's team Vault service is utilized, which requires a github token. Please setup a github token as documented: https://github.com/elastic/infra/blob/master/docs/vault.md#github-auth The github token has to be put into ~/.elastic/github.token, while the file rights must be set to 0600. = Build - Run unit tests: + [source, txt] ----- gradle clean test ----- - Run all tests: + [source, txt] ----- gradle clean check ----- - Run integration tests: + [source, txt] ----- gradle clean integTest ----- - Package X-Pack (without running tests) + [source, txt] ----- gradle clean assemble ----- - Install X-Pack (without running tests) + [source, txt] ----- gradle clean install -----