mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-11 23:45:15 +00:00
0cb2b18265
Aggregated data extraction is done in 2 phases: 1. search 2. process response The first phase cannot be currently cancelled. However, it usually is the fastest of the two. The second phase processes the histogram buckets in the search response into flat JSON and then posts the result stream to the job. This phase can be split into batches where a few buckets are posted to the job at a time. Cancelling can then work between batches. This commit changes the AggregationDataExtractor to process the search response in batches. The definition of a batch is crucial as it has to be short enough to allow for responsive cancelling, yet long enough to minimise overhead due to multiple calls to the post data action. The number of key-value pairs written by the processor is a good candidate for a batch size measure. By testing, 1000 seems to be an effective number. relates elastic/x-pack-elasticsearch#802 Original commit: elastic/x-pack-elasticsearch@ce3a172411
= Elasticsearch X Pack A set of Elastic's commercial plugins: - License - Security - Watcher - Monitoring - Machine Learning = Setup You must checkout x-pack and elasticsearch with a specific directory structure. The elasticsearch checkout will be used when building x-pack. The structure is: - /path/to/elastic/elasticsearch - /path/to/elastic/elasticsearch-extra/x-pack-elasticsearch == Native Code **This is mandatory as tests depend on it** Machine Learning requires platform specific binaries, build from https://github.com/elastic/machine-learning-cpp via CI servers. The native artifacts are stored in S3. To retrieve them infra's team Vault service is utilized, which requires a github token. Please setup a github token as documented: https://github.com/elastic/infra/blob/master/docs/vault.md#github-auth The github token has to be put into ~/.elastic/github.token, while the file rights must be set to 0600. = Build - Run unit tests: + [source, txt] ----- gradle clean test ----- - Run all tests: + [source, txt] ----- gradle clean check ----- - Run integration tests: + [source, txt] ----- gradle clean integTest ----- - Package X-Pack (wihtout running tests) + [source, txt] ----- gradle clean assemble ----- - Install X-Pack (wihtout running tests) + [source, txt] ----- gradle clean install -----