Note that the change in elasticsearch allowed us to store scheduler
config's query and scriptFields as typed objects instead of
ByteReference.
Original commit: elastic/x-pack-elasticsearch@38c5aef2ef
* Check the bulk request contains actions before executing.
This suppresses an validation exception about no requests being added.
* Persist bulk request before refreshing the indexes on a flush acknowledgment
Original commit: elastic/x-pack-elasticsearch@22543e46c8
* Put model state in the .mlstate index
* Revert results index rename
* Put ModelSnapshots in the results index
* Change state index in C++
* Fix logging
* Rename state index ‘.ml-state’
Original commit: elastic/x-pack-elasticsearch@dbe5f6b525
* Strict parse search parts of schedulerConfig
This commit adds methods to build the typed objects
for the search parts of a scheduler config. Those are:
query, aggregations and script_fields.
As scheduler configs are stored in the cluster state and parsing
the search parts requires a SearchRequestParsers object, we cannot
store them as typed fields. Instead, they are now stored as
BytesReferences.
This change is in preparation for switching over to using a
client based data extractor.
Point summary of changes:
- query, aggregations and scriptFields are now stored as BytesReference
- adds methods to build the corresponding typed objects
- putting a scheduler now builds the search parts to to validate that
the config is valid
Relates to elastic/elasticsearch#478
Original commit: elastic/x-pack-elasticsearch@e6d5a85871
This doesn't happen initially when buckets are output by the C++, but
buckets can contain records at the moment they're sent for persistence
during normalization or during integration tests. It's safest if the
persistence code specifically doesn't persist these records.
Original commit: elastic/x-pack-elasticsearch@a93135d8c0
* Add job config option to set the index name
* Check index does not already exist if ’index_name’ is set
* Don’t create alias if ‘index_name’ is the same as ‘job_id’
* Default index_name value
Set it to job_id if null and only create the index alias if job_id != index_name
* Fix compile errors after rebasing
* Address review comments
* Test if the index exists by checking the cluster state
* Update comment
Original commit: elastic/x-pack-elasticsearch@a3e7f1a5bb
The $PRELERT_SRC_HOME environment variable is replaced with $CPP_SRC_HOME,
which points to the cpp sub-directory off the repository root
Original commit: elastic/x-pack-elasticsearch@02ef6d6be6
* Persist quantile documents with the jobId in the document Id
* Add job Id to snapshot Id
* Add job Id to categoriser state document Id
* Rename quantiles doc to start with job id as the other state docs do
* Fix restoring categoriser state
Original commit: elastic/x-pack-elasticsearch@3e5d3368b5
The threadpool that supplies the threads used for job IO cannot be
resized, so the number of jobs cannot be dynamic either
Original commit: elastic/x-pack-elasticsearch@c584bf7147
* Redesign the get anomaly_detectors APIs
This commit redesigns the APIs to get anomaly_detectors.
The new design has 2 GET APIs:
- An API to get the configurations: /anomaly_detectors/{job_id}
- An API to get the stats: /anomaly_detectors/{job_id}/_stats
For both APIs entering "_all" as the job_id returns results for
all jobs.
Note that page params have been removed. They were useful
when the configs were stored in an index. Now that they are part
of cluster state there is no need. Additionally, future support
for wildcard job_id expressions will give users a tool to narrow
down the GET actions to a certain subset of jobs which will be
more useful than the from/size approach.
Follow up:
- Implement similar GET APIs for schedulers
- Remove scheduler_stats from the anomaly_detectors _stats API
as it will be part of the schedulers _stats API
Closeselastic/elasticsearch#548
Original commit: elastic/x-pack-elasticsearch@046a0db8f5
* Added the Elastic copyright header to C++ files
* Added the name of the copyright holder to Java files
Original commit: elastic/x-pack-elasticsearch@aea1b5a656
Aggs does not need to be a separate member field. There can simply
be an aggs parse field which also then stored onto the aggregations
parse field.
Finally, retrieve_whole_source is unnecessary as we move towards
a node client based data extraction.
Original commit: elastic/x-pack-elasticsearch@14024c2ee5
* snake_case model debug output fields
* Rename CategorizerState and BucketInfluencer types to snake_case
Original commit: elastic/x-pack-elasticsearch@da94dc7ec1
changed stop scheduler api to wait with returning a response until the scheduler status has been set to STOPPED.
Original commit: elastic/x-pack-elasticsearch@e20fcd1ae9
* Check use of mappings
* Add unit tests for JobProvider.createJobRelatedIndices
* Remove ‘index: no’ from mappings as no longer required
The entire type mapping has ‘enabled: false’
* Restore “index.analysis.analyzer.default.type” setting
* Remove include_in_all from nested mappings
* Add audit and usage mappings to the job index
* Revert ‘Restore “index.analysis.analyzer.default.type” setting’
Original commit: elastic/x-pack-elasticsearch@c7d62e0c7e
This builds on PR elastic/elasticsearch#526 to get normalization working end-to-end using the
native normalizer process.
The ShortCircuitingRenormalizer class is basically doing what the old
BlockingQueueRenormaliser class did but using the ES threadpool instead
of its own thread.
Also fixed a bug where the C++ was calling the score field of partition_score
documents normalized_probability but the Java was calling it anomaly_score.
Original commit: elastic/x-pack-elasticsearch@d4cecae150
Martijn van Groningen
David Kyle
Dimitrios Athanasiou
David Roberts
Colin Goodheart-Smithe
Zachary Tong