Commit Graph

225 Commits

Author SHA1 Message Date
Martijn van Groningen 0b5b26284b Validate if job index already exists after job validation.
Closes elastic/elasticsearch#594

Original commit: elastic/x-pack-elasticsearch@b3b7b086a7
2016-12-21 08:34:16 +01:00
Martijn van Groningen 999a1eb234 fix JobTests#testEquals_noId by fixing regex expression and including `-`
Original commit: elastic/x-pack-elasticsearch@b295d5f621
2016-12-20 15:15:47 +01:00
David Roberts 8b135226f6 Fix long line
Original commit: elastic/x-pack-elasticsearch@8d030d8cf8
2016-12-20 14:09:39 +00:00
David Kyle 537fea313f Add job config option to set the index name (elastic/elasticsearch#560)
* Add job config option to set the index name

* Check index does not already exist if ’index_name’ is set

* Don’t create alias if ‘index_name’ is the same as ‘job_id’

* Default index_name value

Set it to job_id if null and only create the index alias if job_id != index_name

* Fix compile errors after rebasing

* Address review comments

* Test if the index exists by checking the cluster state

* Update comment

Original commit: elastic/x-pack-elasticsearch@a3e7f1a5bb
2016-12-20 14:03:24 +00:00
David Roberts 3383c67cf2 Ensure automatically generated job IDs meet the validation criteria (elastic/elasticsearch#586)
Original commit: elastic/x-pack-elasticsearch@34b67cb509
2016-12-20 13:53:29 +00:00
David Roberts 5b029ee595 Correct test condition and add a message in case of future failures
Original commit: elastic/x-pack-elasticsearch@55729b39fa
2016-12-20 11:22:03 +00:00
David Roberts 1615be4ff3 Moved the C++ code under the cpp sub-directory (elastic/elasticsearch#584)
The $PRELERT_SRC_HOME environment variable is replaced with $CPP_SRC_HOME,
which points to the cpp sub-directory off the repository root

Original commit: elastic/x-pack-elasticsearch@02ef6d6be6
2016-12-20 11:03:00 +00:00
David Kyle 78fdca65d5 Add job Id to state document IDs (elastic/elasticsearch#579)
* Persist quantile documents with the jobId in the document Id

* Add job Id to snapshot Id

* Add job Id to categoriser state document Id

* Rename quantiles doc to start with job id as the other state docs do

* Fix restoring categoriser state

Original commit: elastic/x-pack-elasticsearch@3e5d3368b5
2016-12-20 10:47:37 +00:00
Dimitrios Athanasiou 316f6bf6cb Remove unused methods and fix test
Original commit: elastic/x-pack-elasticsearch@12290517d0
2016-12-20 10:15:53 +00:00
Dimitris Athanasiou bee58f8b81 Add get API for schedulers (elastic/elasticsearch#582)
Original commit: elastic/x-pack-elasticsearch@59c555d60a
2016-12-20 10:09:45 +00:00
David Roberts 9b3764b7fa Max jobs per node cannot be dynamic due to threadpool (elastic/elasticsearch#578)
The threadpool that supplies the threads used for job IO cannot be
resized, so the number of jobs cannot be dynamic either

Original commit: elastic/x-pack-elasticsearch@c584bf7147
2016-12-19 17:48:22 +00:00
David Roberts f65125df55 No point rethrowing exception in thread pool task
It will just get logged twice, but with a less useful message the
second time

Original commit: elastic/x-pack-elasticsearch@7d32f65cc1
2016-12-19 16:24:45 +00:00
Dimitrios Athanasiou 081bebbbcd Remove unused constants
Original commit: elastic/x-pack-elasticsearch@0f72f1fc6e
2016-12-19 14:22:05 +00:00
Dimitris Athanasiou 8750e5f926 Redesign the get anomaly_detectors APIs (elastic/elasticsearch#570)
* Redesign the get anomaly_detectors APIs

This commit redesigns the APIs to get anomaly_detectors.

The new design has 2 GET APIs:

- An API to get the configurations: /anomaly_detectors/{job_id}
- An API to get the stats: /anomaly_detectors/{job_id}/_stats

For both APIs entering "_all" as the job_id returns results for
all jobs.

Note that page params have been removed. They were useful
when the configs were stored in an index. Now that they are part
of cluster state there is no need. Additionally, future support
for wildcard job_id expressions will give users a tool to narrow
down the GET actions to a certain subset of jobs which will be
more useful than the from/size approach.

Follow up:

- Implement similar GET APIs for schedulers
- Remove scheduler_stats from the anomaly_detectors _stats API
as it will be part of the schedulers _stats API

Closes elastic/elasticsearch#548

Original commit: elastic/x-pack-elasticsearch@046a0db8f5
2016-12-19 13:09:54 +00:00
David Kyle 4eda09d24a Don’t write job description if not set
Original commit: elastic/x-pack-elasticsearch@6e7ff939d3
2016-12-19 11:27:32 +00:00
David Roberts 854dc88cbd Corrections to previous commit
Original commit: elastic/x-pack-elasticsearch@a9e94119e1
2016-12-19 10:28:23 +00:00
David Roberts 028a505fe5 Update copyright headers (elastic/elasticsearch#572)
* Added the Elastic copyright header to C++ files

* Added the name of the copyright holder to Java files

Original commit: elastic/x-pack-elasticsearch@aea1b5a656
2016-12-19 10:19:21 +00:00
Dimitris Athanasiou 6a14f52d2e Remove aggs and retrieve_whole_source from SchedulerConfig (elastic/elasticsearch#569)
Aggs does not need to be a separate member field. There can simply
be an aggs parse field which also then stored onto the aggregations
parse field.

Finally, retrieve_whole_source is unnecessary as we move towards
a node client based data extraction.

Original commit: elastic/x-pack-elasticsearch@14024c2ee5
2016-12-19 09:54:59 +00:00
David Roberts 89e506f050 Merge branch 'analytics-dev' into master
Original commit: elastic/x-pack-elasticsearch@dc9d10a502
2016-12-19 09:07:12 +00:00
David Kyle aacb6f3949 Move JobResultsPersister.getJobIndexName to new class AnomalyDetectorsIndex (elastic/elasticsearch#567)
Original commit: elastic/x-pack-elasticsearch@bf54dd56a9
2016-12-16 17:47:52 +00:00
Martijn van Groningen e3ec908828 run with native process by default
Original commit: elastic/x-pack-elasticsearch@c73beec522
2016-12-16 15:21:11 +01:00
Martijn van Groningen eaca8fb06a Stop local allocated jobs before closing node.
Original commit: elastic/x-pack-elasticsearch@ad3ed7c86a
2016-12-16 14:59:35 +01:00
David Roberts c43a9ba1dd Fix intermittent failure of renormalization (elastic/elasticsearch#558)
The synchronization was flawed

Original commit: elastic/x-pack-elasticsearch@a968c68c7d
2016-12-16 13:45:47 +00:00
Colin Goodheart-Smithe 5e6c63bc51 Removes warnings
Original commit: elastic/x-pack-elasticsearch@59626a3a3a
2016-12-16 12:35:45 +00:00
Dimitris Athanasiou c932588c68 Repackage scheduler classes (elastic/elasticsearch#559)
* Repackage scheduler classes

* Rename and move scheduler status tests

Original commit: elastic/x-pack-elasticsearch@a9d7cc4e48
2016-12-16 09:50:06 +00:00
David Kyle 8d5c8b24d6 Make ModelDebugOutput fields snake case (elastic/elasticsearch#555)
* snake_case model debug output fields

* Rename CategorizerState and BucketInfluencer types to snake_case

Original commit: elastic/x-pack-elasticsearch@da94dc7ec1
2016-12-15 15:42:38 +00:00
David Kyle 8e14aee978 Audit message were being persisted with the type ‘audit_activity’ (elastic/elasticsearch#554)
Original commit: elastic/x-pack-elasticsearch@7c00495c98
2016-12-15 15:31:11 +00:00
Dimitris Athanasiou 8841a97659 Disallow non alphanumeric chars from start or end in IDs (elastic/elasticsearch#552)
Closes elastic/elasticsearch#550

Original commit: elastic/x-pack-elasticsearch@ca0ca9f4da
2016-12-15 13:49:07 +00:00
Dimitrios Athanasiou bc8d966690 Fix integration tests
Original commit: elastic/x-pack-elasticsearch@a5955cfde3
2016-12-15 12:03:51 +00:00
David Kyle b97610f245 Streamed boolean values can on be 0 or 1
Original commit: elastic/x-pack-elasticsearch@16ffa0727b
2016-12-15 12:01:09 +00:00
Martijn van Groningen faf54d4952 renamed test classes
Original commit: elastic/x-pack-elasticsearch@5296de69d7
2016-12-14 19:52:10 +01:00
Martijn van Groningen 985bdf6cb1 Changed rest start scheduler api to wait with returning a response until the the scheduler status has been set to STARTED and
changed stop scheduler api to wait with returning a response until the scheduler status has been set to STOPPED.

Original commit: elastic/x-pack-elasticsearch@e20fcd1ae9
2016-12-14 19:50:02 +01:00
Martijn van Groningen 3b487a268b Add validation for stop scheduler api
Original commit: elastic/x-pack-elasticsearch@4571bbacd9
2016-12-14 19:28:12 +01:00
David Kyle f085b935f6 Custom all mapping for interesting result fields (elastic/elasticsearch#535)
* Check use of mappings

* Add unit tests for JobProvider.createJobRelatedIndices

* Remove ‘index: no’ from mappings as no longer required

The entire type mapping has ‘enabled: false’

* Restore “index.analysis.analyzer.default.type” setting

* Remove include_in_all from nested mappings

* Add audit and usage mappings to the job index

* Revert ‘Restore “index.analysis.analyzer.default.type” setting’

Original commit: elastic/x-pack-elasticsearch@c7d62e0c7e
2016-12-14 16:32:36 +00:00
David Roberts 0a45d846df Add back native process normalizer functionality (elastic/elasticsearch#540)
This builds on PR elastic/elasticsearch#526 to get normalization working end-to-end using the
native normalizer process.

The ShortCircuitingRenormalizer class is basically doing what the old
BlockingQueueRenormaliser class did but using the ES threadpool instead
of its own thread.

Also fixed a bug where the C++ was calling the score field of partition_score
documents normalized_probability but the Java was calling it anomaly_score.

Original commit: elastic/x-pack-elasticsearch@d4cecae150
2016-12-14 16:06:01 +00:00
Dimitris Athanasiou 4732223214 Split SchedulerConfig from Job (elastic/elasticsearch#541)
Original commit: elastic/x-pack-elasticsearch@32a55da726
2016-12-14 14:10:44 +00:00
David Kyle 7efdbd9320 Fix the reported number of bytes read (elastic/elasticsearch#544)
Original commit: elastic/x-pack-elasticsearch@a98902f309
2016-12-14 13:39:00 +00:00
Zachary Tong b386ed33a1 Add ability to get all lists, ensure params are mutually exclusive (elastic/elasticsearch#514)
Also removes some unnecessary stuff in validate()

Related elastic/elasticsearch#291

Original commit: elastic/x-pack-elasticsearch@d9af2d3a30
2016-12-13 11:38:11 -05:00
Zachary Tong 6ef954e3f0 Add mutually-exclusive param validation to GetCategories (elastic/elasticsearch#512)
Also adds body parsing to the REST action, which seemed to be missing (but supported because of the POST)

Closes elastic/elasticsearch#430

Original commit: elastic/x-pack-elasticsearch@e7455ffbb6
2016-12-13 11:37:46 -05:00
Martijn van Groningen 3581fc9d91 removed redundant private method
Original commit: elastic/x-pack-elasticsearch@dbc377d6b4
2016-12-13 13:35:14 +01:00
Martijn van Groningen d15933ac32 test: move java rest test to yaml rest test.
These test are just checking response / status code. It is cleaner to have these tests as yaml tests.

Original commit: elastic/x-pack-elasticsearch@61c323059e
2016-12-13 12:13:24 +01:00
David Kyle 9f73f047eb Test for job existence before updating its state (elastic/elasticsearch#532)
* Test for job existence before updating its state

* Add unit tests covering expected missing job exceptions

Original commit: elastic/x-pack-elasticsearch@bcd270dafd
2016-12-13 09:24:16 +00:00
David Kyle bca06f0ad7 Fix forbidden API errors and logger usage check (elastic/elasticsearch#533)
Original commit: elastic/x-pack-elasticsearch@e7bdaa8b0a
2016-12-12 17:31:39 +00:00
David Kyle 1d67c6ea1d Fix build: rename import Renormali[s]er -> Renormali[z]er
Original commit: elastic/x-pack-elasticsearch@70fa67aaf7
2016-12-12 17:08:39 +00:00
David Kyle 2302dc78ba AutodetectResultProcessor Integration Test (elastic/elasticsearch#516)
* Add results processor integration test

* Integration tests for AutodetectResultProcessor

Original commit: elastic/x-pack-elasticsearch@19e7ec48dd
2016-12-12 16:55:20 +00:00
David Kyle dae0b5625d Add yaml tests assert a 404 is returned when the job id isn’t recognised (elastic/elasticsearch#531)
For open, close, _flush and posting data

Original commit: elastic/x-pack-elasticsearch@be2c29c409
2016-12-12 16:00:20 +00:00
David Kyle 300d7c67d2 Fix failing integration tests
Incorrect job id was used

Original commit: elastic/x-pack-elasticsearch@625a1616ee
2016-12-12 13:03:09 +00:00
David Kyle 3829a83864 Check job exists and return a 404 on flush (elastic/elasticsearch#529)
Original commit: elastic/x-pack-elasticsearch@7721d4510a
2016-12-12 12:50:40 +00:00
David Kyle 46938b281c Check job exists before opening & closing (elastic/elasticsearch#530)
Original commit: elastic/x-pack-elasticsearch@037ea30a12
2016-12-12 12:50:29 +00:00
David Roberts 1d4df3903c Add back parts of the normalizer functionality (elastic/elasticsearch#526)
* Added back Normalizable classes
* Added back normalization process management classes
* Added back the scores updater

Original commit: elastic/x-pack-elasticsearch@ac8edf6ed6
2016-12-12 12:40:13 +00:00