Commit Graph

148 Commits

Author SHA1 Message Date
Martijn van Groningen faf54d4952 renamed test classes
Original commit: elastic/x-pack-elasticsearch@5296de69d7
2016-12-14 19:52:10 +01:00
Martijn van Groningen 985bdf6cb1 Changed rest start scheduler api to wait with returning a response until the the scheduler status has been set to STARTED and
changed stop scheduler api to wait with returning a response until the scheduler status has been set to STOPPED.

Original commit: elastic/x-pack-elasticsearch@e20fcd1ae9
2016-12-14 19:50:02 +01:00
Martijn van Groningen 3b487a268b Add validation for stop scheduler api
Original commit: elastic/x-pack-elasticsearch@4571bbacd9
2016-12-14 19:28:12 +01:00
David Kyle f085b935f6 Custom all mapping for interesting result fields (elastic/elasticsearch#535)
* Check use of mappings

* Add unit tests for JobProvider.createJobRelatedIndices

* Remove ‘index: no’ from mappings as no longer required

The entire type mapping has ‘enabled: false’

* Restore “index.analysis.analyzer.default.type” setting

* Remove include_in_all from nested mappings

* Add audit and usage mappings to the job index

* Revert ‘Restore “index.analysis.analyzer.default.type” setting’

Original commit: elastic/x-pack-elasticsearch@c7d62e0c7e
2016-12-14 16:32:36 +00:00
David Roberts 0a45d846df Add back native process normalizer functionality (elastic/elasticsearch#540)
This builds on PR elastic/elasticsearch#526 to get normalization working end-to-end using the
native normalizer process.

The ShortCircuitingRenormalizer class is basically doing what the old
BlockingQueueRenormaliser class did but using the ES threadpool instead
of its own thread.

Also fixed a bug where the C++ was calling the score field of partition_score
documents normalized_probability but the Java was calling it anomaly_score.

Original commit: elastic/x-pack-elasticsearch@d4cecae150
2016-12-14 16:06:01 +00:00
Dimitris Athanasiou 4732223214 Split SchedulerConfig from Job (elastic/elasticsearch#541)
Original commit: elastic/x-pack-elasticsearch@32a55da726
2016-12-14 14:10:44 +00:00
David Kyle 7efdbd9320 Fix the reported number of bytes read (elastic/elasticsearch#544)
Original commit: elastic/x-pack-elasticsearch@a98902f309
2016-12-14 13:39:00 +00:00
Zachary Tong b386ed33a1 Add ability to get all lists, ensure params are mutually exclusive (elastic/elasticsearch#514)
Also removes some unnecessary stuff in validate()

Related elastic/elasticsearch#291

Original commit: elastic/x-pack-elasticsearch@d9af2d3a30
2016-12-13 11:38:11 -05:00
Zachary Tong 6ef954e3f0 Add mutually-exclusive param validation to GetCategories (elastic/elasticsearch#512)
Also adds body parsing to the REST action, which seemed to be missing (but supported because of the POST)

Closes elastic/elasticsearch#430

Original commit: elastic/x-pack-elasticsearch@e7455ffbb6
2016-12-13 11:37:46 -05:00
Martijn van Groningen 3581fc9d91 removed redundant private method
Original commit: elastic/x-pack-elasticsearch@dbc377d6b4
2016-12-13 13:35:14 +01:00
Martijn van Groningen d15933ac32 test: move java rest test to yaml rest test.
These test are just checking response / status code. It is cleaner to have these tests as yaml tests.

Original commit: elastic/x-pack-elasticsearch@61c323059e
2016-12-13 12:13:24 +01:00
David Kyle 9f73f047eb Test for job existence before updating its state (elastic/elasticsearch#532)
* Test for job existence before updating its state

* Add unit tests covering expected missing job exceptions

Original commit: elastic/x-pack-elasticsearch@bcd270dafd
2016-12-13 09:24:16 +00:00
David Kyle bca06f0ad7 Fix forbidden API errors and logger usage check (elastic/elasticsearch#533)
Original commit: elastic/x-pack-elasticsearch@e7bdaa8b0a
2016-12-12 17:31:39 +00:00
David Kyle 1d67c6ea1d Fix build: rename import Renormali[s]er -> Renormali[z]er
Original commit: elastic/x-pack-elasticsearch@70fa67aaf7
2016-12-12 17:08:39 +00:00
David Kyle 2302dc78ba AutodetectResultProcessor Integration Test (elastic/elasticsearch#516)
* Add results processor integration test

* Integration tests for AutodetectResultProcessor

Original commit: elastic/x-pack-elasticsearch@19e7ec48dd
2016-12-12 16:55:20 +00:00
David Kyle dae0b5625d Add yaml tests assert a 404 is returned when the job id isn’t recognised (elastic/elasticsearch#531)
For open, close, _flush and posting data

Original commit: elastic/x-pack-elasticsearch@be2c29c409
2016-12-12 16:00:20 +00:00
David Kyle 300d7c67d2 Fix failing integration tests
Incorrect job id was used

Original commit: elastic/x-pack-elasticsearch@625a1616ee
2016-12-12 13:03:09 +00:00
David Kyle 3829a83864 Check job exists and return a 404 on flush (elastic/elasticsearch#529)
Original commit: elastic/x-pack-elasticsearch@7721d4510a
2016-12-12 12:50:40 +00:00
David Kyle 46938b281c Check job exists before opening & closing (elastic/elasticsearch#530)
Original commit: elastic/x-pack-elasticsearch@037ea30a12
2016-12-12 12:50:29 +00:00
David Roberts 1d4df3903c Add back parts of the normalizer functionality (elastic/elasticsearch#526)
* Added back Normalizable classes
* Added back normalization process management classes
* Added back the scores updater

Original commit: elastic/x-pack-elasticsearch@ac8edf6ed6
2016-12-12 12:40:13 +00:00
Martijn van Groningen bad8a2beb5 let prelert metadata test extend from AbstractSerializingTestCase, so that we test the writable and xcontent serialization much better.
Original commit: elastic/x-pack-elasticsearch@de8609ad59
2016-12-12 12:09:07 +01:00
Martijn van Groningen f73d22b4b2 removed unused method
Original commit: elastic/x-pack-elasticsearch@bbe4b7e932
2016-12-12 11:19:44 +01:00
Martijn van Groningen 3d8353ac6a replaced `RestActions` helper methods for checking whether there is a request body and getting the request body with getters on RestRequest itself
Original commit: elastic/x-pack-elasticsearch@857d999c8e
2016-12-12 10:51:54 +01:00
David Kyle 45c8aeb0f2 Remove the ES_TIMESTAMP constant (elastic/elasticsearch#523)
Original commit: elastic/x-pack-elasticsearch@04fa354619
2016-12-12 09:28:42 +00:00
Zachary Tong 30bbbf0f78 Convert `jobs` namespace to `anomaly_detectors` (elastic/elasticsearch#513)
This also shuffles results under /anomaly_detectors/.  Note: the cluster state still refers to
"jobs" which should probably be fixed in a separate PR

Original commit: elastic/x-pack-elasticsearch@c9e634621c
2016-12-09 11:54:56 -05:00
Martijn van Groningen a85dfcd91f changed end back to primitive long
renamed private method

Original commit: elastic/x-pack-elasticsearch@fc1849dbc6
2016-12-09 15:56:22 +01:00
Martijn van Groningen de9dbdabe1 fixed typos
Original commit: elastic/x-pack-elasticsearch@21a62161d9
2016-12-09 15:50:58 +01:00
Martijn van Groningen 431a7988fb added more tests
Original commit: elastic/x-pack-elasticsearch@871c13307e
2016-12-09 15:45:25 +01:00
Martijn van Groningen b42f9cccb0 renamed `scheduler_status` to `scheduler_state` in get job response
Original commit: elastic/x-pack-elasticsearch@a7955eb027
2016-12-09 15:36:12 +01:00
Martijn van Groningen d5412627d2 Moved scheduler status to prelertmetadata to make it more independent of job.
Removed SchedulerStats as scheduler status is all we need and start and end times are only needed in start scheduler api.

Original commit: elastic/x-pack-elasticsearch@80c563cb69
2016-12-09 15:03:53 +01:00
Martijn van Groningen e067008a21 make optional flush parameter really optional
Original commit: elastic/x-pack-elasticsearch@6129023c49
2016-12-09 10:24:16 +01:00
Martijn van Groningen 372cb7c964 Let the scheduler use the data transport action instead of directly using autodetect process manager.
Original commit: elastic/x-pack-elasticsearch@2442e222fd
2016-12-09 09:41:04 +01:00
David Roberts 65f03a8888 Use well-defined IDs for records and influencers (elastic/elasticsearch#510)
* Use well-defined IDs for records and influencers

Removes the reliance on ES autogenerated UUIDs for all types that will
be renormalized

* Address some review comments

Original commit: elastic/x-pack-elasticsearch@85fde8b957
2016-12-08 18:39:03 +00:00
David Roberts 256ab7f3e2 Fix another misuse of ParameterizedMessage
Original commit: elastic/x-pack-elasticsearch@9e36900432
2016-12-08 17:32:59 +00:00
David Roberts 3f460b030e Fix a couple of ParameterizedMessage mistakes
Original commit: elastic/x-pack-elasticsearch@0006755c8b
2016-12-08 16:56:49 +00:00
Zachary Tong 8fa17f7ea8 Add mutually-exclusive param validation to GetBuckets (elastic/elasticsearch#497)
Related elastic/elasticsearch#430

Original commit: elastic/x-pack-elasticsearch@2579efca46
2016-12-08 10:28:30 -05:00
Zachary Tong b594e93a5d Remove SingleDocument as it is now unused (elastic/elasticsearch#498)
Closes elastic/elasticsearch#314

Original commit: elastic/x-pack-elasticsearch@88d2d20bdc
2016-12-08 10:28:12 -05:00
David Kyle a55944b284 Make ModelDebugOutput a result type (elastic/elasticsearch#484)
* Make ModelDebugOutput a result type

* Delete unused ElasticsearchBatchedModelDebugOutputIterator

* Add result_type field to ModelDebugOutput

* Address review comments

Original commit: elastic/x-pack-elasticsearch@a48e4cd946
2016-12-08 15:05:06 +00:00
David Kyle d18daf0b3d Make ModelSizeStats a result type (elastic/elasticsearch#500)
* Make ModelSizeStats a result type

* Address review comments

Original commit: elastic/x-pack-elasticsearch@74de7b36d8
2016-12-08 14:44:59 +00:00
Martijn van Groningen 9c2c831996 Hooked scheduler into task api infrastructure.
The start scheduler api call will run until the scheduler has completed. Either when lookback only scheduler completes or the scheduler has been stopped.
 The start scheduler api will first update the scheduler status from STOPPED to STARTED on master node and then start running the scheduler.
 Once a scheduled job completes it updates the scheduler status from STARTED to STOPPED and then the start schedule api returns.
 The STARTING and STOPPING statuses are no longer used, so have been removed.
 The stop scheduler api is a sugar api that uses the task list and cancel apis stop the scheduler.
 Renamed ScheduledJobService to ScheduledJobRunner

Original commit: elastic/x-pack-elasticsearch@ab504fe3d9
2016-12-08 14:37:55 +01:00
Martijn van Groningen c8bda6b1d9 Collapsed JobProvider and JobResultsProvider and renamed ElasticsearchJobProvider to JobProvider
Original commit: elastic/x-pack-elasticsearch@05d5969d73
2016-12-08 10:53:31 +01:00
Dimitris Athanasiou 4990195c90 Convert field references in error messages to snake case (elastic/elasticsearch#495)
* Convert field references in error messages to snake case

Closes elastic/elasticsearch#493

Original commit: elastic/x-pack-elasticsearch@38f4cd91bc
2016-12-08 09:32:10 +00:00
Zachary Tong 4d86670772 Make jobId and from/size mutually exclusive options (elastic/elasticsearch#477)
Make jobId and from/size mutually exclusive options.  

This approach has the main properties of not allowing an invalid Request to be built, and alerting the user if they set an incorrect configuration. It has the downside that PageParams can be null so the consumer will have to check for it. Since jobId could be null before, this seemed acceptable.

Original commit: elastic/x-pack-elasticsearch@106dcdf61a
2016-12-07 12:48:33 -05:00
Zachary Tong 3c711e6dff Add delete_list endpoint (elastic/elasticsearch#409)
Adds a delete_list endpoint.  If a list is currently in use by a job, it is not allowed to be deleted

Original commit: elastic/x-pack-elasticsearch@7d9a984b3a
2016-12-07 12:40:12 -05:00
David Kyle af61a51e22 Fix results not being persisted (elastic/elasticsearch#489)
Original commit: elastic/x-pack-elasticsearch@d0ee02ccf6
2016-12-07 16:52:58 +00:00
Martijn van Groningen 7cc2b8c5ce create allocation when the job has been created instead of creating it when opening the job.
This avoids the confusing situation that a there is no allocation when a job hasn't been opened yet. Now it complains about the fact that the job status is closed.

Original commit: elastic/x-pack-elasticsearch@3159dc6954
2016-12-07 17:15:18 +01:00
David Kyle ccf8cb7e0d Refactor delete interim results (elastic/elasticsearch#470)
* Collapse ElasticsearchBulkDeleter into JobDataDeleter

* Add blocking delete to JobDataDeleter

* Delete interim results only after all the results are parsed.

* Remove unused deleteModelSizeStats and deleteModelDebugOutput methods.

Document missing javadoc tags

Original commit: elastic/x-pack-elasticsearch@1997541673
2016-12-07 11:23:27 +00:00
Martijn van Groningen 14f43af818 moved metric part to be the suffix of the path
Original commit: elastic/x-pack-elasticsearch@6fc1b861ed
2016-12-07 12:09:16 +01:00
Martijn van Groningen e396e8aa68 Changes rest actions with data namespace into the job namespace.
Also added `_status` suffic to get job api as it would otherwise clash with open/close/flush APIs.

Original commit: elastic/x-pack-elasticsearch@6e8ef0ef7d
2016-12-07 12:09:16 +01:00
Martijn van Groningen d807eda9ed replaced custom ClosableIterator with Stream
Stream is closable, which is the reason ClosableIterator was introduced.

Original commit: elastic/x-pack-elasticsearch@b5a4a37e9e
2016-12-07 12:07:59 +01:00