Commit Graph

348 Commits

Author SHA1 Message Date
Dimitris Athanasiou 4128d635f4 Remove explicit _all disabling from mappings (elastic/elasticsearch#719)
The _all field is now deprecated and disabled by default in elasticsearch
6.0.0. We no longer need to disable it explicitly.

Original commit: elastic/x-pack-elasticsearch@c71465083a
2017-01-13 12:39:59 +00:00
Dimitris Athanasiou 3ce48bc7b9 Remove ParseFieldMatcher (elastic/elasticsearch#718)
Original commit: elastic/x-pack-elasticsearch@070fb8146c
2017-01-13 11:54:25 +00:00
Dimitris Athanasiou c1ee50f238 Return 200 on GET requests for all resources when none exists (elastic/elasticsearch#694)
When a user makes a GET request to retrieve all resources of a type
(e.g. anomaly_detectors) and none exists, the response should be an
empty array with 200 status code. This commit fixes this issue for:

* anomaly_detectors and _stats
* schedulers and _stats
* lists
* buckets

All other GETs work fine already.

Original commit: elastic/x-pack-elasticsearch@4daaa91aa4
2017-01-12 17:46:39 +00:00
Martijn van Groningen c4d5cf660d Revert "fixed compile error due upstream changed"
This reverts commit elastic/x-pack@54d8aeadec.

Original commit: elastic/x-pack-elasticsearch@8846dc4ca5
2017-01-12 16:56:30 +01:00
Martijn van Groningen 4550d5f979 fixed compile error due upstream changed
Original commit: elastic/x-pack-elasticsearch@0dd924b287
2017-01-12 16:34:45 +01:00
Martijn van Groningen 9ec22efcba Removed last blocking client calls on network threads.
Closes elastic/elasticsearch#127

Original commit: elastic/x-pack-elasticsearch@3441f51764
2017-01-12 16:34:45 +01:00
Dimitrios Athanasiou d3e4ebcc0e Fix ScheduledJobsIT failure
Original commit: elastic/x-pack-elasticsearch@e43bb12ffb
2017-01-12 14:32:59 +00:00
Dimitris Athanasiou b93ec686f3 Use QUERY_THEN_FETCH (default) as search type for data extractor (elastic/elasticsearch#704)
I thought QUERY_AND_FETCH was the most efficient for the data extractor
but it does not work with sorting. It causes all shard results to be
returned before sorting and thus we may get out-of-order errors.

This commit switches to the default search type.

Original commit: elastic/x-pack-elasticsearch@d8a8155973
2017-01-12 13:36:09 +00:00
Hendrik Muhs af7edd84bb Fix buildbreak caused by upstream change (elastic/elasticsearch#700)
MLPlugin failed to compile du to removal of SearchRequestParsers (https://github.com/elastic/elasticsearch/pull/22538)

fixes elastic/elasticsearch#698

Original commit: elastic/x-pack-elasticsearch@73b7fca28c
2017-01-12 11:24:06 +01:00
Dimitris Athanasiou 3657d8a137 Scheduler robustness improvements (elastic/elasticsearch#687)
* Extract method ScheduledJob#postData

* Remove unreachable else statement

* Restrain usage of DataExtractor in a single thread

Original commit: elastic/x-pack-elasticsearch@5b9b310d9d
2017-01-10 17:09:01 +00:00
Dimitrios Athanasiou 51c50c5840 Remove deprecated use of ParseFieldMatcher
Original commit: elastic/x-pack-elasticsearch@e2150cf0aa
2017-01-10 13:42:58 +00:00
David Roberts c4038b52ed Rename prelert to ml (elastic/elasticsearch#681)
* prelert to ml
* Prelert to Ml
* PRELERT to ML

Exceptions:

* prelert.com - because it generally appears in links to our website, and
  although these will eventually break it will be possible for people to see
  what was there using https://archive.org/web/
* PRELERT_AWS_ACCESS_KEY_ID and PRELERT_AWS_SECRET_ACCESS_KEY - because it
  creates a knock-on effect on infra that will be temporary anyway because once
  we're in x-pack we'll use x-pack keys
* prelert-artifacts - this is the name of the s3 bucket we're currently using
  and you cannot rename s3 buckets - as with the access keys it will become
  obsolete when we merge to x-pack so there's no point changing it now
* prelert-legacy - the name of our legacy Git repo has not changed

Original commit: elastic/x-pack-elasticsearch@720e83c7f2
2017-01-10 13:40:16 +00:00
Martijn van Groningen 10d8a52b23 Made client calls non blocking in JobProvider#modelSnapshots(...)
Original commit: elastic/x-pack-elasticsearch@00790a5336
2017-01-10 12:57:28 +01:00
Martijn van Groningen 1d81509616 remove FixBlockingClientOperations usage in ElasticsearchBatchedDocumentsIterator as it is ok to make blocking from there. It is only used during remormalization which happens from a prelert thread and not a network thread.
Also removed some used code.

Original commit: elastic/x-pack-elasticsearch@2fe506099a
2017-01-10 12:56:09 +01:00
David Kyle 1a0151d020 Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@0696bda55c
2017-01-10 11:54:01 +00:00
David Kyle 9bc839b0fe Update yaml test with job id in path
Original commit: elastic/x-pack-elasticsearch@f74f1250d8
2017-01-10 11:53:48 +00:00
Martijn van Groningen 1a132e2c8b Made client calls non blocking in JobProvider#influencers(...)
and re-enabled some quantiles persistence unit tests (which can remain to be blocking as they aren't used on a network thread)

Original commit: elastic/x-pack-elasticsearch@cf8e78f42d
2017-01-10 12:49:51 +01:00
Dimitris Athanasiou 9e5245fd64 Replace http data extractor with a client extractor (elastic/elasticsearch#619)
* Replace http data extractor with a client extractor

This first implementation replaces the HTTP extractor
with a client extractor that uses search & scroll.

Note that this first implementation has some limitations:

- Only reads data that are in the _source
- Does not handle aggregated searches

These limitations will be addressed in follow up PRs.

Relates to elastic/elasticsearch#154

Original commit: elastic/x-pack-elasticsearch@f692ed961c
2017-01-10 11:45:17 +00:00
David Kyle dee7412044 Add job Id to custom all field (elastic/elasticsearch#598)
* Add job Id to custom all field

* Add yaml test for searching fields copied to custom all

Original commit: elastic/x-pack-elasticsearch@419189460f
2017-01-10 11:33:29 +00:00
Colin Goodheart-Smithe 68b8ce40fd Upgrades to 6.0.0-alpha1-SNAPSHOT (elastic/elasticsearch#672)
* Upgrades to ES 6.0.0-alpha1-SNAPSHOT

* Kibana changes to run upgrade to 6.0.0-alpha1-SNAPSHOT

* Other version changes to 6.0.0-alpha1-SNAPSHOT

Original commit: elastic/x-pack-elasticsearch@574d8573ab
2017-01-10 11:04:29 +00:00
Martijn van Groningen 2d8de6adb2 Fix scheduler config aggregation serialization logic due to upstream change in ES.
Original commit: elastic/x-pack-elasticsearch@3836cf753b
2017-01-10 11:39:04 +01:00
David Kyle fd4d412433 Fix GET bucket params validation (elastic/elasticsearch#661)
Original commit: elastic/x-pack-elasticsearch@66f522588b
2017-01-10 10:04:05 +00:00
David Roberts 45ef535b38 Endpoint adjustments (elastic/elasticsearch#662)
This commit contains around half of the endpoint changes Sophie and Steve
agreed with Clint:

1) Automatic job ID generation is removed

2) Job IDs must now be specified in the URL when putting a job; to avoid
   breaking many test configs, job IDs may also be specified in the job config
   body, but in this case the value specified must match the URL argument

3) The endpoint name for posting data is now post_data instead of job_data

4) The post_data endpoint ends with _data instead of data

5) modelsnapshots is renamed to model_snapshots in all related endpoints

6) PUT model_snapshots/description is changed to POST model_snapshots/_update

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@c379a23f3c
2017-01-09 15:52:07 +00:00
Colin Goodheart-Smithe 0d7ac401f9 Removes duplicate influencer field declaration for results (elastic/elasticsearch#668)
The `influencer_field_name` field was declared two in the results mapping. Once directly from `ElasticsearchMappings.resultsMapping()` and again from `addInfluencerFieldsToMapping(XContentBuilder)` which the `resultsMapping()` method calls.

this change removes the duplicate.

Original commit: elastic/x-pack-elasticsearch@5707a5ee53
2017-01-09 13:59:12 +00:00
Martijn van Groningen c06342b556 use ActionListener.wrap(...) helper rather than anonymous action listener implementations
Original commit: elastic/x-pack-elasticsearch@d3c7e176ef
2017-01-09 13:59:47 +01:00
Martijn van Groningen ade3f6f207 fixed integration tests due to upstream changes
Original commit: elastic/x-pack-elasticsearch@e6bf9bb4a1
2017-01-09 12:03:14 +01:00
Martijn van Groningen 8e6aa2ba3d fixed compile errors due to upstream changes.
Original commit: elastic/x-pack-elasticsearch@9d3a96680d
2017-01-09 10:38:49 +01:00
Zachary Tong a375d90547 Allow deletes to proceed even if index is missing (elastic/elasticsearch#660)
Allow deletes to proceed even if index is missing

Also adds some tests.  All non-IndexNotFound exceptions will still abort the delete.
We can revisit this if we find other edge-cases.

Original commit: elastic/x-pack-elasticsearch@823d00d8a7
2017-01-06 12:26:13 -05:00
Martijn van Groningen d7f6de7133 Made client calls non blocking in JobProvider#modelSizeStats(...)
and FixBlockingClientOperations in two places where blocking client calls are ok,
because these methods aren't called from a network thread.

Original commit: elastic/x-pack-elasticsearch@a6dc34651c
2017-01-06 18:02:07 +01:00
Zachary Tong ae8695a22d Tweak logic to allow Deletes on already-deleting job (elastic/elasticsearch#658)
Original commit: elastic/x-pack-elasticsearch@5352cba1e7
2017-01-06 10:07:28 -05:00
Martijn van Groningen 3c3509d397 create XContentParser instances inside try code block
Original commit: elastic/x-pack-elasticsearch@664cba1633
2017-01-06 15:20:07 +01:00
Martijn van Groningen 092d2e2bdc Made client calls non blocking for JobProvider#getDataCounts(..)
Original commit: elastic/x-pack-elasticsearch@4d6d6360f6
2017-01-06 14:39:52 +01:00
Martijn van Groningen 468402426e Made client calls non blocking in get category apis.
Merged categoryDefinition(...) into categoryDefinitions(...) as the two did similar things. The get call has been replaced with a search with a query on the _uid field and routing on category id, so that the response handling code can be reused.

Original commit: elastic/x-pack-elasticsearch@4243917b00
2017-01-06 11:18:35 +01:00
David Roberts ee9c691858 Switch to tracking Elasticsearch/Kibana 5.3.0-SNAPSHOT (elastic/elasticsearch#651)
Original commit: elastic/x-pack-elasticsearch@75c54427ac
2017-01-06 10:06:08 +00:00
Martijn van Groningen 66a2f999e5 scheduler: prevent stopping scheduler before the scheduler was actually started
The start scheduler waits until the scheduler state has been set to started before returning.
Before this change after the scheduler state has been set to started, the scheduler would link itself to the task the start scheduler api has created.
If the stop scheduler api was called immediately after the start scheduler api then this could lead the stop scheduler api cancelling the task without
stopping the scheduler, as the scheduler could not have been linked to the task.

Now the scheduler gets linked to the task before the scheduler state is set to started, fixing the problematic situation discribed above.

Original commit: elastic/x-pack-elasticsearch@8334ae1967
2017-01-05 21:10:23 +01:00
Colin Goodheart-Smithe b9205142a4 fix checkstyle
Original commit: elastic/x-pack-elasticsearch@b86e94228b
2017-01-05 16:29:18 +00:00
Colin Goodheart-Smithe 8fdeedf61d fix generics compile error in Eclipse
Eclipse compiler needs a hint to tell it the generics for the comparator function

Original commit: elastic/x-pack-elasticsearch@b3d71d3465
2017-01-05 16:23:24 +00:00
David Kyle 70aa238327 Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@88e78a3417
2017-01-05 14:43:37 +00:00
David Kyle 42898ba421 Fix noisy log message about missing mappings
From the scheduler integration test

Original commit: elastic/x-pack-elasticsearch@02d3f58d83
2017-01-05 14:43:29 +00:00
David Roberts 2746edf21b Properly assert that large state documents can be consumed quickly
AssertBusy was not the correct way to do this

Original commit: elastic/x-pack-elasticsearch@c6535e7545
2017-01-05 13:07:56 +00:00
Martijn van Groningen 41f2f51df6 Made client calls in get buckets code non blocking.
Also merged the JobProvider#getBucket(...) method into Jobprovider#getBuckets(...) method, because
it contained a lot of similar logic, otherwise it had to be converted to use non blocking client calls too.

Part of elastic/elasticsearch#127

Original commit: elastic/x-pack-elasticsearch@b1e66b62cb
2017-01-05 13:58:46 +01:00
Martijn van Groningen 51e1199860 Install prelert metadata and create required indices only once.
Original commit: elastic/x-pack-elasticsearch@12c8ba0ce0
2017-01-05 13:38:48 +01:00
David Roberts 27c9f39bf5 Speed up state processing (elastic/elasticsearch#642)
There was an N-squared algorithm in the state processing code, leading
to large state persistence eventually timing out.  Large state documents
are read from the network in 8KB chunks, and the old code was checking
ALL previously read chunks for separators every time a new chunk was read.

Fixes elastic/elasticsearch#635

Original commit: elastic/x-pack-elasticsearch@c814858c2c
2017-01-05 12:37:11 +00:00
David Roberts c7cfa56aaf Rename prelert to ml in endpoint base path (elastic/elasticsearch#639)
Original commit: elastic/x-pack-elasticsearch@ff6745e545
2017-01-05 08:18:43 +00:00
polyfractal f98b4a967c s/nocommit/norelease
Original commit: elastic/x-pack-elasticsearch@6523f5caa1
2017-01-04 16:50:51 -05:00
Zachary Tong 23aef2679d DELETING job status cluster state changes (elastic/elasticsearch#612)
Deleting a job now starts a three-step process:

1. Job status updated to DELETING
2. Physical index is deleted
3. Job removed from cluster state

When jobs are in DELETING, they cannot be modified/updated/changed at all.  Only jobs that are DELETING can actually be removed from the CS.

Original commit: elastic/x-pack-elasticsearch@2cd99a240c
2017-01-04 16:43:24 -05:00
Martijn van Groningen 090c7e792a in case of general exception no need to execute the rest of the method
Original commit: elastic/x-pack-elasticsearch@d0cdaf037d
2017-01-04 15:52:17 +01:00
Martijn van Groningen 5371cc8454 When failures happen during the scheduler's loopback run then make sure we either continue or stop.
Relates to elastic/elasticsearch#628

Original commit: elastic/x-pack-elasticsearch@c5726bd8e4
2017-01-04 15:52:17 +01:00
David Roberts a816ee27bb Fix error message for state index creation
Original commit: elastic/x-pack-elasticsearch@cd36db071e
2017-01-04 13:31:03 +00:00
Martijn van Groningen 567153d73a test: create airline-data index as part of test setup
Original commit: elastic/x-pack-elasticsearch@81dea8b071
2017-01-03 22:35:31 +01:00