Commit Graph

321 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe d0b36fd52a Change pack artifact to be more maven-like
Original commit: elastic/x-pack-elasticsearch@e8eed37553
2017-01-19 14:03:40 +00:00
Colin Goodheart-Smithe 33800bae5e Changes build to make cpp artifact download correct with ml-cpp changes (elastic/elasticsearch#754)
https://github.com/elastic/machine-learning-cpp/pull/3 changes the artifact names and paths for the ml-cpp build. This change makes it so the machine learning build references the artifacts in their new location.

Original commit: elastic/x-pack-elasticsearch@d3916b6a7f
2017-01-19 13:51:01 +00:00
Dimitris Athanasiou 0b084ea0e6 Treat timestamps without timezone as UTC (elastic/elasticsearch#753)
Original commit: elastic/x-pack-elasticsearch@33ab2fb781
2017-01-19 13:49:14 +00:00
David Roberts 36bdcaff5d Rename scheduler/scheduled to datafeed (elastic/elasticsearch#755)
Relates elastic/elasticsearch#630

The more subtle changes to the datafeed endpoints required by elastic/elasticsearch#630
are NOT in this commit, as they would be drowned out by the rename

Original commit: elastic/x-pack-elasticsearch@3318971da9
2017-01-19 13:44:19 +00:00
David Roberts 10441a3e38 More endpoint adjustments (elastic/elasticsearch#750)
This commit contains some more of the endpoint changes Sophie and Steve
agreed with Clint:

1. get_jobs_stats renamed to get_job_stats

2. Revert snapshot must now be done using an ID - other options removed

3. Renamed "categorydefinitions" to "categories" in endpoints

4. get_jobs now has an implicit _all if no job ID/wildcard is specified

5. There is an option to retrieve a specific model snapshot by ID in
   get_model_snapshots

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@9dd71c64a8
2017-01-19 11:41:35 +00:00
David Kyle e826a56212 Make document Ids unique if in a shared index (elastic/elasticsearch#749)
Original commit: elastic/x-pack-elasticsearch@ecc7e876ce
2017-01-19 09:31:03 +00:00
Martijn van Groningen d3c589c33d Moved waiting for scheduler started logic into StartSchedulerAction.TransportAction and moved the logic that was original there to a new action named InternalStartSchedulerAction.
This change prepares for elastic/elasticsearch/elastic/elasticsearch#22575, where we don't have ClusterService available in rest actions.

Original commit: elastic/x-pack-elasticsearch@87658c7fe8
2017-01-19 09:38:10 +01:00
Dimitris Athanasiou c33f26976d Improve field extraction in scheduler (elastic/elasticsearch#748)
This commit performs the following improvements:

- the time field is always requested as doc_value. This makes
specifying a time format for scheduled jobs unnecessary.
- adds DataDescription as a param to the PostDataAction. When set,
it overrides the job's DataDescription. This allows the scheduler to
override the job's DataDescription since it knows the data format (JSON)
and the time format (epoch_ms). This is not exposed in the REST API to
discourage users from using it.
- by default, data extractor search now requests doc_values for analysis fields. This is
expected to result in increased performance.
- a `_source` field is added to the scheduler config. This needs to be
set to true when one or more of the analysis fields do not have
doc_values.
- the ELASTICSEARCH data format is removed as is now redundant.
- fixes the usage of `script_fields`. Previously, setting
`script_fields` would result to none of the source to be returned. Thus,
is the analysis fields were a mixture of script and non-script fields it
would not work.
- ensures nested fields are handled properly

Closes elastic/elasticsearch#679, Closes elastic/elasticsearch#267 

Original commit: elastic/x-pack-elasticsearch@fed35ed354
2017-01-18 18:46:43 +00:00
David Kyle 4c0d2a492d Refactor get methods (elastic/elasticsearch#747)
Original commit: elastic/x-pack-elasticsearch@d300be2dde
2017-01-18 13:35:25 +00:00
James Gowdy b2917376f0 Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@c198cef9d3
2017-01-18 09:53:15 +00:00
Martijn van Groningen 40332c7e1c use client instead of transport action directly in rest actions
Original commit: elastic/x-pack-elasticsearch@4c3380ceb9
2017-01-17 20:38:53 +01:00
Martijn van Groningen d9a75424d0 fixed wrong mockito import in test
Original commit: elastic/x-pack-elasticsearch@c6a7232a87
2017-01-17 20:22:30 +01:00
David Roberts 449a74b2fd C++ log message handler now remembers C++ process copyright message (elastic/elasticsearch#743)
Once we're in x-pack this (or the portion of it containing the version)
can be returned in the ml feature info of the x-pack info endpoint

Relates elastic/elasticsearch#566

Original commit: elastic/x-pack-elasticsearch@b2ea740a6d
2017-01-17 17:19:48 +00:00
David Kyle cfb94b6627 Rename result iterators (elastic/elasticsearch#740)
Original commit: elastic/x-pack-elasticsearch@c462e9595a
2017-01-17 16:02:02 +00:00
David Roberts 92c808fd97 Remove C++ code
It now lives in the machine-learning-cpp repository

Original commit: elastic/x-pack-elasticsearch@11ac8212d7
2017-01-17 14:46:11 +00:00
David Roberts f24e8c6d54 Remove C++ from build files (elastic/elasticsearch#739)
NB: The actual C++ code will be deleted in a separate commit to
avoid swamping this commit.

If you want to have the Java build pick up locally built C++ then:

export CPP_LOCAL_DISTS=$CPP_SRC_HOME/build/distributions

Otherwise, C++ artifacts will be downloaded from S3.

Original commit: elastic/x-pack-elasticsearch@246672e81d
2017-01-17 14:45:00 +00:00
David Kyle 9af2c2cbeb Fix check style error (wildcard import)
Original commit: elastic/x-pack-elasticsearch@d5ba5d8dda
2017-01-17 13:57:08 +00:00
David Kyle bc04bda8d6 Remember the index each result came from (elastic/elasticsearch#727)
* Delete unused batched ModelSnapshot iterator

* Pass source index with normalisable results

* Refactor Normalizable

* Rework persisting renormalised results

* Spell normalize with a ‘z’

* Rename ResultIndex -> ResultWithIndex

* Expand wildcard import

* Make Normalisable child type an enum

Original commit: elastic/x-pack-elasticsearch@52450abafd
2017-01-17 13:11:57 +00:00
Martijn van Groningen 1d891965c1 Stop scheduled job only once.
If scheduled job concurrently gets stopped from within (e.g. lookback) and externally via the stop scheduler api then make sure to execute the stop logic only once.

Original commit: elastic/x-pack-elasticsearch@505c44f515
2017-01-17 08:26:09 +01:00
Martijn van Groningen e0b6a1e493 Removed unneeded task (de)registering as the super class does this already
Original commit: elastic/x-pack-elasticsearch@f5be371e83
2017-01-17 08:19:53 +01:00
David Roberts 5cd79f20fa Add company name to copyright messages
Original commit: elastic/x-pack-elasticsearch@8ee353a550
2017-01-13 19:46:48 +00:00
David Roberts bc03dba9a2 Add facility to cross compile for Mac OS X on Linux (elastic/elasticsearch#717)
Adding a build system that makes it possible for an Ubuntu 16.04 build
server configured according to the instructions in
https://github.com/elastic/prelert-legacy/wiki/Plugin-Development-Setup-for-Mac-OS-X-cross-compiled-on-Linux
to build a Mac OS X version of the product.

Original commit: elastic/x-pack-elasticsearch@c6f39c0395
2017-01-13 17:42:11 +00:00
David Kyle c9a8d9d283 Update unit test after removing _all
Original commit: elastic/x-pack-elasticsearch@59c80a6ba3
2017-01-13 14:25:41 +00:00
Dimitris Athanasiou 4128d635f4 Remove explicit _all disabling from mappings (elastic/elasticsearch#719)
The _all field is now deprecated and disabled by default in elasticsearch
6.0.0. We no longer need to disable it explicitly.

Original commit: elastic/x-pack-elasticsearch@c71465083a
2017-01-13 12:39:59 +00:00
Dimitris Athanasiou 3ce48bc7b9 Remove ParseFieldMatcher (elastic/elasticsearch#718)
Original commit: elastic/x-pack-elasticsearch@070fb8146c
2017-01-13 11:54:25 +00:00
Dimitris Athanasiou c1ee50f238 Return 200 on GET requests for all resources when none exists (elastic/elasticsearch#694)
When a user makes a GET request to retrieve all resources of a type
(e.g. anomaly_detectors) and none exists, the response should be an
empty array with 200 status code. This commit fixes this issue for:

* anomaly_detectors and _stats
* schedulers and _stats
* lists
* buckets

All other GETs work fine already.

Original commit: elastic/x-pack-elasticsearch@4daaa91aa4
2017-01-12 17:46:39 +00:00
Martijn van Groningen c4d5cf660d Revert "fixed compile error due upstream changed"
This reverts commit elastic/x-pack@54d8aeadec.

Original commit: elastic/x-pack-elasticsearch@8846dc4ca5
2017-01-12 16:56:30 +01:00
Martijn van Groningen 4550d5f979 fixed compile error due upstream changed
Original commit: elastic/x-pack-elasticsearch@0dd924b287
2017-01-12 16:34:45 +01:00
Martijn van Groningen 9ec22efcba Removed last blocking client calls on network threads.
Closes elastic/elasticsearch#127

Original commit: elastic/x-pack-elasticsearch@3441f51764
2017-01-12 16:34:45 +01:00
Dimitrios Athanasiou d3e4ebcc0e Fix ScheduledJobsIT failure
Original commit: elastic/x-pack-elasticsearch@e43bb12ffb
2017-01-12 14:32:59 +00:00
Dimitris Athanasiou b93ec686f3 Use QUERY_THEN_FETCH (default) as search type for data extractor (elastic/elasticsearch#704)
I thought QUERY_AND_FETCH was the most efficient for the data extractor
but it does not work with sorting. It causes all shard results to be
returned before sorting and thus we may get out-of-order errors.

This commit switches to the default search type.

Original commit: elastic/x-pack-elasticsearch@d8a8155973
2017-01-12 13:36:09 +00:00
Hendrik Muhs af7edd84bb Fix buildbreak caused by upstream change (elastic/elasticsearch#700)
MLPlugin failed to compile du to removal of SearchRequestParsers (https://github.com/elastic/elasticsearch/pull/22538)

fixes elastic/elasticsearch#698

Original commit: elastic/x-pack-elasticsearch@73b7fca28c
2017-01-12 11:24:06 +01:00
Dimitris Athanasiou 3657d8a137 Scheduler robustness improvements (elastic/elasticsearch#687)
* Extract method ScheduledJob#postData

* Remove unreachable else statement

* Restrain usage of DataExtractor in a single thread

Original commit: elastic/x-pack-elasticsearch@5b9b310d9d
2017-01-10 17:09:01 +00:00
Dimitrios Athanasiou 51c50c5840 Remove deprecated use of ParseFieldMatcher
Original commit: elastic/x-pack-elasticsearch@e2150cf0aa
2017-01-10 13:42:58 +00:00
David Roberts c4038b52ed Rename prelert to ml (elastic/elasticsearch#681)
* prelert to ml
* Prelert to Ml
* PRELERT to ML

Exceptions:

* prelert.com - because it generally appears in links to our website, and
  although these will eventually break it will be possible for people to see
  what was there using https://archive.org/web/
* PRELERT_AWS_ACCESS_KEY_ID and PRELERT_AWS_SECRET_ACCESS_KEY - because it
  creates a knock-on effect on infra that will be temporary anyway because once
  we're in x-pack we'll use x-pack keys
* prelert-artifacts - this is the name of the s3 bucket we're currently using
  and you cannot rename s3 buckets - as with the access keys it will become
  obsolete when we merge to x-pack so there's no point changing it now
* prelert-legacy - the name of our legacy Git repo has not changed

Original commit: elastic/x-pack-elasticsearch@720e83c7f2
2017-01-10 13:40:16 +00:00
Martijn van Groningen 10d8a52b23 Made client calls non blocking in JobProvider#modelSnapshots(...)
Original commit: elastic/x-pack-elasticsearch@00790a5336
2017-01-10 12:57:28 +01:00
Martijn van Groningen 1d81509616 remove FixBlockingClientOperations usage in ElasticsearchBatchedDocumentsIterator as it is ok to make blocking from there. It is only used during remormalization which happens from a prelert thread and not a network thread.
Also removed some used code.

Original commit: elastic/x-pack-elasticsearch@2fe506099a
2017-01-10 12:56:09 +01:00
David Kyle 1a0151d020 Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@0696bda55c
2017-01-10 11:54:01 +00:00
David Kyle 9bc839b0fe Update yaml test with job id in path
Original commit: elastic/x-pack-elasticsearch@f74f1250d8
2017-01-10 11:53:48 +00:00
Martijn van Groningen 1a132e2c8b Made client calls non blocking in JobProvider#influencers(...)
and re-enabled some quantiles persistence unit tests (which can remain to be blocking as they aren't used on a network thread)

Original commit: elastic/x-pack-elasticsearch@cf8e78f42d
2017-01-10 12:49:51 +01:00
Dimitris Athanasiou 9e5245fd64 Replace http data extractor with a client extractor (elastic/elasticsearch#619)
* Replace http data extractor with a client extractor

This first implementation replaces the HTTP extractor
with a client extractor that uses search & scroll.

Note that this first implementation has some limitations:

- Only reads data that are in the _source
- Does not handle aggregated searches

These limitations will be addressed in follow up PRs.

Relates to elastic/elasticsearch#154

Original commit: elastic/x-pack-elasticsearch@f692ed961c
2017-01-10 11:45:17 +00:00
David Kyle dee7412044 Add job Id to custom all field (elastic/elasticsearch#598)
* Add job Id to custom all field

* Add yaml test for searching fields copied to custom all

Original commit: elastic/x-pack-elasticsearch@419189460f
2017-01-10 11:33:29 +00:00
Colin Goodheart-Smithe 68b8ce40fd Upgrades to 6.0.0-alpha1-SNAPSHOT (elastic/elasticsearch#672)
* Upgrades to ES 6.0.0-alpha1-SNAPSHOT

* Kibana changes to run upgrade to 6.0.0-alpha1-SNAPSHOT

* Other version changes to 6.0.0-alpha1-SNAPSHOT

Original commit: elastic/x-pack-elasticsearch@574d8573ab
2017-01-10 11:04:29 +00:00
Martijn van Groningen 2d8de6adb2 Fix scheduler config aggregation serialization logic due to upstream change in ES.
Original commit: elastic/x-pack-elasticsearch@3836cf753b
2017-01-10 11:39:04 +01:00
David Kyle fd4d412433 Fix GET bucket params validation (elastic/elasticsearch#661)
Original commit: elastic/x-pack-elasticsearch@66f522588b
2017-01-10 10:04:05 +00:00
David Roberts 45ef535b38 Endpoint adjustments (elastic/elasticsearch#662)
This commit contains around half of the endpoint changes Sophie and Steve
agreed with Clint:

1) Automatic job ID generation is removed

2) Job IDs must now be specified in the URL when putting a job; to avoid
   breaking many test configs, job IDs may also be specified in the job config
   body, but in this case the value specified must match the URL argument

3) The endpoint name for posting data is now post_data instead of job_data

4) The post_data endpoint ends with _data instead of data

5) modelsnapshots is renamed to model_snapshots in all related endpoints

6) PUT model_snapshots/description is changed to POST model_snapshots/_update

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@c379a23f3c
2017-01-09 15:52:07 +00:00
Colin Goodheart-Smithe 0d7ac401f9 Removes duplicate influencer field declaration for results (elastic/elasticsearch#668)
The `influencer_field_name` field was declared two in the results mapping. Once directly from `ElasticsearchMappings.resultsMapping()` and again from `addInfluencerFieldsToMapping(XContentBuilder)` which the `resultsMapping()` method calls.

this change removes the duplicate.

Original commit: elastic/x-pack-elasticsearch@5707a5ee53
2017-01-09 13:59:12 +00:00
Martijn van Groningen c06342b556 use ActionListener.wrap(...) helper rather than anonymous action listener implementations
Original commit: elastic/x-pack-elasticsearch@d3c7e176ef
2017-01-09 13:59:47 +01:00
Martijn van Groningen ade3f6f207 fixed integration tests due to upstream changes
Original commit: elastic/x-pack-elasticsearch@e6bf9bb4a1
2017-01-09 12:03:14 +01:00
Martijn van Groningen 8e6aa2ba3d fixed compile errors due to upstream changes.
Original commit: elastic/x-pack-elasticsearch@9d3a96680d
2017-01-09 10:38:49 +01:00