Commit Graph

376 Commits

Author SHA1 Message Date
Zachary Tong a11ddd1e04 Integrate domainSplit function into datafeeds (elastic/elasticsearch#841)
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.

The majority of this PR is actually test-related.  Adds a unit test to check for the injected
script/params.  Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly.  The QA test can
be simplified greatly once the Preview API is added.

Original commit: elastic/x-pack-elasticsearch@c7c35a982c
2017-02-01 10:20:00 -05:00
Martijn van Groningen ec902c4dc3 test: change assertion to be more lenient to platform specifics
Original commit: elastic/x-pack-elasticsearch@2131e7f0c7
2017-02-01 14:49:45 +01:00
Martijn van Groningen 051d8d8fdf Moved start and stop datafeeder apis over the persistent task infrastructure
Original commit: elastic/x-pack-elasticsearch@8e15578fb7
2017-01-31 22:50:00 +01:00
Martijn van Groningen 22282e9d56 cleanup toString() methods
Original commit: elastic/x-pack-elasticsearch@17a10ea68f
2017-01-31 22:35:54 +01:00
Martijn van Groningen ce6dc4a506 Make job stats api task aware.
This will allow the job stats api to redirect the request to node where job is running.

Original commit: elastic/x-pack-elasticsearch@9f1d12dfcb
2017-01-31 22:35:54 +01:00
Martijn van Groningen b07e9bbd07 Fixed AOBE caused by fetching model state when opening a job.
This error only occurred for jobs that have been opened before and persisted model state.

Closes elastic/elasticsearch#836

Original commit: elastic/x-pack-elasticsearch@ad76f4167f
2017-01-31 19:56:24 +01:00
David Roberts 7ff3b707a8 Make renormalization thread-safe (elastic/elasticsearch#840)
Each ScoresUpdater needs its own JobRenormalizedResultsPersister, because
each JobRenormalizedResultsPersister has a single BulkRequest that various
methods update.

Fixes elastic/elasticsearch#838

Original commit: elastic/x-pack-elasticsearch@90f4bbd5a0
2017-01-31 16:51:26 +00:00
Colin Goodheart-Smithe f804bb1917 Removes ensureGreen from PlainlessDomainSplitIT (elastic/elasticsearch#839)
This shouldn’t be needed as the cluster no longer goes red when an index is created.

Original commit: elastic/x-pack-elasticsearch@b554ea9caf
2017-01-31 16:09:25 +00:00
David Kyle 97970b94cd Remove the ignoreDowntime parameter from the _data endpoint (elastic/elasticsearch#834)
The parameter only applies when a job is opened

Original commit: elastic/x-pack-elasticsearch@37b902aa2a
2017-01-31 11:48:58 +00:00
David Roberts 34274a30ed Make transport layer names consistent with corresponding endpoints (elastic/elasticsearch#822)
Closes elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@32aae3e1d9
2017-01-31 11:42:06 +00:00
David Kyle c84f227857 Set memory usage log message to trace (elastic/elasticsearch#829)
Original commit: elastic/x-pack-elasticsearch@13412cc4cf
2017-01-31 09:54:56 +00:00
David Roberts ab957b6d91 Adjust validation endpoints (elastic/elasticsearch#812)
Changes are:

1. The detector validation endpoint is changed from /_xpack/ml/_validate/detector
   to /_xpack/ml/anomaly_detectors/_validate/detector
2. A new endpoint is added for validating an entire job config:
   /_xpack/ml/anomaly_detectors/_validate

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@7b2031e746
2017-01-30 17:10:22 +00:00
David Kyle 4eab74ce29 Store input fields for anomaly records and influencers (elastic/elasticsearch#799)
* Store input fields for anomaly records and influencers

* Address review comments

* Remove DotNotationReverser

* Remove duplicated constants

* Can’t use the same date for all records as they will have equivalent Ids

Original commit: elastic/x-pack-elasticsearch@40796b5efc
2017-01-30 14:05:18 +00:00
Colin Goodheart-Smithe 79d1a10a86 Mutes DataFeedJobIT test method that uses painless
This needs to be moved to the single-node-tests qa modules since integTests shouldn’t access modules.

Original commit: elastic/x-pack-elasticsearch@289b697eb8
2017-01-30 10:55:59 +00:00
Colin Goodheart-Smithe 618cb2a1a0 Make all action names cluster actions
Original commit: elastic/x-pack-elasticsearch@815d8f0aac
2017-01-30 10:00:23 +00:00
Igor Motov 827118e154 Adds support for persistent actions
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle.  See TestPersistentActionPlugin for an example implementing a persistent action.

Original commit: elastic/x-pack-elasticsearch@8ef4103cd6
2017-01-27 11:20:54 -05:00
Martijn van Groningen ff65c38253 [TEST] fixed mocking logic to include id
Original commit: elastic/x-pack-elasticsearch@7b20e92fdc
2017-01-27 17:20:18 +01:00
Martijn van Groningen 2059b91620 Workaround for index request without an id being retried that are tripping an assertion in internal engine. (2)
Original commit: elastic/x-pack-elasticsearch@22d5060deb
2017-01-27 17:07:51 +01:00
Martijn van Groningen ad4218320c Workaround for index request without an id being retried that are tripping an assertion in internal engine.
Original commit: elastic/x-pack-elasticsearch@ba44acc28b
2017-01-27 16:15:00 +01:00
David Roberts 64fdb039ab Reduce the controller connect timeout (elastic/elasticsearch#804)
This used to be 60 seconds, dating back to the days when the controller
had to be started manually after starting Elasticsearch.  However, now
Elasticsearch starts it automatically it should already be running when
we try to connect, so the timeout can be much lower.  It just needs to
be long enough to give the C++ process time to create its named pipes.
2 seconds seems reasonable, and matches what we use for autodetect and
normalize.

Original commit: elastic/x-pack-elasticsearch@7300d68482
2017-01-27 14:23:18 +00:00
Zachary Tong 9395ef81b1 Painless DomainSplit tests in new Single-Node QA Module (elastic/elasticsearch#787)
This contains the Painless-based DomainSplit function, generated static maps and basic tests.  Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response


Original commit: elastic/x-pack-elasticsearch@c6c2942e01
2017-01-27 08:52:48 -05:00
Dimitris Athanasiou 91be1e719d Disable stored_fields when possible in ScrollDataExtractor (elastic/elasticsearch#801)
When source fields are not required, stored_fields can be disabled.
This can make the query faster as no stored fields have to be
decompressed. Note that this means no metadata (_id, _index, _type, etc.)
will be returned.

Original commit: elastic/x-pack-elasticsearch@b1ea526d83
2017-01-27 11:38:54 +00:00
Dimitris Athanasiou 5790a6f152 Handle shard failures in extractors (elastic/elasticsearch#794)
Even though a search response may return a 200 status code, things could
still have gone wrong. A search response may report shard failures.

The datafeed extractors should check for that and report an extraction
error accordingly.

Closes elastic/elasticsearch#775

Original commit: elastic/x-pack-elasticsearch@5d6d899738
2017-01-26 16:01:43 +00:00
David Kyle efc47c2a6f Remove Usage classes (elastic/elasticsearch#796)
* Delete usage class

* Delete usage reporter

* Remove unused constant

Original commit: elastic/x-pack-elasticsearch@c7a6c457bd
2017-01-26 11:50:08 +00:00
David Kyle db14d89358 Fix checkstyle
Original commit: elastic/x-pack-elasticsearch@05d59da705
2017-01-26 10:05:03 +00:00
David Kyle e3bb7cfea3 Split ml-int index into .ml-audit and .ml-meta (elastic/elasticsearch#752)
* Audit messages in .ml-audit

* Rename ml-int to .ml-meta

* Remove no release comment

* Fix compilation after classes moved to a different package

* Create the Audit, state and meta indices every time a job is created

* Revert change creating the audit index etc when the job is created

* Rename index .ml-audit -> .ml-notifications

Original commit: elastic/x-pack-elasticsearch@95168fa341
2017-01-26 09:44:54 +00:00
Martijn van Groningen 3a36f94a4a When timeout has been reached, check one more time if the job / datafeed status has the expected value.
Decreased wait timeout from 30s to 20s

Original commit: elastic/x-pack-elasticsearch@b46fb0abe3
2017-01-25 23:32:04 +01:00
Dimitris Athanasiou 86291c12e2 Handle manual aggregations in datafeeds (elastic/elasticsearch#784)
* Handle manual aggregations in datafeeds

Adds a DataExtractor implementation that runs aggregated searches.

The manual aggregations supported have the following limitations:

- each aggregation can hava 0 or 1 sub-aggregations
- the top aggregation has to be a histogram
- sub-aggregations have to be either terms aggregations or single value
metric aggregations.

The response is converted into flat JSON documents that contain only the
fields of interest and can be parsed without additional context from our
JSON parser. The fields in the JSON documents correspond to the names of the aggregations.

Closes elastic/elasticsearch#680

Original commit: elastic/x-pack-elasticsearch@7dfd2d31e6
2017-01-25 19:13:03 +00:00
Colin Goodheart-Smithe 716f543f7b Adds a new constructor to plugin
The new constructor takes an Environment object. This is needed for migration to X-Pack since the environment instance is built by the XPackPlugin and then passed into the feature plugins.

Original commit: elastic/x-pack-elasticsearch@f25225bc6a
2017-01-25 18:45:04 +00:00
David Roberts 4b366f8ef6 Removing transforms and the SINGLE_LINE input format (elastic/elasticsearch#790)
Most transforms will be replaced with Painless scripts.

The exception is the DateTransform, whose functionality is now simplified
to what existed before the other transforms were added.

The SINGLE_LINE format relied on transforms to extract fields, so has also
been removed, but this is reasonable as it strays into Logstash territory.

Relates elastic/elasticsearch#630

Closes elastic/elasticsearch#39

Original commit: elastic/x-pack-elasticsearch@a593d3e0ad
2017-01-25 15:51:50 +00:00
James Gowdy 99c9d3733f Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@c38b4d4c8b
2017-01-25 14:17:00 +00:00
Colin Goodheart-Smithe 603fa47580 Adds an option to disable the ML plugin (elastic/elasticsearch#785)
Adds an `xpack.ml.enabled` node level setting that can be used to enable and disable the plugin. This will be important when we migrate to X-Pack

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@e5c4969a96
2017-01-24 16:14:56 +00:00
Martijn van Groningen e9f899e57a Improved datafeed logging for stopping
Original commit: elastic/x-pack-elasticsearch@94bd5d6a00
2017-01-24 16:00:54 +01:00
Martijn van Groningen b636a4b829 Fixed timeout (de-)serialization for start and stop datafeeder and open job apis.
Original commit: elastic/x-pack-elasticsearch@be054db48c
2017-01-24 15:53:54 +01:00
Martijn van Groningen 5684caece0 [TEST] print hot threads when op job fails
Original commit: elastic/x-pack-elasticsearch@8820758495
2017-01-24 15:51:37 +01:00
David Roberts ba7fa6b69d Top level build.gradle should have build task
Original commit: elastic/x-pack-elasticsearch@bc5aa9e5ae
2017-01-24 14:48:03 +00:00
Martijn van Groningen 8dbaef186e [TEST] use unique job ids to make debugging log files easier
Original commit: elastic/x-pack-elasticsearch@9f04e1b01f
2017-01-24 12:01:21 +01:00
Martijn van Groningen 29451bb7e3 [TEST] select timestamp differently for test documents
Original commit: elastic/x-pack-elasticsearch@679273012c
2017-01-24 11:07:24 +01:00
David Roberts 215410e93f Rename list to filter (elastic/elasticsearch#774)
Part of the endpoint rename Sophie and Steve agreed with Clint

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@6ded117849
2017-01-24 10:01:24 +00:00
Martijn van Groningen a7d1918461 [TEST] added more logging
Original commit: elastic/x-pack-elasticsearch@062a0b41b8
2017-01-23 17:54:14 +01:00
David Roberts d693f8317c Run build as well as check from CI to bundle Kibana plugin (elastic/elasticsearch#773)
The check task bundles the Elasticsearch part of the plugin but not
Kibana

Fixes elastic/elasticsearch#772

Original commit: elastic/x-pack-elasticsearch@ead2ff3686
2017-01-23 15:48:42 +00:00
Colin Goodheart-Smithe db0f4c0977 Fixes projects refs in build (elastic/elasticsearch#771)
The build would fail certain task (e.g. `gradle tasks` and `gradle :prelert-legacy:test`) if run from the elasticsearch root directory because the project references in the build were not abolute project paths from the root, they were relative to the `prelert-legacy` project. This change fixes the references so they are all absolute project paths.

Original commit: elastic/x-pack-elasticsearch@e549533834
2017-01-23 14:09:22 +00:00
David Roberts cd2332730b Move the named pipe no bootstrap test to a separate qa module (elastic/elasticsearch#769)
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch.  This should make merging to
x-pack easier.

Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.

Original commit: elastic/x-pack-elasticsearch@5547f457b6
2017-01-23 12:08:35 +00:00
Dimitris Athanasiou b3b8a7edc9 Restructure packages (elastic/elasticsearch#767)
Restructure packages according to plan described in elastic/elasticsearch#730.
Copying here for completeness.

* Move config classes
* DataCounts/Quantiles/state are process.autodetect.state
* AutodetectProcessManager move to process.autodetect
* lists -> config
* PageParams, QueryPage -> action.util
* StatusReporter -> process.DataCountsReporter
* CountingInputStream -> process
* JobManager -> job
* CppLog* -> process.logging
* DataProcessor -> collapse into implementation
* job.audit -> ml.notifications

Closes elastic/elasticsearch#730

Original commit: elastic/x-pack-elasticsearch@769ea1ed69
2017-01-23 10:12:21 +00:00
Martijn van Groningen 0c40317ed2 Remove unneeded inject annotations, because they are no longer needed since rest controllers have been de-guiced.
Original commit: elastic/x-pack-elasticsearch@d272ae1a1c
2017-01-23 10:05:09 +01:00
polyfractal ae7446f78e De-guice getRestHandlers() to follow upstream changes
Original commit: elastic/x-pack-elasticsearch@04bd9e688f
2017-01-20 17:13:35 -05:00
David Kyle ecd462bf89 Fix bug updating normalised results (elastic/elasticsearch#765)
The bulk request needed resetting after it was executed otherwise stale documents are persisted repeatedly after they have been updated causing a versioning error

Original commit: elastic/x-pack-elasticsearch@263fa9d25d
2017-01-20 17:33:37 +00:00
Colin Goodheart-Smithe 4c6989212a Gets build to use elasticsearch-extras (elastic/elasticsearch#758)
* Gets build to use elasticsearch-extras

Also adds ci script for building repo on CI servers

To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory

* Adds USE_SSH option to ci script

* iter

Original commit: elastic/x-pack-elasticsearch@ea127dfef0
2017-01-20 15:11:21 +00:00
David Roberts bddfac59ed Minor adjustments to datafeed endpoints (elastic/elasticsearch#761)
1. get_datafeeds_stats -> get_datafeed_stats

2. get_datafeeds now accepts implicit _all

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@1fc0f69ee2
2017-01-20 10:57:06 +00:00
David Kyle 2eb0499454 Remove JobDataDeleterFactory and OldDataDeleter (elastic/elasticsearch#759)
Original commit: elastic/x-pack-elasticsearch@ac5b75eb58
2017-01-20 09:49:24 +00:00