Commit Graph

4922 Commits

Author SHA1 Message Date
Zachary Tong a11ddd1e04 Integrate domainSplit function into datafeeds (elastic/elasticsearch#841)
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.

The majority of this PR is actually test-related.  Adds a unit test to check for the injected
script/params.  Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly.  The QA test can
be simplified greatly once the Preview API is added.

Original commit: elastic/x-pack-elasticsearch@c7c35a982c
2017-02-01 10:20:00 -05:00
Martijn van Groningen ec902c4dc3 test: change assertion to be more lenient to platform specifics
Original commit: elastic/x-pack-elasticsearch@2131e7f0c7
2017-02-01 14:49:45 +01:00
Ali Beyad 8cab4fec4b [TEST] adds AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@9e8f6ce047
2017-01-31 22:50:17 -05:00
Ali Beyad 4f4d262323 Adds bwc index for 5.2.0
Original commit: elastic/x-pack-elasticsearch@7b526b241a
2017-01-31 17:13:01 -05:00
Martijn van Groningen 051d8d8fdf Moved start and stop datafeeder apis over the persistent task infrastructure
Original commit: elastic/x-pack-elasticsearch@8e15578fb7
2017-01-31 22:50:00 +01:00
Martijn van Groningen 22282e9d56 cleanup toString() methods
Original commit: elastic/x-pack-elasticsearch@17a10ea68f
2017-01-31 22:35:54 +01:00
Martijn van Groningen ce6dc4a506 Make job stats api task aware.
This will allow the job stats api to redirect the request to node where job is running.

Original commit: elastic/x-pack-elasticsearch@9f1d12dfcb
2017-01-31 22:35:54 +01:00
Jack Conradson 7f0ecc4b30 Change Namespace for Stored Script to Only Use Id (elastic/elasticsearch#4387)
Changes the behavior in x-pack necessary to support the elasticsearch change elastic/elasticsearch#22206.

Original commit: elastic/x-pack-elasticsearch@916e72e263
2017-01-31 13:28:03 -08:00
Martijn van Groningen b07e9bbd07 Fixed AOBE caused by fetching model state when opening a job.
This error only occurred for jobs that have been opened before and persisted model state.

Closes elastic/elasticsearch#836

Original commit: elastic/x-pack-elasticsearch@ad76f4167f
2017-01-31 19:56:24 +01:00
David Roberts 7ff3b707a8 Make renormalization thread-safe (elastic/elasticsearch#840)
Each ScoresUpdater needs its own JobRenormalizedResultsPersister, because
each JobRenormalizedResultsPersister has a single BulkRequest that various
methods update.

Fixes elastic/elasticsearch#838

Original commit: elastic/x-pack-elasticsearch@90f4bbd5a0
2017-01-31 16:51:26 +00:00
Colin Goodheart-Smithe f804bb1917 Removes ensureGreen from PlainlessDomainSplitIT (elastic/elasticsearch#839)
This shouldn’t be needed as the cluster no longer goes red when an index is created.

Original commit: elastic/x-pack-elasticsearch@b554ea9caf
2017-01-31 16:09:25 +00:00
David Kyle 97970b94cd Remove the ignoreDowntime parameter from the _data endpoint (elastic/elasticsearch#834)
The parameter only applies when a job is opened

Original commit: elastic/x-pack-elasticsearch@37b902aa2a
2017-01-31 11:48:58 +00:00
David Roberts 34274a30ed Make transport layer names consistent with corresponding endpoints (elastic/elasticsearch#822)
Closes elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@32aae3e1d9
2017-01-31 11:42:06 +00:00
David Kyle c84f227857 Set memory usage log message to trace (elastic/elasticsearch#829)
Original commit: elastic/x-pack-elasticsearch@13412cc4cf
2017-01-31 09:54:56 +00:00
David Roberts ab957b6d91 Adjust validation endpoints (elastic/elasticsearch#812)
Changes are:

1. The detector validation endpoint is changed from /_xpack/ml/_validate/detector
   to /_xpack/ml/anomaly_detectors/_validate/detector
2. A new endpoint is added for validating an entire job config:
   /_xpack/ml/anomaly_detectors/_validate

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@7b2031e746
2017-01-30 17:10:22 +00:00
David Kyle 4eab74ce29 Store input fields for anomaly records and influencers (elastic/elasticsearch#799)
* Store input fields for anomaly records and influencers

* Address review comments

* Remove DotNotationReverser

* Remove duplicated constants

* Can’t use the same date for all records as they will have equivalent Ids

Original commit: elastic/x-pack-elasticsearch@40796b5efc
2017-01-30 14:05:18 +00:00
Colin Goodheart-Smithe 79d1a10a86 Mutes DataFeedJobIT test method that uses painless
This needs to be moved to the single-node-tests qa modules since integTests shouldn’t access modules.

Original commit: elastic/x-pack-elasticsearch@289b697eb8
2017-01-30 10:55:59 +00:00
Colin Goodheart-Smithe 618cb2a1a0 Make all action names cluster actions
Original commit: elastic/x-pack-elasticsearch@815d8f0aac
2017-01-30 10:00:23 +00:00
Peter Dyson 81171bb6ff Merge pull request elastic/elasticsearch#4601 from elastic/geekpete-patch-1
[DOCS] [Watcher] Add additional detail around auth and settings

Original commit: elastic/x-pack-elasticsearch@ce22ef71e2
2017-01-30 09:54:54 +10:00
Nik Everett cdfa20b83f Handle moving some classes from reindex to core
Original commit: elastic/x-pack-elasticsearch@cdf98a70f7
2017-01-27 16:08:32 -05:00
Igor Motov 827118e154 Adds support for persistent actions
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle.  See TestPersistentActionPlugin for an example implementing a persistent action.

Original commit: elastic/x-pack-elasticsearch@8ef4103cd6
2017-01-27 11:20:54 -05:00
Martijn van Groningen ff65c38253 [TEST] fixed mocking logic to include id
Original commit: elastic/x-pack-elasticsearch@7b20e92fdc
2017-01-27 17:20:18 +01:00
Martijn van Groningen 2059b91620 Workaround for index request without an id being retried that are tripping an assertion in internal engine. (2)
Original commit: elastic/x-pack-elasticsearch@22d5060deb
2017-01-27 17:07:51 +01:00
Martijn van Groningen ad4218320c Workaround for index request without an id being retried that are tripping an assertion in internal engine.
Original commit: elastic/x-pack-elasticsearch@ba44acc28b
2017-01-27 16:15:00 +01:00
Simon Willnauer 795a8a84ed Remove obsolete transport handlers
Original commit: elastic/x-pack-elasticsearch@9a9ce85b76
2017-01-27 16:05:47 +01:00
David Roberts 64fdb039ab Reduce the controller connect timeout (elastic/elasticsearch#804)
This used to be 60 seconds, dating back to the days when the controller
had to be started manually after starting Elasticsearch.  However, now
Elasticsearch starts it automatically it should already be running when
we try to connect, so the timeout can be much lower.  It just needs to
be long enough to give the C++ process time to create its named pipes.
2 seconds seems reasonable, and matches what we use for autodetect and
normalize.

Original commit: elastic/x-pack-elasticsearch@7300d68482
2017-01-27 14:23:18 +00:00
Zachary Tong 9395ef81b1 Painless DomainSplit tests in new Single-Node QA Module (elastic/elasticsearch#787)
This contains the Painless-based DomainSplit function, generated static maps and basic tests.  Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response


Original commit: elastic/x-pack-elasticsearch@c6c2942e01
2017-01-27 08:52:48 -05:00
Dimitris Athanasiou 91be1e719d Disable stored_fields when possible in ScrollDataExtractor (elastic/elasticsearch#801)
When source fields are not required, stored_fields can be disabled.
This can make the query faster as no stored fields have to be
decompressed. Note that this means no metadata (_id, _index, _type, etc.)
will be returned.

Original commit: elastic/x-pack-elasticsearch@b1ea526d83
2017-01-27 11:38:54 +00:00
Tim Brooks 5f84ddee7c Add doPrivileged blocks for socket connect ops (elastic/elasticsearch#4701)
This is related to elastic/elasticsearch#22116. X-pack opens socket connections for a number
of pieces of functionality (Active Directory support, ssl, email
notification, etc). As SocketPermissions are transitioned out of core,
x-pack will require connect permission. This pull request wraps
operations requiring these permissions in doPrivileged blocks.

Original commit: elastic/x-pack-elasticsearch@1bfee97550
2017-01-26 13:18:36 -06:00
Nik Everett ae994c72d3 Fix compilation in Eclipse (elastic/elasticsearch#4745)
Eclipse compiler is silly....

Original commit: elastic/x-pack-elasticsearch@1c7d19bffe
2017-01-26 12:00:52 -05:00
Suyog Rao 31bda61d92 [Monitoring UI] Add cgroups charts to LS advanced view
* Adds Advanced view for Logstash
* Adds 2 cgroups stats based chart in advanced view

Fixes elastic/elasticsearch#4386

Original commit: elastic/x-pack-elasticsearch@453847bcd3
2017-01-26 08:28:49 -08:00
Dimitris Athanasiou 5790a6f152 Handle shard failures in extractors (elastic/elasticsearch#794)
Even though a search response may return a 200 status code, things could
still have gone wrong. A search response may report shard failures.

The datafeed extractors should check for that and report an extraction
error accordingly.

Closes elastic/elasticsearch#775

Original commit: elastic/x-pack-elasticsearch@5d6d899738
2017-01-26 16:01:43 +00:00
David Kyle efc47c2a6f Remove Usage classes (elastic/elasticsearch#796)
* Delete usage class

* Delete usage reporter

* Remove unused constant

Original commit: elastic/x-pack-elasticsearch@c7a6c457bd
2017-01-26 11:50:08 +00:00
David Kyle db14d89358 Fix checkstyle
Original commit: elastic/x-pack-elasticsearch@05d59da705
2017-01-26 10:05:03 +00:00
David Kyle e3bb7cfea3 Split ml-int index into .ml-audit and .ml-meta (elastic/elasticsearch#752)
* Audit messages in .ml-audit

* Rename ml-int to .ml-meta

* Remove no release comment

* Fix compilation after classes moved to a different package

* Create the Audit, state and meta indices every time a job is created

* Revert change creating the audit index etc when the job is created

* Rename index .ml-audit -> .ml-notifications

Original commit: elastic/x-pack-elasticsearch@95168fa341
2017-01-26 09:44:54 +00:00
Stacey Gammon 48d01dad4c Merge pull request elastic/elasticsearch#4736 from Stacey-Gammon/remove-safe-confirm-references
Switch from safeConfirm to confirmModal

Original commit: elastic/x-pack-elasticsearch@feddbd00a2
2017-01-25 17:45:19 -05:00
Martijn van Groningen 3a36f94a4a When timeout has been reached, check one more time if the job / datafeed status has the expected value.
Decreased wait timeout from 30s to 20s

Original commit: elastic/x-pack-elasticsearch@b46fb0abe3
2017-01-25 23:32:04 +01:00
Simon Willnauer 133591a26f [TEST] Remove dfs_query_and_fetch - this mode is internal only and will be removed
Original commit: elastic/x-pack-elasticsearch@f6bb9ce498
2017-01-25 20:18:27 +01:00
Dimitris Athanasiou 86291c12e2 Handle manual aggregations in datafeeds (elastic/elasticsearch#784)
* Handle manual aggregations in datafeeds

Adds a DataExtractor implementation that runs aggregated searches.

The manual aggregations supported have the following limitations:

- each aggregation can hava 0 or 1 sub-aggregations
- the top aggregation has to be a histogram
- sub-aggregations have to be either terms aggregations or single value
metric aggregations.

The response is converted into flat JSON documents that contain only the
fields of interest and can be parsed without additional context from our
JSON parser. The fields in the JSON documents correspond to the names of the aggregations.

Closes elastic/elasticsearch#680

Original commit: elastic/x-pack-elasticsearch@7dfd2d31e6
2017-01-25 19:13:03 +00:00
Colin Goodheart-Smithe 716f543f7b Adds a new constructor to plugin
The new constructor takes an Environment object. This is needed for migration to X-Pack since the environment instance is built by the XPackPlugin and then passed into the feature plugins.

Original commit: elastic/x-pack-elasticsearch@f25225bc6a
2017-01-25 18:45:04 +00:00
Chris Earle 9fad3cf85c [Watcher] Allow Index Action to set _id (elastic/elasticsearch#4694)
This adds a "doc_id" parameter to the index action itself, which can accept a single ID value. This also allows the payload to set _id (or _doc._id) for each document being indexed in order to support this with bulk index actions.

If doc_id and _id are used together, then this blocks the action.

Original commit: elastic/x-pack-elasticsearch@f186ccceb8
2017-01-25 13:03:26 -05:00
David Roberts 4b366f8ef6 Removing transforms and the SINGLE_LINE input format (elastic/elasticsearch#790)
Most transforms will be replaced with Painless scripts.

The exception is the DateTransform, whose functionality is now simplified
to what existed before the other transforms were added.

The SINGLE_LINE format relied on transforms to extract fields, so has also
been removed, but this is reasonable as it strays into Logstash territory.

Relates elastic/elasticsearch#630

Closes elastic/elasticsearch#39

Original commit: elastic/x-pack-elasticsearch@a593d3e0ad
2017-01-25 15:51:50 +00:00
James Gowdy 99c9d3733f Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@c38b4d4c8b
2017-01-25 14:17:00 +00:00
Alexander Reelsen 9d002430b5 Tests: Remove class only used by tests
The `Integers` class was only used in tests (but lurked around in the src) and is not needed.
Also replaced some lambda calls with their shorter equivalents.

Original commit: elastic/x-pack-elasticsearch@a81a5c33d3
2017-01-24 18:10:05 +01:00
Colin Goodheart-Smithe 603fa47580 Adds an option to disable the ML plugin (elastic/elasticsearch#785)
Adds an `xpack.ml.enabled` node level setting that can be used to enable and disable the plugin. This will be important when we migrate to X-Pack

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@e5c4969a96
2017-01-24 16:14:56 +00:00
Alexander Reelsen 984b1b0dd1 Watcher: Add dedicated acknowledged state (elastic/elasticsearch#4588)
In order to display a better UI page for the watch history, the state
of a throttled watch should be visualized. However, right now there
is no way to differentiate between a time throttled watch and a user
acknowledged watch (or action). This commit introduces a new type in a
throttled result, which in turn is used to set the execution state of
a watch.

Closes elastic/elasticsearch#4531

Original commit: elastic/x-pack-elasticsearch@b86e666e54
2017-01-24 16:33:29 +01:00
Luca Cavanna 5e8dd26d93 adapt to "move es. headers to metadata set in ElasticsearchException and stop returning them as response headers" (elastic/elasticsearch#4693)
Original commit: elastic/x-pack-elasticsearch@91abdf73c8
2017-01-24 16:13:03 +01:00
Martijn van Groningen e9f899e57a Improved datafeed logging for stopping
Original commit: elastic/x-pack-elasticsearch@94bd5d6a00
2017-01-24 16:00:54 +01:00
Martijn van Groningen b636a4b829 Fixed timeout (de-)serialization for start and stop datafeeder and open job apis.
Original commit: elastic/x-pack-elasticsearch@be054db48c
2017-01-24 15:53:54 +01:00
Martijn van Groningen 5684caece0 [TEST] print hot threads when op job fails
Original commit: elastic/x-pack-elasticsearch@8820758495
2017-01-24 15:51:37 +01:00