Commit Graph

5159 Commits

Author SHA1 Message Date
David Kyle c84f227857 Set memory usage log message to trace (elastic/elasticsearch#829)
Original commit: elastic/x-pack-elasticsearch@13412cc4cf
2017-01-31 09:54:56 +00:00
David Roberts ab957b6d91 Adjust validation endpoints (elastic/elasticsearch#812)
Changes are:

1. The detector validation endpoint is changed from /_xpack/ml/_validate/detector
   to /_xpack/ml/anomaly_detectors/_validate/detector
2. A new endpoint is added for validating an entire job config:
   /_xpack/ml/anomaly_detectors/_validate

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@7b2031e746
2017-01-30 17:10:22 +00:00
David Kyle 4eab74ce29 Store input fields for anomaly records and influencers (elastic/elasticsearch#799)
* Store input fields for anomaly records and influencers

* Address review comments

* Remove DotNotationReverser

* Remove duplicated constants

* Can’t use the same date for all records as they will have equivalent Ids

Original commit: elastic/x-pack-elasticsearch@40796b5efc
2017-01-30 14:05:18 +00:00
Colin Goodheart-Smithe 79d1a10a86 Mutes DataFeedJobIT test method that uses painless
This needs to be moved to the single-node-tests qa modules since integTests shouldn’t access modules.

Original commit: elastic/x-pack-elasticsearch@289b697eb8
2017-01-30 10:55:59 +00:00
Colin Goodheart-Smithe 618cb2a1a0 Make all action names cluster actions
Original commit: elastic/x-pack-elasticsearch@815d8f0aac
2017-01-30 10:00:23 +00:00
Peter Dyson 81171bb6ff Merge pull request elastic/elasticsearch#4601 from elastic/geekpete-patch-1
[DOCS] [Watcher] Add additional detail around auth and settings

Original commit: elastic/x-pack-elasticsearch@ce22ef71e2
2017-01-30 09:54:54 +10:00
Nik Everett cdfa20b83f Handle moving some classes from reindex to core
Original commit: elastic/x-pack-elasticsearch@cdf98a70f7
2017-01-27 16:08:32 -05:00
Igor Motov 827118e154 Adds support for persistent actions
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle.  See TestPersistentActionPlugin for an example implementing a persistent action.

Original commit: elastic/x-pack-elasticsearch@8ef4103cd6
2017-01-27 11:20:54 -05:00
Martijn van Groningen ff65c38253 [TEST] fixed mocking logic to include id
Original commit: elastic/x-pack-elasticsearch@7b20e92fdc
2017-01-27 17:20:18 +01:00
Martijn van Groningen 2059b91620 Workaround for index request without an id being retried that are tripping an assertion in internal engine. (2)
Original commit: elastic/x-pack-elasticsearch@22d5060deb
2017-01-27 17:07:51 +01:00
Martijn van Groningen ad4218320c Workaround for index request without an id being retried that are tripping an assertion in internal engine.
Original commit: elastic/x-pack-elasticsearch@ba44acc28b
2017-01-27 16:15:00 +01:00
Simon Willnauer 795a8a84ed Remove obsolete transport handlers
Original commit: elastic/x-pack-elasticsearch@9a9ce85b76
2017-01-27 16:05:47 +01:00
David Roberts 64fdb039ab Reduce the controller connect timeout (elastic/elasticsearch#804)
This used to be 60 seconds, dating back to the days when the controller
had to be started manually after starting Elasticsearch.  However, now
Elasticsearch starts it automatically it should already be running when
we try to connect, so the timeout can be much lower.  It just needs to
be long enough to give the C++ process time to create its named pipes.
2 seconds seems reasonable, and matches what we use for autodetect and
normalize.

Original commit: elastic/x-pack-elasticsearch@7300d68482
2017-01-27 14:23:18 +00:00
Zachary Tong 9395ef81b1 Painless DomainSplit tests in new Single-Node QA Module (elastic/elasticsearch#787)
This contains the Painless-based DomainSplit function, generated static maps and basic tests.  Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response


Original commit: elastic/x-pack-elasticsearch@c6c2942e01
2017-01-27 08:52:48 -05:00
Dimitris Athanasiou 91be1e719d Disable stored_fields when possible in ScrollDataExtractor (elastic/elasticsearch#801)
When source fields are not required, stored_fields can be disabled.
This can make the query faster as no stored fields have to be
decompressed. Note that this means no metadata (_id, _index, _type, etc.)
will be returned.

Original commit: elastic/x-pack-elasticsearch@b1ea526d83
2017-01-27 11:38:54 +00:00
Tim Brooks 5f84ddee7c Add doPrivileged blocks for socket connect ops (elastic/elasticsearch#4701)
This is related to elastic/elasticsearch#22116. X-pack opens socket connections for a number
of pieces of functionality (Active Directory support, ssl, email
notification, etc). As SocketPermissions are transitioned out of core,
x-pack will require connect permission. This pull request wraps
operations requiring these permissions in doPrivileged blocks.

Original commit: elastic/x-pack-elasticsearch@1bfee97550
2017-01-26 13:18:36 -06:00
Nik Everett ae994c72d3 Fix compilation in Eclipse (elastic/elasticsearch#4745)
Eclipse compiler is silly....

Original commit: elastic/x-pack-elasticsearch@1c7d19bffe
2017-01-26 12:00:52 -05:00
Suyog Rao 31bda61d92 [Monitoring UI] Add cgroups charts to LS advanced view
* Adds Advanced view for Logstash
* Adds 2 cgroups stats based chart in advanced view

Fixes elastic/elasticsearch#4386

Original commit: elastic/x-pack-elasticsearch@453847bcd3
2017-01-26 08:28:49 -08:00
Dimitris Athanasiou 5790a6f152 Handle shard failures in extractors (elastic/elasticsearch#794)
Even though a search response may return a 200 status code, things could
still have gone wrong. A search response may report shard failures.

The datafeed extractors should check for that and report an extraction
error accordingly.

Closes elastic/elasticsearch#775

Original commit: elastic/x-pack-elasticsearch@5d6d899738
2017-01-26 16:01:43 +00:00
David Kyle efc47c2a6f Remove Usage classes (elastic/elasticsearch#796)
* Delete usage class

* Delete usage reporter

* Remove unused constant

Original commit: elastic/x-pack-elasticsearch@c7a6c457bd
2017-01-26 11:50:08 +00:00
David Kyle db14d89358 Fix checkstyle
Original commit: elastic/x-pack-elasticsearch@05d59da705
2017-01-26 10:05:03 +00:00
David Kyle e3bb7cfea3 Split ml-int index into .ml-audit and .ml-meta (elastic/elasticsearch#752)
* Audit messages in .ml-audit

* Rename ml-int to .ml-meta

* Remove no release comment

* Fix compilation after classes moved to a different package

* Create the Audit, state and meta indices every time a job is created

* Revert change creating the audit index etc when the job is created

* Rename index .ml-audit -> .ml-notifications

Original commit: elastic/x-pack-elasticsearch@95168fa341
2017-01-26 09:44:54 +00:00
Stacey Gammon 48d01dad4c Merge pull request elastic/elasticsearch#4736 from Stacey-Gammon/remove-safe-confirm-references
Switch from safeConfirm to confirmModal

Original commit: elastic/x-pack-elasticsearch@feddbd00a2
2017-01-25 17:45:19 -05:00
Martijn van Groningen 3a36f94a4a When timeout has been reached, check one more time if the job / datafeed status has the expected value.
Decreased wait timeout from 30s to 20s

Original commit: elastic/x-pack-elasticsearch@b46fb0abe3
2017-01-25 23:32:04 +01:00
Simon Willnauer 133591a26f [TEST] Remove dfs_query_and_fetch - this mode is internal only and will be removed
Original commit: elastic/x-pack-elasticsearch@f6bb9ce498
2017-01-25 20:18:27 +01:00
Dimitris Athanasiou 86291c12e2 Handle manual aggregations in datafeeds (elastic/elasticsearch#784)
* Handle manual aggregations in datafeeds

Adds a DataExtractor implementation that runs aggregated searches.

The manual aggregations supported have the following limitations:

- each aggregation can hava 0 or 1 sub-aggregations
- the top aggregation has to be a histogram
- sub-aggregations have to be either terms aggregations or single value
metric aggregations.

The response is converted into flat JSON documents that contain only the
fields of interest and can be parsed without additional context from our
JSON parser. The fields in the JSON documents correspond to the names of the aggregations.

Closes elastic/elasticsearch#680

Original commit: elastic/x-pack-elasticsearch@7dfd2d31e6
2017-01-25 19:13:03 +00:00
Colin Goodheart-Smithe 716f543f7b Adds a new constructor to plugin
The new constructor takes an Environment object. This is needed for migration to X-Pack since the environment instance is built by the XPackPlugin and then passed into the feature plugins.

Original commit: elastic/x-pack-elasticsearch@f25225bc6a
2017-01-25 18:45:04 +00:00
Chris Earle 9fad3cf85c [Watcher] Allow Index Action to set _id (elastic/elasticsearch#4694)
This adds a "doc_id" parameter to the index action itself, which can accept a single ID value. This also allows the payload to set _id (or _doc._id) for each document being indexed in order to support this with bulk index actions.

If doc_id and _id are used together, then this blocks the action.

Original commit: elastic/x-pack-elasticsearch@f186ccceb8
2017-01-25 13:03:26 -05:00
David Roberts 4b366f8ef6 Removing transforms and the SINGLE_LINE input format (elastic/elasticsearch#790)
Most transforms will be replaced with Painless scripts.

The exception is the DateTransform, whose functionality is now simplified
to what existed before the other transforms were added.

The SINGLE_LINE format relied on transforms to extract fields, so has also
been removed, but this is reasonable as it strays into Logstash territory.

Relates elastic/elasticsearch#630

Closes elastic/elasticsearch#39

Original commit: elastic/x-pack-elasticsearch@a593d3e0ad
2017-01-25 15:51:50 +00:00
James Gowdy 99c9d3733f Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@c38b4d4c8b
2017-01-25 14:17:00 +00:00
Alexander Reelsen 9d002430b5 Tests: Remove class only used by tests
The `Integers` class was only used in tests (but lurked around in the src) and is not needed.
Also replaced some lambda calls with their shorter equivalents.

Original commit: elastic/x-pack-elasticsearch@a81a5c33d3
2017-01-24 18:10:05 +01:00
Colin Goodheart-Smithe 603fa47580 Adds an option to disable the ML plugin (elastic/elasticsearch#785)
Adds an `xpack.ml.enabled` node level setting that can be used to enable and disable the plugin. This will be important when we migrate to X-Pack

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@e5c4969a96
2017-01-24 16:14:56 +00:00
Alexander Reelsen 984b1b0dd1 Watcher: Add dedicated acknowledged state (elastic/elasticsearch#4588)
In order to display a better UI page for the watch history, the state
of a throttled watch should be visualized. However, right now there
is no way to differentiate between a time throttled watch and a user
acknowledged watch (or action). This commit introduces a new type in a
throttled result, which in turn is used to set the execution state of
a watch.

Closes elastic/elasticsearch#4531

Original commit: elastic/x-pack-elasticsearch@b86e666e54
2017-01-24 16:33:29 +01:00
Luca Cavanna 5e8dd26d93 adapt to "move es. headers to metadata set in ElasticsearchException and stop returning them as response headers" (elastic/elasticsearch#4693)
Original commit: elastic/x-pack-elasticsearch@91abdf73c8
2017-01-24 16:13:03 +01:00
Martijn van Groningen e9f899e57a Improved datafeed logging for stopping
Original commit: elastic/x-pack-elasticsearch@94bd5d6a00
2017-01-24 16:00:54 +01:00
Martijn van Groningen b636a4b829 Fixed timeout (de-)serialization for start and stop datafeeder and open job apis.
Original commit: elastic/x-pack-elasticsearch@be054db48c
2017-01-24 15:53:54 +01:00
Martijn van Groningen 5684caece0 [TEST] print hot threads when op job fails
Original commit: elastic/x-pack-elasticsearch@8820758495
2017-01-24 15:51:37 +01:00
David Roberts ba7fa6b69d Top level build.gradle should have build task
Original commit: elastic/x-pack-elasticsearch@bc5aa9e5ae
2017-01-24 14:48:03 +00:00
javanna a8bb433aa1 fix compile errors due to upstream changes
Original commit: elastic/x-pack-elasticsearch@40a2561fa3
2017-01-24 13:09:01 +01:00
Martijn van Groningen 8dbaef186e [TEST] use unique job ids to make debugging log files easier
Original commit: elastic/x-pack-elasticsearch@9f04e1b01f
2017-01-24 12:01:21 +01:00
Martijn van Groningen 29451bb7e3 [TEST] select timestamp differently for test documents
Original commit: elastic/x-pack-elasticsearch@679273012c
2017-01-24 11:07:24 +01:00
David Roberts 215410e93f Rename list to filter (elastic/elasticsearch#774)
Part of the endpoint rename Sophie and Steve agreed with Clint

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@6ded117849
2017-01-24 10:01:24 +00:00
Martijn van Groningen a7d1918461 [TEST] added more logging
Original commit: elastic/x-pack-elasticsearch@062a0b41b8
2017-01-23 17:54:14 +01:00
David Roberts d693f8317c Run build as well as check from CI to bundle Kibana plugin (elastic/elasticsearch#773)
The check task bundles the Elasticsearch part of the plugin but not
Kibana

Fixes elastic/elasticsearch#772

Original commit: elastic/x-pack-elasticsearch@ead2ff3686
2017-01-23 15:48:42 +00:00
Simon Willnauer 1998b7ef46 Remove RequestContext from Security (elastic/elasticsearch#4710)
RequestContext is a leftover from when we had no thread context. This
commit removes the last place where it was used and uses the thread context
instead.

Original commit: elastic/x-pack-elasticsearch@50a2bff400
2017-01-23 15:44:40 +01:00
Colin Goodheart-Smithe db0f4c0977 Fixes projects refs in build (elastic/elasticsearch#771)
The build would fail certain task (e.g. `gradle tasks` and `gradle :prelert-legacy:test`) if run from the elasticsearch root directory because the project references in the build were not abolute project paths from the root, they were relative to the `prelert-legacy` project. This change fixes the references so they are all absolute project paths.

Original commit: elastic/x-pack-elasticsearch@e549533834
2017-01-23 14:09:22 +00:00
Simon Willnauer 8651c0ad9f Log exceptions in onFailure instead of bubbling up (elastic/elasticsearch#4709)
Today we wrap the checked IOException in an unchecked exception when
sending back a failure the security transport interceptor. Yet, if that failure handling
in-turn fails due to a broken response channel we should rather log the exception instead
of bubbling it up since it can have unforeseeable side-effects.

Relates to elastic/elasticsearch#4706

* fix line len

Original commit: elastic/x-pack-elasticsearch@03c3826112
2017-01-23 15:08:48 +01:00
David Roberts cd2332730b Move the named pipe no bootstrap test to a separate qa module (elastic/elasticsearch#769)
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch.  This should make merging to
x-pack easier.

Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.

Original commit: elastic/x-pack-elasticsearch@5547f457b6
2017-01-23 12:08:35 +00:00
Dimitris Athanasiou b3b8a7edc9 Restructure packages (elastic/elasticsearch#767)
Restructure packages according to plan described in elastic/elasticsearch#730.
Copying here for completeness.

* Move config classes
* DataCounts/Quantiles/state are process.autodetect.state
* AutodetectProcessManager move to process.autodetect
* lists -> config
* PageParams, QueryPage -> action.util
* StatusReporter -> process.DataCountsReporter
* CountingInputStream -> process
* JobManager -> job
* CppLog* -> process.logging
* DataProcessor -> collapse into implementation
* job.audit -> ml.notifications

Closes elastic/elasticsearch#730

Original commit: elastic/x-pack-elasticsearch@769ea1ed69
2017-01-23 10:12:21 +00:00
Simon Willnauer b6703c1515 Preserve `forceExecution` flag when forking off handler threads after authentication (elastic/elasticsearch#4706)
Today we might get a rejection on a critical operation where `forceExecution=true` but
due to the fact that the forceExecution flag is not passed to the transport interceptor
interface we were not able to preserve this flag when forking off the request after authentication.
This causes serious issues if for instance a replication handler is rejected.

Closes elastic/elasticsearch#4704

Original commit: elastic/x-pack-elasticsearch@f0aad7dede
2017-01-23 11:07:36 +01:00