Commit Graph

5285 Commits

Author SHA1 Message Date
Adrien Grand 52a7f956f4 Tests: Disable `ml/index_layout/Test unrelated index`.
Relates elastic/x-pack-elasticsearch#903

Original commit: elastic/x-pack-elasticsearch@097f559d0b
2017-03-30 19:00:50 +02:00
Simon Willnauer 3abe7a2058 Fix compilation issues after elastic/elasticsearch#23788
Original commit: elastic/x-pack-elasticsearch@ea24170b41
2017-03-30 15:00:41 +02:00
Martijn van Groningen f6d8f92d40 [ML] Use volatile boolean to stop blackhole process
Original commit: elastic/x-pack-elasticsearch@3788c7f0cf
2017-03-30 14:54:21 +02:00
Alexander Reelsen 5b2351fad0 Watcher: Improve assertion output in flaky test
The slack tests seem to fail periodically with not output
This commit tries to add some more verbose output by
making the query more broad and take failures into account
to uncover, what happens in this test.

Relates elastic/x-pack-elasticsearch#836

Original commit: elastic/x-pack-elasticsearch@e601b3a0df
2017-03-30 14:50:55 +02:00
Jason Tedor f52760d2ac Revert "Silence UpgradeClusterClientYamlTestSuiteIT"
This reverts commit elastic/x-pack-elasticsearch@e29f94b24c.

Original commit: elastic/x-pack-elasticsearch@d7d27b20ef
2017-03-30 08:39:10 -04:00
Alexander Reelsen dfc21c56be Watcher: Fix BasicSecurityTests.testWatcherMonitorRole
The test tried to create a watches index, that could have been
created by the test infrastructure already (to check if aliases
are supported).

relates elastic/x-pack-elasticsearch#837

Original commit: elastic/x-pack-elasticsearch@29714b2368
2017-03-30 14:03:37 +02:00
David Roberts f49dee89cb [TEST] Adjust expected error message
This was forgotten in elastic/x-pack-elasticsearch@2dbe41caaa

Original commit: elastic/x-pack-elasticsearch@8f7c4c32f4
2017-03-30 11:56:36 +01:00
Dimitris Athanasiou d33bce2ed2 [ML] Add job closing notification (elastic/x-pack-elasticsearch#893)
relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@921c25537a
2017-03-30 11:53:00 +01:00
Dimitris Athanasiou 12fd8e04e5 [ML] Allow marking a model snapshot with retain (elastic/x-pack-elasticsearch#880)
This change adds a retain field to model snapshots.
A user can set retain to true/false via the update model snapshot API.
Model snapshots with retain set to true will not be deleted by
the daily maintenance service regardless of whether they expired.

This allows users to keep always keep certain snapshots around for
potentially reverting to in the future.

relates elastic/x-pack-elasticsearch#758

Original commit: elastic/x-pack-elasticsearch@2283989a33
2017-03-30 11:48:36 +01:00
Dimitris Athanasiou 379b800c9f [ML] Handle usage call when MlMetadata is not yet installed (elastic/x-pack-elasticsearch#890)
Original commit: elastic/x-pack-elasticsearch@495286d4ba
2017-03-30 10:55:36 +01:00
David Roberts 72f5e92972 [ML] Adjust error to make sense for FAILED as well as CLOSED jobs (elastic/x-pack-elasticsearch#888)
Previously trying to perform an operation on a FAILED job would tell
you that it hadn't been opened.  This was very confusing if you knew
very well you'd just opened the job.

Relates elastic/x-pack-elasticsearch#879

Original commit: elastic/x-pack-elasticsearch@2dbe41caaa
2017-03-30 10:33:56 +01:00
David Roberts c335e79508 [TEST] Enable two blacklisted tests that now work
Previously force closing a job required extra privileges.  Following
the full discussion about what privileges should be required.

Original commit: elastic/x-pack-elasticsearch@4d85314b35
2017-03-30 10:26:15 +01:00
Martijn van Groningen 61b4f6480e [TEST] muted test
Original commit: elastic/x-pack-elasticsearch@550b55739e
2017-03-30 11:24:27 +02:00
Martijn van Groningen 9f63dba26c [ML] No need to create an InternalClient instance, because it can be injected.
Original commit: elastic/x-pack-elasticsearch@0cca0cd569
2017-03-30 09:11:46 +02:00
Martijn van Groningen f2654b5872 [ML] Changed job and datafeed lifecycle management
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
   status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
   The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.

Original commit: elastic/x-pack-elasticsearch@37e778b585
2017-03-29 20:35:10 +02:00
Chris Earle 905fa16f81 [Monitoring] Add .monitoring-alerts-2 Index Template (elastic/x-pack-elasticsearch#819)
Add the `.monitoring-alerts-2` index template via the exporter. This
avoids a very common problem where the user wipes out their monitoring
indices manually, which means that the watches would then create an index
with a dynamic mappings.

This adds a mechanism for posting a template that is not associated with a
Resolver (convenient for the forthcoming work _and_ for a future Logstash
index).

Original commit: elastic/x-pack-elasticsearch@a4cfc48191
2017-03-29 13:24:22 -04:00
Igor Motov 4bb9e00dbf Persistent Tasks: Remove unused stopped and removeOnCompletion flags (elastic/x-pack-elasticsearch#853)
The stopped and removeOnCompletion flags are not currently used, this commit removes them for now to simplify things.

Original commit: elastic/x-pack-elasticsearch@c636c2817e
2017-03-29 12:54:44 -04:00
Zachary Tong 0d202d3d36 Bump ChunkedDataExtractor logging down to DEBUG
Related elastic/x-pack-elasticsearch#866

Original commit: elastic/x-pack-elasticsearch@f510101be2
2017-03-29 12:16:36 -04:00
lcawley 8fc5aadec9 [DOCS] Add watcher rest-api docs to x-pack-elasticsearch
Original commit: elastic/x-pack-elasticsearch@a1cb375599
2017-03-29 09:07:55 -07:00
David Roberts 979d232faa [ML] Set jobs to FAILED state if C++ process dies unexpectedly (elastic/x-pack-elasticsearch#876)
Previously a `kill -9` on the `autodetect` process associated with a
job would leave the job in the OPENED state.

Now if the C++ process dies before a request to close the job is made
then the job state is set to FAILED.

For this purpose C++ process death is defined as end-of-file on the
log stream.  (Technically it would be possible to get end-of-file on
the log stream while the C++ process was still running, but this
would also represent an unexpected and undesirable situation.)

Original commit: elastic/x-pack-elasticsearch@2b74c56a79
2017-03-29 16:23:58 +01:00
Robin Clarke de061597fa Fix grammar in users tool help message
This commit fixes some grammar in the additional help provided by the
users tool.

Relates elastic/x-pack-elasticsearch#826

Original commit: elastic/x-pack-elasticsearch@49c86576d6
2017-03-29 10:42:19 -04:00
David Kyle a85b18e70b Don’t write the usePerPartitionNormalization field unless it is set to true (elastic/x-pack-elasticsearch#873)
Original commit: elastic/x-pack-elasticsearch@db7cd6993d
2017-03-29 15:09:14 +01:00
Daniel Mitterdorfer 2355338625 Silence UpgradeClusterClientYamlTestSuiteIT
Relates elastic/x-pack-elasticsearch#870
Relates elastic/x-pack-kibana#690

Original commit: elastic/x-pack-elasticsearch@e29f94b24c
2017-03-29 10:38:28 +02:00
Daniel Mitterdorfer e5b7148fee Silence ExpirationCallbackTests#testPreExpirationScheduleTime()
Relates elastic/x-pack-elasticsearch#869

Original commit: elastic/x-pack-elasticsearch@fcba6847a0
2017-03-29 09:00:35 +02:00
lcawl 84645fc578 [DOCS] Copy watcher docs from x-pack to x-pack-elasticsearch repo (elastic/x-pack-elasticsearch#863)
Original commit: elastic/x-pack-elasticsearch@c4b2bbc975
2017-03-28 14:23:01 -07:00
Dimitrios Athanasiou 991e5d8833 [TEST] Set required euery on delete-by-query request
Original commit: elastic/x-pack-elasticsearch@b9b6234f3d
2017-03-28 17:15:23 +01:00
David Roberts 7eece908b2 [ML] Remove obsolete TODO
It has been decided that filebeat will be used to index ES logs

Original commit: elastic/x-pack-elasticsearch@98f2169429
2017-03-28 16:58:41 +01:00
Martijn van Groningen 2357afd91b [TEST] Use search api to retrieve data counts instead of via get job stats api.
This to avoid to lose data counts when the job gets restarted on another node.
The job stats api returns live data counts, which may not have been persisted to an index,
so getting the data counts via search api will give us a better guarantee that when
the job gets restarted the datacounts are there too. During job restart a get call is being
done to get data counts in the order to initialize the job.

Original commit: elastic/x-pack-elasticsearch@901952da85
2017-03-28 17:10:25 +02:00
David Kyle 42e3eb7ba5 [ML] Use default number of replicas (elastic/x-pack-elasticsearch#861)
Original commit: elastic/x-pack-elasticsearch@7d3d00d196
2017-03-28 15:50:16 +01:00
Tanguy Leroux 642bdb6cd9 [Test] Mute IndicesStatsTests
See https://github.com/elastic/x-pack-elasticsearch/issues/496

Original commit: elastic/x-pack-elasticsearch@80aa60bc27
2017-03-28 16:29:57 +02:00
Colin Goodheart-Smithe b5dadc733b Make Filter actions extend HandledTransportAction (elastic/x-pack-elasticsearch#859)
Previously the GET/PUT/DELETE filters actions were master node actions. This is not necessary since the filters are stored in an index rather than the cluster state. This change makes the actions extend `HandledTransportAction` so they can be run on any node.

The change also makes PutFilterAction.TransportAction use the TransportBulkAction instead of the deprecated TransportIndexAction.

relates elastic/x-pack-elasticsearch#756

Original commit: elastic/x-pack-elasticsearch@c6df04382e
2017-03-28 15:02:39 +01:00
Tanguy Leroux b642ba6351 [Monitoring] Make MonitoringDoc immutable (elastic/x-pack-elasticsearch#785)
This commit makes the MonitoringDoc immutable and removes the type() and id() methods from "resolvers" so that they are not anymore in charge of computing the documents types and ids. Now each MonitoringDoc knows its type and is able to compute its own id if needed.

Original commit: elastic/x-pack-elasticsearch@5161cedcc8
2017-03-28 15:20:08 +02:00
David Roberts 70f621ed7e Remove duplicate actions from transport actions list
Original commit: elastic/x-pack-elasticsearch@da1226ee46
2017-03-28 11:10:44 +01:00
David Kyle fca0feb02d Tidy up (elastic/x-pack-elasticsearch#854)
Original commit: elastic/x-pack-elasticsearch@701407ecf5
2017-03-28 09:56:54 +01:00
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Jason Tedor eac00c6a9d Mark x-pack plugin as having native controller
This commit marks the x-pack plugin as having a native controller. This
is now a requirement in core for any plugin that forks a native process
to display a warning to the user when they install the plugin.

Relates elastic/x-pack-elasticsearch#839

Original commit: elastic/x-pack-elasticsearch@3529250023
2017-03-27 15:53:03 -04:00
Igor Motov 9fc6ce83ee Persistent Tasks: Merge NodePersistentTask and RunningPersistentTask (elastic/x-pack-elasticsearch#842)
Refactors NodePersistentTask and RunningPersistentTask into a single AllocatedPersistentTask. Makes it possible to update Persistent Task Status via AllocatedPersistentTask.

Original commit: elastic/x-pack-elasticsearch@8f59d7b819
2017-03-27 14:21:01 -04:00
David Kyle 9b4d399fc3 [ML] Fix bug deleting job with missing alias (elastic/x-pack-elasticsearch#850)
Original commit: elastic/x-pack-elasticsearch@44fd88a834
2017-03-27 17:47:29 +01:00
Alexander Reelsen f3c4ec8a81 Watcher: Allow multiple email addresses for to/cc/bcc (elastic/x-pack-elasticsearch#661)
All of our code supports configuring email addresses in the
email action not only via a JSON array, but also via a comma
separated value (we also have tests for this). However in one bit
we did not support this, where an email template is rendered to
a concrete email.

This commit fixes the last piece, so that users will be able to
specify comma separated email adresses.

The main use case for this is having an array of email addresses,
that can be joined in mustache with a comma in order to send to
several recipients.

Original commit: elastic/x-pack-elasticsearch@19794ba612
2017-03-27 16:56:43 +02:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
Dimitrios Athanasiou c0a6b6741e [ML] Improve notification message for shard failures in datafeed
relates elastic/x-pack-elasticsearch#804

Original commit: elastic/x-pack-elasticsearch@2114f94937
2017-03-27 13:52:18 +01:00
Dimitris Athanasiou 27a313c07d [ML] Correctly initialise model size stats on job reopen (elastic/x-pack-elasticsearch#822)
This commit ensures that upon reopening a job, the in-memory
model size stats are correctly initialized from the ones
last persisted in the results index.

This fixes the bug that could be seen upon opening a job
that has processed data and immediately calling its _stats
API only to see the model size stats are zero.

In addition, this PR refactors getting the parameters needed to
open an autodetect job:

- Previously, there was a method chaining together multiple
callbacks to the job provider.
- These methods were retrieving data via GETs which is not
going to work with index rollover.

Note, this PR is not eliminating all GETs. More work is needed
to fully support index rollover.

relates elastic/x-pack-elasticsearch#801

Original commit: elastic/x-pack-elasticsearch@1ef1d44b32
2017-03-27 13:17:44 +01:00
Dimitris Athanasiou 9cd21cca4b [ML] Use default number_of_shards for results and state indices (elastic/x-pack-elasticsearch#849)
Like other x-pack features, these indices are best created with
default number_of_shards. Users can adjust the setting by editing
the respective templates.

relates elastic/x-pack-elasticsearch#723

Original commit: elastic/x-pack-elasticsearch@1d688993be
2017-03-27 13:08:19 +01:00
David Roberts ecbfaace38 [TEST] Improve ML security test comments and account for new close assertion
Original commit: elastic/x-pack-elasticsearch@004845bf2f
2017-03-27 10:24:05 +01:00
Alexander Reelsen 374be8b732 Watcher: Support arrays in http response payload (elastic/x-pack-elasticsearch#793)
The xcontent parser was only set to read all data to a map
which did not work, when the returned data was in form of an
array (for example the cat API is doing this, if the response
format is set to JSON).

relates elastic/x-pack-elasticsearch#351

Original commit: elastic/x-pack-elasticsearch@08ad457bf6
2017-03-27 10:22:22 +02:00
Martijn van Groningen 6709b43b97 [ML] Remove redundant comment
Original commit: elastic/x-pack-elasticsearch@7d3d1fdfb5
2017-03-27 09:05:11 +02:00
Ryan Ernst 0255c45f1f Build: Use a deep clone of ES for ci (elastic/x-pack-elasticsearch#840)
WIth the addition of building the backcompat version in elasticsearch, a
shallow clone no longer works, as we need to have eg 5.x branch
available. This change removes the depth argument from cloning
elasticsearch.

Original commit: elastic/x-pack-elasticsearch@3d89fe92ec
2017-03-24 11:50:02 -07:00
Martijn van Groningen ea273ab67e [ML] Made fields volatile or final
Original commit: elastic/x-pack-elasticsearch@c9478e2da4
2017-03-24 19:36:12 +01:00
David Roberts c0445fac4d [TEST] Silence failing ML security tests
Original commit: elastic/x-pack-elasticsearch@af4ed2019d
2017-03-24 17:58:30 +00:00
Zachary Tong 4285335dfb [ML] Add a 'force' parameter to CloseJob and StopDataFeed endpoints (elastic/x-pack-elasticsearch#710)
If forced, the internal RemovePersistentTasks API is invoked instead of going through
ML.  This will remove the task, which should trigger the task framework to do
necessary cleanup.

At that point, the Delete* APIs interpret a missing task as CLOSED/STOPPED,
so they can be removed regardless of the original state.

Original commit: elastic/x-pack-elasticsearch@bff23c7840
2017-03-24 13:10:20 -04:00