Commit Graph

6234 Commits

Author SHA1 Message Date
Chris Earle 1f59efa6a7 [Monitoring] Cluster Alert should reference cluster_stats in 5.5+ (elastic/x-pack-elasticsearch#1812)
The cluster alert did not get updated when we dropped the logical 'type' for cluster_state in favor of merging it with cluster_stats in 5.5+.

Original commit: elastic/x-pack-elasticsearch@c7105be36f
2017-06-21 12:02:51 -04:00
Simon Willnauer 9e5a8f3df5 Ensure transport impls read SSLConfiguration early (elastic/x-pack-elasticsearch#1797)
Today we parse / construct SSLConfiguration late when client / server
channels are created. This is problematic if we try to read from secure settings
in the future since we need to read all secure settings as soon as the node is
constructed. If we keep on reading late, we will access a closed keystore
since channel creation happens during node startup.

Original commit: elastic/x-pack-elasticsearch@78d6061990
2017-06-21 16:58:20 +02:00
Alexander Reelsen 889ee11c83 Watcher: Do not start, if index meta data version is wrong (elastic/x-pack-elasticsearch#1770)
If the internal index version of an index is not the right one, do
not start watcher.

Also, add the internal index version of 6 to all our index templates.

Original commit: elastic/x-pack-elasticsearch@20b50aa82b
2017-06-21 13:51:36 +02:00
Dimitris Athanasiou c661ee0934 [ML] Flush job even when datafeed retrieves no data (elastic/x-pack-elasticsearch#1786)
In order to detect anomalies due to the absence of data we
need to flush the job even if the datafeed retrieves no data.

relates elastic/x-pack-elasticsearch#1794

Original commit: elastic/x-pack-elasticsearch@645e393976
2017-06-21 10:37:29 +01:00
David Roberts a8e394c3b5 [ML] Fix race condition between multiple job deletion and immediate recreation (elastic/x-pack-elasticsearch#1793)
If multiple job deletion requests were sent in quick succession, there was a
race condition that meant they could both get through the check to enforce
one active deletion request at a time.  Then, if the job was immediately
recreated after the first deletion request returned, the second, still running,
deletion request could interfere with it and delete the aliases that the put
job request created.

This problem can be avoided by using the "ask forgiveness, not permission"
idiom when checking if the job is already being deleted at the beginning of
each deletion request.

Additionally, now even force delete requests will wait for a certain amount
of time for a prior delete request to complete.  This is to avoid the same
race conditions.  However, force delete requests will eventually start an
(unsafe) parallel delete to provide a get-out in case a delete request
completely dies.

relates elastic/x-pack-elasticsearch#1765

Original commit: elastic/x-pack-elasticsearch@b5c8f26a0e
2017-06-21 09:14:51 +01:00
David Kyle 410b210736 [ML] Retry after SearchPhaseExecutionException in ScrollDataExtractor (elastic/x-pack-elasticsearch#1788)
Original commit: elastic/x-pack-elasticsearch@bbe287b9c3
2017-06-21 08:55:09 +01:00
Lisa Cawley 8103a1bf8b [DOCS] Fix broken links (elastic/x-pack-elasticsearch#1799)
Original commit: elastic/x-pack-elasticsearch@7ff7ff1dbc
2017-06-20 17:41:16 -07:00
Lisa Cawley 1b3b7f2944 [DOCS] Update links in X-Pack Reference for Reporting info (elastic/x-pack-elasticsearch#1584)
* [DOCS] Update links to Reporting docs

* [DOCS] Fix incomplete link

Original commit: elastic/x-pack-elasticsearch@c3ccafb191
2017-06-20 15:38:27 -07:00
Lisa Cawley fe0c5cd3dc [DOCS] Update doc build info in README
Original commit: elastic/x-pack-elasticsearch@c93986436c
2017-06-20 11:49:15 -07:00
Alexander Reelsen 1e7f61b4c8 Tests: Ensure watcher index templates are installed in REST tests (elastic/x-pack-elasticsearch#1784)
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.

relates elastic/x-pack-elasticsearch#1762

Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
2017-06-20 14:17:36 +02:00
David Roberts 96927cc1b6 [ML] Handle failures in the idiomatic way (elastic/x-pack-elasticsearch#1785)
This commit changes a couple of places where our ExceptionsHelper
class was throwing exceptions to instead return the exceptions.
Then they can be passed to onFailure() methods or thrown depending
on what's appropriate for the caller.  This is the standard Elastic
way of handling failures.

Original commit: elastic/x-pack-elasticsearch@fce07eb075
2017-06-20 13:06:28 +01:00
Alexander Reelsen d6254c9fd3 Watcher: Add proxy support for reporting attachment action (elastic/x-pack-elasticsearch#1781)
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.

Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
2017-06-20 13:49:32 +02:00
Alexander Reelsen 6e6629bd18 Tests: Log HTTP requests/response for external hipchat tests
Relates elastic/x-pack-elasticsearch#1774

Original commit: elastic/x-pack-elasticsearch@bdd86236e2
2017-06-20 10:47:42 +02:00
Boaz Leskes 27731eb523 remove NOTCONSOLE markers from installing-xes.asciidoc
It can only be used in shells that actually use curl

Original commit: elastic/x-pack-elasticsearch@b6bf9a98e0
2017-06-20 09:59:38 +02:00
lcawley 28bebe48bf [DOCS] Fix links to ML APIs
Original commit: elastic/x-pack-elasticsearch@406c9ace57
2017-06-19 20:28:56 -07:00
Lisa Cawley 0e133ada9e [DOCS]Fix links to ML APIs (elastic/x-pack-elasticsearch#1779)
Original commit: elastic/x-pack-elasticsearch@81be1e49b6
2017-06-19 19:31:39 -07:00
Lisa Cawley b41ff62bcc [DOCS] Update ML APIs for Elasticsearch Reference (elastic/x-pack-elasticsearch#1718)
* [DOCS] Update ML APIs for Elasticsearch Reference

* [DOCS] Add X-Pack icon for ML APIs

* [DOCS] Add role attribute to ML APIs

Original commit: elastic/x-pack-elasticsearch@997ea39759
2017-06-19 18:23:58 -07:00
Lisa Cawley 551f2f6ac2 [DOCS] Create X-Pack installation and introduction for Elasticsearch Ref (elastic/x-pack-elasticsearch#1698)
* [DOCS] Create X-Pack installation and introduction for Elasticsearch Reference

* [DOCS] Address feedback in X-Pack install and intro info

* [DOCS] Add X-Pack setup pages

* [DOCS] Add ML settings to Elasticsearch

* [DOCS] Add table for X-Pack settings

* [DOCS]Add logstash settings

Original commit: elastic/x-pack-elasticsearch@65786cff78
2017-06-19 18:01:52 -07:00
lcawley 68c1c15912 [DOCS] Rename index-all.asciidoc to index.asciidoc
Original commit: elastic/x-pack-elasticsearch@8c8cbdcc2a
2017-06-19 14:25:06 -07:00
Lisa Cawley 59d7484611 [DOCS] Call shared index files to build Elasticsearch Ref (elastic/x-pack-elasticsearch#1776)
* [DOCS] Call shared index files to build Elasticsearch Ref

* [DOCS] Fix path and edit URL for Elasticsearch build

Original commit: elastic/x-pack-elasticsearch@626d6d79d8
2017-06-19 13:33:00 -07:00
Jay Modi 3854e0aee7 Validate filename both before and after resolving the path (elastic/x-pack-elasticsearch#1775)
This change ensures we fail early with validation before we even attempt to resolve a path as the
path resolution can fail with invalid characters on some operating systems.

relates elastic/x-pack-elasticsearch#1757

Original commit: elastic/x-pack-elasticsearch@3a4243b6d6
2017-06-19 14:09:48 -06:00
Martijn van Groningen b07a903afa test: make sure analysis-common is also installed as a plugin
Original commit: elastic/x-pack-elasticsearch@3c3141a734
2017-06-19 20:57:47 +02:00
jaymode 7cb0dbe1e6 Revert "Test: use System.lineSeperator instead of `\n`"
The MockTerminal used in tests uses \n always and the reverted commit re-introduced the bug which
had been fixed earlier.

Original commit: elastic/x-pack-elasticsearch@09b93b5565
2017-06-19 12:05:44 -06:00
Dimitris Athanasiou 168b566844 [ML] Allow closing a job that is restoring state (elastic/x-pack-elasticsearch#1746)
This change enables closing a job while it is in
the middle of restoring its state. This is has the
benefit of allowing users to close jobs that due to
relocation are `opened` but they are still restoring
state. It also helps avoiding race conditions in tests.

Part of this change also includes restoring the state
as a separate step from the process creation. This means
we no longer block the job map while the process is
restoring its state.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@1713a4a7c4
2017-06-19 18:39:45 +01:00
Dimitrios Athanasiou c39b35d4a0 [ML] Increase ScrollDataExtractor context duration
In cases where the job is bound on the analytics performance
the datafeed can fail because the scroll expires. This is
commit increases the scroll context duration from 10 to 30 minutes
as a temporary solution that will avoid most cases.

Original commit: elastic/x-pack-elasticsearch@fd277bbaa1
2017-06-19 18:36:34 +01:00
jaymode 0532c9069b Test: use System.lineSeperator instead of `\n`
This changes the native realm migrate tool tests to use the System.lineSeperator instead of `\n`
so that the tests will pass on Windows.

Original commit: elastic/x-pack-elasticsearch@d3f9a71ac4
2017-06-19 10:34:30 -06:00
David Kyle cbceae4a33 [ML] Call initialising constructor of BaseTasksRequest (elastic/x-pack-elasticsearch#1771)
Original commit: elastic/x-pack-elasticsearch@216527c959
2017-06-19 16:40:32 +01:00
David Roberts 7ab6a7e83a [ML] Fix NullPointerException when deleting a job with state "opening" (elastic/x-pack-elasticsearch#1769)
relates elastic/x-pack-elasticsearch#1768

Original commit: elastic/x-pack-elasticsearch@ae1f431316
2017-06-19 16:24:10 +01:00
Alexander Reelsen c1685c70ea Revert "X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)"
This reverts commit elastic/x-pack-elasticsearch@e430691c51.

The setting is going to be applied in core, so that only parts
of these changes need to be applied here (in a seperate PR).

Original commit: elastic/x-pack-elasticsearch@9e178c87fa
2017-06-19 16:13:36 +02:00
Shaunak Kashyap 160fb8ee53 Index logstash_stats.pipelines.events.duration_in_millis (elastic/x-pack-elasticsearch#1752)
Original commit: elastic/x-pack-elasticsearch@c731badb20
2017-06-19 07:12:12 -07:00
Luca Cavanna 6344bff121 Remove (deprecated) support for '+' in index expressions (elastic/x-pack-elasticsearch#1749)
Original commit: elastic/x-pack-elasticsearch@fc3fa2b842
2017-06-19 15:20:02 +02:00
David Kyle 40700f22b1 [ML] Fix NPE by calling super constructor that inits members. (elastic/x-pack-elasticsearch#1763)
Original commit: elastic/x-pack-elasticsearch@ee47ad8322
2017-06-19 14:18:14 +01:00
Alexander Reelsen 22d57ca96e Tests: Fix line endings to match mock terminal line endings
The MockTerminal class always uses "\n" as line endings, so
the tests should not use `System.lineSeparator`, but also
a newline.

relates elastic/x-pack-elasticsearch#1758

Original commit: elastic/x-pack-elasticsearch@364a4e43ad
2017-06-19 13:48:54 +02:00
Dimitris Athanasiou ae70a657dd [ML] Add documentation to rest spec (elastic/x-pack-elasticsearch#1756)
Original commit: elastic/x-pack-elasticsearch@7d7de84a5a
2017-06-19 10:37:17 +01:00
David Roberts 03652e7497 [ML] Kill autodetect on force close and isolated node rejoining (elastic/x-pack-elasticsearch#1742)
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job.  This was not ideal, because then the job
would persist state from the autodetect process that was isolated.  This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.

In order to achieve this a change has also been made to the behaviour of
force-close.  Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request.  However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster.  Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it.  The documentation
has been changed to reflect this.  It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.

relates elastic/x-pack-elasticsearch#1186

Original commit: elastic/x-pack-elasticsearch@578c944371
2017-06-19 10:16:51 +01:00
Alexander Reelsen 44c3c6b992 Watcher: Fix NPE when search input did not contain search body (elastic/x-pack-elasticsearch#1736)
This came up in a forum post. An NPE was raised, when a search input
contained a search that did not contain a body, but just specified
indices or types.

This commit allows for empty bodies, and also makes sure there are
no null pointer exceptions by using empty bytes references otherwise.

In addition a suite scoped integration test was converted to a unit
test.

Original commit: elastic/x-pack-elasticsearch@29be2976fc
2017-06-19 10:07:10 +02:00
Simon Willnauer 35eb70e113 Catch up with renamings in `TcpTransport`
This is a followup from elastic/elasticsearch#25250

Original commit: elastic/x-pack-elasticsearch@325d10b973
2017-06-19 09:29:20 +02:00
Alexander Reelsen 65228e4379 X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)
This introduces a new index setting called xpack.internal.format to
x-pack, which is configured for all of our index templates and set to
"v6". This indicates the version of compatibility of this index.

In addition a setting named index.xpack.version has been removed,
as it was unused.

Watcher does not start, if the watches and the triggered watches
index is not compatible with this setting.

Original commit: elastic/x-pack-elasticsearch@e430691c51
2017-06-19 08:34:43 +02:00
Lisa Cawley b7d4600c3c [DOCS] Add Anomaly Explorer limitation re scripted fields (elastic/x-pack-elasticsearch#1676)
* [DOCS] Add Anomaly Explorer limitation re scripted fields

* [DOCS] Update Anomaly Explorer limitation re scripted fields

Original commit: elastic/x-pack-elasticsearch@7c9971ada0
2017-06-16 15:44:36 -07:00
Nik Everett 1559f85c73 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.

Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
2017-06-16 17:19:47 -04:00
Andrew Cholakian 4967d424f6 Insert ephemeral IDs + correct pipeline event stats into the logstash_stats document (elastic/x-pack-elasticsearch#1747)
Before the event stats were mislabeled, so were not being indexed, and
the ephemeral_id was only in the _state document.

Original commit: elastic/x-pack-elasticsearch@ca0ec81aa5
2017-06-16 14:18:28 -05:00
Luca Cavanna f0c026d1d9 [TEST] adapt test: delete index doesn't resolve to aliases anymore (elastic/x-pack-elasticsearch#1735)
Original commit: elastic/x-pack-elasticsearch@49629ccea6
2017-06-16 17:46:45 +02:00
Nik Everett d526461bd2 Add basic full cluster restart tests for x-pack (elastic/x-pack-elasticsearch#1743)
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.

The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.

Relates to elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@8994bec8e7
2017-06-16 11:44:51 -04:00
Jay Modi ed382807c3 Remove unnecessary initialization of the system key (elastic/x-pack-elasticsearch#1734)
This commit removes unnecessary initialization of the system key in tests that no longer make use
of the system key. It also removes the feature usage for the system key in the SecurityFeatureSet.

Original commit: elastic/x-pack-elasticsearch@b9fffe0bd3
2017-06-16 09:24:03 -06:00
Pier-Hugues Pellerin 76857d7bbe Add support for Logstash' ephemeral_id (elastic/x-pack-elasticsearch#1697)
Logstash now has ephemeral id at the instance level and also at the
pipeline level, we need to add them to the logstash monitoring template.

Original commit: elastic/x-pack-elasticsearch@dfac702d59
2017-06-16 11:08:08 -04:00
David Kyle 02da8e7cd9 [ML] Use bulk request to persist model plots (elastic/x-pack-elasticsearch#1714)
* Use bulk request to persist model plots and model size stats

* Revert persisting model size stats in the bulk request

* Refactor results persister

Original commit: elastic/x-pack-elasticsearch@f51297bfc2
2017-06-16 15:18:16 +01:00
Tim Vernum c63d4e306b Reduce logging for reserved realm authc failures (elastic/x-pack-elasticsearch#1711)
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.

However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.

Original commit: elastic/x-pack-elasticsearch@686a98010b
2017-06-16 10:46:07 +10:00
Martijn van Groningen 8cc4f29f33 test: make sure analysis-common module is also available on the cluster 1 and cluster2 nodes.
Original commit: elastic/x-pack-elasticsearch@eef5d2b566
2017-06-15 23:25:45 +02:00
jaymode ee3d17adfd Test: fix security rolling upgrade tests that were failing
This commit fixes the default password migration tests that had been failing reproducibly. The
first fix skips tests using the set enabled api when running against a version prior to 5.1.2 as
this api would otherwise trip an assertion that the xcontent builder was not closed. The second
fix is to ensure we set the password field in the user object.

relates elastic/x-pack-elasticsearch#1529
relates elastic/x-pack-elasticsearch#1516

Original commit: elastic/x-pack-elasticsearch@2f9c804309
2017-06-15 14:50:26 -06:00
Martijn van Groningen 863755d2da Make sure that the module are installed too.
Original commit: elastic/x-pack-elasticsearch@f581d0902c
2017-06-15 22:03:18 +02:00