Commit Graph

969 Commits

Author SHA1 Message Date
Igor Motov 568802bbcd Add Kibana upgrade procedure to Upgrade API (elastic/x-pack-elasticsearch#1498)
Adds upgrade infrastructure and support for Kibana index upgrades.

Original commit: elastic/x-pack-elasticsearch@91f343b00f
2017-06-27 17:53:58 -04:00
Igor Motov ad6b86481e Make Upgrade API license aware (elastic/x-pack-elasticsearch#1401)
Upgrade API should only work with non-trial licenses

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@28fef0feeb
2017-06-27 17:53:58 -04:00
Igor Motov bb034f42b8 Add Upgrade API Index Upgrade Info action (elastic/x-pack-elasticsearch#1264)
Adds a new Upgrade API with the first action, index upgrade info, that returns that list of indices that require upgrade in the current cluster before the cluster can be upgraded to the next major version.

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@761e7d2128
2017-06-27 17:53:58 -04:00
Ryan Ernst 84574c0367 Another fix for security tribe tests when ssl is disabled
Original commit: elastic/x-pack-elasticsearch@db2eca3522
2017-06-27 14:53:12 -07:00
Suyog Rao e8472908c1 [Logstash] Change config management license to Gold (elastic/x-pack-elasticsearch#1843)
* [Logstash] Change management license to Gold

Previously the license type for LS config management was `BASIC`. In order to use the security features in Standard/Gold, we had to bump Logstash as well to Gold license.

relates elastic/x-pack-elasticsearch#1841

Original commit: elastic/x-pack-elasticsearch@29194b2417
2017-06-27 14:21:00 -07:00
Ryan Ernst ad9797393a Fix tribe tests with secure ssl settings
Original commit: elastic/x-pack-elasticsearch@a06ef174b0
2017-06-27 14:01:21 -07:00
Tal Levy 8145b100f1 Introduce the deprecation API (elastic/x-pack-elasticsearch#1833)
Adds REST endpoint and Transport Action for retrieving breaking-changes deprecations that exist in current version. This PR is just the framework for such an API, future checks will be added to the appropriate branches.

Original commit: elastic/x-pack-elasticsearch@990e3468e9
2017-06-27 13:51:45 -07:00
Jason Tedor b710f5906f Register primary context handoff as known handler
This commit registers the primary context handoff handler as known.

relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@f91142a521
2017-06-27 15:18:44 -04:00
Ryan Ernst 9b3fb66394 Settings: Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#1852)
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.

relates elastic/elasticsearch#22475

Original commit: elastic/x-pack-elasticsearch@be5275fa3d
2017-06-27 10:15:12 -07:00
Tal Levy 7c7bf475c1 reflect core change to method signature in sourceAsMap (elastic/x-pack-elasticsearch#1862)
x-pack change for https://github.com/elastic/elasticsearch/pull/25410

Original commit: elastic/x-pack-elasticsearch@908ddd3413
2017-06-27 08:59:29 -07:00
Alexander Reelsen 403cf8eba3 Watcher: Refactoring of TriggeredWatchStore (elastic/x-pack-elasticsearch#1848)
* The TriggeredWatchStore now only has one method to put triggered
  watches
* All code is async in TriggeredWatchStore, locking has been removed
* The dedicated WatchRecord.Fields interface has been removed
* TriggeredWatchTests integration test has been moved to a unit test

Original commit: elastic/x-pack-elasticsearch@bc4b5820fb
2017-06-27 17:47:00 +02:00
Simon Willnauer 9b0639d7f0 Remove usage of `mapper.single_type` from xpack tests (elastic/x-pack-elasticsearch#1853)
`index.mapper.single_type` will be removed in master. While there is still
one usage in the security template that we are working on, this change
will remove the remaining usage.

Original commit: elastic/x-pack-elasticsearch@6e7f63b9e0
2017-06-27 13:29:34 +02:00
Daniel Mitterdorfer 54345e6b8e Mute MlDistributedFailureIT#testFullClusterRestart()
Relates elastic/elasticsearch#25415

Original commit: elastic/x-pack-elasticsearch@01fa0e7601
2017-06-27 12:44:22 +02:00
Daniel Mitterdorfer 10a95c4268 Add AwaitsFix to KnownActionsTests#testAllTransportHandlersAreKnown
Relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@ea16c49af0
2017-06-27 08:58:55 +02:00
Jason Tedor c22494bcb7 Remove path.conf setting
This commit is a response to a change in core removing path.conf as a
valid setting.

Relates elastic/x-pack-elasticsearch#1844

Original commit: elastic/x-pack-elasticsearch@477a7eab71
2017-06-26 15:18:49 -04:00
Alexander Reelsen 0912a85778 Tests: Add test for templatable URLs, that are URL encoded
This is just a workaround at the moment, but allows to use
mustache if you only provide the `url` part of a request,
instead of scheme, port, path, host, etc.

Original commit: elastic/x-pack-elasticsearch@3a4aa26665
2017-06-26 18:24:53 +02:00
David Kyle 2ad9ce4840 [ML] Remove duplicated test classes (elastic/x-pack-elasticsearch#1850)
Original commit: elastic/x-pack-elasticsearch@cb5a1b544a
2017-06-26 16:20:59 +01:00
Dimitris Athanasiou bd06a7b9b4 [ML] Do not write JSON records when metric value is not finite (elastic/x-pack-elasticsearch#1849)
relates elastic/x-pack-elasticsearch#1847

Original commit: elastic/x-pack-elasticsearch@26a69b840f
2017-06-26 14:45:00 +01:00
Alexander Reelsen 3db3cd0f0b Watcher: Reduce logging noise if no local watcher shards exist (elastic/x-pack-elasticsearch#1846)
Every cluster state update resulted in a log message, that watcher
pauses execution. This has been fixed to only log, if there was an
actual state switch from executing to pausing, but do nothing if
there are no local shards anyway.

This will reduce the logging noise in tests a lot.

Original commit: elastic/x-pack-elasticsearch@32ab86610c
2017-06-26 12:49:08 +02:00
David Kyle 232d59b855 [ML] Give kill a chance to stop autodetect before closing input (elastic/x-pack-elasticsearch#1824)
* Give kill a chance to kill the process before closing input

* Remove variable that can be refactored out

Original commit: elastic/x-pack-elasticsearch@42f7a3cece
2017-06-26 10:17:44 +01:00
Colin Goodheart-Smithe 27aa3094f6 Adds GraphExploreRequest as a remote index request (elastic/x-pack-elasticsearch#1836)
The graph API needs to be able to search in remote indices. Although it uses the Search API to perform the search and so doesn’t need to deal with remote indexes directly, the security feature needs to know it can be used with remote indexes so it knows to include remote indices in the list of indices accessible from the API for index level security

Original commit: elastic/x-pack-elasticsearch@e3cd84963e
2017-06-26 09:59:00 +01:00
Tim Vernum a73be456ec [Security] Don't apply local restrictions on remote cluster search (elastic/x-pack-elasticsearch#1609)
This change removes all local security checks against remote cluster names.
Any user is allowed to attempt a cross-cluster search, and it is the responsibility of the remote cluster to authorise the search (or not).

This includes support for remote searches even if you have _no_ local search privileges.

Original commit: elastic/x-pack-elasticsearch@1620c3a8fa
2017-06-26 12:30:31 +10:00
Dimitris Athanasiou 701dc53c2a [ML] Refactor aggregation response handling to make more flexible (elastic/x-pack-elasticsearch#1795)
Currently, aggregated datafeeds construct JSON from the aggregation
response by traversing all nested aggregations. In order to
achieve this, multiple leaf aggregations are not supported. Also,
scenarios it makes it impossible to effectively use pipeline
aggregations as it will not ignore the intermediate bucket
aggregations.

This commit refactors AggregationToJsonProcessor in order to
support the above scenarios. This is achieved by only converting
the fields of interest, that is the job analysis fields.

Original commit: elastic/x-pack-elasticsearch@8b575956ca
2017-06-23 12:36:32 +01:00
Tim Vernum 5c157fcd7b [TESTS] Fix potential for negative sleep (elastic/x-pack-elasticsearch#1818)
Original commit: elastic/x-pack-elasticsearch@cb19908a61
2017-06-23 11:45:09 +10:00
Andy Bristol 855c63dbc7 User/role names can be longer with more characters (elastic/x-pack-elasticsearch#1745)
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:

A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.

Original commit: elastic/x-pack-elasticsearch@f77640f269
2017-06-22 13:05:56 -07:00
Simon Willnauer 2fef2c72eb Simplify SSL server configuratin validation (elastic/x-pack-elasticsearch#1826)
Today we have some hidden complexity related to default configurations
might specify NO_KEY which is in some cases valid for server configuration.
This change removes the leniencey paramenters on the validation methods and removes
obsolet asserts.

Original commit: elastic/x-pack-elasticsearch@17ed4b1d20
2017-06-22 21:48:51 +02:00
David Kyle 12eec0e911 [ML] Wait for job to close in test (elastic/x-pack-elasticsearch#1825)
Original commit: elastic/x-pack-elasticsearch@3dbeed58f3
2017-06-22 17:21:16 +01:00
Alexander Reelsen 404cac7b6b Tests: Disabling another failing OpenLDAP test
Relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@88ee3d3eac
2017-06-22 16:16:50 +02:00
Alexander Reelsen f105cc0755 Tests: Added missing handlers for KnownActionsTest
Added in core in https://github.com/elastic/elasticsearch/pull/24841

Original commit: elastic/x-pack-elasticsearch@c70eb073ac
2017-06-22 15:41:07 +02:00
Alexander Reelsen 2147e2aa4f Tests: Another OpenLDAP tests that needs to be muted for now
Relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@406eee05fe
2017-06-22 15:26:07 +02:00
Alexander Reelsen 217ac5bac5 Tests: Further muting of OpenLDAP dependent tests
Relates elastic/x-pack-elasticsearch#182

Original commit: elastic/x-pack-elasticsearch@10826c4c20
2017-06-22 14:41:40 +02:00
Alexander Reelsen 8666f96b62 Tests: Muting OpenLDAP tests due to network issues
Original commit: elastic/x-pack-elasticsearch@c32d397a20
2017-06-22 14:18:01 +02:00
Dimitris Athanasiou 0399be6406 [ML] Respect requested start time when datafeed is resumed (elastic/x-pack-elasticsearch#1822)
This commit fixes a bug where if a datafeed is resumed with `now`
as the `start` time, the datafeed still pulls all data from last
seen record.

relates elastic/x-pack-elasticsearch#1821

Original commit: elastic/x-pack-elasticsearch@ff066c0e04
2017-06-22 12:13:42 +01:00
Adrien Grand bd973aaadb Upgrade to lucene-7.0.0-snapshot-ad2cb77. (elastic/x-pack-elasticsearch#1820)
Sibling of elastic/elasticsearch#25349.

Original commit: elastic/x-pack-elasticsearch@2ab35a91e5
2017-06-22 12:35:51 +02:00
David Kyle 2eb939842d [ML] More informative logging for condition that prevents job creation (elastic/x-pack-elasticsearch#1811)
Original commit: elastic/x-pack-elasticsearch@fb223d3406
2017-06-22 09:48:48 +01:00
Chris Earle 0c7e802704 [Monitoring] Allow Cluster Alerts to be disabled in LocalExporter (elastic/x-pack-elasticsearch#1808)
You can now disable cluster alerts in the local exporter, which you can do in the HTTP exporter already.

This helps users that mess up their watcher configuration (e.g., disabling scripts) can turn off the feature to avoid log spam.

Original commit: elastic/x-pack-elasticsearch@f2096b553d
2017-06-21 12:46:28 -04:00
Chris Earle 1f59efa6a7 [Monitoring] Cluster Alert should reference cluster_stats in 5.5+ (elastic/x-pack-elasticsearch#1812)
The cluster alert did not get updated when we dropped the logical 'type' for cluster_state in favor of merging it with cluster_stats in 5.5+.

Original commit: elastic/x-pack-elasticsearch@c7105be36f
2017-06-21 12:02:51 -04:00
Simon Willnauer 9e5a8f3df5 Ensure transport impls read SSLConfiguration early (elastic/x-pack-elasticsearch#1797)
Today we parse / construct SSLConfiguration late when client / server
channels are created. This is problematic if we try to read from secure settings
in the future since we need to read all secure settings as soon as the node is
constructed. If we keep on reading late, we will access a closed keystore
since channel creation happens during node startup.

Original commit: elastic/x-pack-elasticsearch@78d6061990
2017-06-21 16:58:20 +02:00
Alexander Reelsen 889ee11c83 Watcher: Do not start, if index meta data version is wrong (elastic/x-pack-elasticsearch#1770)
If the internal index version of an index is not the right one, do
not start watcher.

Also, add the internal index version of 6 to all our index templates.

Original commit: elastic/x-pack-elasticsearch@20b50aa82b
2017-06-21 13:51:36 +02:00
Dimitris Athanasiou c661ee0934 [ML] Flush job even when datafeed retrieves no data (elastic/x-pack-elasticsearch#1786)
In order to detect anomalies due to the absence of data we
need to flush the job even if the datafeed retrieves no data.

relates elastic/x-pack-elasticsearch#1794

Original commit: elastic/x-pack-elasticsearch@645e393976
2017-06-21 10:37:29 +01:00
David Roberts a8e394c3b5 [ML] Fix race condition between multiple job deletion and immediate recreation (elastic/x-pack-elasticsearch#1793)
If multiple job deletion requests were sent in quick succession, there was a
race condition that meant they could both get through the check to enforce
one active deletion request at a time.  Then, if the job was immediately
recreated after the first deletion request returned, the second, still running,
deletion request could interfere with it and delete the aliases that the put
job request created.

This problem can be avoided by using the "ask forgiveness, not permission"
idiom when checking if the job is already being deleted at the beginning of
each deletion request.

Additionally, now even force delete requests will wait for a certain amount
of time for a prior delete request to complete.  This is to avoid the same
race conditions.  However, force delete requests will eventually start an
(unsafe) parallel delete to provide a get-out in case a delete request
completely dies.

relates elastic/x-pack-elasticsearch#1765

Original commit: elastic/x-pack-elasticsearch@b5c8f26a0e
2017-06-21 09:14:51 +01:00
David Kyle 410b210736 [ML] Retry after SearchPhaseExecutionException in ScrollDataExtractor (elastic/x-pack-elasticsearch#1788)
Original commit: elastic/x-pack-elasticsearch@bbe287b9c3
2017-06-21 08:55:09 +01:00
Alexander Reelsen 1e7f61b4c8 Tests: Ensure watcher index templates are installed in REST tests (elastic/x-pack-elasticsearch#1784)
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.

relates elastic/x-pack-elasticsearch#1762

Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
2017-06-20 14:17:36 +02:00
David Roberts 96927cc1b6 [ML] Handle failures in the idiomatic way (elastic/x-pack-elasticsearch#1785)
This commit changes a couple of places where our ExceptionsHelper
class was throwing exceptions to instead return the exceptions.
Then they can be passed to onFailure() methods or thrown depending
on what's appropriate for the caller.  This is the standard Elastic
way of handling failures.

Original commit: elastic/x-pack-elasticsearch@fce07eb075
2017-06-20 13:06:28 +01:00
Alexander Reelsen d6254c9fd3 Watcher: Add proxy support for reporting attachment action (elastic/x-pack-elasticsearch#1781)
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.

Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
2017-06-20 13:49:32 +02:00
Alexander Reelsen 6e6629bd18 Tests: Log HTTP requests/response for external hipchat tests
Relates elastic/x-pack-elasticsearch#1774

Original commit: elastic/x-pack-elasticsearch@bdd86236e2
2017-06-20 10:47:42 +02:00
Jay Modi 3854e0aee7 Validate filename both before and after resolving the path (elastic/x-pack-elasticsearch#1775)
This change ensures we fail early with validation before we even attempt to resolve a path as the
path resolution can fail with invalid characters on some operating systems.

relates elastic/x-pack-elasticsearch#1757

Original commit: elastic/x-pack-elasticsearch@3a4243b6d6
2017-06-19 14:09:48 -06:00
Martijn van Groningen b07a903afa test: make sure analysis-common is also installed as a plugin
Original commit: elastic/x-pack-elasticsearch@3c3141a734
2017-06-19 20:57:47 +02:00
jaymode 7cb0dbe1e6 Revert "Test: use System.lineSeperator instead of `\n`"
The MockTerminal used in tests uses \n always and the reverted commit re-introduced the bug which
had been fixed earlier.

Original commit: elastic/x-pack-elasticsearch@09b93b5565
2017-06-19 12:05:44 -06:00
Dimitris Athanasiou 168b566844 [ML] Allow closing a job that is restoring state (elastic/x-pack-elasticsearch#1746)
This change enables closing a job while it is in
the middle of restoring its state. This is has the
benefit of allowing users to close jobs that due to
relocation are `opened` but they are still restoring
state. It also helps avoiding race conditions in tests.

Part of this change also includes restoring the state
as a separate step from the process creation. This means
we no longer block the job map while the process is
restoring its state.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@1713a4a7c4
2017-06-19 18:39:45 +01:00
Dimitrios Athanasiou c39b35d4a0 [ML] Increase ScrollDataExtractor context duration
In cases where the job is bound on the analytics performance
the datafeed can fail because the scroll expires. This is
commit increases the scroll context duration from 10 to 30 minutes
as a temporary solution that will avoid most cases.

Original commit: elastic/x-pack-elasticsearch@fd277bbaa1
2017-06-19 18:36:34 +01:00
jaymode 0532c9069b Test: use System.lineSeperator instead of `\n`
This changes the native realm migrate tool tests to use the System.lineSeperator instead of `\n`
so that the tests will pass on Windows.

Original commit: elastic/x-pack-elasticsearch@d3f9a71ac4
2017-06-19 10:34:30 -06:00
David Kyle cbceae4a33 [ML] Call initialising constructor of BaseTasksRequest (elastic/x-pack-elasticsearch#1771)
Original commit: elastic/x-pack-elasticsearch@216527c959
2017-06-19 16:40:32 +01:00
David Roberts 7ab6a7e83a [ML] Fix NullPointerException when deleting a job with state "opening" (elastic/x-pack-elasticsearch#1769)
relates elastic/x-pack-elasticsearch#1768

Original commit: elastic/x-pack-elasticsearch@ae1f431316
2017-06-19 16:24:10 +01:00
Alexander Reelsen c1685c70ea Revert "X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)"
This reverts commit elastic/x-pack-elasticsearch@e430691c51.

The setting is going to be applied in core, so that only parts
of these changes need to be applied here (in a seperate PR).

Original commit: elastic/x-pack-elasticsearch@9e178c87fa
2017-06-19 16:13:36 +02:00
Shaunak Kashyap 160fb8ee53 Index logstash_stats.pipelines.events.duration_in_millis (elastic/x-pack-elasticsearch#1752)
Original commit: elastic/x-pack-elasticsearch@c731badb20
2017-06-19 07:12:12 -07:00
Luca Cavanna 6344bff121 Remove (deprecated) support for '+' in index expressions (elastic/x-pack-elasticsearch#1749)
Original commit: elastic/x-pack-elasticsearch@fc3fa2b842
2017-06-19 15:20:02 +02:00
David Kyle 40700f22b1 [ML] Fix NPE by calling super constructor that inits members. (elastic/x-pack-elasticsearch#1763)
Original commit: elastic/x-pack-elasticsearch@ee47ad8322
2017-06-19 14:18:14 +01:00
Alexander Reelsen 22d57ca96e Tests: Fix line endings to match mock terminal line endings
The MockTerminal class always uses "\n" as line endings, so
the tests should not use `System.lineSeparator`, but also
a newline.

relates elastic/x-pack-elasticsearch#1758

Original commit: elastic/x-pack-elasticsearch@364a4e43ad
2017-06-19 13:48:54 +02:00
Dimitris Athanasiou ae70a657dd [ML] Add documentation to rest spec (elastic/x-pack-elasticsearch#1756)
Original commit: elastic/x-pack-elasticsearch@7d7de84a5a
2017-06-19 10:37:17 +01:00
David Roberts 03652e7497 [ML] Kill autodetect on force close and isolated node rejoining (elastic/x-pack-elasticsearch#1742)
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job.  This was not ideal, because then the job
would persist state from the autodetect process that was isolated.  This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.

In order to achieve this a change has also been made to the behaviour of
force-close.  Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request.  However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster.  Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it.  The documentation
has been changed to reflect this.  It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.

relates elastic/x-pack-elasticsearch#1186

Original commit: elastic/x-pack-elasticsearch@578c944371
2017-06-19 10:16:51 +01:00
Alexander Reelsen 44c3c6b992 Watcher: Fix NPE when search input did not contain search body (elastic/x-pack-elasticsearch#1736)
This came up in a forum post. An NPE was raised, when a search input
contained a search that did not contain a body, but just specified
indices or types.

This commit allows for empty bodies, and also makes sure there are
no null pointer exceptions by using empty bytes references otherwise.

In addition a suite scoped integration test was converted to a unit
test.

Original commit: elastic/x-pack-elasticsearch@29be2976fc
2017-06-19 10:07:10 +02:00
Simon Willnauer 35eb70e113 Catch up with renamings in `TcpTransport`
This is a followup from elastic/elasticsearch#25250

Original commit: elastic/x-pack-elasticsearch@325d10b973
2017-06-19 09:29:20 +02:00
Alexander Reelsen 65228e4379 X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)
This introduces a new index setting called xpack.internal.format to
x-pack, which is configured for all of our index templates and set to
"v6". This indicates the version of compatibility of this index.

In addition a setting named index.xpack.version has been removed,
as it was unused.

Watcher does not start, if the watches and the triggered watches
index is not compatible with this setting.

Original commit: elastic/x-pack-elasticsearch@e430691c51
2017-06-19 08:34:43 +02:00
Nik Everett 1559f85c73 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.

Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
2017-06-16 17:19:47 -04:00
Andrew Cholakian 4967d424f6 Insert ephemeral IDs + correct pipeline event stats into the logstash_stats document (elastic/x-pack-elasticsearch#1747)
Before the event stats were mislabeled, so were not being indexed, and
the ephemeral_id was only in the _state document.

Original commit: elastic/x-pack-elasticsearch@ca0ec81aa5
2017-06-16 14:18:28 -05:00
Luca Cavanna f0c026d1d9 [TEST] adapt test: delete index doesn't resolve to aliases anymore (elastic/x-pack-elasticsearch#1735)
Original commit: elastic/x-pack-elasticsearch@49629ccea6
2017-06-16 17:46:45 +02:00
Nik Everett d526461bd2 Add basic full cluster restart tests for x-pack (elastic/x-pack-elasticsearch#1743)
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.

The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.

Relates to elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@8994bec8e7
2017-06-16 11:44:51 -04:00
Jay Modi ed382807c3 Remove unnecessary initialization of the system key (elastic/x-pack-elasticsearch#1734)
This commit removes unnecessary initialization of the system key in tests that no longer make use
of the system key. It also removes the feature usage for the system key in the SecurityFeatureSet.

Original commit: elastic/x-pack-elasticsearch@b9fffe0bd3
2017-06-16 09:24:03 -06:00
Pier-Hugues Pellerin 76857d7bbe Add support for Logstash' ephemeral_id (elastic/x-pack-elasticsearch#1697)
Logstash now has ephemeral id at the instance level and also at the
pipeline level, we need to add them to the logstash monitoring template.

Original commit: elastic/x-pack-elasticsearch@dfac702d59
2017-06-16 11:08:08 -04:00
David Kyle 02da8e7cd9 [ML] Use bulk request to persist model plots (elastic/x-pack-elasticsearch#1714)
* Use bulk request to persist model plots and model size stats

* Revert persisting model size stats in the bulk request

* Refactor results persister

Original commit: elastic/x-pack-elasticsearch@f51297bfc2
2017-06-16 15:18:16 +01:00
Tim Vernum c63d4e306b Reduce logging for reserved realm authc failures (elastic/x-pack-elasticsearch#1711)
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.

However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.

Original commit: elastic/x-pack-elasticsearch@686a98010b
2017-06-16 10:46:07 +10:00
Martijn van Groningen 9dddce2e08 Add analysis-common as test dependency to xpack plugin module.
Original commit: elastic/x-pack-elasticsearch@2472585037
2017-06-15 20:50:31 +02:00
Tim Brooks 7c7e47aa0f Add tool to setup passwords for internal users (elastic/x-pack-elasticsearch#1434)
This is related to elastic/x-pack-elasticsearch#1217. This change introduces a tool
bin/x-pack/setup-passwords that will streamline the setting of
internal user passwords. There are two modes of operation. One mode
called auto, automatically generates passwords and prints them to
the console. The second mode called interactive allows the user to 
enter passwords.

All passwords are changed using the elastic superuser. The elastic
password is the first password to be set.

Original commit: elastic/x-pack-elasticsearch@00974234a2
2017-06-15 10:48:02 -05:00
Jay Modi d920cc7348 Remove signing code completely from master (elastic/x-pack-elasticsearch#1719)
After improving the authorization of scroll requests and backporting to 5.x, we no longer need to
have any signing code in master. This commit removes it.

Original commit: elastic/x-pack-elasticsearch@8b65fd9338
2017-06-15 07:51:39 -06:00
Jim Ferenczi 07fcf75dd9 [Test] Fix DocumentLevelSecurityTests and FieldLevelSecurityTests to work with the new format of the parent-join field
Original commit: elastic/x-pack-elasticsearch@cd9f26887b
2017-06-15 15:40:41 +02:00
David Roberts 291aa27562 [ML] Kill autodetect on error writing headers (elastic/x-pack-elasticsearch#1730)
If an exception occurs while sending the initial setup messages to the autodetect
such that it fails rather than reaching the open state then the autodetect process
needs to be killed to prevent it hogging resources.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@1ee80ed9b0
2017-06-15 14:03:41 +01:00
David Roberts b748da1880 [ML] Prevent time_field and control field name in analysis_config (elastic/x-pack-elasticsearch#1729)
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer.  Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.

Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@e6750a2cda
2017-06-15 13:04:25 +01:00
Boaz Leskes c76bdf61d7 move assertBusy to use CheckException (elastic/x-pack-elasticsearch#1727)
We use assertBusy in many places where the underlying code throw exceptions. Currently we need to wrap those exceptions in a RuntimeException which is ugly.

Companion PR to https://github.com/elastic/elasticsearch/pull/25246

Original commit: elastic/x-pack-elasticsearch@056857273f
2017-06-15 13:24:34 +02:00
Tim Vernum e36f86cf95 Improve upgrade process for reserved users (elastic/x-pack-elasticsearch#1712)
- Don't attempt to upgrade from 2.x
- Attempt up to 10 retries if the migration fails (with increasing back-off between attempts)
- If a cached user is disabled, recheck with the underlying store

The last change is required if the migration takes a long time.
While users are being migrated, they might be marked as disabled, but when the migration is complete they need to be usable immediately.

Original commit: elastic/x-pack-elasticsearch@2621867014
2017-06-15 10:27:23 +10:00
Nik Everett 0970c509bc Remove the assemble task from projects not published (elastic/x-pack-elasticsearch#1721)
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.

Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
2017-06-14 19:57:26 -04:00
Igor Motov 8abf4c3119 Tests: Remove QUERY_AND_FETCH BWC actions from the list of known handlers
Relates to elastic/elasticsearch#25223

Original commit: elastic/x-pack-elasticsearch@2da097d1ea
2017-06-14 19:01:14 -04:00
Simon Willnauer 97693b9357 Add scroll support for cross cluster search (elastic/x-pack-elasticsearch#1706)
Original commit: elastic/x-pack-elasticsearch@eadffa396b
2017-06-14 20:38:58 +02:00
Dimitrios Athanasiou 50de99374c [TEST] Start non ML node in BasicDistributedJobsIT.testMlIndicesNotAvailable
Original commit: elastic/x-pack-elasticsearch@71e63b6ea2
2017-06-14 16:43:57 +01:00
Dimitrios Athanasiou 1eb785d8ee [TEST] Minor refactorings in BasicDistributedJobsIT
Original commit: elastic/x-pack-elasticsearch@eab0fe759e
2017-06-14 16:04:06 +01:00
Dimitrios Athanasiou cd119c488f [TEST] Remove test logging for resolved issue elastic/x-pack-elasticsearch#1048
Original commit: elastic/x-pack-elasticsearch@012d2b53c9
2017-06-14 15:21:20 +01:00
David Kyle 1010f73ae7 [ML] Retry after broken scroll (elastic/x-pack-elasticsearch#1713)
Original commit: elastic/x-pack-elasticsearch@b4fc329c52
2017-06-14 15:04:14 +01:00
Alexander Reelsen 4e085f03b7 Tests: Disable watcher in ServerTransportFilterIntegrationTests
These tests are starting their own nodes and do not use the testing
trigger schedule class.

There are occasional test failure due to a race condition where watcher
is in the process of being started, but cannot be shut down properly,
because starting up was not finished when the shut down was called for.

These filter tests do not rely on watcher, so we can disable them for
now, but we still need to fix a race condition in starting/stopping
watcher.

relates elastic/x-pack-elasticsearch#1422

Original commit: elastic/x-pack-elasticsearch@f13bb7a6fb
2017-06-14 15:00:37 +02:00
Chris Earle bb31c09ed0 [Monitoring] Add Mapping for cfs_quota_micros field (elastic/x-pack-elasticsearch#1710)
This maps the cgroup field, which was unmapped (and now mappings are no longer dynamic).

Original commit: elastic/x-pack-elasticsearch@86b4333ebb
2017-06-14 07:37:15 -04:00
Alexander Reelsen 457af97d1f Watcher: Remove handling of _status field (elastic/x-pack-elasticsearch#1701)
There is no need to handle any _status field in
the 6.0 release from now on, as everything has been
taken care in the upgrade API.

Original commit: elastic/x-pack-elasticsearch@606581f4d7
2017-06-14 10:36:46 +02:00
David Roberts e7af52fb8e [ML] Let core ES stop the native controller when ML is disabled (elastic/x-pack-elasticsearch#1704)
This changes part of the logic that was added in elastic/x-pack-elasticsearch#644 and extended
in elastic/x-pack-elasticsearch#1495 so that when ML is disabled we never try to communicate with
the native controller during node shutdown.

The original reason for needing to communicate with the native controller
when ML is disabled was the problem of elastic/prelert-legacy#803.
However, this was fixed in a better way in elastic/elasticsearch#24579.
Now there is considerable benefit in never talking to the native
controller from the plugin code when ML is disabled, because it means
anyone suffering some obscure problem with ML can disable it without
running the risk of uncovering some other obscure problem with shutdown.

Original commit: elastic/x-pack-elasticsearch@9d329483a7
2017-06-14 08:54:57 +01:00
Alexander Reelsen 4346431156 Tests: Ensure watcher is started in REST tests (elastic/x-pack-elasticsearch#1702)
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.

This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.

Original commit: elastic/x-pack-elasticsearch@fc547d49b4
2017-06-14 08:34:09 +02:00
Ryan Ernst abe217ebc3 Remove uses of Strings.cleanPath (elastic/x-pack-elasticsearch#1709)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25209

Original commit: elastic/x-pack-elasticsearch@4f59f011e9
2017-06-13 21:10:02 -07:00
Suyog Rao fe72991c70 [Logstash] Add new component to x-pack to handle LS features (elastic/x-pack-elasticsearch#1530)
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.

On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.

relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471

Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
2017-06-13 10:30:30 -07:00
Dimitris Athanasiou f2e2ccae01 [ML] Extract parent field when job has text multi-field (elastic/x-pack-elasticsearch#1705)
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.

However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.

Original commit: elastic/x-pack-elasticsearch@f116e89921
2017-06-13 18:00:24 +01:00
Simon Willnauer 65bac10eed Fix API change in SecurityNetty4Transport
Original commit: elastic/x-pack-elasticsearch@f152fb1813
2017-06-13 10:15:26 +02:00
Dimitrios Athanasiou 29811ea1d8 [ML] Write null instead of false for optional boolean that was removed
Original commit: elastic/x-pack-elasticsearch@43f5fc04bd
2017-06-12 17:35:25 +01:00
Dimitrios Athanasiou 1d33fb8b8b [TEST] Add now required mapping for data index in datafeed test
Original commit: elastic/x-pack-elasticsearch@67cf68c5a0
2017-06-12 15:50:16 +01:00
Dimitris Athanasiou 8eb62eac27 [ML] Automate detection of way to extract fields (elastic/x-pack-elasticsearch#1686)
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.

This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.

This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.

relates elastic/x-pack-elasticsearch#1649

Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
2017-06-12 14:56:31 +01:00
Dimitris Athanasiou c9bbc17742 [ML] Make datafeed types optional (elastic/x-pack-elasticsearch#1690)
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.

Original commit: elastic/x-pack-elasticsearch@f61ac01b45
2017-06-12 14:46:53 +01:00
Alexander Reelsen 27b5142de6 Watcher: Fix croneval tool for packaging (elastic/x-pack-elasticsearch#1689)
The croneval script used an old parameter to start up.
This commit removes this parameter, that is used, when a
package is used.

In addition a concrete vagrant test has been added.

relates elastic/x-pack-elasticsearch#1635

Original commit: elastic/x-pack-elasticsearch@ea7b8a08f4
2017-06-12 13:56:26 +02:00
Alexander Reelsen 83a5e022cb Watcher: Reduce logging noise when watcher might be stopped (elastic/x-pack-elasticsearch#1685)
Only log an entry when an actual stop is executed instead of
always logging.

Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.

Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
2017-06-12 10:24:16 +02:00
David Kyle 516696bbec [ML][TEST] Refresh after indexing docs
Original commit: elastic/x-pack-elasticsearch@b64ad1b2c6
2017-06-10 19:12:52 +01:00
Jason Tedor cb62d81bdc Fix compilation for CreateIndexResponse
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.

Original commit: elastic/x-pack-elasticsearch@b078d80cd9
2017-06-09 14:59:55 -04:00
jaymode 741bf85fbf Remove incorrect bwc branch logic from master
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.

Original commit: elastic/x-pack-elasticsearch@9a77269fa6
2017-06-09 10:02:43 -06:00
Alexander Reelsen 709ed7d50e Watcher: Only try to load triggered watches index, if it exists (elastic/x-pack-elasticsearch#1569)
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.

This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.

Original commit: elastic/x-pack-elasticsearch@9f26d557d0
2017-06-09 17:51:23 +02:00
Alexander Reelsen d769ee0813 Watcher: Do not pause watcher if not needed (elastic/x-pack-elasticsearch#1681)
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.

This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.

Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
2017-06-09 17:50:35 +02:00
Alexander Reelsen 7f48337bf6 Watcher: Remove index template configuration via cluster settings (elastic/x-pack-elasticsearch#1680)
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.

As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.

This feature was not exposed in our documentation at all.

Original commit: elastic/x-pack-elasticsearch@32e1769925
2017-06-09 17:50:01 +02:00
Ryan Ernst c1a3f50e19 Convert script uses to use source/id keys (elastic/x-pack-elasticsearch#1670)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25127

Original commit: elastic/x-pack-elasticsearch@e25bd90825
2017-06-09 08:29:36 -07:00
David Kyle 02a241bdf5 [ML] Refresh index after delete (elastic/x-pack-elasticsearch#1675)
Original commit: elastic/x-pack-elasticsearch@fbbcd0343e
2017-06-09 12:27:30 +01:00
David Kyle d64bea14dc [ML] Closing an unknown job should throw resource not found exception (elastic/x-pack-elasticsearch#1673)
Original commit: elastic/x-pack-elasticsearch@c244d2809b
2017-06-09 10:02:51 +01:00
David Kyle d6e92c19da Fix closing/stopping unassigned jobs/datafeeds (elastic/x-pack-elasticsearch#1672)
Original commit: elastic/x-pack-elasticsearch@9f032ac98f
2017-06-09 09:33:36 +01:00
Chris Earle b31c8e2661 [Monitoring] Remove Specific ClusterStateCollector/Resolver and Tests (elastic/x-pack-elasticsearch#1664)
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).

This makes the tests a little more stable and removes an extra network hop during monitoring data collection.

Original commit: elastic/x-pack-elasticsearch@44851d2dd6
2017-06-08 15:55:23 -04:00
Chris Earle 3f5ae2d54f Always use the lower version in resource tests
Original commit: elastic/x-pack-elasticsearch@47aa744acf
2017-06-08 15:39:52 -04:00
jaymode dad15b8d6c Fix branch logic for bwc tests in the same major version
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.

Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
2017-06-08 11:40:45 -06:00
David Roberts f097ff906d Ensure only 1 in-flight request to create each watcher index template (elastic/x-pack-elasticsearch#1660)
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening.  Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates.  After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.

Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650

Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
2017-06-08 10:05:39 +01:00
Tim Vernum 29c11c30f3 Settings filter should rely directly on realm settings (elastic/x-pack-elasticsearch#1669)
The security module had special cases for realm settings that should be filtered,
but since elastic/x-pack-elasticsearch@2c76a13 / elastic/elasticsearch#4311 it's possible to use the existing realm
setting objects to do that.

Original commit: elastic/x-pack-elasticsearch@0651afe987
2017-06-08 18:05:02 +10:00
David Roberts 7aa1114eca [ML] Make get_job_stats robust to missing results indices (elastic/x-pack-elasticsearch#1662)
Although the job stats for jobs with missing results indices are clearly
ruined, it's better to provide zeroes for the missing values and show the
stats for other jobs than to fail the whole request. This means the UI
can continue to function.

relates elastic/x-pack-elasticsearch#1656

Original commit: elastic/x-pack-elasticsearch@a06fa994a5
2017-06-08 08:33:06 +01:00
Jack Conradson d7658bd9a2 Generate Painless Factory for Creating Script Instances. (elastic/x-pack-elasticsearch#1667)
Original commit: elastic/x-pack-elasticsearch@bda1668eec
2017-06-07 16:06:30 -07:00
Chris Earle adc82e7323 make racey test a bit more flexible
Original commit: elastic/x-pack-elasticsearch@6d5c1110bf
2017-06-07 11:03:44 -04:00
David Kyle ba3e258470 [ML] Wait for job deletion if it is in the deleting state (elastic/x-pack-elasticsearch#1651)
* Wait for job deletion if it is in the deleting  state

* Tolerate errors if multiple force delete requests

Original commit: elastic/x-pack-elasticsearch@1f0c9fbb86
2017-06-07 15:41:29 +01:00
David Kyle ae299f633e [ML] Validate initial job settings (elastic/x-pack-elasticsearch#1646)
* [ML] Validate initial job settings

* Add same job creation checks to the validate endpoint

Original commit: elastic/x-pack-elasticsearch@ab76cf9ea2
2017-06-07 09:34:58 +01:00
Alexander Reelsen 87edc4bfdd Watcher: Only delete triggered watch if watch was known (elastic/x-pack-elasticsearch#1562)
When a user executes a watch and specifies it as part of the
execute watch API, no triggered watch is created, as the watch
cannot be picked up anymore (it only leaves for the duration of
the request).

However until now the TriggeredWatchStore was invoked and tried
to delete this non-existing triggered watch, resulting in some
log cluttering.

This commit removes this try to delete a non-existing triggered
watch.

Original commit: elastic/x-pack-elasticsearch@3db125cea2
2017-06-07 09:19:24 +02:00
Chris Earle 4b2d4a1e3b Simplify ClusterStateTests until resolvers are removed to make less racey
Original commit: elastic/x-pack-elasticsearch@bd44bf6a3b
2017-06-07 00:37:27 -04:00
Jason Tedor 4517892951 Remove uses of unnecessary callback interface
This commit removes uses of a callback interface that is unnecessary in
core as we can use java.util.function.Consumer instead.

Relates elastic/x-pack-elasticsearch#1654

Original commit: elastic/x-pack-elasticsearch@21f470e974
2017-06-06 20:50:16 -04:00
Igor Motov 55a8bc87e4 Add missed action to KnownActionsTests (elastic/x-pack-elasticsearch#1633)
Patching KnownActionsTests for now until we can decided what we want to do with it in a long term in elastic/x-pack-elasticsearch#1489

Original commit: elastic/x-pack-elasticsearch@9f9288c0e2
2017-06-06 16:42:00 -04:00
jaymode 48c34ec8ac Update version for signing scrolls after backport
This commit updates the version in security that we use to check a version that needs a signed
scroll id. After backporting we can talk to 5.5 without signing.

Relates elastic/x-pack-elasticsearch#1416

Original commit: elastic/x-pack-elasticsearch@8653fce1e5
2017-06-06 13:55:36 -06:00
Chris Earle e5ee80c292 [Monitoring] Upgrade Indices to remove usage of _type (elastic/x-pack-elasticsearch#1616)
This is just the culmination of all of the minor PRs associated with 1068. It will:

- Drop the `.monitoring-data-N` index
- Drop use of `_type` in all cases (replaced by `doc` and a new `type` field)
- Drop the API version from the template name (e.g., instead of `.monitoring-es-6` we now use `.monitoring-es`).
- Change API version to `-6-` from `-2-`.
- Both exporters handle versioned resources (templates, pipelines, and watches)
- HTTP exporters will optionally (true by default) publish placeholders for the old, `-2` templates.

When this is backported, it will need to:

- Change `index_patterns` to `template` within the templates.
- Downgrade the version requirements for the templates, pipeline, and watches _and_ the HTTP exporter itself (all require 6.0)

This is a companion to the feature branch in X-Pack Kibana elastic/x-pack-kibana/pull/1318 and they need to be merged at the same time.

Original commit: elastic/x-pack-elasticsearch@6031cfffa4
2017-06-06 14:29:52 -04:00
Jay Modi 2d893df7e9 Add better authorization for scroll requests and remove signing (elastic/x-pack-elasticsearch#1416)
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.

Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.

Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
2017-06-06 10:23:18 -06:00
Martijn van Groningen 15022670e2 security: Add FLS & DLS tests for join field
Original commit: elastic/x-pack-elasticsearch@3a35743ccc
2017-06-06 16:44:19 +02:00
Dimitris Athanasiou 3f6e640f90 [ML] Add force delete datafeed action (elastic/x-pack-elasticsearch#1623)
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.

This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.

relates elastic/x-pack-elasticsearch#1533

Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
2017-06-06 13:39:36 +01:00
David Roberts c2575288d8 [ML] Account for the possibility of no ML metadata existing (elastic/x-pack-elasticsearch#1648)
We try to install empty ML metadata as soon as possible after startup
if none exists.  However, this still leaves a short gap when the cluster
is active with no ML metadata.  To avoid problems, functions that use
the ML metadata should treat this situation as equivalent to having
empty ML metadata.

relates elastic/x-pack-elasticsearch#1643

Original commit: elastic/x-pack-elasticsearch@8f0e00cda8
2017-06-06 13:27:23 +01:00
David Roberts 0d2b127fd7 [TEST] Wait for stable cluster to avoid spurious test failures
Original commit: elastic/x-pack-elasticsearch@4c0d9a0fac
2017-06-06 10:23:54 +01:00
Luca Cavanna f09ccbc6cb Adapt indices resolution to new ignoreAliases index option (elastic/x-pack-elasticsearch#1622)
ignoreAliases allows to resolve index expressions against concrete indices only, rather than against indices and aliases. It is used for now only in IndicesAliasesRequest and the indices resolution code in the security plugin needs to be adapted accordingly.

Original commit: elastic/x-pack-elasticsearch@ae964eade9
2017-06-06 11:02:07 +02:00
David Roberts 41ef0b827f [ML] Add a write alias for persisting job results (elastic/x-pack-elasticsearch#1636)
This commit switches over to two index aliases per job: one for reading
and one for writing.  In the future this will allow the addition of a
rollover endpoint for ML results indices.  (Rollover is still not possible
following this change, but the change to make it possible in the future
should not be a breaking change now.)

Relates elastic/x-pack-elasticsearch#1599
relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@d648f4631f
2017-06-06 09:44:11 +01:00
David Kyle ce0315abc4 [ML] Add force delete job option (elastic/x-pack-elasticsearch#1612)
* Add force delete job option

* Can’t kill a process on a 5.4 node

* Address review comments

* Rename KillAutodetectAction -> KillProcessAction

* Review comments

* Cancelling task is superfluous after it has been killed

* Update docs

* Revert "Cancelling task is superfluous after it has been killed"

This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.

* Remove unnecessary TODOs and logic that doesn't alwasys force close

Original commit: elastic/x-pack-elasticsearch@f8c8b38217
2017-06-06 09:41:33 +01:00
Tim Vernum a12b384906 [TEST] Force LDAP connection to close at end of test (elastic/x-pack-elasticsearch#1620)
This test would sometime leak threads.
The "Timer thread for LDAPConnection" is created by the unboundid SDK - closing the connection should force the thread to terminate

Original commit: elastic/x-pack-elasticsearch@bd58a17a59
2017-06-06 17:47:28 +10:00
Alexander Reelsen 376c9be6fa Watcher: Ensure reloading happens based on watch index instead of alias (elastic/x-pack-elasticsearch#1544)
The cluster state listener to decide if watcher should be reloaded was
assuming that no aliases could be used and thus wrongly could trigger
a reload, which could have lead to wrong test results.

During debugging I also added a reason for reloading and fixed another
wrong test assumption.

Also the listener does not rely on previous cluster state, but stores this
in instance variable, as we need to compare with local state and not the
previous cluster state.

Original commit: elastic/x-pack-elasticsearch@582783a66d
2017-06-06 09:39:11 +02:00
Tim Vernum fe37109c3f [DOCS] [Security] Documentation for Role Mapping API (elastic/x-pack-elasticsearch#1474)
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping 
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings 
- Updates to LDAP/AD realms: new setting for "metadata"

Original commit: elastic/x-pack-elasticsearch@6349f665f5
2017-06-06 14:12:31 +10:00
Tim Vernum 6e7102845b [TESTS] run-as in SecurityServerTransportInterceptorTests (elastic/x-pack-elasticsearch#1475)
Changed existing tests to randomly include a separate "authenticating-user" to verify that the behaviours are correct when run-as is used.
Also includes random roles to completeness.

Related to: elastic/x-pack-elasticsearch@637a865 elastic/x-pack-elasticsearch#1391

Original commit: elastic/x-pack-elasticsearch@e4006bc80a
2017-06-06 14:08:14 +10:00
Tim Vernum 98cdc15038 [Security] Support anon-bind without pooling (elastic/x-pack-elasticsearch#1491)
Make LDAP User-Search work with anonymous bind (bind_dn not set) and connection pooling disabled.

Original commit: elastic/x-pack-elasticsearch@b2c7703fb0
2017-06-06 14:07:07 +10:00
David Kyle 5f76bbd58d [ML] Validate that no documents exist with the new job_id (elastic/x-pack-elasticsearch#1624)
* Validate that no documents exist with the new job_id

Original commit: elastic/x-pack-elasticsearch@acdfb7b5a9
2017-06-05 14:11:34 +01:00
David Roberts 955968c53c [ML] Delete one type at a time when deleting model snapshots (elastic/x-pack-elasticsearch#1637)
This avoids log spam about being unable to create new mappings in indices
that are set to only allow one type.  (It doesn't actually have any effect
on the deletion, which was working before despite the failure to create new
mappings for the legacy types referenced by the delete request.)

relates elastic/x-pack-elasticsearch#1634

Original commit: elastic/x-pack-elasticsearch@061ce7acf1
2017-06-05 13:46:24 +01:00
Ryan Ernst 7ee8eccf95 Script: Convert uses of CompiledTemplate to TemplateScript (elastic/x-pack-elasticsearch#1630)
This is the xpack side of elastic/elasticsearch#25032

Original commit: elastic/x-pack-elasticsearch@ba7df4f6ce
2017-06-02 13:41:33 -07:00
Nik Everett a27ded98c6 Test: crank up logging on LicensingTests
We've seen some failures of the LicensingTests that we don't have enough
information to debug:
https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+master+periodic/2220/consoleFull

So this cranks up the logging of the bits we expect to see the failures
in.

Original commit: elastic/x-pack-elasticsearch@d28c1051a4
2017-06-02 11:38:26 -04:00
David Roberts 67ddbf1fac [ML] Don't serialise the detector_index field to cluster state (elastic/x-pack-elasticsearch#1628)
Because:

1. It's pointless, as new detector_index values are assigned when an
   analysis_config is parsed
2. It creates a backwards compatibility issue when upgrading from v5.4

Original commit: elastic/x-pack-elasticsearch@2f61aa457e
2017-06-02 16:38:18 +01:00
Chris Earle 8e76265c26 Always Accumulate Transport Exceptions (elastic/x-pack-elasticsearch#1619)
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.

There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.

Original commit: elastic/x-pack-elasticsearch@9b57321549
2017-06-02 10:01:21 -04:00
David Roberts babe27afe0 [ML] Add a detector_index field to detectors, and use it for updates (elastic/x-pack-elasticsearch#1613)
Detectors now have a field called detector_index.  This is also now the
field that needs to be supplied when updating a detector.  (Previously
it was simply index, which was confusing.)

When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0.  The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future.  This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.

relates elastic/x-pack-elasticsearch#1275

Original commit: elastic/x-pack-elasticsearch@20a660b07b
2017-06-02 10:26:01 +01:00
Colin Goodheart-Smithe 36b8fe9b29 Adds nodes usage action to known actions list (elastic/x-pack-elasticsearch#1111)
* Adds nodes usage action to known actions list

* Adds name to all rest actions

Original commit: elastic/x-pack-elasticsearch@77ae827a66
2017-06-02 08:46:48 +01:00
Tanguy Leroux 261bf8d78d [Test] LocalExporterTests should wait for exporters to terminate in a finally block (elastic/x-pack-elasticsearch#1581)
At the end of the test, LocalExporterTests checks if no more monitoring
 data are exporter by checking multiple times the last time nodes_stats
 documents were exported, stopping after 10 seconds. It does this in a
 @After annotated method but it would be better to do this in a finally
 block. Also, it should search for node_stats documents only if the
 monitoring indices exist and are searchable to avoid some "all shards
 failed" failures.

Original commit: elastic/x-pack-elasticsearch@90ffb4affd
2017-06-02 09:12:49 +02:00