Commit Graph

1010 Commits

Author SHA1 Message Date
Nik Everett 1af9f56834 Remove `IndexResponse.created`
Core removed this in elastic/elasticsearch#25516.

Original commit: elastic/x-pack-elasticsearch@b1e7040fa2
2017-07-07 13:57:43 -04:00
Lisa Cawley 2aa53f253f [DOCS] Update doc links for graph API (elastic/x-pack-elasticsearch#1884)
Original commit: elastic/x-pack-elasticsearch@9eebb6c9a6
2017-07-07 10:11:06 -07:00
Tim Sullivan 66ebdff447 [Monitoring] Email actions for Cluster Alerts (elastic/x-pack-elasticsearch#1879)
* [Monitoring] Email actions for Cluster Alerts

* fix quotations in email fields

* move email vars to transform, and rename for snake_case

* add state to email subject for cluster status alert

* remove types field in kibana_settings search

* simplify email action condition script

* uppercase the state for the email subject

* only append state to email subject if alert is new

* show state in email subject even when alert is resolved

Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
2017-07-07 09:55:45 -07:00
Simon Willnauer c87d9278a6 Add validation for all `transport.profile.*` settings (elastic/x-pack-elasticsearch#1909)
Follow-up from elasticsearch/elastic#25508

Original commit: elastic/x-pack-elasticsearch@fe08e74ccc
2017-07-07 09:41:34 +02:00
Jason Tedor b636dcc366 Upgrade to Netty 4.1.13.Final
This commit updates the SHAs here as core upgraded the Netty dependency
from 4.1.11.Final to 4.1.13.Final.

Relates elastic/x-pack-elasticsearch#1936

Original commit: elastic/x-pack-elasticsearch@d1803fc331
2017-07-06 15:37:17 -04:00
Lisa Cawley 0e322b525d [DOCS] Update doc links for security APIs (elastic/x-pack-elasticsearch#1883)
Original commit: elastic/x-pack-elasticsearch@660410008c
2017-07-06 11:06:03 -07:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Dimitris Athanasiou 4e03c657a3 [ML] Write model_memory_limit with units into the cluster state (elastic/x-pack-elasticsearch#1934)
This is step 2 of elastic/x-pack-elasticsearch#1604

This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.

Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
2017-07-06 15:40:19 +01:00
David Roberts 984d2ca2ba [ML] Ignore unknown fields when parsing ML cluster state (elastic/x-pack-elasticsearch#1924)
ML has two types of custom cluster state:

1. jobs
2. datafeeds

These need to be parsed from JSON in two situations:

1. Create/update of the job/datafeed
2. Restoring cluster state on startup

Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features.  This is because the
parser was very strict.  This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.

This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state.  Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.

relates elastic/x-pack-elasticsearch#1732

Original commit: elastic/x-pack-elasticsearch@754e51d1ec
2017-07-06 13:36:11 +01:00
Dimitris Athanasiou adc6fd5a0f [ML] Parse model_memory_limit both as number and string with units (elastic/x-pack-elasticsearch#1921)
This is the first step for elastic/x-pack-elasticsearch#1604.

Original commit: elastic/x-pack-elasticsearch@70010a216d
2017-07-06 12:29:12 +01:00
Lee Hinman b66560fa85 [TEST] Use better REST endpoints for testing watch actions
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437

It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.

The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.

Original commit: elastic/x-pack-elasticsearch@3be78d9aea
2017-07-05 10:55:25 -06:00
Igor Motov c035adb568 Rename upgrade.* client methods to migration.* (elastic/x-pack-elasticsearch#1881)
This makes client names consistent with REST APIs and makes it simplifies client development.

Original commit: elastic/x-pack-elasticsearch@90913f485b
2017-07-05 09:47:58 -04:00
Dimitrios Athanasiou 3057f7f4b6 [TEST] Fix post data request in post_data.yml
Original commit: elastic/x-pack-elasticsearch@14e9083c02
2017-07-05 12:06:53 +01:00
Dimitris Athanasiou 15f9b1ed9c [ML] Impove mechanism for ignoring maintenance windows (elastic/x-pack-elasticsearch#1914)
Currently, the autodetect process has an `ignoreDowntime`
parameter which, when set to true, results to time being
skipped over to the end of the bucket of the first data
point received. After that, skipping time requires closing
and opening the job. With regard to datafeeds, this does not
work well with real-time requests which use the advance-time
API in order to ensure results are created for data gaps.

This commit improves this functionality by making it more
flexible and less ambiguous.

- flush API now supports skip_time parameter which
sends a control message to the autodetect process
telling it to skip time to a given value
- the flush API now also returns the last_finalized_bucket_end
time which allows clients to resume data searches correctly
- the datafeed start API issues a skip_time request when the
given start time is after the resume point. It then resumes
the search from the last_finalized_bucket_end time.

relates elastic/x-pack-elasticsearch#1913


Original commit: elastic/x-pack-elasticsearch@caa5fe8016
2017-07-05 11:33:42 +01:00
Lisa Cawley 595468a505 [DOCS] Update doc links for watcher APIs (elastic/x-pack-elasticsearch#1880)
Original commit: elastic/x-pack-elasticsearch@47521f1261
2017-07-04 11:16:47 -07:00
Lisa Cawley 04ff94a180 [DOCS] Update doc links for ML APIs (elastic/x-pack-elasticsearch#1882)
Original commit: elastic/x-pack-elasticsearch@cdc45f282d
2017-07-04 11:10:34 -07:00
David Roberts b93890b0df [ML] Add license downgrade warning (elastic/x-pack-elasticsearch#1912)
When applying a license that will cause ML to be disabled, a warning
to this effect is now included in the acknowledgement messages.

relates elastic/x-pack-elasticsearch#1888

Original commit: elastic/x-pack-elasticsearch@e453f20f58
2017-07-04 15:10:18 +01:00
Dimitrios Athanasiou 05a73a32bd [TEST] Correctly verify that aliases were deleted in MlJobIT
This fixes `testDeleteJobAfterMissingAliases` to not fail randomly.
The reason the test was failing is that at some point some aliases
are deleted and the cat-aliases API is called to verify they were
indeed deleted. This was checked by asserting an
index_not_found_exception was thrown by the cat-aliases request.
This was some times working as there were no other aliases. However,
that depends on whether other x-pack features had time to create their
infrastructure. For example, security creates an alias. When other
aliases had the time to be created, the cat-aliases request does not
fail and the test fails.

This commit simply changes the verification that the read/write
aliases were deleted by replacing the cat-aliases request with
two single get-alias requests.

Original commit: elastic/x-pack-elasticsearch@fe2c7b0cb4
2017-07-04 13:14:38 +01:00
Alexander Reelsen fff72256a5 Watcher: Fix wrong logging in reporting attachment parser (elastic/x-pack-elasticsearch#1900)
The logging shows a wrong HTTP response status code from a previous
request. In addition the body now also gets logged, as debugging
is impossible otherwise.

Original commit: elastic/x-pack-elasticsearch@cc998cd587
2017-07-04 13:01:14 +02:00
Christoph Büscher 8d26996afd Remove QueryParseContext (elastic/x-pack-elasticsearch#1895)
This is the x-pack side of elastic/elasticsearch#25486.

Original commit: elastic/x-pack-elasticsearch@c90a3e096b
2017-07-03 17:31:18 +02:00
Simon Willnauer 20f6d66294 Adopt to network settings cleanup in elastic/elasticsearch#25489
Original commit: elastic/x-pack-elasticsearch@364bb260ee
2017-07-02 10:17:30 +02:00
Chris Earle 02c0ad2aad [Monitoring] Reduce NodeStats Collection to required Data (elastic/x-pack-elasticsearch#1240)
This changes from collecting every index statistic to only what we actually want. This should help to reduce the performance impact of the lookup.

Original commit: elastic/x-pack-elasticsearch@80ae20f382
2017-06-30 19:42:44 -04:00
Tim Brooks 7b3b2d5f02 Localhost check: check if addr bound to interface (elastic/x-pack-elasticsearch#1901)
This is related to elastic/x-pack-elasticsearch#1217 and elastic/x-pack-elasticsearch#1896. Right now we are checking if an
incoming address is the loopback address or a special local addres. It
appears that we also need to check if that address is bound to a
network interface to be thorough in our localhost check.

This change mimicks how we check if localhost in `PatternRule`.

Original commit: elastic/x-pack-elasticsearch@a8947d6174
2017-06-30 14:19:49 -05:00
Dimitrios Athanasiou 8264cbf72f [TEST] Stabilise UpdateInterimResultsIT
Depending on the random numbers fed to the analytics,
it is possible that the first planted anomaly ends up
in a different bucket due to the overlapping buckets feature.
Then that may result to a single interim bucket being available
due to overlapping buckets blocking the other interim bucket
from being considered.

I am removing the initial anomaly from the test as it is not useful
and it makes the test unstable.

relates elastic/x-pack-elasticsearch#1897

Original commit: elastic/x-pack-elasticsearch@aca7870708
2017-06-30 17:11:54 +01:00
Christoph Büscher e7e24c453c Reenable SecurityIndexSearcherWrapperIntegrationTests (elastic/x-pack-elasticsearch#1894)
Original commit: elastic/x-pack-elasticsearch@03ff1bf9a5
2017-06-30 16:52:47 +02:00
David Roberts 10c37f0fa4 [TEST] Improve diagnostics for ML interim results test failure
Original commit: elastic/x-pack-elasticsearch@2ccc9d71ae
2017-06-30 11:47:53 +01:00
Jay Modi a9707a461d Use a secure setting for the watcher encryption key (elastic/x-pack-elasticsearch#1831)
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.

Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
2017-06-29 14:58:35 -06:00
Tim Brooks f2cbe20ea0 Remove default passwords from reserved users (elastic/x-pack-elasticsearch#1665)
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.

This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.

Original commit: elastic/x-pack-elasticsearch@e1e101a237
2017-06-29 15:27:57 -05:00
Christoph Büscher 075eda4fc1 Temporarily disable SecurityIndexSearcherWrapperIntegrationTests
Original commit: elastic/x-pack-elasticsearch@bcef6ae8c6
2017-06-29 20:46:30 +02:00
Christoph Büscher 7c6b8ffa36 Adapting to changes in https://github.com/elastic/elasticsearch/pull/25448 (elastic/x-pack-elasticsearch#1887)
Original commit: elastic/x-pack-elasticsearch@5cdf5a2372
2017-06-29 17:10:34 +02:00
Christoph Büscher 3ff5ee3f47 Adapting to merging GetField and SearchHitField to DocumentField (elastic/x-pack-elasticsearch#1860)
Follow up to changes in https://github.com/elastic/elasticsearch/pull/25361

Original commit: elastic/x-pack-elasticsearch@5b1ca009f6
2017-06-29 11:36:20 +02:00
Simon Willnauer 2f10afa40e Use explicit settings for simple bool settings (elastic/x-pack-elasticsearch#1871)
We made the mistake to generate way to many settings in xpack which makes
finding out the right string and where it's defined super difficult. If
we use constants we can just use commandline tools to find where the settings
are defined. This also removes 1.x and 2.x BWC from the enabled settings which should
be removed in 6.x

Original commit: elastic/x-pack-elasticsearch@ec25e6c40c
2017-06-28 22:07:08 +02:00
Tim Brooks f275a3f07b Support bootstrap password when in container (elastic/x-pack-elasticsearch#1832)
This is related to elastic/x-pack-elasticsearch#1217. This commit reads two environment variables on
startup: BOOTSTRAP_PWD and ELASTIC_CONTAINER. If BOOTSTRAP_PWD is
present, ELASTIC_CONTAINER must be set to true. Otherwise a new
bootstrap check will fail.

If ELASTIC_CONTAINER is set to true, the elastic user can be
authenticated with the BOOTSTRAP_PWD variable when its password
has not been explicitly set.

Original commit: elastic/x-pack-elasticsearch@78f53fd232
2017-06-28 12:48:49 -05:00
Chris Earle 231634251f [Monitoring] Enable read-only access for kibana_system user (elastic/x-pack-elasticsearch#1851)
Add read-only `.monitoring-*` to the `kibana_system` user.

Original commit: elastic/x-pack-elasticsearch@0991c062b8
2017-06-28 11:44:04 -04:00
Alexander Reelsen edd5fa4ab4 Watcher: Allow unmet action conditions to reset action status (elastic/x-pack-elasticsearch#1859)
The logic of resetting acknowledgements is only executed, if the watch
wide condition is not met. However, if you dont specify a condition
(which makes it always true), but create a condition in your action
(this might make sense because it allows you to execute a transform and
then execute the condition), then after acking this action, it will
never get be unacked, because the watch wide condition is always met.

relates elastic/x-pack-elasticsearch#1857

Original commit: elastic/x-pack-elasticsearch@95aa402c27
2017-06-28 14:52:26 +02:00
Dimitris Athanasiou 1a076e2eb9 [ML] Improve naming of flush related methods (elastic/x-pack-elasticsearch#1872)
Original commit: elastic/x-pack-elasticsearch@eb3eb80b6e
2017-06-28 13:32:51 +01:00
Ali Beyad a68fb27a23 Upgrade security index to use only one (the default) index type (elastic/x-pack-elasticsearch#1780)
The .security index used several different types to differentiate the
documents added to the index (users, reserved-users, roles, etc).  Since
types are deprecated in 6.x, this commit changes the .security index
access layer to only use a single type and have all documents in the
index be of that single type.  To differentiate documents that may have
the same id (e.g. the same user name and role name), the appropriate
type of the document is prepended to the id.  For example, a user named
"jdoe" will now have the document id "user-jdoe".  

This commit also ensures that any native realm security index operations
that lead to auto creation of the security index first go through the process
of creating the internal security index (.security-v6) and creating the alias
.security to point to the internal index. 

Lastly, anytime the security index is accessed without having been
upgraded, an exception is thrown notifying the user to use the
upgrade API to upgrade the security index.

Original commit: elastic/x-pack-elasticsearch@cc0a474aed
2017-06-27 17:53:58 -04:00
Igor Motov d2bdd99308 Update upgrade API endpoints (elastic/x-pack-elasticsearch#1724)
Switch from /_xpack/_upgrade to /_xpack/migration/assistance and /_xpack/migration/upgrade

Original commit: elastic/x-pack-elasticsearch@b15217135c
2017-06-27 17:53:58 -04:00
Igor Motov 0de8cf47b1 Add REST point names
Relates to elastic/elasticsearch#24169

Original commit: elastic/x-pack-elasticsearch@d18a24a035
2017-06-27 17:53:58 -04:00
Igor Motov 3502a9901b Add watcher upgrade procedure (elastic/x-pack-elasticsearch#1603)
Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@1017d60df4
2017-06-27 17:53:58 -04:00
Igor Motov b564e6e102 Upgrade API should work with TRIAL license as well (elastic/x-pack-elasticsearch#1567)
Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@2ad7e389fb
2017-06-27 17:53:58 -04:00
Igor Motov 568802bbcd Add Kibana upgrade procedure to Upgrade API (elastic/x-pack-elasticsearch#1498)
Adds upgrade infrastructure and support for Kibana index upgrades.

Original commit: elastic/x-pack-elasticsearch@91f343b00f
2017-06-27 17:53:58 -04:00
Igor Motov ad6b86481e Make Upgrade API license aware (elastic/x-pack-elasticsearch#1401)
Upgrade API should only work with non-trial licenses

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@28fef0feeb
2017-06-27 17:53:58 -04:00
Igor Motov bb034f42b8 Add Upgrade API Index Upgrade Info action (elastic/x-pack-elasticsearch#1264)
Adds a new Upgrade API with the first action, index upgrade info, that returns that list of indices that require upgrade in the current cluster before the cluster can be upgraded to the next major version.

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@761e7d2128
2017-06-27 17:53:58 -04:00
Ryan Ernst 84574c0367 Another fix for security tribe tests when ssl is disabled
Original commit: elastic/x-pack-elasticsearch@db2eca3522
2017-06-27 14:53:12 -07:00
Suyog Rao e8472908c1 [Logstash] Change config management license to Gold (elastic/x-pack-elasticsearch#1843)
* [Logstash] Change management license to Gold

Previously the license type for LS config management was `BASIC`. In order to use the security features in Standard/Gold, we had to bump Logstash as well to Gold license.

relates elastic/x-pack-elasticsearch#1841

Original commit: elastic/x-pack-elasticsearch@29194b2417
2017-06-27 14:21:00 -07:00
Ryan Ernst ad9797393a Fix tribe tests with secure ssl settings
Original commit: elastic/x-pack-elasticsearch@a06ef174b0
2017-06-27 14:01:21 -07:00
Tal Levy 8145b100f1 Introduce the deprecation API (elastic/x-pack-elasticsearch#1833)
Adds REST endpoint and Transport Action for retrieving breaking-changes deprecations that exist in current version. This PR is just the framework for such an API, future checks will be added to the appropriate branches.

Original commit: elastic/x-pack-elasticsearch@990e3468e9
2017-06-27 13:51:45 -07:00
Jason Tedor b710f5906f Register primary context handoff as known handler
This commit registers the primary context handoff handler as known.

relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@f91142a521
2017-06-27 15:18:44 -04:00
Ryan Ernst 9b3fb66394 Settings: Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#1852)
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.

relates elastic/elasticsearch#22475

Original commit: elastic/x-pack-elasticsearch@be5275fa3d
2017-06-27 10:15:12 -07:00
Tal Levy 7c7bf475c1 reflect core change to method signature in sourceAsMap (elastic/x-pack-elasticsearch#1862)
x-pack change for https://github.com/elastic/elasticsearch/pull/25410

Original commit: elastic/x-pack-elasticsearch@908ddd3413
2017-06-27 08:59:29 -07:00
Alexander Reelsen 403cf8eba3 Watcher: Refactoring of TriggeredWatchStore (elastic/x-pack-elasticsearch#1848)
* The TriggeredWatchStore now only has one method to put triggered
  watches
* All code is async in TriggeredWatchStore, locking has been removed
* The dedicated WatchRecord.Fields interface has been removed
* TriggeredWatchTests integration test has been moved to a unit test

Original commit: elastic/x-pack-elasticsearch@bc4b5820fb
2017-06-27 17:47:00 +02:00
Simon Willnauer 9b0639d7f0 Remove usage of `mapper.single_type` from xpack tests (elastic/x-pack-elasticsearch#1853)
`index.mapper.single_type` will be removed in master. While there is still
one usage in the security template that we are working on, this change
will remove the remaining usage.

Original commit: elastic/x-pack-elasticsearch@6e7f63b9e0
2017-06-27 13:29:34 +02:00
Daniel Mitterdorfer 54345e6b8e Mute MlDistributedFailureIT#testFullClusterRestart()
Relates elastic/elasticsearch#25415

Original commit: elastic/x-pack-elasticsearch@01fa0e7601
2017-06-27 12:44:22 +02:00
Daniel Mitterdorfer 10a95c4268 Add AwaitsFix to KnownActionsTests#testAllTransportHandlersAreKnown
Relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@ea16c49af0
2017-06-27 08:58:55 +02:00
Jason Tedor c22494bcb7 Remove path.conf setting
This commit is a response to a change in core removing path.conf as a
valid setting.

Relates elastic/x-pack-elasticsearch#1844

Original commit: elastic/x-pack-elasticsearch@477a7eab71
2017-06-26 15:18:49 -04:00
Alexander Reelsen 0912a85778 Tests: Add test for templatable URLs, that are URL encoded
This is just a workaround at the moment, but allows to use
mustache if you only provide the `url` part of a request,
instead of scheme, port, path, host, etc.

Original commit: elastic/x-pack-elasticsearch@3a4aa26665
2017-06-26 18:24:53 +02:00
David Kyle 2ad9ce4840 [ML] Remove duplicated test classes (elastic/x-pack-elasticsearch#1850)
Original commit: elastic/x-pack-elasticsearch@cb5a1b544a
2017-06-26 16:20:59 +01:00
Dimitris Athanasiou bd06a7b9b4 [ML] Do not write JSON records when metric value is not finite (elastic/x-pack-elasticsearch#1849)
relates elastic/x-pack-elasticsearch#1847

Original commit: elastic/x-pack-elasticsearch@26a69b840f
2017-06-26 14:45:00 +01:00
Alexander Reelsen 3db3cd0f0b Watcher: Reduce logging noise if no local watcher shards exist (elastic/x-pack-elasticsearch#1846)
Every cluster state update resulted in a log message, that watcher
pauses execution. This has been fixed to only log, if there was an
actual state switch from executing to pausing, but do nothing if
there are no local shards anyway.

This will reduce the logging noise in tests a lot.

Original commit: elastic/x-pack-elasticsearch@32ab86610c
2017-06-26 12:49:08 +02:00
David Kyle 232d59b855 [ML] Give kill a chance to stop autodetect before closing input (elastic/x-pack-elasticsearch#1824)
* Give kill a chance to kill the process before closing input

* Remove variable that can be refactored out

Original commit: elastic/x-pack-elasticsearch@42f7a3cece
2017-06-26 10:17:44 +01:00
Colin Goodheart-Smithe 27aa3094f6 Adds GraphExploreRequest as a remote index request (elastic/x-pack-elasticsearch#1836)
The graph API needs to be able to search in remote indices. Although it uses the Search API to perform the search and so doesn’t need to deal with remote indexes directly, the security feature needs to know it can be used with remote indexes so it knows to include remote indices in the list of indices accessible from the API for index level security

Original commit: elastic/x-pack-elasticsearch@e3cd84963e
2017-06-26 09:59:00 +01:00
Tim Vernum a73be456ec [Security] Don't apply local restrictions on remote cluster search (elastic/x-pack-elasticsearch#1609)
This change removes all local security checks against remote cluster names.
Any user is allowed to attempt a cross-cluster search, and it is the responsibility of the remote cluster to authorise the search (or not).

This includes support for remote searches even if you have _no_ local search privileges.

Original commit: elastic/x-pack-elasticsearch@1620c3a8fa
2017-06-26 12:30:31 +10:00
Dimitris Athanasiou 701dc53c2a [ML] Refactor aggregation response handling to make more flexible (elastic/x-pack-elasticsearch#1795)
Currently, aggregated datafeeds construct JSON from the aggregation
response by traversing all nested aggregations. In order to
achieve this, multiple leaf aggregations are not supported. Also,
scenarios it makes it impossible to effectively use pipeline
aggregations as it will not ignore the intermediate bucket
aggregations.

This commit refactors AggregationToJsonProcessor in order to
support the above scenarios. This is achieved by only converting
the fields of interest, that is the job analysis fields.

Original commit: elastic/x-pack-elasticsearch@8b575956ca
2017-06-23 12:36:32 +01:00
Tim Vernum 5c157fcd7b [TESTS] Fix potential for negative sleep (elastic/x-pack-elasticsearch#1818)
Original commit: elastic/x-pack-elasticsearch@cb19908a61
2017-06-23 11:45:09 +10:00
Andy Bristol 855c63dbc7 User/role names can be longer with more characters (elastic/x-pack-elasticsearch#1745)
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:

A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.

Original commit: elastic/x-pack-elasticsearch@f77640f269
2017-06-22 13:05:56 -07:00
Simon Willnauer 2fef2c72eb Simplify SSL server configuratin validation (elastic/x-pack-elasticsearch#1826)
Today we have some hidden complexity related to default configurations
might specify NO_KEY which is in some cases valid for server configuration.
This change removes the leniencey paramenters on the validation methods and removes
obsolet asserts.

Original commit: elastic/x-pack-elasticsearch@17ed4b1d20
2017-06-22 21:48:51 +02:00
David Kyle 12eec0e911 [ML] Wait for job to close in test (elastic/x-pack-elasticsearch#1825)
Original commit: elastic/x-pack-elasticsearch@3dbeed58f3
2017-06-22 17:21:16 +01:00
Alexander Reelsen 404cac7b6b Tests: Disabling another failing OpenLDAP test
Relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@88ee3d3eac
2017-06-22 16:16:50 +02:00
Alexander Reelsen f105cc0755 Tests: Added missing handlers for KnownActionsTest
Added in core in https://github.com/elastic/elasticsearch/pull/24841

Original commit: elastic/x-pack-elasticsearch@c70eb073ac
2017-06-22 15:41:07 +02:00
Alexander Reelsen 2147e2aa4f Tests: Another OpenLDAP tests that needs to be muted for now
Relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@406eee05fe
2017-06-22 15:26:07 +02:00
Alexander Reelsen 217ac5bac5 Tests: Further muting of OpenLDAP dependent tests
Relates elastic/x-pack-elasticsearch#182

Original commit: elastic/x-pack-elasticsearch@10826c4c20
2017-06-22 14:41:40 +02:00
Alexander Reelsen 8666f96b62 Tests: Muting OpenLDAP tests due to network issues
Original commit: elastic/x-pack-elasticsearch@c32d397a20
2017-06-22 14:18:01 +02:00
Dimitris Athanasiou 0399be6406 [ML] Respect requested start time when datafeed is resumed (elastic/x-pack-elasticsearch#1822)
This commit fixes a bug where if a datafeed is resumed with `now`
as the `start` time, the datafeed still pulls all data from last
seen record.

relates elastic/x-pack-elasticsearch#1821

Original commit: elastic/x-pack-elasticsearch@ff066c0e04
2017-06-22 12:13:42 +01:00
Adrien Grand bd973aaadb Upgrade to lucene-7.0.0-snapshot-ad2cb77. (elastic/x-pack-elasticsearch#1820)
Sibling of elastic/elasticsearch#25349.

Original commit: elastic/x-pack-elasticsearch@2ab35a91e5
2017-06-22 12:35:51 +02:00
David Kyle 2eb939842d [ML] More informative logging for condition that prevents job creation (elastic/x-pack-elasticsearch#1811)
Original commit: elastic/x-pack-elasticsearch@fb223d3406
2017-06-22 09:48:48 +01:00
Chris Earle 0c7e802704 [Monitoring] Allow Cluster Alerts to be disabled in LocalExporter (elastic/x-pack-elasticsearch#1808)
You can now disable cluster alerts in the local exporter, which you can do in the HTTP exporter already.

This helps users that mess up their watcher configuration (e.g., disabling scripts) can turn off the feature to avoid log spam.

Original commit: elastic/x-pack-elasticsearch@f2096b553d
2017-06-21 12:46:28 -04:00
Chris Earle 1f59efa6a7 [Monitoring] Cluster Alert should reference cluster_stats in 5.5+ (elastic/x-pack-elasticsearch#1812)
The cluster alert did not get updated when we dropped the logical 'type' for cluster_state in favor of merging it with cluster_stats in 5.5+.

Original commit: elastic/x-pack-elasticsearch@c7105be36f
2017-06-21 12:02:51 -04:00
Simon Willnauer 9e5a8f3df5 Ensure transport impls read SSLConfiguration early (elastic/x-pack-elasticsearch#1797)
Today we parse / construct SSLConfiguration late when client / server
channels are created. This is problematic if we try to read from secure settings
in the future since we need to read all secure settings as soon as the node is
constructed. If we keep on reading late, we will access a closed keystore
since channel creation happens during node startup.

Original commit: elastic/x-pack-elasticsearch@78d6061990
2017-06-21 16:58:20 +02:00
Alexander Reelsen 889ee11c83 Watcher: Do not start, if index meta data version is wrong (elastic/x-pack-elasticsearch#1770)
If the internal index version of an index is not the right one, do
not start watcher.

Also, add the internal index version of 6 to all our index templates.

Original commit: elastic/x-pack-elasticsearch@20b50aa82b
2017-06-21 13:51:36 +02:00
Dimitris Athanasiou c661ee0934 [ML] Flush job even when datafeed retrieves no data (elastic/x-pack-elasticsearch#1786)
In order to detect anomalies due to the absence of data we
need to flush the job even if the datafeed retrieves no data.

relates elastic/x-pack-elasticsearch#1794

Original commit: elastic/x-pack-elasticsearch@645e393976
2017-06-21 10:37:29 +01:00
David Roberts a8e394c3b5 [ML] Fix race condition between multiple job deletion and immediate recreation (elastic/x-pack-elasticsearch#1793)
If multiple job deletion requests were sent in quick succession, there was a
race condition that meant they could both get through the check to enforce
one active deletion request at a time.  Then, if the job was immediately
recreated after the first deletion request returned, the second, still running,
deletion request could interfere with it and delete the aliases that the put
job request created.

This problem can be avoided by using the "ask forgiveness, not permission"
idiom when checking if the job is already being deleted at the beginning of
each deletion request.

Additionally, now even force delete requests will wait for a certain amount
of time for a prior delete request to complete.  This is to avoid the same
race conditions.  However, force delete requests will eventually start an
(unsafe) parallel delete to provide a get-out in case a delete request
completely dies.

relates elastic/x-pack-elasticsearch#1765

Original commit: elastic/x-pack-elasticsearch@b5c8f26a0e
2017-06-21 09:14:51 +01:00
David Kyle 410b210736 [ML] Retry after SearchPhaseExecutionException in ScrollDataExtractor (elastic/x-pack-elasticsearch#1788)
Original commit: elastic/x-pack-elasticsearch@bbe287b9c3
2017-06-21 08:55:09 +01:00
Alexander Reelsen 1e7f61b4c8 Tests: Ensure watcher index templates are installed in REST tests (elastic/x-pack-elasticsearch#1784)
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.

relates elastic/x-pack-elasticsearch#1762

Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
2017-06-20 14:17:36 +02:00
David Roberts 96927cc1b6 [ML] Handle failures in the idiomatic way (elastic/x-pack-elasticsearch#1785)
This commit changes a couple of places where our ExceptionsHelper
class was throwing exceptions to instead return the exceptions.
Then they can be passed to onFailure() methods or thrown depending
on what's appropriate for the caller.  This is the standard Elastic
way of handling failures.

Original commit: elastic/x-pack-elasticsearch@fce07eb075
2017-06-20 13:06:28 +01:00
Alexander Reelsen d6254c9fd3 Watcher: Add proxy support for reporting attachment action (elastic/x-pack-elasticsearch#1781)
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.

Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
2017-06-20 13:49:32 +02:00
Alexander Reelsen 6e6629bd18 Tests: Log HTTP requests/response for external hipchat tests
Relates elastic/x-pack-elasticsearch#1774

Original commit: elastic/x-pack-elasticsearch@bdd86236e2
2017-06-20 10:47:42 +02:00
Jay Modi 3854e0aee7 Validate filename both before and after resolving the path (elastic/x-pack-elasticsearch#1775)
This change ensures we fail early with validation before we even attempt to resolve a path as the
path resolution can fail with invalid characters on some operating systems.

relates elastic/x-pack-elasticsearch#1757

Original commit: elastic/x-pack-elasticsearch@3a4243b6d6
2017-06-19 14:09:48 -06:00
Martijn van Groningen b07a903afa test: make sure analysis-common is also installed as a plugin
Original commit: elastic/x-pack-elasticsearch@3c3141a734
2017-06-19 20:57:47 +02:00
jaymode 7cb0dbe1e6 Revert "Test: use System.lineSeperator instead of `\n`"
The MockTerminal used in tests uses \n always and the reverted commit re-introduced the bug which
had been fixed earlier.

Original commit: elastic/x-pack-elasticsearch@09b93b5565
2017-06-19 12:05:44 -06:00
Dimitris Athanasiou 168b566844 [ML] Allow closing a job that is restoring state (elastic/x-pack-elasticsearch#1746)
This change enables closing a job while it is in
the middle of restoring its state. This is has the
benefit of allowing users to close jobs that due to
relocation are `opened` but they are still restoring
state. It also helps avoiding race conditions in tests.

Part of this change also includes restoring the state
as a separate step from the process creation. This means
we no longer block the job map while the process is
restoring its state.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@1713a4a7c4
2017-06-19 18:39:45 +01:00
Dimitrios Athanasiou c39b35d4a0 [ML] Increase ScrollDataExtractor context duration
In cases where the job is bound on the analytics performance
the datafeed can fail because the scroll expires. This is
commit increases the scroll context duration from 10 to 30 minutes
as a temporary solution that will avoid most cases.

Original commit: elastic/x-pack-elasticsearch@fd277bbaa1
2017-06-19 18:36:34 +01:00
jaymode 0532c9069b Test: use System.lineSeperator instead of `\n`
This changes the native realm migrate tool tests to use the System.lineSeperator instead of `\n`
so that the tests will pass on Windows.

Original commit: elastic/x-pack-elasticsearch@d3f9a71ac4
2017-06-19 10:34:30 -06:00
David Kyle cbceae4a33 [ML] Call initialising constructor of BaseTasksRequest (elastic/x-pack-elasticsearch#1771)
Original commit: elastic/x-pack-elasticsearch@216527c959
2017-06-19 16:40:32 +01:00
David Roberts 7ab6a7e83a [ML] Fix NullPointerException when deleting a job with state "opening" (elastic/x-pack-elasticsearch#1769)
relates elastic/x-pack-elasticsearch#1768

Original commit: elastic/x-pack-elasticsearch@ae1f431316
2017-06-19 16:24:10 +01:00
Alexander Reelsen c1685c70ea Revert "X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)"
This reverts commit elastic/x-pack-elasticsearch@e430691c51.

The setting is going to be applied in core, so that only parts
of these changes need to be applied here (in a seperate PR).

Original commit: elastic/x-pack-elasticsearch@9e178c87fa
2017-06-19 16:13:36 +02:00
Shaunak Kashyap 160fb8ee53 Index logstash_stats.pipelines.events.duration_in_millis (elastic/x-pack-elasticsearch#1752)
Original commit: elastic/x-pack-elasticsearch@c731badb20
2017-06-19 07:12:12 -07:00
Luca Cavanna 6344bff121 Remove (deprecated) support for '+' in index expressions (elastic/x-pack-elasticsearch#1749)
Original commit: elastic/x-pack-elasticsearch@fc3fa2b842
2017-06-19 15:20:02 +02:00
David Kyle 40700f22b1 [ML] Fix NPE by calling super constructor that inits members. (elastic/x-pack-elasticsearch#1763)
Original commit: elastic/x-pack-elasticsearch@ee47ad8322
2017-06-19 14:18:14 +01:00
Alexander Reelsen 22d57ca96e Tests: Fix line endings to match mock terminal line endings
The MockTerminal class always uses "\n" as line endings, so
the tests should not use `System.lineSeparator`, but also
a newline.

relates elastic/x-pack-elasticsearch#1758

Original commit: elastic/x-pack-elasticsearch@364a4e43ad
2017-06-19 13:48:54 +02:00
Dimitris Athanasiou ae70a657dd [ML] Add documentation to rest spec (elastic/x-pack-elasticsearch#1756)
Original commit: elastic/x-pack-elasticsearch@7d7de84a5a
2017-06-19 10:37:17 +01:00
David Roberts 03652e7497 [ML] Kill autodetect on force close and isolated node rejoining (elastic/x-pack-elasticsearch#1742)
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job.  This was not ideal, because then the job
would persist state from the autodetect process that was isolated.  This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.

In order to achieve this a change has also been made to the behaviour of
force-close.  Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request.  However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster.  Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it.  The documentation
has been changed to reflect this.  It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.

relates elastic/x-pack-elasticsearch#1186

Original commit: elastic/x-pack-elasticsearch@578c944371
2017-06-19 10:16:51 +01:00
Alexander Reelsen 44c3c6b992 Watcher: Fix NPE when search input did not contain search body (elastic/x-pack-elasticsearch#1736)
This came up in a forum post. An NPE was raised, when a search input
contained a search that did not contain a body, but just specified
indices or types.

This commit allows for empty bodies, and also makes sure there are
no null pointer exceptions by using empty bytes references otherwise.

In addition a suite scoped integration test was converted to a unit
test.

Original commit: elastic/x-pack-elasticsearch@29be2976fc
2017-06-19 10:07:10 +02:00
Simon Willnauer 35eb70e113 Catch up with renamings in `TcpTransport`
This is a followup from elastic/elasticsearch#25250

Original commit: elastic/x-pack-elasticsearch@325d10b973
2017-06-19 09:29:20 +02:00
Alexander Reelsen 65228e4379 X-Pack: Add index.internal.format index setting (elastic/x-pack-elasticsearch#1700)
This introduces a new index setting called xpack.internal.format to
x-pack, which is configured for all of our index templates and set to
"v6". This indicates the version of compatibility of this index.

In addition a setting named index.xpack.version has been removed,
as it was unused.

Watcher does not start, if the watches and the triggered watches
index is not compatible with this setting.

Original commit: elastic/x-pack-elasticsearch@e430691c51
2017-06-19 08:34:43 +02:00
Nik Everett 1559f85c73 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.

Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
2017-06-16 17:19:47 -04:00
Andrew Cholakian 4967d424f6 Insert ephemeral IDs + correct pipeline event stats into the logstash_stats document (elastic/x-pack-elasticsearch#1747)
Before the event stats were mislabeled, so were not being indexed, and
the ephemeral_id was only in the _state document.

Original commit: elastic/x-pack-elasticsearch@ca0ec81aa5
2017-06-16 14:18:28 -05:00
Luca Cavanna f0c026d1d9 [TEST] adapt test: delete index doesn't resolve to aliases anymore (elastic/x-pack-elasticsearch#1735)
Original commit: elastic/x-pack-elasticsearch@49629ccea6
2017-06-16 17:46:45 +02:00
Nik Everett d526461bd2 Add basic full cluster restart tests for x-pack (elastic/x-pack-elasticsearch#1743)
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.

The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.

Relates to elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@8994bec8e7
2017-06-16 11:44:51 -04:00
Jay Modi ed382807c3 Remove unnecessary initialization of the system key (elastic/x-pack-elasticsearch#1734)
This commit removes unnecessary initialization of the system key in tests that no longer make use
of the system key. It also removes the feature usage for the system key in the SecurityFeatureSet.

Original commit: elastic/x-pack-elasticsearch@b9fffe0bd3
2017-06-16 09:24:03 -06:00
Pier-Hugues Pellerin 76857d7bbe Add support for Logstash' ephemeral_id (elastic/x-pack-elasticsearch#1697)
Logstash now has ephemeral id at the instance level and also at the
pipeline level, we need to add them to the logstash monitoring template.

Original commit: elastic/x-pack-elasticsearch@dfac702d59
2017-06-16 11:08:08 -04:00
David Kyle 02da8e7cd9 [ML] Use bulk request to persist model plots (elastic/x-pack-elasticsearch#1714)
* Use bulk request to persist model plots and model size stats

* Revert persisting model size stats in the bulk request

* Refactor results persister

Original commit: elastic/x-pack-elasticsearch@f51297bfc2
2017-06-16 15:18:16 +01:00
Tim Vernum c63d4e306b Reduce logging for reserved realm authc failures (elastic/x-pack-elasticsearch#1711)
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.

However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.

Original commit: elastic/x-pack-elasticsearch@686a98010b
2017-06-16 10:46:07 +10:00
Martijn van Groningen 9dddce2e08 Add analysis-common as test dependency to xpack plugin module.
Original commit: elastic/x-pack-elasticsearch@2472585037
2017-06-15 20:50:31 +02:00
Tim Brooks 7c7e47aa0f Add tool to setup passwords for internal users (elastic/x-pack-elasticsearch#1434)
This is related to elastic/x-pack-elasticsearch#1217. This change introduces a tool
bin/x-pack/setup-passwords that will streamline the setting of
internal user passwords. There are two modes of operation. One mode
called auto, automatically generates passwords and prints them to
the console. The second mode called interactive allows the user to 
enter passwords.

All passwords are changed using the elastic superuser. The elastic
password is the first password to be set.

Original commit: elastic/x-pack-elasticsearch@00974234a2
2017-06-15 10:48:02 -05:00
Jay Modi d920cc7348 Remove signing code completely from master (elastic/x-pack-elasticsearch#1719)
After improving the authorization of scroll requests and backporting to 5.x, we no longer need to
have any signing code in master. This commit removes it.

Original commit: elastic/x-pack-elasticsearch@8b65fd9338
2017-06-15 07:51:39 -06:00
Jim Ferenczi 07fcf75dd9 [Test] Fix DocumentLevelSecurityTests and FieldLevelSecurityTests to work with the new format of the parent-join field
Original commit: elastic/x-pack-elasticsearch@cd9f26887b
2017-06-15 15:40:41 +02:00
David Roberts 291aa27562 [ML] Kill autodetect on error writing headers (elastic/x-pack-elasticsearch#1730)
If an exception occurs while sending the initial setup messages to the autodetect
such that it fails rather than reaching the open state then the autodetect process
needs to be killed to prevent it hogging resources.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@1ee80ed9b0
2017-06-15 14:03:41 +01:00
David Roberts b748da1880 [ML] Prevent time_field and control field name in analysis_config (elastic/x-pack-elasticsearch#1729)
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer.  Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.

Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@e6750a2cda
2017-06-15 13:04:25 +01:00
Boaz Leskes c76bdf61d7 move assertBusy to use CheckException (elastic/x-pack-elasticsearch#1727)
We use assertBusy in many places where the underlying code throw exceptions. Currently we need to wrap those exceptions in a RuntimeException which is ugly.

Companion PR to https://github.com/elastic/elasticsearch/pull/25246

Original commit: elastic/x-pack-elasticsearch@056857273f
2017-06-15 13:24:34 +02:00
Tim Vernum e36f86cf95 Improve upgrade process for reserved users (elastic/x-pack-elasticsearch#1712)
- Don't attempt to upgrade from 2.x
- Attempt up to 10 retries if the migration fails (with increasing back-off between attempts)
- If a cached user is disabled, recheck with the underlying store

The last change is required if the migration takes a long time.
While users are being migrated, they might be marked as disabled, but when the migration is complete they need to be usable immediately.

Original commit: elastic/x-pack-elasticsearch@2621867014
2017-06-15 10:27:23 +10:00
Nik Everett 0970c509bc Remove the assemble task from projects not published (elastic/x-pack-elasticsearch#1721)
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.

Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
2017-06-14 19:57:26 -04:00
Igor Motov 8abf4c3119 Tests: Remove QUERY_AND_FETCH BWC actions from the list of known handlers
Relates to elastic/elasticsearch#25223

Original commit: elastic/x-pack-elasticsearch@2da097d1ea
2017-06-14 19:01:14 -04:00
Simon Willnauer 97693b9357 Add scroll support for cross cluster search (elastic/x-pack-elasticsearch#1706)
Original commit: elastic/x-pack-elasticsearch@eadffa396b
2017-06-14 20:38:58 +02:00
Dimitrios Athanasiou 50de99374c [TEST] Start non ML node in BasicDistributedJobsIT.testMlIndicesNotAvailable
Original commit: elastic/x-pack-elasticsearch@71e63b6ea2
2017-06-14 16:43:57 +01:00
Dimitrios Athanasiou 1eb785d8ee [TEST] Minor refactorings in BasicDistributedJobsIT
Original commit: elastic/x-pack-elasticsearch@eab0fe759e
2017-06-14 16:04:06 +01:00
Dimitrios Athanasiou cd119c488f [TEST] Remove test logging for resolved issue elastic/x-pack-elasticsearch#1048
Original commit: elastic/x-pack-elasticsearch@012d2b53c9
2017-06-14 15:21:20 +01:00
David Kyle 1010f73ae7 [ML] Retry after broken scroll (elastic/x-pack-elasticsearch#1713)
Original commit: elastic/x-pack-elasticsearch@b4fc329c52
2017-06-14 15:04:14 +01:00
Alexander Reelsen 4e085f03b7 Tests: Disable watcher in ServerTransportFilterIntegrationTests
These tests are starting their own nodes and do not use the testing
trigger schedule class.

There are occasional test failure due to a race condition where watcher
is in the process of being started, but cannot be shut down properly,
because starting up was not finished when the shut down was called for.

These filter tests do not rely on watcher, so we can disable them for
now, but we still need to fix a race condition in starting/stopping
watcher.

relates elastic/x-pack-elasticsearch#1422

Original commit: elastic/x-pack-elasticsearch@f13bb7a6fb
2017-06-14 15:00:37 +02:00
Chris Earle bb31c09ed0 [Monitoring] Add Mapping for cfs_quota_micros field (elastic/x-pack-elasticsearch#1710)
This maps the cgroup field, which was unmapped (and now mappings are no longer dynamic).

Original commit: elastic/x-pack-elasticsearch@86b4333ebb
2017-06-14 07:37:15 -04:00
Alexander Reelsen 457af97d1f Watcher: Remove handling of _status field (elastic/x-pack-elasticsearch#1701)
There is no need to handle any _status field in
the 6.0 release from now on, as everything has been
taken care in the upgrade API.

Original commit: elastic/x-pack-elasticsearch@606581f4d7
2017-06-14 10:36:46 +02:00
David Roberts e7af52fb8e [ML] Let core ES stop the native controller when ML is disabled (elastic/x-pack-elasticsearch#1704)
This changes part of the logic that was added in elastic/x-pack-elasticsearch#644 and extended
in elastic/x-pack-elasticsearch#1495 so that when ML is disabled we never try to communicate with
the native controller during node shutdown.

The original reason for needing to communicate with the native controller
when ML is disabled was the problem of elastic/prelert-legacy#803.
However, this was fixed in a better way in elastic/elasticsearch#24579.
Now there is considerable benefit in never talking to the native
controller from the plugin code when ML is disabled, because it means
anyone suffering some obscure problem with ML can disable it without
running the risk of uncovering some other obscure problem with shutdown.

Original commit: elastic/x-pack-elasticsearch@9d329483a7
2017-06-14 08:54:57 +01:00
Alexander Reelsen 4346431156 Tests: Ensure watcher is started in REST tests (elastic/x-pack-elasticsearch#1702)
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.

This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.

Original commit: elastic/x-pack-elasticsearch@fc547d49b4
2017-06-14 08:34:09 +02:00
Ryan Ernst abe217ebc3 Remove uses of Strings.cleanPath (elastic/x-pack-elasticsearch#1709)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25209

Original commit: elastic/x-pack-elasticsearch@4f59f011e9
2017-06-13 21:10:02 -07:00
Suyog Rao fe72991c70 [Logstash] Add new component to x-pack to handle LS features (elastic/x-pack-elasticsearch#1530)
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.

On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.

relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471

Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
2017-06-13 10:30:30 -07:00
Dimitris Athanasiou f2e2ccae01 [ML] Extract parent field when job has text multi-field (elastic/x-pack-elasticsearch#1705)
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.

However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.

Original commit: elastic/x-pack-elasticsearch@f116e89921
2017-06-13 18:00:24 +01:00
Simon Willnauer 65bac10eed Fix API change in SecurityNetty4Transport
Original commit: elastic/x-pack-elasticsearch@f152fb1813
2017-06-13 10:15:26 +02:00
Dimitrios Athanasiou 29811ea1d8 [ML] Write null instead of false for optional boolean that was removed
Original commit: elastic/x-pack-elasticsearch@43f5fc04bd
2017-06-12 17:35:25 +01:00
Dimitrios Athanasiou 1d33fb8b8b [TEST] Add now required mapping for data index in datafeed test
Original commit: elastic/x-pack-elasticsearch@67cf68c5a0
2017-06-12 15:50:16 +01:00
Dimitris Athanasiou 8eb62eac27 [ML] Automate detection of way to extract fields (elastic/x-pack-elasticsearch#1686)
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.

This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.

This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.

relates elastic/x-pack-elasticsearch#1649

Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
2017-06-12 14:56:31 +01:00
Dimitris Athanasiou c9bbc17742 [ML] Make datafeed types optional (elastic/x-pack-elasticsearch#1690)
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.

Original commit: elastic/x-pack-elasticsearch@f61ac01b45
2017-06-12 14:46:53 +01:00
Alexander Reelsen 27b5142de6 Watcher: Fix croneval tool for packaging (elastic/x-pack-elasticsearch#1689)
The croneval script used an old parameter to start up.
This commit removes this parameter, that is used, when a
package is used.

In addition a concrete vagrant test has been added.

relates elastic/x-pack-elasticsearch#1635

Original commit: elastic/x-pack-elasticsearch@ea7b8a08f4
2017-06-12 13:56:26 +02:00
Alexander Reelsen 83a5e022cb Watcher: Reduce logging noise when watcher might be stopped (elastic/x-pack-elasticsearch#1685)
Only log an entry when an actual stop is executed instead of
always logging.

Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.

Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
2017-06-12 10:24:16 +02:00
David Kyle 516696bbec [ML][TEST] Refresh after indexing docs
Original commit: elastic/x-pack-elasticsearch@b64ad1b2c6
2017-06-10 19:12:52 +01:00
Jason Tedor cb62d81bdc Fix compilation for CreateIndexResponse
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.

Original commit: elastic/x-pack-elasticsearch@b078d80cd9
2017-06-09 14:59:55 -04:00
jaymode 741bf85fbf Remove incorrect bwc branch logic from master
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.

Original commit: elastic/x-pack-elasticsearch@9a77269fa6
2017-06-09 10:02:43 -06:00
Alexander Reelsen 709ed7d50e Watcher: Only try to load triggered watches index, if it exists (elastic/x-pack-elasticsearch#1569)
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.

This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.

Original commit: elastic/x-pack-elasticsearch@9f26d557d0
2017-06-09 17:51:23 +02:00
Alexander Reelsen d769ee0813 Watcher: Do not pause watcher if not needed (elastic/x-pack-elasticsearch#1681)
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.

This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.

Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
2017-06-09 17:50:35 +02:00
Alexander Reelsen 7f48337bf6 Watcher: Remove index template configuration via cluster settings (elastic/x-pack-elasticsearch#1680)
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.

As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.

This feature was not exposed in our documentation at all.

Original commit: elastic/x-pack-elasticsearch@32e1769925
2017-06-09 17:50:01 +02:00
Ryan Ernst c1a3f50e19 Convert script uses to use source/id keys (elastic/x-pack-elasticsearch#1670)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25127

Original commit: elastic/x-pack-elasticsearch@e25bd90825
2017-06-09 08:29:36 -07:00