Commit Graph

1140 Commits

Author SHA1 Message Date
Alexander Reelsen 547b0ebc1b X-Pack: Remove deprecated handlers, remove watcher restart action (elastic/x-pack-elasticsearch#2133)
The deprecated handlers should have been removed earlier, but are now
going to to away finally.

Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.

Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
2017-08-01 14:14:51 +02:00
Alexander Reelsen 681ce6e2aa Watcher: Replace flaky WatchStatsTests with unit tests (elastic/x-pack-elasticsearch#2118)
This flaky tests was using sleep, latches and a custom script plugin,
causing issues with stopping/starting tests.

This removes the integration tests and replaces it with a unit test.

Also removed a couple of unused ctor/method parameters as cleanup.

relates elastic/x-pack-elasticsearch#1639

Original commit: elastic/x-pack-elasticsearch@2a42faf2db
2017-08-01 09:50:32 +02:00
Alexander Reelsen 9ea032998b Watcher: Remove REST hijacking operations (elastic/x-pack-elasticsearch#2132)
As all the triggering is now done on the shards, people can use
bulk, *-by-query and regular index/delete operations on the watcher
index and do not need to use the dedicated APIs.

We can now remove the long existing rest operation hijacking code.

Original commit: elastic/x-pack-elasticsearch@08f4f4c3de
2017-08-01 09:49:28 +02:00
Jay Modi 7fecca6329 Fix handling of exceptions for concurrent attempts to create security index (elastic/x-pack-elasticsearch#2120)
This commit fixes the handling of some exceptions when we attempt to create the security index and
alias. The issue here is provoked by a test that is currently muted with an AwaitsFix,
GroupMappingTests, which will be unmuted in another change.

Original commit: elastic/x-pack-elasticsearch@55f6b656cb
2017-07-31 12:34:14 -06:00
Luca Cavanna dac28fe4e0 Adapt to AbstractQueryBuilder upstream changes
AbstractQueryBuilder doesn't extend ToXContentToBytes anymore

Original commit: elastic/x-pack-elasticsearch@6c1b948ccb
2017-07-31 17:38:56 +02:00
Jason Tedor c51e6e1cd6 Remove unnecessary directory push/pop from scripts
This commit proposes removing an unnecessary directory push/pop from the
X-Pack scripts. It is not clear exactly why these were added, the
original change was almost three years ago in
elastic/x-pack@ea9ba7cdd0 but
unfortunately the commit message does not elucidate the exact the
problem, nor is there an associated pull request. This change has
propogated into all of the X-Pack scripts yet still the reasons are
unclear. The little that we can glean from the commit message is that
there was a problem with the default paths if the script was executed
outside of the Elasticsearch home. It seems that such issues have been
addressed by the recent introduction of elasticsearch-env so maybe we
can simplify these scripts here?

Relates elastic/x-pack-elasticsearch#2125

Original commit: elastic/x-pack-elasticsearch@9548c47743
2017-07-31 23:13:22 +09:00
Jason Tedor a80b1e4de1 Exit Windows scripts promptly on failure
When invoking the elasticsearch-env.bat or x-pack-env.bat batch scripts
on Windows, if these scripts exits due to an error (e.g., Java can not
be found, or the wrong version of Java is found), then the script
exits. Sadly, on Windows, this does not also terminate the caller,
instead returning control. This means we have to explicitly exit so that
is what we do in this commit.

Relates elastic/x-pack-elasticsearch#2126

Original commit: elastic/x-pack-elasticsearch@18645db62c
2017-07-31 20:39:59 +09:00
Jason Tedor 351febe031 Introduce elasticsearch-env for Windows
This commit refactors the batch scripts on Windows to use the
elasticsearch-env.bat script, and introduces a Windows version of
x-pack-env.bat for reuse in the scripts as well.

Relates elastic/x-pack-elasticsearch#2124

Original commit: elastic/x-pack-elasticsearch@faacb40dca
2017-07-30 09:31:52 +09:00
Chris Earle 45964561eb [Security] Add 'read_cross_cluster' privilege for .monitoring indices (elastic/x-pack-elasticsearch#2111)
This allows 6.0+ monitoring clusters to be used out of the box with CCS for extended read-only access.

Original commit: elastic/x-pack-elasticsearch@2b1e4ca4e4
2017-07-28 16:02:20 -04:00
Jay Modi db4c00b565 Update the full cluster restart tests to be more generic (elastic/x-pack-elasticsearch#2107)
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.

Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
2017-07-28 10:31:44 -06:00
David Kyle 694cfd6f0e Fix javadoc typo
Original commit: elastic/x-pack-elasticsearch@ef901af9cc
2017-07-28 16:24:24 +01:00
Lisa Cawley e7248c579c [DOCS] Fix licensing documentation links (elastic/x-pack-elasticsearch#2108)
Original commit: elastic/x-pack-elasticsearch@7b0f74348f
2017-07-28 08:19:55 -07:00
David Kyle f9104b7127 [ML] Lookback one extra bucket for histograms (elastic/x-pack-elasticsearch#2084)
Original commit: elastic/x-pack-elasticsearch@b9b4d3977f
2017-07-28 16:14:27 +01:00
Yannick Welsch 3a265de458 Revert "Mute IndexAuditTrailTests"
This reverts commit elastic/x-pack-elasticsearch@2e4a837dae.

Original commit: elastic/x-pack-elasticsearch@d8841cd2ce
2017-07-28 14:56:48 +02:00
Yannick Welsch 40174e4dcf [TEST] Set proper "transport.type" for transport client in IndexAuditTrailTests
Fixes issues seen on CI.

relates elastic/x-pack-elasticsearch#2115

Original commit: elastic/x-pack-elasticsearch@0094225a2e
2017-07-28 14:56:36 +02:00
David Kyle f99838e1bf Mute IndexAuditTrailTests
Original commit: elastic/x-pack-elasticsearch@2e4a837dae
2017-07-28 13:41:20 +01:00
David Kyle 21b437e140 Mute ShardsCollectorTests.testShardsCollectorMultipleIndices
Original commit: elastic/x-pack-elasticsearch@c2a6c05536
2017-07-28 11:42:51 +01:00
Martijn van Groningen 5f51dd813c XPack changes for: https://github.com/elastic/elasticsearch/pull/25456
Original commit: elastic/x-pack-elasticsearch@9d1d3c3fb4
2017-07-28 12:25:27 +02:00
Yannick Welsch 51e87bf290 Move tribe to a module (elastic/x-pack-elasticsearch#2088)
Companion PR to elastic/elasticsearch#25778

Original commit: elastic/x-pack-elasticsearch@a3355802e9
2017-07-28 11:23:52 +02:00
Jason Tedor 8176adfc26 Fix retrieve roles and users tests
This commit removes the use of a now removed --path.conf command-line
flag from the retrieve roles and users tests.

Original commit: elastic/x-pack-elasticsearch@30d5f5f648
2017-07-28 18:04:05 +09:00
Jason Tedor e77db8faf4 Fix users tool tests
This commit fixes the users tool command tests which were broken because
of a guard added that es.path.conf is set. We do not want to set this
system property in tests so instead we override createEnv where the
problematic guard exists.

Original commit: elastic/x-pack-elasticsearch@78b757695b
2017-07-28 17:32:39 +09:00
Jason Tedor d8277942ac Fix list X-Pack extension command tests
This commit fixes the list X-Pack extension command tests which were
broken because of a guard added that es.path.conf is set. We do not want
to set this system property in tests so instead we override createEnv
where the problematic guard exists.

Original commit: elastic/x-pack-elasticsearch@b1bc4ddcb8
2017-07-28 17:15:23 +09:00
Jason Tedor aade36eff3 Fix failing CLI tests
This commit fixes some failing CLI tests. The failure here is that a
guard against the system property es.path.conf was added yet these tests
were not adapted for this change. This commit implements this adapation
which overrides the createEnv method where the problematic guard is
invoked. We do this to avoid having to set es.path.conf in tests.

Original commit: elastic/x-pack-elasticsearch@20e1724823
2017-07-28 17:00:25 +09:00
Jason Tedor af7ec7a213 Pass config path as a system property
This commit responds to an upstream change which removes the --path.conf
command-line flag and instead uses the replacement mechanism for setting
the configuration path via the system property es.path.conf.

Relates elastic/x-pack-elasticsearch#2113

Original commit: elastic/x-pack-elasticsearch@4fefbffecb
2017-07-28 12:15:37 +09:00
Jay Modi 3a1b64bb12 Create the cryptoservice later in startup process (elastic/x-pack-elasticsearch#2087)
This commit moves the creation of the CryptoService to the createComponents method so that bootstrap
checks have been checked before the crypto service is instantiated. The cryptoservice was changed to
expect that the bootstrap check has passed before being instantiated in elastic/x-pack-elasticsearch#1831.

Original commit: elastic/x-pack-elasticsearch@cf11cf4782
2017-07-27 14:03:05 -06:00
David Kyle b95d3f7bf4 Unmute test after fix
Original commit: elastic/x-pack-elasticsearch@cceb3e7237
2017-07-27 11:34:52 +01:00
Tim Brooks 1603823f2c Ensure that security client is used in x-pack
This commit is fixing an issue with the build. Currently we have a
scenario where a call to build a transport client is not using the
security client. This modifies the settings source to ensure that the
security client is used.

Original commit: elastic/x-pack-elasticsearch@2d6ea1f4e4
2017-07-26 13:36:20 -05:00
Luca Cavanna cb20de5d95 Adapt to removal of XContentHelper#toString(ToXContent) (elastic/x-pack-elasticsearch#2072)
Original commit: elastic/x-pack-elasticsearch@35f6ac23a1
2017-07-26 16:01:09 +02:00
Igor Motov 4da7ae21bc Persistent Tasks: remove unused isCurrentStatus method (elastic/x-pack-elasticsearch#2076)
Removes a method that is no longer used in production code.

Relates to elastic/x-pack-elasticsearch#957

Original commit: elastic/x-pack-elasticsearch@84fcb9db8a
2017-07-26 08:40:22 -04:00
Simon Willnauer 3914d66cea Followup for elastic/elasticsearch#25885
Original commit: elastic/x-pack-elasticsearch@0397e68efd
2017-07-26 09:17:40 +02:00
David Kyle 14c88ca15f Correct long line length
Original commit: elastic/x-pack-elasticsearch@e77f473b5b
2017-07-25 16:59:42 +01:00
David Kyle 74d06216c2 [ML] Accept more varied Datafeed Aggregations (elastic/x-pack-elasticsearch#2038)
Original commit: elastic/x-pack-elasticsearch@ec1477f41c
2017-07-25 16:45:47 +01:00
David Kyle 8f6d9df96e [ML] Check influencer names are valid (elastic/x-pack-elasticsearch#2073)
Original commit: elastic/x-pack-elasticsearch@75869cacb3
2017-07-25 10:41:51 +01:00
Tim Vernum 9ab6d3cbc3 [Security] Support PKCSelastic/x-pack-elasticsearch#12 keystores (elastic/x-pack-elasticsearch#2066)
Adds support for reading PKCSelastic/x-pack-elasticsearch#12 files as SSL keystores/truststores.

Original commit: elastic/x-pack-elasticsearch@1855ad6173
2017-07-25 17:31:37 +10:00
Michael Basnight 6a7e51d9c0 Use shaded rest client dependencies
This commit modifies all org.apache.http to use the shaded rest clients
org.elasticsearch.client.http packages. It also removes a few unused
licenses due to the change.

Relates elastic/elasticsearch#25780

Original commit: elastic/x-pack-elasticsearch@8605560232
2017-07-24 12:56:17 -05:00
Chris Earle b302ff32e8 [Monitoring] Remove BWC Layer for Marvel (2.3 - 2.4) in 6.0 (elastic/x-pack-elasticsearch#1990)
This removes all BWC code that assisted in reading Marvel indices in 6.0.

Original commit: elastic/x-pack-elasticsearch@253fbf9a73
2017-07-24 10:23:06 -04:00
Simon Willnauer 9078c0b244 [TEST] use true unique transport address in test to prevent address collision
Original commit: elastic/x-pack-elasticsearch@91e5e1c046
2017-07-23 21:20:18 +02:00
Tal Levy 04ace4f1df add Migration Deprecation Info API Documentation (elastic/x-pack-elasticsearch#2064)
Original commit: elastic/x-pack-elasticsearch@0b90dfc97f
2017-07-21 16:20:13 -07:00
jaymode 9357369d57 Test: TokenAuthIntegTests ensures security index writable before invalidating
This commit makes sure the TokenAuthIntegTests wait for the security index to be writable before
each test method as invalidation requires writing to the security index.

Relates elastic/x-pack-elasticsearch#1551

Original commit: elastic/x-pack-elasticsearch@6e22885102
2017-07-21 14:01:43 -06:00
Ali Beyad 5190a05b75 Improves error message on non-upgraded security index (elastic/x-pack-elasticsearch#2061)
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.

Original commit: elastic/x-pack-elasticsearch@710b7634b4
2017-07-21 15:56:15 -04:00
Igor Motov 66a723d134 Docs: Add Upgrade API documentation (elastic/x-pack-elasticsearch#2063)
Original commit: elastic/x-pack-elasticsearch@f5a7c9ee5d
2017-07-21 15:26:23 -04:00
Jay Modi 2f7142ccc7 Provide a message for callers of the EmptyTrustManager (elastic/x-pack-elasticsearch#2052)
This change will provide a exception with a message to any callers of the empty trust manager for
better visibility into issues.

Original commit: elastic/x-pack-elasticsearch@c8241aea98
2017-07-21 10:10:14 -06:00
Tim Vernum b29f7a9ddb [Security] Handle non-existent user in native realm (elastic/x-pack-elasticsearch#2044)
Since change elastic/x-pack-elasticsearch@f796949 authentication is not allowed to respond with null, it must be AuthenticationResult.notHandled()

- Fixes 1 case where the native realm would respond null if the user was not found
- Fixes some edge cases in the LDAP realm.

Original commit: elastic/x-pack-elasticsearch@bc739a1d40
2017-07-21 22:25:50 +10:00
Tim Vernum 1752104140 Grant `getProxySelector` permission to rest client (elastic/x-pack-elasticsearch#2059)
Monitoring uses the low level rest client, which was recently updated to need an additional permission

Relates: elasticsearch#25757
relates elastic/x-pack-elasticsearch#2058

Original commit: elastic/x-pack-elasticsearch@eb9578792c
2017-07-21 19:06:40 +10:00
Tim Brooks 495fc21c37 Add log when elastic password boostrapped (elastic/x-pack-elasticsearch#2053)
This is related to elastic/x-pack-elasticsearch#1217. This adds a log message to inform the user
when the elastic user's password is bootsrapped successfully.

Original commit: elastic/x-pack-elasticsearch@8d30e163ec
2017-07-20 23:00:08 -05:00
Tim Vernum 5056d4e3df Revert unintended changes to syskeygen from elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch#2055)
During the development of elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch@5ac95c6) there was an intention to replace `SystemKeyTool` with `EncKeyTool`.
The java change was reverted and never committed to master, but the script change was accidentally left in place.

Also removes redundant "properties" variable (refer `elastic/x-pack-elasticsearch@b0a3b89`)

Original commit: elastic/x-pack-elasticsearch@f240479748
2017-07-21 12:51:25 +10:00
Jason Tedor 470e81ce64 Use elasticsearch-env
This commit utilizes the elasticsearch-env script that is added in core
Elasticsearch for significantly simplifying the scripts used in x-pack.

Relates elastic/x-pack-elasticsearch#2049

Original commit: elastic/x-pack-elasticsearch@8ef041d077
2017-07-21 09:39:46 +09:00
Andrew Cholakian c8e42a0db6 Update logstash mappings to use pipeline.id + correct metric types (elastic/x-pack-elasticsearch#2045)
In logstash parlance there really is no pipeline.name, its pipeline.id

This also removes support for deprecated gauge types `text` and `boolean` we will be removing those soon in logstash

This also renames `counters` to `long_counters` to be more explicit and for consistency with the gauge type. Also, if we ever decide to add other types of counters this will be more clear

Original commit: elastic/x-pack-elasticsearch@8f44a94579
2017-07-20 14:30:37 -05:00
Chris Earle e0fe6da303 [Security] Remove beats_system User and Role until needed (elastic/x-pack-elasticsearch#2042)
The user/role was added to simplify they setup surrounding Beats monitoring, but Beats monitoring has been delayed until the UI work is begun.

Original commit: elastic/x-pack-elasticsearch@1c0c85562e
2017-07-20 12:24:15 -04:00
Jack Conradson e007fee9fb remove lang url parameter from stored script requests (elastic/x-pack-elasticsearch#2029)
Original commit: elastic/x-pack-elasticsearch@1044c3ba53
2017-07-20 08:51:27 -07:00
Jason Tedor 5ecbbbd46d Use plain old exception if security init fails
When security initialization fails during startup today we throw an
Error. This triggers the uncaught exception handler immediately killing
the node. While the node is going to die either way, we should not be
triggering the killer and in general we should avoid using Throwables
that extend Error.

Relates elastic/x-pack-elasticsearch#2035

Original commit: elastic/x-pack-elasticsearch@546f7f9002
2017-07-21 00:26:22 +09:00
Simon Willnauer eb5631b981 Followup for elastic/elasticsearch#25791
Original commit: elastic/x-pack-elasticsearch@806de5670f
2017-07-20 16:45:48 +02:00
Jason Tedor 3f08fad603 Remove CONF_FILE check
This commit removes a legacy check for an unsupported environment
variable. This environment variable has not been supported since 1.x so
it is safe to stop checking for the existence of this setting.

Relates elastic/x-pack-elasticsearch#2048

Original commit: elastic/x-pack-elasticsearch@023230fa9e
2017-07-20 22:42:21 +09:00
Jason Tedor 814c1a21f9 Stop exporting HOSTNAME from scripts
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.

Relates elastic/x-pack-elasticsearch#2047

Original commit: elastic/x-pack-elasticsearch@7b833e061c
2017-07-20 22:28:01 +09:00
Alexander Reelsen dce13b87c3 Watcher: Remove stream serialization version checks, happen in 5.x (elastic/x-pack-elasticsearch#2002)
Those checks were moved into 5.x in elastic/x-pack-elasticsearch#2001

Original commit: elastic/x-pack-elasticsearch@7664bf35c5
2017-07-20 14:48:08 +02:00
Jason Tedor 581293aed5 Fix croneval script to respect CONF_DIR
This commit fixes the croneval script to respect the CONF_DIR
environment variable used to locate the configuration directory.

Original commit: elastic/x-pack-elasticsearch@79974947f9
2017-07-20 19:49:28 +09:00
Jason Tedor 5fe584472e Fix certgen script usage of ES_HOME
This commit fixes an issue with the usage of the environment variable
ES_HOME in the certgen script; the script was missing the use of $ to
obtain the value of the environment variable ES_HOME.

Relates elastic/x-pack-elasticsearch#2046

Original commit: elastic/x-pack-elasticsearch@63128db0eb
2017-07-20 16:02:21 +09:00
Jason Tedor 6552c9a5f7 Remove use of ES_INCLUDE
This commit removes all uses of ES_INCLUDE as this functionality has
been removed from core Elasticsearch in favor of a dedicated include
script for establishing the environment.

Relates elastic/x-pack-elasticsearch#2046

Original commit: elastic/x-pack-elasticsearch@92f8470e44
2017-07-20 15:41:59 +09:00
Tim Vernum 776f7cec65 [TEST] Improve SSLTrustRestrictionsTests (elastic/x-pack-elasticsearch#2015)
Use assertBusy rather than a fixed sleep time to wait for SSL reloading.

relates elastic/x-pack-elasticsearch#2007

Original commit: elastic/x-pack-elasticsearch@c8f789c327
2017-07-20 13:10:27 +10:00
Ali Beyad fe9d99daed Removes NativeRealmMigrator since its no longer needed (elastic/x-pack-elasticsearch#2028)
With the new template and mapping update mechanisms introduced as part
of the Upgrade API work, the NativeRealmMigrator is no longer needed or
used.  This commit removes the NativeRealmMigrator code and the
associated tests for it.

Original commit: elastic/x-pack-elasticsearch@5d2d7a582c
2017-07-19 12:19:48 -04:00
Tim Brooks 7ed5df4068 Make self-generated license type configurable (elastic/x-pack-elasticsearch#2000)
This is related to elastic/x-pack-elasticsearch#1778. This commit adds a setting
(xpack.license.self_generated.type) which allows the user to specify
what type of license will be self-generated on node startup. The allowed
types are basic or trial.

Original commit: elastic/x-pack-elasticsearch@0a16a59e10
2017-07-19 10:56:10 -05:00
Adrien Grand f15147b87b Remove assumption about how the timeout feature is implemented.
Original commit: elastic/x-pack-elasticsearch@7a4fa000c1
2017-07-19 17:29:49 +02:00
Tim Brooks a6bf79bd31 Remove the container work (elastic/x-pack-elasticsearch#2030)
This is related to elastic/x-pack-elasticsearch#1217. This commit removes the features that were
introduced in elastic/x-pack-elasticsearch#1832. This is because the containerized version of
x-pack can be have its password bootstrapped using the generalized
keystore mechanism introduced in elastic/x-pack-elasticsearch#1942.

Original commit: elastic/x-pack-elasticsearch@a9b6e870c1
2017-07-19 09:17:24 -05:00
Alexander Reelsen a64f9afa30 Tests: Ensure correct client is picked when security is enabled
Original commit: elastic/x-pack-elasticsearch@67d05553f8
2017-07-19 16:11:03 +02:00
markharwood ffe6966ccd Test fix - ensure forceMerged to single segment to avoid scoring variations in tests.
Removed test muting.

Original commit: elastic/x-pack-elasticsearch@3a584aadda
2017-07-19 13:27:07 +01:00
Alexander Reelsen 5e6c56bfc1 Watcher: Restore old WatcherStatsAction for BWC (elastic/x-pack-elasticsearch#2022)
To achieve backwards compatibility the easiest way is
to restore the old watcher stats, which are supposed to run
on the master node only.

The distributed watcher stats have been moved under the statsdist
package and the action name has been changed as well.

This way there is no need to have a serialization BWC layer,
we can just call different actions.

Note: With the current approach developers still need to change
their java applications if they try to receive watcher stats,
as by default we are now using the distributed stats in the
watcher client.

Original commit: elastic/x-pack-elasticsearch@49b3a45452
2017-07-19 13:34:14 +02:00
Simon Willnauer aeed4cb3e4 Followup refactoring for elastic/elasticsearch#25787
Original commit: elastic/x-pack-elasticsearch@ab0e5c45ef
2017-07-19 12:30:27 +02:00
David Roberts ac46b0b0a5 [ML] Include closing jobs in node capacity check (elastic/x-pack-elasticsearch#2034)
Closing jobs can still use some or all of the threads that communicate
with the C++ process - the number of threads used will decrease as the
close progresses, but at the beginning of the closure all are still in
use.  Therefore, to prevent the risk of EsRejectedExecution exceptions
for the autodetect communications threadpool, closing jobs need to be
considered when checking that enough threads exist to start a new
process.  An explicit check produces a much more understandable error
message than an EsRejectedExecution exception.

relates elastic/x-pack-elasticsearch#1364

Original commit: elastic/x-pack-elasticsearch@845bfe0188
2017-07-19 11:25:02 +01:00
David Roberts a41c33dd95 [ML] Allow jobs in the "closing" state to be killed (elastic/x-pack-elasticsearch#2026)
This is important for two reasons:

1. If a job hangs in the closing state for any reason there is now a
   way to get rid of it
2. The force delete endpoint (as used by the UI) killed open jobs before
   deleting them, but could not kill closing jobs, which created the
   possibility that if a job was deleted from the UI while in the closing
   state then the last few results could be indexed after the deletion
   completed

relates elastic/x-pack-elasticsearch#1796

Original commit: elastic/x-pack-elasticsearch@1471106e06
2017-07-19 08:54:24 +01:00
Tim Vernum 1bbc579cf3 [Security] [certgen] Option to generate PKCSelastic/x-pack-elasticsearch#12 (elastic/x-pack-elasticsearch#2013)
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.

These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.

Also adds documentation for all the command-line options in certgen.

Original commit: elastic/x-pack-elasticsearch@d10f88f12d
2017-07-19 12:04:31 +10:00
Tim Sullivan ae62a67e61 [Monitoring] Add Cluster Alert for X-Pack License Expiration (elastic/x-pack-elasticsearch#1998)
* [Monitoring] Add Cluster Alert for X-Pack License Expiration

* work on scripts round 1

* updates per feedback

* spaces

* fix NPE error in transform

* condition to allow updating metadata in the alerts index in every interval

* custom subject message

* update name of indexing actions

* ensure ctx.metadata is updated even if alert is not resolved

* fix omission of absoluteTime

* skip info-level alerts for trial-type license

* move break above `fromNow` declaration

* fix test

Original commit: elastic/x-pack-elasticsearch@f13718f5b5
2017-07-18 15:39:13 -07:00
Andrew Cholakian ebc37feaeb [Logstash] new pipeline viewer schema (elastic/x-pack-elasticsearch#1845)
New metric types for logstash pipeline viewer.

Original commit: elastic/x-pack-elasticsearch@8e44b1fa5a
2017-07-18 11:28:01 -05:00
Alexander Reelsen 139513fdd3 Tests: Fix WatchBackwardsCompatibilityIT temporarily
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests

Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
2017-07-18 18:00:55 +02:00
Ali Beyad 37cc602aef Adds upgrade API functionality for security (elastic/x-pack-elasticsearch#2012)
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.

Original commit: elastic/x-pack-elasticsearch@4abe9f1263
2017-07-18 11:44:28 -04:00
Jay Modi 8b608ef23b Restricted trust config delegates files to monitor to wrapped trust configuration (elastic/x-pack-elasticsearch#2017)
This change makes the restricted trust configuration delegate the list of files to monitor to the
trust configuration that it wraps so that all files that should be monitored for changes are
monitored for changes.

Relates elastic/x-pack-elasticsearch#1919

Original commit: elastic/x-pack-elasticsearch@227db92ac0
2017-07-18 08:52:04 -06:00
Luca Cavanna 7c58130eb2 Wipe security index using its concrete name (elastic/x-pack-elasticsearch#2011)
We were catching IndexNotFoundException, which was hiding the fact that delete index and update aliases APIs don't accept aliases anymore. Now that the exception changed this problem popped up. We now rather call get index providing .security as index name, then delete the concrete indices returned in the response.

Original commit: elastic/x-pack-elasticsearch@18f64f9a41
2017-07-18 15:41:32 +02:00
Igor Motov b4031ee96f Upgrade API: add support for wait_for_completion in upgrade request (elastic/x-pack-elasticsearch#2019)
The wait_for_completion can be now specified with upgrade request to make it async

Original commit: elastic/x-pack-elasticsearch@b768a13ebd
2017-07-17 17:06:16 -04:00
Jay Modi 653b927628 LDAP calls that create a new connection use privilegedConnect (elastic/x-pack-elasticsearch#2018)
This change fixes some cases where calls to the LDAP library can result in a new connection being
created that were not wrapped in privileged connect calls. This would result in permission denied
errors when trying to make the connection.

Original commit: elastic/x-pack-elasticsearch@182c790dd4
2017-07-17 13:15:12 -06:00
Adrien Grand 44c9bba39c Remove `randomDynamicTemplates` usage.
It has been removed it core.

Original commit: elastic/x-pack-elasticsearch@ba9fd16ed6
2017-07-17 16:55:39 +02:00
Tim Brooks 1abc40c645 Rename elastic bootstrap password setting (elastic/x-pack-elasticsearch#2009)
This is related to elastic/x-pack-elasticsearch#1991.

Original commit: elastic/x-pack-elasticsearch@b265211e4a
2017-07-14 16:41:42 -05:00
jaymode 1a6e6411cf Test: use the same settings object to construct tribe settings
This changes the SecurityTribeIT tests to use the same settings object when creating the settings
for the tribe node. Previously two different objects were being created and we would read regular
settings from one and secure settings from another. This causes problems since randomization means
that there may be settings added on the first call that do not get added on the second call. One
example is the randomization of when to only use a keystore or to use both a keystore and a
truststore. On the first call, we would add settings for both a keystore and a truststore but on
the second call only the keystore settings were added. This lead to failures as we would not be
able to open a password protected truststore since the password was never added to the secure
settings.

relates elastic/x-pack-elasticsearch#2005

Original commit: elastic/x-pack-elasticsearch@bbdb3ec662
2017-07-14 14:12:32 -06:00
David Roberts 8365038584 [TEST] Mute failing test: SSLTrustRestrictionsTests testRestrictionsAreReloaded
See elastic/x-pack-elasticsearch#2007

Original commit: elastic/x-pack-elasticsearch@1deb4a7162
2017-07-14 16:30:47 +01:00
Jay Modi 6b4468ea5c Clear security caches on security index health changes (elastic/x-pack-elasticsearch#1957)
This change clears the caches in the native realm and the composite roles store when there is a
a change in the health of the security index that necessitates this. When the security index goes
to a red state, the caches are left in tact as this allows for management operations to be
performed for a limited amount of time. When the index transitions out of the red state or exists
when it didn't exist before, the caches will be cleared so that we remove any stale values.

relates elastic/x-pack-elasticsearch#1789

Original commit: elastic/x-pack-elasticsearch@914959ea6b
2017-07-14 09:28:28 -06:00
Tim Brooks 8ab167cccb Fix ReservedRealm test for failed authentication
Original commit: elastic/x-pack-elasticsearch@5759d9268c
2017-07-14 09:28:44 -05:00
Yannick Welsch dbbec0d37e Let primary own its replication group
Companion commit for elastic/x-pack-elasticsearch#25692

Original commit: elastic/x-pack-elasticsearch@ed93c56f07
2017-07-14 13:51:43 +02:00
David Roberts 81ec1a7ba5 [TEST] Mute failing test
See elastic/x-pack-elasticsearch#2003

Original commit: elastic/x-pack-elasticsearch@5f3611b1d3
2017-07-14 09:49:30 +01:00
Tim Vernum 1686add7ce The configured role-mapping file must be valid. (elastic/x-pack-elasticsearch#1940)
This adds a bootstrap-check that makes it an error to configure a role mapping file that doesn't exist or cannot be parsed.

We are still lenient on dynamic reload because
(a) killing a running node is quite drastic
(b) file writes aren't atomic, so we might be picking up a file that is half way through being written (etc).

If you rely on the default role mapping filename, then it doesn't need to exist (because you might be using the role mapping API instead) but if it does exist it has to parse successfully

Original commit: elastic/x-pack-elasticsearch@5424dea4c4
2017-07-14 15:04:26 +10:00
Tim Vernum d57e38fbed [LDAP] Support explicit "dn" attribute in group search (elastic/x-pack-elasticsearch#1995)
The default for group_search.attribute is to search by DN, but explicitly setting that to dn
wouldn't work because the DN is returned in a special value in the result, and not in the attributes list.

This change detects when user_attribute is set to dn and treats it the same way as the default value.

Original commit: elastic/x-pack-elasticsearch@1933410a0b
2017-07-14 13:12:28 +10:00
Tim Vernum 44a104cb26 [TESTS] Fix SSLTrustRestriction IntegrationTest
- Changes the reloading test to always trust the "trusted" cert so that the health-ping works
- Adds some more logging in case we get new failures

Original commit: elastic/x-pack-elasticsearch@993bf9c721
2017-07-14 13:10:29 +10:00
Tim Brooks 6d04eacdec Require elastic password be bootstrapped (elastic/x-pack-elasticsearch#1962)
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.

Additionally, this commit updates the tests to work with this
functionality.

Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
2017-07-13 19:59:50 -05:00
Igor Motov 4de6d9ebe5 Upgrade API: upgrade assistance shouldn't throw 404 on an empty cluster (elastic/x-pack-elasticsearch#1997)
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.

Original commit: elastic/x-pack-elasticsearch@2ea9707867
2017-07-13 17:01:50 -04:00
Igor Motov dd11fc3d0a Upgrade API: fix parent task propagation for upgrade (elastic/x-pack-elasticsearch#1986)
Ensures that parent task is propagated to child operations to ensure that reindex operation can be cancelled if needed.

Original commit: elastic/x-pack-elasticsearch@fa40b5a951
2017-07-13 16:25:38 -04:00
jaymode 20c06578f6 Do not add domain to the bind user when it is a DN
This change fixes the creation of the bind DN string for active directory realms so that they do
not add the `@domain` suffix to the bind DN, when it is a actual DN value.

Original commit: elastic/x-pack-elasticsearch@bd04c07e16
2017-07-13 12:15:08 -06:00
Chris Earle f4b9dff71a [Monitoring] Support new Kibana mappings for Detecting Cloud (elastic/x-pack-elasticsearch#1989)
This adds the Cloud detection mappings so that phone home can take advantage of their existence.

It also sets the system load fields to use `half_floats`.

Original commit: elastic/x-pack-elasticsearch@75f7992d38
2017-07-13 13:59:40 -04:00
Chris Earle 924ff446bf [Monitoring] Stop checking for old alerts in upgraded version (elastic/x-pack-elasticsearch#1992)
This stops checking for older alerts now that we support emailing based on state changes. This only applies to 6.0 because the upgrade _can_ still happen usefully in 5.6 without the noise.

Original commit: elastic/x-pack-elasticsearch@9d73c64daa
2017-07-13 13:24:32 -04:00
Jay Modi 6fdad6039f Allow the Active Directory UPN authenticator to work with suffixes (elastic/x-pack-elasticsearch#1958)
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.

relates elastic/x-pack-elasticsearch#1744

Original commit: elastic/x-pack-elasticsearch@de00c4817e
2017-07-13 10:08:22 -06:00
Tim Vernum c753ddf7f2 Enable TLS trust restrictions by subject alternative name (elastic/x-pack-elasticsearch#1919)
Optional restrictions can be applied on top of an existing certificate trust scheme (PEM CAs, JKS TrustStore etc).
The restrictions reduce the set of certificate that would be otherwise trusted.

In this commit, the only supported restrictions are to filter by the certificate's SubjectAlternativeNames that are tagged as "other name" with an object-identifier of "cn"

Original commit: elastic/x-pack-elasticsearch@c6105a47df
2017-07-13 23:45:00 +10:00
Colin Goodheart-Smithe 6abe6e5b54 Changes to support removal of Field stats API (elastic/x-pack-elasticsearch#1953)
* Changes to support removal of Field stats API

* iter

Original commit: elastic/x-pack-elasticsearch@bac97b8701
2017-07-13 11:56:43 +01:00
Luca Cavanna 56cfaf8cb9 Rename client artifacts (elastic/x-pack-elasticsearch#1985)
Adapt to upstream rename of rest and sniffer artifacts

Original commit: elastic/x-pack-elasticsearch@f43368b3fb
2017-07-13 09:44:53 +02:00
Tim Vernum e4c8851a24 Convert Realm.authenticate to provide a richer result (elastic/x-pack-elasticsearch#1932)
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.

Original commit: elastic/x-pack-elasticsearch@f796949cfb
2017-07-13 14:24:08 +10:00
Simon Willnauer 84ee21ed26 Followup for elastic/elasticsearch#25658 (elastic/x-pack-elasticsearch#1984)
This is the xpack side fo elastic/elasticsearch#25658 which is mainly refactorings
of a ctor and added tests.

Original commit: elastic/x-pack-elasticsearch@d8e2a2a057
2017-07-12 22:19:34 +02:00
Jay Modi e686d8a3bf Add active directory bind user and user lookup support (elastic/x-pack-elasticsearch#1956)
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.

relates elastic/x-pack-elasticsearch#179

Original commit: elastic/x-pack-elasticsearch@40b07b3422
2017-07-12 14:01:39 -06:00
Brandon Kobel ef25568b2a Removing indices privileges from the reporting_user (elastic/x-pack-elasticsearch#1806)
* Removing indices privileges from the reporting_user

* Updating tests

* Adding comment

Original commit: elastic/x-pack-elasticsearch@ba7688b1dd
2017-07-12 15:56:35 -04:00
Alexander Reelsen e64cf23b13 Watcher: Start watcher on master node only with mixed versions (elastic/x-pack-elasticsearch#1983)
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.

Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.

Original commit: elastic/x-pack-elasticsearch@65cecb6d69
2017-07-12 20:58:47 +02:00
Tim Sullivan 51adbf0f51 [Monitoring] Update template version to 6.0.0 beta1 version (elastic/x-pack-elasticsearch#1948)
* [Monitoring] Update template version to 6000003

* update LAST_UPDATED_VERSION

* use the 6.0.0-beta1 version constant

* use the 6.0.0-beta1 version value (6000026)

* fix cluster uuid reference in terms filters

Original commit: elastic/x-pack-elasticsearch@40c822d396
2017-07-12 09:19:54 -07:00
David Roberts 176786de54 [ML] Improve mapping clash error message (elastic/x-pack-elasticsearch#1968)
relates elastic/x-pack-elasticsearch#1751

Original commit: elastic/x-pack-elasticsearch@01c221bf42
2017-07-12 15:58:37 +01:00
Jack Conradson 32dbfba0c2 Disallow lang to specified in requests where a stored script is used. (elastic/x-pack-elasticsearch#1949)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.

Original commit: elastic/x-pack-elasticsearch@926a7b2d86
2017-07-12 07:56:08 -07:00
Igor Motov 41f30b0ae4 Upgrade API: add common upgrade check method (elastic/x-pack-elasticsearch#1970)
Moves internal index format check to a central location.

Original commit: elastic/x-pack-elasticsearch@0869baac51
2017-07-12 08:46:59 -04:00
Hendrik Muhs f5cfb98ea2 do not update mapping if it is newer than the current version (elastic/x-pack-elasticsearch#1971)
In case of mixed version in a cluster the update could potentially downgrade the mappings. This CL changes the check to avoid this situation.

Original commit: elastic/x-pack-elasticsearch@9ecd5df70e
2017-07-12 09:19:06 +02:00
Chris Earle 753690bf5c [TEST][Monitoring] Use specific LAST_UPDATED constant to avoid issues when they're not the same (elastic/x-pack-elasticsearch#1976)
Original commit: elastic/x-pack-elasticsearch@a41bff564a
2017-07-11 20:47:36 -04:00
Chris Earle 44eda0962e [Monitoring] Use ctx.metadata.xpack not just ctx.metadata (elastic/x-pack-elasticsearch#1974)
This tweaks a recent fix so that it uses the right metadata variable.

Original commit: elastic/x-pack-elasticsearch@af9d9b0e41
2017-07-11 18:04:38 -04:00
Jason Tedor 5ed683538d Name the log tail thread and mark it as a daemon
This commit provides an explicit name for the log tail thread, otherwise
the thread shows up with a generic name like Thread-12 in stack
dumps. While the stack trace provides enough information to conclude
what this thread is doing, it is better to be more explicit about the
purpose of this thread so that that can be discerned directly from the
name of the thread.

Additionally, we mark this thread as a daemon thread. Since this thread
is created by the main thread during node initialization it defaults to
being a user thread. Since the JVM only exits when the only threads
running are not user threads, if this thread were to somehow block then
it could prevent the server JVM from exiting. As such, this thread
should be marked as a daemon thread.

Relates elastic/x-pack-elasticsearch#1969

Original commit: elastic/x-pack-elasticsearch@b3761c7d91
2017-07-11 16:05:07 -04:00
Tim Brooks c22f618a1a Modify password tool to pull password from keystore (elastic/x-pack-elasticsearch#1951)
This is related to elastic/x-pack-elasticsearch#1217. This commit modifies the password setup tool
to pull the password for the `elastic` user from the keystore.

Original commit: elastic/x-pack-elasticsearch@fbc71f393d
2017-07-11 12:50:53 -05:00
Igor Motov 9030c3ae77 Upgrade API: remove kibana upgraders (elastic/x-pack-elasticsearch#1963)
Also removes handling of extra parameters in the upgrade api

Original commit: elastic/x-pack-elasticsearch@9edb825da7
2017-07-11 11:29:31 -04:00
Jay Modi 03ed2bbbd0 Add setting for the LDAP user search filter and deprecate user attribute (elastic/x-pack-elasticsearch#1959)
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.

relates elastic/x-pack-elasticsearch#1861

Original commit: elastic/x-pack-elasticsearch@e9d797e81c
2017-07-11 09:27:24 -06:00
Alexander Reelsen 22b0560ccb Watcher: Ensure no template deletion race in mixed cluster environment (elastic/x-pack-elasticsearch#1964)
When having a mixed cluster with 5.6 and 6.0 nodes, the template upgrade
service has a cluster state listener that deletes the old watches and
triggered_watches index template. However during that time the 5.6 node
WatcherIndexTemplateRegistry checks if the templates are missing and
adds them back. This results in a race, because the new .watches index
template does not get added by the WatcherIndexTemplateRegistry when the
6.0 node is node a master node.

This commit circumvents this issue, by only deleting the watches and
triggered watches template during the upgrade process.

Original commit: elastic/x-pack-elasticsearch@71380f460a
2017-07-11 16:50:41 +02:00
David Roberts 5288791f41 [ML] Specify only concrete indices during deletion (elastic/x-pack-elasticsearch#1966)
Following elastic/elasticsearch#23997 this was only working due to
the way we were suppressing certain errors during job deletion.

This PR makes the situations we want to ignore during job deletion
clearer and adheres to the intention of elastic/elasticsearch#23997
by only specifying concrete indices to the `indices` arguments of
deletion calls.

Original commit: elastic/x-pack-elasticsearch@2458c3db40
2017-07-11 14:07:48 +01:00
Adrien Grand 2e7c725d03 Fix compile errors in src/test.
Original commit: elastic/x-pack-elasticsearch@2ab67bbbdc
2017-07-11 14:31:35 +02:00
Alexander Reelsen 82499b6e55 Fixing compilation issue after lucene snapshot update
Original commit: elastic/x-pack-elasticsearch@1d9e73ad9a
2017-07-11 14:27:21 +02:00
Yannick Welsch 6b02197ca1 [TEST] Ensure stable cluster before disabling licensing in testEnableDisableBehaviour
If the cluster is still making cluster state updates while disabling the license, cluster state updates might not go through,
triggering an assertion failure at the end of the test that checks if all cluster states have been applied.

relates elastic/x-pack-elasticsearch#1627

Original commit: elastic/x-pack-elasticsearch@e11863fd02
2017-07-11 11:48:27 +02:00
Simon Willnauer 16f2de017d Trivial refactorings after elastic/elasticsearch#25627
Original commit: elastic/x-pack-elasticsearch@1cb9c2f959
2017-07-11 11:42:51 +02:00
Hendrik Muhs 261d65a9c4 [ML] add meta data field into mappings and update procedure (elastic/x-pack-elasticsearch#1925)
Adds a update procedure for ML index mappings in order to allow adding new fields. The ES version is stored in the "_meta" field of the ML mappings which then get applied to every index. When opening a job, this version is checked on the (shared) index and a mapping update is performed in case that the version is older than current.

Original commit: elastic/x-pack-elasticsearch@211608c7ad
2017-07-11 07:44:41 +02:00
Tal Levy 4fdfa5cf6f add index checks (elastic/x-pack-elasticsearch#1938)
* add index checks

The following checks apply to both 5.x as well as 6.0

index checks:
- coercion of boolean fields
- the _all meta field is now disabled by default
- the include_in_all mapping parameter is now disallowed
- unrecognized match_mapping_type options not silently ignored
- similarity settings
- store settings
- store throttling settings
- shadow replicas have been removed

Original commit: elastic/x-pack-elasticsearch@5583b21763
2017-07-10 12:19:02 -07:00
Tim Brooks f9eabcdf08 Load bootstrap elastic user password from keystore (elastic/x-pack-elasticsearch#1942)
This is related to elastic/x-pack-elasticsearch#1217. This commit adds a ClusterStateListener at
node startup. Once the cluster and security index are ready, this
listener will attempt to set the elastic user's password with the
bootstrap password pulled from the keystore. If the password is not in
the keystore or the elastic password has already been set, nothing will
be done.

Original commit: elastic/x-pack-elasticsearch@7fc4943c45
2017-07-10 11:15:39 -05:00
Alexander Reelsen 1a57120a6b Security: Add index template permissions for system user (elastic/x-pack-elasticsearch#1937)
As the TemplateUpgradeService requires permissions to add and
delete index template, we have to grant those to the _system user.

This commit adds such permissions plus an integration test.

Original commit: elastic/x-pack-elasticsearch@a76ca9c738
2017-07-10 16:15:13 +02:00
Colin Goodheart-Smithe f70f432695 hanges to accompany agg package structure change in ES
Original commit: elastic/x-pack-elasticsearch@5b79242656
2017-07-10 11:57:11 +01:00
Alexander Reelsen bc5be696e4 Monitoring: Fix cluster monitoring watches to prevent compilations (elastic/x-pack-elasticsearch#1944)
The monitoring watches are roughly executing the same queries even when
they run against different clusters. However the way they were created,
where the cluster name is replaced via search & replace instead of using
watch metadata implies, that every watch is different from a script
compilation cache perspective. On top of that every of those watches is
executed once a minute. So if a new node becomes master and you monitor
three clusters, this results in a fair share of compilations in the first minute. The
reason for the compilation is the fact, that the search input uses
mustache for being able to add dynamic parts into a search using
mustache.

Several of those watches also need to compile more than one search
request.

The maximum default value for script compilations is only 15 and thus at
least one watch will not be executed due to failing script compilations.

This commit changes the four watches, so that the search requests are
cacheable. This means, no matter how many clusters you monitor, there
will be only needed four compilations for the different watches and
that's it.

Relates elastic/support-dev-help#2090

Original commit: elastic/x-pack-elasticsearch@6c877421bb
2017-07-10 09:20:16 +02:00
Jason Tedor c0305727c2 Bump version to 6.0.0-beta1
This commit follows core where the version was bumped from 6.0.0-alpha3
to 6.0.0-beta1 and the 6.0.0-alpha3 version constant was replaced by
6.0.0-beta1.

Relates elastic/x-pack-elasticsearch#1950

Original commit: elastic/x-pack-elasticsearch@a7816fec70
2017-07-09 18:12:31 -04:00
Igor Motov 37075bd201 Upgrade API: Add end-to-end bwc test for watcher upgrade (elastic/x-pack-elasticsearch#1939)
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.

Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
2017-07-09 11:21:57 -04:00
Nik Everett 65a0d44b2f Handle "current" being dropped from the list of released versions
Original commit: elastic/x-pack-elasticsearch@46668ce672
2017-07-07 16:01:21 -04:00
Nik Everett 1af9f56834 Remove `IndexResponse.created`
Core removed this in elastic/elasticsearch#25516.

Original commit: elastic/x-pack-elasticsearch@b1e7040fa2
2017-07-07 13:57:43 -04:00
Lisa Cawley 2aa53f253f [DOCS] Update doc links for graph API (elastic/x-pack-elasticsearch#1884)
Original commit: elastic/x-pack-elasticsearch@9eebb6c9a6
2017-07-07 10:11:06 -07:00
Tim Sullivan 66ebdff447 [Monitoring] Email actions for Cluster Alerts (elastic/x-pack-elasticsearch#1879)
* [Monitoring] Email actions for Cluster Alerts

* fix quotations in email fields

* move email vars to transform, and rename for snake_case

* add state to email subject for cluster status alert

* remove types field in kibana_settings search

* simplify email action condition script

* uppercase the state for the email subject

* only append state to email subject if alert is new

* show state in email subject even when alert is resolved

Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
2017-07-07 09:55:45 -07:00
Simon Willnauer c87d9278a6 Add validation for all `transport.profile.*` settings (elastic/x-pack-elasticsearch#1909)
Follow-up from elasticsearch/elastic#25508

Original commit: elastic/x-pack-elasticsearch@fe08e74ccc
2017-07-07 09:41:34 +02:00
Jason Tedor b636dcc366 Upgrade to Netty 4.1.13.Final
This commit updates the SHAs here as core upgraded the Netty dependency
from 4.1.11.Final to 4.1.13.Final.

Relates elastic/x-pack-elasticsearch#1936

Original commit: elastic/x-pack-elasticsearch@d1803fc331
2017-07-06 15:37:17 -04:00
Lisa Cawley 0e322b525d [DOCS] Update doc links for security APIs (elastic/x-pack-elasticsearch#1883)
Original commit: elastic/x-pack-elasticsearch@660410008c
2017-07-06 11:06:03 -07:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Dimitris Athanasiou 4e03c657a3 [ML] Write model_memory_limit with units into the cluster state (elastic/x-pack-elasticsearch#1934)
This is step 2 of elastic/x-pack-elasticsearch#1604

This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.

Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
2017-07-06 15:40:19 +01:00
David Roberts 984d2ca2ba [ML] Ignore unknown fields when parsing ML cluster state (elastic/x-pack-elasticsearch#1924)
ML has two types of custom cluster state:

1. jobs
2. datafeeds

These need to be parsed from JSON in two situations:

1. Create/update of the job/datafeed
2. Restoring cluster state on startup

Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features.  This is because the
parser was very strict.  This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.

This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state.  Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.

relates elastic/x-pack-elasticsearch#1732

Original commit: elastic/x-pack-elasticsearch@754e51d1ec
2017-07-06 13:36:11 +01:00
Dimitris Athanasiou adc6fd5a0f [ML] Parse model_memory_limit both as number and string with units (elastic/x-pack-elasticsearch#1921)
This is the first step for elastic/x-pack-elasticsearch#1604.

Original commit: elastic/x-pack-elasticsearch@70010a216d
2017-07-06 12:29:12 +01:00
Lee Hinman b66560fa85 [TEST] Use better REST endpoints for testing watch actions
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437

It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.

The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.

Original commit: elastic/x-pack-elasticsearch@3be78d9aea
2017-07-05 10:55:25 -06:00
Igor Motov c035adb568 Rename upgrade.* client methods to migration.* (elastic/x-pack-elasticsearch#1881)
This makes client names consistent with REST APIs and makes it simplifies client development.

Original commit: elastic/x-pack-elasticsearch@90913f485b
2017-07-05 09:47:58 -04:00
Dimitrios Athanasiou 3057f7f4b6 [TEST] Fix post data request in post_data.yml
Original commit: elastic/x-pack-elasticsearch@14e9083c02
2017-07-05 12:06:53 +01:00
Dimitris Athanasiou 15f9b1ed9c [ML] Impove mechanism for ignoring maintenance windows (elastic/x-pack-elasticsearch#1914)
Currently, the autodetect process has an `ignoreDowntime`
parameter which, when set to true, results to time being
skipped over to the end of the bucket of the first data
point received. After that, skipping time requires closing
and opening the job. With regard to datafeeds, this does not
work well with real-time requests which use the advance-time
API in order to ensure results are created for data gaps.

This commit improves this functionality by making it more
flexible and less ambiguous.

- flush API now supports skip_time parameter which
sends a control message to the autodetect process
telling it to skip time to a given value
- the flush API now also returns the last_finalized_bucket_end
time which allows clients to resume data searches correctly
- the datafeed start API issues a skip_time request when the
given start time is after the resume point. It then resumes
the search from the last_finalized_bucket_end time.

relates elastic/x-pack-elasticsearch#1913


Original commit: elastic/x-pack-elasticsearch@caa5fe8016
2017-07-05 11:33:42 +01:00
Lisa Cawley 595468a505 [DOCS] Update doc links for watcher APIs (elastic/x-pack-elasticsearch#1880)
Original commit: elastic/x-pack-elasticsearch@47521f1261
2017-07-04 11:16:47 -07:00
Lisa Cawley 04ff94a180 [DOCS] Update doc links for ML APIs (elastic/x-pack-elasticsearch#1882)
Original commit: elastic/x-pack-elasticsearch@cdc45f282d
2017-07-04 11:10:34 -07:00
David Roberts b93890b0df [ML] Add license downgrade warning (elastic/x-pack-elasticsearch#1912)
When applying a license that will cause ML to be disabled, a warning
to this effect is now included in the acknowledgement messages.

relates elastic/x-pack-elasticsearch#1888

Original commit: elastic/x-pack-elasticsearch@e453f20f58
2017-07-04 15:10:18 +01:00
Dimitrios Athanasiou 05a73a32bd [TEST] Correctly verify that aliases were deleted in MlJobIT
This fixes `testDeleteJobAfterMissingAliases` to not fail randomly.
The reason the test was failing is that at some point some aliases
are deleted and the cat-aliases API is called to verify they were
indeed deleted. This was checked by asserting an
index_not_found_exception was thrown by the cat-aliases request.
This was some times working as there were no other aliases. However,
that depends on whether other x-pack features had time to create their
infrastructure. For example, security creates an alias. When other
aliases had the time to be created, the cat-aliases request does not
fail and the test fails.

This commit simply changes the verification that the read/write
aliases were deleted by replacing the cat-aliases request with
two single get-alias requests.

Original commit: elastic/x-pack-elasticsearch@fe2c7b0cb4
2017-07-04 13:14:38 +01:00
Alexander Reelsen fff72256a5 Watcher: Fix wrong logging in reporting attachment parser (elastic/x-pack-elasticsearch#1900)
The logging shows a wrong HTTP response status code from a previous
request. In addition the body now also gets logged, as debugging
is impossible otherwise.

Original commit: elastic/x-pack-elasticsearch@cc998cd587
2017-07-04 13:01:14 +02:00
Christoph Büscher 8d26996afd Remove QueryParseContext (elastic/x-pack-elasticsearch#1895)
This is the x-pack side of elastic/elasticsearch#25486.

Original commit: elastic/x-pack-elasticsearch@c90a3e096b
2017-07-03 17:31:18 +02:00