Commit Graph

1235 Commits

Author SHA1 Message Date
David Kyle 6ba02e8087 [ML] Use the internal client for DeleteModelSnapshotAction (elastic/x-pack-elasticsearch#2407)
Original commit: elastic/x-pack-elasticsearch@7d3cbfa0cf
2017-08-31 18:16:30 +01:00
Tim Vernum 57a07d6b5a Authorize on shard requests for bulk actions (elastic/x-pack-elasticsearch#2369)
* Add support for authz checks at on shard requests

* Add Rest Tests for authorization

* Bulk security - Only reject individual items, rather than a whole shard

* Sync with core change

* Grant "delete" priv in ML smoketest

This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)

Original commit: elastic/x-pack-elasticsearch@830e89e652
2017-08-31 11:49:46 -04:00
Alexander Reelsen 2033b027ed Tests: Ensure watcher is enabled/disabled during tests (elastic/x-pack-elasticsearch#2392)
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.

This fixes the test to ensure that always the same value
is returned and documents this requirement.

relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
2017-08-31 12:55:05 +02:00
Alexander Reelsen 4cada797d7 Watcher: Replace System.currentTimeMillis() with nanotime() (elastic/x-pack-elasticsearch#2393)
Enjoy the luxury of monotonically increasing clocks. So that
the duration will never be zero.

Original commit: elastic/x-pack-elasticsearch@c934ff0adb
2017-08-31 11:57:25 +02:00
Alexander Reelsen 1121a15eb7 Watcher: Ensure emit_stacktraces parameter works in watcher stats (elastic/x-pack-elasticsearch#2399)
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.

This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.

relates elastic/x-pack-elasticsearch#2396

Original commit: elastic/x-pack-elasticsearch@8638df336c
2017-08-31 09:19:07 +02:00
Hendrik Muhs 381bb4ed7d add a test case to ensure NaN's aren't accepted (elastic/x-pack-elasticsearch#2395)
Original commit: elastic/x-pack-elasticsearch@72879c4f7c
2017-08-31 07:41:48 +02:00
David Kyle b818ee285b Revert "[ML] Check influencer names are valid (elastic/x-pack-elasticsearch#2073)"
This reverts commit elastic/x-pack-elasticsearch@75869cacb3.

Original commit: elastic/x-pack-elasticsearch@1f582ed260
2017-08-30 15:19:49 +01:00
Alexander Reelsen 03ec9df1d8 Tests: Remove is_true assertion that can return zero
Original commit: elastic/x-pack-elasticsearch@220d87fe4e
2017-08-30 15:46:38 +02:00
Dimitrios Athanasiou 5c145224a5 [ML] Add integrations tests for memory limit compliance
Relates elastic/x-pack-elasticsearch#2098

Original commit: elastic/x-pack-elasticsearch@730835edfa
2017-08-30 14:27:06 +01:00
Alexander Reelsen 200cb6a5b3 Tests: Remove is_true check in REST test
If the duration time was 0 (and this might happen due to
using System.currentTimeMillis), the is_true check
still returns false.

The correct fix will be done later to replace the offending
measurement calls and replace them. Then we can add back this
line.

Original commit: elastic/x-pack-elasticsearch@076a9a37cc
2017-08-30 14:04:24 +02:00
Alexander Reelsen 58d16c3579 Tests: Disable watcher in tribe TribeTransportTestCase
Watcher does not need to run in those tests. Disabling it,
no need for careful stopping then.

Relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@ecca7f2ebd
2017-08-30 13:48:54 +02:00
Alexander Reelsen 80df309bbd Tests: Dont check for execution speed, only that field exists
Execution might be fast in combination with the clock not being counted
fast enough. As we are only interested in the field existing, we dont
need to check for the value.

relates elastic/x-pack-elasticsearch#2391

Original commit: elastic/x-pack-elasticsearch@8f3df967d9
2017-08-30 13:38:50 +02:00
Alexander Reelsen 4ac13fa9c5 Watcher: Replace integration tests with unit and REST tests (elastic/x-pack-elasticsearch#2322)
A bunch of integration tests should have been built as unit tests
or already have unit test equivalents.

This commit removes integration tests as well as adding REST equivalents
or creating unit tests instead of extending from AbstractWatcherIntegrationTestCase

Original commit: elastic/x-pack-elasticsearch@a97b99467d
2017-08-30 11:46:28 +02:00
Alexander Reelsen 17de601e21 Watcher: Remove the BWC compatible old watcher stats (elastic/x-pack-elasticsearch#2323)
Those were needed for the 5.x to 6.x transition phase, but can
be removed for 7.x.

Original commit: elastic/x-pack-elasticsearch@75572bd0c8
2017-08-30 11:45:57 +02:00
Ryan Ernst 3ef93c55b0 Build: Add build metadata generation and usage to xpack (elastic/x-pack-elasticsearch#2368)
This commit adds generation of build_metadata files from bwc checkouts
as well as the sibling ES checkout of the xpack ci script.

relates elastic/elasticsearch#26397

Original commit: elastic/x-pack-elasticsearch@e17d904beb
2017-08-28 14:03:02 -07:00
Jim Ferenczi 27d8b4c79c Remove the _all metadata field (elastic/x-pack-elasticsearch#2356)
This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).

Relates https://github.com/elastic/elasticsearch/pull/26356

Original commit: elastic/x-pack-elasticsearch@a47133c94e
2017-08-28 13:01:27 +02:00
Michael Basnight e18f04f3eb Revert "Use shaded rest client dependencies" (elastic/x-pack-elasticsearch#2352)
This reverts commit elastic/x-pack-elasticsearch@8605560232.

Relates elastic/elasticsearch#26367

Original commit: elastic/x-pack-elasticsearch@e4cd960504
2017-08-25 14:13:16 -05:00
Alexander Reelsen 9b0a1a34e0 Upgrade: Remove watcher/security upgrade checks (elastic/x-pack-elasticsearch#2338)
The checks are used for the 5.6 to 6.x transition, thus they do
not make sense to keep in 7.x.

Original commit: elastic/x-pack-elasticsearch@c6c6fa819e
2017-08-25 17:24:49 +02:00
David Kyle 5a63c4d9a2 [ML] Remove trailing slashes from ML rest spec (elastic/x-pack-elasticsearch#2350)
Original commit: elastic/x-pack-elasticsearch@f2a3d70562
2017-08-25 09:35:07 +01:00
David Kyle 175e8db3aa [ML] Don’t count incomplete buckets in data stream diagnostics (elastic/x-pack-elasticsearch#2351)
* Don’t count incomplete buckets in data stream diagnostics

* Fix tests now bucket_count doesn’t include partial buckets

Original commit: elastic/x-pack-elasticsearch@bc1a7bd9e7
2017-08-25 09:25:15 +01:00
Ryan Ernst 13672dad13 Build: Set xpack to require keystore
See https://github.com/elastic/elasticsearch/pull/26329

Original commit: elastic/x-pack-elasticsearch@e77361a6d5
2017-08-24 14:09:07 -07:00
Jay Modi 8e4ded82c0 Adapt to resync rename and add assertion for missing user when sending a request (elastic/x-pack-elasticsearch#2345)
This commit adapts to the renaming of the TransportResyncReplicationAction in core and also adds
an assertion to the check for a user being present when sending a request. The assertion is added
so that we can hopefully catch these scenarios in our testing as the assertion error will cause the
node to die but the ISE will just be seen in the logs. Since we do not run with assertions enabled
in production, the ISE is left to handle those cases.

relates elastic/x-pack-elasticsearch#2335

Original commit: elastic/x-pack-elasticsearch@c5ce0c93af
2017-08-24 11:04:41 -06:00
Colin Goodheart-Smithe d0103d6cab [TEST] Fixes PageParamsTests to no underflow from and size
In `mutateInstance()` the from or size could become negative if the other one was pushed over the limit for `from + size`. This change fixes this case to make sure after the mutate method is called the from and size obey the limit but are also both `>= 0`

relates elastic/x-pack-elasticsearch#2344

Original commit: elastic/x-pack-elasticsearch@a8a7072fcc
2017-08-24 09:29:36 +01:00
Christoph Büscher f05568e7b3 Revert adding bin to .gitignore (elastic/x-pack-elasticsearch#2346)
Original commit: elastic/x-pack-elasticsearch@cfd2be3831
2017-08-23 22:43:01 +02:00
Jason Tedor 85e494d90f Add x-pack-env.bat
This file was missing from the commit which depends on this file due to
a .gitignore rule. This commit adds this vital file.

Relates elastic/x-pack-elasticsearch#2124

Original commit: elastic/x-pack-elasticsearch@78125b56de
2017-08-23 14:16:36 -04:00
Stacey Gammon 1bfa4cb8bb Add reserved dashboards_only_user role (elastic/x-pack-elasticsearch#2250)
* Add reserved dashboards_only_user role

* Fix line too long

* add tests for new reserved role

* rename role, hopefully fix tests

* Fix test

Original commit: elastic/x-pack-elasticsearch@99f6718c7c
2017-08-23 13:16:17 -04:00
David Kyle 71063825be [MLFix bucket setting count (elastic/x-pack-elasticsearch#2339)
Original commit: elastic/x-pack-elasticsearch@6801e90d54
2017-08-23 15:46:51 +01:00
Alexander Reelsen 6ee4fe6a0b Watcher: Improve upgrade API logging message to include index names
Original commit: elastic/x-pack-elasticsearch@9cecc11f88
2017-08-23 16:11:49 +02:00
Albert Zaharovits 026729e911 TOKEN_SERVICE_ENABLED_SETTING enabled if HTTP_SSL_ENABLED (elastic/x-pack-elasticsearch#2321)
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.

* TokenService default enabled if HTTP_ENABLED == false

* Fixed tests that need TokenService explicitly enabled

* [DOC] Default value for `xpack.security.authc.token.enabled`

Original commit: elastic/x-pack-elasticsearch@bd154d16eb
2017-08-23 13:21:30 +03:00
Colin Goodheart-Smithe de6c275dca Refactor/to x content fragments2 (elastic/x-pack-elasticsearch#2329)
* Moves more classes over to ToXContentObject/Fragment

* Removes ToXContentToBytes

* Removes ToXContent from Enums

* review comment fix

* slight change to use XContantHelper

Original commit: elastic/x-pack-elasticsearch@0f2d3f328b
2017-08-23 08:17:28 +01:00
David Kyle 18d15ef2d2 Revert "[ML] Missing validations in analysis config (elastic/x-pack-elasticsearch#2103)"
This reverts commit elastic/x-pack-elasticsearch@461be12f9f.

Original commit: elastic/x-pack-elasticsearch@1ce7196710
2017-08-22 13:33:40 +01:00
Yannick Welsch b4353b55ad Allow build to directly run under JDK 9 (elastic/x-pack-elasticsearch#2320)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

Relates to elastic/elasticsearch#25859

Original commit: elastic/x-pack-elasticsearch@4bf266e0b0
2017-08-22 14:46:37 +09:30
Alexander Reelsen 27f39c615b Watcher: Create two index ugprade checks for watcher upgrade (elastic/x-pack-elasticsearch#2298)
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.

There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.

This also means, that there will be two entries popping up in the kibana
UI.

Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.

relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@2c92040ed6
2017-08-21 17:36:16 +02:00
Jason Tedor 8a5be5b58e Replace atomic move writer
I noticed this while working on a previous issue with atomic move writer
(silent swallowing of exceptions). Namely, atomic move writer has
dangerous semantics. The problem is as follows: atomic move writer works
by writing lines to a temporary file, and then in its close method it
replaces the target path with the temporary file. However, the close
method is invoked whether or not all writes to the temporary file
succeeded (because writers obtained from atomic move writer are used in
try-with-resources blocks, as they should be). There is no way to
distinguish that the writer is being closed in a successful scenario
versus a failure scenario. In the close method for atomic move writer,
the target file is replaced by the temporary file. This means that the
target file is replaced whether or not writing to the temporary file
actually succeeded. Since these atomic move writers are used for user
configuration files (users and user_roles), a failure here can lead to
data loss for the user, a tragedy!

There is another (less serious) problem with the atomic move
writer. Since the close method tries to move the temporary file in place
of the existing file, the temporary file can be left behind if there is
another failure in the close method (e.g., closing the underlying file
after writing, or setting the permissions on the temporary file). This
means that in some situations, atomic move writer will leave temporary
files behind (which is not definitively not atomic).

This commit replaces the atomic move writer with a safer mechanism. We
still perform the write atomically in the following sense: we write to a
temporary file. Either writing to that file succeeds or it fails. If
writing succeeds, we replace the existing file with the temporary
file. If writing fails, we clean up the temporary file and the existing
file remains in place.

Relates elastic/x-pack-elasticsearch#2299


Original commit: elastic/x-pack-elasticsearch@3199decb0a
2017-08-21 10:35:37 -04:00
David Roberts e428c58dff [TEST] Fix logic in one branch of random mutate test
Certain types of datafeeds cannot have null chunking configs, so
setting chunking config to null sometimes doesn't stick as null

Original commit: elastic/x-pack-elasticsearch@3a52bad460
2017-08-21 14:51:04 +01:00
Colin Goodheart-Smithe 0a1225f934 [TEST] Adds mutate function to some tests (elastic/x-pack-elasticsearch#2263)
Original commit: elastic/x-pack-elasticsearch@560d7e9a80
2017-08-21 11:20:14 +01:00
Alexander Reelsen 1b6d9d430c Watcher: Load for watch for execution as late as possible (elastic/x-pack-elasticsearch#2151)
When a watch is executed currently, it gets passed an in-memory
watch object, that was loaded, before the execution started.

This means there is a window of time, where an old watch could still
be executing, then a watch gets loaded for execution, then the old watch
execution finishes and updates the watch status and thus reindexes the
watch.

Now the watch, that got loaded for execution, executes and tries to
store its watch status, but fails, because the version of the watch
has changed.

This commit changes the point in time where the watch is loaded. Now
this only happens, while a watch is in its protected execution block,
and thus we can be sure, that there is no other execution of the watch
happening.

This will primarily impact watches, that execute often, but their
runtime is longer than the configured interval between executions.

Side fix: Removed some duplicate testing method and moved into 
WatcherTestUtils, fixed a tests with a ton of if's with random booleans
into separate tests.

relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bf393023d7
2017-08-21 10:43:41 +02:00
Simon Willnauer 44c3d5b3d9 Allow to change bwc refspec in xpack (elastic/x-pack-elasticsearch#2311)
Today it's impossible to run xpack bwc tests against any other branch
or remote than upstream. This allows to pass `-Dtests.bwc.refspec` to
change the refspec to use for the bwc tests.

Original commit: elastic/x-pack-elasticsearch@4d365f5a6e
2017-08-21 09:59:45 +02:00
Simon Willnauer 846f40ba15 [TEST] allow test to start an additional node even if max number of random nodes is reached
relates elastic/x-pack-elasticsearch#2315

Original commit: elastic/x-pack-elasticsearch@d989c06198
2017-08-19 11:03:46 +02:00
Simon Willnauer 71827b70a0 Bump token service BWC version to 6.0.0-beta2
Original commit: elastic/x-pack-elasticsearch@ef688f02cb
2017-08-18 17:06:45 +02:00
David Roberts 4f17335629 [ML] Add unit tests for ML roles (elastic/x-pack-elasticsearch#2309)
I forgot to add these when the roles were originally added

Original commit: elastic/x-pack-elasticsearch@0e72141b11
2017-08-18 14:04:41 +01:00
Simon Willnauer ac9ab974f4 Ensure token service can boostrap itself without a pre-shared key (elastic/x-pack-elasticsearch#2240)
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.

This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.

This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.

Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
2017-08-18 14:23:43 +02:00
David Roberts 269f0f6a19 [ML] Default model memory limit to 1GB for newly created jobs (elastic/x-pack-elasticsearch#2300)
This change means that newly created jobs will get an explicit 1GB
model memory limit if no model memory limit is specified when creating
the job.  Existing jobs that had a null model memory limit will carry
on using the default model memory limit defined in the C++ code.

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@a4e6b73c2b
2017-08-18 09:36:09 +01:00
Jason Tedor 5125978f60 Enable command extensions
We rely on command extensions in our scripts but we do not actually
guarantee that they are enabled (usually they are, by default, but they
can be disabled outside of our control). This commit ensures that they
are enabled.

Relates elastic/x-pack-elasticsearch#2307

Original commit: elastic/x-pack-elasticsearch@a5eec8ca7b
2017-08-17 22:50:52 -04:00
Simon Willnauer 724325f161 Fallback to `keystore.seed` as a bootstrap password if actual password is not present (elastic/x-pack-elasticsearch#2295)
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.

Relates to elastic/elasticsearch#26253

Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
2017-08-17 16:42:32 +02:00
David Roberts 2ae5634dc9 [ML] Improve error message when auto-close isn't attempted (elastic/x-pack-elasticsearch#2296)
The old message of "Cannot auto close job" implied the problem was with
closing the job.  This change makes it clearer that the problem is that
the datafeed could not be stopped and hence auto-close will not even be
attempted.

Original commit: elastic/x-pack-elasticsearch@065e9930ce
2017-08-17 15:07:06 +01:00
David Roberts 44857d71b3 Make AllocatedPersistentTask members volatile (elastic/x-pack-elasticsearch#2297)
These members are default initialized on contruction and then set by the
init() method.  It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.

Original commit: elastic/x-pack-elasticsearch@668121e274
2017-08-17 14:54:31 +01:00
Colin Goodheart-Smithe 751680e7b2 Moves more classes over to ToXContentObject/Fragment (elastic/x-pack-elasticsearch#2283)
Original commit: elastic/x-pack-elasticsearch@73c6802523
2017-08-17 11:16:56 +01:00
Alexander Reelsen 6d30806996 Watcher: Improvements on the rolling restart tests (elastic/x-pack-elasticsearch#2286)
This improves the rolling restart tests (tests different paths in
different ways) and aligns the upgrade code with the 5.6 branch from

Relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@01b0954558
2017-08-17 11:41:11 +02:00
Simon Willnauer 8f15324a08 Don't bootstrap security index on start-up but authenticate bootstrap password locally (elastic/x-pack-elasticsearch#2272)
Today we try to bootstrap the security index with the bootstrap password and recommend the user to change the password with the user tool. This is trappy for instance if you happen to configure multiple nodes with a different bootstrap passwords (which is possible) it's unclear which password made it too bootstrap. Yet, we tell in the logs but it can still be very confusing. In general it should be possible to bootstrap with the user tool from any node unless the user is already created in the native user store. This change uses the bootstrap.password from the local node and always authenticate against it until the user is bootstrapped even if the passwords are different on different nodes. This will also work for authenticating against the cluster for instance if a user deletes the .security index or if that index has not been upgraded.

Original commit: elastic/x-pack-elasticsearch@8cebecb287
2017-08-17 08:36:26 +02:00
Jason Tedor 2bf8f4b0bc Remove print writer wrapping for users tools
When writing the users and users_roles files, we wrap a custom writer in
a print writer. There is a problem with this though: when print writer
closes it closes our underlying custom writer and the close
implementation for our custom writer is not trivial, it executes code
that can throw an I/O exception. When print writer invokes this close
and an I/O exception is thrown, it swallows that exception and sets the
status on the print writer to error. One would think that we could
simply check this status but alas print writer is broken here. The act
of checking the status causes print writer to try to flush the
underyling stream which is going to be completely undefined because the
underlying stream might or might not be closed. This might cause another
exception to be thrown, losing the original. Print writer screwed the
pooch here, there is no good reason to try to do any I/O after the
underlying writer entered a failed state. To address this we remove the
use of print writer, we use our custom writer directly. This allows any
thrown exceptions to bubble up.

Relates elastic/x-pack-elasticsearch#2288

Original commit: elastic/x-pack-elasticsearch@11b8dd5641
2017-08-16 12:50:39 -04:00
David Roberts 6fcc3be438 [ML] Preserve _meta on results index mapping update (elastic/x-pack-elasticsearch#2274)
When mappings are updated for an index are updated most settings are
merged, but not _meta.  This change ensures that _meta is set when we
add per-job term mappings to our results index mappings.  In order to
keep the logic for updating mappings after upgrade working, we now
have to put ALL the mappings for our results along with the latest _meta
section when updating per-job term mappings.

relates elastic/x-pack-elasticsearch#2265

Original commit: elastic/x-pack-elasticsearch@f58c11a13e
2017-08-16 16:16:30 +01:00
Simon Willnauer 8b23f133c7 Create security bootstrap checks early to access secure settings safely (elastic/x-pack-elasticsearch#2282)
We close the secure settings in core before we pull bootstrap checks.
This means if a bootstrap check like the `TokenPassphraseBootstrapCheck`
accesses a secure setting that late it will fail due to an exception in
the `PKCS12KeyStore`. This change moves the bootstrap check creation
to the plugin constructor and adds a dummy setting to the integTest
that triggers the bootstrap checks.

Original commit: elastic/x-pack-elasticsearch@2b20865d1c
2017-08-16 13:01:52 +02:00
Yannick Welsch fd76651d92 Expose timeout of acknowledged requests in REST layer (elastic/x-pack-elasticsearch#2259)
Companion PR to elastic/elasticsearch#26189

Original commit: elastic/x-pack-elasticsearch@f561e22835
2017-08-16 07:43:18 +08:00
David Roberts db8885a46e [ML] Do not download the ml-cpp zip when building it locally (elastic/x-pack-elasticsearch#2262)
When the machine-learning-cpp repo is built locally, the zip file it
creates is preferred over that downloaded from s3 when creating the
overall x-pack-elasticsearch zip.  However, prior to this change the
build would ALSO download an ml-cpp zip from s3, and just not use it.

Original commit: elastic/x-pack-elasticsearch@bd71637edd
2017-08-15 16:31:23 +01:00
Jason Tedor f3a7d46698 Rename CONF_DIR to ES_PATH_CONF
This commit is following upstream Elasticsearch which has renamed the
environment variable used to specify a custom configuration directory
from CONF_DIR to ES_PATH_CONF.

Relates elastic/x-pack-elasticsearch#2261

Original commit: elastic/x-pack-elasticsearch@9ae29941e5
2017-08-15 06:19:39 +09:00
Tim Vernum a27dc257c9 Gracefully handle no content(-type) in Put License (elastic/x-pack-elasticsearch#2258)
PUT /_xpack/license with no content or content-type should fail with an appropriate error message rather than throwing NPE.

Original commit: elastic/x-pack-elasticsearch@f8c744d2a2
2017-08-14 20:39:39 +10:00
Alexander Reelsen 5416a6afd4 Tests: Fix timeout in watcher history template test
Due to an invalid timeout the test failed earlier than it should have.

relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@2265c419e3
2017-08-14 09:53:52 +02:00
David Roberts b5d159bc1c [ML] Handle simultaneous force delete datafeed and stop datafeed (elastic/x-pack-elasticsearch#2243)
This is an important case as the UI force stops datafeeds now.

Fixes elastic/x-pack-kibana#2083

Original commit: elastic/x-pack-elasticsearch@4d0f62ad2d
2017-08-11 14:43:24 +01:00
David Roberts cb3f3d2d04 [ML] Switch from max_running_jobs to xpack.ml.max_open_jobs (elastic/x-pack-elasticsearch#2232)
This change makes 2 improvements to the max_running_jobs setting:

1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
   is not used elsewhere

The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified.  max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@18c539f9bb
2017-08-11 09:00:33 +01:00
Alexander Reelsen 5f30508efd Tests: Remove AwaitsFix annotation and increase logging
This test does not reproduce locally but fails regularly in CI.
Added more logging and proper comments.

Relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@bf6b590629
2017-08-11 09:44:09 +02:00
Alexander Reelsen 11334b2df3 Tests: Fix TimeThrottleIntegrationTests to not rely on shard actions (elastic/x-pack-elasticsearch#2234)
These tests used to fail rarely, because during a watch execution
one of the watcher shards was relocated resulting in a second execution
of watch.

In order to prevent this, the tests do not need to actually create any
shards, which causes watcher potentially to be rebalanced.

This simplifies and speeds up the test as well.

relates elastic/x-pack-elasticsearch#1608

Original commit: elastic/x-pack-elasticsearch@1cfac1145d
2017-08-11 09:19:25 +02:00
Alexander Reelsen 35ee552923 Tests: Fix logger initialization to include hostname in timewarped watcher
This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.

Original commit: elastic/x-pack-elasticsearch@c8c2819d86
2017-08-10 14:09:14 +02:00
Alexander Reelsen ceb13988e3 Watcher: Do not update watch status state during execution (elastic/x-pack-elasticsearch#2204)
When a watch is executed, it sends an update request to the watch to
udpate its status.

This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.

This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.

This will fix not only the current behaviour but also some flaky tests.

Original commit: elastic/x-pack-elasticsearch@ca69109ecb
2017-08-10 13:16:39 +02:00
Alexander Reelsen 4012da662d Tests: Add @AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@545155ec13
2017-08-10 10:17:22 +02:00
Colin Goodheart-Smithe a4dd177978 Migrates ToXContent classes (elastic/x-pack-elasticsearch#2024)
* Migrates ToXContent classes

* review comments

Original commit: elastic/x-pack-elasticsearch@74ce3755ab
2017-08-09 15:55:04 +01:00
Alexander Reelsen 0b5909fc65 Watcher: Stop swallowing exceptions, always return them instead of message (elastic/x-pack-elasticsearch#1933)
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.

This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.

This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy

In addition a few Field interface classes were removed, that only contained parse fields.

relates elastic/x-pack-elasticsearch#1816

Original commit: elastic/x-pack-elasticsearch@b2ce680139
2017-08-08 18:36:22 +02:00
David Roberts 22da5cf89e [ML] Add max open jobs per node as a node attribute (elastic/x-pack-elasticsearch#2203)
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs.  Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster.  However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.

For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@1e62b89183
2017-08-08 16:16:27 +01:00
Dimitris Athanasiou 2864078da2 [ML] Move job group validation after parsing (elastic/x-pack-elasticsearch#2207)
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.

Original commit: elastic/x-pack-elasticsearch@a03f002bdc
2017-08-08 15:59:04 +01:00
David Roberts 6a159d2127 [ML] Fix fallout from bulk action requiring newlines (elastic/x-pack-elasticsearch#2205)
Only unit tests were broken.  Production ML code was always terminating
bulk requests with newlines.

Original commit: elastic/x-pack-elasticsearch@96ed06fed3
2017-08-08 11:07:13 +01:00
Alexander Reelsen 55e88d6857 Watcher: Ignore if template is missing when upgrade is running (elastic/x-pack-elasticsearch#2199)
If one of the old watcher templates does not exist when we try 
to delete it, the upgrade should just continue.

Original commit: elastic/x-pack-elasticsearch@6a52bad329
2017-08-08 10:17:58 +02:00
Chris Earle 1d6f82dbe3 [Monitoring] Remove Dedicated IndicesStatsCollector (elastic/x-pack-elasticsearch#2192)
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.

On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.

Original commit: elastic/x-pack-elasticsearch@93b09878e4
2017-08-07 13:00:41 -04:00
Tim Sullivan 7d86d75aa4 [Monitoring/Cluster Alerts] default admin email is snake_case (elastic/x-pack-elasticsearch#2177)
Original commit: elastic/x-pack-elasticsearch@d6129a0065
2017-08-07 09:15:14 -07:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Chris Earle 87bc215b91 [ML] Use try-with-resources for InputStream (elastic/x-pack-elasticsearch#2182)
Original commit: elastic/x-pack-elasticsearch@88d5e73fec
2017-08-04 13:26:35 -04:00
Alexander Reelsen 36bad2079d Tests: Fix watcher test when aliases were used
Original commit: elastic/x-pack-elasticsearch@d1c38cb85a
2017-08-04 18:01:26 +02:00
Jay Modi 8b0fb5eae8 Re-enable OpenLDAP tests and run against vagrant instance (elastic/x-pack-elasticsearch#2121)
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.

relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@ac9bc82297
2017-08-04 09:44:08 -06:00
David Roberts 39bc5886f2 [ML] Remove record_count from ML mappings (elastic/x-pack-elasticsearch#2183)
record_count is no longer written to new results, but is still tolerated
for backwards compatibility.  However, in the backwards compatibility case
the results index must already contain the required mapping.  There's no
need to add this mapping to newly created results indices.

Original commit: elastic/x-pack-elasticsearch@e586f3ba96
2017-08-04 10:52:35 +01:00
Jay Modi a7d6138f83 Fix the building of the default URL for the setup password tool (elastic/x-pack-elasticsearch#2176)
This commit fixes the building of the default URL for the setup password tool so that a default
elasticsearch.yml file will still result in a succesful run of the tool.

relates elastic/x-pack-elasticsearch#2174

Original commit: elastic/x-pack-elasticsearch@2291b14875
2017-08-03 15:14:24 -06:00
Dimitris Athanasiou 2d9d4c41d8 [ML] Add ability to assign groups to jobs (elastic/x-pack-elasticsearch#2155)
In particular:

  - adds a `groups` field in a job
  - group names can be used in multi-job APIs

relates elastic/x-pack-elasticsearch#2097

Original commit: elastic/x-pack-elasticsearch@c8517221ae
2017-08-03 17:32:05 +01:00
David Kyle 4b531c4dbb [ML] Check histogram interval is a divisor of bucketspan (elastic/x-pack-elasticsearch#2153)
Original commit: elastic/x-pack-elasticsearch@356dfa719c
2017-08-03 11:58:58 +01:00
Alexander Reelsen 65ea1b3bc4 Tests: Remove unneeded classes, fix AwaitsFix watcher tests (elastic/x-pack-elasticsearch#2139)
Fix TemplateTransformMappingTests to work, even if date rolls over
during execution.

Reenable test in BootStrapTests, was forgotten.

Remove the SecurityF/MonitoringF/WatcherF classes, as there is a gradle
command to easily start elasticsearch with xpack

Remove HasherBenchmark, as it is not a test and relies on RandomContext
that is not available anymore (also I think a JMH benchmark would be
needed here).

Remove ManualPublicSmtpServersTester, was not usable anymore.

Remove OldWatcherIndicesBackwardsCompatibilityTests, now in dedicated
rolling upgrade tests.

Remove unused EvalCron class.

Original commit: elastic/x-pack-elasticsearch@100fa9e9b0
2017-08-03 12:51:35 +02:00
Tim Brooks dd899956e2 Use nio transport in x-pack tests (elastic/x-pack-elasticsearch#2159)
This commit is related to elastic/elasticsearch#25986. It updates x-pack
to use the randomized transport work in elasticsearch.

Original commit: elastic/x-pack-elasticsearch@eba2c0f815
2017-08-02 11:29:53 -05:00
Jay Modi 573f365b56 Fix validation of username and password in CreateTokenRequest (elastic/x-pack-elasticsearch#2145)
This change fixes the validation of the the username and password field in the CreateTokenRequest
and adds a unit test to validate the fix.

relates elastic/x-pack-elasticsearch#2127

Original commit: elastic/x-pack-elasticsearch@b870683d39
2017-08-02 07:58:54 -06:00
David Kyle 7795d70414 [ML] Get Jobs/Datafeed Actions can run on local node (elastic/x-pack-elasticsearch#2095)
* No need to execute Get Jobs/Datafeed Actions on master node

Original commit: elastic/x-pack-elasticsearch@9d19010c5f
2017-08-02 11:24:28 +01:00
Dimitris Athanasiou 0125a332a1 [ML] Add support for mutli-job/multi-datafeed APIs (elastic/x-pack-elasticsearch#2079)
This commit enhances job/datafeed APIs that support acting
on multiple jobs/datafeeds at once so that they accept
expressions that may contain comma-separated lists or
wildcard patterns.

The APIs that are enhances are:

  - get jobs API
  - get job stats API
  - close job API
  - get datafeeds API
  - get datafeed stats API
  - stop datafeed API

relates elastic/x-pack-elasticsearch#1876

Original commit: elastic/x-pack-elasticsearch@45a1139d97
2017-08-02 11:10:06 +01:00
David Roberts c09430f3bf [TEST] Fix ML licensing tests following change in core test framework (elastic/x-pack-elasticsearch#2152)
The change made in elastic/elasticsearch#25986 means that half the time
tests will use the NIO transport rather than the mock TCP transport.
But the NIO plugin was not added to the TestXPackTransportClient.

Original commit: elastic/x-pack-elasticsearch@e465b0f801
2017-08-02 09:51:03 +01:00
Luca Cavanna 9fcb230d90 Adapt to upstream changes made to AbstractStreamableXContentTestCase (elastic/x-pack-elasticsearch#2117)
Original commit: elastic/x-pack-elasticsearch@0b1be31ffa
2017-08-02 08:43:09 +02:00
Jay Modi a35234de56 Setup password tool builds default URL from settings (elastic/x-pack-elasticsearch#2146)
This change makes the setup password tool build the default URL from the settings provided by the
environment. This will ease the amount of work a user would have to do in order to run the tool as
http vs https will be selected automatically and the port/host will as well.

Original commit: elastic/x-pack-elasticsearch@79affe4a79
2017-08-01 14:13:08 -06:00
Ryan Ernst 45a55d16cf Bump master version to 7.0.0-alpha1 (elastic/x-pack-elasticsearch#2135)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25876

Original commit: elastic/x-pack-elasticsearch@c86ea25009
2017-08-01 15:48:04 -04:00
Jay Modi ec11799003 Read the token passphrase earlier in the bootstrap check (elastic/x-pack-elasticsearch#2144)
This commit moves the reading of the token passphrase to the creation of the bootstrap check to
avoid issues with the secure settings keystore already being closed and thus causing issues during
startup.

Original commit: elastic/x-pack-elasticsearch@bba1cc832d
2017-08-01 13:04:34 -06:00
David Kyle 466d421abe [ML] Missing validations in analysis config (elastic/x-pack-elasticsearch#2103)
* Don’t set detector field names to empty strings
* Check summary count field and categorisation field names are not empty strings
* Check a detector has a by field when using multivariate by fields

Original commit: elastic/x-pack-elasticsearch@461be12f9f
2017-08-01 15:54:46 +01:00
David Roberts 8487e1e0fe [ML] Better handling of errors if native controller dies (elastic/x-pack-elasticsearch#2141)
If the native controller dies or is killed then requests to open jobs
now immediately return with an error that says what the problem is.
The error that is logged also now clearly records the problem.

Previously open job requests would time out if the native controller
was not running, and logged errors were not easy to understand without
in-depth knowledge of the ML code.

relates elastic/x-pack-elasticsearch#2140

Original commit: elastic/x-pack-elasticsearch@fc7f074d4a
2017-08-01 15:53:58 +01:00
Jay Modi 7291eb55fe Automatically enable AES 256 bit TLS ciphers when available (elastic/x-pack-elasticsearch#2137)
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.

Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
2017-08-01 07:36:35 -06:00
Alexander Reelsen 4bf5d9536a Tests: Remove @ClusterScope tests, create REST tests (elastic/x-pack-elasticsearch#2131)
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods

Original commit: elastic/x-pack-elasticsearch@265966d80c
2017-08-01 14:15:36 +02:00
Alexander Reelsen 547b0ebc1b X-Pack: Remove deprecated handlers, remove watcher restart action (elastic/x-pack-elasticsearch#2133)
The deprecated handlers should have been removed earlier, but are now
going to to away finally.

Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.

Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
2017-08-01 14:14:51 +02:00
Alexander Reelsen 681ce6e2aa Watcher: Replace flaky WatchStatsTests with unit tests (elastic/x-pack-elasticsearch#2118)
This flaky tests was using sleep, latches and a custom script plugin,
causing issues with stopping/starting tests.

This removes the integration tests and replaces it with a unit test.

Also removed a couple of unused ctor/method parameters as cleanup.

relates elastic/x-pack-elasticsearch#1639

Original commit: elastic/x-pack-elasticsearch@2a42faf2db
2017-08-01 09:50:32 +02:00
Alexander Reelsen 9ea032998b Watcher: Remove REST hijacking operations (elastic/x-pack-elasticsearch#2132)
As all the triggering is now done on the shards, people can use
bulk, *-by-query and regular index/delete operations on the watcher
index and do not need to use the dedicated APIs.

We can now remove the long existing rest operation hijacking code.

Original commit: elastic/x-pack-elasticsearch@08f4f4c3de
2017-08-01 09:49:28 +02:00
Jay Modi 7fecca6329 Fix handling of exceptions for concurrent attempts to create security index (elastic/x-pack-elasticsearch#2120)
This commit fixes the handling of some exceptions when we attempt to create the security index and
alias. The issue here is provoked by a test that is currently muted with an AwaitsFix,
GroupMappingTests, which will be unmuted in another change.

Original commit: elastic/x-pack-elasticsearch@55f6b656cb
2017-07-31 12:34:14 -06:00
Luca Cavanna dac28fe4e0 Adapt to AbstractQueryBuilder upstream changes
AbstractQueryBuilder doesn't extend ToXContentToBytes anymore

Original commit: elastic/x-pack-elasticsearch@6c1b948ccb
2017-07-31 17:38:56 +02:00
Jason Tedor c51e6e1cd6 Remove unnecessary directory push/pop from scripts
This commit proposes removing an unnecessary directory push/pop from the
X-Pack scripts. It is not clear exactly why these were added, the
original change was almost three years ago in
elastic/x-pack@ea9ba7cdd0 but
unfortunately the commit message does not elucidate the exact the
problem, nor is there an associated pull request. This change has
propogated into all of the X-Pack scripts yet still the reasons are
unclear. The little that we can glean from the commit message is that
there was a problem with the default paths if the script was executed
outside of the Elasticsearch home. It seems that such issues have been
addressed by the recent introduction of elasticsearch-env so maybe we
can simplify these scripts here?

Relates elastic/x-pack-elasticsearch#2125

Original commit: elastic/x-pack-elasticsearch@9548c47743
2017-07-31 23:13:22 +09:00
Jason Tedor a80b1e4de1 Exit Windows scripts promptly on failure
When invoking the elasticsearch-env.bat or x-pack-env.bat batch scripts
on Windows, if these scripts exits due to an error (e.g., Java can not
be found, or the wrong version of Java is found), then the script
exits. Sadly, on Windows, this does not also terminate the caller,
instead returning control. This means we have to explicitly exit so that
is what we do in this commit.

Relates elastic/x-pack-elasticsearch#2126

Original commit: elastic/x-pack-elasticsearch@18645db62c
2017-07-31 20:39:59 +09:00
Jason Tedor 351febe031 Introduce elasticsearch-env for Windows
This commit refactors the batch scripts on Windows to use the
elasticsearch-env.bat script, and introduces a Windows version of
x-pack-env.bat for reuse in the scripts as well.

Relates elastic/x-pack-elasticsearch#2124

Original commit: elastic/x-pack-elasticsearch@faacb40dca
2017-07-30 09:31:52 +09:00
Chris Earle 45964561eb [Security] Add 'read_cross_cluster' privilege for .monitoring indices (elastic/x-pack-elasticsearch#2111)
This allows 6.0+ monitoring clusters to be used out of the box with CCS for extended read-only access.

Original commit: elastic/x-pack-elasticsearch@2b1e4ca4e4
2017-07-28 16:02:20 -04:00
Jay Modi db4c00b565 Update the full cluster restart tests to be more generic (elastic/x-pack-elasticsearch#2107)
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.

Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
2017-07-28 10:31:44 -06:00
David Kyle 694cfd6f0e Fix javadoc typo
Original commit: elastic/x-pack-elasticsearch@ef901af9cc
2017-07-28 16:24:24 +01:00
Lisa Cawley e7248c579c [DOCS] Fix licensing documentation links (elastic/x-pack-elasticsearch#2108)
Original commit: elastic/x-pack-elasticsearch@7b0f74348f
2017-07-28 08:19:55 -07:00
David Kyle f9104b7127 [ML] Lookback one extra bucket for histograms (elastic/x-pack-elasticsearch#2084)
Original commit: elastic/x-pack-elasticsearch@b9b4d3977f
2017-07-28 16:14:27 +01:00
Yannick Welsch 3a265de458 Revert "Mute IndexAuditTrailTests"
This reverts commit elastic/x-pack-elasticsearch@2e4a837dae.

Original commit: elastic/x-pack-elasticsearch@d8841cd2ce
2017-07-28 14:56:48 +02:00
Yannick Welsch 40174e4dcf [TEST] Set proper "transport.type" for transport client in IndexAuditTrailTests
Fixes issues seen on CI.

relates elastic/x-pack-elasticsearch#2115

Original commit: elastic/x-pack-elasticsearch@0094225a2e
2017-07-28 14:56:36 +02:00
David Kyle f99838e1bf Mute IndexAuditTrailTests
Original commit: elastic/x-pack-elasticsearch@2e4a837dae
2017-07-28 13:41:20 +01:00
David Kyle 21b437e140 Mute ShardsCollectorTests.testShardsCollectorMultipleIndices
Original commit: elastic/x-pack-elasticsearch@c2a6c05536
2017-07-28 11:42:51 +01:00
Martijn van Groningen 5f51dd813c XPack changes for: https://github.com/elastic/elasticsearch/pull/25456
Original commit: elastic/x-pack-elasticsearch@9d1d3c3fb4
2017-07-28 12:25:27 +02:00
Yannick Welsch 51e87bf290 Move tribe to a module (elastic/x-pack-elasticsearch#2088)
Companion PR to elastic/elasticsearch#25778

Original commit: elastic/x-pack-elasticsearch@a3355802e9
2017-07-28 11:23:52 +02:00
Jason Tedor 8176adfc26 Fix retrieve roles and users tests
This commit removes the use of a now removed --path.conf command-line
flag from the retrieve roles and users tests.

Original commit: elastic/x-pack-elasticsearch@30d5f5f648
2017-07-28 18:04:05 +09:00
Jason Tedor e77db8faf4 Fix users tool tests
This commit fixes the users tool command tests which were broken because
of a guard added that es.path.conf is set. We do not want to set this
system property in tests so instead we override createEnv where the
problematic guard exists.

Original commit: elastic/x-pack-elasticsearch@78b757695b
2017-07-28 17:32:39 +09:00
Jason Tedor d8277942ac Fix list X-Pack extension command tests
This commit fixes the list X-Pack extension command tests which were
broken because of a guard added that es.path.conf is set. We do not want
to set this system property in tests so instead we override createEnv
where the problematic guard exists.

Original commit: elastic/x-pack-elasticsearch@b1bc4ddcb8
2017-07-28 17:15:23 +09:00
Jason Tedor aade36eff3 Fix failing CLI tests
This commit fixes some failing CLI tests. The failure here is that a
guard against the system property es.path.conf was added yet these tests
were not adapted for this change. This commit implements this adapation
which overrides the createEnv method where the problematic guard is
invoked. We do this to avoid having to set es.path.conf in tests.

Original commit: elastic/x-pack-elasticsearch@20e1724823
2017-07-28 17:00:25 +09:00
Jason Tedor af7ec7a213 Pass config path as a system property
This commit responds to an upstream change which removes the --path.conf
command-line flag and instead uses the replacement mechanism for setting
the configuration path via the system property es.path.conf.

Relates elastic/x-pack-elasticsearch#2113

Original commit: elastic/x-pack-elasticsearch@4fefbffecb
2017-07-28 12:15:37 +09:00
Jay Modi 3a1b64bb12 Create the cryptoservice later in startup process (elastic/x-pack-elasticsearch#2087)
This commit moves the creation of the CryptoService to the createComponents method so that bootstrap
checks have been checked before the crypto service is instantiated. The cryptoservice was changed to
expect that the bootstrap check has passed before being instantiated in elastic/x-pack-elasticsearch#1831.

Original commit: elastic/x-pack-elasticsearch@cf11cf4782
2017-07-27 14:03:05 -06:00
David Kyle b95d3f7bf4 Unmute test after fix
Original commit: elastic/x-pack-elasticsearch@cceb3e7237
2017-07-27 11:34:52 +01:00
Tim Brooks 1603823f2c Ensure that security client is used in x-pack
This commit is fixing an issue with the build. Currently we have a
scenario where a call to build a transport client is not using the
security client. This modifies the settings source to ensure that the
security client is used.

Original commit: elastic/x-pack-elasticsearch@2d6ea1f4e4
2017-07-26 13:36:20 -05:00
Luca Cavanna cb20de5d95 Adapt to removal of XContentHelper#toString(ToXContent) (elastic/x-pack-elasticsearch#2072)
Original commit: elastic/x-pack-elasticsearch@35f6ac23a1
2017-07-26 16:01:09 +02:00
Igor Motov 4da7ae21bc Persistent Tasks: remove unused isCurrentStatus method (elastic/x-pack-elasticsearch#2076)
Removes a method that is no longer used in production code.

Relates to elastic/x-pack-elasticsearch#957

Original commit: elastic/x-pack-elasticsearch@84fcb9db8a
2017-07-26 08:40:22 -04:00
Simon Willnauer 3914d66cea Followup for elastic/elasticsearch#25885
Original commit: elastic/x-pack-elasticsearch@0397e68efd
2017-07-26 09:17:40 +02:00
David Kyle 14c88ca15f Correct long line length
Original commit: elastic/x-pack-elasticsearch@e77f473b5b
2017-07-25 16:59:42 +01:00
David Kyle 74d06216c2 [ML] Accept more varied Datafeed Aggregations (elastic/x-pack-elasticsearch#2038)
Original commit: elastic/x-pack-elasticsearch@ec1477f41c
2017-07-25 16:45:47 +01:00
David Kyle 8f6d9df96e [ML] Check influencer names are valid (elastic/x-pack-elasticsearch#2073)
Original commit: elastic/x-pack-elasticsearch@75869cacb3
2017-07-25 10:41:51 +01:00
Tim Vernum 9ab6d3cbc3 [Security] Support PKCSelastic/x-pack-elasticsearch#12 keystores (elastic/x-pack-elasticsearch#2066)
Adds support for reading PKCSelastic/x-pack-elasticsearch#12 files as SSL keystores/truststores.

Original commit: elastic/x-pack-elasticsearch@1855ad6173
2017-07-25 17:31:37 +10:00
Michael Basnight 6a7e51d9c0 Use shaded rest client dependencies
This commit modifies all org.apache.http to use the shaded rest clients
org.elasticsearch.client.http packages. It also removes a few unused
licenses due to the change.

Relates elastic/elasticsearch#25780

Original commit: elastic/x-pack-elasticsearch@8605560232
2017-07-24 12:56:17 -05:00
Chris Earle b302ff32e8 [Monitoring] Remove BWC Layer for Marvel (2.3 - 2.4) in 6.0 (elastic/x-pack-elasticsearch#1990)
This removes all BWC code that assisted in reading Marvel indices in 6.0.

Original commit: elastic/x-pack-elasticsearch@253fbf9a73
2017-07-24 10:23:06 -04:00
Simon Willnauer 9078c0b244 [TEST] use true unique transport address in test to prevent address collision
Original commit: elastic/x-pack-elasticsearch@91e5e1c046
2017-07-23 21:20:18 +02:00
Tal Levy 04ace4f1df add Migration Deprecation Info API Documentation (elastic/x-pack-elasticsearch#2064)
Original commit: elastic/x-pack-elasticsearch@0b90dfc97f
2017-07-21 16:20:13 -07:00
jaymode 9357369d57 Test: TokenAuthIntegTests ensures security index writable before invalidating
This commit makes sure the TokenAuthIntegTests wait for the security index to be writable before
each test method as invalidation requires writing to the security index.

Relates elastic/x-pack-elasticsearch#1551

Original commit: elastic/x-pack-elasticsearch@6e22885102
2017-07-21 14:01:43 -06:00
Ali Beyad 5190a05b75 Improves error message on non-upgraded security index (elastic/x-pack-elasticsearch#2061)
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.

Original commit: elastic/x-pack-elasticsearch@710b7634b4
2017-07-21 15:56:15 -04:00
Igor Motov 66a723d134 Docs: Add Upgrade API documentation (elastic/x-pack-elasticsearch#2063)
Original commit: elastic/x-pack-elasticsearch@f5a7c9ee5d
2017-07-21 15:26:23 -04:00
Jay Modi 2f7142ccc7 Provide a message for callers of the EmptyTrustManager (elastic/x-pack-elasticsearch#2052)
This change will provide a exception with a message to any callers of the empty trust manager for
better visibility into issues.

Original commit: elastic/x-pack-elasticsearch@c8241aea98
2017-07-21 10:10:14 -06:00
Tim Vernum b29f7a9ddb [Security] Handle non-existent user in native realm (elastic/x-pack-elasticsearch#2044)
Since change elastic/x-pack-elasticsearch@f796949 authentication is not allowed to respond with null, it must be AuthenticationResult.notHandled()

- Fixes 1 case where the native realm would respond null if the user was not found
- Fixes some edge cases in the LDAP realm.

Original commit: elastic/x-pack-elasticsearch@bc739a1d40
2017-07-21 22:25:50 +10:00
Tim Vernum 1752104140 Grant `getProxySelector` permission to rest client (elastic/x-pack-elasticsearch#2059)
Monitoring uses the low level rest client, which was recently updated to need an additional permission

Relates: elasticsearch#25757
relates elastic/x-pack-elasticsearch#2058

Original commit: elastic/x-pack-elasticsearch@eb9578792c
2017-07-21 19:06:40 +10:00
Tim Brooks 495fc21c37 Add log when elastic password boostrapped (elastic/x-pack-elasticsearch#2053)
This is related to elastic/x-pack-elasticsearch#1217. This adds a log message to inform the user
when the elastic user's password is bootsrapped successfully.

Original commit: elastic/x-pack-elasticsearch@8d30e163ec
2017-07-20 23:00:08 -05:00
Tim Vernum 5056d4e3df Revert unintended changes to syskeygen from elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch#2055)
During the development of elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch@5ac95c6) there was an intention to replace `SystemKeyTool` with `EncKeyTool`.
The java change was reverted and never committed to master, but the script change was accidentally left in place.

Also removes redundant "properties" variable (refer `elastic/x-pack-elasticsearch@b0a3b89`)

Original commit: elastic/x-pack-elasticsearch@f240479748
2017-07-21 12:51:25 +10:00
Jason Tedor 470e81ce64 Use elasticsearch-env
This commit utilizes the elasticsearch-env script that is added in core
Elasticsearch for significantly simplifying the scripts used in x-pack.

Relates elastic/x-pack-elasticsearch#2049

Original commit: elastic/x-pack-elasticsearch@8ef041d077
2017-07-21 09:39:46 +09:00
Andrew Cholakian c8e42a0db6 Update logstash mappings to use pipeline.id + correct metric types (elastic/x-pack-elasticsearch#2045)
In logstash parlance there really is no pipeline.name, its pipeline.id

This also removes support for deprecated gauge types `text` and `boolean` we will be removing those soon in logstash

This also renames `counters` to `long_counters` to be more explicit and for consistency with the gauge type. Also, if we ever decide to add other types of counters this will be more clear

Original commit: elastic/x-pack-elasticsearch@8f44a94579
2017-07-20 14:30:37 -05:00
Chris Earle e0fe6da303 [Security] Remove beats_system User and Role until needed (elastic/x-pack-elasticsearch#2042)
The user/role was added to simplify they setup surrounding Beats monitoring, but Beats monitoring has been delayed until the UI work is begun.

Original commit: elastic/x-pack-elasticsearch@1c0c85562e
2017-07-20 12:24:15 -04:00
Jack Conradson e007fee9fb remove lang url parameter from stored script requests (elastic/x-pack-elasticsearch#2029)
Original commit: elastic/x-pack-elasticsearch@1044c3ba53
2017-07-20 08:51:27 -07:00
Jason Tedor 5ecbbbd46d Use plain old exception if security init fails
When security initialization fails during startup today we throw an
Error. This triggers the uncaught exception handler immediately killing
the node. While the node is going to die either way, we should not be
triggering the killer and in general we should avoid using Throwables
that extend Error.

Relates elastic/x-pack-elasticsearch#2035

Original commit: elastic/x-pack-elasticsearch@546f7f9002
2017-07-21 00:26:22 +09:00
Simon Willnauer eb5631b981 Followup for elastic/elasticsearch#25791
Original commit: elastic/x-pack-elasticsearch@806de5670f
2017-07-20 16:45:48 +02:00
Jason Tedor 3f08fad603 Remove CONF_FILE check
This commit removes a legacy check for an unsupported environment
variable. This environment variable has not been supported since 1.x so
it is safe to stop checking for the existence of this setting.

Relates elastic/x-pack-elasticsearch#2048

Original commit: elastic/x-pack-elasticsearch@023230fa9e
2017-07-20 22:42:21 +09:00
Jason Tedor 814c1a21f9 Stop exporting HOSTNAME from scripts
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.

Relates elastic/x-pack-elasticsearch#2047

Original commit: elastic/x-pack-elasticsearch@7b833e061c
2017-07-20 22:28:01 +09:00
Alexander Reelsen dce13b87c3 Watcher: Remove stream serialization version checks, happen in 5.x (elastic/x-pack-elasticsearch#2002)
Those checks were moved into 5.x in elastic/x-pack-elasticsearch#2001

Original commit: elastic/x-pack-elasticsearch@7664bf35c5
2017-07-20 14:48:08 +02:00