Commit Graph

6745 Commits

Author SHA1 Message Date
David Kyle 5a63c4d9a2 [ML] Remove trailing slashes from ML rest spec (elastic/x-pack-elasticsearch#2350)
Original commit: elastic/x-pack-elasticsearch@f2a3d70562
2017-08-25 09:35:07 +01:00
David Kyle 175e8db3aa [ML] Don’t count incomplete buckets in data stream diagnostics (elastic/x-pack-elasticsearch#2351)
* Don’t count incomplete buckets in data stream diagnostics

* Fix tests now bucket_count doesn’t include partial buckets

Original commit: elastic/x-pack-elasticsearch@bc1a7bd9e7
2017-08-25 09:25:15 +01:00
Ryan Ernst b30c634326 Build: Configure the license/notice file for xpack (elastic/x-pack-elasticsearch#2357)
This commit uses the new license/notice file settings in gradle to
specify the xpack license to be placed in jar files.

See https://github.com/elastic/elasticsearch/pull/26373

Original commit: elastic/x-pack-elasticsearch@209a1e9a47
2017-08-24 22:47:10 -07:00
Ryan Ernst 13672dad13 Build: Set xpack to require keystore
See https://github.com/elastic/elasticsearch/pull/26329

Original commit: elastic/x-pack-elasticsearch@e77361a6d5
2017-08-24 14:09:07 -07:00
Jay Modi 8e4ded82c0 Adapt to resync rename and add assertion for missing user when sending a request (elastic/x-pack-elasticsearch#2345)
This commit adapts to the renaming of the TransportResyncReplicationAction in core and also adds
an assertion to the check for a user being present when sending a request. The assertion is added
so that we can hopefully catch these scenarios in our testing as the assertion error will cause the
node to die but the ISE will just be seen in the logs. Since we do not run with assertions enabled
in production, the ISE is left to handle those cases.

relates elastic/x-pack-elasticsearch#2335

Original commit: elastic/x-pack-elasticsearch@c5ce0c93af
2017-08-24 11:04:41 -06:00
Colin Goodheart-Smithe d0103d6cab [TEST] Fixes PageParamsTests to no underflow from and size
In `mutateInstance()` the from or size could become negative if the other one was pushed over the limit for `from + size`. This change fixes this case to make sure after the mutate method is called the from and size obey the limit but are also both `>= 0`

relates elastic/x-pack-elasticsearch#2344

Original commit: elastic/x-pack-elasticsearch@a8a7072fcc
2017-08-24 09:29:36 +01:00
Christoph Büscher f05568e7b3 Revert adding bin to .gitignore (elastic/x-pack-elasticsearch#2346)
Original commit: elastic/x-pack-elasticsearch@cfd2be3831
2017-08-23 22:43:01 +02:00
Jason Tedor 85e494d90f Add x-pack-env.bat
This file was missing from the commit which depends on this file due to
a .gitignore rule. This commit adds this vital file.

Relates elastic/x-pack-elasticsearch#2124

Original commit: elastic/x-pack-elasticsearch@78125b56de
2017-08-23 14:16:36 -04:00
Stacey Gammon 1bfa4cb8bb Add reserved dashboards_only_user role (elastic/x-pack-elasticsearch#2250)
* Add reserved dashboards_only_user role

* Fix line too long

* add tests for new reserved role

* rename role, hopefully fix tests

* Fix test

Original commit: elastic/x-pack-elasticsearch@99f6718c7c
2017-08-23 13:16:17 -04:00
David Kyle 71063825be [MLFix bucket setting count (elastic/x-pack-elasticsearch#2339)
Original commit: elastic/x-pack-elasticsearch@6801e90d54
2017-08-23 15:46:51 +01:00
Alexander Reelsen 6ee4fe6a0b Watcher: Improve upgrade API logging message to include index names
Original commit: elastic/x-pack-elasticsearch@9cecc11f88
2017-08-23 16:11:49 +02:00
Albert Zaharovits 026729e911 TOKEN_SERVICE_ENABLED_SETTING enabled if HTTP_SSL_ENABLED (elastic/x-pack-elasticsearch#2321)
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.

* TokenService default enabled if HTTP_ENABLED == false

* Fixed tests that need TokenService explicitly enabled

* [DOC] Default value for `xpack.security.authc.token.enabled`

Original commit: elastic/x-pack-elasticsearch@bd154d16eb
2017-08-23 13:21:30 +03:00
Colin Goodheart-Smithe de6c275dca Refactor/to x content fragments2 (elastic/x-pack-elasticsearch#2329)
* Moves more classes over to ToXContentObject/Fragment

* Removes ToXContentToBytes

* Removes ToXContent from Enums

* review comment fix

* slight change to use XContantHelper

Original commit: elastic/x-pack-elasticsearch@0f2d3f328b
2017-08-23 08:17:28 +01:00
David Kyle 18d15ef2d2 Revert "[ML] Missing validations in analysis config (elastic/x-pack-elasticsearch#2103)"
This reverts commit elastic/x-pack-elasticsearch@461be12f9f.

Original commit: elastic/x-pack-elasticsearch@1ce7196710
2017-08-22 13:33:40 +01:00
Yannick Welsch b4353b55ad Allow build to directly run under JDK 9 (elastic/x-pack-elasticsearch#2320)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

Relates to elastic/elasticsearch#25859

Original commit: elastic/x-pack-elasticsearch@4bf266e0b0
2017-08-22 14:46:37 +09:30
Alexander Reelsen 3f541fa556 Tests: Ensure watcher is started via awaitBusy in bwc test
Original commit: elastic/x-pack-elasticsearch@a8c0cf04c9
2017-08-22 00:39:11 +02:00
Alexander Reelsen 17980ab360 Tests: Remove randomized unsupported code upgrading two indices at once
Original commit: elastic/x-pack-elasticsearch@c86b87927d
2017-08-22 00:09:01 +02:00
Lisa Cawley cb50b2c539 [DOCS] Update README with new release notes scripts
Original commit: elastic/x-pack-elasticsearch@78ec39322d
2017-08-21 14:58:56 -07:00
lcawley 357a79f44c [DOCS] Add release notes script that merges two repos
Original commit: elastic/x-pack-elasticsearch@935c7312f1
2017-08-21 14:54:05 -07:00
David Roberts ea94ef3aa9 [ML] Update docs for change to default model memory limit (elastic/x-pack-elasticsearch#2308)
This is the doc change for elastic/x-pack-elasticsearch#2300

Original commit: elastic/x-pack-elasticsearch@acd683d06a
2017-08-21 16:51:30 +01:00
Alexander Reelsen 27f39c615b Watcher: Create two index ugprade checks for watcher upgrade (elastic/x-pack-elasticsearch#2298)
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.

There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.

This also means, that there will be two entries popping up in the kibana
UI.

Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.

relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@2c92040ed6
2017-08-21 17:36:16 +02:00
Jason Tedor 8a5be5b58e Replace atomic move writer
I noticed this while working on a previous issue with atomic move writer
(silent swallowing of exceptions). Namely, atomic move writer has
dangerous semantics. The problem is as follows: atomic move writer works
by writing lines to a temporary file, and then in its close method it
replaces the target path with the temporary file. However, the close
method is invoked whether or not all writes to the temporary file
succeeded (because writers obtained from atomic move writer are used in
try-with-resources blocks, as they should be). There is no way to
distinguish that the writer is being closed in a successful scenario
versus a failure scenario. In the close method for atomic move writer,
the target file is replaced by the temporary file. This means that the
target file is replaced whether or not writing to the temporary file
actually succeeded. Since these atomic move writers are used for user
configuration files (users and user_roles), a failure here can lead to
data loss for the user, a tragedy!

There is another (less serious) problem with the atomic move
writer. Since the close method tries to move the temporary file in place
of the existing file, the temporary file can be left behind if there is
another failure in the close method (e.g., closing the underlying file
after writing, or setting the permissions on the temporary file). This
means that in some situations, atomic move writer will leave temporary
files behind (which is not definitively not atomic).

This commit replaces the atomic move writer with a safer mechanism. We
still perform the write atomically in the following sense: we write to a
temporary file. Either writing to that file succeeds or it fails. If
writing succeeds, we replace the existing file with the temporary
file. If writing fails, we clean up the temporary file and the existing
file remains in place.

Relates elastic/x-pack-elasticsearch#2299


Original commit: elastic/x-pack-elasticsearch@3199decb0a
2017-08-21 10:35:37 -04:00
David Roberts e428c58dff [TEST] Fix logic in one branch of random mutate test
Certain types of datafeeds cannot have null chunking configs, so
setting chunking config to null sometimes doesn't stick as null

Original commit: elastic/x-pack-elasticsearch@3a52bad460
2017-08-21 14:51:04 +01:00
Colin Goodheart-Smithe 0a1225f934 [TEST] Adds mutate function to some tests (elastic/x-pack-elasticsearch#2263)
Original commit: elastic/x-pack-elasticsearch@560d7e9a80
2017-08-21 11:20:14 +01:00
Alexander Reelsen 1b6d9d430c Watcher: Load for watch for execution as late as possible (elastic/x-pack-elasticsearch#2151)
When a watch is executed currently, it gets passed an in-memory
watch object, that was loaded, before the execution started.

This means there is a window of time, where an old watch could still
be executing, then a watch gets loaded for execution, then the old watch
execution finishes and updates the watch status and thus reindexes the
watch.

Now the watch, that got loaded for execution, executes and tries to
store its watch status, but fails, because the version of the watch
has changed.

This commit changes the point in time where the watch is loaded. Now
this only happens, while a watch is in its protected execution block,
and thus we can be sure, that there is no other execution of the watch
happening.

This will primarily impact watches, that execute often, but their
runtime is longer than the configured interval between executions.

Side fix: Removed some duplicate testing method and moved into 
WatcherTestUtils, fixed a tests with a ton of if's with random booleans
into separate tests.

relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bf393023d7
2017-08-21 10:43:41 +02:00
Simon Willnauer 44c3d5b3d9 Allow to change bwc refspec in xpack (elastic/x-pack-elasticsearch#2311)
Today it's impossible to run xpack bwc tests against any other branch
or remote than upstream. This allows to pass `-Dtests.bwc.refspec` to
change the refspec to use for the bwc tests.

Original commit: elastic/x-pack-elasticsearch@4d365f5a6e
2017-08-21 09:59:45 +02:00
Simon Willnauer 846f40ba15 [TEST] allow test to start an additional node even if max number of random nodes is reached
relates elastic/x-pack-elasticsearch#2315

Original commit: elastic/x-pack-elasticsearch@d989c06198
2017-08-19 11:03:46 +02:00
Lisa Cawley 702c192c35 [DOCS] Fix model_plot_config in ML create job API (elastic/x-pack-elasticsearch#2316)
Original commit: elastic/x-pack-elasticsearch@2ef32e52cd
2017-08-18 15:06:24 -07:00
Lisa Cawley 17b4720df7 [DOCS] Add missing ML create job API options (elastic/x-pack-elasticsearch#2268)
* [DOCS] Add missing ML create job API options

* [DOCS] Small fixes in ML create job API

Original commit: elastic/x-pack-elasticsearch@369bd5bfbb
2017-08-18 13:00:15 -07:00
Lisa Cawley d76969a431 [DOCS] Update example for default model memory limit (elastic/x-pack-elasticsearch#2312)
Original commit: elastic/x-pack-elasticsearch@815d63d1bf
2017-08-18 11:29:08 -07:00
Simon Willnauer 71827b70a0 Bump token service BWC version to 6.0.0-beta2
Original commit: elastic/x-pack-elasticsearch@ef688f02cb
2017-08-18 17:06:45 +02:00
David Roberts 4f17335629 [ML] Add unit tests for ML roles (elastic/x-pack-elasticsearch#2309)
I forgot to add these when the roles were originally added

Original commit: elastic/x-pack-elasticsearch@0e72141b11
2017-08-18 14:04:41 +01:00
Simon Willnauer ac9ab974f4 Ensure token service can boostrap itself without a pre-shared key (elastic/x-pack-elasticsearch#2240)
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.

This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.

This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.

Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
2017-08-18 14:23:43 +02:00
David Roberts 269f0f6a19 [ML] Default model memory limit to 1GB for newly created jobs (elastic/x-pack-elasticsearch#2300)
This change means that newly created jobs will get an explicit 1GB
model memory limit if no model memory limit is specified when creating
the job.  Existing jobs that had a null model memory limit will carry
on using the default model memory limit defined in the C++ code.

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@a4e6b73c2b
2017-08-18 09:36:09 +01:00
Jason Tedor 5125978f60 Enable command extensions
We rely on command extensions in our scripts but we do not actually
guarantee that they are enabled (usually they are, by default, but they
can be disabled outside of our control). This commit ensures that they
are enabled.

Relates elastic/x-pack-elasticsearch#2307

Original commit: elastic/x-pack-elasticsearch@a5eec8ca7b
2017-08-17 22:50:52 -04:00
Lisa Cawley 8e7bd27186 [DOCS] Describe severity threshold and interval in anomaly table (elastic/x-pack-elasticsearch#2266)
* [DOCS] Describe severity threshold and interval in anomaly table

* [DOCS] Describe aggregation in anomaly table

* [DOCS] Fixed capitalization in ML getting started

Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
2017-08-17 12:56:58 -07:00
Lisa Cawley e97b6dcc47 [DOCS] Add job groups to ML create/update job APIs (elastic/x-pack-elasticsearch#2290)
* [DOCS] Add job groups to ML create/update job APIs

* [DOCS] Fix ML update job API example

* [DOCS] Address feedback for ML create/update job APIs

Original commit: elastic/x-pack-elasticsearch@0e7bb47342
2017-08-17 12:52:29 -07:00
Lisa Cawley 29fc38b0e2 [DOCS] Add xpack.ml.max_open_jobs setting (elastic/x-pack-elasticsearch#2303)
Original commit: elastic/x-pack-elasticsearch@9b82043c34
2017-08-17 12:23:34 -07:00
Simon Willnauer 724325f161 Fallback to `keystore.seed` as a bootstrap password if actual password is not present (elastic/x-pack-elasticsearch#2295)
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.

Relates to elastic/elasticsearch#26253

Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
2017-08-17 16:42:32 +02:00
David Roberts 2ae5634dc9 [ML] Improve error message when auto-close isn't attempted (elastic/x-pack-elasticsearch#2296)
The old message of "Cannot auto close job" implied the problem was with
closing the job.  This change makes it clearer that the problem is that
the datafeed could not be stopped and hence auto-close will not even be
attempted.

Original commit: elastic/x-pack-elasticsearch@065e9930ce
2017-08-17 15:07:06 +01:00
David Roberts 44857d71b3 Make AllocatedPersistentTask members volatile (elastic/x-pack-elasticsearch#2297)
These members are default initialized on contruction and then set by the
init() method.  It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.

Original commit: elastic/x-pack-elasticsearch@668121e274
2017-08-17 14:54:31 +01:00
Colin Goodheart-Smithe 751680e7b2 Moves more classes over to ToXContentObject/Fragment (elastic/x-pack-elasticsearch#2283)
Original commit: elastic/x-pack-elasticsearch@73c6802523
2017-08-17 11:16:56 +01:00
Alexander Reelsen 6d30806996 Watcher: Improvements on the rolling restart tests (elastic/x-pack-elasticsearch#2286)
This improves the rolling restart tests (tests different paths in
different ways) and aligns the upgrade code with the 5.6 branch from

Relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@01b0954558
2017-08-17 11:41:11 +02:00
Simon Willnauer 8f15324a08 Don't bootstrap security index on start-up but authenticate bootstrap password locally (elastic/x-pack-elasticsearch#2272)
Today we try to bootstrap the security index with the bootstrap password and recommend the user to change the password with the user tool. This is trappy for instance if you happen to configure multiple nodes with a different bootstrap passwords (which is possible) it's unclear which password made it too bootstrap. Yet, we tell in the logs but it can still be very confusing. In general it should be possible to bootstrap with the user tool from any node unless the user is already created in the native user store. This change uses the bootstrap.password from the local node and always authenticate against it until the user is bootstrapped even if the passwords are different on different nodes. This will also work for authenticating against the cluster for instance if a user deletes the .security index or if that index has not been upgraded.

Original commit: elastic/x-pack-elasticsearch@8cebecb287
2017-08-17 08:36:26 +02:00
Deb Adair 4ed7b5473c [DOCS] Changed to link specifically to 6.0 branch.
Original commit: elastic/x-pack-elasticsearch@247fd4b909
2017-08-16 15:43:00 -07:00
Deb Adair 6aac2b9f11 [DOCS] Fixed hardcoded link to ES ref.
Original commit: elastic/x-pack-elasticsearch@045561b93d
2017-08-16 14:25:16 -07:00
Jason Tedor 2bf8f4b0bc Remove print writer wrapping for users tools
When writing the users and users_roles files, we wrap a custom writer in
a print writer. There is a problem with this though: when print writer
closes it closes our underlying custom writer and the close
implementation for our custom writer is not trivial, it executes code
that can throw an I/O exception. When print writer invokes this close
and an I/O exception is thrown, it swallows that exception and sets the
status on the print writer to error. One would think that we could
simply check this status but alas print writer is broken here. The act
of checking the status causes print writer to try to flush the
underyling stream which is going to be completely undefined because the
underlying stream might or might not be closed. This might cause another
exception to be thrown, losing the original. Print writer screwed the
pooch here, there is no good reason to try to do any I/O after the
underlying writer entered a failed state. To address this we remove the
use of print writer, we use our custom writer directly. This allows any
thrown exceptions to bubble up.

Relates elastic/x-pack-elasticsearch#2288

Original commit: elastic/x-pack-elasticsearch@11b8dd5641
2017-08-16 12:50:39 -04:00
David Roberts 6fcc3be438 [ML] Preserve _meta on results index mapping update (elastic/x-pack-elasticsearch#2274)
When mappings are updated for an index are updated most settings are
merged, but not _meta.  This change ensures that _meta is set when we
add per-job term mappings to our results index mappings.  In order to
keep the logic for updating mappings after upgrade working, we now
have to put ALL the mappings for our results along with the latest _meta
section when updating per-job term mappings.

relates elastic/x-pack-elasticsearch#2265

Original commit: elastic/x-pack-elasticsearch@f58c11a13e
2017-08-16 16:16:30 +01:00
Alexander Reelsen 0a86f00d7e Tests: Ensure responses are closed in watch backwards compat tests
The HTTP client has to have it's response entities closed, otherwise
it might block further requests because the underlying connection cannot
be reused.

Relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@a24ecb9764
2017-08-16 16:29:52 +02:00
Simon Willnauer 8b23f133c7 Create security bootstrap checks early to access secure settings safely (elastic/x-pack-elasticsearch#2282)
We close the secure settings in core before we pull bootstrap checks.
This means if a bootstrap check like the `TokenPassphraseBootstrapCheck`
accesses a secure setting that late it will fail due to an exception in
the `PKCS12KeyStore`. This change moves the bootstrap check creation
to the plugin constructor and adds a dummy setting to the integTest
that triggers the bootstrap checks.

Original commit: elastic/x-pack-elasticsearch@2b20865d1c
2017-08-16 13:01:52 +02:00