Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.
This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.
This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.
Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
This change means that newly created jobs will get an explicit 1GB
model memory limit if no model memory limit is specified when creating
the job. Existing jobs that had a null model memory limit will carry
on using the default model memory limit defined in the C++ code.
Relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@a4e6b73c2b
We rely on command extensions in our scripts but we do not actually
guarantee that they are enabled (usually they are, by default, but they
can be disabled outside of our control). This commit ensures that they
are enabled.
Relates elastic/x-pack-elasticsearch#2307
Original commit: elastic/x-pack-elasticsearch@a5eec8ca7b
* [DOCS] Describe severity threshold and interval in anomaly table
* [DOCS] Describe aggregation in anomaly table
* [DOCS] Fixed capitalization in ML getting started
Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
* [DOCS] Add job groups to ML create/update job APIs
* [DOCS] Fix ML update job API example
* [DOCS] Address feedback for ML create/update job APIs
Original commit: elastic/x-pack-elasticsearch@0e7bb47342
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.
Relates to elastic/elasticsearch#26253
Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
The old message of "Cannot auto close job" implied the problem was with
closing the job. This change makes it clearer that the problem is that
the datafeed could not be stopped and hence auto-close will not even be
attempted.
Original commit: elastic/x-pack-elasticsearch@065e9930ce
These members are default initialized on contruction and then set by the
init() method. It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.
Original commit: elastic/x-pack-elasticsearch@668121e274
Today we try to bootstrap the security index with the bootstrap password and recommend the user to change the password with the user tool. This is trappy for instance if you happen to configure multiple nodes with a different bootstrap passwords (which is possible) it's unclear which password made it too bootstrap. Yet, we tell in the logs but it can still be very confusing. In general it should be possible to bootstrap with the user tool from any node unless the user is already created in the native user store. This change uses the bootstrap.password from the local node and always authenticate against it until the user is bootstrapped even if the passwords are different on different nodes. This will also work for authenticating against the cluster for instance if a user deletes the .security index or if that index has not been upgraded.
Original commit: elastic/x-pack-elasticsearch@8cebecb287
When writing the users and users_roles files, we wrap a custom writer in
a print writer. There is a problem with this though: when print writer
closes it closes our underlying custom writer and the close
implementation for our custom writer is not trivial, it executes code
that can throw an I/O exception. When print writer invokes this close
and an I/O exception is thrown, it swallows that exception and sets the
status on the print writer to error. One would think that we could
simply check this status but alas print writer is broken here. The act
of checking the status causes print writer to try to flush the
underyling stream which is going to be completely undefined because the
underlying stream might or might not be closed. This might cause another
exception to be thrown, losing the original. Print writer screwed the
pooch here, there is no good reason to try to do any I/O after the
underlying writer entered a failed state. To address this we remove the
use of print writer, we use our custom writer directly. This allows any
thrown exceptions to bubble up.
Relates elastic/x-pack-elasticsearch#2288
Original commit: elastic/x-pack-elasticsearch@11b8dd5641
When mappings are updated for an index are updated most settings are
merged, but not _meta. This change ensures that _meta is set when we
add per-job term mappings to our results index mappings. In order to
keep the logic for updating mappings after upgrade working, we now
have to put ALL the mappings for our results along with the latest _meta
section when updating per-job term mappings.
relates elastic/x-pack-elasticsearch#2265
Original commit: elastic/x-pack-elasticsearch@f58c11a13e
We close the secure settings in core before we pull bootstrap checks.
This means if a bootstrap check like the `TokenPassphraseBootstrapCheck`
accesses a secure setting that late it will fail due to an exception in
the `PKCS12KeyStore`. This change moves the bootstrap check creation
to the plugin constructor and adds a dummy setting to the integTest
that triggers the bootstrap checks.
Original commit: elastic/x-pack-elasticsearch@2b20865d1c
* [DOCS] Add custom URL instructions
* [DOCS] Update custom URL examples
* [DOCS] Add example for custom URL
* [DOCS] Add ML category terms custom URL example
* [DOCS] Add tip for pop-up ML custom URLs
* [DOCS] Address feedback about ML custom URLs
* [DOCS] Clarify impact of interval on custom URL
Original commit: elastic/x-pack-elasticsearch@eee192226f
When the machine-learning-cpp repo is built locally, the zip file it
creates is preferred over that downloaded from s3 when creating the
overall x-pack-elasticsearch zip. However, prior to this change the
build would ALSO download an ml-cpp zip from s3, and just not use it.
Original commit: elastic/x-pack-elasticsearch@bd71637edd
PUT /_xpack/license with no content or content-type should fail with an appropriate error message rather than throwing NPE.
Original commit: elastic/x-pack-elasticsearch@f8c744d2a2
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.
relates elastic/x-pack-elasticsearch#2255
Original commit: elastic/x-pack-elasticsearch@56765a649e
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.
As the smoke tests are only four tests this will not lead to a
log explosion.
Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874
Original commit: elastic/x-pack-elasticsearch@5832dc7990
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
These tests used to fail rarely, because during a watch execution
one of the watcher shards was relocated resulting in a second execution
of watch.
In order to prevent this, the tests do not need to actually create any
shards, which causes watcher potentially to be rebalanced.
This simplifies and speeds up the test as well.
relates elastic/x-pack-elasticsearch#1608
Original commit: elastic/x-pack-elasticsearch@1cfac1145d
This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.
Original commit: elastic/x-pack-elasticsearch@c8c2819d86
When a watch is executed, it sends an update request to the watch to
udpate its status.
This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.
This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.
This will fix not only the current behaviour but also some flaky tests.
Original commit: elastic/x-pack-elasticsearch@ca69109ecb
* [DOCS] Add user setup to X-Pack install info
* [DOCS] Add TSL steps to X-Pack install
* [DOCS] Clarify SSL settings in Xpack install
Original commit: elastic/x-pack-elasticsearch@eee37729ff
* [DOCS] Update APIs for multiple jobs or datafeeds
* [DOCS] Fix syntax diagrams for ML stop/close APIs
* [DOCS] Removed TBD authorization for ML APIs
Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
Simon Willnauer
David Roberts
Jason Tedor
Lisa Cawley
Colin Goodheart-Smithe
Alexander Reelsen
Deb Adair
Yannick Welsch
Tim Vernum
David Kyle