Our tests currently rely on waiting for the security index to be available in some cases and in
CI, these checks have been timing out. This commit increases the amount of time that we will wait
for the index before failing to account for slow machines.
Original commit: elastic/x-pack-elasticsearch@639dccd5cb
This checks if `apm-*` indices exist in the cluster to try to determine if APM is in use on the Elasticsearch cluster.
Original commit: elastic/x-pack-elasticsearch@7f9a9a4eee
On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe. Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).
Fixeselastic/machine-learning-cpp#385
Original commit: elastic/x-pack-elasticsearch@89237d7125
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.
Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.
original commit message:
Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.
modifications:
Test with 7.x vs 6.x nodes.
Original commit: elastic/x-pack-elasticsearch@56f81bfb20
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.
Original commit: elastic/x-pack-elasticsearch@2d75d15c41
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.
This forces a few things to happen, thereby testing the behavior:
1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.
Original commit: elastic/x-pack-elasticsearch@012e5738bb
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.
Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
This change should have been made in elastic/x-pack-elasticsearch#2913. Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.
Original commit: elastic/x-pack-elasticsearch@79672b0825
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.
relates elastic/x-pack-elasticsearch#2656
Original commit: elastic/x-pack-elasticsearch@649a8033e4
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.
Original commit: elastic/x-pack-elasticsearch@a2e72bed12
I imagine this needless indirection arose from accepting the wrong
IntelliJ suggestion for an import.
Original commit: elastic/x-pack-elasticsearch@54d7e854d3
The 5.6 Upgrade API will reindex .security to .security-6 and create a .security alias.
But the 6.0 default was to create a .security-v6 index and a .security alias if none existed (e.g. fresh x-pack install)
Having two different index names based on the method of install/upgrade complicates the code and testing, so we're unifying on the .security-6 index name that already exists in the wild.
Original commit: elastic/x-pack-elasticsearch@d78f569c5f
When simultaneous close requests were made for the same job it was possible
that one of the requests would inappropriately log error messages about the
job having failed. This change prevents that problem, whilst continuing to
adhere to the requirement that close requests for already closing jobs do not
return until the close request that is doing the work completes.
relates elastic/x-pack-elasticsearch#2912
Original commit: elastic/x-pack-elasticsearch@513b7fa1d6
The default internal XPack user no longer has access to the security index, but it should have read-only access to the audit log so that watches can be triggered based on audit events (but cannot write audit records)
Original commit: elastic/x-pack-elasticsearch@5c37720dad
This commit changes the handling of exceptions when retrieving roles from the native roles store.
Previously, exceptions would have caused the request to terminate and the exception would be
sent back to the user. This makes for a bad experience when a cluster hasn't been upgraded to the
latest index format and anonymous access is enabled with a native role as all requests without
preemptive basic authentication would result in an exception. The change here is to allow the
request to continue processing. Once the security index is up to date, the roles cache is cleared
so that the native roles can be picked up.
relates elastic/x-pack-elasticsearch#2686
Original commit: elastic/x-pack-elasticsearch@ef5149140f
Certgen was generating "other name" SANs without the explicit [0] tag that is required.
This was masked by the fact that the JRE X.509 classes always wrap the "other name" name-value in a [0] tag (even if it already has one)
Also switched to a UTF8 String from an IA5 string to match the configuration being used for testing in openssl.
Original commit: elastic/x-pack-elasticsearch@1b87964ec7
This is the X-Pack side of elastic/elasticsearch#27235. To force people
who construct an Environment object in production code to think about the
correct setting of configPath there is no longer a single argument
constructor in the Environment class. Instead there is a factory method
in the test framework to replace it. Having this in the test framework
ensures that there is no way to use it in production code.
Original commit: elastic/x-pack-elasticsearch@4860e92d90
Collapses a package into its parent package because they only contain a
single class together. No need for two layers.
Original commit: elastic/x-pack-elasticsearch@c947e57952
Organizes the SQL translate action to match the way that x-pack has been
organizing new actions for a while. All of the pieces are put into the
same class file.
Original commit: elastic/x-pack-elasticsearch@def911c0ab
We should not be constructing a temporary Environment object in production
code. This currently isn't causing any problems, but it might in the future
if elastic/elasticsearch#27144 or something similar is ever merged. Instead
the master Environment of the node should always be used.
Original commit: elastic/x-pack-elasticsearch@6276a54a45
This adds the data necessary to add a warning to the alerts UI representing each cluster when xpack.security.transport.tls.enabled is not set to true for a trial licensed cluster running with
xpack.security.enabled.
Original commit: elastic/x-pack-elasticsearch@28fe8bad76
This adds details about the shards and the health of the index. By adding these stats directly to the document, the UI can avoid many aggregations and enable better searching and sorting against indices.
Original commit: elastic/x-pack-elasticsearch@f38ae5ce69
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.
Original commit: elastic/x-pack-elasticsearch@7026d83c06
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.
Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.
Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.
relates elastic/x-pack-elasticsearch#2658
Original commit: elastic/x-pack-elasticsearch@27e2ce7223
Adding this field enables a very simple mechanism for detecting node changes in the cluster state via Watcher (and other mechanisms). The next step is to add the cluster alert that uses it.
Original commit: elastic/x-pack-elasticsearch@1eacc25cff
This commit adds a new `certutil` command and deprecates the `certgen` command.
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.
Original commit: elastic/x-pack-elasticsearch@3f57687da9
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.
This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.
Original commit: elastic/x-pack-elasticsearch@351e64e14d
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.
Original commit: elastic/x-pack-elasticsearch@34cfd11df8
This commit adds a new interval_ms field to the monitoring documents.
This field indicates the current collection interval for Elasticsearch or
external monitored systems. The value is indexed as a long.
Related to elastic/x-pack-elasticsearch#212
Original commit: elastic/x-pack-elasticsearch@2ceb20455c
This change fixes an incorrect check for a missing password setting for the PKI realm. The check
only allowed the secure setting to be used for the PkiRealm password even though the legacy setting
is still valid. This change fixes the check.
Relates elastic/x-pack-elasticsearch#2487
Original commit: elastic/x-pack-elasticsearch@a4524c2c05
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.
This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.
relates elastic/x-pack-elasticsearch#2459
Original commit: elastic/x-pack-elasticsearch@2d349e840f
Provides more verbose messaging around errors and possible causes when the tool aborts.
This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.
Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).
Validation is now applied after parsing the file and if errors are found the tool exits.
Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node. This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.
This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649
Original commit: elastic/x-pack-elasticsearch@9c381801d9
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
This adds a dynamic setting, which defaults to `false` currently, that can be used to delete all `.watcher-history*` indices that match the same age requirements as Monitoring indices.
Original commit: elastic/x-pack-elasticsearch@8ca3bdbca3
Do not execute bind on on the LDAP reader thread
Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.
Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620
Original commit: elastic/x-pack-elasticsearch@404a3d8737
Uses the appropriate overload of `generateRandomStringArray` to disallow empty arrays from being returned.
Original commit: elastic/x-pack-elasticsearch@2596653ca1
Since the transport ssl enabled setting is usable in 6.x again, this change adds back the value to
the xpack security usage so that it can be included in phone home data.
Original commit: elastic/x-pack-elasticsearch@52f6176df0
The upgrade API adds a "type" field to role mapping documents.
The parser would reject these docs due to an unexpected field. We now ignore the "type" field instead.
Original commit: elastic/x-pack-elasticsearch@538f5adab2
Those classes used to be elasticsearch-agnostic wrappers
of the query parameters. However, we now do not need that
layer of abstraction. Instead we can make those builders
own the building of the SearchSourceBuilder, which
simplifies the JobProvider and makes them reusable.
Original commit: elastic/x-pack-elasticsearch@b079cce1d6
Since we are authorising on a single shard of a single index,
and there are only 3 possible actions that an item might represent,
we can test which items are authorised with a maximum of 3 permission
evaluations, regardless of how many items are actually in the shard
request. Previously we would test them all independently which had
a much higher overhead for large bulk requests.
Relates: elastic/x-pack-elasticsearch#2369
Original commit: elastic/x-pack-elasticsearch@aceacf0aa3
A number of REST requests require a body but did not explicitly validate for it.
This would typically cause a NPE if they were called with no body.
Original commit: elastic/x-pack-elasticsearch@863ac89429
The test also used the timewarp trigger for watches to be executed, but it is sufficient to just call the execute watch API to make this test faster.
Original commit: elastic/x-pack-elasticsearch@3a4165f72c
When getting a single bucket, the get_buckets API can take a timestamp
either in the body or in the URL. Prior to this change, if a timestamp
was specified in the URL but a body not containing a timestamp was specified
(either empty or containing other parameters like exclude_interim or sort)
then it would cause a bad_request exception. This in turn causes problems
for clients that cannot send a body when GETting and always send a body when
POSTing.
This change fixes get_buckets to always read any timestamp in the URL, even
when a body is sent.
relates elastic/x-pack-elasticsearch#2515
Original commit: elastic/x-pack-elasticsearch@5c23dd972e
If a job close is requested after a job was opened but before
its process was launched, the job close returns successfully
without doing anything. The result is that the process hangs
around. This has been causing test failures as documented
int elastic/x-pack-elasticsearch#2360 and elastic/x-pack-elasticsearch#1270.
This commit fixes this problem by refactoring the
AutodetectProcessManager. It introduces a state pattern
to make clear the states of the process and it uses locking
to ensure a close waits for the job process to be created.
relates elastic/x-pack-elasticsearch#1270
Original commit: elastic/x-pack-elasticsearch@ff858bd136
The AuthenticationService#testInvalidToken would cause a suite timeout in the case of an exception
due to a incorrect stream size as the latch was never counted down. This fixes the missing latch
countdown.
relates elastic/x-pack-elasticsearch#2615
Original commit: elastic/x-pack-elasticsearch@e838e6e912
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
* I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.
Original commit: elastic/x-pack-elasticsearch@3b12afd11c
Checking the size of the map doesn't make sense when each filter is
checked independently right after.
Original commit: elastic/x-pack-elasticsearch@58e5d3401d
This is related to elastic/x-pack-elasticsearch#1941.
Currently we support self-generating either a basic or trial license at
cluster startup. With the addition of the basic option, it is possible
that a user would choose to self-generate and eventually register a
basic license.
This commit allows a user to upgrade to a 30-day trial license if they
have not already utilized this 30-day trial license before. Additionally
it adds a get route to check if the user is eligible to upgrade. This
route will allow kibana to implement a cleaner UI.
Original commit: elastic/x-pack-elasticsearch@7f19b33a08
This removes the creation and handling of the Beats monitoring template and its data until we actually expect to support it (most likely 6.2 - 6.3).
Original commit: elastic/x-pack-elasticsearch@2dc8abbb37
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.
Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.
I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.
This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.
Original commit: elastic/x-pack-elasticsearch@560c6815e3
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved
* fix making payload.message show only when new
Original commit: elastic/x-pack-elasticsearch@6d54b02913
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.
relates elastic/x-pack-elasticsearch#2519
Original commit: elastic/x-pack-elasticsearch@e03dde5fea
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.
Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario
relates elastic/x-pack-elasticsearch#2527.
Original commit: elastic/x-pack-elasticsearch@7581e8d699
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket. Prior to this change we would set the "actual" value to 0 for such
model plot documents. Following this change no "actual" value will be
present in these documents.
Only newly created model plot documents are affected. Model plot documents
that were incorrectly written in the past will remain wrong forever.
relates elastic/x-pack-elasticsearch#2528
Original commit: elastic/x-pack-elasticsearch@47a7365f59
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.
This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.
relates elastic/x-pack-elasticsearch#2519
Original commit: elastic/x-pack-elasticsearch@15c7d2655f
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.
This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores
Original commit: elastic/x-pack-elasticsearch@fc361f0d87
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.
Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.
This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.
Original commit: elastic/x-pack-elasticsearch@8d99f5518b
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.
Original commit: elastic/x-pack-elasticsearch@d66145de54
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
* Move CLI to TransportSqlAction
* Moves REST endpoint from `/_cli` to `/_sql/cli`
* Removes the special purpose CLI transport action instead
implements the CLI entirely on the REST layer, delegating
all SQL stuff to the same action that backs the `/_sql` REST
API.
* Reworks "embedded testing mode" to use a `FilterClient` to
bounce capture the sql transport action and execute in embedded.
* Switches CLI formatting from consuming the entire response
to consuming just the first page of the response and returning
a `cursor` that can be used to read the next page. That read is
not yet implemented.
* Switch CLI formatting from the consuming the `RowSetCursor` to
consuming the `SqlResponse` object.
* Adds tests for CLI formatting.
* Support next page in the cli
* Rename cli's CommandRequest/CommandResponse to
QueryInitRequest/QueryInitResponse to line up with jdbc
* Implement QueryPageRequest/QueryPageResponse in cli
* Use `byte[]` to represent the cursor in the cli. Those bytes
mean something, but only to the server. The only reasonint that
the client does about them is "if length == 0 then there isn't a
next page."
* Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
QueryInitResponse, and QueryPageResponse into the shared-proto
project
* By implication this switches jdbc's QueryPageRequest to using
the same cursor implementation as the cli
Original commit: elastic/x-pack-elasticsearch@193586f1ee
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.
Original commit: elastic/x-pack-elasticsearch@ad906bc913
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@5765b7cd21
integrate forecasting feature branch into master
- add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
- current parameters: end time
- persists forecast results into shared or own index
- different runs are separated by a 'forecast id'
relates elastic/x-pack-elasticsearch#1838
Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.
Relates elastic/support-dev-help#2505
Original commit: elastic/x-pack-elasticsearch@54da044a27
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@21f5a58472
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.
This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".
relates elastic/x-pack-elasticsearch#2273
Original commit: elastic/x-pack-elasticsearch@b538923aca
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.
relates elastic/x-pack-elasticsearch#2472
Original commit: elastic/x-pack-elasticsearch@3224e836fa
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.
Relates elastic/x-pack-elasticsearch#2426
Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
"Established" memory use will be one of the building blocks for smarter node
allocation.
In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
time period covered by the last 20 buckets, which is defined as having a
coefficient of variation of no more than 0.1
Relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@5032eb01d8
Instead of throwing and catching an exception for invalid
indices this returns *why* they are invalid in a convenient
object form that can be thrown as an exception when the index
is required or the entire index can be ignored when listing
indices.
Original commit: elastic/x-pack-elasticsearch@f45cbce647
This integrates SQL's metadata calls with security by creating
`SqlIndicesAction` and routing all of SQL's metadata calls through
it. Since it *does* know up from which indices it is working against
it can be an `IndicesRequest.Replaceable` and integrate with the
existing security infrastructure for filtering indices.
This request is implemented fairly similarly to the `GetIndexAction`
with the option to read from the master or from a local copy of
cluster state. Currently SQL forces it to run on the local copy
because the request doesn't properly support serialization. I'd
like to implement that in a followup.
Original commit: elastic/x-pack-elasticsearch@15f9512820
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests. A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't. Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.
Original commit: elastic/x-pack-elasticsearch@4de42d9e54
When watcher is loading it must only load the watches
which are active instead of all possible watches.
This loading happens on start up as well as when shards
relocate.
Original commit: elastic/x-pack-elasticsearch@29df56b99d
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented. This change fixes some instances where this rule was
not adhered to.
Original commit: elastic/x-pack-elasticsearch@01c8f5172c
* Add support for authz checks at on shard requests
* Add Rest Tests for authorization
* Bulk security - Only reject individual items, rather than a whole shard
* Sync with core change
* Grant "delete" priv in ML smoketest
This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)
Original commit: elastic/x-pack-elasticsearch@830e89e652
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.
This fixes the test to ensure that always the same value
is returned and documents this requirement.
relates elastic/x-pack-elasticsearch#1783
Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.
This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.
relates elastic/x-pack-elasticsearch#2396
Original commit: elastic/x-pack-elasticsearch@8638df336c
If the duration time was 0 (and this might happen due to
using System.currentTimeMillis), the is_true check
still returns false.
The correct fix will be done later to replace the offending
measurement calls and replace them. Then we can add back this
line.
Original commit: elastic/x-pack-elasticsearch@076a9a37cc
A bunch of integration tests should have been built as unit tests
or already have unit test equivalents.
This commit removes integration tests as well as adding REST equivalents
or creating unit tests instead of extending from AbstractWatcherIntegrationTestCase
Original commit: elastic/x-pack-elasticsearch@a97b99467d
Scrolling was only implemented for the `SqlAction` (not jdbc or cli)
and it was implemented by keeping request state on the server. On
principle we try to avoid adding extra state to elasticsearch where
possible because it creates extra points of failure and tends to
have lots of hidden complexity.
This replaces the state on the server with serializing state to the
client. This looks to the user like a "next_page" key with fairly
opaque content. It actually consists of an identifier for the *kind*
of scroll, the scroll id, and a base64 string containing the field
extractors.
Right now this only implements scrolling for `SqlAction`. The plan
is to reuse the same implementation for jdbc and cli in a followup.
This also doesn't implement all of the required serialization.
Specifically it doesn't implement serialization of
`ProcessingHitExtractor` because I haven't implemented serialization
for *any* `ColumnProcessors`.
Original commit: elastic/x-pack-elasticsearch@a8567bc5ec
* Don’t count incomplete buckets in data stream diagnostics
* Fix tests now bucket_count doesn’t include partial buckets
Original commit: elastic/x-pack-elasticsearch@bc1a7bd9e7
This commit adapts to the renaming of the TransportResyncReplicationAction in core and also adds
an assertion to the check for a user being present when sending a request. The assertion is added
so that we can hopefully catch these scenarios in our testing as the assertion error will cause the
node to die but the ISE will just be seen in the logs. Since we do not run with assertions enabled
in production, the ISE is left to handle those cases.
relates elastic/x-pack-elasticsearch#2335
Original commit: elastic/x-pack-elasticsearch@c5ce0c93af
In `mutateInstance()` the from or size could become negative if the other one was pushed over the limit for `from + size`. This change fixes this case to make sure after the mutate method is called the from and size obey the limit but are also both `>= 0`
relates elastic/x-pack-elasticsearch#2344
Original commit: elastic/x-pack-elasticsearch@a8a7072fcc
* Add reserved dashboards_only_user role
* Fix line too long
* add tests for new reserved role
* rename role, hopefully fix tests
* Fix test
Original commit: elastic/x-pack-elasticsearch@99f6718c7c
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* Moves more classes over to ToXContentObject/Fragment
* Removes ToXContentToBytes
* Removes ToXContent from Enums
* review comment fix
* slight change to use XContantHelper
Original commit: elastic/x-pack-elasticsearch@0f2d3f328b
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.
Relates to elastic/elasticsearch#25859
Original commit: elastic/x-pack-elasticsearch@4bf266e0b0
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.
There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.
This also means, that there will be two entries popping up in the kibana
UI.
Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.
relates elastic/x-pack-elasticsearch#2238
Original commit: elastic/x-pack-elasticsearch@2c92040ed6
I noticed this while working on a previous issue with atomic move writer
(silent swallowing of exceptions). Namely, atomic move writer has
dangerous semantics. The problem is as follows: atomic move writer works
by writing lines to a temporary file, and then in its close method it
replaces the target path with the temporary file. However, the close
method is invoked whether or not all writes to the temporary file
succeeded (because writers obtained from atomic move writer are used in
try-with-resources blocks, as they should be). There is no way to
distinguish that the writer is being closed in a successful scenario
versus a failure scenario. In the close method for atomic move writer,
the target file is replaced by the temporary file. This means that the
target file is replaced whether or not writing to the temporary file
actually succeeded. Since these atomic move writers are used for user
configuration files (users and user_roles), a failure here can lead to
data loss for the user, a tragedy!
There is another (less serious) problem with the atomic move
writer. Since the close method tries to move the temporary file in place
of the existing file, the temporary file can be left behind if there is
another failure in the close method (e.g., closing the underlying file
after writing, or setting the permissions on the temporary file). This
means that in some situations, atomic move writer will leave temporary
files behind (which is not definitively not atomic).
This commit replaces the atomic move writer with a safer mechanism. We
still perform the write atomically in the following sense: we write to a
temporary file. Either writing to that file succeeds or it fails. If
writing succeeds, we replace the existing file with the temporary
file. If writing fails, we clean up the temporary file and the existing
file remains in place.
Relates elastic/x-pack-elasticsearch#2299
Original commit: elastic/x-pack-elasticsearch@3199decb0a
Certain types of datafeeds cannot have null chunking configs, so
setting chunking config to null sometimes doesn't stick as null
Original commit: elastic/x-pack-elasticsearch@3a52bad460
When a watch is executed currently, it gets passed an in-memory
watch object, that was loaded, before the execution started.
This means there is a window of time, where an old watch could still
be executing, then a watch gets loaded for execution, then the old watch
execution finishes and updates the watch status and thus reindexes the
watch.
Now the watch, that got loaded for execution, executes and tries to
store its watch status, but fails, because the version of the watch
has changed.
This commit changes the point in time where the watch is loaded. Now
this only happens, while a watch is in its protected execution block,
and thus we can be sure, that there is no other execution of the watch
happening.
This will primarily impact watches, that execute often, but their
runtime is longer than the configured interval between executions.
Side fix: Removed some duplicate testing method and moved into
WatcherTestUtils, fixed a tests with a ton of if's with random booleans
into separate tests.
relates elastic/x-pack-elasticsearch#395
Original commit: elastic/x-pack-elasticsearch@bf393023d7
Today it's impossible to run xpack bwc tests against any other branch
or remote than upstream. This allows to pass `-Dtests.bwc.refspec` to
change the refspec to use for the bwc tests.
Original commit: elastic/x-pack-elasticsearch@4d365f5a6e
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.
This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.
This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.
Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
This change means that newly created jobs will get an explicit 1GB
model memory limit if no model memory limit is specified when creating
the job. Existing jobs that had a null model memory limit will carry
on using the default model memory limit defined in the C++ code.
Relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@a4e6b73c2b
We rely on command extensions in our scripts but we do not actually
guarantee that they are enabled (usually they are, by default, but they
can be disabled outside of our control). This commit ensures that they
are enabled.
Relates elastic/x-pack-elasticsearch#2307
Original commit: elastic/x-pack-elasticsearch@a5eec8ca7b
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.
Relates to elastic/elasticsearch#26253
Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
The old message of "Cannot auto close job" implied the problem was with
closing the job. This change makes it clearer that the problem is that
the datafeed could not be stopped and hence auto-close will not even be
attempted.
Original commit: elastic/x-pack-elasticsearch@065e9930ce
These members are default initialized on contruction and then set by the
init() method. It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.
Original commit: elastic/x-pack-elasticsearch@668121e274
Today we try to bootstrap the security index with the bootstrap password and recommend the user to change the password with the user tool. This is trappy for instance if you happen to configure multiple nodes with a different bootstrap passwords (which is possible) it's unclear which password made it too bootstrap. Yet, we tell in the logs but it can still be very confusing. In general it should be possible to bootstrap with the user tool from any node unless the user is already created in the native user store. This change uses the bootstrap.password from the local node and always authenticate against it until the user is bootstrapped even if the passwords are different on different nodes. This will also work for authenticating against the cluster for instance if a user deletes the .security index or if that index has not been upgraded.
Original commit: elastic/x-pack-elasticsearch@8cebecb287
When writing the users and users_roles files, we wrap a custom writer in
a print writer. There is a problem with this though: when print writer
closes it closes our underlying custom writer and the close
implementation for our custom writer is not trivial, it executes code
that can throw an I/O exception. When print writer invokes this close
and an I/O exception is thrown, it swallows that exception and sets the
status on the print writer to error. One would think that we could
simply check this status but alas print writer is broken here. The act
of checking the status causes print writer to try to flush the
underyling stream which is going to be completely undefined because the
underlying stream might or might not be closed. This might cause another
exception to be thrown, losing the original. Print writer screwed the
pooch here, there is no good reason to try to do any I/O after the
underlying writer entered a failed state. To address this we remove the
use of print writer, we use our custom writer directly. This allows any
thrown exceptions to bubble up.
Relates elastic/x-pack-elasticsearch#2288
Original commit: elastic/x-pack-elasticsearch@11b8dd5641
When mappings are updated for an index are updated most settings are
merged, but not _meta. This change ensures that _meta is set when we
add per-job term mappings to our results index mappings. In order to
keep the logic for updating mappings after upgrade working, we now
have to put ALL the mappings for our results along with the latest _meta
section when updating per-job term mappings.
relates elastic/x-pack-elasticsearch#2265
Original commit: elastic/x-pack-elasticsearch@f58c11a13e
We close the secure settings in core before we pull bootstrap checks.
This means if a bootstrap check like the `TokenPassphraseBootstrapCheck`
accesses a secure setting that late it will fail due to an exception in
the `PKCS12KeyStore`. This change moves the bootstrap check creation
to the plugin constructor and adds a dummy setting to the integTest
that triggers the bootstrap checks.
Original commit: elastic/x-pack-elasticsearch@2b20865d1c
Adapts audit logging to actions that delay getting index access control until the action is started. The audit log will contain an entry for the action itself starting without any associated indices because the indices are not yet known. The audit log will also contain an entry for every time the action resolved security for a set of indices. Since sql resolves indices one at a time it will contain an entry per index.
All of this customization is entirely in the security code. The only SQL change in this PR is to add audit logging support to the integration test.
Original commit: elastic/x-pack-elasticsearch@539bb3c2a8
When the machine-learning-cpp repo is built locally, the zip file it
creates is preferred over that downloaded from s3 when creating the
overall x-pack-elasticsearch zip. However, prior to this change the
build would ALSO download an ml-cpp zip from s3, and just not use it.
Original commit: elastic/x-pack-elasticsearch@bd71637edd
PUT /_xpack/license with no content or content-type should fail with an appropriate error message rather than throwing NPE.
Original commit: elastic/x-pack-elasticsearch@f8c744d2a2
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
These tests used to fail rarely, because during a watch execution
one of the watcher shards was relocated resulting in a second execution
of watch.
In order to prevent this, the tests do not need to actually create any
shards, which causes watcher potentially to be rebalanced.
This simplifies and speeds up the test as well.
relates elastic/x-pack-elasticsearch#1608
Original commit: elastic/x-pack-elasticsearch@1cfac1145d
Adds a granular licensing support to SQL. JDBC now requires a platinum license, everything else work with any non-expired license.
Original commit: elastic/x-pack-elasticsearch@a30470e2c9
This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.
Original commit: elastic/x-pack-elasticsearch@c8c2819d86
When a watch is executed, it sends an update request to the watch to
udpate its status.
This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.
This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.
This will fix not only the current behaviour but also some flaky tests.
Original commit: elastic/x-pack-elasticsearch@ca69109ecb
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.
This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.
This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy
In addition a few Field interface classes were removed, that only contained parse fields.
relates elastic/x-pack-elasticsearch#1816
Original commit: elastic/x-pack-elasticsearch@b2ce680139
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs. Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster. However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.
For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@1e62b89183
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.
Original commit: elastic/x-pack-elasticsearch@a03f002bdc
Only unit tests were broken. Production ML code was always terminating
bulk requests with newlines.
Original commit: elastic/x-pack-elasticsearch@96ed06fed3
If one of the old watcher templates does not exist when we try
to delete it, the upgrade should just continue.
Original commit: elastic/x-pack-elasticsearch@6a52bad329
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.
On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.
Original commit: elastic/x-pack-elasticsearch@93b09878e4
This adds support for field level security to SQL by creating a new type of flow for securing requests that look like sql requests. `AuthorizationService` verifies that the user can execute the request but doesn't check the indices in the request because they are not yet ready. Instead, it adds a `BiFunction` to the context that can be used to check permissions for an index while servicing the request. This allows requests to cooperatively secure themselves. SQL does this by implementing filtering on top of its `Catalog` abstraction and backing that filtering with security's filters. This minimizes the touch points between security and SQL.
Stuff I'd like to do in followups:
What doesn't work at all still:
1. `SHOW TABLES` is still totally unsecured
2. `DESCRIBE TABLE` is still totally unsecured
3. JDBC's metadata APIs are still totally unsecured
What kind of works but not well:
1. The audit trail doesn't show the index being authorized for SQL.
Original commit: elastic/x-pack-elasticsearch@86f88ba2f5
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.
relates elastic/x-pack-elasticsearch#1823
Original commit: elastic/x-pack-elasticsearch@ac9bc82297
Igor Motov
jaymode
Chris Earle
Simon Willnauer
Hendrik Muhs
Ryan Ernst
David Roberts
Nik Everett
Tim Vernum
Tal Levy
Luca Cavanna
Jay Modi
David Kyle
Christoph Büscher
Tim Vernum
Chris Earle
Albert Zaharovits
Tanguy Leroux
Martijn van Groningen
Nhat
Alexander Reelsen
Dimitris Athanasiou
Jason Tedor
Lee Hinman
Costin Leau
Nik Everett
Costin Leau
Nhat
Tim Brooks
Tim Sullivan
Yannick Welsch
Suyog Rao
Michael Basnight
Andy Bristol
Jim Ferenczi
Colin Goodheart-Smithe
Stacey Gammon