* add QA-style Rest tests to index-lifecycle
This PR introduces a `qa` module within the index-lifecycle project.
the idea is to have both complex policies tested, as well as
policies with isolated/singular actions. So far, only tests with policies
containing one action are implemented.
Following Actions have implemented tests in this commit
- AllocateAction
- DeleteAction
- ForceMergeAction
- ReadOnlyAction
- ReplicasAction
tests to be added later
- RolloverAction
- ShrinkAction
* respond to review and enable integTests
* fix dependsOn fiasco
* fix license
* update to new proj structure
* move to new integTest with x-pack-core as module
* remove unused imports
* update to use module instead of plugin
This commit removes xpack from being a meta-plugin-as-a-module.
It also fixes a couple tests which were missing task dependencies, which
failed once the gradle execution order changed.
Removes dependency for server's version from the JDBC driver code. This
should allow us to dramatically reduce driver's size by removing the
server dependency from the driver.
Relates #29856
This commit increases the logging level around search to aid in
debugging failures in LicensingTests#testSecurityActionsByLicenseType
where we are seeing all shards failed error while trying to search the
security index.
See #30301
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the
task management API.
Related to #27205
* Refactors ClientHelper to combine header logic
This change removes all the `*ClientHelper` classes which were
repeating logic between plugins and instead adds
`ClientHelper.executeWithHeaders()` and
`ClientHelper.executeWithHeadersAsync()` methods to centralise the
logic for executing requests with stored security headers.
* Removes Watcher headers constant
When the encrpytion of sensitive date is enabled, test that a
scheduled watch is executed as expected and produces the correct value
from a secret in the basic auth header.
The `ClusterStateWaitStep.isConditionMet()` method now returns a
`Result` object which contains a boolean for if the condition is met
and an `ToXContentObject` to provide information in the case where the
condition is not met.
If the condition is not met, the step information is stored in the
cluster state
Make SSLContext reloadable
This commit replaces all customKeyManagers and TrustManagers
(ReloadableKeyManager,ReloadableTrustManager,
EmptyKeyManager, EmptyTrustManager) with instances of
X509ExtendedKeyManager and X509ExtendedTrustManager.
This change was triggered by the effort to allow Elasticsearch to
run in a FIPS-140 environment. In JVMs running in FIPS approved
mode, only SunJSSE TrustManagers and KeyManagers can be used.
Reloadability is now ensured by a volatile instance of SSLContext
in SSLContectHolder.
SSLConfigurationReloaderTests use the reloadable SSLContext to
initialize HTTP Clients and Servers and use these for testing the
key material and trust relations.
This commit is related to #28898. It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
Due to the way composite aggregation works, ordering in GROUP BY can be
applied only through grouped columns which now the analyzer verifier
enforces.
Fix 29900
This commit removes the SecurityLifecycleService, relegating its former
functions of listening for cluster state updates to SecurityIndexManager
and IndexAuditTrail.
This change adds a grok_pattern field to the GET categories API
output in ML. It's calculated using the regex and examples in the
categorization result, and applying a list of candidate Grok
patterns to the bits in between the tokens that are considered to
define the category.
This can currently be considered a prototype, as the Grok patterns
it produces are not optimal. However, enough people have said it
would be useful for it to be worthwhile exposing it as experimental
functionality for interested parties to try out.
This is fixing an issue that has come up in some builds. In some
scenarios I see an assertion failure that we are trying to move to
application mode when we are not in handshake mode. What I think is
happening is that we are in handshake mode and have received the
completed handshake message AND an application message. While reading in
handshake mode we switch to application mode. However, there is still
data to be consumed so we attempt to continue to read in handshake mode.
This leads to us attempting to move to application mode again throwing
an assertion.
This commit fixes this by immediatly exiting the handshake mode read
method if we are not longer in handshake mode. Additionally if we swap
modes during a read we attempt to read with the new mode to see if there
is data that needs to be handled.
The errors were caused because release tests would use a copy of
the public key that was formatted differently. The change to the
public key format was introduced in [1].
Release tests Jenkins job has now been updated to use the correct
key format depending on the branch they run on [2]
Closes#30430
[1] https://github.com/elastic/elasticsearch/pull/30251
[2] https://github.com/elastic/infra/pull/4944
* Adds `index.lifecycle.step_info` setting and uses it on ERROR
This change make a new `index.lifecycle.step_info` setting which can be
used to store a JSON blob of containing context about the current step.
It then adds code so that when we move to the error step we serialise
the exception to JSON and store it in this setting so the user can get
information on why the step failed.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycle.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
* Adds step information for AsyncWaitSteps
The Listener for AsyncWaitStep now takes a ToXContentObject which
represents information about the status of the condition if it has not
completed. This object is then serialised to a JSON string and stored
in the `index.lifecycle.step_info` index setting. This information is
only stored if the step is not complete. If the step is complete the
step info is ignored sice we will move straight to the next step where
the info is no longer relevant.
Changes for the `ClusterStateWaitStep` will be very similar but will be
made in a following commit after this approach has been agreed. I do
not intend to have information for `AsyncActionStep` to have the
ability to set step info since actions should either be done or not
done and if they error they should transition to the ERROR step.
* Clear step info when transitioning to next step
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
* Addresses review comments
Since adding back the per-watch statistics, we do not need to access
every trigger engine implementation to get the current total job count.
This commit removes the unused methods to do so.
Tweak the return data, in particular with regards for ODBC columns to
better align with the spec
Fix order for SYS TYPES and TABLES according to the JDBC/ODBC spec
Fix#30386Fix#30521
This commit cleans up some code in the FileUserPasswdStore and the
FileUserRolesStore classes. The maps used in these classes are volatile
so we need to make sure that we don't perform multiple operations with
the map unless we are sure we are using a reference to the same map.
The maps are also never null, but there were a few null checks in the
code that were not needed. These checks have been removed.
The TokenMetaData equals method compared byte arrays using `.equals` on
the arrays themselves, which is the equivalent of an `==` check. This
means that a seperate byte[] with the same contents would not be
considered equivalent to the existing one, even though it should be.
The method has been updated to use `Array#equals` and similarly the
hashcode method has been updated to call `Arrays#hashCode` instead of
calling hashcode on the array itself.
These tests are both in the file `watcher/stats/10_basic`, and have been
failing fairly frequently over the last month with a start-up issue.
The issue is being tracked in #30298.
Dates internally contain milliseconds (which appear when converting them
to Strings) however parsing does not accept them (and is being strict).
The parser has been changed so that Date is mandatory but the time
(including its fractions such as millis) are optional.
Fix#30002
This commit adds a general state listener to the SecurityIndexManager,
and replaces the existing health and up-to-date listeners with that. It
also moves helper methods relating to health to SecurityIndexManager
from SecurityLifecycleService.
As conformance to best practices, this changes ensures that if a
SAML Response is signed, we verify the signature before processing
it any further. We were only checking the InResponseTo and
Destination attributes before potential signature validation but
there was no reason to do that up front either.
With the opening of xpack, we still retained a run task within
:x-pack:plugin. However, the root level run task also runs with the
default distribution. This change removes the extra run task inside
xpack in favor of using the root level task, and moves the
license/configuration code for run into the main run configuration.
This commit removes the hardcoded list of unconfigured ciphers in the
SslIntegrationTests. This list may include ciphers that are not
supported on certain JVMs. This list is replaced with code that
dynamically computes the set of ciphers that are not configured for
use by default.
Makes most logging at debug rather than warn and error.
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleService.java
The HTTPClient used in watcher is based on the apache http client. The
current client is using a lot of defaults - which are not always
optimal. Two of those defaults are the maximum number of total
connections and the maximum number of connections to a single route.
If one of those limits is reached, the HTTPClient waits for a connection
to be finished thus acting in a blocking fashion. In order to prevent
this when many requests are being executed, we increase the limit of
total connections as well as the connections per route (a route is
basically an endpoint, which also contains proxy information, not
containing an URL, just hosts).
On top of that an additional option has been set to evict
long running connections, which can potentially be reused after some
time. As this requires an additional background thread, this required
some changes to ensure that the httpclient is closed properly. Also the
timeout for this can be configured.
This commit renames IndexLifecycleManager to SecurityIndexManager as it
is not actually a general purpose class, but specific to security. It
also removes indirection in code calling the lifecycle service, instead
calling the security index manager directly.
Tal Levy
Ryan Ernst
Dimitris Athanasiou
Igor Motov
jaymode
Van0SS
Colin Goodheart-Smithe
Alexander Reelsen
Ioannis Kakavas
Tim Brooks
Costin Leau
Costin Leau
Ioannis Kakavas
David Roberts
David Kyle
Jay Modi
Julie Tibshirani
Nhat Nguyen
Jack Conradson