With #31020 we introduced the ability for transport clients to indicate what features they support
in order to make sure we don't serialize object to them they don't support. This PR adapts the
serialization logic of persistent tasks to be aware of those features and not serialize tasks that
aren't supported.
Also, a version check is added for the future where we may add new tasks implementations and
need to be able to indicate they shouldn't be serialized both to nodes and clients.
As the implementation relies on the interface of `PersistentTaskParams`, these are no longer
optional. That's acceptable as all current implementation have them and we plan to make
`PersistentTaskParams` more central in the future.
Relates to #30731
If you invoke elasticsearch-plugin (or any other CLI script on Windows)
with a path that has a percent-encoded space (or any other
percent-encoded character) because the CLI scripts now shell into a
common shell script (elasticsearch-cli) the percent-encoded space ends
up being interpreted as a parameter. For example passing install --batch
file:/c:/encoded%20%space/analysis-icu-7.0.0.zip to elasticsearch-plugin
leads to the %20 being interpreted as %2 followed by a zero. Here, the
%2 is interpreted as the second parameter (--batch) and the
InstallPluginCommand class ends up seeing
file:/c/encoded--batch0space/analysis-icu-7.0.0.zip as the path which
will not exist. This commit addresses this by escaping the %* that is
used to pass the parameters to the common CLI script so that the common
script sees the correct parameters without the %2 being substituted.
* Adds an explain API endpoint
This endpoint can be used to explain the current lifecycle state of an
index
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/ExplainLifecycleAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/IndexExplainResponse.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/action/TransportExplainLifecycleAction.java
x-pack/plugin/src/test/resources/rest-api-spec/test/index_lifecycle/20_m
ove_to_step.yml
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/ExplainLifecycleAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/IndexExplainResponse.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RandomStepInfo.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/ExplainLifecycleRequestTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/ExplainLifecycleResponseTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/action/IndexExplainResponseTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycle.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/action/RestExplainLifecycleAction.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/action/TransportExplainLifecycleAction.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/src/test/resources/rest-api-spec/api/xpack.index_lifecycle
.explain_lifecycle.json
x-pack/plugin/src/test/resources/rest-api-spec/test/index_lifecycle/20_m
ove_to_step.yml
x-pack/plugin/src/test/resources/rest-api-spec/test/index_lifecycle/30_e
xplain_lifecycle.yml
* Adds tests for explain API
* Addresses Review comments and fixes REST tests
* Removes RequestBuilder from ExplainLifecycleAction
This commit introduces the ability for a client to communicate to the
server features that it can support and for these features to be used in
influencing the decisions that the server makes when communicating with
the client. To this end we carry the features from the client to the
underlying stream as we carry the version of the client today. This
enables us to enhance the logic where we make protocol decisions on the
basis of the version on the stream to also make protocol decisions on
the basis of the features on the stream. With such functionality, the
client can communicate to the server if it is a transport client, or if
it has, for example, X-Pack installed. This enables us to support
rolling upgrades from the OSS distribution to the default distribution
without breaking client connectivity as we can now elect to serialize
customs in the cluster state depending on whether or not the client
reports to us using the feature capabilities that it can under these
customs. This means that we would avoid sending a client pieces of the
cluster state that it can not understand. However, we want to take care
and always send the full cluster state during node-to-node communication
as otherwise we would end up with different understanding of what is in
the cluster state across nodes depending on which features they reported
to have. This is why when deciding whether or not to write out a custom
we always send the custom if the client is not a transport client and
otherwise do not send the custom if the client is transport client that
does not report to have the feature required by the custom.
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Otherwise we could end up with persistent tasks metadata in the cluster that some of the nodes
might not understand in case where the cluster is during rolling upgrade from the default 6.2 to the
default 6.3 distribution.
Follow-up to #30743
* Retain the expiryDate for trial licenses
While updating the license signature to the new license spec retain
the trial license expiration date to that of the existing license.
Resolves#30882
* Ensure that a purposefully wrong key is used
Uses a specific keypair for tests that require a purposefully wrong
keypair instead of selecting one randomly from the same pull from
which the correct one is selected. Entropy is low because of the
small space and the same key can be randomly selected as both the
correct one and the wrong one, causing the tests to fail.
The purposefully wrong key is also used in
testSigningKeyIsReloadedForEachRequest and needs to be cleaned
up afterwards so the rest of the tests don't use that for signing.
Resolves#30970
This commit removes the RequestBuilder generic type from Action. It was
needed to be used by the newRequest method, which in turn was used by
client.prepareExecute. Both of these methods are now removed, along with
the existing users of prepareExecute constructing the appropriate
builder directly.
This commit reworks the Sniffer component to simplify it and make it possible to test it.
In particular, it no longer takes out the host that failed when sniffing on failure, but rather relies on whatever the cluster returns. This is the result of some valid comments from #27985. Taking out one single host is too naive, hard to test and debug.
A new Scheduler abstraction is introduced to abstract the tasks scheduling away and make it possible to plug in any test implementation and take out timing aspects when testing.
Concurrency aspects have also been improved, synchronized methods are no longer required. At the same time, we were able to take #27697 and #25701 into account and fix them, especially now that we can more easily add tests.
Last but not least, unit tests are added for the Sniffer component, long overdue.
Closes#27697Closes#25701
This code is from an Apache 2.0 licensed codebase and when we imported
it into our codebase it carried the Apache 2.0 license as well. However,
during the migration of the X-Pack codebase from the internal private
repository to the elastic/elasticsearch repository, the migration tool
mistakently changed the license on this source file from the Apache 2.0
license to the Elastic license. This commit addresses this mistake by
reapplying the Apache 2.0 license.
This commit renames methods in the PersistentTasksService, to
make obvious that the methods send requests in order to change
the state of persistent tasks.
Relates to #29608.
ML has dedicated APIs for datafeeds and jobs yet base test classes and
some tests were relying on the cluster state for this state. This commit
removes this usage in favor of using the dedicated endpoints.
Limits the scope of the runtime dependency on
BouncyCastle so that it can be eventually removed.
* Splits functionality related to reading and generating certificates
and keys in two utility classes so that reading certificates and
keys doesn't require BouncyCastle.
* Implements a class for parsing PEM Encoded key material (which also
adds support for reading PKCS8 encoded encrypted private keys).
* Removes BouncyCastle dependency for all of our test suites(except
for the tests that explicitly test certificate generation) by using
pre-generated keys/certificates/keystores.
Removes the last remaining server dependencies from jdbc client. In order to do that it introduces the new project sql-shared-proto that contains only XContent-serializable classes. HTTP Client and JDBC now depend only on sql-shared-proto. I had to keep the original sql-proto project since it is used as a dependency by sql-cli and security integration tests.
Relates #29856
This is a bug that was identified by the kibana team. Currently on a
get-license call we do not serialize the hard-coded expiration for basic
licenses. However, the kibana team calls the x-pack info route which
still does serialize the expiration date. This commit removes that
serialization in the rest response.
As we are preparing to support policy updates/changes, we noticed
that restricting allocation wait steps with pinned replicas/shard
counts makes this difficult to continue from. For example,
as user may update or switch a policy to increase replicas. If this
is done, then the check will never pass and user intervention will
be required. If we simply remove this restriction, we still check
that the index is allocated correctly, but without depending on
the newly configured replicas setting in the policy.
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
This commit reworks the way our realms perform caching in order to
limit each principal to a single ongoing authentication per realm. In
other words, this means that multiple requests made by the same user
will not trigger more that one authentication attempt at a time if no
entry has been stored in the cache. If an entry is present in our
cache, there is no restriction on the number of concurrent
authentications performed for this user.
This change enables us to limit the load we place on an external system
like an LDAP server and also preserve resources such as CPU on
expensive operations such as BCrypt authentication.
Closes#30355
This commit fixes an issue with dynamic mapping updates when an index
operation is performed against an alias and when the user only has
permissions to the alias. Dynamic mapping updates resolve the concrete
index early to prevent issues so the information about the alias that
the triggering operation was being executed against is lost. When
security is enabled and a user only has privileges to the alias, this
dynamic mapping update would be rejected as it is executing against the
concrete index and not the alias. In order to handle this situation,
the security code needs to look at the concrete index and the
authorized indices of the user; if the concrete index is not authorized
the code will attempt to find an alias that the user has permissions to
update the mappings of.
Closes#30597
The .watcher-history-* template is currently using a plugin-custom index setting xpack.watcher.template.version,
which prevents this template from being installed in a mixed OSS / X-Pack cluster, ultimately
leading to the situation where an X-Pack node is constantly spamming an OSS master with (failed)
template updates. Other X-Pack templates (e.g. security-index-template or security_audit_log)
achieve the same versioning functionality by using a custom _meta field in the mapping instead.
This commit switches the .watcher-history-* template to use the _meta field instead.
* Changes PhaseAfterStep to take the name of the previous phase
This changes the way the phase after step is built so its key has the
phase name of the phase that preceeds it rather than the phase that
follows it. This is more intuitive to the user since the index is in
the warm phase until the after condition for the cold phase is met.
* Fixes REST tests
x-pack/plugin/src/test/resources/rest-api-spec/test/index_lifecycle/20_m
ove_to_step.yml
x-pack/plugin/src/test/resources/rest-api-spec/test/index_lifecycle/20_m
ove_to_step.yml
Persistent tasks was moved from X-Pack to core in #28455.
However, registration of the named writables and named
X-content was left in X-Pack.
This change moves the registration of the named writables
and named X-content into core. Additionally, the persistent
task actions are no longer registered in the X-Pack client
plugin, as they are already registered in ActionModule.
This change adds a simple header to the transport client
that is present on the servers thread context that ensures
we can detect if a transport client talks to the server in a
specific request. This change also adds a header for xpack
to detect if the client has xpack installed.
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
This reverts commit 31251c9 introduced in #30695.
We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.
Closes#27740
Enables a rolling restart from the OSS distribution to the x-pack based distribution by preventing
x-pack code from installing custom metadata into the cluster state until all nodes are capable of
deserializing this metadata.
Colin Goodheart-Smithe
Boaz Leskes
Jason Tedor
Tal Levy
Yannick Welsch
Nik Everett
Ioannis Kakavas
Ryan Ernst
Luca Cavanna
Igor Motov
Tanguy Leroux
Nhat Nguyen
Christoph Büscher
Tim Brooks
Jay Modi
David Roberts
Simon Willnauer
Adrien Grand