* master:
992c7889ee Uncouple persistent task state and status (#31031)
8c6ee7db54 Describe how to add a plugin in Dockerfile (#31340)
1c5cec0ac7 Remove http status code maps (#31350)
87a676e4d5 Do not set vm.max_map_count when unnecessary (#31285)
e5b7137508 TEST: getCapturedRequestsAndClear should be atomic (#31312)
0324103737 Painless: Fix bug for static method calls on interfaces (#31348)
d6d0727aac QA: Fix resolution of default distribution (#31351)
fcf1e41e42 Extract common http logic to server (#31311)
6dd81ead74 Build: Fix the license in the pom zip and tar (#31336)
8f886cd4be Treat ack timeout more like a publish timeout (#31303)
9b293275af [ML] Add description to ML filters (#31330)
f7a0cafe55 SQL: Fix build on Java 10
375d09c588 [TEST] Fix RemoteClusterClientTests#testEnsureWeReconnect
4877cec3e8 More detailed tracing when writing metadata (#31319)
bbfe1eccc7 [Tests] Mutualize fixtures code in BaseHttpFixture (#31210)
This pull request removes the relationship between the state
of persistent task (as stored in the cluster state) and the status
of the task (as reported by the Task APIs and used in various
places) that have been confusing for some time (#29608).
In order to do that, a new PersistentTaskState interface is added.
This interface represents the persisted state of a persistent task.
The methods used to update the state of persistent tasks are
renamed: updatePersistentStatus() becomes updatePersistentTaskState()
and now takes a PersistentTaskState as a parameter. The
Task.Status type as been changed to PersistentTaskState in all
places were it make sense (in persistent task customs in cluster
state and all other methods that deal with the state of an allocated
persistent task).
This is related to #28898. With the addition of the http nio transport,
we now have two different modules that provide http transports.
Currently most of the http logic lives at the module level. However,
some of this logic can live in server. In particular, some of the
setting of headers, cors, and pipelining. This commit begins this moving
in that direction by introducing lower level abstraction (HttpChannel,
HttpRequest, and HttpResonse) that is implemented by the modules. The
higher level rest request and rest channel work can live entirely in
server.
For 6.3 we renamed the `tar` and `zip` distributions to `oss-tar` and
`oss-zip`. Then we added new `tar` and `zip` distributions that contain
x-pack and are licensed under the Elastic License. Unfortunately we
accidentally generated POM files along side the new `tar` and `zip`
distributions that incorrectly claimed that they were Apache 2 licensed.
Oooops.
This fixes the license on the POMs generated for the `tar` and `zip`
distributions.
This adds a `description` to ML filters in order
to allow users to describe their filters in a human
readable form which is also editable (filter updates
to be added shortly).
Due to a runtime classpath clash, featureAware task was failing on JVMs
higher than 1.8 (since the ASM version from Painless was used instead
which does not recognized Java 9 or 10 bytecode) causing the task to
fail.
This commit excludes the ASM dependency (since it's not used by SQL
itself).
* master:
Remove RestGetAllAliasesAction (#31308)
Temporary fix for broken build
Reenable Checkstyle's unused import rule (#31270)
Remove remaining unused imports before merging #31270
Fix non-REST doc snippet
[DOC] Extend SQL docs
Immediately flush channel after writing to buffer (#31301)
[DOCS] Shortens ML API intros
Use quotes in the call invocation (#31249)
move security ingest processors to a sub ingest directory (#31306)
Add 5.6.11 version constant.
Fix version detection.
SQL: Whitelist SQL utility class for better scripting (#30681)
[Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)
CCS: don't proxy requests for already connected node (#31273)
Mute ScriptedMetricAggregatorTests testSelfReferencingAggStateAfterMap
[test] opensuse packaging turn up debug logging
Add unreleased version 6.3.1
Removes experimental tag from scripted_metric aggregation (#31298)
[Rollup] Metric config parser must use builder so validation runs (#31159)
[ML] Check licence when datafeeds use cross cluster search (#31247)
Add notion of internal index settings (#31286)
Test: Remove broken yml test feature (#31255)
REST hl client: cluster health to default to cluster level (#31268)
[ML] Update test thresholds to account for changes to memory control (#31289)
Log warnings when cluster state publication failed to some nodes (#31233)
Fix AntFixture waiting condition (#31272)
Ignore numeric shard count if waiting for ALL (#31265)
[ML] Implement new rules design (#31110)
index_prefixes back-compat should test 6.3 (#30951)
Core: Remove plain execute method on TransportAction (#30998)
Update checkstyle to 8.10.1 (#31269)
Set analyzer version in PreBuiltAnalyzerProviderFactory (#31202)
Modify pipelining handlers to require full requests (#31280)
Revert upgrade to Netty 4.1.25.Final (#31282)
Use armored input stream for reading public key (#31229)
Fix Netty 4 Server Transport tests. Again.
REST hl client: adjust wait_for_active_shards param in cluster health (#31266)
REST high-level Client: remove deprecated API methods (#31200)
[DOCS] Mark SQL feature as experimental
[DOCS] Updates machine learning custom URL screenshots (#31222)
Fix naming conventions check for XPackTestCase
Fix security Netty 4 transport tests
Fix race in clear scroll (#31259)
[DOCS] Clarify audit index settings when remote indexing (#30923)
Delete typos in SAML docs (#31199)
REST high-level client: add Cluster Health API (#29331)
[ML][TEST] Mute tests using rules (#31204)
Support RequestedAuthnContext (#31238)
SyncedFlushResponse to implement ToXContentObject (#31155)
Add Get Aliases API to the high-level REST client (#28799)
Remove some line length supressions (#31209)
Validate xContentType in PutWatchRequest. (#31088)
[INGEST] Interrupt the current thread if evaluation grok expressions take too long (#31024)
Suppress extras FS on caching directory tests
Revert "[DOCS] Added 6.3 info & updated the upgrade table. (#30940)"
Revert "Fix snippets in upgrade docs"
Fix snippets in upgrade docs
[DOCS] Added 6.3 info & updated the upgrade table. (#30940)
LLClient: Support host selection (#30523)
Upgrade to Netty 4.1.25.Final (#31232)
Enable custom credentials for core REST tests (#31235)
Move ESIndexLevelReplicationTestCase to test framework (#31243)
Encapsulate Translog in Engine (#31220)
HLRest: Add get index templates API (#31161)
Remove all unused imports and fix CRLF (#31207)
[Tests] Fix self-referencing tests
[TEST] Fix testRecoveryAfterPrimaryPromotion
[Docs] Remove mention pattern files in Grok processor (#31170)
Use stronger write-once semantics for Azure repository (#30437)
Don't swallow exceptions on replication (#31179)
Limit the number of concurrent requests per node (#31206)
Call ensureNoSelfReferences() on _agg state variable after scripted metric agg script executions (#31044)
Move java version checker back to its own jar (#30708)
[test] add fix for rare virtualbox error (#31212)
We currently have a specific REST action to retrieve all aliaes, which
uses internally the get index API. This doesn't seem to be required
anymore though as the existing RestGetAliaesAction could as well take
the requests with no indices and aliases specified.
This commit removes the RestGetAllAliasesAction in favour of using
RestGetAliasesAction also for requests that don't specify indices nor
aliases. Similar to #31129.
x-pack/sql depends on lang-painless which depends on ASM 5.1
FeatureAwareCheck needs ASM 6
This is a hack to strip ASM5 from the classpath for FeatureAwareCheck
This is related to #27260. Currently when we queue a write with a
channel we set OP_WRITE and wait until the next selection loop to flush
the write. However, if the channel does not have a pending write, it
is probably ready to flush. This PR implements an optimistic flush logic
that will attempt this flush.
- All rollup pages should be marked as experimental instead of just
the top page
- While the job config docs state which aggregations are allowed, adding
a section which specifically details this in one place is more convenient
for the user
- Add a clarification that the DeleteJob API does not delete the rollup
data, just the rollup job.
The parser for the Metric config was directly instantiating
the config object, rather than using the builder. That means it was
bypassing the validation logic built into the builder, and would allow
users to create invalid metric configs (like using unsupported metrics).
The job would later blow up and abort due to bad configs, but this isn't
immediately obvious to the user since the PutJob API succeeded.
This change prevents a datafeed using cross cluster search from starting if the remote cluster
does not have x-pack installed and a sufficient license. The check is made only when starting a
datafeed.
Rules allow users to supply a detector with domain
knowledge that can improve the quality of the results.
The model detects statistically anomalous results but it
has no knowledge of the meaning of the values being modelled.
For example, a detector that performs a population analysis
over IP addresses could benefit from a list of IP addresses
that the user knows to be safe. Then anomalous results for
those IP addresses will not be created and will not affect
the quantiles either.
Another example would be a detector looking for anomalies
in the median value of CPU utilization. A user might want
to inform the detector that any results where the actual
value is less than 5 is not interesting.
This commit introduces a `custom_rules` field to the `Detector`.
A detector may have multiple rules which are combined with `or`.
A rule has 3 fields: `actions`, `scope` and `conditions`.
Actions is a list of what should happen when the rule applies.
The current options include `skip_result` and `skip_model_update`.
The default value for `actions` is the `skip_result` action.
Scope is optional and allows for applying filters on any of the
partition/over/by field. When not defined the rule applies to
all series. The `filter_id` needs to be specified to match the id
of the filter to be used. Optionally, the `filter_type` can be specified
as either `include` (default) or `exclude`. When set to `include`
the rule applies to entities that are in the filter. When set to
`exclude` the rule only applies to entities not in the filter.
There may be zero or more conditions. A condition requires `applies_to`,
`operator` and `value` to be specified. The `applies_to` value can be
either `actual`, `typical` or `diff_from_typical` and it specifies
the numerical value to which the condition applies. The `operator`
(`lt`, `lte`, `gt`, `gte`) and `value` complete the definition.
Conditions are combined with `and` and allow to specify numerical
conditions for when a rule applies.
A rule must either have a scope or one or more conditions. Finally,
a rule with scope and conditions applies when all of them apply.
TransportAction has many variants of execute. One of those variants
executes by returning a future, which is then often blocked on by
calling get(). This commit removes this variant of execute, instead
using a helper method for tests that want to block, or having tests
pass in a PlainActionFuture directly as a listener.
Co-authored-by: Simon Willnauer <simonw@apache.org>
This is in preparation of pushing the new
rules design in the `ml-cpp` side. These
tests will be switched on again after merging
in the new rules implementation.
* Support RequestedAuthnContext
This implements limited support for RequestedAuthnContext by :
- Allowing SP administrators to define a list of authnContextClassRef
to be included in the RequestedAuthnContext of a SAML Authn Request
- Veirifying that the authnContext in the incoming SAML Asertion's
AuthnStatement contains one of the requested authnContextClassRef
- Only EXACT comparison is supported as the semantics of validating
the incoming authnContextClassRef are deployment dependant and
require pre-established rules for MINIMUM, MAXIMUM and BETTER
Also adds necessary AuthnStatement validation as indicated by [1] and
[2]
[1] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
3.4.1.4, line 2250-2253
[2] https://kantarainitiative.github.io/SAMLprofiles/saml2int.html
[SDP-IDP10]
Trying to post a new watch without any body currently results in a
NullPointerException. This change fixes that by validating that
Post and Put requests always have a body.
Closes#30057
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```
This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.
I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.
Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:
```
---
"example test":
- skip:
features: host_selector
- do:
host_selector:
version: " - 7.0.0" # same syntax as skip
apiname:
something: true
```
The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.
The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.
Closes#21888
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
The core REST tests with security currently use a hardcoded username and
password. This is not amenable to running these tests in scenarios where
the user controls the creation of the cluster and owns the credentials
for this cluster. This commit enables running the core REST tests with
security with a custom username and password.
* master:
Move default location of dependencies report (#31228)
Remove dependencies report task dependencies (#31227)
Add recognition of MPL 2.0 (#31226)
Fix unknown licenses (#31223)
Remove version from license file name for GCS SDK (#31221)
Fully encapsulate LocalCheckpointTracker inside of the engine (#31213)
[DOCS] Added 'fail_on_unsupported_field' param to MLT. Closes#28008 (#31160)
Add licenses for transport-nio (#31218)
Remove DocumentFieldMappers#simpleMatchToFullName. (#31041)
Allow to trim all ops above a certain seq# with a term lower than X, post backport fix (#31211)
Compliant SAML Response destination check (#31175)
Remove DocumentFieldMappers#smartNameFieldMapper, as it is no longer needed. (#31018)
Remove extraneous references to 'tokenized' in the mapper code. (#31010)
Allow to trim all ops above a certain seq# with a term lower than X (#30176)
SQL: Make a single JDBC driver jar (#31012)
Enhance license detection for various licenses (#31198)
[DOCS] Add note about long-lived idle connections (#30990)
Move number of language analyzers to analysis-common module (#31143)
Default max concurrent search req. numNodes * 5 (#31171)
flush job to ensure all results have been written (#31187)
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
* Remove DocumentFieldMappers#simpleMatchToFullName, as it is duplicative of MapperService#simpleMatchToIndexNames.
* Rename MapperService#simpleMatchToIndexNames -> simpleMatchToFullName for consistency.
* Simplify EsIntegTestCase#assertConcreteMappingsOnAll to accept concrete fields instead of wildcard patterns.
Make SAML Response Destination check compliant
Only validate the Destination element of an incoming SAML Response
if Destination is present and the SAML Response is signed.
The standard [1] - 3.5.5.2 and [2] - 3.2.2 does mention that the
Destination element is optional and should only be verified when
the SAML Response is signed. Some Identity Provider implementations
are known to not set a Destination XML Attribute in their SAML
responses when those are not signed, so this change also aims to
enhance interoperability.
[1] https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf
[2] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
* elastic/master: (53 commits)
Painless: Restructure/Clean Up of Spec Documentation (#31013)
Update ignore_unmapped serialization after backport
Add back dropped substitution on merge
high level REST api: cancel task (#30745)
Enable engine factory to be pluggable (#31183)
Remove vestiges of animal sniffer (#31178)
Rename elasticsearch-nio to nio (#31186)
Rename elasticsearch-core to core (#31185)
Move cli sub-project out of server to libs (#31184)
[DOCS] Fixes broken link in auditing settings
QA: Better seed nodes for rolling restart
[DOCS] Moves ML content to stack-docs
[DOCS] Clarifies recommendation for audit index output type (#31146)
Add nio-transport as option for http smoke tests (#31162)
QA: Set better node names on rolling restart tests
Add support for ignore_unmapped to geo sort (#31153)
Share common parser in some AcknowledgedResponses (#31169)
Fix random failure on SearchQueryIT#testTermExpansionExceptionOnSpanFailure
Remove reference to multiple fields with one name (#31127)
Remove BlobContainer.move() method (#31100)
...
Tanguy Leroux
Martijn van Groningen
Tim Brooks
Nik Everett
Dimitris Athanasiou
Nhat Nguyen
Costin Leau
Luca Cavanna
Tim Vernum
lcawl
Tal Levy
Costin Leau
Zachary Tong
David Kyle
Tom Veasey
Ryan Ernst
Jason Tedor
James Baiera
Albert Zaharovits
Yuri Tceretian
Ioannis Kakavas
Aditya Dhulipala
Julie Tibshirani
Igor Motov
Hendrik Muhs