Here is the problem: if two threads are racing and one hits a failure
freeing a context and the other succeeded, we can expose the value of
the has failure marker to the succeeding thread before the failing
thread has had a chance to set the failure marker. This is a problem if
the failing thread counted down the expected number of operations, then
be put to sleep by a gentle lullaby from the OS, and then the other
thread could count down to zero. Since the failing thread did not get to
set the failure marker, the succeeding thread would respond that the
clear scroll succeeded and that makes that thread a liar. This commit
addresses by first setting the failure marker before we potentially
expose its value to another thread.
This is in preparation of pushing the new
rules design in the `ml-cpp` side. These
tests will be switched on again after merging
in the new rules implementation.
* Support RequestedAuthnContext
This implements limited support for RequestedAuthnContext by :
- Allowing SP administrators to define a list of authnContextClassRef
to be included in the RequestedAuthnContext of a SAML Authn Request
- Veirifying that the authnContext in the incoming SAML Asertion's
AuthnStatement contains one of the requested authnContextClassRef
- Only EXACT comparison is supported as the semantics of validating
the incoming authnContextClassRef are deployment dependant and
require pre-established rules for MINIMUM, MAXIMUM and BETTER
Also adds necessary AuthnStatement validation as indicated by [1] and
[2]
[1] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
3.4.1.4, line 2250-2253
[2] https://kantarainitiative.github.io/SAMLprofiles/saml2int.html
[SDP-IDP10]
The response currently implements ToXContentFragment although the only time it's used
it is supposed to print out a complete object rather than a fragment. Note that this
is the client version of the response, used only in the high-level client.
Given the weirdness of the response returned by the get alias API, we went for a client specific response, which allows us to hold the error message, exception and status returned as part of the response together with aliases. See #30536 .
Relates to #27205
Trying to post a new watch without any body currently results in a
NullPointerException. This change fixes that by validating that
Post and Put requests always have a body.
Closes#30057
This adds a thread interrupter that allows us to encapsulate calls to org.joni.Matcher#search()
This method can hang forever if the regex expression is too complex.
The thread interrupter in the background checks every 3 seconds whether there are threads
execution the org.joni.Matcher#search() method for longer than 5 seconds and
if so interrupts these threads.
Joni has checks that that for every 30k iterations it checks if the current thread is interrupted and
if so returns org.joni.Matcher#INTERRUPTED
Closes#28731
This filesystem needs to be suppressed during these tests because it
adds random files to the directory upon directory creation. That means
that the size of these directories is off from what we expect them to
be. Rather than loosening the assertion which could hide bugs on real
directories, this commit suppresses this file system in this test suite.
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```
This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.
I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.
Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:
```
---
"example test":
- skip:
features: host_selector
- do:
host_selector:
version: " - 7.0.0" # same syntax as skip
apiname:
something: true
```
The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.
The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.
Closes#21888
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
The core REST tests with security currently use a hardcoded username and
password. This is not amenable to running these tests in scenarios where
the user controls the creation of the cluster and owns the credentials
for this cluster. This commit enables running the core REST tests with
security with a custom username and password.
This removes the abstract `getTranslog` method in `Engine`, instead leaving it
to the abstract implementations of the other methods that use the translog. This
allows future Engines not to have a Translog, as instead they must implement the
methods that use the translog pieces to return necessary values.
This test was failing from time to time due to a ConcurrentModificationException, which
was triggered due to the primary-replica resync running concurrently with shards being
removed.
Closes#30767
There's no need for an extra blobExists() call when writing a blob to the Azure service. Azure
provides an option (with stronger consistency guarantees) on the upload method that guarantees
that the blob that's uploaded does not already exist. This saves one network roundtrip.
Relates to #19749
With `max_concurrent_shard_requests` we used to throttle / limit
the number of concurrent shard requests a high level search request
can execute per node. This had several problems since it limited the
number on a global level based on the number of nodes. This change
now throttles the number of concurrent requests per node while still
allowing concurrency across multiple nodes.
Closes#31192
Previously this was called for the combine script only. This change checks for self references for
init, map, and reduce scripts as well, and adds unit test coverage for the init, map, and combine cases.
The java version checker requires being written with java 7 APIs.
In order to use java 8 apis in other launcher utilities, this commit
moves the java version checker back to its own jar.
See the vagrant issue mentioned in this commit for details. This error
has happened a couple times in packaging test CI builds with workers
using virtualbox 5.2.10r122088
This commit moves the default location of the full dependencies report
to be under the reports directory to align it with the location for the
dependenciesInfo task output.
A previous commit tried to add task dependencies for the
:distribution:generateDependenciesReport task so that a user did not
have to run "dependenciesInfo
:distribution:generateDependenciesReport". However this method did not
reliably add all task dependencies due to task ordering issues in
previous versions of Gradle and our build. This commit removes this for
now and a user will continue to have to run "dependenciesInfo
:distribution:generateDependenciesReport".
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
Most of our license file names strip the version off the artifact name
when deducing the license filename. However, the version on the GCS SDK
(google-api-services-storage) does not match the usual format and
instead starts with a vee. This means that the license filename for this
license ended up carrying the version and we should not do that. This
commit adjusts the regex the deduces the license filename to account for
this case, and adjusts the google-api-services-storage license files
accordingly.
* Fully encapsulate LocalCheckpointTracker inside of the engine
This makes the Engine interface not expose the `LocalCheckpointTracker`, instead
exposing the pieces needed (like retrieving the local checkpoint) as individual
methods.
* Remove DocumentFieldMappers#simpleMatchToFullName, as it is duplicative of MapperService#simpleMatchToIndexNames.
* Rename MapperService#simpleMatchToIndexNames -> simpleMatchToFullName for consistency.
* Simplify EsIntegTestCase#assertConcreteMappingsOnAll to accept concrete fields instead of wildcard patterns.
Make SAML Response Destination check compliant
Only validate the Destination element of an incoming SAML Response
if Destination is present and the SAML Response is signed.
The standard [1] - 3.5.5.2 and [2] - 3.2.2 does mention that the
Destination element is optional and should only be verified when
the SAML Response is signed. Some Identity Provider implementations
are known to not set a Destination XML Attribute in their SAML
responses when those are not signed, so this change also aims to
enhance interoperability.
[1] https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf
[2] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
This commit enhances the license detection that we have for various
licenses. Here we improve the detection for all licenses (especially the
Apache 2.0 License), the BSD 2-clause license, the MIT (with
attribution) license, and we add detection for the BSD 3-clause
license. One way that we achieved this improvement is by changing how
the license files are read so that rather than reading them as a
multi-line string which ended up represented as "[line1, line2, line3,
...]" internally, we read the full bytes of the license text and replace
all whitespace with a single space so the license text is now loaded as
"line1 line2 line3". For the MIT license we add the actual license text
and remove the "MIT" string as not all copies of the license clearly
indicate that the text is the MIT license. We take a similar strategy
for the BSD-2 and BSD-3 clause licenses. With this change, we reduce the
number of "custom" licenses in the codebase from 31 to 2. The two
remaining appear to be truly custom licenses, not carrying licenses
identifiable by SPDX. A follow-up will address "unknown" licenses.
The following analyzers were moved from server module to analysis-common module:
`snowball`, `arabic`, `armenian`, `basque`, `bengali`, `brazilian`, `bulgarian`,
`catalan`, `chinese`, `cjk`, `czech`, `danish`, `dutch`, `english`, `finnish`,
`french`, `galician` and `german`.
Relates to #23658
Jason Tedor
Albert Zaharovits
Yuri Tceretian
Van0SS
Dimitris Athanasiou
Ioannis Kakavas
Luca Cavanna
olcbean
Christoph Büscher
Aditya Dhulipala
Martijn van Groningen
debadair
Nik Everett
Nhat Nguyen
Lee Hinman
Tanguy Leroux
Yannick Welsch
Simon Willnauer
rationull
Ryan Ernst
Andy Bristol
Sue Gallagher
Julie Tibshirani
Vladimir Dolzhenko
Igor Motov
David Turner