Hadoop's security model uses the OS level authentication modules to collect
information about the current user. In JDK 11, the UnixLoginModule makes
use of a new permission to determine if the executing code is allowed to load
the libraries required to pull the user information from the OS. This PR adds
that permission and re-enables the tests that were previously failing when
testing against JDK 11.
This recreates a test that was added to the bats packaging tests
in #31343 but didn't make it over to the java project during when the
linux package tests were ported in #31943
When packages are installed but can not locate the java executable, they
should fail with a descriptive message
Explicitly include all subdirectories of these folders in
/usr/share/elasticsearch in package distributions so that they are
managed by the package manager. This change does really have an
effect in the 7.x series, where there are no subdirectories in bin, and
we were already doing this in lib and modules. It does have an effect in
the 6.x series where the bin/x-pack subdirectory was not previously
tracked by the package manager and could be left behind on removal in
rpm distributions.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:core` project to use the new versions.
This is largely mechanical change that cleans up the addConstructor, addMethod, and
addFields methods in PainlessLookup. Changes include renamed variables, better error
messages, and some minor code movement to make it more maintainable long term.
ClassCastException can be thrown by callers of TransportActions.isShardNotAvailableException(e) as e is not always an instance of ElasticSearchException
fixes#32173
New data is reported from Beats to the monitoring endpoint. This PR adds the template change necessary for it. See https://github.com/elastic/beats/issues/7521 for more details.
Queue data is skipped for now as implementation is not finished yet.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `qa/full-cluster-restart` project to use the new
versions.
It also fixes a small bug in the test for explain on the `_all` field
that was causing it to not properly invoke `_explain`.
* Remove BouncyCastle dependency from runtime
This commit introduces a new gradle project that contains
the classes that have a dependency on BouncyCastle. For
the default distribution, It builds a jar from those and
in puts it in a subdirectory of lib
(/tools/security-cli) along with the BouncyCastle jars.
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools
that use these classes.
BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
* INGEST: Extend KV Processor (#31789)
Added more capabilities supported by LS to the KV processor:
* Stripping of brackets and quotes from values (`include_brackets` in corresponding LS filter)
* Adding key prefixes
* Trimming specified chars from keys and values
Refactored the way the filter is configured to avoid conditionals during execution.
Refactored Tests a little to not have to add more redundant getters for new parameters.
Relates #31786
* Add documentation
* INGEST: Make a few Processors callable by Painless
* Extracted a few stateless String processors as well as the json processor to static methods and whitelisted them in Painless
* provide whitelist from processors plugin
Currently we check that the queries that QueryStringQueryBuilder#toQuery returns
is one out of a list of many Lucene query classes. This list has extended a lot over time,
since QueryStringQueryBuilder can build all sort of queries. This makes the test hard to
maintain. The recent addition of alias fields which build a BlendedTermQuery show how
easy this test breaks. Also the current assertions doesn't add a lot in terms of catching
errors. This is why we decided to remove this check.
Closes#32234
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `qa/rolling-upgrade` project to use the new
versions.
The parent filter for nested sort should always match **all** parents regardless
of the child queries. It is used to find the boundaries of a single parent and we use
the child query to match all the filters set in the nested tree so there is no need to
repeat the nested filters.
With this change we ensure that we build bitset filters
only to find the root docs (or the docs at the level where the sort applies) that can be reused
among queries.
Closes#31554Closes#32130Closes#31783
Co-authored-by: Dominic Bevacqua <bev@treatwell.com>
* Enhance Parent circuit breaker error message
This adds information about either the current real usage (if tracking "real"
memory usage) or the child breaker usages to the exception message when the
parent circuit breaker trips.
The messages now look like:
```
[parent] Data too large, data for [my_request] would be [211288064/201.5mb], which is larger than the limit of [209715200/200mb], usages [request=157286400/150mb, fielddata=54001664/51.5mb, in_flight_requests=0/0b, accounting=0/0b]
```
Or when tracking real memory usage:
```
[parent] Data too large, data for [request] would be [251/251b], which is larger than the limit of [200/200b], real usage: [181/181b], new bytes reserved: [70/70b]
```
* Only call currentMemoryUsage once by returning structured object
The initial decision to use async durability was made a long time ago
for performance reasons. That argument no longer applies and we
prefer the safety of request durability.
With this commit we remove the documentation for the setting
`xpack.monitoring.collection.indices.stats.timeout` which has already
been removed in code.
Closes#32133
Relates #32229
Currently the ranking evaluation response contains a 'unknown_docs' section
for each search use case in the evaluation set. It contains document ids for
results in the search hits that currently don't have a quality rating.
This change renames it to `unrated_docs`, which better reflects its purpose.
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.
Relates to #31516
* Enabled advanced leak detection when loading `EsTestCase`
* Added custom `Appender` to collect leak logs and check for logged errors in a way similar to what is done for the `StatusLogger`
* Fixes#20398
Prior to 6.3 a trial license default to security enabled. Since 6.3
they default to security disabled. If a cluster is upgraded from <6.3
to >6.3, then we detect this and mimic the old behaviour with respect
to security.
This removes some extraneous naming syntax and makes clear the meaning of certain
naming conventions without ambiguities (stricter) within the lookup package. Purely
mechanical change. Note this does not cover a large portion of the
PainlessLookupBuilder and PainlessLookup yet as there are several more follow up PRs for these incoming.
Throw an exception for doc['field'].value
if this document is missing a value for the field.
After deprecation changes have been backported to 6.x,
make this a default behaviour in 7.0
Closes#29286
Now write operations like Index, Delete, Update rely on the write-index associated with
an alias to operate against. This means writes will be accepted even when an alias points to multiple indices, so long as one is the write index. Routing values will be used from the AliasMetaData for the alias in the write-index. All read operations are left untouched.
We do not support intra-cluster connections on multiple interfaces, but the
documentation indicates that we will in future. In fact there is currently no
plan to support this, so the forward-looking documentation is misleading. This
commit
- removes the misleading sentence
- fixes that a transport profile affects outbound connections, not inbound ones
- tidies up some nearby text
Relates #29827
This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.
This commit also changes the license type to trial, so that watcher is available in high level rest client tests.
/cc @hub-cap
In 1.x and 2.x, plugins were published to maven and the plugin
installer downloaded them from there. This was later changed to install
from the download service, and in 5.0 plugin zips were no longer
published to maven. However, the build still currently produces an
unused pom file. This is troublesome in the special case when the main
jar of a plugin needs to be published (and thus needs a pom file of
the same name).
closes#31946
The rollup indexer uses a range query to select the next page
of results based on the last time bucket of the previous round
and the `delay` configured on the rollup job. This query uses
the `epoch_millis` format implicitly but doesn't set the `format`.
This result in errors during the rollup job if the field
definition doesn't allow this format. It can also miss documents
if the format is not accepted but another format in the field
definition is able to parse the query (e.g.: `epoch_second`).
This change ensures that we use `epoch_millis` as the only format
to parse the rollup range query.
* Detect and prevent configuration that triggers a Gradle bug
As we found in #31862, this can lead to a lot of wasted time as it's not
immediatly obvius what's going on.
Givent how many projects we have it's getting increasingly easier to run
into gradle/gradle#847.
Add packaging tests for the linux package distributions to the java test
project and remove them from bats. Most of the tests that lived in
30_deb_package.bats and 40_rpm_package.bats are applicable to both
package types and are combined into a single type of test case. Others
are separated out into separate cases to make their intent more clear
For #26741
James Baiera
Jason Tedor
Andy Bristol
Nik Everett
Jack Conradson
Andrey Ershov
Nhat Nguyen
Christoph Büscher
itsnotv
Nicolas Ruflin
Alpar Torok
Luca Cavanna
debadair
Ioannis Kakavas
Armin Braun
Nick Peihl
Julie Tibshirani
Paul Sanwald
Jim Ferenczi
Lee Hinman
David Kyle
Daniel Mitterdorfer
Alexander Reelsen
Tim Vernum
Tim Vernum
Mayya Sharipova
Tal Levy
Vladimir Dolzhenko
David Turner
Boaz Leskes
Ryan Ernst