This commit adds a new fixture that emulates a S3 service in order to
improve the existing integration tests. This is very similar to what has
been made for Google Cloud Storage in #28788, and such tests would
have helped a lot to catch bugs like #22534.
The AmazonS3Fixture is brittle and only implements the very necessary
stuff for the S3 repository to work, but at least it works and can be
adapted for specific tests needs.
Fixes and edge case where DiscountedCumulativeGain can return NaN as
result of the quality metric calculation. This can happen when the
search result set is empty and normalization is used. We should return 0
in this case. Also adding related unit tests to the other two metrics.
* Pass script level params into scripted metric aggs (#28819)
Now params that are passed at the script level and at the aggregation level
are merged and can both be used in the aggregation scripts. If there are
any conflicts, aggregation level params will win. This may be followed
by another change detecting that case and throwing an exception to
disallow such conflicts.
* Disallow duplicate parameter names between scripted agg and script (#28819)
If a scripted metric aggregation has aggregation params and script params
which have the same name, throw an IllegalArgumentException when merging
the parameter lists.
This commit fixes an issue in the security nio transport tests where
renegotiation was not actually being tested. We were not waiting on the
handshake complete latches. This commit fixes this issue.
Original commit: elastic/x-pack-elasticsearch@47bebc5c13
I am not sure why we have this leniency for HTTP max content length, it
has been there since the beginning
(5ac51ee93f) with no explanation of its
source. That said, our philosophy today is different than the philosophy
of the past where Elasticsearch would be quite lenient in its handling
of settings and today we aim for predictability for both users and
us. This commit removes leniency in the parsing of
http.max_content_length.
* Begin moving XContent to a separate lib/artifact
This commit moves a large portion of the XContent code from the `server` project
to the `libs/xcontent` project. For the pieces that have been moved, some
helpers have been duplicated to allow them to be decoupled from ES helper
classes. In addition, `Booleans` and `CheckedFunction` have been moved to the
`elasticsearch-core` project.
This decoupling is a move so that we can eventually make things like the
high-level REST client not rely on the entire ES jar, only the parts it needs.
There are some pieces that are still not decoupled, in particular some of the
XContent tests still remain in the server project, this is because they test a
large portion of the pluggable xcontent pieces through
`XContentElasticsearchException`. They may be decoupled in future work.
Additionally, there may be more piecese that we want to move to the xcontent lib
in the future that are not part of this PR, this is a starting point.
Relates to #28504
* Add test matrix axis files for periodic java testing
* Add properties file defining java versions to use
Original commit: elastic/x-pack-elasticsearch@d679b9ab3e
* Add test matrix axis files for periodic java testing
* Add properties file defining java versions to use
* We have no openjdk8
* Remove openjdk
Oracle Java and OpenJDK basically only differ in license, so we don't
need to test both.
Fix a couple of minor things in the InternalEngine:
* Rename loadOrGenerateHistoryUUID to reflect that it always generates a UUID
* Move .acquire() call next to the associated try {} block.
Today this part of the documentation just says that Geo queries are not 100%
accurate, but in fact we can be more precise about which kinds of queries see
which kinds of error. This commit clarifies this point.
At time of writing, GeoJSON did not enforce a specific ordering of vertices in
a polygon, but it now does. We occasionally get reports of Elasticsearch
rejecting apparently-valid GeoJSON because of badly oriented polygons, and it's
helpful to be able to point at this bit of the documentation when responding.
Serialization assertions in ElasticsearchAssertions, a transport
interceptor that used them, and a plugin that added that interceptor
were removed from the test framework. This test case no longer needs to
exclude them from its plugins.
Original commit: elastic/x-pack-elasticsearch@07e5c58983
Removes a set of assertions in the test framework that verified that
Streamable objects could be serialized and deserialized across different
versions. When this was discussed the consensus was that this approach
has not caught many bugs in a long time and that serialization testing of
objects was best left to their respective unit and integration tests.
This commit also removes a transport interceptor that was used in
ESIntegTestCase tests to make these assertions about objects coming in
or off the wire.
If a job is deleted and then GetJobs API is immediately called,
it is possible for a job to be returned in the response. This is likely
due to the GetJobs API being executed on a node with a slightly
stale cluster state which shows the job as still existing.
So we delegate to the master node so the list of jobs/tasks is current.
After routing to the master, we need to check if the rollup job
is in the PersistentTask's CS. A job can be acknowledged canceled,
removed from the CS, but the allocated task is still alive. So we
first check the CS to make sure it's really there before going to the
allocated task to get the status.
As extra precaution, when running local to the task, we also make
sure the task isn't canceled before including it in the response.
relates elastic/x-pack-elasticsearch#4041
Original commit: elastic/x-pack-elasticsearch@3b6fb65e12
The parsing code for script query currently silently skips by any tokens
it does not know about within its parsing loop. The only token it does
not catch is an array, which means pasing multiple scripts in via an
array will cause the last script to be parsed and one, silently dropping
the others. This commit adds validation that arrays are not seen while
parsing.
We use a latch when sending requests during tests so that we do not hang
forever waiting for replies on those requests. This commit increases the
timeout on that latch to 30 seconds because sometimes 10 seconds is just
not enough.
This commit changes the sysprop for overriding the branch bwc builds use
to be branch specific. There are 3 different bwc branches built, but all
of them currently read the exact same sysprop. For example, with this change
and current branches, you can now specify eg `-Dtests.bwc.refspec.6.x=my_6x`
and it will build only next-minor-snapshot with that branch, while
next-bugfix-snapshot will continue to use 5.6.
Since #29260, unsafe commits must be trimmed before opening an engine.
This makes the engine constructor follow Lucene standard semantics and
use the last commit. However, we haven't fully applied this change in some
tests.
Relates #29260
As follow up to #28245 , this PR removes the logic for selecting the
right start commit from the Engine constructor in favor of explicitly
trimming them in the Store, before the engine is opened. This makes the
constructor in engine follow standard Lucene semantics and use the last
commit.
Relates #28245
Relates #29156
All ML objects stored in internal indices are currently parsed
strictly. This means unknown fields lead to parsing failures.
In turn, this means we cannot add new fields in any of those
objects (e.g. bucket, record, calendar, etc.) as it is not
backwards compatible.
This commit changes this by introducing lenient parsing when
it comes to reading those objects from the internal indices.
Note we still use strict parsing for the objects we read from
the c++ process, which is nice as it guarantees we would detect
if any of the fields were renamed on one side but not the other.
Also note that even though this is going in from 6.3, we cannot
introduce new fields until 7.0.
relates elastic/x-pack-elasticsearch#4232
Original commit: elastic/x-pack-elasticsearch@3f95d3c7b9
This commit sets the order of the audit log template to 1000 instead of
using the max value. This will allow a user to define a template that
adds an alias.
Original commit: elastic/x-pack-elasticsearch@2267322755
Due to special treatment for the 0xFFFFFF... value in GeoHashUtils'
encodeLatLon method, the hashcode for lat 90, lon 180 is incorrectly
encoded as `"000000000000"` instead of "zzzzzzzzzzzz". This commit
removes the special treatment and fixes the issue.
Closes#22163
When deleting a snapshot, it is not necessary to load and to parse the
global metadata of the snapshot to delete. Now indices are stored in
the snapshot metadata file, we have all the information to resolve the
shards files to delete.
This commit removes the readSnapshotMetaData() method that was used to
load both global and index metadata files. Test coverage should be
enough as SharedClusterSnapshotRestoreIT already contains several
deletion tests.
Related to #28934
Adds a SecureSetting option for the "bind_password" in LDAP/AD realms
and deprecates the non-secure version.
LDAP bind passwords should now be configured with the setting
`xpack.security.authc.realms.REALM_NAME.secure_bind_password`
in the elasticsearch keystore.
Original commit: elastic/x-pack-elasticsearch@1a0cebd77e
The sysprop repos.mavenLocal may be used to add the local .m2 maven
repository for testing snapshots of locally build dependencies.
Unfortunately this has to be checked in two different places (they cannot
be shared, due to buildSrc being built essentially as a separate
project), and the casing of the string sysprop lookups did not align.
This commit fixes BuildPlugin's checking of repos.mavenLocal to use the
correct casing (camelCase, to match the gradle dsl element).
- Changes in CertUtils to add algorithm parameter to
generateSignedCertificates
- Changes in Tests to randomly pick signature algorithms
- Changes in Tests to randomly pick encryption algorithms
relates elastic/x-pack-elasticsearch#3983
Original commit: elastic/x-pack-elasticsearch@d1b5f3a166
If a user has roles that grant access to a large number of disparate
index patterns, then the resulting Automaton can become large and
too costly to determinise. This happens rarely, and is usually a sign
of a poorly implemented security model, so we have no immediate plans
to change the implementation. However the resulting error message is
not clear and does not provide sufficient information for users to
resolve the underlying problem.
This commit catches the underlying exception and provides a more
specific error message, with DEBUG logging of the offending index
patterns.
Original commit: elastic/x-pack-elasticsearch@532be70efc
Today we have a few problems with how we handle bad requests:
- handling requests with bad encoding
- handling requests with invalid value for filter_path/pretty/human
- handling requests with a garbage Content-Type header
There are two problems:
- in every case, we give an empty response to the client
- in most cases, we leak the byte buffer backing the request!
These problems are caused by a broader problem: poor handling preparing
the request for handling, or the channel to write to when the response
is ready. This commit addresses these issues by taking a unified
approach to all of them that ensures that:
- we respond to the client with the exception that blew us up
- we do not leak the byte buffer backing the request
Tanguy Leroux
Christoph Büscher
rationull
Adrien Grand
Jack Conradson
Tim Brooks
Jason Tedor
Lee Hinman
David Leatherman
Alexander Reelsen
Aaron Bull Schaefer
Igor Motov
David Turner
Mayya Sharipova
Andy Bristol
Zachary Tong
David Kyle
Fabien Baligand
Zachary Tong
javanna
olcbean
Ryan Ernst
Sue Gallagher
Nhat Nguyen
Boaz Leskes
Dimitris Athanasiou
Jay Modi
Tim Vernum
Uwe Schindler
Yogesh Gaikwad
Bolarinwa Saheed Olayemi