Date histograms on non-fixed timezones such as `Europe/Paris` proved much slower
than histograms on fixed timezones in #28727. This change mitigates the issue by
using a fixed time zone instead when shard data doesn't cross a transition so
that all timestamps share the same fixed offset. This should be a common case
with daily indices.
NOTE: Rewriting the aggregation doesn't work since the timezone is then also
used on the coordinating node to create empty buckets, which might be out of the
range of data that exists on the shard.
NOTE: In order to be able to get a shard context in the tests, I reused code
from the base query test case by creating a new parent test case for both
queries and aggregations: `AbstractBuilderTestCase`.
Mitigates #28727
This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points. It is the scripted version of MovingAvg pipeline agg.
Through custom script contexts, we expose a number of convenience methods:
- MovingFunctions.max()
- MovingFunctions.min()
- MovingFunctions.sum()
- MovingFunctions.unweightedAvg()
- MovingFunctions.linearWeightedAvg()
- MovingFunctions.ewma()
- MovingFunctions.holt()
- MovingFunctions.holtWinters()
- MovingFunctions.stdDev()
The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
The TemplateUpgradeService is a system service that allows for plugins
to register templates that need to be upgraded. These template upgrades
should always happen in a system context as they are not a user
initiated action. For security integrations, the lack of running this
in a system context could lead to unexpected failures. The changes in
this commit set an empty system context for the execution of the
template upgrades performed by this service.
Relates #30603
When processing a top-level sibling pipeline, we destructively sublist
the path by assigning back onto the same variable. But if aggs are
specified such:
A. Multi-bucket agg in the first entry of our internal list
B. Regular agg as the immediate child of the multi-bucket in A
C. Regular agg with the same name as B at the top level, listed as the
second entry in our internal list
D. Finally, a pipeline agg with the path down to B
We'll get class cast exception. The first agg will sublist the path
from [A,B] to [B], and then when we loop around to check agg C,
the sublisted path [B] matches the name of C and it fails.
The fix is simple: we just need to store the sublist in a new object
so that the old path remains valid for the rest of the aggs in the loop
Closes#30608
* Fixes IndiceOptionsTests to serialise correctly
Previous to this change `IndicesOptionsTests.testSerialisation()` would
select a complete random version for both the `StreamOutput` and the
`StreamInput`. This meant that the output could be selected as 7.0+
while the input was selected as <7.0 causing the stream to be written
in the new format and read in teh old format (or vica versa). This
change splits the two cases into different test methods ensuring that
the Streams are at least on compatibile versions even if they are on
different versions.
* Use same random version for input and output streams
server/src/test/java/org/elasticsearch/action/support/IndicesOptionsTest
s.java
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the
task management API.
Related to #27205
* Refactors ClientHelper to combine header logic
This change removes all the `*ClientHelper` classes which were
repeating logic between plugins and instead adds
`ClientHelper.executeWithHeaders()` and
`ClientHelper.executeWithHeadersAsync()` methods to centralise the
logic for executing requests with stored security headers.
* Removes Watcher headers constant
When the encrpytion of sensitive date is enabled, test that a
scheduled watch is executed as expected and produces the correct value
from a secret in the basic auth header.
Allows the setting to be specified using proper array syntax, for example:
"cluster.routing.allocation.awareness.attributes": [ "foo", "bar", "baz" ]
Closes#30617
Make SSLContext reloadable
This commit replaces all customKeyManagers and TrustManagers
(ReloadableKeyManager,ReloadableTrustManager,
EmptyKeyManager, EmptyTrustManager) with instances of
X509ExtendedKeyManager and X509ExtendedTrustManager.
This change was triggered by the effort to allow Elasticsearch to
run in a FIPS-140 environment. In JVMs running in FIPS approved
mode, only SunJSSE TrustManagers and KeyManagers can be used.
Reloadability is now ensured by a volatile instance of SSLContext
in SSLContectHolder.
SSLConfigurationReloaderTests use the reloadable SSLContext to
initialize HTTP Clients and Servers and use these for testing the
key material and trust relations.
This commit adds Create Repository, the associated docs and tests
for the high level REST API client. A few small changes to the
PutRepository Request and Response went into the commit as well.
A 6.x node can send a deprecation message that the default number of
shards will change from five to one in 7.0.0. In a mixed cluster,
whether or not a create index request sees five or one shard and
produces a deprecation message depends on the version of the master
node. This means that during BWC tests a test can see this deprecation
message depending on the version of the master node. In 6.x when we
introduced this deprecation message we assumed that whereever we see
this deprecation message is expected. However, in a mixed cluster test
we need a similar mechanism but it would only apply if the version of
the master node is earlier than 7.0.0. This commit takes advantage of a
recent change to expose the version of the master node to do sections of
REST tests. With this in hand, we can skip asserting on the deprecation
message if the version of the master node is before 7.0.0 and otherwise
seeing that deprecation message would be completely unexpected.
This commit is related to #28898. It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the the old requests. This
changes many calls in the `qa` projects to use the new version.
This commit exposes the master version to the REST test context. This
will be needed in a follow-up where the master version will be used to
determine whether or not a certain warning header is expected.
This configures all `qa` projects to use the distribution contained in
the `tests.distribution` system property if it is set. The goal is to
create a simple way to run tests against the default distribution which
has x-pack basic features enabled while not forcing these tests on all
contributors. You run these tests by doing something like:
```
./gradlew -p qa -Dtests.distribution=zip check
```
or
```
./gradlew -p qa -Dtests.distribution=zip bwcTest
```
x-pack basic *shouldn't* get in the way of any of these tests but
nothing is ever perfect so this we have to disable a few when running
with the zip distribution.
* Refactor IndicesOptions to not be byte-based
This refactors IndicesOptions to be enum/enummap based rather than using a byte
as a bitmap for each of the options. This is necessary because we'd like to add
additional options, but we ran out of bits.
Backwards compatibility is kept for earlier versions so the option serialization
does not change the options.
Relates sort of to #30188
Due to the way composite aggregation works, ordering in GROUP BY can be
applied only through grouped columns which now the analyzer verifier
enforces.
Fix 29900
This commit removes the SecurityLifecycleService, relegating its former
functions of listening for cluster state updates to SecurityIndexManager
and IndexAuditTrail.
This does away with the deprecated `com.google.api-client:google-api-client:1.23`
and replaces it with `com.google.cloud:google-cloud-storage:1.28.0`.
It also changes security permissions for the repository-gcs plugin.
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.
Closes#30416
The order in which double values are added in java can give different results
for the sum, so we need to allow a certain delta in the test assertions. The
current value was still a bit too low, resulting in rare test failures. This
change increases the allowed margin of error by a factor of ten.
This change adds a grok_pattern field to the GET categories API
output in ML. It's calculated using the regex and examples in the
categorization result, and applying a list of candidate Grok
patterns to the bits in between the tokens that are considered to
define the category.
This can currently be considered a prototype, as the Grok patterns
it produces are not optimal. However, enough people have said it
would be useful for it to be worthwhile exposing it as experimental
functionality for interested parties to try out.
We are starting over on the changelog with a different approach. This
commit removes the existing incarnation of the changelog to remove
confusion that we need to continue adding entries to it.
This is fixing an issue that has come up in some builds. In some
scenarios I see an assertion failure that we are trying to move to
application mode when we are not in handshake mode. What I think is
happening is that we are in handshake mode and have received the
completed handshake message AND an application message. While reading in
handshake mode we switch to application mode. However, there is still
data to be consumed so we attempt to continue to read in handshake mode.
This leads to us attempting to move to application mode again throwing
an assertion.
This commit fixes this by immediatly exiting the handshake mode read
method if we are not longer in handshake mode. Additionally if we swap
modes during a read we attempt to read with the new mode to see if there
is data that needs to be handled.
In #28255 the implementation of the elasticsearch.keystore was changed
to no longer be built on top of a PKCS#12 keystore. A side effect of
that change was that calling getString or getFile on a closed
KeyStoreWrapper ceased to throw an exception, and would instead return
a value consisting of all 0 bytes.
This change restores the previous behaviour as closely as possible.
It is possible to retrieve the _keys_ from a closed keystore, but any
attempt to get or set the entries will throw an IllegalStateException.
Zachary Tong
Ke Li
Adrien Grand
Zachary Tong
Colin Goodheart-Smithe
Jay Modi
Albert Zaharovits
Boaz Leskes
Van0SS
David Kyle
Alexander Reelsen
Yannick Welsch
lukens
Ioannis Kakavas
Vladimir Dolzhenko
Nik Everett
Michael Basnight
Jason Tedor
Ryan Ernst
Tim Brooks
Lee Hinman
Julie Tibshirani
Costin Leau
Costin Leau
lcawl
Nhat Nguyen
Ioannis Kakavas
javanna
Simon Willnauer
David Roberts
srini-raman
Christoph Büscher
Tim Vernum
Tim Vernum