This commit adds retries for windows cleanup after tests, which may fail
due to file locks not being immediately released after a windows process
exits.
closes#50825
This commit improves the performance of warning value extraction in the
low-level REST client, and is similar to the approach taken in
#24114. There are some differences since the low-level REST client might
be connected to Elasticsearch through a proxy that injects its own
warnings.
This commit changes the default behavior for
xpack.security.ssl.diagnose.trust when running in a FIPS 140 JVM.
More specifically, when xpack.security.fips_mode.enabled is true:
- If xpack.security.ssl.diagnose.trust is not explicitly set, the
default value of it becomes false and a log message is printed
on info level, notifying of the fact that the TLS/SSL diagnostic
messages are not enabled when in a FIPS 140 JVM.
- If xpack.security.ssl.diagnose.trust is explicitly set, the value of
it is honored, even in FIPS mode.
This is relevant only for 7.x where we support Java 8 in which
SunJSSE can still be used as a FIPS 140 provider for TLS. SunJSSE
in FIPS mode, disallows the use of other TrustManager implementations
than the one shipped with SunJSSE.
Follow up to #50692 that starts writing a `min_version` field to
the `RepositoryData` so that pre-7.6 ES versions can not read it
(and potentially corrupt it if they attempt to modify the repo contents)
after the repository moved to the new metadata format.
The system created and models we provide now use the `_xpack` user for uniformity with our other features
The `PUT` action is now an admin cluster action
And XPackClient class now references the action instance.
Hide the `.async-search-*` in Security by making it a restricted index namespace.
The namespace is hard-coded.
To grant privileges on restricted indices, one must explicitly toggle the
`allow_restricted_indices` flag in the indices permission in the role definition.
As is the case with any other index, if a certain user lacks all permissions for an
index, that index is effectively nonexistent for that user.
When deserializing time zones in the Rounding classes we used to include a tiny
normalization step via `DateUtils.of(in.readString())` that was lost in #50609.
Its at least necessary for some tests, e.g. the cause of #50827 is that when
sending the default time zone ZoneOffset.UTC on a stream pre 7.0 we convert it
to a "UTC" string id via `DateUtils.zoneIdToDateTimeZone`. This gets then read
back as a UTC ZoneRegion, which should behave the same but fails the equality
tests in our serialization tests. Reverting to the previous behaviour with an
additional normalization step on 7.x.
Co-authored-by: Nik Everett <nik9000@gmail.com>
Closes#50827
The OpenIdConnectRealm had a bug which would cause it not to populate
User metadata for collections contained in the user JWT claims.
This commit fixes that bug.
Backport of: #50521
* [ML][Inference] PUT API (#50852)
This adds the `PUT` API for creating trained models that support our format.
This includes
* HLRC change for the API
* API creation
* Validations of model format and call
* fixing backport
Today we make multiple attempts to corrupt the translog header in
`TranslogHeaderTests#testCurrentHeaderVersion`, but if we are extraordinarily
unlucky then this sequence of corruptions may restore the file to its original
state. This change adjusts the test to only corrupt the file once, which is
certain not to leave the file in its original state.
The test checked queue size and active count, however,
ThreadPoolExecutor pulls out the request from the queue before marking
the worker active, risking that we think all tasks are done when they
are not. Now check on completed-tasks metric instead, which is
guaranteed to be monotonic.
Relates #50769
Today we periodically check the indexing buffer memory every 5 seconds
or after we have used 1/30 of the configured memory. If the total used
memory is over the threshold, then we refresh the "largest" shards. If
refreshing takes longer these intervals (i.e., 5s or 1/30 buffer), then
we continue to enqueue refreshes to these shards. This leads to two
issues:
- The refresh thread pool can be exhausted and other shards can't refresh
- Execute too many refreshes for the "largest" shards
With this change, we only refresh the largest shards if they are not refreshing.
Here we rely on the periodic check to trigger another refresh if needed. We can
harden this by making the ongoing refresh triggers the memory check when
it's completed. I opted out this option in this PR for simplicity.
See: https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/
* Fix SLM check for restore in progress (#50868)
* Fix SLM check for restore in progress
This commit fixes the check in SLM where the `RestoreInProgress`
metadata was checked for existence. Rather than check existence we
should instead check the `isEmpty` method. Prior to this, a successful
restore for a repository that used SLM retention would prevent SLM
retention from running in subsequent invocations, due to SLM thinking
that a restore was still running.
* Fix 7.x-isms
When a composite aggregation is reduced using the results from an
index that has one of the fields unmapped we were throwing away the
formatter. This is mildly annoying, except in the case of IP addresses
which were coming out as non-utf-8-characters. And tripping assertions.
This carefully preserves the formatter from the working bucket.
Closes#50600
This change fixes the upgrade of index metadata that contain
a custom similarity with options that are not compatible with BM25.
The upgrade doesn't need a real similarity service so we fake one that
resolves all custom similarity to BM25 but this logic fails because the
BM25 provider checks that all options are compatible. This commit removes
the verification step as it is not needed during the upgrade (the verification
is done when the index is restored/opened).
Closes#50763
Since 6.0, the 'template' field has been deprecated in put template requests in
favour of index_patterns. Previously, the PutIndexTemplateRequest would accept
the 'template' field in its 'source' methods and silently convert it to
'index_patterns'. This meant that users specifying 'template' in the source
would not receive a deprecation warning from the server.
This PR makes a small change to no longer silently convert 'template' to
'index_patterns', which ensures that users receive a deprecation warning.
Follow-up to #49460.
The packed-refs support was using the original .git path, changed to use
the real .git directory after reference from worktree has been followed.
Relates #47464
This commit removes validation logic of source and dest indices
for data frame analytics and replaces it with using the common
`SourceDestValidator` class which is already used by transforms.
This way the validations and their messages become consistent
while we reduce code.
This means that where these validations fail the error messages
will be slightly different for data frame analytics.
Backport of #50841
* Fix Snapshot Shard Status Request Deduplication
The request deduplication didn't actually work for these requests
since they had no `equals` and `hashCode` so the deduplicator wouldn't
actually recognize equal requests.
Replaces the "funny"
`Function<String, ConstructingObjectParser<T, Void>>` with a much
simpler `ConstructingObjectParser<T, String>`. This makes pretty much
all of our object parsers static.
* Fix Snapshot Repository Corruption in Downgrade Scenarios (#50692)
This PR introduces test infrastructure for downgrading a cluster while interacting with a given repository.
It fixes the fact that repository metadata in the new format could be written while there's still older snapshots in the repository that require the old-format metadata to be restorable.
A very large number of recursive calls can cause a stack overflow
exception. This commit forks the recursive calls for non-async
processors. Once forked, each thread will handle at most 10
recursive calls to help keep the stack size and thread count
down to a reasonable size.
Adds support for the `offset` parameter to the `date_histogram` source
of composite aggs. The `offset` parameter is supported by the normal
`date_histogram` aggregation and is useful for folks that need to
measure things from, say, 6am one day to 6am the next day.
This is implemented by creating a new `Rounding` that knows how to
handle offsets and delegates to other rounding implementations. That
implementation doesn't fully implement the `Rounding` contract, namely
`nextRoundingValue`. That method isn't used by composite aggs so I can't
be sure that any implementation that I add will be correct. I propose to
leave it throwing `UnsupportedOperationException` until I need it.
Closes#48757
This adds the necessary named XContent classes to the HLRC for the lang ident model. This is so the HLRC can call `GET _ml/inference/lang_ident_model_1?include_definition=true` without XContent parsing errors.
The constructors are package private as since this classes are used exclusively within the pre-packaged model (and require the specific weights, etc. to be of any use).
If a pipeline referenced by a transform does not exist, we should not allow the transform to be created.
We do allow the pipeline existence check to be skipped with defer_validations, but if the pipeline still does not exist on `_start`, the pipeline will fail to start.
relates: #50135
It's impossible to tell why #50754 fails without this change.
We're failing to close the `exchange` somewhere and there is no
write timeout in the GCS SDK (something to look into separately)
only a read timeout on the socket so if we're failing on an assertion without
reading the full request body (at least into the read-buffer) we're locking up
waiting forever on `write0`.
This change ensure the `exchange` is closed in the tests where we could lock up
on a write and logs the failure so we can find out what broke #50754.
Ryan Ernst
Mark Vieira
Nhat Nguyen
Darren Foong
Ioannis Kakavas
junmuz
Armin Braun
Larry Gregory
Benjamin Trent
Albert Zaharovits
Christoph Büscher
Tim Vernum
David Turner
Henning Andersen
Lee Hinman
Nik Everett
Julie Tibshirani
Jim Ferenczi
James Rodewig
Dimitris Athanasiou
debadair
Matt Braymer-Hayes
Lisa Cawley
Jake Landis
Sean Story
István Zoltán Szabó