MapperService carries a lot of weight and is only used to determine if loading of field data for the id field is enabled, which can be done in a different way.
Getting the API key document form the security index is the most time consuing part
of the API Key authentication flow (>60% if index is local and >90% if index is remote).
This traffic is now avoided by caching added with this PR.
Additionally, we add a cache invalidator registry so that clearing of different caches will
be managed in a single place (requires follow-up PRs).
Just a few spots where we can dry up these tests using the snapshot test infrastructure
in core that I found while studying the existing searchable snapshot tests.
This fixes fields retrieval on unsigned_long field
1) For docvalue_fields a custom UnsignedLongLeafFieldData::getLeafValueFetcher
is implemented that correctly retrieves doc values.
2) For stored fields, an error was fixed in UnsignedLongFieldMapper
how stored values were stored. Before they were incorrectly
stored in the shifted format, now they are stored as original
values in String format.
Relates to #60050
Backport for #63119
If the connection between clusters is disconnected or the leader cluster
is offline, then CCR shard-follow tasks can stop with "no seed node
left". CCR should retry on this error.
The copy constructors previously used were hard to read and the exact state changes
were not obvious at all.
Refactored those into a number of named constructors instead, added additional assertions
and moved the snapshot abort logic into `SnapshotsInProgress`.
Since match (for matching regex) is not currently in use remove it for
now.
Close#63263
(cherry picked from commit 6abd531cf457f3c5686f59709647bed3276e3c6b)
Make EQL case sensitive by default and adapt some of the string functions
Remove the case sensitive option from Between string function
Add case_insensitive option to term and wildcard queries usage
(cherry picked from commit 7550e0664c8c2f1f13519036c759b1e76345551f)
We only ever use this with `XContentParser` no need to make it inline
worse by forcing the lambda and hence dynamic callsite here.
=> Extraced the exception formatting code path that is likely very cold
to a separate method and removed the lambda usage in hot loops by simplifying
the signature here.
* Just some obvious drying up of these super complex tests.
* Mainly just shortening the diff of #61839 here by moving test utilities
to the abstract test case.
Also, making use of the now available functionality to simplify existing tests
and improve logging in them.
In the latest version of the GCS SDK the `404` exception is wrapped
in an `IOException` making it not pass to the unwrapping added in
the previous fix#63168.
We can't be handling `IOException` differently here now that GCS uses it
for `404`s so I adjusted the exception unwrapping accordingly.
Closes#63091
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.
Brings us one step closer to #57918
this adds the new field `feature_importance_baseline` and allows it to be optionally be included in the model's metadata.
Related to: https://github.com/elastic/ml-cpp/pull/1522
Add a unit test to verify that an identifier surrounded with backquotes
is not a valid syntax for eventType value, as eventType is
schemantically a string literal and not a field identifier.
Follows: #63169
(cherry picked from commit ff12c1340b3890ac52251f31259fa9a719d9eacc)
Ignore by default unavailable indices (same as ES) and verify that
allowNoIndices is set to false since at least one index is required
for validating the query.
Fix#62986
(cherry picked from commit fd75ac27223cd1b699b8d9c311dc401a39f9e0c8)
Do not filter by tiebreaker while searching sequence matches as
it's not monotonic and thus can filter out valid data.
Fix#62781
(cherry picked from commit 4d62198df70f3b70f8b6e7730e888057652c18a8)
Point-In-Time queries cannot be ran on individual indices but on all.
Thus all PIT queries move their index from the request level to a filter
so this condition is fulfilled while keeping the query scoped
accordingly.
Fix#63132
(cherry picked from commit c8eb4f724d5dcc0fcc172c6219ecfbc1dc1fbbae)
For runtime fields, we will want to do all search-time interaction with
a field definition via a MappedFieldType, rather than a FieldMapper, to
avoid interfering with the logic of document parsing. Currently, fetching
values for runtime scripts and for building top hits responses need to
call a method on FieldMapper. This commit moves this method to
MappedFieldType, incidentally simplifying the current call sites and freeing
us up to implement runtime fields as pure MappedFieldType objects.
This commit adds a tier preference when mounting a searchable
snapshot. This sets a preference that a searchable snapshot is mounted
to a node with the cold role if one exists, then the warm role, then the
hot role, assuming that no other allocation rules are in place. This
means that by default, searchable snapshots are mounted to a node with
the cold role.
Note that depending on how we implement frozen functionality of
searchable snapshots (not pre-cached/not fully-cached), we might need to
adjust this to prefer frozen if mounting a not pre-cached/fully-cached
searchable snapshot versus mounting a pre-cached/fully-cached searchable
snapshot. This is a later concern since neither this nor the frozen role
are implemented currently.
We need to complete the search before closing the iterator, which
internally closes the point in time; otherwise, the search will fail
with a missing context error.
Closes#62451
Backport #63182 to 7.x branch.
The `randomEnrichPolicy(...)` helper method stores the policy and creates the source indices.
If a source index already exists, because it was creates for a random policy created earlier then
skipping the source index fails, but that is ignored and the test continues. However if the policy
has a match field that doesn't exist in the previous random policy then the mapping is never updated
and the put policy api fails with the fact that the match field can't be found.
This pr fixes that by execute a put mapping call in the event that the source index already exists.
Closes#63126
* [ML] optimize delete expired snapshots (#63134)
When deleting expired snapshots, we do an individual delete action per snapshot per job.
We should instead gather the expired snapshots and delete them in a single call.
This commit achieves this and a side-effect is there is less audit log spam on nightly cleanup
closes https://github.com/elastic/elasticsearch/issues/62875
Event type is actually a string value for event.category which can
contain any kind of characters, or start with a digit, which currently
is not supported, so we introduce the possibility to be able to use the
usual syntax of " and """ for strings and raw strings.
Make the grammar a bit cleaner by using the identifier only where it's
actually an identifier in terms of query scemantics.
Fixes: #62933
(cherry picked from commit 306e1d76da3db652db57f11f847705b3995609ff)
Adds normalisation of ngram tokens to reduce disk space.
All punctuation becomes / char and for A-Z0-9 chars turn even codepoints to prior odd e.g. aab becomes aaa
Closes#62817
Currently, CAST will use the first keyword subfield of a text field for
an expression in WHERE clause that gets translated to a painless script
which will lead to an exception thrown:
```
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:759)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:308)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:101)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:98)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:98)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"org.elasticsearch.xpack.sql.expression.function.scalar.whitelist.InternalSqlScriptUtils.docValue(InternalSqlScriptUtils.java:79)",
"InternalSqlScriptUtils.cast(InternalSqlScriptUtils.docValue(doc,params.v0),params.v1)",
" ^---- HERE"
],
"script": "InternalSqlScriptUtils.cast(InternalSqlScriptUtils.docValue(doc,params.v0),params.v1)",
"lang": "painless"
}
],
```
Instead of allowing a painless translation using the first underlying
keyword silently, which can be confusing, we detect such usage and throw\
an error early.
Relates to #60178
(cherry picked from commit 7402e8267ba564e52dc672c25b262824b6048b40)
Backport #63170 to 7.x branch.
The _index field is a special field that allows using queries against the name of an index or alias.
Data stream names were not included, this pr fixes that by changing SearchIndexNameMatcher
(which used via IndexFieldMapper) to also include data streams.
We recently upgraded the SDK which resulted in the storage exception to be
wrapped now so we must unwrap to check for whether it's a 404 or not.
Closes#63091
Use triple double quotes enclosing a string literal to interpret it
as unescaped, in order to use `?` for marking query params and avoid
user confusion. `?` also usually implies regex expressions.
Any character inside the `"""` beginning-closing markings is considered
raw and the only thing that is not permitted is the `"""` sequence itself.
If a user wants to use that, needs to resort to the normal `"` string literal
and use proper escaping.
Relates to #61659
(cherry picked from commit d87c2ca2eacab5552bca1e520d33cf71da40bcfd)
* [ML] renames */inference* apis to */trained_models* (#63097)
This commit renames all `inference` CRUD APIs to `trained_models`.
This aligns with internal terminology, documentation, and use-cases.
In the codebase there is the non-written convention that classes that extend `MappedFieldType` are generally called `*FieldType`. With this commit we adopt the same convention for runtime field types which allows us to shorten their names by removing the `Mapped` portion which is implicit.
We would only check for a null value and not for an empty string so
that meant that we were not actually enforcing this mandatory
setting. This commits ensures we check for both and fail
accordingly if necessary, on startup
Introduce : operator for doing case insensitive string comparisons.
Recognizes "*" for wildcard matches in string literals.
Restricted only to string types.
Relates #62941
(cherry picked from commit 201e577e65f26a9b958a6197fe6c7268da39de29)
Since `fork` is not used, is undocumented in Python EQL and
there is no plan at the moment to implement it in the future,
removing it from the grammar. User will get parsing exceptions
instead of higher level messages about unsupported features
which can lead to wrong expectations.
(cherry picked from commit f6a0f8f01c1b1893bab86629d1de73e9f9dae8dc)
Benjamin Trent
Luca Cavanna
Yang Wang
Armin Braun
Mayya Sharipova
David Kyle
Nhat Nguyen
Costin Leau
Andrei Stefan
David Roberts
Rene Groeschke
Marios Trivyzas
Costin Leau
Alan Woodward
Jason Tedor
Martijn van Groningen
markharwood
Przemysław Witek
Joe Gallo
Ioannis Kakavas
Christoph Büscher
Igor Motov
Lee Hinman