This commit updates the rest API specs for
- security.get_role_mapping to specify name as a list type.
- security.get_role to specify name as a list type.
Co-authored-by: Russ Cam <russ.cam@elastic.co>
* Async search should retry updates on version conflict
The _async_search APIs can throw version conflict exception when the internal response
is updated concurrently. That can happen if the final response is written while the user
extends the expiration time. That scenario should be rare but it happened in Kibana for
several users so this change ensures that updates are retried at least 5 times. That
should resolve the transient errors for Kibana. This change also preserves the version
conflict exception in case the retry didn't work instead of returning a confusing 404.
This commit also ensures that we don't delete the response if the search was cancelled
internally and not deleted explicitly by the user.
Closes#63213
The `remote_monitoring_agent` reserved role is extended to grant more privileges
over the metricbeat-* index pattern.
In addition to the index and create_index index privileges that it granted already,
it now also grants the view_index_metadata privilege.
Closes#63203
This commit ensures that jobs within the SchedulerEngine do not
continue to run after they are cancelled. There was no synchronization
between the cancel method of an ActiveSchedule and the run method, so
an actively running schedule would go ahead and reschedule itself even
if the cancel method had been called.
This commit adds synchronization between cancelling and the scheduling
of the next run to ensure that the job is cancelled. In real life
scenarios this could manifest as a job running multiple times for
SLM. This could happen if a job had been triggered and was cancelled
prior to completing its run such as if the node was no longer the
master node or if SLM was stopping/stopped.
Closes#63754
Backport of #63762
This commit updates the APIs in the logstash plugin to handle
IndexNotFoundExceptions that are returned by client calls. Until we
have the creation of this index in place, we need to handle this case
and not let the exception propagate out of the API.
Backport of #63698
* SQL: integer parameter validation in string functions (#58923)
In insert, locate, substring function, when argument `start` or `length` is greater than Integer.MAX_INT OR less then Integer.MIN_INT + 1 (note that `start` need to minus 1), it causes overflow and leads to unexpected results.
* Add range checks for BinaryStringNumericProcessors
- Add range checks for Left, Right, Repeat.
- Minor refactorings on initial PR changes.
Co-authored-by: yinanwu <yinanwu@tencent.com>
(cherry picked from commit bf6dc58b93529f977d035a846d083b1c31867694)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Add a new gradle module under eql/qa which runs and validates a set of
queries over a 4m event dataset (restored from a snapshot residing in a
gcs bucket). The results are providing by running the exact set of queries
with Python EQL against the same dataset.
Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
(cherry picked from commit 1cf789e5fcfb0f364f665bfaac021e24a4c2f556)
Co-authored-by: Mark Vieira <portugee@gmail.com>
This commit fixes two issues in dealing with bool fields in EQL:
- avoid simplifications of field == true expressions
- adding comparison to clauses on fields missing logic (where bool)
Fix#63693
(cherry picked from commit d10a5d0e842bbd4e0031834de948ceb24da3872b)
(cherry picked from commit 0227da3a275c7f22ff524d99d53e1a79146f9e28)
* Allow all indices options variants
Irrespective of allow_no_indices value, throw VerificationException when
there is no index validated
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Mentions the list of wildchars in case a wildchar is used as an
`ESCAPE` character.
Relates #63428
(cherry picked from commit 74cbcf871e9593b3640e382ae6845168fd14966b)
For a query like `SELECT name FROM test WHERE name LIKE ''%c*'` ES SQL
generates an error. `*` is not a special character in a `LIKE` construct
and it's expected to not needing to be escaped, so the previous query
should work as is.
In the LIKE pattern any `*` character was treated as invalid character
and the usage of `%` or `_` was suggested instead. But `*` is a valid,
acceptable non-wildcard on the right side of the `LIKE` operator.
Fix: #55108
(cherry picked from commit 190d9fe3deb31aed0d8f312007360625d4fff217)
Now that deprecation logs get indexed to a data stream, if we
do not load the data stream plugin in our tests and any test
generates a deprecation log message then millions of exceptions
get logged, slowing down the tests to the extent that they can
fail.
This change loads the data streams plugin during the ML internal
cluster tests. (It should already be present in external cluster
tests.)
Fixes#63548
Adds validation that the dest pipeline exists when a transform
is updated. Refactors the pipeline check into the `SourceDestValidator`.
Fixes#59587
Backport of #63494
This PR adds factory methods for the most common implementations:
* `SourceValueFetcher.identity` to pass through the source value untouched.
* `SourceValueFetcher.toString` to simply convert the source value to a string.
* Remove FlatObjectFieldTypeTests, as it's redundant.
* Do not apply null_value when fetching root-level values.
* Remove a TODO in favor of opening an issue.
Do not filter by tiebreaker while searching sequence matches as
it's not monotonic and thus can filter out valid data.
Add handling for data 'near' the boundary that has the same timestamp
but different tie-breaker and thus can be just outside the window.
Fix#62781
Relates #63215
(cherry picked from commit 36f834600d4d9ded0fb7b1440274b2e597733770)
(cherry picked from commit 72a2ce825f3bfd13f87423ba7f3c739ea64c57f6)
As a result of this, we can remove a chunk of code from TypeParsers as well. Tests
for search/index mode analyzers have moved into their own file. This commit also
rationalises the serialization checks for parameters into a single SerializerCheck
interface that takes the values includeDefaults, isConfigured and the value
itself.
Relates to #62988
Strings in the watcher context may use the `.sha1()` and `.sha256()`
augmentation added for ingest.
Ref: #59633, #59671Fixes: #61244
Backport of: 380ee6f
This PR updates the `logstash_admin` role to include the recently-added Logstash Pipeline Management APIs, as well as access to the `.logstash*` index pattern.
Co-authored-by: William Brafford <williamrandolphbrafford@gmail.com>
When constructing a value fetcher, the 'parsesArrayValue' flag must match
`FieldMapper#parsesArrayValue`. However there is nothing in code or tests to
help enforce this.
This PR reworks the value fetcher constructors so that `parsesArrayValue` is
'false' by default. Just as for `FieldMapper#parsesArrayValue`, field types must
explicitly set it to true and ensure the behavior is covered by tests.
Follow-up to #62974.
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
Determines the shard size of shards before allocating shards that are
recovering from snapshots. It ensures during shard allocation that the
target node that is selected as recovery target will have enough free
disk space for the recovery event. This applies to regular restores,
CCR bootstrap from remote, as well as mounting searchable snapshots.
The InternalSnapshotInfoService is responsible for fetching snapshot
shard sizes from repositories. It provides a getShardSize() method
to other components of the system that can be used to retrieve the
latest known shard size. If the latest snapshot shard size retrieval
failed, the getShardSize() returns
ShardRouting.UNAVAILABLE_EXPECTED_SHARD_SIZE. While
we'd like a better way to handle such failures, returning this value
allows to keep the existing behavior for now.
Note that this PR does not address an issues (we already have today)
where a replica is being allocated without knowing how much disk
space is being used by the primary.
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Add a new ids field to the API of invalidating API keys so that it supports bulk
invalidation with a list of IDs.
Note the existing id field is kept as is and it is an error if both id and ids are specified.
MapperService carries a lot of weight and is only used to determine if loading of field data for the id field is enabled, which can be done in a different way.
Getting the API key document form the security index is the most time consuing part
of the API Key authentication flow (>60% if index is local and >90% if index is remote).
This traffic is now avoided by caching added with this PR.
Additionally, we add a cache invalidator registry so that clearing of different caches will
be managed in a single place (requires follow-up PRs).
Just a few spots where we can dry up these tests using the snapshot test infrastructure
in core that I found while studying the existing searchable snapshot tests.
This fixes fields retrieval on unsigned_long field
1) For docvalue_fields a custom UnsignedLongLeafFieldData::getLeafValueFetcher
is implemented that correctly retrieves doc values.
2) For stored fields, an error was fixed in UnsignedLongFieldMapper
how stored values were stored. Before they were incorrectly
stored in the shifted format, now they are stored as original
values in String format.
Relates to #60050
Backport for #63119
If the connection between clusters is disconnected or the leader cluster
is offline, then CCR shard-follow tasks can stop with "no seed node
left". CCR should retry on this error.
The copy constructors previously used were hard to read and the exact state changes
were not obvious at all.
Refactored those into a number of named constructors instead, added additional assertions
and moved the snapshot abort logic into `SnapshotsInProgress`.
Since match (for matching regex) is not currently in use remove it for
now.
Close#63263
(cherry picked from commit 6abd531cf457f3c5686f59709647bed3276e3c6b)
Make EQL case sensitive by default and adapt some of the string functions
Remove the case sensitive option from Between string function
Add case_insensitive option to term and wildcard queries usage
(cherry picked from commit 7550e0664c8c2f1f13519036c759b1e76345551f)
We only ever use this with `XContentParser` no need to make it inline
worse by forcing the lambda and hence dynamic callsite here.
=> Extraced the exception formatting code path that is likely very cold
to a separate method and removed the lambda usage in hot loops by simplifying
the signature here.
* Just some obvious drying up of these super complex tests.
* Mainly just shortening the diff of #61839 here by moving test utilities
to the abstract test case.
Also, making use of the now available functionality to simplify existing tests
and improve logging in them.
this adds the new field `feature_importance_baseline` and allows it to be optionally be included in the model's metadata.
Related to: https://github.com/elastic/ml-cpp/pull/1522
Add a unit test to verify that an identifier surrounded with backquotes
is not a valid syntax for eventType value, as eventType is
schemantically a string literal and not a field identifier.
Follows: #63169
(cherry picked from commit ff12c1340b3890ac52251f31259fa9a719d9eacc)
Ignore by default unavailable indices (same as ES) and verify that
allowNoIndices is set to false since at least one index is required
for validating the query.
Fix#62986
(cherry picked from commit fd75ac27223cd1b699b8d9c311dc401a39f9e0c8)
Do not filter by tiebreaker while searching sequence matches as
it's not monotonic and thus can filter out valid data.
Fix#62781
(cherry picked from commit 4d62198df70f3b70f8b6e7730e888057652c18a8)
Point-In-Time queries cannot be ran on individual indices but on all.
Thus all PIT queries move their index from the request level to a filter
so this condition is fulfilled while keeping the query scoped
accordingly.
Fix#63132
(cherry picked from commit c8eb4f724d5dcc0fcc172c6219ecfbc1dc1fbbae)
For runtime fields, we will want to do all search-time interaction with
a field definition via a MappedFieldType, rather than a FieldMapper, to
avoid interfering with the logic of document parsing. Currently, fetching
values for runtime scripts and for building top hits responses need to
call a method on FieldMapper. This commit moves this method to
MappedFieldType, incidentally simplifying the current call sites and freeing
us up to implement runtime fields as pure MappedFieldType objects.
This commit adds a tier preference when mounting a searchable
snapshot. This sets a preference that a searchable snapshot is mounted
to a node with the cold role if one exists, then the warm role, then the
hot role, assuming that no other allocation rules are in place. This
means that by default, searchable snapshots are mounted to a node with
the cold role.
Note that depending on how we implement frozen functionality of
searchable snapshots (not pre-cached/not fully-cached), we might need to
adjust this to prefer frozen if mounting a not pre-cached/fully-cached
searchable snapshot versus mounting a pre-cached/fully-cached searchable
snapshot. This is a later concern since neither this nor the frozen role
are implemented currently.
We need to complete the search before closing the iterator, which
internally closes the point in time; otherwise, the search will fail
with a missing context error.
Closes#62451
Backport #63182 to 7.x branch.
The `randomEnrichPolicy(...)` helper method stores the policy and creates the source indices.
If a source index already exists, because it was creates for a random policy created earlier then
skipping the source index fails, but that is ignored and the test continues. However if the policy
has a match field that doesn't exist in the previous random policy then the mapping is never updated
and the put policy api fails with the fact that the match field can't be found.
This pr fixes that by execute a put mapping call in the event that the source index already exists.
Closes#63126
* [ML] optimize delete expired snapshots (#63134)
When deleting expired snapshots, we do an individual delete action per snapshot per job.
We should instead gather the expired snapshots and delete them in a single call.
This commit achieves this and a side-effect is there is less audit log spam on nightly cleanup
closes https://github.com/elastic/elasticsearch/issues/62875
Event type is actually a string value for event.category which can
contain any kind of characters, or start with a digit, which currently
is not supported, so we introduce the possibility to be able to use the
usual syntax of " and """ for strings and raw strings.
Make the grammar a bit cleaner by using the identifier only where it's
actually an identifier in terms of query scemantics.
Fixes: #62933
(cherry picked from commit 306e1d76da3db652db57f11f847705b3995609ff)
Adds normalisation of ngram tokens to reduce disk space.
All punctuation becomes / char and for A-Z0-9 chars turn even codepoints to prior odd e.g. aab becomes aaa
Closes#62817
Currently, CAST will use the first keyword subfield of a text field for
an expression in WHERE clause that gets translated to a painless script
which will lead to an exception thrown:
```
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:759)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:308)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:101)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:98)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:98)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"org.elasticsearch.xpack.sql.expression.function.scalar.whitelist.InternalSqlScriptUtils.docValue(InternalSqlScriptUtils.java:79)",
"InternalSqlScriptUtils.cast(InternalSqlScriptUtils.docValue(doc,params.v0),params.v1)",
" ^---- HERE"
],
"script": "InternalSqlScriptUtils.cast(InternalSqlScriptUtils.docValue(doc,params.v0),params.v1)",
"lang": "painless"
}
],
```
Instead of allowing a painless translation using the first underlying
keyword silently, which can be confusing, we detect such usage and throw\
an error early.
Relates to #60178
(cherry picked from commit 7402e8267ba564e52dc672c25b262824b6048b40)
Backport #63170 to 7.x branch.
The _index field is a special field that allows using queries against the name of an index or alias.
Data stream names were not included, this pr fixes that by changing SearchIndexNameMatcher
(which used via IndexFieldMapper) to also include data streams.
Use triple double quotes enclosing a string literal to interpret it
as unescaped, in order to use `?` for marking query params and avoid
user confusion. `?` also usually implies regex expressions.
Any character inside the `"""` beginning-closing markings is considered
raw and the only thing that is not permitted is the `"""` sequence itself.
If a user wants to use that, needs to resort to the normal `"` string literal
and use proper escaping.
Relates to #61659
(cherry picked from commit d87c2ca2eacab5552bca1e520d33cf71da40bcfd)
* [ML] renames */inference* apis to */trained_models* (#63097)
This commit renames all `inference` CRUD APIs to `trained_models`.
This aligns with internal terminology, documentation, and use-cases.
In the codebase there is the non-written convention that classes that extend `MappedFieldType` are generally called `*FieldType`. With this commit we adopt the same convention for runtime field types which allows us to shorten their names by removing the `Mapped` portion which is implicit.
We would only check for a null value and not for an empty string so
that meant that we were not actually enforcing this mandatory
setting. This commits ensures we check for both and fail
accordingly if necessary, on startup
Introduce : operator for doing case insensitive string comparisons.
Recognizes "*" for wildcard matches in string literals.
Restricted only to string types.
Relates #62941
(cherry picked from commit 201e577e65f26a9b958a6197fe6c7268da39de29)
Since `fork` is not used, is undocumented in Python EQL and
there is no plan at the moment to implement it in the future,
removing it from the grammar. User will get parsing exceptions
instead of higher level messages about unsupported features
which can lead to wrong expectations.
(cherry picked from commit f6a0f8f01c1b1893bab86629d1de73e9f9dae8dc)
This test accesses system indices for 2 reasons.
First, it creates a filter that has a different type. This was done
to assert that filter is not returned from the APIs. However,
now that access to the `.ml-meta` index is restricted,
it is not really a concern.
Second, it creates a `.ml-meta` index without mappings to test
the get API does not fail due to lack of mappings on a sorted field,
namely the `filter_id`. Once again, this test is less useful once
system indices have restricted access.
Relates #62501
Backport of #63111
This fixes a bug introduced when moving from mget to ids query. While
mget returns all the ids given, id query is a search query and thus by
default returns only 10 documents.
The fix correctly sets the expected size so all the information is
returned inside the response.
Fix#63030
(cherry picked from commit 09ba85548a0142a1fe8376efea9cc4e7764a207c)
It extracts the query capabilities from AbstractGeometryFieldType into two new interfaces, GeoshapeQueryable and ShapeQueryable. Those interfaces are implemented by the final mappers.
Bwc tests can consume much time to build and to run so it's nice to be
able to skip them when running the `check` task on the SQL module.
Introduce a new task `checkNoBwc` so one can use:
```
./gradlew -p x-pack/plugin/sql checkNoBwc
```
to skip them.
(cherry picked from commit a52e1846f338f6869273181c6f248579581fa68c)
Previously, backquote couldn't not be used inside an escaped identifier,
e.g.:
```
`my`identifier` = "some_value"
```
was not allowed. Introduce escaping of the backquote by using a
double backquote:
```
`my``identifier` = "some_value"
```
(cherry picked from commit 49514121486f42a58674b3e5901de4021fda5c15)
When an opened anomaly detection job is updated with a detection
rule that references a filter, apart from updating the c++ process
with the rule, we also need to update it with the referenced filter.
This commit fixes a bug which led to the job not applying such updates
on-the-fly.
Fixes#62948
Backport of #63057
Remove Count class and related artifacts since that functionality is not
(yet) available.
Update parser name for better error reporting.
Fix#62131
(cherry picked from commit 060f500346788c4c5d0b3b9c045facec5d677d3d)
Test testSortDifferentFormatsShouldFail was occasionally failing for
2 reasons:
1) Documents on "idx2" were not available for search before a
search request started
2) Running a test multiple times was causing
occasional ResourceAlreadyExistsException for idx2,
as idx2 was not deleted for a test.
This patch makes the following fixes:
1) Sets up immediate refresh policy for docs in the index"idx2"
2) Creates an index idx2 only once per cluster
Closes: #62997
This is a very minor optimization but trivial to implement, so might as well.
```
Benchmark (nGramStrs) Mode Cnt Score Error Units
NGramProcessorBenchmark.ngramInnerLoop 1,2,3 avgt 20 4415092.443 ± 31302.115 ns/op
NGramProcessorBenchmark.ngramOuterLoop 1,2,3 avgt 20 4235550.340 ± 103393.465 ns/op
```
This measurement is in nanoseconds, consequently, the overall performance of inference is dominated by other factors (i.e. map#put). But, this optimization adds up overtime and is simple.
Data frame analytics results format changed in version `7.10.0`.
If existing jobs that were not completed are restarted, it is
possible the destination index had already been created. That index's
mappings are not suitable for the new results format.
This commit checks the version of the destination index and deletes
it when the version is outdated. The job will then continue by
recreating the destination index and reindexing.
Backport of #62960
This PR adds timeouts to the named pipe connections of the
autodetect, normalize and data_frame_analyzer processes.
This argument requires the changes of elastic/ml-cpp#1514 in
order to work, so that PR will be merged before this one.
(The controller process already had a different mechanism,
tied to the ES JVM lifetime.)
Backport of #62993
Fix query creation inside sequences with any queries due to lacking a
clause to combine, which lead to an invalid request being created.
Fix#62967
(cherry picked from commit ff59d8823919a6e70928816e5c3687308ebde33f)
Classification feature importance supports various types in the class name:
- string
- boolean
- numerical
The xcontent parsing on the server side and the HLRC side should support and test these types.
* Use compilation as validation for painless role template (#62845)
Role template validation now performs only compilation if the script is painless.
It no longer attempts to execute the script with empty input which is problematic.
The compliation process will catch things like invalid syntax, undefined variables,
which still provide certain level of protection against ill-defined role templates.
Behaviour for Mustache script is unchanged.
* Checkstyle
* [ML] fixing testTwoJobsWithSameRandomizeSeedUseSameTrainingSet tests (#62976)
This fixes the two test failures.
The shard failure seems to be due to the .ml-stats index being in the middle of being created.
50_script_values/Script query fails sometimes
as resulting hits will be ordered differently from expected.
This patch ensures consisten ordering of hits.
Closes#62975
This refactors the loading of monitoring templates slightly so that they aren't loaded over and
over again (from disk) on CS updates. This isn't an important optimization in production for obvious
reasons since it only affects the install stage, but this turned out to cause some slow CS applies
in tests.
Relates #62853
If the transform grouping is a script then exclude the field from the source index
mappings fields caps request. A null object caused an NPE in the serialisation of
FieldCapabilitiesIndexRequest.
As we have decided top level importance for classification is not useful,
it has been removed from the results from the training job. This commit
also removes them from inference.
Backport of #62486
Passing FieldMappers to point parsing functions makes trying to build source-only
fields from MappedFieldTypes more complicated. This small refactoring changes
things so that the relevant parsing and factory functions from
AbstractGeometryFieldMapper are instead passed as lambdas to the PointParser
constructor.
Currently we duplicate our specialized cors logic in all transport
plugins. This is unnecessary as it could be implemented in a single
place. This commit moves the logic to server. Additionally it fixes a
but where we are incorrectly closing http channels on early Cors
responses.
Introduce 64-bit unsigned long field type
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
to double and can be imprecise for large values.
Backport for #60050Closes#32434
This commit adds a mechanism to MapperTestCase that allows implementing
test classes to check that their parameters can be updated, or throw conflict
errors as advertised. Child classes override the registerParameters method
and tell the passed-in UpdateChecker class about their parameters. Simple
conflicts can be checked, using the existing minimal mappings as a base to
compare against, or alternatively a particular initial mapping can be provided
to check edge cases (eg, norms can be updated from true to false, but not
vice versa). Updates are registered with a predicate that checks that the update
has in fact been applied to the resulting FieldMapper.
Fixes#61631
Same as in the normal Netty tests we have to disable the runtime proc
setting in the normal tests task just like we do for the internal cluster tests.
Closes#61919Closes#62298
This commit allows coordinating node to account the memory used to perform partial and final reduce of
aggregations in the request circuit breaker. The search coordinator adds the memory that it used to save
and reduce the results of shard aggregations in the request circuit breaker. Before any partial or final
reduce, the memory needed to reduce the aggregations is estimated and a CircuitBreakingException} is thrown
if exceeds the maximum memory allowed in this breaker.
This size is estimated as roughly 1.5 times the size of the serialized aggregations that need to be reduced.
This estimation can be completely off for some aggregations but it is corrected with the real size after
the reduce completes.
If the reduce is successful, we update the circuit breaker to remove the size of the source aggregations
and replace the estimation with the serialized size of the newly reduced result.
As a follow up we could trigger partial reduces based on the memory accounted in the circuit breaker instead
of relying on a static number of shard responses. A simpler follow up that could be done in the mean time is
to [reduce the default batch reduce size](https://github.com/elastic/elasticsearch/issues/51857) of blocking
search request to a more sane number.
Closes#37182
for get trained models include_model_definition is now deprecated.
This commit writes a deprecation warning if that parameter is used and suggests the caller to utilize the replacement
Backport #62825 to 7.x branch.
Today if a data stream is auto created, but an index with same name as the
first backing index already exists then internally that error is ignored,
which then result that later in the execution of a bulk request, the
bulk item fails due to that the data stream hasn't been auto created.
This situation can only occur if an index with same is created that
will be the backing index of a data stream prior to the creation
of the data stream.
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
The `migrate` action will now configure the
`index.routing.allocation.include._tier_preference` setting to the corresponding
tiers. For the HOT phase it will configure `data_hot`, for the WARM phase it will
configure `data_warm,data_hot` and for the COLD phase
`data_cold,data_warm,data_cold`.
(cherry picked from commit 9dbf0e6f0c267e40c5bcfb568bb2254da103ae40)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Replace common Like and RLike queries that match all characters with
IsNotNull (exists) queries
Fix#62585
(cherry picked from commit 4c23fad0468a9edd7325b06c6a96f7af37625dbf)
In case of more than 500 transforms, get and stats return paged results which can be requested using
page parameters. For >500 transforms count wasn't parsed out of the server response but taken from
size of the list of transforms.
The change also adds client/server hlrc tests and fixes a wrong type for count in get.
fixes#56245
Backport #62766 to 7.x branch.
The bulk api cache the resolved concrete indices when resolving the user provided
index name into the actual index name. The validation that prevents write ops other
than create from being executed in a data stream was only performed if the result
wasn't cached. In case of cached resolvings, the validation never occurs.
The validation would be skipped for all bulk items for a data stream after a create
operation for that same data stream. This commit ensures that the validation is always
performed for all bulk items (whether the concrete index resolution has been cached or
not cached).
Closes#62762
When state persistence was first implemented for data frame analytics
we had the assumption that state would always fit in a single document.
However this is not the case any more.
This commit adds handling of state that spreads over multiple documents.
Backport of #62564
This fixes reindexing progress in the scenario when a DFA job that had not finished
reindexing is resumed (either because the user called stop and start or because the
job was reassigned in the middle of reindexing). Before the fix reindexing progress
stays to the value it had reached before until it surpasses that value.
When we resume a data frame analytics job we want to preserve reindexing progress
and reset all other phases. Except for when reindexing was not completed.
In that case we are deleting the destination index and starting reindexing
from scratch. Thus we need to reset reindexing progress too.
Backport of #62772
To better align the plugin naming with other mapper plugins under x-pack (e.g.
mapper-flattened) this PR changes the plugin name and the containing directory
to "mapper-version"
This change adds support for the recently introduced case insensitivity flag for
wildcard and prefix queries. Since version field values are encoded differently we
need to adapt our own AutomatonQuery variation to add both cases if case insensitivity
is turned on.
Most of our field types have the same implementation for their `existsQuery` method which relies on doc_values if present, otherwise it queries norms if available or uses a term query against the _field_names meta field. This standard implementation is repeated in many different mappers.
There are field types that only query doc_values, because they always have them, and field types that always query _field_names, because they never have norms nor doc_values. We could apply the same standard logic to all of these field types as `MappedFieldType` has the knowledge about what data structures are available.
This commit introduces a standard implementation that does the right thing depending on the data structure that is available. With that only field types that require a different behaviour need to override the existsQuery method.
At the same time, this no longer forces subclasses to override `existsQuery`, which could be forgotten when needed. To address this we introduced a new test method in `MapperTestCase` that verifies the `existsQuery` being generated and its consistency with the available data structures.
Julie Tibshirani
Lisa Cawley
Andrei Stefan
Enrico Zimuel
Ioannis Kakavas
Jim Ferenczi
Albert Zaharovits
Jay Modi
Marios Trivyzas
Bogdan Pintea
Marios Trivyzas
Costin Leau
Hendrik Muhs
Tanguy Leroux
Ryland Herrick
Lee Hinman
Andras Palinkas
Andras Palinkas
Przemysław Witek
David Roberts
Dimitris Athanasiou
Tomas Della Vedova
Benjamin Trent
Mayya Sharipova
Luca Cavanna
Alan Woodward
Tim Vernum
Stuart Tettemer
Gordon Brown
Igor Motov
David Kyle
Yang Wang
Armin Braun
Nhat Nguyen
Costin Leau
Jason Tedor
Martijn van Groningen
markharwood
Joe Gallo
Christoph Büscher
Ignacio Vera
Howard
Tim Brooks
Daniel Mitterdorfer
Andrei Dan
Rory Hunter
James Rodewig