The new fields option allows to fetch the value of all fields in the mapping.
However, internal fields that are used by some field mappers are also shown when
concrete fields retrieved through a pattern (`*` or `foo*`).
We have a [long term plan](https://github.com/elastic/elasticsearch/issues/63446) to hide these fields in field_caps and from pattern resolution
so this change is just a hot fix to ensure that they don't break the retrieval in the meantime.
The `flattened._keyed field will show up as an empty field when using a pattern that match the
flattened field.
Relates #63446
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.
To fully specifiy the behavior for the tiered data roles then:
- starting a new node with the defaults: enabled
- starting a new node with node.roles configured: enabled if and only
if the tiered data roles are explicitly configured, independently
of the node having the data role
- starting a new node with node.data enabled: enabled unless the
tiered data roles are explicitly disabled
- starting a new node with node.data disabled: disabled unless the
tiered data roles are explicitly enabled
XPack usage starts out on management threads, but depending on the
implementation of the usage plugin, they could end up running on
transport threads instead. Fixed to always reschedule on a management
thread.
With this change, we will always return the same point in time in a
search response as its input until we implement the retry mechanism
for the point in times.
The `NodeNotConnectedException` exception can be nested as well in the
fairly unlikley case of the disconnect occuring between the connected check
and actually sending the request in the transport service.
Closes#63233
There is a small chance that the file deletion will run
on the searchable snapshot thread pool and not on the test
thread now that the cache is non-blocking in which case
we fail the assertion unless we wait for that thread.
Adds support for the unsigned_long type to data frame analytics.
This type is handled in the same way as the long type. Values
sent to the ML native processes are converted to floats and
hence will lose accuracy when outside the range where a float
can uniquely represent long values.
Backport of #64066
Replacing the mechanism for eviction and listener references via a read-write lock by
a reference counting implementation.
This fixes a bug that caused test failure #63586 in which concurrently trying to acquire or release
an eviction listener while doing a file operation would sometimes lead to throwing an exception
since the `tryLock` call on the read lock would fail in this case.
Also this removes the possibility of blocking cluster state updates as a result of them waiting
on the write-lock which might take a long time if a slow read operation executes concurrently.
Closes#63586
The deprecation indexing code was writing to a regular data stream,
and it is not yet possible to hide a data stream or prefix it with
a period. This functionality we be re-added once it is possible to
mark a data stream as hidden, and also to not rely on the standard
logs template since that can be disabled.
The officially supported way to clearing all entries from a cache is to use
wildcard of either * or _all. Though empty string has the same effect, it was
never intended. Therefore the tests should not use empty string and this PR
changes them to use *.
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long
fixes#63871
backport #63940
When calculating feature importance, the leaf values directly correlate the value of the importance.
Consequently, positive leaf values -> positive feature importance
negative leaf values -> negative feature importance.
It follows that for binary classification, this is done such that the importance relates to the leaf values, which relate directly to the "probability of class 1".
So, the feature importance calculated is always for the importance as it relates to class 1.
The inverse is the importance as it relates to class 0.
Max and min aggs were producing wrong results for unsigned_long field
if field was indexed. If field is indexed for max/min aggs instead of
field data, we use values from indexed Points, values of which
are derived using method pointReaderIfPossible. Before
UnsignedLongFieldType#pointReaderIfPossible was incorrectly
producing values, as it failed to shift them back to original
values.
This patch fixes method pointReaderIfPossible to produce
correct original values.
Relates to #60050
* [DOCS] Combining important config settings into a single page (#63849)
* Combining important config settings into a single page.
* Updating ids for two pages causing link errors and implementing redirects.
* Updating links to use IDs instead of xrefs.
This commit updates the rest API specs for
- security.get_role_mapping to specify name as a list type.
- security.get_role to specify name as a list type.
Co-authored-by: Russ Cam <russ.cam@elastic.co>
* Async search should retry updates on version conflict
The _async_search APIs can throw version conflict exception when the internal response
is updated concurrently. That can happen if the final response is written while the user
extends the expiration time. That scenario should be rare but it happened in Kibana for
several users so this change ensures that updates are retried at least 5 times. That
should resolve the transient errors for Kibana. This change also preserves the version
conflict exception in case the retry didn't work instead of returning a confusing 404.
This commit also ensures that we don't delete the response if the search was cancelled
internally and not deleted explicitly by the user.
Closes#63213
The `remote_monitoring_agent` reserved role is extended to grant more privileges
over the metricbeat-* index pattern.
In addition to the index and create_index index privileges that it granted already,
it now also grants the view_index_metadata privilege.
Closes#63203
This commit ensures that jobs within the SchedulerEngine do not
continue to run after they are cancelled. There was no synchronization
between the cancel method of an ActiveSchedule and the run method, so
an actively running schedule would go ahead and reschedule itself even
if the cancel method had been called.
This commit adds synchronization between cancelling and the scheduling
of the next run to ensure that the job is cancelled. In real life
scenarios this could manifest as a job running multiple times for
SLM. This could happen if a job had been triggered and was cancelled
prior to completing its run such as if the node was no longer the
master node or if SLM was stopping/stopped.
Closes#63754
Backport of #63762
This commit updates the APIs in the logstash plugin to handle
IndexNotFoundExceptions that are returned by client calls. Until we
have the creation of this index in place, we need to handle this case
and not let the exception propagate out of the API.
Backport of #63698
* SQL: integer parameter validation in string functions (#58923)
In insert, locate, substring function, when argument `start` or `length` is greater than Integer.MAX_INT OR less then Integer.MIN_INT + 1 (note that `start` need to minus 1), it causes overflow and leads to unexpected results.
* Add range checks for BinaryStringNumericProcessors
- Add range checks for Left, Right, Repeat.
- Minor refactorings on initial PR changes.
Co-authored-by: yinanwu <yinanwu@tencent.com>
(cherry picked from commit bf6dc58b93529f977d035a846d083b1c31867694)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Add a new gradle module under eql/qa which runs and validates a set of
queries over a 4m event dataset (restored from a snapshot residing in a
gcs bucket). The results are providing by running the exact set of queries
with Python EQL against the same dataset.
Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
(cherry picked from commit 1cf789e5fcfb0f364f665bfaac021e24a4c2f556)
Co-authored-by: Mark Vieira <portugee@gmail.com>
This commit fixes two issues in dealing with bool fields in EQL:
- avoid simplifications of field == true expressions
- adding comparison to clauses on fields missing logic (where bool)
Fix#63693
(cherry picked from commit d10a5d0e842bbd4e0031834de948ceb24da3872b)
(cherry picked from commit 0227da3a275c7f22ff524d99d53e1a79146f9e28)
* Allow all indices options variants
Irrespective of allow_no_indices value, throw VerificationException when
there is no index validated
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Mentions the list of wildchars in case a wildchar is used as an
`ESCAPE` character.
Relates #63428
(cherry picked from commit 74cbcf871e9593b3640e382ae6845168fd14966b)
For a query like `SELECT name FROM test WHERE name LIKE ''%c*'` ES SQL
generates an error. `*` is not a special character in a `LIKE` construct
and it's expected to not needing to be escaped, so the previous query
should work as is.
In the LIKE pattern any `*` character was treated as invalid character
and the usage of `%` or `_` was suggested instead. But `*` is a valid,
acceptable non-wildcard on the right side of the `LIKE` operator.
Fix: #55108
(cherry picked from commit 190d9fe3deb31aed0d8f312007360625d4fff217)
Now that deprecation logs get indexed to a data stream, if we
do not load the data stream plugin in our tests and any test
generates a deprecation log message then millions of exceptions
get logged, slowing down the tests to the extent that they can
fail.
This change loads the data streams plugin during the ML internal
cluster tests. (It should already be present in external cluster
tests.)
Fixes#63548
Armin Braun
jimczi
Jason Tedor
Costin Leau
Henning Andersen
Nhat Nguyen
David Kyle
David Roberts
Rory Hunter
James Rodewig
Yang Wang
Hendrik Muhs
Ignacio Vera
Benjamin Trent
Mayya Sharipova
Adam Locke
Julie Tibshirani
Lisa Cawley
Andrei Stefan
Enrico Zimuel
Ioannis Kakavas
Albert Zaharovits
Jay Modi
Marios Trivyzas
Bogdan Pintea
Marios Trivyzas
Tanguy Leroux
Ryland Herrick
Lee Hinman
Andras Palinkas
Andras Palinkas
Przemysław Witek