ForecastIT.testOverflowToDisk has been observed to fail a few
times in FIPS JVMs because it takes longer than the permitted
30 seconds. This PR bumps the timeout up to 60 seconds.
Fixes#63793
This test failed sometimes for various reasons: an empty bulk request
that can't be validated, a background force-merge that completes after
the store stats were collected and finally an assertBusy() that waits
10 seconds while we usually wait 60s on the follower cluster in CCR
tests.
Closes#64167
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.
Assert the index is restored correctly and use new snapshot.
(cherry picked from commit fc8c6dd56d602b4a62ee1ff484f00caab92dc6e2)
These tests were added to do a proper end-to-end test of the
memory usage of the geotile_grid and geohash_grid aggregations
on `geo_shape` fields. Although this was asperational,
the truth is — the test environment does not run these aggregations
in isolation. This means that the memory overhead is variable and
too flaky to rely on over time. The unit tests for circuit-breaking
remain.
Closes#63158.
This commit internalizes whether or not a role represents the ability to
contain data. In the future, this will let us remove the compatibility
role notion.
Wrap a verification_exception in case there is no valid index available in an index_not_found_exception providing also the original index pattern that may be lost in the chain of filters involving the Security one.
(cherry picked from commit 9c9da2f2f9a4ad12704f7d3a273f067e96cd2054)
The new fields option allows to fetch the value of all fields in the mapping.
However, internal fields that are used by some field mappers are also shown when
concrete fields retrieved through a pattern (`*` or `foo*`).
We have a [long term plan](https://github.com/elastic/elasticsearch/issues/63446) to hide these fields in field_caps and from pattern resolution
so this change is just a hot fix to ensure that they don't break the retrieval in the meantime.
The `flattened._keyed field will show up as an empty field when using a pattern that match the
flattened field.
Relates #63446
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.
To fully specifiy the behavior for the tiered data roles then:
- starting a new node with the defaults: enabled
- starting a new node with node.roles configured: enabled if and only
if the tiered data roles are explicitly configured, independently
of the node having the data role
- starting a new node with node.data enabled: enabled unless the
tiered data roles are explicitly disabled
- starting a new node with node.data disabled: disabled unless the
tiered data roles are explicitly enabled
XPack usage starts out on management threads, but depending on the
implementation of the usage plugin, they could end up running on
transport threads instead. Fixed to always reschedule on a management
thread.
With this change, we will always return the same point in time in a
search response as its input until we implement the retry mechanism
for the point in times.
The `NodeNotConnectedException` exception can be nested as well in the
fairly unlikley case of the disconnect occuring between the connected check
and actually sending the request in the transport service.
Closes#63233
There is a small chance that the file deletion will run
on the searchable snapshot thread pool and not on the test
thread now that the cache is non-blocking in which case
we fail the assertion unless we wait for that thread.
Adds support for the unsigned_long type to data frame analytics.
This type is handled in the same way as the long type. Values
sent to the ML native processes are converted to floats and
hence will lose accuracy when outside the range where a float
can uniquely represent long values.
Backport of #64066
Replacing the mechanism for eviction and listener references via a read-write lock by
a reference counting implementation.
This fixes a bug that caused test failure #63586 in which concurrently trying to acquire or release
an eviction listener while doing a file operation would sometimes lead to throwing an exception
since the `tryLock` call on the read lock would fail in this case.
Also this removes the possibility of blocking cluster state updates as a result of them waiting
on the write-lock which might take a long time if a slow read operation executes concurrently.
Closes#63586
The deprecation indexing code was writing to a regular data stream,
and it is not yet possible to hide a data stream or prefix it with
a period. This functionality we be re-added once it is possible to
mark a data stream as hidden, and also to not rely on the standard
logs template since that can be disabled.
The officially supported way to clearing all entries from a cache is to use
wildcard of either * or _all. Though empty string has the same effect, it was
never intended. Therefore the tests should not use empty string and this PR
changes them to use *.
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long
fixes#63871
backport #63940
When calculating feature importance, the leaf values directly correlate the value of the importance.
Consequently, positive leaf values -> positive feature importance
negative leaf values -> negative feature importance.
It follows that for binary classification, this is done such that the importance relates to the leaf values, which relate directly to the "probability of class 1".
So, the feature importance calculated is always for the importance as it relates to class 1.
The inverse is the importance as it relates to class 0.
Max and min aggs were producing wrong results for unsigned_long field
if field was indexed. If field is indexed for max/min aggs instead of
field data, we use values from indexed Points, values of which
are derived using method pointReaderIfPossible. Before
UnsignedLongFieldType#pointReaderIfPossible was incorrectly
producing values, as it failed to shift them back to original
values.
This patch fixes method pointReaderIfPossible to produce
correct original values.
Relates to #60050
* [DOCS] Combining important config settings into a single page (#63849)
* Combining important config settings into a single page.
* Updating ids for two pages causing link errors and implementing redirects.
* Updating links to use IDs instead of xrefs.
This commit updates the rest API specs for
- security.get_role_mapping to specify name as a list type.
- security.get_role to specify name as a list type.
Co-authored-by: Russ Cam <russ.cam@elastic.co>
* Async search should retry updates on version conflict
The _async_search APIs can throw version conflict exception when the internal response
is updated concurrently. That can happen if the final response is written while the user
extends the expiration time. That scenario should be rare but it happened in Kibana for
several users so this change ensures that updates are retried at least 5 times. That
should resolve the transient errors for Kibana. This change also preserves the version
conflict exception in case the retry didn't work instead of returning a confusing 404.
This commit also ensures that we don't delete the response if the search was cancelled
internally and not deleted explicitly by the user.
Closes#63213
The `remote_monitoring_agent` reserved role is extended to grant more privileges
over the metricbeat-* index pattern.
In addition to the index and create_index index privileges that it granted already,
it now also grants the view_index_metadata privilege.
Closes#63203
This commit ensures that jobs within the SchedulerEngine do not
continue to run after they are cancelled. There was no synchronization
between the cancel method of an ActiveSchedule and the run method, so
an actively running schedule would go ahead and reschedule itself even
if the cancel method had been called.
This commit adds synchronization between cancelling and the scheduling
of the next run to ensure that the job is cancelled. In real life
scenarios this could manifest as a job running multiple times for
SLM. This could happen if a job had been triggered and was cancelled
prior to completing its run such as if the node was no longer the
master node or if SLM was stopping/stopped.
Closes#63754
Backport of #63762
This commit updates the APIs in the logstash plugin to handle
IndexNotFoundExceptions that are returned by client calls. Until we
have the creation of this index in place, we need to handle this case
and not let the exception propagate out of the API.
Backport of #63698
* SQL: integer parameter validation in string functions (#58923)
In insert, locate, substring function, when argument `start` or `length` is greater than Integer.MAX_INT OR less then Integer.MIN_INT + 1 (note that `start` need to minus 1), it causes overflow and leads to unexpected results.
* Add range checks for BinaryStringNumericProcessors
- Add range checks for Left, Right, Repeat.
- Minor refactorings on initial PR changes.
Co-authored-by: yinanwu <yinanwu@tencent.com>
(cherry picked from commit bf6dc58b93529f977d035a846d083b1c31867694)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Add a new gradle module under eql/qa which runs and validates a set of
queries over a 4m event dataset (restored from a snapshot residing in a
gcs bucket). The results are providing by running the exact set of queries
with Python EQL against the same dataset.
Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
(cherry picked from commit 1cf789e5fcfb0f364f665bfaac021e24a4c2f556)
Co-authored-by: Mark Vieira <portugee@gmail.com>
This commit fixes two issues in dealing with bool fields in EQL:
- avoid simplifications of field == true expressions
- adding comparison to clauses on fields missing logic (where bool)
Fix#63693
(cherry picked from commit d10a5d0e842bbd4e0031834de948ceb24da3872b)
(cherry picked from commit 0227da3a275c7f22ff524d99d53e1a79146f9e28)
David Roberts
Tanguy Leroux
Marios Trivyzas
Tal Levy
Jason Tedor
James Rodewig
Armin Braun
Andrei Stefan
jimczi
Costin Leau
Henning Andersen
Nhat Nguyen
David Kyle
Rory Hunter
Yang Wang
Hendrik Muhs
Ignacio Vera
Benjamin Trent
Mayya Sharipova
Adam Locke
Julie Tibshirani
Lisa Cawley
Enrico Zimuel
Ioannis Kakavas
Albert Zaharovits
Jay Modi
Marios Trivyzas
Bogdan Pintea