Today this test fails because the sizes of the snapshot
shards are only kept in a very short period of time in
the InternalSnapshotsInfoService and are not
guaranteed to exist once the shards are correctly
assigned.
closes#64167
If we encounter an exception during extracting data in a data
frame analytics job, we retry once. However, we were not catching
exceptions thrown from processing the search response. This may
result in an infinite loop that causes a stack overflow.
This commit fixes this problem.
Backport of #64947
Fixes the inconsistency between the type of the object returned by the
`SIGN()/SIGNUM()` SQL functions and the specified `DataType`.
In the Class Sign, DataType is DataTypes.INTEGER. The source code is as
follows：
```
public DataType dataType() {
return DataTypes.INTEGER;
}
```
But In the Class MathProcessor, the source code of SIGN((Object l),
Parameter and return value types are the same. Therefore, when using
double or float parameters to test, there is a little problem, the test
method is like the following curl :
```
curl -XPOST 127.0.0.1:9200/_sql -d "{\"query\":\"select SIGN(1.0) \"}" \
-H 'Content-Type: application/json'
```
The result is:
```
{"columns":[{"name":"SIGN(1.0)","type":"integer"}],"rows":[[1.0]]}
```
The result value is `1.0`, but the type is `integer`.
Signed-off-by: mantuliu <240951888@qq.com>
Co-authored-by: Marios Trivyzas <matriv@gmail.com>
(cherry picked from commits aa78301e71f, ced3c1281c7, 40e5b9b)
If the deleted index has N shards, then ShardFollowTaskCleaner can send
N*(N-1)/2 requests to remove N shard-follow tasks. I think that's fine
as the implementation is straightforward. The test failed when the
deleted index has 8 shards. This commit increases the timeout in the
test.
Closes#64311
When using custom processors, the field names extracted from the documents are not the
same as the feature names used for training.
Consequently, it is possible for the stratified sampler to have an incorrect view of the feature rows.
This can lead to the wrong column being read for the class label, and thus throw errors on training
row extraction.
This commit changes the training row feature names used by the stratified sampler so that it matches
the names (and their order) that are sent to the analytics process.
It is possible that a value mapped as a `keyword` has any scalar value type. This includes any numerical value, String, or boolean.
This commit allows `boolean` types to be considered as a part of the categorical feature collection when this is the case.
* Consistency in writing style
Removing spaces before and after brackets for consistency.
* Remove typo
Remove one of two consecutive "the"s
Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co>
This commit converts build code that downloads distributions or other
artifacts to use the new no-kpi subdomain, and removes the formerly used
no-kpi header.
Random the compression factor starting with 1 (to elimitinate nearly 0 values)
which will only use one centroid (and yield 0 for MAD as the aproximate median
is the same as the single centroid mean value)
(cherry picked from commit 940e0f1fde0f40f99af117dd03ab0891c9eedae6)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Summary of the issue and the root cause:
```
(1) SELECT 100, 100 -> success
(2) SELECT ?, ? (with params: 100, 100) -> success
(3) SELECT 100, 100 FROM test -> Unknown output attribute exception for the second 100
(4) SELECT ?, ? FROM test (params: 100, 100) -> Unknown output attribute exception for the second ?
(5) SELECT field1 as "x", field1 as "x" FROM test -> Unknown output attribute exception for the second "x"
```
There are two separate issues at play here:
1. Construction of `AttributeMap`s keeps only one of the `Attribute`s with the same name even if the `id`s are different (see the `AttributeMapTests` in this PR). This should be fixed no matter what, we should not overwrite attributes with one another during the construction of the `AttributeMap`.
2. The `id` on the `Alias`es is not the same in case the `Alias`es have the same `name` and same `child`
It was considered to simpy fix the second issue by just reassigning the same `id`s to the `Alias`es with the same name and child, but it would not solve the `unknown output attribute exception` (see notes below). This PR covers the fix for the first issue.
Relates to #56013
The SchedulerEngine used by SLM uses a custom runnable that will
schedule itself for its next execution if there is one to run. For the
majority of jobs, this scheduling could be many hours or days away. Due
to the scheduling so far in advance, there is a chance that time drifts
on the machine or even that time varies core to core so there is no
guarantee that the job actually runs on or after the scheduled time.
This can cause some jobs to reschedule themselves for the same
scheduled time even if they ran only a millisecond prior to the
scheduled time, which causes unexpected actions to be taken such as
what appears as duplicated snapshots.
This change resolves this by checking the triggered time against the
scheduled time and using the appropriate value to ensure that we do
not have unexpected job runs.
Relates #63754
Backport of #64501
testMoveToStepRereadsPolicy relied on an updated ILM policy that had
a rollover condition that enabled the index to be rolled after one second.
This changes the test to use a `max_doc`:1 condition so it's under the
test's control to trigger the condition.
(cherry picked from commit 73ab35a411bcdf5a92eb3d2b3bae5b1132a2bb56)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
The yml "Test Invalid Move To Step With Invalid Next Step" worked based on
assuming the current step is a particular one. As we can't control the
timing of ILM and we can't busy assert in yml test, this converts the
test to a java test and makes use of `assertBusy`
This converts the explain lifecycle yml tests that depende on ILM having run
at least once to a java integration test that makes use of `assertBusy`.
(cherry picked from commit 6afd0422ed5ff0e3a2e5661f0e6d192bdad9af4f)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
This field is added from version 7.11 onwards. We are
adding it to the list of ignored fields for data frame analytics
in 7.10 to avoid failing to start an outlier detection job
in a mixed cluster environment.
Relates #64503
ForecastIT.testOverflowToDisk has been observed to fail a few
times in FIPS JVMs because it takes longer than the permitted
30 seconds. This PR bumps the timeout up to 60 seconds.
Fixes#63793
This test failed sometimes for various reasons: an empty bulk request
that can't be validated, a background force-merge that completes after
the store stats were collected and finally an assertBusy() that waits
10 seconds while we usually wait 60s on the follower cluster in CCR
tests.
Closes#64167
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.
Assert the index is restored correctly and use new snapshot.
(cherry picked from commit fc8c6dd56d602b4a62ee1ff484f00caab92dc6e2)
These tests were added to do a proper end-to-end test of the
memory usage of the geotile_grid and geohash_grid aggregations
on `geo_shape` fields. Although this was asperational,
the truth is — the test environment does not run these aggregations
in isolation. This means that the memory overhead is variable and
too flaky to rely on over time. The unit tests for circuit-breaking
remain.
Closes#63158.
This commit internalizes whether or not a role represents the ability to
contain data. In the future, this will let us remove the compatibility
role notion.
Wrap a verification_exception in case there is no valid index available in an index_not_found_exception providing also the original index pattern that may be lost in the chain of filters involving the Security one.
(cherry picked from commit 9c9da2f2f9a4ad12704f7d3a273f067e96cd2054)
The new fields option allows to fetch the value of all fields in the mapping.
However, internal fields that are used by some field mappers are also shown when
concrete fields retrieved through a pattern (`*` or `foo*`).
We have a [long term plan](https://github.com/elastic/elasticsearch/issues/63446) to hide these fields in field_caps and from pattern resolution
so this change is just a hot fix to ensure that they don't break the retrieval in the meantime.
The `flattened._keyed field will show up as an empty field when using a pattern that match the
flattened field.
Relates #63446
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.
To fully specifiy the behavior for the tiered data roles then:
- starting a new node with the defaults: enabled
- starting a new node with node.roles configured: enabled if and only
if the tiered data roles are explicitly configured, independently
of the node having the data role
- starting a new node with node.data enabled: enabled unless the
tiered data roles are explicitly disabled
- starting a new node with node.data disabled: disabled unless the
tiered data roles are explicitly enabled
XPack usage starts out on management threads, but depending on the
implementation of the usage plugin, they could end up running on
transport threads instead. Fixed to always reschedule on a management
thread.
With this change, we will always return the same point in time in a
search response as its input until we implement the retry mechanism
for the point in times.
The `NodeNotConnectedException` exception can be nested as well in the
fairly unlikley case of the disconnect occuring between the connected check
and actually sending the request in the transport service.
Closes#63233
There is a small chance that the file deletion will run
on the searchable snapshot thread pool and not on the test
thread now that the cache is non-blocking in which case
we fail the assertion unless we wait for that thread.
Adds support for the unsigned_long type to data frame analytics.
This type is handled in the same way as the long type. Values
sent to the ML native processes are converted to floats and
hence will lose accuracy when outside the range where a float
can uniquely represent long values.
Backport of #64066
Tanguy Leroux
Dimitris Athanasiou
Andras Palinkas
Daniel Mitterdorfer
Nhat Nguyen
Benjamin Trent
Adam Locke
James Rodewig
Ryan Ernst
Lee Hinman
Jim Ferenczi
Dan Hermann
Andrei Dan
Jay Modi
Mayya Sharipova
Ignacio Vera
David Roberts
Marios Trivyzas
Tal Levy
Jason Tedor
Armin Braun
Andrei Stefan
Costin Leau
Henning Andersen
David Kyle