When changing a job state using a mechanism that doesn't
wait for the desired state to be reached within the production
code the test code needs to loop until the cluster state has
been updated.
Closes#52451
Closes#52450. `setRandomIndexSettings(...)` in `ESIntegTestCase` was
using both a thread-local and a supplied source of randomness. Fix this
method to only use the supplied source.
Backport from #52179
Don't rely on the delete index api to resolve all the enrich indices for a particular enrich policy using a '[policy_name]-*' wildcard expression. With this change, the delete policy api will resolve the indices to remove and pass that directly to the delete index api.
This resolves a bug, that if `action.destructive_requires_name` setting has been set to true then the delete policy api is unable to remove the enrich indices related to the policy being deleted.
Closes#51228
Co-authored-by: bellengao <gbl_long@163.com>
audit messages are stored in the notifications index, so audit information is lost for integration
tests. This change forwards audit messages to logs, so they can help to debug issues.
relates: #51627
Following the change to store cluster state in Lucene indices
(#50907) it can take longer for all the cluster state updates
associated with node failure scenarios to be processed during
internal cluster tests where several nodes all run in the same
JVM.
Refactor the code to allow contextual parameterization of dateFormat and
name.
Separate aggs/query implementation though there's room for improvement
in the future
(cherry picked from commit e086f81b688875b33d01e4504ce7377031c8cf28)
When `FilterStreamInput` wraps a Netty `ByteBuf` based stream it
did not forward the bulk primitive reads to the delegate.
These are optimized on the delegate but if they're not forwarded
then the delegate will be called e.g. 4 times to read an `int`.
This happens for essentially all network reads prior to this
change because they all run from a `NamedWritableAwareStreamInput`.
This also required optimising `BufferedChecksumStreamInput` individually to use bulk reads from the buffer because it implicitly assumed that the filter stream input wouldn't override any of the bulk operations.
The shard follow task cleaner executes on behalf of the user to clean up
a shard follow task after the follower index has been
deleted. Otherwise, these persistent tasks are left laying around, and
they fail to execute because the follower index has been deleted. In the
face of security, attempts to complete these persistent tasks would
fail. This is because these cleanups are executed under the system
context (this makes sense, they are happening on behalf of the user
after the user has executed an action) but the system role was never
granted the permission for persistent task completion. This commit
addresses this by adding this cluster privilege to the system role.
Reading the startup scripts does not elucidate how JVM options are
applied. Instead, the reader must consult the source for the JVM options
parser. This commit adds some transparency around this process so that
it easier to understand reading the startup scripts how the final JVM
options to start Elasticsearch are constructed.
We marked the `init` ILM step as retryable but our test used `waitUntil`
without an assert so we didn’t catch the fact that we were not actually
able to retry this step as our ILM state didn’t contain any information
about the policy execution (as we were in the process of initialising
it).
This commit manually sets the current step to `init` when we’re moving
the ilm policy into the ERROR step (this enables us to successfully
move to the error step and later retry the step)
* ShrunkenIndexCheckStep: Use correct logger
(cherry picked from commit f78d4b3d91345a2a8fc0f48b90dd66c9959bd7ff)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
The oss and default bats tests were removed, but these references to
them remained, causing gradle failures when trying to run packaging
tests. While the upgrade and plugins bats tests should still be tested,
that is being handled in #51565. This commit removes the outdated
references.
closes#51974
Translate to an agg query even if only literals are selected,
so that the correct number of rows is returned (number of buckets).
Fix issue with key only in GROUP BY (not in select) and WHERE clause:
Resolve aggregates and groupings based on the child plan which holds
the info info for all the fields of the underlying table.
Fixes: #41951Fixes: #41413
(cherry picked from commit 45b85809678b34a448639a420b97e25436ae851f)
* Set default ES_PATH_CONF for package scriptlets
Our packages use scriptlets to create or update the Elasticsearch
keystore as necessary when installing or upgrading an Elasticsearch
package. If these scriptlets don't work as expected, Elasticsearch may
try and fail to create or upgrade the keystore at startup time. This
will prevent Elasticsearch from starting up at all.
These scriptlets use the Elasticsearch keystore command-line tools. Like
most of our command-line tools, the keystore tools will by default get
their value for ES_PATH_CONF from a system configuration file:
/etc/sysconfig/elasticsearch for RPMs, /etc/default/elasticsearch for
debian packages. Previously, if the user removed ES_PATH_CONF from that
system configuration file (perhaps thinking that it is obsolete when
the same variables is also defined in the systemd unit file), the
keystore command-line tools would fail. Scriptlet errors do not seem to
cause the installation to fail, and for RPMs the error message is easy
to miss in command output.
This commit adds a line of bash to scriptlets that will set ES_PATH_CONF
to a default when ES_PATH_CONF is unset, rather than only when the
system configuration file is missing.
* Fix "Description"s for various sections in the functions pages.
* Added a TIP for searching using a routing key.
* Other small polishings
(cherry picked from commit 9fad0b1ac4409a42c435ed040f41cbaea18930a3)
* Filter empty lines from docker ls response
In order to cut down on test time, our docker/vagrant tests build the
docker image outside of the vagrant VM. When we get around to launching
the Vagrant VM, we mount that already-built docker image to a known
location. At that point, we need to load the docker image. But we only
want to load it once. As we're running tests, we use "docker ls" to
check whether the local image is loaded for use. Empty output from the
particular ls invocation means no image is loaded.
There was a bug in how we checked this. In Java, splitting an empty
string will yield an array containing one empty string. So when we're
counting the output from the docker ls command, we need to filter out
empty lines in order to proceed to loading the image for docker tests.
The `top_metrics` agg is kind of like `top_hits` but it only works on
doc values so it *should* be faster.
At this point it is fairly limited in that it only supports a single,
numeric sort and a single, numeric metric. And it only fetches the "very
topest" document worth of metric. We plan to support returning a
configurable number of top metrics, requesting more than one metric and
more than one sort. And, eventually, non-numeric sorts and metrics. The
trick is doing those things fairly efficiently.
Co-Authored by: Zachary Tong <zach@elastic.co>
ML mappings and index templates have so far been created
programmatically. While this had its merits due to static typing,
there is consensus it would be clear to maintain those in json files.
In addition, we are going to adding ILM policies to these indices
and the component for a plugin to register ILM policies is
`IndexTemplateRegistry`. It expects the templates to be in resource
json files.
For the above reasons this commit refactors ML mappings and index
templates into json resource files that are registered via
`MlIndexTemplateRegistry`.
Backport of #51765
Fixes the the no-query optimization for `min` and `max` aggregations
for `date_nanos` fields by delegating decoding dates "through" their
`resolution` member.
Closes#52220
This commit makes the names of fetch subphases more consistent:
* Now the names end in just 'Phase', whereas before some ended in
'FetchSubPhase'. This matches the query subphases like AggregationPhase.
* Some names include 'fetch' like FetchScorePhase to avoid ambiguity about what
they do.
Ironically PreventFailingBuildIT.testSoThatTestsDoNotFail is causing failures
as documented in #52197. The no longer serves a purpose and can now be removed.
This is to support the ML categorization wizard.
Currently cluster:admin/analyze is only provided with the
"manage" cluster privilege, which is an excessive privilege
level to provide access to this single feature. It means
that the ML categorization wizard only works for extremely
highly privileged users.
Following this change the Kibana system user will be
permitted to run the _analyze endpoint on supplied strings
(not on an index). The ML UI will then call the _analyze
endpoint as the Kibana system user after first checking
that the logged-in user is permitted to create an ML job.
This will mean that users with the more reasonable
"manage_ml" cluster privilege will be permitted to use
the ML categorization wizard.
(This is also consistent with the way the ML UI will access
_all_ Elasticsearch functionality when the "ML in Spaces"
project is completed.)
Closes#51391
Relates elastic/kibana#57375
This adds a builder and parsed results for the `string_stats`
aggregation directly to the high level rest client. Without this the
HLRC can't access the `string_stats` API without the elastic licensed
`analytics` module.
While I'm in there this adds a few of our usual unit tests and
modernizes the parsing.
The example of how to access the nano value of a date_nanos field has
been broken since it was created. This commit fixes it to use the
correct scripting methods.
closes#51931
When `date_histogram` attempts to optimize itself it for a particular
time zone it checks to see if the entire shard is within the same
"transition". Most time zone transition once every size months or
thereabouts so the optimization can usually kicks in.
*But* it crashes when you attempt feed it a time zone who's last DST
transition was before epoch. The reason for this is a little twisted:
before this patch it'd find the next and previous transitions in
milliseconds since epoch. Then it'd cast them to `Long`s and pass them
into the `DateFieldType` to check if the shard's contents were within
the range. The trouble is they are then converted to `String`s which are
*then* parsed back to `Instant`s which are then convertd to `long`s. And
the parser doesn't like most negative numbers. And everything before
epoch is negative.
This change removes the
`long` -> `Long` -> `String` -> `Instant` -> `long` chain in favor of
passing the `long` -> `Instant` -> `long` which avoids the fairly complex
parsing code and handles a bunch of interesting edge cases around
epoch. And other edge cases around `date_nanos`.
Closes#50265
David Roberts
Przemysław Witek
Rory Hunter
Armin Braun
Przemko Robakowski
Martijn van Groningen
Hendrik Muhs
Nhat Nguyen
Costin Leau
Jason Tedor
Andrei Dan
Ryan Ernst
Hicham Mallah
William Brafford
OriGlassman
James Rodewig
Andrei Stefan
Andrei Stefan
Lisa Cawley
Nik Everett
Dimitris Athanasiou
bellengao
Marios Trivyzas
Ioannis Kakavas
Igor Motov
Mark Vieira
Julie Tibshirani
David Kyle
debadair
Ryan Ernst