Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.
Assert the index is restored correctly and use new snapshot.
(cherry picked from commit fc8c6dd56d602b4a62ee1ff484f00caab92dc6e2)
* Add tests for using ES_JAVA_OPTS with windows service
* Relocate ES_JAVA_OPTS delimiter munging
* Don't use equals for -Xmx and -Xms args
* Write newlines in temporary configs
These tests were added to do a proper end-to-end test of the
memory usage of the geotile_grid and geohash_grid aggregations
on `geo_shape` fields. Although this was asperational,
the truth is — the test environment does not run these aggregations
in isolation. This means that the memory overhead is variable and
too flaky to rely on over time. The unit tests for circuit-breaking
remain.
Closes#63158.
- Replaces more abstract docs about object structure and values source with task-based examples.
- Relocates several sections from the current `misc.asciidoc` file.
- Alphabetically sorts agg categories in the nav.
- Removes the matrix agg family. Moves the stats matrix agg under the metric agg family
This commit internalizes whether or not a role represents the ability to
contain data. In the future, this will let us remove the compatibility
role notion.
This commit updates the bundled jdk to 15.0.1, and at the same time once
again switches the bundled jdk back to adoptopenjdk, which has fixed
their build problem and regained support for glibc 2.12.
closes#64026
The docs for the geoip processor database_file option appear to indicate
that all geoip databases are in the config directory. This is leftover
legacy from when this was the case when ingest-geoip was a plugin, but
it is no longer true as the built-in databases now ship inside the
ingest-geoip module that is bundled by default. This commit clarifies
those docs.
Co-authored-by: Jakob Reiter <jakommo@users.noreply.github.com>
Wrap a verification_exception in case there is no valid index available in an index_not_found_exception providing also the original index pattern that may be lost in the chain of filters involving the Security one.
(cherry picked from commit 9c9da2f2f9a4ad12704f7d3a273f067e96cd2054)
* [DOCS] Add top-level Data management section.
* Edits
* Edits
* Fixed xrefs
* Apply suggestions from code review
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
* Update docs/reference/datatiers.asciidoc
* Update docs/reference/datatiers.asciidoc
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
This commit clarifies that the preferred method for setting the heap
size is via jvm.options.d and that using the ES_JAVA_OPTS environment
variable is discouraged for production deployments.
Now that we're consistently using `cat_match` to filter which shards we
run on we can get this confusing case:
1. You have a search with, say, a range and a sub-agg.
2. That search has a query that `can_match` can recognize will match no
docs. On *any* shard.
3. So we dutifully run it on a single shard so it can produce the
"empty" aggs.
4. The shard we pick happens to not have the target of the range mapped.
5. This kicks in the special range aggregator that doesn't collect any
documents.
6. Before this commit, that range aggregator *also* never produced any
sub-aggs.
So, without this change, it was quite possible for a search that
happened to match no documents to "throw away" the sub-aggs of a range
and a few other aggs.
We've had this problem for a long, long time but it is more confusing
now because `can_match` is really kicking in and causing us to see cases
where it looks like you are targeting a lot of shards but you really are
only targeting a couple. It used to be that to get the "no sub-aggs"
behavior you had to explicitly target only shards that didn't map the
target field of the `range` agg. And, like, in that case it isn't too
bad because you targeted a sort of degenerate shard. But now that
`can_match` is doing its thing you can end up with the confusing steps
above. It took me several hours to track down what what happening I know
how the individual pieces of all of this works. It took four hours to
figure out how they fit together in this case....
Anyway! This replaces all the aggregator implementations that throw out
the sub-aggregators with ones that keep them. I think this'll be less
confusing in the future.
Closes#64142
This commit adds logging to indicate whether or not we are using the
bundled JDK. We distinguish between using a distribution that bundles
the JDK versus using a distribution that does not bundle the JDK.
In 7.x we can't just by default generate this setting as it might not be
supported by data nodes that are assigned shards for an older version in mixed version
clusters.
Closes#64152
The new fields option allows to fetch the value of all fields in the mapping.
However, internal fields that are used by some field mappers are also shown when
concrete fields retrieved through a pattern (`*` or `foo*`).
We have a [long term plan](https://github.com/elastic/elasticsearch/issues/63446) to hide these fields in field_caps and from pattern resolution
so this change is just a hot fix to ensure that they don't break the retrieval in the meantime.
The `flattened._keyed field will show up as an empty field when using a pattern that match the
flattened field.
Relates #63446
This commit fixes an issue with the detection on macOS for whether or
not the bundled JDK is being used. The logic between macOS and non-macOS
is different because the JDK has a different directory structure on
macOS versus non-macOS. However, due to notarization issues, we changed
the top-level directory from jdk to jdk.app, yet never updated this
detection logic to account for that.
Ideally, we would have a packaging test that asserts that we have the
behavior here correct, and it maintains over time. Alas, we do not
currently have packaging tests on macOS.
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.
To fully specifiy the behavior for the tiered data roles then:
- starting a new node with the defaults: enabled
- starting a new node with node.roles configured: enabled if and only
if the tiered data roles are explicitly configured, independently
of the node having the data role
- starting a new node with node.data enabled: enabled unless the
tiered data roles are explicitly disabled
- starting a new node with node.data disabled: disabled unless the
tiered data roles are explicitly enabled
James Rodewig
Andrei Dan
Marios Trivyzas
Armin Braun
William Brafford
Tal Levy
István Zoltán Szabó
Jason Tedor
Yang Wang
Ryan Ernst
Andrei Stefan
debadair
Mark Vieira
Nik Everett
jimczi
Jake Landis
Costin Leau