2020-01-29 11:35:02 -05:00
|
|
|
|
[[release-highlights-7.7.0]]
|
|
|
|
|
== 7.7.0 release highlights
|
|
|
|
|
++++
|
|
|
|
|
<titleabbrev>7.7.0</titleabbrev>
|
|
|
|
|
++++
|
|
|
|
|
|
|
|
|
|
//NOTE: The notable-highlights tagged regions are re-used in the
|
|
|
|
|
//Installation and Upgrade Guide
|
|
|
|
|
|
2020-04-17 07:37:37 -04:00
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== Fixed index corruption on shrunk indices
|
|
|
|
|
|
|
|
|
|
Applying deletes or updates on an index after it had been shrunk would likely
|
|
|
|
|
corrupt the index. We advise users of Elasticsearch 6.x who opt in for soft
|
|
|
|
|
deletes on some of their indices and all users of Elasticsearch 7.x to upgrade
|
|
|
|
|
to 7.7 as soon as possible to no longer be subject to this corruption bug. In
|
|
|
|
|
case upgrading in the near future is not an option, we recommend to completely
|
|
|
|
|
stop using `_shrink` on read-write indices and to do a force-merge right after
|
|
|
|
|
shrinking on read-only indices, which significantly reduces the likeliness of
|
|
|
|
|
being affected by this bug in case deletes or updates get applied by mistake.
|
|
|
|
|
This bug is fixed as of {es} 7.7.0. Low-level details can be found on the
|
|
|
|
|
https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
2020-05-13 14:08:14 -04:00
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== Significant reduction of heap usage of segments
|
|
|
|
|
|
|
|
|
|
This release of Elasticsearch significantly reduces the amount of heap memory
|
|
|
|
|
that is needed to keep Lucene segments open. In addition to helping with cluster
|
|
|
|
|
stability, this helps reduce costs by storing much more data per node before
|
|
|
|
|
hitting memory limits.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
2020-01-29 11:35:02 -05:00
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[discrete]
|
2020-04-23 05:58:32 -04:00
|
|
|
|
=== {transforms-cap} – now in GA!
|
2020-01-29 11:35:02 -05:00
|
|
|
|
|
2020-04-23 05:58:32 -04:00
|
|
|
|
In 7.7, we move {transforms} from beta to general availability.
|
|
|
|
|
|
|
|
|
|
{ref}/transforms.html[{transforms-cap}] enable you to pivot existing {es}
|
|
|
|
|
indices using group-by and aggregations into a destination feature index, which
|
|
|
|
|
provides opportunities for new insights and analytics. For example, you can use
|
|
|
|
|
{transforms} to pivot your data into entity-centric indices that summarize the
|
|
|
|
|
behavior of users or sessions or other entities in your data.
|
|
|
|
|
|
|
|
|
|
{transforms-cap} now include support for cross-cluster search. Allowing you to
|
|
|
|
|
create your destination feature index on a separate cluster from the source
|
|
|
|
|
indices.
|
|
|
|
|
|
|
|
|
|
Aggregation support has been expanded within {transforms} to include support for
|
|
|
|
|
{ref}/search-aggregations-metrics-percentile-aggregation.html[multi-value (percentiles)]
|
|
|
|
|
and
|
|
|
|
|
{ref}/search-aggregations-bucket-filter-aggregation.html[filter aggregations].
|
|
|
|
|
We also optimized the performance of the
|
|
|
|
|
{ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram aggregations].
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[discrete]
|
|
|
|
|
=== Introducing multiclass {classification}
|
|
|
|
|
|
|
|
|
|
{ml-docs}/dfa-classification.html[{classification-cap}] using multiple classes
|
|
|
|
|
is now available in {dfanalytics}. {classification-cap} is a supervised {ml}
|
|
|
|
|
technique which has been already available as a binary process in the previous
|
|
|
|
|
release. Multiclass {classification} works well with up to 30 distinct
|
|
|
|
|
categories.
|
2020-01-29 11:35:02 -05:00
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
2020-04-23 05:58:32 -04:00
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[discrete]
|
|
|
|
|
=== {feat-imp-cap} at {infer} time
|
|
|
|
|
|
|
|
|
|
{feat-imp-cap} now can be calculated at {infer} time. This value provides
|
|
|
|
|
further insight into the results of a {classification} or {regression} job and
|
|
|
|
|
therefore helps interpret these results.
|
|
|
|
|
|
2020-05-13 14:08:14 -04:00
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== Finer memory control for bucket aggregations
|
|
|
|
|
|
2020-06-01 14:58:20 -04:00
|
|
|
|
While building buckets, aggregations will now periodically check the
|
|
|
|
|
real-memory circuit breaker before continuing to allocate more buckets. This
|
|
|
|
|
allows better responsivity to memory pressure and avoids `OutOfMemory`
|
|
|
|
|
situations due to allocating more buckets than the node can handle.
|
2020-05-13 14:08:14 -04:00
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== A new way of searching: asynchronously
|
|
|
|
|
|
|
|
|
|
You can now submit {ref}/async-search-intro.html[long-running searches] using
|
|
|
|
|
the new {ref}/async-search.html[`_async_search` API]. The new API accepts the
|
|
|
|
|
same parameters and request body as the {ref}/search-search.html[Search API].
|
|
|
|
|
However, instead of blocking and returning the final response only when it's
|
|
|
|
|
entirely finished, you can retrieve results from an async search as they become
|
|
|
|
|
available.
|
|
|
|
|
|
|
|
|
|
The request takes a parameter, `wait_for_completion`, which controls how long
|
|
|
|
|
the server will wait until it sends back a response. The first response
|
|
|
|
|
contains among others a search unique ID, a response version, an indication if
|
|
|
|
|
this response is partial or not, plus the usual metadata (shards involved,
|
|
|
|
|
number of hits etc) and potentially results. If the response is not complete
|
|
|
|
|
and final, the client can continue polling for results, issuing a new request
|
|
|
|
|
using the provided search ID. If new results are available, the returned
|
|
|
|
|
version is incremented and the new batch of results are returned. This can
|
|
|
|
|
continue until all the results are fetched.
|
|
|
|
|
|
|
|
|
|
Unless deleted earlier by the user, the asynchronous searches are kept alive
|
|
|
|
|
for a given interval. This defaults to 5 days and can be controlled by another
|
|
|
|
|
request parameter, `keep_alive`.
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== Password protection for the keystore
|
|
|
|
|
|
|
|
|
|
{es} uses a custom on-disk {ref}/secure-settings.html[keystore] for secure settings such as
|
|
|
|
|
passwords and SSL certificates. Up until now, this prevented users with
|
|
|
|
|
{ref}/elasticsearch-keystore.html[command-line access] from viewing secure files by listing commands, but nothing
|
|
|
|
|
prevented such users from changing values in the keystore, or removing values
|
|
|
|
|
from it. Furthermore, the values were only obfuscated by a hash; no
|
|
|
|
|
user-specific secret protected the secure settings.
|
|
|
|
|
|
|
|
|
|
This new feature changes all of that by adding password-protection to the
|
|
|
|
|
keystore. This is not be a breaking change: if a keystore has no password,
|
|
|
|
|
there won’t be any new prompts. A user must choose to password-protect their
|
|
|
|
|
keystore in order to benefit from the new behavior.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== A new aggregation: `top_metrics`
|
|
|
|
|
|
|
|
|
|
The new {ref}//search-aggregations-metrics-top-metrics.html[`top_metrics` aggregation] "selects" a metric from a document according
|
|
|
|
|
to a criteria on a given, different field. That criteria is currently the
|
|
|
|
|
largest or smallest "sort" value. It is fairly similar to `top_hits` in spirit,
|
|
|
|
|
but because it is more limited, `top_metrics` uses less memory and
|
|
|
|
|
is often faster.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== Query speed-up for sorted queries on time-based indices
|
|
|
|
|
|
|
|
|
|
We've optimized sorted, top-documents-only queries run on time-based indices.
|
|
|
|
|
The optimization stems from the fact that the ranges of (document) timestamps
|
|
|
|
|
in the shards don't overlap. It is implemented by rewriting the shard search
|
|
|
|
|
requests based on the partial results already available from other shards, if
|
|
|
|
|
it can be determined that the query will not yield any result from the current
|
|
|
|
|
shard; i.e. we know in advance that the bottom entry of the (sorted) result set
|
|
|
|
|
after a partial merge is better than the values contained in this current
|
|
|
|
|
shard.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== A new aggregation: `boxplot`
|
|
|
|
|
|
|
|
|
|
The https://en.wikipedia.org/wiki/Interquartile_range[interquartile range (IQR)] is a common robust measure of statistical dispersion.
|
|
|
|
|
Compared to the standard deviation, the IQR is less sensitive to outliers in
|
|
|
|
|
the data, with a breakdown point of 0.25. Along with the median, it is often
|
|
|
|
|
used in creating a box plot, a simple yet common way to summarize data and
|
|
|
|
|
identify potential outliers.
|
|
|
|
|
|
|
|
|
|
The new {ref}/search-aggregations-metrics-boxplot-aggregation.html[`boxplot`
|
|
|
|
|
aggregation] calculates the min, max, and medium as well as the first and third
|
|
|
|
|
quartiles of a given data set.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|
|
|
|
|
|
|
|
|
|
// tag::notable-highlights[]
|
|
|
|
|
[float]
|
|
|
|
|
=== AArch64 support
|
|
|
|
|
|
|
|
|
|
{es} now provides AArch64 packaging, including bundling an AArch64 JDK
|
|
|
|
|
distribution. There are some restrictions in place, namely no {ml} support and
|
|
|
|
|
depending on underlying page sizes, class data sharing is disabled.
|
|
|
|
|
|
|
|
|
|
// end::notable-highlights[]
|