diff --git a/docs/reference/release-notes/highlights-7.0.0.asciidoc b/docs/reference/release-notes/highlights-7.0.0.asciidoc deleted file mode 100644 index 7165aa09ed1..00000000000 --- a/docs/reference/release-notes/highlights-7.0.0.asciidoc +++ /dev/null @@ -1,371 +0,0 @@ -[[release-highlights-7.0.0]] -== 7.0.0 release highlights -++++ -7.0.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -//tag::notable-highlights[] -[float] -==== Adaptive replica selection enabled by default - -In Elasticsearch 6.x and prior, a series of search requests to the same shard -would be forwarded to the primary and each replica in round-robin fashion. This -could prove problematic if one node starts a long garbage collection --- search -requests could still be forwarded to the slow node regardless and would have an -impact on search latency. - -In 6.1, we added an experimental -{ref}/search.html#search-adaptive-replica[adaptive replica selection] feature. -Each node tracks and compares how long search requests to -other nodes take, and uses this information to adjust how frequently to send -requests to shards on particular nodes. In our benchmarks, this results in an -overall improvement in search throughput and reduced 99th percentile latencies. - -This option was disabled by default throughout 6.x, but we’ve heard feedback -from our users that have found the setting to be very beneficial, so we’ve -turned it on by default starting in Elasticsearch 7.0.0. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Skip shard refreshes if a shard is "search idle" - -Elasticsearch 6.x and prior {ref}/indices-refresh.html[refreshed] indices -automatically in the background, by default every second. This provides the -“near real-time” search capabilities Elasticsearch is known for: results are -available for search requests within one second after they'd been added, by -default. However, this behavior has a significant impact on indexing performance -if the refreshes are not needed, (e.g., if Elasticsearch isn’t servicing any -active searches). - -Elasticsearch 7.0 is much smarter about this behavior by introducing the -notion of a shard being "search idle". A shard now transitions to being search -idle after it hasn't had any searches for -{ref}/index-modules.html#dynamic-index-settings[thirty seconds], by default. -Once a shard is search idle, all scheduled refreshes will -be skipped until a search comes through, which will trigger the next scheduled -refresh. We know that this is going to significantly increase the indexing -throughput for many users. The new behavior is only applied if there is no -explicit {ref}/index-modules.html#dynamic-index-settings[refresh interval set], -so do set the refresh -interval explicitly for any indices on which you prefer the old behavior. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Default to one shard - -One of the biggest sources of troubles we’ve seen over the years from our users -has been over-sharding and defaults play a big role in that. In Elasticsearch -6.x and prior, we defaulted to five shards by default per index. If you had one -daily index for ten different applications and each had the default of five -shards, you were creating fifty shards per day and it wasn't long before you had -thousands of shards even if you were only indexing a few gigabytes of data per -day. Index Lifecycle Management was a first step to help with this: providing -native rollover functions to create indexes by size instead of (just) by day and -built-in shrink functionality to shrink the number of shards per -index. Defaulting indices to one shard is the next step in helping to reduce -over-sharding. Of course, if you have another preferred primary shard count, you -can set it via the index settings. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Lucene 8 - -As with every major release, we look to support the latest major version of -Lucene, along with all the goodness that comes with it. That includes all the -developments that we contributed to the new Lucene version. Elasticsearch 7.0 -bundles Lucene 8, which is the latest version of Lucene. Lucene version 8 serves -as the foundation for many functional improvements in the rest of Elasticsearch, -including improved search performance for top-k queries and better ways to -combine relevance signals for your searches while still maintaining speed. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Introduce the ability to minimize round-trips in {ccs} - -In Elasticsearch 5.3, we released a feature called -{ref}/modules-cross-cluster-search.html[{ccs}] for users to query across multiple -clusters. We’ve since improved on the {ccs} framework, adding features to -ultimately use it to deprecate and replace tribe nodes as a way to federate -queries. In Elasticsearch 7.0, we’re adding a new execution mode for {ccs}: one -which has fewer round-trips when they aren't necessary. This mode -(`ccs_minimize_roundtrips`) can result in faster searches when the {ccs} query -spans high-latencies (e.g., across a WAN). -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== New cluster coordination implementation - -Since the beginning, we focused on making Elasticsearch easy to scale and -resilient to catastrophic failures. To support these requirements, we created a -pluggable cluster coordination system, with the default implementation known as -Zen Discovery. Zen Discovery was meant to be effortless, and give our users -peace of mind (as the name implies). The meteoric rise in Elasticsearch usage -has taught us a great deal. For instance, Zen's `minimum_master_nodes` setting -was often misconfigured, which put clusters at a greater risk of split brains -and losing data. Maintaining this setting across large and dynamically resizing -clusters was also difficult. - -In Elasticsearch 7.0, we have completely rethought and rebuilt the cluster -coordination layer. The new implementation gives safe sub-second master election -times, where Zen may have taken several seconds to elect a new master, valuable -time for a mission-critical deployment. With the `minimum_master_nodes` setting -removed, growing and shrinking clusters becomes safer and easier, and leaves -much less room to misconfigure the system. Most importantly, the new cluster -coordination layer gives us strong building blocks for the future of -Elasticsearch, ensuring we can build functionality for even more advanced -use-cases to come. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Better support for small heaps (the real-memory circuit breaker) - -Elasticsearch 7.0 adds an all-new {ref}/circuit-breaker.html[circuit breaker] -that keeps track of the total memory used by the JVM and will reject requests if -they would cause the reserved plus actual heap usage to exceed 95%. We'll also -be changing the default maximum buckets to return as part of an aggregation -(`search.max_buckets`) to 10,000, which is unbounded by default in 6.x and -prior. These two show great signs at seriously improving the out-of-memory -protection of Elasticsearch in 7.x, helping you keep your cluster alive even in -the face of adversarial or novice users running large queries and aggregations. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== {ccr-cap} is production-ready - -We introduced {ccr-cap} as a beta feature in Elasticsearch -6.5. {ccr-cap} was the most heavily requested features for Elasticsearch. We're -excited to announce {ccr-cap} is now generally available and ready for production use -in Elasticsearch 6.7 and 7.0! {ccr-cap} has a variety of use cases, including -cross-datacenter and cross-region replication, replicating data to get closer to -the application server and user, and maintaining a centralized reporting cluster -replicated from a large number of smaller clusters. - -In addition to maturing to a GA feature, there were a number of important -technical advancements in CCR for 6.7 and 7.0. Previous versions of {ccr-cap} required -replication to start on new indices only: existing indices could not be -replicated. {ccr-cap} can now start replicating existing indices that have soft -deletes enabled in 6.7 and 7.0, and new indices default to having soft deletes -enabled. We also introduced new technology to prevent a follower index from -falling fatally far behind its leader index. We’ve added a management UI in -Kibana for configuring remote clusters, indices to replicate, and index naming -patterns for automatic replication (e.g. for replicating `metricbeat-*` -indices). We've also added a monitoring UI for insight into {ccr} progress and -alerting on errors. Check out the Getting started with {ccr} -guide, or visit the reference documentation to learn more. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== {ilm-cap} is production-ready - -Index Lifecycle Management (ILM) was -https://www.elastic.co/blog/elastic-stack-6-6-0-released[released] as a beta -feature in Elasticsearch 6.6. We’ve officially moved ILM out of beta and into -GA, ready for production usage! ILM makes it easy to manage the lifecycle of -data in Elasticsearch, including how data progresses between -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/ilm-policy-definition.html[hot, warm, cold, and deletion phases]. -Specific rules regarding how data moves through these phases can be created via -APIs in Elasticsearch, or a beautiful management UI in Kibana. - -In Elasticsearch 6.7 and 7.0, ILM can now manage frozen indices. Frozen indices -are valuable for long term data storage in Elasticsearch, and require a smaller -amount of memory (heap) in relation to the amount of data managed by a node. In -6.7 and 7.0, -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/_actions.html[frozen indices] -can now be frozen as part of the cold phase in ILM. In addition, ILM now works -directly with Cross-Cluster Replication (CCR), which also GA’d in the -Elasticsearch 6.7 and 7.0 releases. The potential actions available in each ILM -phase can be found in the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/_actions.html[documentation]. -ILM is free to use and part of the default distribution of Elasticsearch. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== SQL is production-ready - -The SQL interface to Elasticsearch is now GA. -https://www.elastic.co/blog/elasticsearch-6-3-0-released[Introduced in 6.3] as -an alpha release, the SQL interface allows developers and data scientists -familiar with SQL to use the speed, scalability, and full-text power of -Elasticsearch that others know and love. It also allows BI tools using SQL to -easily access data in Elasticsearch. In addition to approving SQL access as a GA -feature in Elasticsearch, we’ve designated our -https://www.elastic.co/downloads/jdbc-client[JDBC] and -https://www.elastic.co/downloads/odbc-client[ODBC] drivers as GA. There are four -methods to access Elasticsearch SQL: through the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/sql-rest.html[Elasticsearch -REST endpoints], the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/sql-cli.html[Elasticsearch -SQL command line interface], the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/sql-jdbc.html[JDBC -driver], and the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/sql-odbc.html[ODBC -driver]. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== High-level REST client is feature-complete - -If you’ve been following our -https://www.elastic.co/blog/the-elasticsearch-java-high-level-rest-client-is-out[blog] -or our https://github.com/elastic/elasticsearch/issues/27205[GitHub repository], -you may be aware of a task we’ve been working on for quite a while now: creating -a next-generation Java client for accessing an Elasticsearch cluster. We -started off by working on the most commonly-used features like search and -aggregations, and have been working our way through administrative and -monitoring APIs. Many of you that use Java are already using this new client, -but for those that are still using the TransportClient, now is a great time to -upgrade to our High Level REST Client, or HLRC. - -As of 7.0.0, the HLRC now has all the API checkboxes checked to call it -“complete” so those of you still using the TransportClient should be able to -migrate. We’ll of course continue to develop our REST APIs and will add them to -this client as we go. For a list of all of the APIs that are available, have a -look at our -https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.0/java-rest-high.html[HLRC -documentation]. To get started, have a look at the -https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.0/java-rest-high-getting-started.html[getting -started with the HLRC] section of our docs and if you need help migrating from -the TransportClient, have a look at our -https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.0/java-rest-high-level-migration.html[migration -guide]. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Support nanosecond timestamps - -Up until 7.0 Elasticsearch could only store timestamps with millisecond -precision. If you wanted to process events that occur at a higher rate -- for -example if you want to store and analyze tracing or network packet data in -Elasticsearch -- you may want higher precision. Historically, we have used the -https://www.joda.org/joda-time/[Joda time library] to handle dates and times, -and Joda lacked support for such high precision timestamps. - -With JDK 8, an official Java time API has been introduced which can also handle -nanosecond precision timestamps and over the past year, we’ve been working to -migrate our Joda time usage to the native Java time while trying to maintain -backwards compatibility. As of 7.0.0, you can now make use of these nanosecond -timestamps via a dedicated -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/date_nanos.html[date_nanos -field mapper]. Note that aggregations are still on a millisecond resolution -with this field to avoid having an explosion of buckets. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Faster retrieval of top hits - -When it comes to search, query performance is a key feature. We have achieved a -significant improvement to search performance in Elasticsearch 7.0 for -situations in which the exact hit count is not needed and it is sufficient to -set a lower boundary to the number of results. For example, if your users -typically just look at the first page of results on your site and don’t care -about exactly how many documents matched, you may be able to show them “more -than 10,000 hits” and then provide them with paginated results. It’s quite -common to have users enter frequently-occurring terms like “the” and “a” in -their queries, which has historically forced Elasticsearch to score a lot of -documents even when those frequent terms couldn’t possibly add much to the -score. - -In these conditions Elasticsearch can now skip calculating scores for records -that are identified at an early stage as records that will not be ranked at the -top of the result set. This can significantly improve the query speed. The -actual number of top results that are scored is -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/search-request-track-total-hits.html[configurable], -but the default is 10,000. The behavior of queries that have a result set that -is smaller than this threshold will not change - i.e. the results count is -accurate but there is no performance improvement for queries that match a small -number of documents. Because the improvement is based on skipping low ranking -records, it does not apply to aggregations. You can read more about this -powerful algorithmic development in our blog post -https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand[Magic -WAND: Faster Retrieval of Top Hits in Elasticsearch]. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Support for TLS 1.3 - -Elasticsearch has supported encrypted communications for a long time, however, -we recently started https://www.elastic.co/support/matrix#matrix_jvm[supporting -JDK 11], which gives us new capabilities. JDK 11 now has TLSv1.3 support so -starting with 7.0, we’re now supporting TLSv1.3 within Elasticsearch for those -of you running JDK 11. In order to help new users from inadvertently running -with low security, we’ve also dropped TLSv1.0 from our defaults. For those -running older versions of Java, we have default options of TLSv1.2 and -TLSv1.1. Have a look at our -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/ssl-tls.html[TLS -setup instructions] if you need help getting started. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Bundle JDK in Elasticsearch distribution - -One of the more prominent "getting started hurdles" we’ve seen users run into -has been not knowing that Elasticsearch is a Java application and that they need -to install one of the supported JDKs first. With 7.0, we’re now bundling a -distribution of OpenJDK to help users get started with Elasticsearch even -faster. We understand that some users have preferred JDK distributions, so we -also support bringing your own JDK. If you want to bring your own JDK, you can -still do so by -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/setup.html#jvm-version[setting -JAVA_HOME] before starting Elasticsearch. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Rank features - -Elasticsearch 7.0 has several new field types to get the most out of your data. -Two to help with core search use cases are -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/rank-feature.html[`rank_feature`] -and -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/rank-features.html[`rank_features`]. -These can be used to boost documents based on numeric or categorical values -while still maintaining the performance of the new fast top hits query -capabilities. For more information on these fields and how to use them, read our -https://www.elastic.co/blog/easier-relevance-tuning-elasticsearch-7-0[blog -post]. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== JSON logging - -JSON logging is now enabled in Elasticsearch in addition to plaintext -logs. Starting in 7.0, you will find new files with `.json` extensions in your -log directory. This means you can now use filtering tools like -https://stedolan.github.io/jq/[`jq`] to pretty print and process your logs in a -much more structured manner. You can also expect finding additional information -like `node.id`, `cluster.uuid`, `type` (and more) in each log line. The `type` -field per each JSON log line will let you to distinguish log streams when -running on docker. -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -=== Script score query (aka function score 2.0) - -With 7.0, we are introducing the -https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-script-score-query.html[next -generation of our function score capability]. This new script_score query -provides a new, simpler, and more flexible way to generate a ranking score per -record. The script_score query is constructed of a set of functions, including -arithmetic and distance functions, which the user can mix and match to construct -arbitrary function score calculations. The modular structure is simpler to use -and will open this important functionality to additional users. -//end::notable-highlights[] diff --git a/docs/reference/release-notes/highlights-7.1.0.asciidoc b/docs/reference/release-notes/highlights-7.1.0.asciidoc deleted file mode 100644 index f4b27d57e98..00000000000 --- a/docs/reference/release-notes/highlights-7.1.0.asciidoc +++ /dev/null @@ -1,40 +0,0 @@ -[[release-highlights-7.1.0]] -== 7.1.0 release highlights -++++ -7.1.0 -++++ - -See also <>. - -//tag::notable-highlights[] -[float] -==== TLS is now licensed under the Elastic Basic license - -Transport Layer Security (TLS), commonly referred to as SSL, is now -licensed under the free-of-charge Elastic Basic license. Previously, this security feature -required a paid Gold-tier subscription. With the default distribution, -you can now encrypt all Elasticsearch communication, within a cluster and across remotes -clusters. Download https://www.elastic.co/downloads/elasticsearch[Elasticsearch], -https://www.elastic.co/guide/en/elasticsearch/reference/7.1/configuring-tls.html[configure TLS], -and run your cluster in production, knowing all Elasticsearch communication is safely encrypted. -For details, see https://www.elastic.co/subscriptions -//end::notable-highlights[] - -//tag::notable-highlights[] -[float] -==== RBAC is now licensed under the Elastic Basic license - -RBAC (Role Based Access Control) is now licenced under the free-of-charge Elastic Basic licence. -Previously, this security feature required a paid Gold-tier subscription. -With the default distribution you can take advantage of RBAC by configuring users, groups, roles -and permissions for any user from the -https://www.elastic.co/guide/en/elasticsearch/reference/7.1/configuring-file-realm.html[file realm] -or the https://www.elastic.co/guide/en/elasticsearch/reference/7.1/configuring-native-realm.html[native realm] -. Download https://www.elastic.co/downloads/elasticsearch[Elasticsearch], -https://www.elastic.co/guide/en/elasticsearch/reference/7.1/authorization.html[configure RBAC], -and run your cluster in production, knowing your private data stays private. -Note that our advanced security features, such as single sign-on and Active Directory/LDAP -authentication to field-level and document-level security, remain paid features. -For details, see https://www.elastic.co/subscriptions - -//end::notable-highlights[] diff --git a/docs/reference/release-notes/highlights-7.2.0.asciidoc b/docs/reference/release-notes/highlights-7.2.0.asciidoc deleted file mode 100644 index ea98c34d847..00000000000 --- a/docs/reference/release-notes/highlights-7.2.0.asciidoc +++ /dev/null @@ -1,101 +0,0 @@ -[[release-highlights-7.2.0]] -== 7.2.0 release highlights -++++ -7.2.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[discrete] -==== {dataframes-cap} - -beta[] You can now transform your data with -{ref}/transforms.html[data frames]. There is a new {kib} wizard that -guides you through the process of creating a {dataframe-transform} to pivot and -summarize your data and store it in a new index. Alternatively, you can use -{ref}/data-frame-apis.html[{dataframe} APIs] to preview, create, and manage -the transforms. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Closed indices are now replicated - -Elasticsearch 7.2.0 brings better support for closed indices by allowing -shards of closed indices to be replicated. -As soon as an index is closed, Elasticsearch takes care of safely tearing down -the "opened" shards before reinitializing them as "closed" shards, which require -fewer resources. Closed shards can later be promoted to primary shards or -automatically recovered during {ref}/recovery.html[peer recovery] -The data is also automatically replicated by the -cluster to ensure that enough shard copies are safely kept around at all -times (configurable with `index.number_of_replicas`). - -In addition to that, it is now possible to snapshot closed indices using -the {ref}/modules-snapshots.html[Snapshot/Restore API]. To include a closed index -when creating a snapshot on Elasticsearch 7.2+, the `expand_wildcards` -parameter must be explicitly set to either `all` or `closed` . - -Note that only indices closed in Elasticsearch 7.2+ are automatically -replicated. Indices closed on previous versions of Elasticsearch will -remain non replicated unless they are opened and closed again in 7.2+. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Geo features in SQL -beta[] This release introduces the first set of geo features to SQL. -The implementation is based on the OpenGIS® Implementation Standard for Geographic -information - "Simple feature access". This is the current de-facto standard for GIS -system implementation. This release includes a small subset of SQL option AKA ISO 19125-2. - -For this initial beta release, we added support for returning -geo_shapes and geo_points as results, added support for a few geo functions -(ST_AsText, ST_Distance, ST_GeometryType, ST_GeometryFromText, ST_X, ST_Y, and ST_Z) -, and added a limited support for using geo_points in distance queries. For example: -SELECT * FROM my_index WHERE ST_Distance(point, ST_WKTToSQL('point (10 20)')) < 20. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== OpenId Connect authentication realm - -This release introduces OpenId Connect as an authentication realm. -Elasticsearch (with the assistance of Kibana or another web component) can now serve as an -OpenID Connect Relying Party (RP). {es} supports the Authorization Code Grant and Implicit -flows as described in http://ela.st/oidc-spec. It also supports consuming and verifying signed ID Tokens -, RP initiated single sign on (SSO), 3rd party initiated SSO, and RP initiated single logout. - - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Search as you type field mapping type - -The `search_as_you_type` field type is a text-like field optimized to -provide out-of-the-box support for queries that serve an as-you-type completion -use case. It creates a series of subfields that are analyzed to index terms -that can be efficiently matched by a query that partially matches the entire -indexed text value. Both prefix completion (i.e matching terms starting at the -beginning of the input) and infix completion (i.e. matching terms at any -position within the input) are supported. - -// end::notable-highlights[] - - -// tag::notable-highlights[] -[float] -==== Distance Feature Query - -The `distance_feature` query is a specialized query that only works on `date`, `date_nanos`, or `geo_point` -fields. The query boosts documents scores based on proximity to some given origin. -For example, you can use this query to give higher scores to documents with dates -closer to a certain date or locations closer to a certain location. - -// end::notable-highlights[] diff --git a/docs/reference/release-notes/highlights-7.3.0.asciidoc b/docs/reference/release-notes/highlights-7.3.0.asciidoc deleted file mode 100644 index 0f69d828125..00000000000 --- a/docs/reference/release-notes/highlights-7.3.0.asciidoc +++ /dev/null @@ -1,186 +0,0 @@ -[[release-highlights-7.3.0]] -== 7.3.0 release highlights -++++ -7.3.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -==== Voting-only master nodes - -A new {ref}/modules-node.html#voting-only-node[`node.voting-only`] role has been -introduced that allows nodes to participate in elections even though they are -not eligible to become the master. -The benefit is that these nodes still help with high availability while -requiring less CPU and heap than master nodes. - -NOTE: The `node.voting-only` role is only available with the default -distribution of {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Reloading of search-time synonyms - -A new {ref}/indices-reload-analyzers.html[Analyzer reload API] allows to reload -the definition of search-time analyzers and their associated resources. A common -use-case for this API is the reloading of search-time synonyms. In earlier -versions of Elasticsearch, users could force synonyms to be reloaded by closing -the index and then opening it again. With this new API, synonyms can be updated -without closing the index. - -NOTE: The Analyzer reload API is only available with the default distribution -of {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== New `flattened` field type - -A new {ref}/flattened.html[`flattened`] field type has been added, which can index -arbitrary `json` objects into a single field. This helps avoid hitting issues -due to many fields in mappings, at the cost of more limited search -functionality. - -NOTE: The `flattened` field type is only available with the -default distribution of {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Functions on vector fields - -Painless now support computing the -{ref}/query-dsl-script-score-query.html#vector-functions[cosine similarity] and -the {ref}/query-dsl-script-score-query.html#vector-functions[dot product] of a -query vector and either values of a -{ref}/sparse-vector.html[`sparse_vector`] or -{ref}/dense-vector.html[`dense_vector`] field. - -NOTE: These functions are only available with the default distribution of {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Prefix and wildcard support for intervals - -{ref}/query-dsl-intervals-query.html[Intervals] now support querying by -{ref}/query-dsl-intervals-query.html#intervals-prefix[prefix] or -{ref}/query-dsl-intervals-query.html#intervals-wildcard[wildcard]. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Rare terms aggregation - -A new -{ref}/search-aggregations-bucket-rare-terms-aggregation.html[Rare Terms aggregation] -allows to find the least frequent values in a field. It is intended to replace -the `"order" : { "_count" : "asc" }` option of the -{ref}/search-aggregations-bucket-terms-aggregation.html[terms aggregations]. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Aliases are replicated via {ccr} - -Read aliases are now replicated via {ref}/ccr-put-follow.html[{ccr}]. Note that -write aliases are still not replicated since they only make sense for indices that -are being written to while follower indices do not receive direct writes. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== SQL supports frozen indices - -{es-sql} now supports querying {ref}/frozen-indices.html[frozen indices] via the -new {ref}/sql-index-frozen.html[`FROZEN`] keyword. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Fixed memory leak when using templates in document-level security - -{ref}/document-level-security.html[Document-level security] was using an -unbounded cache for the set of visible documents. This could lead to a memory -leak when using a templated query as a role query. The cache has been fixed to -evict based on memory usage and has a limit of 50MB. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== More memory-efficient aggregations on `keyword` fields - -{ref}/search-aggregations-bucket-terms-aggregation.html[Terms aggregations] -generally need to build -{ref}/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-execution-hint[global ordinals] -in order to run. Unfortunately this operation became more memory-intensive in -6.0 due to the move to doc-value iterators in order to improve handling of -sparse fields. Memory pressure of global ordinals now goes back to a more -similar level as what you could have on pre-6.0 releases. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[discrete] -[[release-highlights-7.3.0-transforms]] -==== {dataframes-cap}: transform and pivot your streaming data - -beta[] {ref}/transforms.html[{dataframe-transforms-cap}] are a core new -feature in {es} that enable you to transform an existing index to a secondary, -summarized index. {dataframe-transforms-cap} enable you to pivot your data and -create entity-centric indices that can summarize the behavior of an entity. This -organizes the data into an analysis-friendly format. - -{dataframe-transforms-cap} were originally available in 7.2. With 7.3 they can -now run either as a single batch transform or continuously incorporating new -data as it is ingested. - -{dataframes-cap} enable new possibilities for {ml} analysis (such as -_outlier detection_), but they can also be useful for other types of -visualizations and custom types of analysis. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[discrete] -[[release-highlights-7.3.0-outlier-detection]] -==== Discover your most unusual data using {oldetection} - -The goal of {ml-docs}/dfa-outlier-detection.html[{oldetection}] is to find -the most unusual data points in an index. We analyse the numerical fields of -each data point (document in an index) and annotate them with how unusual they -are. - -We use unsupervised {oldetection} which means there is no need to provide a -training data set to teach {oldetection} to recognize outliers. In practice, -this is achieved by using an ensemble of distance based and density based -techniques to identify those data points which are the most different from the -bulk of the data in the index. We assign to each analysed data point an -{olscore}, which captures how different the entity is from other entities in the -index. - -In addition to new {oldetection} functionality, we are introducing the -{ref}/evaluate-dfanalytics.html[evaluate {dfanalytics} API], which enables you -to compute a range of performance metrics such -as confusion matrices, precision, recall, the -https://en.wikipedia.org/wiki/Receiver_operating_characteristic[receiver-operating characteristics (ROC) curve] -and the area under the ROC curve. If you are running {oldetection} on a source -index that has already been labeled to indicate which points are truly outliers -and which are normal, you can use the -evaluate {dfanalytics} API to assess the performance of the -{oldetection} analytics on your dataset. - -// end::notable-highlights[] diff --git a/docs/reference/release-notes/highlights-7.4.0.asciidoc b/docs/reference/release-notes/highlights-7.4.0.asciidoc deleted file mode 100644 index f26c73811c9..00000000000 --- a/docs/reference/release-notes/highlights-7.4.0.asciidoc +++ /dev/null @@ -1,156 +0,0 @@ -[[release-highlights-7.4.0]] -== 7.4.0 release highlights -++++ -7.4.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -==== Results pinning - -You can use the new {ref}/query-dsl-pinned-query.html[pinned query] -to define the first records -(and the order in which they are returned) -in a result set directly within {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== New `shape` field type - -A new {ref}/shape.html[`shape`] field type has been added, -which allows you to position and query shapes -in a geometry of your choosing. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Circle ingest processor - -A new {ref}/ingest-circle-processor.html[circle ingest processor] has been added, -which translates circles into regular polygons (bounded by the circles). -This makes ingesting, indexing, searching, and aggregating circles both easy and efficient. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Aggregations on range fields - -The {ref}/search-aggregations-bucket-histogram-aggregation.html[histogram] -and {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram] -aggregations now support the {ref}/range.html[`range`] field type. - -Range aggregations are useful -when counting ranges that overlap with specific buckets -(e.g. the number of phone calls that took place during a specific minute). - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Cumulative cardinality aggregation - -A new {ref}/search-aggregations-pipeline-cumulative-cardinality-aggregation.html[cumulative cardinality aggregation] -has been added -as part of our ongoing effort to provide advanced aggregations. - -You can use this new pipeline aggregation -to calculate a net-new total of document occurrences -within a given time range. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Snapshot lifecycle management - -We’re introducing {ref}/getting-started-snapshot-lifecycle-management.html[snapshot lifecycle management (SLM)], -which allows an administrator to define policies, -via API or {kibana-ref}/index-lifecycle-policies.html[{kib} UI], -that manage when and how often snapshots are taken. -You can use SLM -to ensure that appropriate, recent backups are ready -if disaster strikes -or you need to restore {es} data. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== API key management - -New {ref}/security-privileges.html[cluster privileges] to manage API keys have been added, -allowing cluster administrators to manage everything, -and regular users to manage their own keys. -Users can create API keys -and use them to provide long-term credentials -while interacting with {es}. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== TLS settings for email notifications - -Notifications may contain sensitive information that must be protected over the wire. This requires that communication with the mail server is encrypted and authenticated properly. -{es} now supports custom {ref}/notification-settings.html#ssl-notification-smtp-settings[TLS settings] for email notifications, -allowing secure connections to servers with custom security configuration. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Automatic query cancellation - -{es} now automatically terminates queries -sent through the `_search` endpoint -when the initiating connection is closed. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Support for AdoptOpenJDK - -AdoptOpenJDK 13 is now supported and shipped with {es} as the pre-bundled JDK. - -If you want to use your own JDK, -you can still do so by setting `JAVA_HOME` before starting Elasticsearch. - -The availability of a notarized AdoptOpenJDK package -(per the new requirements for software running on macOS Catalina) -facilitates notarization of {es} for continued support on macOS. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Regression analysis - Experimental - -{ml-docs}/dfa-regression.html[Regression analysis] is an experimental machine learning process -for estimating the relationships among a number of feature variables and a dependent variable, -then making further predictions based on the described relationship. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== New vector distance functions for document script scoring - Experimental - -Two experimential similarity measurements— -Manhattan distance (L1 norm) -and Euclidean distance (L2 norm)— -have been added. -Like the dot product and cosine similarity, -the Euclidean and Manhattan distances are provided as {ref}/query-dsl-script-score-query.html#vector-functions[predefined Painless functions] -so that they may be incorporated with other query elements -as part of a {ref}/query-dsl-script-score-query.html[script_score] query. - -// end::notable-highlights[] - diff --git a/docs/reference/release-notes/highlights-7.5.0.asciidoc b/docs/reference/release-notes/highlights-7.5.0.asciidoc deleted file mode 100644 index 974a088d2fb..00000000000 --- a/docs/reference/release-notes/highlights-7.5.0.asciidoc +++ /dev/null @@ -1,64 +0,0 @@ -[[release-highlights-7.5.0]] -== 7.5.0 release highlights -++++ -7.5.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -==== Enrich processor - -A new {ref}/enrich-processor.html[enrich ingest processor] has been added, -which can enrich documents with data from another index. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Shape support in SQL - -{ref}/xpack-sql.html[SQL] functionality that worked for geo_shape will -now work for the {ref}/shape.html[`shape`] field type introduced in 7.4. - - -// end::notable-highlights[] - - -// tag::notable-highlights[] -[float] -==== Snapshot lifecycle management retention - -There is a new {ref}/slm-retention.html[snapshot lifecycle management retention], -which allows you to delete older snapshots automatically through a custom -policy’s schedule. - -// end::notable-highlights[] - - -// tag::notable-highlights[] -[float] -==== Pause {ccr} - -New {ref}/ccr-post-pause-follow.html[pause] and -{ref}/ccr-post-pause-follow.html[resume] API endpoints for -{ref}/xpack-ccr.html[{ccr}] have been added, which enable you to temporarily -pause auto-follow patterns. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== {ml-cap} {classanalysis} - -{ml-docs}/dfa-classification.html[{classanalysis-cap}] is a supervised {ml} -process for predicting a class or category of a given data point in a dataset. -For example, it can determine whether an email is spam or not. -{classification-cap} is for predicting discrete, categorical values, unlike -{reganalysis}, which predicts continuous, numerical values. 7.5.0 introduces -binary classification, which can label data points into two possible categories. - -// end::notable-highlights[] - diff --git a/docs/reference/release-notes/highlights-7.6.0.asciidoc b/docs/reference/release-notes/highlights-7.6.0.asciidoc deleted file mode 100644 index 3a38256c8b9..00000000000 --- a/docs/reference/release-notes/highlights-7.6.0.asciidoc +++ /dev/null @@ -1,72 +0,0 @@ -[[release-highlights-7.6.0]] -== 7.6.0 release highlights -++++ -7.6.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -==== New histogram field type - -A new {ref}/histogram.html[histogram field type] has been added. The new `histogram` field accepts -pre-aggregated histograms which can later be used directly in the -{ref}/search-aggregations-metrics-percentile-aggregation.html[percentiles] and -{ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile_ranks] aggregations. -This allows users to pre-aggregate histogram data locally and only send the final -data structure, saving storage and network bandwidth while retaining the ability to -aggregate it like any other data. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Optimized sorting on long field types - -Sorting on a {ref}/number.html[`long`] field now internally rewrites into a Lucene `DistanceFeatureQuery`. -This lets {es} skip non-competitive hits, which often improves query speed. -In benchmarking tests, this sped up sorts on `long` fields by 10x. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== Simplifying and operationalizing machine learning - -With the release of 7.6 the {stack} delivers an end-to-end {ml} pipeline -providing the path from raw data to building, testing, and deploying {ml} models -in production. Up to this point {ml} in the {stack} had primarily focused on -unsupervised techniques by using sophisticated pattern recognition that builds -time series models used for {anomaly-detect}. With the new {dfanalytics}, you -can now use labelled data to train and test your own models, store those models -as {es} indices, and use {ml-docs}/ml-inference.html[inference] to add predicted -values to the indices based on your trained models. - -One packaged model that we are releasing in 7.6 is -{ml-docs}/ml-lang-ident.html[{lang-ident}]. If you have documents or sources -that come in a variety of languages, {lang-ident} can be used to determine the -language of text so you can improve the overall search relevance. -{lang-ident-cap} is a trained model that can provide a prediction of the -language of any text field. -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -==== {ccs-cap} in {transforms} - -{ref}/transforms.html[{transforms-cap}] can now use {ccs} (CCS) for the source -index. Now you can have separate clusters (for example, project clusters) build -entity-centric or feature indices against a primary cluster. - -// end::notable-highlights[] - -[float] -=== Learn more - -Get more details on these features in the -https://www.elastic.co/blog/elasticsearch-7-6-0-released[{es} 7.6 release blog]. -For a complete list of enhancements and other changes, check out the -<>. - diff --git a/docs/reference/release-notes/highlights-7.7.0.asciidoc b/docs/reference/release-notes/highlights-7.7.0.asciidoc deleted file mode 100644 index a1b8f835782..00000000000 --- a/docs/reference/release-notes/highlights-7.7.0.asciidoc +++ /dev/null @@ -1,191 +0,0 @@ -[[release-highlights-7.7.0]] -== 7.7.0 release highlights -++++ -7.7.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -=== Fixed index corruption on shrunk indices - -Applying deletes or updates on an index after it had been shrunk would likely -corrupt the index. We advise users of Elasticsearch 6.x who opt in for soft -deletes on some of their indices and all users of Elasticsearch 7.x to upgrade -to 7.7 as soon as possible to no longer be subject to this corruption bug. In -case upgrading in the near future is not an option, we recommend to completely -stop using `_shrink` on read-write indices and to do a force-merge right after -shrinking on read-only indices, which significantly reduces the likeliness of -being affected by this bug in case deletes or updates get applied by mistake. -This bug is fixed as of {es} 7.7.0. Low-level details can be found on the -https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue]. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Significant reduction of heap usage of segments - -This release of Elasticsearch significantly reduces the amount of heap memory -that is needed to keep Lucene segments open. In addition to helping with cluster -stability, this helps reduce costs by storing much more data per node before -hitting memory limits. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[discrete] -=== {transforms-cap} – now in GA! - -In 7.7, we move {transforms} from beta to general availability. - -{ref}/transforms.html[{transforms-cap}] enable you to pivot existing {es} -indices using group-by and aggregations into a destination feature index, which -provides opportunities for new insights and analytics. For example, you can use -{transforms} to pivot your data into entity-centric indices that summarize the -behavior of users or sessions or other entities in your data. - -{transforms-cap} now include support for cross-cluster search. Allowing you to -create your destination feature index on a separate cluster from the source -indices. - -Aggregation support has been expanded within {transforms} to include support for -{ref}/search-aggregations-metrics-percentile-aggregation.html[multi-value (percentiles)] -and -{ref}/search-aggregations-bucket-filter-aggregation.html[filter aggregations]. -We also optimized the performance of the -{ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram aggregations]. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[discrete] -=== Introducing multiclass {classification} - -{ml-docs}/dfa-classification.html[{classification-cap}] using multiple classes -is now available in {dfanalytics}. {classification-cap} is a supervised {ml} -technique which has been already available as a binary process in the previous -release. Multiclass {classification} works well with up to 30 distinct -categories. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[discrete] -=== {feat-imp-cap} at {infer} time - -{feat-imp-cap} now can be calculated at {infer} time. This value provides -further insight into the results of a {classification} or {regression} job and -therefore helps interpret these results. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Finer memory control for bucket aggregations - -While building buckets, aggregations will now periodically check the -real-memory circuit breaker before continuing to allocate more buckets. This -allows better responsivity to memory pressure and avoids `OutOfMemory` -situations due to allocating more buckets than the node can handle. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== A new way of searching: asynchronously - -You can now submit {ref}/async-search-intro.html[long-running searches] using -the new {ref}/async-search.html[`_async_search` API]. The new API accepts the -same parameters and request body as the {ref}/search-search.html[Search API]. -However, instead of blocking and returning the final response only when it's -entirely finished, you can retrieve results from an async search as they become -available. - -The request takes a parameter, `wait_for_completion`, which controls how long -the server will wait until it sends back a response. The first response -contains among others a search unique ID, a response version, an indication if -this response is partial or not, plus the usual metadata (shards involved, -number of hits etc) and potentially results. If the response is not complete -and final, the client can continue polling for results, issuing a new request -using the provided search ID. If new results are available, the returned -version is incremented and the new batch of results are returned. This can -continue until all the results are fetched. - -Unless deleted earlier by the user, the asynchronous searches are kept alive -for a given interval. This defaults to 5 days and can be controlled by another -request parameter, `keep_alive`. -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Password protection for the keystore - -{es} uses a custom on-disk {ref}/secure-settings.html[keystore] for secure settings such as -passwords and SSL certificates. Up until now, this prevented users with -{ref}/elasticsearch-keystore.html[command-line access] from viewing secure files by listing commands, but nothing -prevented such users from changing values in the keystore, or removing values -from it. Furthermore, the values were only obfuscated by a hash; no -user-specific secret protected the secure settings. - -This new feature changes all of that by adding password-protection to the -keystore. This is not be a breaking change: if a keystore has no password, -there won’t be any new prompts. A user must choose to password-protect their -keystore in order to benefit from the new behavior. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== A new aggregation: `top_metrics` - -The new {ref}//search-aggregations-metrics-top-metrics.html[`top_metrics` aggregation] "selects" a metric from a document according -to a criteria on a given, different field. That criteria is currently the -largest or smallest "sort" value. It is fairly similar to `top_hits` in spirit, -but because it is more limited, `top_metrics` uses less memory and -is often faster. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Query speed-up for sorted queries on time-based indices - -We've optimized sorted, top-documents-only queries run on time-based indices. -The optimization stems from the fact that the ranges of (document) timestamps -in the shards don't overlap. It is implemented by rewriting the shard search -requests based on the partial results already available from other shards, if -it can be determined that the query will not yield any result from the current -shard; i.e. we know in advance that the bottom entry of the (sorted) result set -after a partial merge is better than the values contained in this current -shard. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== A new aggregation: `boxplot` - -The https://en.wikipedia.org/wiki/Interquartile_range[interquartile range (IQR)] is a common robust measure of statistical dispersion. -Compared to the standard deviation, the IQR is less sensitive to outliers in -the data, with a breakdown point of 0.25. Along with the median, it is often -used in creating a box plot, a simple yet common way to summarize data and -identify potential outliers. - -The new {ref}/search-aggregations-metrics-boxplot-aggregation.html[`boxplot` -aggregation] calculates the min, max, and medium as well as the first and third -quartiles of a given data set. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== AArch64 support - -{es} now provides AArch64 packaging, including bundling an AArch64 JDK -distribution. There are some restrictions in place, namely no {ml} support and -depending on underlying page sizes, class data sharing is disabled. - -// end::notable-highlights[] diff --git a/docs/reference/release-notes/highlights-7.8.0.asciidoc b/docs/reference/release-notes/highlights-7.8.0.asciidoc deleted file mode 100644 index 7cb76785b8f..00000000000 --- a/docs/reference/release-notes/highlights-7.8.0.asciidoc +++ /dev/null @@ -1,167 +0,0 @@ -[[release-highlights-7.8.0]] -== 7.8.0 release highlights -++++ -7.8.0 -++++ - -//NOTE: The notable-highlights tagged regions are re-used in the -//Installation and Upgrade Guide - -// tag::notable-highlights[] -[float] -=== Geo improvements - -We have made several improvements to geo support in {es} 7.8. - -- You can now run an aggregation that finds the bounding box (top left point and -bottom right point) that contains all shapes matching a query. A shape is -anything that is defined by multiple points. See -{ref}/search-aggregations-metrics-geobounds-aggregation.html[Geo Bounds Aggregations]. -- {ref}/search-aggregations-bucket-geohashgrid-aggregation.html[GeoHash grid aggregations] -and {ref}/search-aggregations-bucket-geotilegrid-aggregation.html[map tile grid aggregations] -allow you to group geo_points into buckets. -- {ref}/search-aggregations-metrics-geocentroid-aggregation.html[Geo centroid aggregations] -allow you to compute the weighted https://en.wikipedia.org/wiki/Centroid[centroid] -from all coordinate values for a geo_point field. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Add support for t-test aggregations - -{es} now supports a `t_test` metrics -aggregation, which performs a statistical hypothesis test in which the test -statistic follows a -https://en.wikipedia.org/wiki/Student%27s_t-distribution[Student’s -t-distribution] under the null hypothesis on numeric values extracted from -the aggregated documents or generated by provided scripts. In practice, -this will tell you if the difference between two population means are -statistically significant and did not occur by chance alone. See -{ref}/search-aggregations-metrics-ttest-aggregation.html[T-Test Aggregation]. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Expose aggregation usage in feature usage API - -It is now possible to fetch a count of aggregations that have been executed -via the {ref}/cluster-nodes-usage.html[node features API]. This is broken down per -combination of aggregation and data type, per shard on each node, from the -last restart until the time when the counts are fetched. When trying to -analyze how {es} is being used in practice, it is useful to know -the usage distribution across aggregations and field types. For example, -you might be able to conclude that a certain part of an index is not used a -lot and could perhaps can be eliminated. - - -// end::notable-highlights[] - - -// tag::notable-highlights[] -[float] -=== Support `value_count` and `avg` aggregations over histogram fields - -{es} now implements `value_count` and `avg` aggregations over histogram -fields. - -When the `value_count` aggregation is computed on {ref}/histogram.html[histogram -fields], the result of the aggregation is the sum of all numbers in the -`counts` array of the histogram. - -When the average is computed on histogram fields, the result of the -aggregation is the weighted average of all elements in the `values` array -taking into consideration the number in the same position in the `counts` -array. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Reduce aggregation memory consumption - -{es} now attempts to save memory on the coordinating node by delaying -deserialization of the shard results for an aggregation until the last -second. This is helpful as it makes the shard-aggregations results "short -lived" garbage. It also should shrink the memory usage of aggregations when -they are waiting to be merged. - -Additionally, when the search is in batched reduce mode, {es} will force -the results to be serialized between batch reduces in an attempt to keep -the memory usage as low as possible between reductions. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -=== Scalar functions now supported in SQL aggregations - -When querying {es} using SQL, it is now possible to use scalar functions -inside aggregations. This allows for more complex expressions, including -within `GROUP BY` or `HAVING` clauses. For example: - -[source, sql] ----- -SELECT - MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) AS max, - b -FROM test -GROUP BY b -HAVING - MAX(CASE WHEN a IS NULL then -1 ELSE abs(a * 10) + 1 END) > 5 ----- - -// end::notable-highlights[] -// tag::notable-highlights[] -[float] -[[release-highlights-7.8.0-throttling]] -=== Increase the performance and scalability of {transforms} with throttling - -{transforms-cap} achieved GA status in 7.7 and now in 7.8 they are even better -with the introduction of -{ref}/transform-overview.html#transform-performance[throttling]. You can spread -out the impact of the {transforms} on your cluster by defining the rate at which -they perform search and index requests. Set the `docs_per_second` limit when you -create or update your {transform}. - -// end::notable-highlights[] -// tag::notable-highlights[] -[float] -[[release-highlights-7.8.0-mml]] -=== Better estimates for {ml} model memory usage - -For 7.8, we introduce dynamic estimation of the model memory limit for jobs in -{ml-docs}/ootb-ml-jobs.html[ML solution modules]. The estimate is generated -during the job creation. It uses a calculation based on the specific detectors -of the job and the cardinality of the partitioning and influencer fields. It -means the job setup has better default values depending on the size of the data -being analyzed. - -// end::notable-highlights[] -// tag::notable-highlights[] -[float] -[[release-highlights-7.8.0-loss-functions]] -=== Additional loss functions for {regression} - -{ml-docs}/dfa-regression.html#dfa-regression-lossfunction[Loss functions] -measure how well a {ml} model fits a specific data set. In 7.8, we added two new -loss functions for {regression} analysis. In addition to the existing mean -squared error function, there are now mean squared logarithmic error and -Pseudo-Huber loss functions. These additions enable you to choose the -loss function that fits best with your data set. - -// end::notable-highlights[] - -// tag::notable-highlights[] -[float] -[[release-highlights-7.8.0-data-visualizer]] -=== Extended upload limit and explanations for Data Visualizer - -You can now upload files up to 1 GB in Data Visualizer. The file structure -finder functionality of the Data Visualizer provides more detailed explanations -after both successful and unsuccessful analysis which makes it easier to -diagnose issues with file upload. - -// end::notable-highlights[] -