diff --git a/docs/reference/migration/migrate_7_7.asciidoc b/docs/reference/migration/migrate_7_7.asciidoc index a65ca7ea3f7..33d18eaf4d3 100644 --- a/docs/reference/migration/migrate_7_7.asciidoc +++ b/docs/reference/migration/migrate_7_7.asciidoc @@ -9,15 +9,10 @@ your application to Elasticsearch 7.7. See also <> and <>. -coming[7.7.0] - //NOTE: The notable-breaking-changes tagged regions are re-used in the //Installation and Upgrade Guide //tag::notable-breaking-changes[] - -//end::notable-breaking-changes[] - [discrete] [[breaking_77_logging_changes]] === Logging changes @@ -30,7 +25,22 @@ loggers in the `org.elasticsearch.action.*` hierarchy emitted log messages at `DEBUG` level by default. This sometimes resulted in a good deal of unnecessary log noise. From 7.7 onwards the default log level for logger in this hierarchy is now `INFO`, in line with most other loggers. If needed, you can recover the -pre-7.7 default behaviour by adjusting your <>. +pre-7.7 default behaviour by adjusting your {ref}/logging.html[logging]. + +[discrete] +[[breaking_77_mapping_changes]] +=== Mapping changes + +[discrete] +[[stricter-mapping-validation]] +==== Validation for dynamic templates + +So far misconfiguration of dynamic templates have been discovered when indexing +a document with an unmapped field only. In {es} 8.0 and later versions, dynamic mappings +have stricter validation, done at mapping update time. Invalid updates, such as using +incorrect analyzer settings or unknown field types, fail. For +indices created in {es} 7.7 and later version, the update succeeds but emits a warning. + [discrete] [[breaking_77_settings_changes]] @@ -86,7 +96,14 @@ happen when using date math or dates that don't specify up to the last millisecond. While queries on `date` field round up to the latest millisecond for `gt` and `lte` boundaries, the same queries on `date_range` fields didn't do this. The behavior is now the same for both field types like documented in -<>. +{ref}/query-dsl-range-query.html#range-query-date-math-rounding[Date math and rounding]. + +[discrete] +==== Pipeline aggregation validation errors +The pipeline aggregation validation has been moved to the coordinating node. +Those errors that used to return HTTP 500s/Internal Server Errors now return +400/Bad Request and we now return a list of validation errors rather than the +first one we encounter. [discrete] [[breaking_77_highlighters_changes]] @@ -98,3 +115,4 @@ If a keyword value was ignored during indexing because of its length (`ignore_above` parameter was applied), {es} doesn't attempt to highlight it anymore, which means no highlights are produced for ignored values. +//end::notable-breaking-changes[] diff --git a/docs/reference/release-notes/7.7.asciidoc b/docs/reference/release-notes/7.7.asciidoc index 458c9809a2d..a6555e0caa4 100644 --- a/docs/reference/release-notes/7.7.asciidoc +++ b/docs/reference/release-notes/7.7.asciidoc @@ -131,6 +131,7 @@ Search:: === Enhancements Aggregations:: +* Fixed rewrite of time zone without DST {pull}54398[#54398] * Try to save memory on aggregations {pull}53793[#53793] * Speed up partial reduce of terms aggregations {pull}53216[#53216] (issue: {issue}51857[#51857]) * Simplify SiblingPipelineAggregator {pull}53144[#53144] @@ -182,6 +183,8 @@ Features/Features:: * Enable deprecation checks for removed settings {pull}53317[#53317] Features/ILM+SLM:: +* Use Priority.IMMEDIATE for stop ILM cluster update {pull}54909[#54909] +* Add cluster update timeout on step retry {pull}54878[#54878] * Hide ILM & SLM history aliases {pull}53564[#53564] * Avoid race condition in ILMHistorySotre {pull}53039[#53039] (issues: {issue}50353[#50353], {issue}52853[#52853]) * Make FreezeStep retryable {pull}52540[#52540] @@ -213,9 +216,6 @@ Features/Monitoring:: * Secure password for monitoring HTTP exporter {pull}50919[#50919] (issue: {issue}50197[#50197]) * Validate SSL settings at parse time {pull}49196[#49196] (issue: {issue}47711[#47711]) -Features/Stats:: -* Add analysis components and mapping types to the usage API. {pull}51031[#51031] - Features/Watcher:: * Make watch history indices hidden {pull}52962[#52962] (issue: {issue}50251[#50251]) * Upgrade to the latest OWASP HTML sanitizer {pull}50765[#50765] (issue: {issue}50395[#50395]) @@ -235,6 +235,7 @@ Infra/Logging:: * Capture stdout and stderr to log4j log {pull}50259[#50259] (issue: {issue}50156[#50156]) Infra/Packaging:: +* Use AdoptOpenJDK API to Download JDKs {pull}55127[#55127] (issue: {issue}55125[#55125]) * Introduce aarch64 Docker image {pull}53936[#53936] (issue: {issue}53914[#53914]) * Introduce jvm.options.d for customizing JVM options {pull}51882[#51882] (issue: {issue}51626[#51626]) @@ -414,6 +415,7 @@ Audit:: * Logfile audit settings validation {pull}52537[#52537] (issues: {issue}47038[#47038], {issue}47711[#47711], {issue}52357[#52357]) Authentication:: +* Fix responses for the token APIs {pull}54532[#54532] (issue: {issue}53323[#53323]) * Fix potential bug in concurrent token refresh support {pull}53668[#53668] * Respect runas realm for ApiKey security operations {pull}52178[#52178] (issue: {issue}51975[#51975]) * Preserve ApiKey credentials for async verification {pull}51244[#51244] @@ -425,12 +427,14 @@ Authorization:: * Allow _rollup_search with read privilege {pull}52043[#52043] (issue: {issue}50245[#50245]) CCR:: +* Clear recent errors when auto-follow successfully {pull}54997[#54997] * Put CCR tasks on (data && remote cluster clients) {pull}54146[#54146] (issue: {issue}53924[#53924]) * Handle no such remote cluster exception in ccr {pull}53415[#53415] (issue: {issue}53225[#53225]) * Fix shard follow task cleaner under security {pull}52347[#52347] (issues: {issue}44702[#44702], {issue}51971[#51971]) CRUD:: * Force execution of finish shard bulk request {pull}51957[#51957] (issue: {issue}51904[#51904]) +* Block too many concurrent mapping updates {pull}51038[#51038] (issue: {issue}50670[#50670]) * Return 429 status code when there's a read_only cluster block {pull}50166[#50166] (issue: {issue}49393[#49393]) Cluster Coordination:: @@ -444,21 +448,28 @@ Distributed:: * Exclude nested documents in LuceneChangesSnapshot {pull}51279[#51279] Engine:: +* Update translog policy before the next safe commit {pull}54839[#54839] (issue: {issue}52223[#52223]) * Fix doc_stats and segment_stats of ReadOnlyEngine {pull}53345[#53345] (issues: {issue}51303[#51303], {issue}51331[#51331]) * Do not wrap soft-deletes reader for segment stats {pull}51331[#51331] (issues: {issue}51192[#51192], {issue}51303[#51303]) * Account soft-deletes in FrozenEngine {pull}51192[#51192] (issue: {issue}50775[#50775]) * Fixed an index corruption bug that would occur when applying deletes or updates on an index after it has been shrunk. More details can be found on the https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue]. Features/CAT APIs:: +* Cat tasks output should respect time display settings {pull}54536[#54536] * Fix NPE in RestPluginsAction {pull}52620[#52620] (issue: {issue}45321[#45321]) Features/ILM+SLM:: +* Ensure error handler is called during SLM retention callback failure {pull}55252[#55252] (issue: {issue}55217[#55217]) +* Ignore ILM indices in the TerminalPolicyStep {pull}55184[#55184] (issue: {issue}51631[#51631]) * Disallow negative TimeValues {pull}53913[#53913] (issue: {issue}54041[#54041]) * Fix null config in SnapshotLifecyclePolicy.toRequest {pull}53328[#53328] (issues: {issue}44465[#44465], {issue}53171[#53171]) * Freeze step retry when not acknowledged {pull}53287[#53287] * Make the set-single-node-allocation retryable {pull}52077[#52077] (issue: {issue}43401[#43401]) * Fix the init step to actually be retryable {pull}52076[#52076] +Features/Indices APIs:: +* Read the index.number_of_replicas from template so that wait_for_active_shards is interpreted correctly {pull}54231[#54231] + Features/Ingest:: * Fix ingest pipeline _simulate api with empty docs never returns a response {pull}52937[#52937] (issue: {issue}52833[#52833]) * Handle errors when evaluating if conditions in processors {pull}52543[#52543] (issue: {issue}52339[#52339]) @@ -492,6 +503,8 @@ Highlighting:: * Fix highlighter support in PinnedQuery and added test {pull}53716[#53716] (issue: {issue}53699[#53699]) Infra/Core:: +* Make feature usage version aware {pull}55246[#55246] (issues: {issue}44589[#44589], {issue}55248[#55248]) +* Avoid StackOverflowError if write circular reference exception {pull}54147[#54147] (issue: {issue}53589[#53589]) * Fix Joda compatibility in stream protocol {pull}53823[#53823] (issue: {issue}53586[#53586]) * Avoid self-suppression on grouped action listener {pull}53262[#53262] (issue: {issue}53174[#53174]) * Ignore virtual ethernet devices that disappear {pull}51581[#51581] (issue: {issue}49914[#49914]) @@ -537,6 +550,7 @@ Mapping:: * Throw better exception on wrong `dynamic_templates` syntax {pull}51783[#51783] (issue: {issue}51486[#51486]) Network:: +* Add support for more named curves {pull}55179[#55179] (issue: {issue}55031[#55031]) * Allow proxy mode server name to be updated {pull}54107[#54107] * Invoke response handler on failure to send {pull}53631[#53631] * Do not log no-op reconnections at DEBUG {pull}53469[#53469] @@ -553,6 +567,7 @@ Reindex:: * Allow comma separated source indices {pull}52044[#52044] (issue: {issue}51949[#51949]) SQL:: +* Fix ODBC metadata for DATE & TIME data types {pull}55316[#55316] (issue: {issue}41086[#41086]) * Fix NPE for parameterized LIKE/RLIKE {pull}53573[#53573] (issue: {issue}53557[#53557]) * Add support for index aliases for SYS COLUMNS command {pull}53525[#53525] (issue: {issue}31609[#31609]) * Fix issue with LIKE/RLIKE as painless script {pull}53495[#53495] (issue: {issue}53486[#53486]) @@ -567,6 +582,7 @@ SQL:: * Selecting a literal from grouped by query generates error {pull}41964[#41964] (issues: {issue}41413[#41413], {issue}41951[#41951]) Search:: +* Improve robustness of Query Result serializations {pull}54692[#54692] (issue: {issue}54665[#54665]) * Fix Term Vectors with artificial docs and keyword fields {pull}53504[#53504] (issue: {issue}53494[#53494]) * Fix concurrent requests race over scroll context limit {pull}53449[#53449] * Fix pre-sorting of shards in the can_match phase {pull}53397[#53397] @@ -576,6 +592,8 @@ Search:: * Correct boost in `script_score` query and error on negative scores {pull}52478[#52478] (issue: {issue}48465[#48465]) Snapshot/Restore:: +* Exclude Snapshot Shard Status Update Requests from Circuit Breaker {pull}55376[#55376] (issue: {issue}54714[#54714]) +* Fix Snapshot Completion Listener Lost on Master Failover {pull}54286[#54286] * Fix Non-Verbose Snapshot List Missing Empty Snapshots {pull}52433[#52433] * Fix Inconsistent Shard Failure Count in Failed Snapshots {pull}51416[#51416] (issue: {issue}47550[#47550]) * Fix Overly Aggressive Request DeDuplication {pull}51270[#51270] (issue: {issue}51253[#51253]) diff --git a/docs/reference/release-notes/highlights-7.7.0.asciidoc b/docs/reference/release-notes/highlights-7.7.0.asciidoc index 75f0f8d2278..823d26dfb54 100644 --- a/docs/reference/release-notes/highlights-7.7.0.asciidoc +++ b/docs/reference/release-notes/highlights-7.7.0.asciidoc @@ -24,6 +24,17 @@ https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue]. // end::notable-highlights[] +// tag::notable-highlights[] +[float] +=== Significant reduction of heap usage of segments + +This release of Elasticsearch significantly reduces the amount of heap memory +that is needed to keep Lucene segments open. In addition to helping with cluster +stability, this helps reduce costs by storing much more data per node before +hitting memory limits. + +// end::notable-highlights[] + // tag::notable-highlights[] [discrete] === {transforms-cap} – now in GA! @@ -69,4 +80,113 @@ categories. further insight into the results of a {classification} or {regression} job and therefore helps interpret these results. -// end::notable-highlights[] \ No newline at end of file +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== Finer memory control for bucket aggregations + +We introduced a new `search.check_buckets_step_size` setting to +better control how the coordinating node allocates memory when aggregating +buckets. The allocation of buckets is now be done in steps, each step +allocating a number of buckets equal to this setting. To avoid an `OutOfMemory` +error, a parent circuit breaker check is performed on allocation. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== A new way of searching: asynchronously + +You can now submit {ref}/async-search-intro.html[long-running searches] using +the new {ref}/async-search.html[`_async_search` API]. The new API accepts the +same parameters and request body as the {ref}/search-search.html[Search API]. +However, instead of blocking and returning the final response only when it's +entirely finished, you can retrieve results from an async search as they become +available. + +The request takes a parameter, `wait_for_completion`, which controls how long +the server will wait until it sends back a response. The first response +contains among others a search unique ID, a response version, an indication if +this response is partial or not, plus the usual metadata (shards involved, +number of hits etc) and potentially results. If the response is not complete +and final, the client can continue polling for results, issuing a new request +using the provided search ID. If new results are available, the returned +version is incremented and the new batch of results are returned. This can +continue until all the results are fetched. + +Unless deleted earlier by the user, the asynchronous searches are kept alive +for a given interval. This defaults to 5 days and can be controlled by another +request parameter, `keep_alive`. +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== Password protection for the keystore + +{es} uses a custom on-disk {ref}/secure-settings.html[keystore] for secure settings such as +passwords and SSL certificates. Up until now, this prevented users with +{ref}/elasticsearch-keystore.html[command-line access] from viewing secure files by listing commands, but nothing +prevented such users from changing values in the keystore, or removing values +from it. Furthermore, the values were only obfuscated by a hash; no +user-specific secret protected the secure settings. + +This new feature changes all of that by adding password-protection to the +keystore. This is not be a breaking change: if a keystore has no password, +there won’t be any new prompts. A user must choose to password-protect their +keystore in order to benefit from the new behavior. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== A new aggregation: `top_metrics` + +The new {ref}//search-aggregations-metrics-top-metrics.html[`top_metrics` aggregation] "selects" a metric from a document according +to a criteria on a given, different field. That criteria is currently the +largest or smallest "sort" value. It is fairly similar to `top_hits` in spirit, +but because it is more limited, `top_metrics` uses less memory and +is often faster. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== Query speed-up for sorted queries on time-based indices + +We've optimized sorted, top-documents-only queries run on time-based indices. +The optimization stems from the fact that the ranges of (document) timestamps +in the shards don't overlap. It is implemented by rewriting the shard search +requests based on the partial results already available from other shards, if +it can be determined that the query will not yield any result from the current +shard; i.e. we know in advance that the bottom entry of the (sorted) result set +after a partial merge is better than the values contained in this current +shard. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== A new aggregation: `boxplot` + +The https://en.wikipedia.org/wiki/Interquartile_range[interquartile range (IQR)] is a common robust measure of statistical dispersion. +Compared to the standard deviation, the IQR is less sensitive to outliers in +the data, with a breakdown point of 0.25. Along with the median, it is often +used in creating a box plot, a simple yet common way to summarize data and +identify potential outliers. + +The new {ref}/search-aggregations-metrics-boxplot-aggregation.html[`boxplot` +aggregation] calculates the min, max, and medium as well as the first and third +quartiles of a given data set. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[float] +=== AArch64 support + +{es} now provides AArch64 packaging, including bundling an AArch64 JDK +distribution. There are some restrictions in place, namely no {ml} support and +depending on underlying page sizes, class data sharing is disabled. + +// end::notable-highlights[]