[[release-notes-7.9.0]] == {es} version 7.9.0 Also see <>. coming::[7.9.0] [[breaking-7.9.0]] [discrete] === Breaking changes Script Cache:: * Script cache size and rate limiting are per-context {es-pull}55753[#55753] (issue: {es-issue}50152[#50152]) Field capabilities API:: * Constant_keyword fields are now described by their family type `keyword` instead of `constant_keyword` {es-pull}58483[#58483] (issue: {es-issue}53175[#53175]) Snapshot restore throttling:: * Restoring from a snapshot (which is a particular form of recovery) is now properly taking recovery throttling into account (i.e. the `indices.recovery.max_bytes_per_sec` setting). The `max_restore_bytes_per_sec` setting is also now defaulting to unlimited, whereas previously it was set to `40mb`, which is the default that's used for `indices.recovery.max_bytes_per_sec`. This means that no behavioral change will be observed by clusters where the recovery and restore settings had not been adapted from the defaults. {es-pull}58658[#58658] Thread pool write queue size:: * The WRITE thread pool default queue size (`thread_pool.write.size`) has been increased from 200 to 10000. A small queue size (200) caused issues when users wanted to send small indexing requests with a high client count. Additional memory-oriented back pressure has been introduced with the `indexing_pressure.memory.limit` setting. This setting configures a limit to the number of bytes allowed to be consumed by outstanding indexing requests. {es-issue}59263[#59263] Dangling indices:: * Automatically importing dangling indices is now deprecated, disabled by default, and will be removed in {es} 8.0. See the <>. {es-pull}58176[#58176] {es-pull}58898[#58898] (issue: {es-issue}48366[#48366]) [[breaking-java-7.9.0]] [discrete] === Breaking Java changes Aggregations:: * Improve cardinality measure used to build aggs {es-pull}56533[#56533] (issue: {es-issue}56487[#56487]) Features/Ingest:: * Add optional description parameter to ingest processors. {es-pull}57906[#57906] (issue: {es-issue}56000[#56000]) [[feature-7.9.0]] [discrete] === New features Aggregations:: * Add moving percentiles pipeline aggregation {es-pull}55441[#55441] (issue: {es-issue}49452[#49452]) * Add normalize pipeline aggregation {es-pull}56399[#56399] (issue: {es-issue}51005[#51005]) * Add variable width histogram aggregation {es-pull}42035[#42035] (issues: {es-issue}9572[#9572], {es-issue}50863[#50863]) * Add pipeline inference aggregation {es-pull}58193[#58193] * Speed up time interval arounding around daylight savings time (DST) {es-pull}56371[#56371] (issue: {es-issue}55559[#55559]) Geo:: * Override doc_value parameter in Spatial XPack module {es-pull}53286[#53286] (issue: {es-issue}37206[#37206]) Machine Learning:: * Add update data frame analytics jobs API {es-pull}58302[#58302] (issue: {es-issue}45720[#45720]) * Introduce model_plot_config.annotations_enabled setting for anomaly detection jobs {es-pull}57539[#57539] (issue: {es-issue}55781[#55781]) * Report significant changes to anomaly detection models in annotations of the results {ml-pull}1247[#1247], {es-pull}56342[#56342], {es-pull}56417[#56417], {es-pull}57144[#57144], {es-pull}57278[#57278], {es-pull}57539[#57539] Mapping:: * Merge mappings for composable index templates {es-pull}58521[#58521] (issue: {es-issue}53101[#53101]) * Wildcard field optimised for wildcard queries {es-pull}49993[#49993] (issue: {es-issue}48852[#48852]) Search:: * Allow index filtering in field capabilities API {es-pull}57276[#57276] (issue: {es-issue}56195[#56195]) [[enhancement-7.9.0]] [discrete] === Enhancements Aggregations:: * Add support for numeric range keys {es-pull}56452[#56452] (issue: {es-issue}56402[#56402]) * Added standard deviation / variance sampling to extended stats {es-pull}49782[#49782] (issue: {es-issue}49554[#49554]) * Give significance lookups their own home {es-pull}57903[#57903] * Increase search.max_buckets to 65,535 {es-pull}57042[#57042] (issue: {es-issue}51731[#51731]) * Optimize date_histograms across daylight savings time {es-pull}55559[#55559] * Return clear error message if aggregation type is invalid {es-pull}58255[#58255] (issue: {es-issue}58146[#58146]) * Save memory on numeric significant terms when not top {es-pull}56789[#56789] (issue: {es-issue}55873[#55873]) * Save memory when auto_date_histogram is not on top {es-pull}57304[#57304] (issue: {es-issue}56487[#56487]) * Save memory when date_histogram is not on top {es-pull}56921[#56921] (issues: {es-issue}55873[#55873], {es-issue}56487[#56487]) * Save memory when histogram agg is not on top {es-pull}57277[#57277] * Save memory when numeric terms agg is not top {es-pull}55873[#55873] * Save memory when parent and child are not on top {es-pull}57892[#57892] (issue: {es-issue}55873[#55873]) * Save memory when rare_terms is not on top {es-pull}57948[#57948] (issue: {es-issue}55873[#55873]) * Save memory when significant_text is not on top {es-pull}58145[#58145] (issue: {es-issue}55873[#55873]) * Save memory when string terms are not on top {es-pull}57758[#57758] * Speed up reducing auto_date_histo with a time zone {es-pull}57933[#57933] (issue: {es-issue}56124[#56124]) * Speed up rounding in auto_date_histogram {es-pull}56384[#56384] (issue: {es-issue}55559[#55559]) Allocation:: * Account for remaining recovery in disk allocator {es-pull}58029[#58029] Analysis:: * Add max_token_length setting to the CharGroupTokenizer {es-pull}56860[#56860] (issue: {es-issue}56676[#56676]) * Expose discard_compound_token option to kuromoji_tokenizer {es-pull}57421[#57421] * Support multiple tokens on LHS in stemmer_override rules (#56113) {es-pull}56484[#56484] (issue: {es-issue}56113[#56113]) Authentication:: * Add http proxy support for OIDC realm {es-pull}57039[#57039] (issue: {es-issue}53379[#53379]) * Improve threadpool usage and error handling for API key validation {es-pull}58090[#58090] (issue: {es-issue}58088[#58088]) * Support handling LogoutResponse from SAML idP {es-pull}56316[#56316] (issues: {es-issue}40901[#40901], {es-issue}43264[#43264]) Authorization:: * Add cache for application privileges {es-pull}55836[#55836] (issue: {es-issue}54317[#54317]) * Add monitor and view_index_metadata privileges to built-in `kibana_system` role {es-pull}57755[#57755] * Improve role cache efficiency for API key roles {es-pull}58156[#58156] (issue: {es-issue}53939[#53939]) CCR:: * Allow follower indices to override leader settings {es-pull}58103[#58103] CRUD:: * Retry failed replication due to transient errors {es-pull}55633[#55633] Engine:: * Don't log on RetentionLeaseSync error handler after an index has been deleted {es-pull}58098[#58098] (issue: {es-issue}57864[#57864]) Features/Data streams:: * Add support for snapshot and restore to data streams {es-pull}57675[#57675] (issues: {es-issue}53100[#53100], {es-issue}57127[#57127]) * Data stream creation validation allows for prefixed indices {es-pull}57750[#57750] (issue: {es-issue}53100[#53100]) * Disallow deletion of composable template if in use by data stream {es-pull}57957[#57957] (issue: {es-issue}57004[#57004]) * Validate alias operations don't target data streams {es-pull}58327[#58327] (issue: {es-issue}53100[#53100]) Features/ILM+SLM:: * Add data stream support to searchable snapshot action {es-pull}57873[#57873] (issue: {es-issue}53100[#53100]) * Add data stream support to the shrink action {es-pull}57616[#57616] (issue: {es-issue}53100[#53100]) * Add support for rolling over data streams {es-pull}57295[#57295] (issues: {es-issue}53100[#53100], {es-issue}53488[#53488]) * Check the managed index is not a data stream write index {es-pull}58239[#58239] (issue: {es-issue}53100[#53100]) Features/Indices APIs:: * Add default composable templates for new indexing strategy {es-pull}57629[#57629] (issue: {es-issue}56709[#56709]) * Add index block api {es-pull}58094[#58094] * Add new flag to check whether alias exists on remove {es-pull}58100[#58100] * Add prefer_v2_templates parameter to reindex {es-pull}56253[#56253] (issue: {es-issue}53101[#53101]) * Add template simulation API for simulating template composition {es-pull}56842[#56842] (issues: {es-issue}53101[#53101], {es-issue}55686[#55686], {es-issue}56255[#56255], {es-issue}56390[#56390]) Features/Ingest:: * Add ignore_empty_value parameter in set ingest processor {es-pull}57030[#57030] (issue: {es-issue}54783[#54783]) * Support `if_seq_no` and `if_primary_term` for ingest {es-pull}55430[#55430] (issue: {es-issue}41255[#41255]) Features/Java High Level REST Client:: * Add support for data streams {es-pull}58106[#58106] (issue: {es-issue}53100[#53100]) * Enable decompression of response within LowLevelRestClient {es-pull}55413[#55413] (issues: {es-issue}24349[#24349], {es-issue}53555[#53555]) Features/Java Low Level REST Client:: * Add isRunning method to RestClient {es-pull}57973[#57973] (issue: {es-issue}42133[#42133]) * Add RequestConfig support to RequestOptions {es-pull}57972[#57972] Infra/Circuit Breakers:: * Enhance real memory circuit breaker with G1 GC {es-pull}58674[#58674] (issue: {es-issue}57202[#57202]) Infra/Core:: * Introduce node.roles setting {es-pull}54998[#54998] Infra/Packaging:: * Remove DEBUG-level logging from actions in Docker {es-pull}57389[#57389] (issues: {es-issue}51198[#51198], {es-issue}51459[#51459]) Infra/Plugins:: * Improved ExtensiblePlugin {es-pull}58234[#58234] Infra/Resiliency:: * Adds resiliency to read-only filesystems #45286 {es-pull}52680[#52680] (issue: {es-issue}45286[#45286]) Machine Learning:: * Accounting for model size when models are not cached. {es-pull}58670[#58670] * Adds new for_export flag to GET _ml/inference API {es-pull}57351[#57351] * Adds WKT geometry detection in find_file_structure {es-pull}57014[#57014] (issue: {es-issue}56967[#56967]) * Calculate cache misses for inference and return in stats {es-pull}58252[#58252] * Delete auto-generated annotations when job is deleted. {es-pull}58169[#58169] (issue: {es-issue}57976[#57976]) * Delete auto-generated annotations when model snapshot is reverted {es-pull}58240[#58240] (issue: {es-issue}57982[#57982]) * Delete expired data by job {es-pull}57337[#57337] * Introduce Annotation.event field {es-pull}57144[#57144] (issue: {es-issue}55781[#55781]) * Add support for larger forecasts in memory via max_model_memory setting {ml-pull}1238[#1238], {es-pull}57254[#57254] * Don't lose precision when saving model state {ml-pull}1274[#1274] * Parallelize the feature importance calculation for classification and regression over trees {ml-pull}1277[#1277] * Add an option to do categorization independently for each partition {ml-pull}1293[#1293], {ml-pull}1318[#1318], {ml-pull}1356[#1356], {es-pull}57683[#57683] * Memory usage is reported during job initialization {ml-pull}1294[#1294] * More realistic memory estimation for classification and regression means that these analyses will require lower memory limits than before {ml-pull}1298[#1298] * Checkpoint state to allow efficient failover during coarse parameter search for classification and regression {ml-pull}1300[#1300] * Improve data access patterns to speed up classification and regression {ml-pull}1312[#1312] * Performance improvements for classification and regression, particularly running multithreaded {ml-pull}1317[#1317] * Improve runtime and memory usage training deep trees for classification and regression {ml-pull}1340[#1340] * Improvement in handling large inference model definitions {ml-pull}1349[#1349] * Add a peak_model_bytes field to model_size_stats {ml-pull}1389[#1389] Mapping:: * Add regex query support to wildcard field {es-pull}55548[#55548] (issue: {es-issue}54725[#54725]) * Make `keyword` a family of field types {es-pull}58315[#58315] (issue: {es-issue}53175[#53175]) * Store parsed mapping settings in IndexSettings {es-pull}57492[#57492] (issue: {es-issue}57395[#57395]) * Wildcard field - add support for custom null values {es-pull}57047[#57047] Network:: * Make the number of transport threads equal to the number of available CPUs {es-pull}56488[#56488] Recovery:: * Implement dangling indices API {es-pull}50920[#50920] (issue: {es-issue}48366[#48366]) * Reestablish peer recovery after network errors {es-pull}55274[#55274] * Sending operations concurrently in peer recovery {es-pull}58018[#58018] (issue: {es-issue}58011[#58011]) Reindex:: * Throw an illegal_argument_exception when max_docs is less than slices {es-pull}54901[#54901] (issue: {es-issue}52786[#52786]) SQL:: * Implement TIME_PARSE function for parsing strings into TIME values {es-pull}55223[#55223] (issues: {es-issue}54963[#54963], {es-issue}55095[#55095]) * Implement TOP as an alternative to LIMIT {es-pull}57428[#57428] (issue: {es-issue}41195[#41195]) * Implement TRIM function {es-pull}57518[#57518] (issue: {es-issue}41195[#41195]) * Improve performances of LTRIM/RTRIM {es-pull}57603[#57603] (issue: {es-issue}57594[#57594]) * Make CASTing string to DATETIME more lenient {es-pull}57451[#57451] * Redact credentials in connection exceptions {es-pull}58650[#58650] (issue: {es-issue}56474[#56474]) * Relax parsing of date/time escaped literals {es-pull}58336[#58336] (issue: {es-issue}58262[#58262]) * Add support for scalars within LIKE/RLIKE {es-pull}56495[#56495] (issue: {es-issue}55058[#55058]) Search:: * Add description to submit and get async search, as well as cancel tasks {es-pull}57745[#57745] * Add matchBoolPrefix static method in query builders {es-pull}58637[#58637] (issue: {es-issue}58388[#58388]) * Add range query support to wildcard field {es-pull}57881[#57881] (issue: {es-issue}57816[#57816]) * Group docIds by segment in FetchPhase to better use LRU cache {es-pull}57273[#57273] * Improve error handling when decoding async execution ids {es-pull}56285[#56285] * Specify reason whenever async search gets cancelled {es-pull}57761[#57761] * Use index sort range query when possible. {es-pull}56657[#56657] (issue: {es-issue}48665[#48665]) Security:: * Add machine learning admin permissions to the kibana_system role {es-pull}58061[#58061] * Just log 401 stacktraces {es-pull}55774[#55774] Snapshot/Restore:: * Deduplicate Index Metadata in BlobStore {es-pull}50278[#50278] (issues: {es-issue}45736[#45736], {es-issue}46250[#46250], {es-issue}49800[#49800]) * Default to zero replicas for searchable snapshots {es-pull}57802[#57802] (issue: {es-issue}50999[#50999]) * Enable fully concurrent snapshot operations {es-pull}56911[#56911] * Support cloning of searchable snapshot indices {es-pull}56595[#56595] * Track GET/LIST Azure Storage API calls {es-pull}56773[#56773] * Track GET/LIST GoogleCloudStorage API calls {es-pull}56585[#56585] * Track PUT/PUT_BLOCK operations on AzureBlobStore. {es-pull}56936[#56936] * Track multipart/resumable uploads GCS API calls {es-pull}56821[#56821] * Track upload requests on S3 repositories {es-pull}56826[#56826] Task Management:: * Add index name to refresh mapping task {es-pull}57598[#57598] * Cancel task and descendants on channel disconnects {es-pull}56620[#56620] (issues: {es-issue}56327[#56327], {es-issue}56619[#56619]) Transform:: * Add support for terms agg in transforms {es-pull}56696[#56696] * Adds geotile_grid support in group_by {es-pull}56514[#56514] (issue: {es-issue}56121[#56121]) [[bug-7.9.0]] [discrete] === Bug fixes Aggregations:: * Fix auto_date_histogram interval {es-pull}56252[#56252] (issue: {es-issue}56116[#56116]) * Fix bug in faster interval rounding {es-pull}56433[#56433] (issue: {es-issue}56400[#56400]) * Fix bug in parent and child aggregators when parent field not defined {es-pull}57089[#57089] (issue: {es-issue}42997[#42997]) * Fix missing null values for std_deviation_bounds in ext. stats aggs {es-pull}58000[#58000] Allocation:: * Reword INDEX_READ_ONLY_ALLOW_DELETE_BLOCK message {es-pull}58410[#58410] (issues: {es-issue}42559[#42559], {es-issue}50166[#50166], {es-issue}58376[#58376]) Authentication:: * Map only specific type of OIDC Claims {es-pull}58524[#58524] Authorization:: * Change privilege of enrich stats API to monitor {es-pull}52027[#52027] (issue: {es-issue}51677[#51677]) Engine:: * Fix local translog recovery not updating safe commit in edge case {es-pull}57350[#57350] (issue: {es-issue}57010[#57010]) * Hide AlreadyClosedException on IndexCommit release {es-pull}57986[#57986] (issue: {es-issue}57797[#57797]) Features/ILM+SLM:: * Normalized prefix for rollover API {es-pull}57271[#57271] (issue: {es-issue}53388[#53388]) Features/Indices APIs:: * Don't allow invalid template combinations {es-pull}56397[#56397] (issues: {es-issue}53101[#53101], {es-issue}56314[#56314]) * Handle `cluster.max_shards_per_node` in YAML config {es-pull}57234[#57234] (issue: {es-issue}40803[#40803]) Features/Ingest:: * Fix ingest simulate verbose on failure with conditional {es-pull}56478[#56478] (issue: {es-issue}56004[#56004]) Geo:: * Check for degenerated lines when calculating the centroid {es-pull}58027[#58027] (issue: {es-issue}55851[#55851]) * Fix bug in circuit-breaker check for geoshape grid aggregations {es-pull}57962[#57962] (issue: {es-issue}57847[#57847]) Infra/Scripting:: * Fix source return bug in scripting {es-pull}56831[#56831] (issue: {es-issue}52103[#52103]) Machine Learning:: * Fix wire serialization for flush acknowledgements {es-pull}58413[#58413] * Make waiting for renormalization optional for internally flushing job {es-pull}58537[#58537] (issue: {es-issue}58395[#58395]) * Tail the C++ logging pipe before connecting other pipes {es-pull}56632[#56632] (issue: {es-issue}56366[#56366]) * Fix numerical issues leading to blow up of the model plot bounds {ml-pull}1268[#1268] * Fix causes for inverted forecast confidence interval bounds {ml-pull}1369[#1369] (issue: {ml-issue}1357[#1357]) * Restrict growth of max matching string length for categories {ml-pull}1406[#1406] Mapping:: * Wildcard field fix for scripts - changed value type from BytesRef to String {es-pull}58060[#58060] (issue: {es-issue}58044[#58044]) SQL:: * Introduce JDBC option for meta pattern escaping {es-pull}40661[#40661] (issue: {es-issue}40640[#40640]) Search:: * Don't omit empty arrays when filtering _source {es-pull}56527[#56527] (issues: {es-issue}20736[#20736], {es-issue}22593[#22593], {es-issue}23796[#23796]) * Fix casting of scaled_float in sorts {es-pull}57207[#57207] Snapshot/Restore:: * Account for recovery throttling when restoring snapshot {es-pull}58658[#58658] (issue: {es-issue}57023[#57023]) * Fix noisy logging during snapshot delete {es-pull}56264[#56264] * Fix S3ClientSettings leak {es-pull}56703[#56703] (issue: {es-issue}56702[#56702]) [[upgrade-7.9.0]] [discrete] === Upgrades Search:: * Update to lucene snapshot e7c625430ed {es-pull}57981[#57981]