[[release-notes-7.0.0-beta1]] == {es} version 7.0.0-beta1 Also see <>. [[breaking-7.0.0-beta1]] [float] === Breaking changes Audit:: * Remove index audit output type {es-pull}37707[#37707] (issues: {es-issue}29881[#29881], {es-issue}37301[#37301]) Authentication:: * Remove bwc logic for token invalidation {es-pull}36893[#36893] (issue: {es-issue}36727[#36727]) Authorization:: * Remove implicit index monitor privilege {es-pull}37774[#37774] CCR:: * Follow stats api should return a 404 when requesting stats for a non existing index {es-pull}37220[#37220] (issue: {es-issue}37021[#37021]) CRUD:: * Remove support for internal versioning for concurrency control {es-pull}38254[#38254] (issue: {es-issue}1078[#1078]) Features/Ingest:: * Add ECS schema for user-agent ingest processor (#37727) {es-pull}37984[#37984] (issues: {es-issue}37329[#37329], {es-issue}37727[#37727]) * Remove special handling for ingest plugins {es-pull}36967[#36967] (issues: {es-issue}36898[#36898], {es-issue}36956[#36956]) Features/Java Low Level REST Client:: * Drop support for the low-level REST client on JDK 7 {es-pull}38540[#38540] (issue: {es-issue}29607[#29607]) Features/Watcher:: * Remove Watcher Account "unsecure" settings {es-pull}36736[#36736] (issue: {es-issue}36403[#36403]) Infra/Logging:: * Elasticsearch json logging {es-pull}36833[#36833] (issue: {es-issue}32850[#32850]) Infra/Packaging:: * Package ingest-user-agent as a module {es-pull}36956[#36956] * Package ingest-geoip as a module {es-pull}36898[#36898] Machine Learning:: * [ML] Remove types from datafeed {es-pull}36538[#36538] (issue: {es-issue}34265[#34265]) Mapping:: * Make sure to reject mappings with type _doc when include_type_name is false. {es-pull}38270[#38270] (issue: {es-issue}38266[#38266]) * Update the default for include_type_name to false. {es-pull}37285[#37285] * Support 'include_type_name' in RestGetIndicesAction {es-pull}37149[#37149] Network:: * Remove TLS 1.0 as a default SSL protocol {es-pull}37512[#37512] (issue: {es-issue}36021[#36021]) * Security: remove SSL settings fallback {es-pull}36846[#36846] (issue: {es-issue}29797[#29797]) Ranking:: * Forbid negative field boosts in analyzed queries {es-pull}37930[#37930] (issue: {es-issue}33309[#33309]) Search:: * Track total hits up to 10,000 by default {es-pull}37466[#37466] (issue: {es-issue}33028[#33028]) * Use mappings to format doc-value fields by default. {es-pull}30831[#30831] (issues: {es-issue}26948[#26948], {es-issue}29639[#29639]) Security:: * Remove heuristics that enable security on trial licenses {es-pull}38075[#38075] (issue: {es-issue}38009[#38009]) ZenDiscovery:: * Remove DiscoveryPlugin#getDiscoveryTypes {es-pull}38414[#38414] (issue: {es-issue}38410[#38410]) [[breaking-java-7.0.0-beta1]] [float] === Breaking Java changes Features/Java Low Level REST Client:: * Remove support for maxRetryTimeout from low-level REST client {es-pull}38085[#38085] (issues: {es-issue}25951[#25951], {es-issue}31834[#31834], {es-issue}33342[#33342]) Infra/Core:: * Handle scheduler exceptions {es-pull}38014[#38014] (issues: {es-issue}28667[#28667], {es-issue}36137[#36137], {es-issue}37708[#37708]) [[deprecation-7.0.0-beta1]] [float] === Deprecations Aggregations:: * Deprecate dots in aggregation names {es-pull}31468[#31468] (issues: {es-issue}17600[#17600], {es-issue}19040[#19040]) Analysis:: * [Analysis] Deprecate Standard Html Strip Analyzer in master {es-pull}26719[#26719] (issue: {es-issue}4704[#4704]) Audit:: * Deprecate index audit output type {es-pull}37301[#37301] (issue: {es-issue}29881[#29881]) Features/Indices APIs:: * Reject setting index.optimize_auto_generated_id after version 7.0.0 {es-pull}28895[#28895] (issue: {es-issue}27600[#27600]) Features/Ingest:: * Deprecate `_type` in simulate pipeline requests {es-pull}37949[#37949] (issue: {es-issue}37731[#37731]) Features/Java High Level REST Client:: * Deprecate HLRC security methods {es-pull}37883[#37883] (issues: {es-issue}36938[#36938], {es-issue}37540[#37540]) * Deprecate HLRC EmptyResponse used by security {es-pull}37540[#37540] (issue: {es-issue}36938[#36938]) Features/Watcher:: * Deprecate xpack.watcher.history.cleaner_service.enabled {es-pull}37782[#37782] (issue: {es-issue}32041[#32041]) * deprecate types for watcher {es-pull}37594[#37594] (issue: {es-issue}35190[#35190]) Infra/Core:: * Core: Deprecate negative epoch timestamps {es-pull}36793[#36793] * Core: Deprecate use of scientific notation in epoch time parsing {es-pull}36691[#36691] Infra/Scripting:: * Add types deprecation to script contexts {es-pull}37554[#37554] * Deprecate _type from LeafDocLookup {es-pull}37491[#37491] * Scripting: Remove deprecated params.ctx {es-pull}36848[#36848] (issue: {es-issue}34059[#34059]) Machine Learning:: * Adding ml_settings entry to HLRC and Docs for deprecation_info {es-pull}38118[#38118] * [ML] Datafeed deprecation checks {es-pull}38026[#38026] (issue: {es-issue}37932[#37932]) * [ML] Remove "8" prefixes from file structure finder timestamp formats {es-pull}38016[#38016] * [ML] Adjust structure finder for Joda to Java time migration {es-pull}37306[#37306] * [ML] Resolve 7.0.0 TODOs in ML code {es-pull}36842[#36842] (issue: {es-issue}29963[#29963]) Mapping:: * Deprecate types in rollover index API {es-pull}38039[#38039] (issue: {es-issue}35190[#35190]) * Deprecate types in get field mapping API {es-pull}37667[#37667] (issue: {es-issue}35190[#35190]) * Deprecate types in the put mapping API. {es-pull}37280[#37280] (issues: {es-issue}29453[#29453], {es-issue}37285[#37285]) * Support include_type_name in the field mapping and index template APIs. {es-pull}37210[#37210] * Deprecate types in create index requests. {es-pull}37134[#37134] (issues: {es-issue}29453[#29453], {es-issue}37285[#37285]) * Deprecate use of the _type field in aggregations. {es-pull}37131[#37131] (issue: {es-issue}36802[#36802]) * Deprecate reference to _type in lookup queries {es-pull}37016[#37016] (issue: {es-issue}35190[#35190]) * Deprecate the document create endpoint. {es-pull}36863[#36863] * Deprecate types in index API {es-pull}36575[#36575] (issues: {es-issue}35190[#35190], {es-issue}35790[#35790]) * Deprecate types in update APIs {es-pull}36225[#36225] Search:: * Deprecate use of type in reindex request body {es-pull}36823[#36823] * Add typless endpoints for get_source and exist_source {es-pull}36426[#36426] [[feature-7.0.0-beta1]] [float] === New features Authentication:: * Add support for API keys to access Elasticsearch {es-pull}38291[#38291] (issue: {es-issue}34383[#34383]) * OIDC realm authentication flows {es-pull}37787[#37787] * [WIP] OIDC Realm JWT+JWS related functionality {es-pull}37272[#37272] (issues: {es-issue}35339[#35339], {es-issue}37009[#37009]) * OpenID Connect Realm base functionality {es-pull}37009[#37009] (issue: {es-issue}35339[#35339]) Authorization:: * Allow custom authorization with an authorization engine {es-pull}38358[#38358] (issues: {es-issue}32435[#32435], {es-issue}36245[#36245], {es-issue}37328[#37328], {es-issue}37495[#37495], {es-issue}37785[#37785], {es-issue}38137[#38137], {es-issue}38219[#38219]) * WIldcard IndicesPermissions don't cover .security {es-pull}36765[#36765] CCR:: * Add ccr follow info api {es-pull}37408[#37408] (issue: {es-issue}37127[#37127]) Features/ILM:: * [ILM] Add unfollow action {es-pull}36970[#36970] (issue: {es-issue}34648[#34648]) Geo:: * geotile_grid implementation {es-pull}37842[#37842] (issue: {es-issue}30240[#30240]) * [GEO] Fork Lucene's LatLonShape Classes to local lucene package {es-pull}36794[#36794] * [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {es-pull}36751[#36751] (issue: {es-issue}35320[#35320]) * [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {es-pull}35320[#35320] (issue: {es-issue}32039[#32039]) Machine Learning:: * ML: Adds set_upgrade_mode API endpoint {es-pull}37837[#37837] Mapping:: * Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. {es-pull}37871[#37871] (issue: {es-issue}37773[#37773]) * Add nanosecond field mapper {es-pull}37755[#37755] (issues: {es-issue}27330[#27330], {es-issue}32601[#32601]) SQL:: * SQL: Allow sorting of groups by aggregates {es-pull}38042[#38042] (issue: {es-issue}35118[#35118]) * SQL: Implement FIRST/LAST aggregate functions {es-pull}37936[#37936] (issue: {es-issue}35639[#35639]) * SQL: Introduce SQL DATE data type {es-pull}37693[#37693] (issue: {es-issue}37340[#37340]) Search:: * Introduce ability to minimize round-trips in CCS {es-pull}37828[#37828] (issues: {es-issue}32125[#32125], {es-issue}37566[#37566]) * Add script filter to intervals {es-pull}36776[#36776] * Add the ability to set the number of hits to track accurately {es-pull}36357[#36357] (issue: {es-issue}33028[#33028]) * Add a maximum search request size. {es-pull}26423[#26423] [[enhancement-7.0.0-beta1]] [float] === Enhancements Aggregations:: * Add Composite to AggregationBuilders {es-pull}38207[#38207] (issue: {es-issue}38020[#38020]) * Allow nested fields in the composite aggregation {es-pull}37178[#37178] (issue: {es-issue}28611[#28611]) * Remove single shard optimization when suggesting shard_size {es-pull}37041[#37041] (issue: {es-issue}32125[#32125]) * Use List instead of priority queue for stable sorting in bucket sort aggregator {es-pull}36748[#36748] (issue: {es-issue}36322[#36322]) * Keys are compared in BucketSortPipelineAggregation so making key type… {es-pull}36407[#36407] Allocation:: * Fail start on obsolete indices documentation {es-pull}37786[#37786] (issue: {es-issue}27073[#27073]) * Fail start on invalid index metadata {es-pull}37748[#37748] (issue: {es-issue}27073[#27073]) * Fail start of non-data node if node has data {es-pull}37347[#37347] (issue: {es-issue}27073[#27073]) Analysis:: * Allow word_delimiter_graph_filter to not adjust internal offsets {es-pull}36699[#36699] (issues: {es-issue}33710[#33710], {es-issue}34741[#34741]) Audit:: * Security Audit includes HTTP method for requests {es-pull}37322[#37322] (issue: {es-issue}29765[#29765]) * Add X-Forwarded-For to the logfile audit {es-pull}36427[#36427] Authentication:: * Security: propagate auth result to listeners {es-pull}36900[#36900] (issue: {es-issue}30794[#30794]) * Security: reorder realms based on last success {es-pull}36878[#36878] * Improve error message for 6.x style realm settings {es-pull}36876[#36876] (issues: {es-issue}30241[#30241], {es-issue}36026[#36026]) * Change missing authn message to not mention tokens {es-pull}36750[#36750] * Invalidate Token API enhancements - HLRC {es-pull}36362[#36362] (issue: {es-issue}35388[#35388]) * Enhance Invalidate Token API {es-pull}35388[#35388] (issues: {es-issue}34556[#34556], {es-issue}35115[#35115]) Authorization:: * Add apm_user reserved role {es-pull}38206[#38206] * Permission for restricted indices {es-pull}37577[#37577] (issue: {es-issue}34454[#34454]) * Remove kibana_user and kibana_dashboard_only_user index privileges {es-pull}37441[#37441] * Create snapshot role {es-pull}35820[#35820] (issue: {es-issue}34454[#34454]) CCR:: * Concurrent file chunk fetching for CCR restore {es-pull}38495[#38495] * Tighten mapping syncing in ccr remote restore {es-pull}38071[#38071] (issues: {es-issue}36879[#36879], {es-issue}37887[#37887]) * Do not allow put mapping on follower {es-pull}37675[#37675] (issue: {es-issue}30086[#30086]) * Added ccr to xpack usage infrastructure {es-pull}37256[#37256] (issue: {es-issue}37221[#37221]) * [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned {es-pull}37213[#37213] * [CCR] Added auto_follow_exception.timestamp field to auto follow stats {es-pull}36947[#36947] * [CCR] Add time since last auto follow fetch to auto follow stats {es-pull}36542[#36542] (issues: {es-issue}33007[#33007], {es-issue}35895[#35895]) CRUD:: * Add Seq# based optimistic concurrency control to UpdateRequest {es-pull}37872[#37872] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Introduce ssl settings to reindex from remote {es-pull}37527[#37527] (issues: {es-issue}29755[#29755], {es-issue}37287[#37287]) * Use Sequence number powered OCC for processing updates {es-pull}37308[#37308] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Document Seq No powered optimistic concurrency control {es-pull}37284[#37284] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Enable IPv6 URIs in reindex from remote {es-pull}36874[#36874] * Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm {es-pull}36757[#36757] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Expose Sequence Number based Optimistic Concurrency Control in the rest layer {es-pull}36721[#36721] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Add doc's sequence number + primary term to GetResult and use it for updates {es-pull}36680[#36680] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Add seq no powered optimistic locking support to the index and delete transport actions {es-pull}36619[#36619] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * Set acking timeout to 0 on dynamic mapping update {es-pull}31140[#31140] (issues: {es-issue}30672[#30672], {es-issue}30844[#30844]) Distributed:: * Recover retention leases during peer recovery {es-pull}38435[#38435] (issue: {es-issue}37165[#37165]) * Lift retention lease expiration to index shard {es-pull}38380[#38380] (issues: {es-issue}37165[#37165], {es-issue}37963[#37963], {es-issue}38070[#38070]) * Introduce retention lease background sync {es-pull}38262[#38262] (issue: {es-issue}37165[#37165]) * Allow shards of closed indices to be replicated as regular shards {es-pull}38024[#38024] (issue: {es-issue}33888[#33888]) * Expose retention leases in shard stats {es-pull}37991[#37991] (issue: {es-issue}37165[#37165]) * Introduce retention leases versioning {es-pull}37951[#37951] (issue: {es-issue}37165[#37165]) * Soft-deletes policy should always fetch latest leases {es-pull}37940[#37940] (issues: {es-issue}37165[#37165], {es-issue}37375[#37375]) * Sync retention leases on expiration {es-pull}37902[#37902] (issue: {es-issue}37165[#37165]) * Ignore shard started requests when primary term does not match {es-pull}37899[#37899] (issue: {es-issue}33888[#33888]) * Move update and delete by query to use seq# for optimistic concurrency control {es-pull}37857[#37857] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148], {es-issue}37639[#37639]) * Introduce retention lease serialization {es-pull}37447[#37447] (issues: {es-issue}37165[#37165], {es-issue}37398[#37398]) * Add run under primary permit method {es-pull}37440[#37440] (issue: {es-issue}37398[#37398]) * Introduce retention lease syncing {es-pull}37398[#37398] (issue: {es-issue}37165[#37165]) * Introduce retention lease persistence {es-pull}37375[#37375] (issue: {es-issue}37165[#37165]) * Add validation for retention lease construction {es-pull}37312[#37312] (issue: {es-issue}37165[#37165]) * Introduce retention lease expiration {es-pull}37195[#37195] (issue: {es-issue}37165[#37165]) * Introduce shard history retention leases {es-pull}37167[#37167] (issue: {es-issue}37165[#37165]) * [Close Index API] Add unique UUID to ClusterBlock {es-pull}36775[#36775] * [Close Index API] Mark shard copy as stale if needed during shard verification {es-pull}36755[#36755] * [Close Index API] Propagate tasks ids between Freeze, Close and Verify Shard actions {es-pull}36630[#36630] * Always initialize the global checkpoint {es-pull}34381[#34381] Engine:: * Specialize pre-closing checks for engine implementations {es-pull}38702[#38702] * Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines {es-pull}37426[#37426] * Enable Bulk-Merge if all source remains {es-pull}37269[#37269] * Rename setting to enable mmap {es-pull}37070[#37070] (issue: {es-issue}36668[#36668]) * Add hybridfs store type {es-pull}36668[#36668] * Introduce time-based retention policy for soft-deletes {es-pull}34943[#34943] (issue: {es-issue}34908[#34908]) * handle AsyncAfterWriteAction failure on primary in the same way as failures on replicas {es-pull}31969[#31969] (issues: {es-issue}31716[#31716], {es-issue}31755[#31755]) Features/CAT APIs:: * Expose `search.throttled` on `_cat/indices` {es-pull}37073[#37073] (issue: {es-issue}34352[#34352]) Features/Features:: * Run Node deprecation checks locally (#38065) {es-pull}38250[#38250] (issue: {es-issue}38065[#38065]) Features/ILM:: * Ensure ILM policies run safely on leader indices {es-pull}38140[#38140] (issue: {es-issue}34648[#34648]) * Skip Shrink when numberOfShards not changed {es-pull}37953[#37953] (issue: {es-issue}33275[#33275]) * Inject Unfollow before Rollover and Shrink {es-pull}37625[#37625] (issue: {es-issue}34648[#34648]) * Add set_priority action to ILM {es-pull}37397[#37397] (issue: {es-issue}36905[#36905]) * [ILM] Add Freeze Action {es-pull}36910[#36910] (issue: {es-issue}34630[#34630]) Features/Indices APIs:: * New mapping signature and mapping string source fixed. {es-pull}37401[#37401] Features/Ingest:: * ingest: compile mustache template only if field includes '{{'' {es-pull}37207[#37207] (issue: {es-issue}37120[#37120]) * Move ingest-geoip default databases out of config {es-pull}36949[#36949] (issue: {es-issue}36898[#36898]) * Make the ingest-geoip databases even lazier to load {es-pull}36679[#36679] * Updates the grok patterns to be consistent with the logstash {es-pull}27181[#27181] Features/Java High Level REST Client:: * HLRC: Fix strict setting exception handling {es-pull}37247[#37247] (issue: {es-issue}37090[#37090]) * HLRC: Use nonblocking entity for requests {es-pull}32249[#32249] Features/Monitoring:: * Adding mapping for hostname field {es-pull}37288[#37288] Features/Stats:: * Stats: Add JVM dns cache expiration config to JvmInfo {es-pull}36372[#36372] Features/Watcher:: * Move watcher to use seq# and primary term for concurrency control {es-pull}37977[#37977] (issues: {es-issue}10708[#10708], {es-issue}37872[#37872]) * Use ILM for Watcher history deletion {es-pull}37443[#37443] (issue: {es-issue}32041[#32041]) * Watcher: Add whitelist to HttpClient {es-pull}36817[#36817] (issue: {es-issue}29937[#29937]) Infra/Core:: * fix a few versionAdded values in ElasticsearchExceptions {es-pull}37877[#37877] * Add simple method to write collection of writeables {es-pull}37448[#37448] (issue: {es-issue}37398[#37398]) * Date/Time parsing: Use java time API instead of exception handling {es-pull}37222[#37222] * [API] spelling: interruptible {es-pull}37049[#37049] (issue: {es-issue}37035[#37035]) Infra/Logging:: * Trim the JSON source in indexing slow logs {es-pull}38081[#38081] (issue: {es-issue}38080[#38080]) * Optimize warning header de-duplication {es-pull}37725[#37725] (issues: {es-issue}35754[#35754], {es-issue}37530[#37530], {es-issue}37597[#37597], {es-issue}37622[#37622]) * Remove warn-date from warning headers {es-pull}37622[#37622] (issues: {es-issue}35754[#35754], {es-issue}37530[#37530], {es-issue}37597[#37597]) * Add some deprecation optimizations {es-pull}37597[#37597] (issues: {es-issue}35754[#35754], {es-issue}37530[#37530]) * Only update response headers if we have a new one {es-pull}37590[#37590] (issues: {es-issue}35754[#35754], {es-issue}37530[#37530]) Infra/Packaging:: * Add OS/architecture classifier to distributions {es-pull}37881[#37881] * Change file descriptor limit to 65535 {es-pull}37537[#37537] (issue: {es-issue}35839[#35839]) * Exit batch files explictly using ERRORLEVEL {es-pull}29583[#29583] (issue: {es-issue}29582[#29582]) Infra/Scripting:: * Add getZone to JodaCompatibleZonedDateTime {es-pull}37084[#37084] * [Painless] Add boxed type to boxed type casts for method/return {es-pull}36571[#36571] Infra/Settings:: * Separate out validation of groups of settings {es-pull}34184[#34184] License:: * Handle malformed license signatures {es-pull}37137[#37137] (issue: {es-issue}35340[#35340]) Machine Learning:: * Move ML Optimistic Concurrency Control to Seq No {es-pull}38278[#38278] (issues: {es-issue}10708[#10708], {es-issue}36148[#36148]) * [ML] Add explanation so far to file structure finder exceptions {es-pull}38191[#38191] (issue: {es-issue}29821[#29821]) * ML: Add reason field in JobTaskState {es-pull}38029[#38029] (issue: {es-issue}34431[#34431]) * [ML] Add _meta information to all ML indices {es-pull}37964[#37964] * ML: Add upgrade mode docs, hlrc, and fix bug {es-pull}37942[#37942] * [ML] Tighten up use of aliases rather than concrete indices {es-pull}37874[#37874] * ML: Add support for single bucket aggs in Datafeeds {es-pull}37544[#37544] (issue: {es-issue}36838[#36838]) * [ML] Create the ML annotations index {es-pull}36731[#36731] (issues: {es-issue}26034[#26034], {es-issue}33376[#33376]) * [ML] Merge the Jindex master feature branch {es-pull}36702[#36702] (issue: {es-issue}32905[#32905]) * [FEATURE][ML] Add cluster setting to enable/disable config migration {es-pull}36700[#36700] (issue: {es-issue}32905[#32905]) Mapping:: * Log document id when MapperParsingException occurs {es-pull}37800[#37800] (issue: {es-issue}37658[#37658]) * [API] spelling: unknown {es-pull}37056[#37056] (issue: {es-issue}37035[#37035]) * Make SourceToParse immutable {es-pull}36971[#36971] * Use index-prefix fields for terms of length min_chars - 1 {es-pull}36703[#36703] Network:: * Enable TLSv1.3 by default for JDKs with support {es-pull}38103[#38103] (issue: {es-issue}32276[#32276]) Recovery:: * SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence {es-pull}37691[#37691] (issue: {es-issue}33888[#33888]) * Make prepare engine step of recovery source non-blocking {es-pull}37573[#37573] (issue: {es-issue}37174[#37174]) * Make recovery source send operations non-blocking {es-pull}37503[#37503] (issue: {es-issue}37458[#37458]) * Prepare to make send translog of recovery non-blocking {es-pull}37458[#37458] (issue: {es-issue}37291[#37291]) * Make finalize step of recovery source non-blocking {es-pull}37388[#37388] (issue: {es-issue}37291[#37291]) * Make recovery source partially non-blocking {es-pull}37291[#37291] (issue: {es-issue}36195[#36195]) * Do not mutate RecoveryResponse {es-pull}37204[#37204] (issue: {es-issue}37174[#37174]) * Don't block on peer recovery on the target side {es-pull}37076[#37076] (issue: {es-issue}36195[#36195]) * Reduce recovery time with compress or secure transport {es-pull}36981[#36981] (issue: {es-issue}33844[#33844]) * Translog corruption marker {es-pull}33415[#33415] (issue: {es-issue}31389[#31389]) Rollup:: * Replace the TreeMap in the composite aggregation {es-pull}36675[#36675] SQL:: * SQL: Allow look-ahead resolution of aliases for WHERE clause {es-pull}38450[#38450] (issue: {es-issue}29983[#29983]) * SQL: Implement CURRENT_DATE {es-pull}38175[#38175] (issue: {es-issue}38160[#38160]) * SQL: Generate relevant error message when grouping functions are not used in GROUP BY {es-pull}38017[#38017] (issue: {es-issue}37952[#37952]) * SQL: Skip the nested and object field types in case of an ODBC request {es-pull}37948[#37948] (issue: {es-issue}37801[#37801]) * SQL: Add protocol tests and remove jdbc_type from drivers response {es-pull}37516[#37516] (issues: {es-issue}36635[#36635], {es-issue}36882[#36882]) * SQL: Remove slightly used meta commands {es-pull}37506[#37506] (issue: {es-issue}37409[#37409]) * SQL: Describe aliases as views {es-pull}37496[#37496] (issue: {es-issue}37422[#37422]) * SQL: Make `FULL` non-reserved keyword in the grammar {es-pull}37377[#37377] (issue: {es-issue}37376[#37376]) * SQL: Use declared source for error messages {es-pull}37161[#37161] * SQL: Improve error message when unable to translate to ES query DSL {es-pull}37129[#37129] (issue: {es-issue}37040[#37040]) * [API] spelling: subtract {es-pull}37055[#37055] (issue: {es-issue}37035[#37035]) * [API] spelling: similar {es-pull}37054[#37054] (issue: {es-issue}37035[#37035]) * [API] spelling: input {es-pull}37048[#37048] (issue: {es-issue}37035[#37035]) * SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg {es-pull}36933[#36933] (issue: {es-issue}36903[#36903]) * SQL: Preserve original source for each expression {es-pull}36912[#36912] (issue: {es-issue}36894[#36894]) * SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms {es-pull}36753[#36753] (issue: {es-issue}36740[#36740]) * SQL: Fix wrong appliance of StackOverflow limit for IN {es-pull}36724[#36724] (issue: {es-issue}36592[#36592]) Search:: * Tie break on cluster alias when merging shard search failures {es-pull}38715[#38715] (issue: {es-issue}38672[#38672]) * Add finalReduce flag to SearchRequest {es-pull}38104[#38104] (issues: {es-issue}37000[#37000], {es-issue}37838[#37838]) * Streamline skip_unavailable handling {es-pull}37672[#37672] (issue: {es-issue}32125[#32125]) * Expose sequence number and primary terms in search responses {es-pull}37639[#37639] * Add support for merging multiple search responses into one {es-pull}37566[#37566] (issue: {es-issue}32125[#32125]) * Allow field types to optimize phrase prefix queries {es-pull}37436[#37436] (issue: {es-issue}31921[#31921]) * Add support for providing absolute start time to SearchRequest {es-pull}37142[#37142] (issue: {es-issue}32125[#32125]) * Ensure that local cluster alias is never treated as remote {es-pull}37121[#37121] (issues: {es-issue}32125[#32125], {es-issue}36997[#36997]) * [API] spelling: cacheable {es-pull}37047[#37047] (issue: {es-issue}37035[#37035]) * Add ability to suggest shard_size on coord node rewrite {es-pull}37017[#37017] (issues: {es-issue}32125[#32125], {es-issue}36997[#36997], {es-issue}37000[#37000]) * Skip final reduction if SearchRequest holds a cluster alias {es-pull}37000[#37000] (issues: {es-issue}32125[#32125], {es-issue}36997[#36997]) * Add support for local cluster alias to SearchRequest {es-pull}36997[#36997] (issue: {es-issue}32125[#32125]) * Use SearchRequest copy constructor in ExpandSearchPhase {es-pull}36772[#36772] (issue: {es-issue}36641[#36641]) * Add raw sort values to SearchSortValues transport serialization {es-pull}36617[#36617] (issue: {es-issue}32125[#32125]) Security:: * Move CAS operations in TokenService to sequence numbers {es-pull}38311[#38311] (issues: {es-issue}10708[#10708], {es-issue}37872[#37872]) * Cleanup construction of interceptors {es-pull}38294[#38294] * Add passphrase support to elasticsearch-keystore {es-pull}37472[#37472] (issue: {es-issue}32691[#32691]) Snapshot/Restore:: * RestoreService should update primary terms when restoring shards of existing indices {es-pull}38177[#38177] (issue: {es-issue}33888[#33888]) * Allow open indices to be restored {es-pull}37733[#37733] * Create specific exception for when snapshots are in progress {es-pull}37550[#37550] (issue: {es-issue}37541[#37541]) * SNAPSHOT: Make Atomic Blob Writes Mandatory {es-pull}37168[#37168] (issues: {es-issue}37011[#37011], {es-issue}37066[#37066]) * SNAPSHOT: Speed up HDFS Repository Writes {es-pull}37069[#37069] * Implement Atomic Blob Writes for HDFS Repository {es-pull}37066[#37066] (issue: {es-issue}37011[#37011]) * [API] spelling: repositories {es-pull}37053[#37053] (issue: {es-issue}37035[#37035]) * SNAPSHOT: Use CancellableThreads to Abort {es-pull}35901[#35901] (issue: {es-issue}21759[#21759]) * WIP: S3 client encryption {es-pull}30513[#30513] (issues: {es-issue}11128[#11128], {es-issue}16843[#16843]) Suggesters:: * Remove unused empty constructors from suggestions classes {es-pull}37295[#37295] * [API] spelling: likelihood {es-pull}37052[#37052] (issue: {es-issue}37035[#37035]) ZenDiscovery:: * Add elasticsearch-node detach-cluster tool {es-pull}37979[#37979] * Deprecate minimum_master_nodes {es-pull}37868[#37868] * Step down as master when configured out of voting configuration {es-pull}37802[#37802] (issue: {es-issue}37712[#37712]) * Enforce cluster UUIDs {es-pull}37775[#37775] * Bubble exceptions up in ClusterApplierService {es-pull}37729[#37729] * Use m_m_nodes from Zen1 master for Zen2 bootstrap {es-pull}37701[#37701] * Add tool elasticsearch-node unsafe-bootstrap {es-pull}37696[#37696] * Report terms and version if cluster does not form {es-pull}37473[#37473] * Bootstrap a Zen2 cluster once quorum is discovered {es-pull}37463[#37463] * Zen2: Add join validation {es-pull}37203[#37203] * Publish cluster states in chunks {es-pull}36973[#36973] [[bug-7.0.0-beta1]] [float] === Bug fixes Aggregations:: * Don't load global ordinals with the `map` execution_hint {es-pull}37833[#37833] (issue: {es-issue}37705[#37705]) * Issue #37303 - Invalid variance fix {es-pull}37384[#37384] (issue: {es-issue}37303[#37303]) Allocation:: * Fix _host based require filters {es-pull}38173[#38173] * ALLOC: Fail Stale Primary Alloc. Req. without Data {es-pull}37226[#37226] (issue: {es-issue}37098[#37098]) Audit:: * Fix NPE in Logfile Audit Filter {es-pull}38120[#38120] (issue: {es-issue}38097[#38097]) Authentication:: * Enhance parsing of StatusCode in SAML Responses {es-pull}38628[#38628] * Limit token expiry to 1 hour maximum {es-pull}38244[#38244] * Fix expired token message in Exception header {es-pull}37196[#37196] * Fix NPE in CachingUsernamePasswordRealm {es-pull}36953[#36953] (issue: {es-issue}36951[#36951]) CCR:: * Prevent CCR recovery from missing documents {es-pull}38237[#38237] * Fix file reading in ccr restore service {es-pull}38117[#38117] * Correct argument names in update mapping/settings from leader {es-pull}38063[#38063] * Ensure changes requests return the latest mapping version {es-pull}37633[#37633] * Do not set fatal exception when shard follow task is stopped. {es-pull}37603[#37603] * Add fatal_exception field for ccr stats in monitoring mapping {es-pull}37563[#37563] * Do not add index event listener if CCR disabled {es-pull}37432[#37432] * When removing an AutoFollower also mark it as removed. {es-pull}37402[#37402] (issue: {es-issue}36761[#36761]) * [CCR] Make shard follow tasks more resilient for restarts {es-pull}37239[#37239] (issue: {es-issue}37231[#37231]) * [CCR] Resume follow Api should not require a request body {es-pull}37217[#37217] (issue: {es-issue}37022[#37022]) * [CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled {es-pull}36886[#36886] (issue: {es-issue}33007[#33007]) * Remote cluster license checker and no license info. {es-pull}36837[#36837] (issue: {es-issue}36815[#36815]) * Make CCR resilient against missing remote cluster connections {es-pull}36682[#36682] (issues: {es-issue}36255[#36255], {es-issue}36667[#36667]) * [CCR] AutoFollowCoordinator and follower index already created {es-pull}36540[#36540] (issue: {es-issue}33007[#33007]) CRUD:: * Fix Reindex from remote query logic {es-pull}36908[#36908] * Synchronize WriteReplicaResult callbacks {es-pull}36770[#36770] Distributed:: * TransportVerifyShardBeforeCloseAction should force a flush {es-pull}38401[#38401] (issues: {es-issue}33888[#33888], {es-issue}37961[#37961]) * Fix limit on retaining sequence number {es-pull}37992[#37992] (issue: {es-issue}37165[#37165]) * Close Index API should force a flush if a sync is needed {es-pull}37961[#37961] (issues: {es-issue}33888[#33888], {es-issue}37426[#37426]) * Force Refresh Listeners when Acquiring all Operation Permits {es-pull}36835[#36835] * Replaced the word 'shards' with 'replicas' in an error message. (#36234) {es-pull}36275[#36275] (issue: {es-issue}36234[#36234]) Engine:: * Subclass NIOFSDirectory instead of using FileSwitchDirectory {es-pull}37140[#37140] (issues: {es-issue}36668[#36668], {es-issue}37111[#37111]) Features/ILM:: * Preserve ILM operation mode when creating new lifecycles {es-pull}38134[#38134] (issues: {es-issue}38229[#38229], {es-issue}38230[#38230]) * Retry ILM steps that fail due to SnapshotInProgressException {es-pull}37624[#37624] (issues: {es-issue}37541[#37541], {es-issue}37552[#37552]) * Remove `indexing_complete` when removing policy {es-pull}36620[#36620] Features/Indices APIs:: * Reject delete index requests with a body {es-pull}37501[#37501] (issue: {es-issue}8217[#8217]) * Fix duplicate phrase in shrink/split error message {es-pull}36734[#36734] (issue: {es-issue}36729[#36729]) * Get Aliases with wildcard exclusion expression {es-pull}34230[#34230] (issues: {es-issue}33518[#33518], {es-issue}33805[#33805], {es-issue}34144[#34144]) Features/Ingest:: * Support unknown fields in ingest pipeline map configuration {es-pull}38352[#38352] (issue: {es-issue}36938[#36938]) * Ingest node - user_agent, move device parsing to an object {es-pull}38115[#38115] (issues: {es-issue}37329[#37329], {es-issue}38094[#38094]) * ingest: fix on_failure with Drop processor {es-pull}36686[#36686] (issue: {es-issue}36151[#36151]) * ingest: support default pipelines + bulk upserts {es-pull}36618[#36618] (issue: {es-issue}36219[#36219]) Features/Java High Level REST Client:: * Update IndexTemplateMetadata to allow unknown fields {es-pull}38448[#38448] (issue: {es-issue}36938[#36938]) * `if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API {es-pull}38411[#38411] * Update Rollup Caps to allow unknown fields {es-pull}38339[#38339] (issue: {es-issue}36938[#36938]) * Fix ILM explain response to allow unknown fields {es-pull}38054[#38054] (issue: {es-issue}36938[#36938]) * Fix ILM status to allow unknown fields {es-pull}38043[#38043] (issue: {es-issue}36938[#36938]) * Fix ILM Lifecycle Policy to allow unknown fields {es-pull}38041[#38041] (issue: {es-issue}36938[#36938]) * Update authenticate to allow unknown fields {es-pull}37713[#37713] (issue: {es-issue}36938[#36938]) * Update verify repository to allow unknown fields {es-pull}37619[#37619] (issue: {es-issue}36938[#36938]) * Update get users to allow unknown fields {es-pull}37593[#37593] (issue: {es-issue}36938[#36938]) * Update Execute Watch to allow unknown fields {es-pull}37498[#37498] (issue: {es-issue}36938[#36938]) * Update Put Watch to allow unknown fields {es-pull}37494[#37494] (issue: {es-issue}36938[#36938]) * Update Delete Watch to allow unknown fields {es-pull}37435[#37435] (issue: {es-issue}36938[#36938]) * Fix rest reindex test for IPv4 addresses {es-pull}37310[#37310] * Fix weighted_avg parser not found for RestHighLevelClient {es-pull}37027[#37027] (issue: {es-issue}36861[#36861]) Features/Java Low Level REST Client:: * Fix potential IllegalCapacityException in LLRC when selecting nodes {es-pull}37821[#37821] Features/Monitoring:: * Allow built-in monitoring_user role to call GET _xpack API {es-pull}38060[#38060] (issue: {es-issue}37970[#37970]) Features/Watcher:: * Support merge nested Map in list for JIRA configurations {es-pull}37634[#37634] (issue: {es-issue}30068[#30068]) * Watcher accounts constructed lazily {es-pull}36656[#36656] * Ensures watch definitions are valid json {es-pull}30692[#30692] (issue: {es-issue}29746[#29746]) Geo:: * Fix GeoHash PrefixTree BWC {es-pull}38584[#38584] (issue: {es-issue}38494[#38494]) * Geo: Do not normalize the longitude with value -180 for Lucene shapes {es-pull}37299[#37299] (issue: {es-issue}37297[#37297]) Infra/Core:: * Bubble-up exceptions from scheduler {es-pull}38317[#38317] (issue: {es-issue}38014[#38014]) * Core: Revert back to joda's multi date formatters {es-pull}36814[#36814] (issues: {es-issue}36447[#36447], {es-issue}36602[#36602]) * Propagate Errors in executors to uncaught exception handler {es-pull}36137[#36137] (issue: {es-issue}28667[#28667]) Infra/Packaging:: * Remove NOREPLACE for /etc/elasticsearch in rpm and deb {es-pull}37839[#37839] * Packaging: Update marker used to allow ELASTIC_PASSWORD {es-pull}37243[#37243] (issue: {es-issue}37240[#37240]) * Packaging: Remove permission editing in postinst {es-pull}37242[#37242] (issue: {es-issue}37143[#37143]) Infra/REST API:: * Reject all requests that have an unconsumed body {es-pull}37504[#37504] (issues: {es-issue}30792[#30792], {es-issue}37501[#37501], {es-issue}8217[#8217]) Infra/Scripting:: * Fix Painless void return bug {es-pull}38046[#38046] Infra/Settings:: * Fix setting by time unit {es-pull}37192[#37192] * Fix handling of fractional byte size value settings {es-pull}37172[#37172] * Fix handling of fractional time value settings {es-pull}37171[#37171] Machine Learning:: * [ML] Report index unavailable instead of waiting for lazy node {es-pull}38423[#38423] * ML: Fix error race condition on stop _all datafeeds and close _all jobs {es-pull}38113[#38113] (issue: {es-issue}37959[#37959]) * [ML] Update ML results mappings on process start {es-pull}37706[#37706] (issue: {es-issue}37607[#37607]) * [ML] Prevent submit after autodetect worker is stopped {es-pull}37700[#37700] (issue: {es-issue}37108[#37108]) * [ML] Fix ML datafeed CCS with wildcarded cluster name {es-pull}37470[#37470] (issue: {es-issue}36228[#36228]) * [ML] Update error message for process update {es-pull}37363[#37363] * [ML] Wait for autodetect to be ready in the datafeed {es-pull}37349[#37349] (issues: {es-issue}36810[#36810], {es-issue}37227[#37227]) * [ML] Stop datafeeds running when their jobs are stale {es-pull}37227[#37227] (issue: {es-issue}36810[#36810]) * [ML] Order GET job stats response by job id {es-pull}36841[#36841] (issue: {es-issue}36683[#36683]) * [ML] Make GetJobStats work with arbitrary wildcards and groups {es-pull}36683[#36683] (issue: {es-issue}34745[#34745]) Mapping:: * Treat put-mapping calls with `_doc` as a top-level key as typed calls. {es-pull}38032[#38032] * Correct deprec log in RestGetFieldMappingAction {es-pull}37843[#37843] (issue: {es-issue}37667[#37667]) * Restore a noop _all metadata field for 6x indices {es-pull}37808[#37808] (issue: {es-issue}37429[#37429]) * Make sure PutMappingRequest accepts content types other than JSON. {es-pull}37720[#37720] * Make sure to use the resolved type in DocumentMapperService#extractMappings. {es-pull}37451[#37451] (issue: {es-issue}36811[#36811]) * MAPPING: Improve Precision for scaled_float {es-pull}37169[#37169] (issue: {es-issue}32570[#32570]) * Make sure to accept empty unnested mappings in create index requests. {es-pull}37089[#37089] * Stop automatically nesting mappings in index creation requests. {es-pull}36924[#36924] * Rewrite SourceToParse with resolved docType {es-pull}36921[#36921] (issues: {es-issue}35790[#35790], {es-issue}36769[#36769]) Network:: * Reload SSL context on file change for LDAP {es-pull}36937[#36937] (issues: {es-issue}30509[#30509], {es-issue}36923[#36923]) * Do not resolve addresses in remote connection info {es-pull}36671[#36671] (issue: {es-issue}35658[#35658]) Ranking:: * QueryRescorer should keep the window size when rewriting {es-pull}36836[#36836] Recovery:: * RecoveryMonitor#lastSeenAccessTime should be volatile {es-pull}36781[#36781] Rollup:: * Fix Rollup's metadata parser {es-pull}36791[#36791] (issue: {es-issue}36726[#36726]) * Fix rollup search statistics {es-pull}36674[#36674] SQL:: * SQL: Prevent grouping over grouping functions {es-pull}38649[#38649] (issue: {es-issue}38308[#38308]) * SQL: Relax StackOverflow circuit breaker for constants {es-pull}38572[#38572] (issue: {es-issue}38571[#38571]) * SQL: Fix issue with IN not resolving to underlying keyword field {es-pull}38440[#38440] (issue: {es-issue}38424[#38424]) * SQL: change the Intervals milliseconds precision to 3 digits {es-pull}38297[#38297] (issue: {es-issue}37423[#37423]) * SQL: Fix esType for DATETIME/DATE and INTERVALS {es-pull}38179[#38179] (issue: {es-issue}38051[#38051]) * SQL: Added SSL configuration options tests {es-pull}37875[#37875] (issue: {es-issue}37711[#37711]) * SQL: Fix casting from date to numeric type to use millis {es-pull}37869[#37869] (issue: {es-issue}37655[#37655]) * SQL: Fix BasicFormatter NPE {es-pull}37804[#37804] * SQL: Return Intervals in SQL format for CLI {es-pull}37602[#37602] (issues: {es-issue}29970[#29970], {es-issue}36186[#36186], {es-issue}36432[#36432]) * SQL: fix object extraction from sources {es-pull}37502[#37502] (issue: {es-issue}37364[#37364]) * SQL: Fix issue with field names containing "." {es-pull}37364[#37364] (issue: {es-issue}37128[#37128]) * SQL: Fix bug regarding alias fields with dots {es-pull}37279[#37279] (issue: {es-issue}37224[#37224]) * SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {es-pull}37254[#37254] (issue: {es-issue}30285[#30285]) * SQL: fix COUNT DISTINCT filtering {es-pull}37176[#37176] (issue: {es-issue}37086[#37086]) * SQL: Fix issue with wrong NULL optimization {es-pull}37124[#37124] (issue: {es-issue}35872[#35872]) * SQL: Fix issue with complex expression as args of PERCENTILE/_RANK {es-pull}37102[#37102] (issue: {es-issue}37099[#37099]) * SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless {es-pull}37024[#37024] (issue: {es-issue}37023[#37023]) * SQL: Fix bug regarding histograms usage in scripting {es-pull}36866[#36866] * SQL: Fix issue with always false filter involving functions {es-pull}36830[#36830] (issue: {es-issue}35980[#35980]) * SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {es-pull}36800[#36800] (issue: {es-issue}36756[#36756]) * SQL: Enhance Verifier to prevent aggregate or grouping functions from {es-pull}36799[#36799] (issue: {es-issue}36798[#36798]) * SQL: Fix translation of LIKE/RLIKE keywords {es-pull}36672[#36672] (issues: {es-issue}36039[#36039], {es-issue}36584[#36584]) * SQL: Scripting support for casting functions CAST and CONVERT {es-pull}36640[#36640] (issue: {es-issue}36061[#36061]) * SQL: Concat should be always not nullable {es-pull}36601[#36601] (issue: {es-issue}36169[#36169]) * SQL: Fix issue with complex HAVING and GROUP BY ordinal {es-pull}36594[#36594] (issue: {es-issue}36059[#36059]) Search:: * Look up connection using the right cluster alias when releasing contexts {es-pull}38570[#38570] * Fix fetch source option in expand search phase {es-pull}37908[#37908] (issue: {es-issue}23829[#23829]) * Change `rational` to `saturation` in script_score {es-pull}37766[#37766] (issue: {es-issue}37714[#37714]) * Throw if two inner_hits have the same name {es-pull}37645[#37645] (issue: {es-issue}37584[#37584]) * Ensure either success or failure path for SearchOperationListener is called {es-pull}37467[#37467] (issue: {es-issue}37185[#37185]) * `query_string` should use indexed prefixes {es-pull}36895[#36895] * Avoid duplicate types deprecation messages in search-related APIs. {es-pull}36802[#36802] Security:: * Fix exit code for Security CLI tools {es-pull}37956[#37956] (issue: {es-issue}37841[#37841]) * Fix potential NPE in UsersTool {es-pull}37660[#37660] Snapshot/Restore:: * Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {es-pull}38368[#38368] (issue: {es-issue}38226[#38226]) * Fix Two Races that Lead to Stuck Snapshots {es-pull}37686[#37686] (issues: {es-issue}32265[#32265], {es-issue}32348[#32348]) * Fix Race in Concurrent Snapshot Delete and Create {es-pull}37612[#37612] (issue: {es-issue}37581[#37581]) * Streamline S3 Repository- and Client-Settings {es-pull}37393[#37393] Suggesters:: * Fix duplicate removal when merging completion suggestions {es-pull}36996[#36996] (issue: {es-issue}35836[#35836]) Task Management:: * Un-assign persistent tasks as nodes exit the cluster {es-pull}37656[#37656] ZenDiscovery:: * Fix size of rolling-upgrade bootstrap config {es-pull}38031[#38031] * Always return metadata version if metadata is requested {es-pull}37674[#37674] * [Zen2] Elect freshest master in upgrade {es-pull}37122[#37122] (issue: {es-issue}40[#40]) * Fix cluster state persistence for single-node discovery {es-pull}36825[#36825] [[regression-7.0.0-beta1]] [float] === Regressions Infra/Core:: * Restore date aggregation performance in UTC case {es-pull}38221[#38221] (issue: {es-issue}37826[#37826]) * Speed up converting of temporal accessor to zoned date time {es-pull}37915[#37915] (issue: {es-issue}37826[#37826]) Mapping:: * Performance fix. Reduce deprecation calls for the same bulk request {es-pull}37415[#37415] (issue: {es-issue}37411[#37411]) [[upgrade-7.0.0-beta1]] [float] === Upgrades Engine:: * Upgrade to lucene-8.0.0-snapshot-83f9835. {es-pull}37668[#37668] Machine Learning:: * [ML] No need to add state doc mapping on job open in 7.x {es-pull}37759[#37759]