[[release-notes-7.0.0-beta1]] == 7.0.0-beta1 Release Notes coming[7.0.0-beta1] Also see <>. [[breaking-7.0.0-beta1]] [float] === Breaking changes Audit:: * Remove index audit output type {pull}37707[#37707] (issues: {issue}29881[#29881], {issue}37301[#37301]) Authentication:: * Remove bwc logic for token invalidation {pull}36893[#36893] (issue: {issue}36727[#36727]) Authorization:: * Remove implicit index monitor privilege {pull}37774[#37774] CCR:: * Follow stats api should return a 404 when requesting stats for a non existing index {pull}37220[#37220] (issue: {issue}37021[#37021]) CRUD:: * Remove support for internal versioning for concurrency control {pull}38254[#38254] (issue: {issue}1078[#1078]) Features/Ingest:: * Add ECS schema for user-agent ingest processor (#37727) {pull}37984[#37984] (issues: {issue}37329[#37329], {issue}37727[#37727]) * Remove special handling for ingest plugins {pull}36967[#36967] (issues: {issue}36898[#36898], {issue}36956[#36956]) Features/Java Low Level REST Client:: * Drop support for the low-level REST client on JDK 7 {pull}38540[#38540] (issue: {issue}29607[#29607]) Features/Watcher:: * Remove Watcher Account "unsecure" settings {pull}36736[#36736] (issue: {issue}36403[#36403]) Infra/Logging:: * Elasticsearch json logging {pull}36833[#36833] (issue: {issue}32850[#32850]) Infra/Packaging:: * Package ingest-user-agent as a module {pull}36956[#36956] * Package ingest-geoip as a module {pull}36898[#36898] Machine Learning:: * [ML] Remove types from datafeed {pull}36538[#36538] (issue: {issue}34265[#34265]) Mapping:: * Make sure to reject mappings with type _doc when include_type_name is false. {pull}38270[#38270] (issue: {issue}38266[#38266]) * Update the default for include_type_name to false. {pull}37285[#37285] * Support 'include_type_name' in RestGetIndicesAction {pull}37149[#37149] Network:: * Remove TLS 1.0 as a default SSL protocol {pull}37512[#37512] (issue: {issue}36021[#36021]) * Security: remove SSL settings fallback {pull}36846[#36846] (issue: {issue}29797[#29797]) Ranking:: * Forbid negative field boosts in analyzed queries {pull}37930[#37930] (issue: {issue}33309[#33309]) Search:: * Track total hits up to 10,000 by default {pull}37466[#37466] (issue: {issue}33028[#33028]) * Use mappings to format doc-value fields by default. {pull}30831[#30831] (issues: {issue}26948[#26948], {issue}29639[#29639]) Security:: * Remove heuristics that enable security on trial licenses {pull}38075[#38075] (issue: {issue}38009[#38009]) ZenDiscovery:: * Remove DiscoveryPlugin#getDiscoveryTypes {pull}38414[#38414] (issue: {issue}38410[#38410]) [[breaking-java-7.0.0-beta1]] [float] === Breaking Java changes Features/Java Low Level REST Client:: * Remove support for maxRetryTimeout from low-level REST client {pull}38085[#38085] (issues: {issue}25951[#25951], {issue}31834[#31834], {issue}33342[#33342]) Infra/Core:: * Handle scheduler exceptions {pull}38014[#38014] (issues: {issue}28667[#28667], {issue}36137[#36137], {issue}37708[#37708]) [[deprecation-7.0.0-beta1]] [float] === Deprecations Aggregations:: * Deprecate dots in aggregation names {pull}31468[#31468] (issues: {issue}17600[#17600], {issue}19040[#19040]) Analysis:: * [Analysis] Deprecate Standard Html Strip Analyzer in master {pull}26719[#26719] (issue: {issue}4704[#4704]) Audit:: * Deprecate index audit output type {pull}37301[#37301] (issue: {issue}29881[#29881]) Features/Indices APIs:: * Reject setting index.optimize_auto_generated_id after version 7.0.0 {pull}28895[#28895] (issue: {issue}27600[#27600]) Features/Ingest:: * Deprecate `_type` in simulate pipeline requests {pull}37949[#37949] (issue: {issue}37731[#37731]) Features/Java High Level REST Client:: * Deprecate HLRC security methods {pull}37883[#37883] (issues: {issue}36938[#36938], {issue}37540[#37540]) * Deprecate HLRC EmptyResponse used by security {pull}37540[#37540] (issue: {issue}36938[#36938]) Features/Watcher:: * Deprecate xpack.watcher.history.cleaner_service.enabled {pull}37782[#37782] (issue: {issue}32041[#32041]) * deprecate types for watcher {pull}37594[#37594] (issue: {issue}35190[#35190]) Infra/Core:: * Core: Deprecate negative epoch timestamps {pull}36793[#36793] * Core: Deprecate use of scientific notation in epoch time parsing {pull}36691[#36691] Infra/Scripting:: * Add types deprecation to script contexts {pull}37554[#37554] * Deprecate _type from LeafDocLookup {pull}37491[#37491] * Scripting: Remove deprecated params.ctx {pull}36848[#36848] (issue: {issue}34059[#34059]) Machine Learning:: * Adding ml_settings entry to HLRC and Docs for deprecation_info {pull}38118[#38118] * [ML] Datafeed deprecation checks {pull}38026[#38026] (issue: {issue}37932[#37932]) * [ML] Remove "8" prefixes from file structure finder timestamp formats {pull}38016[#38016] * [ML] Adjust structure finder for Joda to Java time migration {pull}37306[#37306] * [ML] Resolve 7.0.0 TODOs in ML code {pull}36842[#36842] (issue: {issue}29963[#29963]) Mapping:: * Deprecate types in rollover index API {pull}38039[#38039] (issue: {issue}35190[#35190]) * Deprecate types in get field mapping API {pull}37667[#37667] (issue: {issue}35190[#35190]) * Deprecate types in the put mapping API. {pull}37280[#37280] (issues: {issue}29453[#29453], {issue}37285[#37285]) * Support include_type_name in the field mapping and index template APIs. {pull}37210[#37210] * Deprecate types in create index requests. {pull}37134[#37134] (issues: {issue}29453[#29453], {issue}37285[#37285]) * Deprecate use of the _type field in aggregations. {pull}37131[#37131] (issue: {issue}36802[#36802]) * Deprecate reference to _type in lookup queries {pull}37016[#37016] (issue: {issue}35190[#35190]) * Deprecate the document create endpoint. {pull}36863[#36863] * Deprecate types in index API {pull}36575[#36575] (issues: {issue}35190[#35190], {issue}35790[#35790]) * Deprecate types in update APIs {pull}36225[#36225] Search:: * Deprecate use of type in reindex request body {pull}36823[#36823] * Add typless endpoints for get_source and exist_source {pull}36426[#36426] [[feature-7.0.0-beta1]] [float] === New features Authentication:: * Add support for API keys to access Elasticsearch {pull}38291[#38291] (issue: {issue}34383[#34383]) * OIDC realm authentication flows {pull}37787[#37787] * [WIP] OIDC Realm JWT+JWS related functionality {pull}37272[#37272] (issues: {issue}35339[#35339], {issue}37009[#37009]) * OpenID Connect Realm base functionality {pull}37009[#37009] (issue: {issue}35339[#35339]) Authorization:: * Allow custom authorization with an authorization engine {pull}38358[#38358] (issues: {issue}32435[#32435], {issue}36245[#36245], {issue}37328[#37328], {issue}37495[#37495], {issue}37785[#37785], {issue}38137[#38137], {issue}38219[#38219]) * WIldcard IndicesPermissions don't cover .security {pull}36765[#36765] CCR:: * Add ccr follow info api {pull}37408[#37408] (issue: {issue}37127[#37127]) Features/ILM:: * [ILM] Add unfollow action {pull}36970[#36970] (issue: {issue}34648[#34648]) Geo:: * geotile_grid implementation {pull}37842[#37842] (issue: {issue}30240[#30240]) * [GEO] Fork Lucene's LatLonShape Classes to local lucene package {pull}36794[#36794] * [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}36751[#36751] (issue: {issue}35320[#35320]) * [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}35320[#35320] (issue: {issue}32039[#32039]) Machine Learning:: * ML: Adds set_upgrade_mode API endpoint {pull}37837[#37837] Mapping:: * Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. {pull}37871[#37871] (issue: {issue}37773[#37773]) * Add nanosecond field mapper {pull}37755[#37755] (issues: {issue}27330[#27330], {issue}32601[#32601]) SQL:: * SQL: Allow sorting of groups by aggregates {pull}38042[#38042] (issue: {issue}35118[#35118]) * SQL: Implement FIRST/LAST aggregate functions {pull}37936[#37936] (issue: {issue}35639[#35639]) * SQL: Introduce SQL DATE data type {pull}37693[#37693] (issue: {issue}37340[#37340]) Search:: * Introduce ability to minimize round-trips in CCS {pull}37828[#37828] (issues: {issue}32125[#32125], {issue}37566[#37566]) * Add script filter to intervals {pull}36776[#36776] * Add the ability to set the number of hits to track accurately {pull}36357[#36357] (issue: {issue}33028[#33028]) * Add a maximum search request size. {pull}26423[#26423] [[enhancement-7.0.0-beta1]] [float] === Enhancements Aggregations:: * Add Composite to AggregationBuilders {pull}38207[#38207] (issue: {issue}38020[#38020]) * Allow nested fields in the composite aggregation {pull}37178[#37178] (issue: {issue}28611[#28611]) * Remove single shard optimization when suggesting shard_size {pull}37041[#37041] (issue: {issue}32125[#32125]) * Use List instead of priority queue for stable sorting in bucket sort aggregator {pull}36748[#36748] (issue: {issue}36322[#36322]) * Keys are compared in BucketSortPipelineAggregation so making key type… {pull}36407[#36407] Allocation:: * Fail start on obsolete indices documentation {pull}37786[#37786] (issue: {issue}27073[#27073]) * Fail start on invalid index metadata {pull}37748[#37748] (issue: {issue}27073[#27073]) * Fail start of non-data node if node has data {pull}37347[#37347] (issue: {issue}27073[#27073]) Analysis:: * Allow word_delimiter_graph_filter to not adjust internal offsets {pull}36699[#36699] (issues: {issue}33710[#33710], {issue}34741[#34741]) Audit:: * Security Audit includes HTTP method for requests {pull}37322[#37322] (issue: {issue}29765[#29765]) * Add X-Forwarded-For to the logfile audit {pull}36427[#36427] Authentication:: * Security: propagate auth result to listeners {pull}36900[#36900] (issue: {issue}30794[#30794]) * Security: reorder realms based on last success {pull}36878[#36878] * Improve error message for 6.x style realm settings {pull}36876[#36876] (issues: {issue}30241[#30241], {issue}36026[#36026]) * Change missing authn message to not mention tokens {pull}36750[#36750] * Invalidate Token API enhancements - HLRC {pull}36362[#36362] (issue: {issue}35388[#35388]) * Enhance Invalidate Token API {pull}35388[#35388] (issues: {issue}34556[#34556], {issue}35115[#35115]) Authorization:: * Add apm_user reserved role {pull}38206[#38206] * Permission for restricted indices {pull}37577[#37577] (issue: {issue}34454[#34454]) * Remove kibana_user and kibana_dashboard_only_user index privileges {pull}37441[#37441] * Create snapshot role {pull}35820[#35820] (issue: {issue}34454[#34454]) CCR:: * Concurrent file chunk fetching for CCR restore {pull}38495[#38495] * Tighten mapping syncing in ccr remote restore {pull}38071[#38071] (issues: {issue}36879[#36879], {issue}37887[#37887]) * Do not allow put mapping on follower {pull}37675[#37675] (issue: {issue}30086[#30086]) * Added ccr to xpack usage infrastructure {pull}37256[#37256] (issue: {issue}37221[#37221]) * [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned {pull}37213[#37213] * [CCR] Added auto_follow_exception.timestamp field to auto follow stats {pull}36947[#36947] * [CCR] Add time since last auto follow fetch to auto follow stats {pull}36542[#36542] (issues: {issue}33007[#33007], {issue}35895[#35895]) CRUD:: * Add Seq# based optimistic concurrency control to UpdateRequest {pull}37872[#37872] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Introduce ssl settings to reindex from remote {pull}37527[#37527] (issues: {issue}29755[#29755], {issue}37287[#37287]) * Use Sequence number powered OCC for processing updates {pull}37308[#37308] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Document Seq No powered optimistic concurrency control {pull}37284[#37284] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Enable IPv6 URIs in reindex from remote {pull}36874[#36874] * Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm {pull}36757[#36757] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Expose Sequence Number based Optimistic Concurrency Control in the rest layer {pull}36721[#36721] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Add doc's sequence number + primary term to GetResult and use it for updates {pull}36680[#36680] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Add seq no powered optimistic locking support to the index and delete transport actions {pull}36619[#36619] (issues: {issue}10708[#10708], {issue}36148[#36148]) * Set acking timeout to 0 on dynamic mapping update {pull}31140[#31140] (issues: {issue}30672[#30672], {issue}30844[#30844]) Distributed:: * Recover retention leases during peer recovery {pull}38435[#38435] (issue: {issue}37165[#37165]) * Lift retention lease expiration to index shard {pull}38380[#38380] (issues: {issue}37165[#37165], {issue}37963[#37963], {issue}38070[#38070]) * Introduce retention lease background sync {pull}38262[#38262] (issue: {issue}37165[#37165]) * Allow shards of closed indices to be replicated as regular shards {pull}38024[#38024] (issue: {issue}33888[#33888]) * Expose retention leases in shard stats {pull}37991[#37991] (issue: {issue}37165[#37165]) * Introduce retention leases versioning {pull}37951[#37951] (issue: {issue}37165[#37165]) * Soft-deletes policy should always fetch latest leases {pull}37940[#37940] (issues: {issue}37165[#37165], {issue}37375[#37375]) * Sync retention leases on expiration {pull}37902[#37902] (issue: {issue}37165[#37165]) * Ignore shard started requests when primary term does not match {pull}37899[#37899] (issue: {issue}33888[#33888]) * Move update and delete by query to use seq# for optimistic concurrency control {pull}37857[#37857] (issues: {issue}10708[#10708], {issue}36148[#36148], {issue}37639[#37639]) * Introduce retention lease serialization {pull}37447[#37447] (issues: {issue}37165[#37165], {issue}37398[#37398]) * Add run under primary permit method {pull}37440[#37440] (issue: {issue}37398[#37398]) * Introduce retention lease syncing {pull}37398[#37398] (issue: {issue}37165[#37165]) * Introduce retention lease persistence {pull}37375[#37375] (issue: {issue}37165[#37165]) * Add validation for retention lease construction {pull}37312[#37312] (issue: {issue}37165[#37165]) * Introduce retention lease expiration {pull}37195[#37195] (issue: {issue}37165[#37165]) * Introduce shard history retention leases {pull}37167[#37167] (issue: {issue}37165[#37165]) * [Close Index API] Add unique UUID to ClusterBlock {pull}36775[#36775] * [Close Index API] Mark shard copy as stale if needed during shard verification {pull}36755[#36755] * [Close Index API] Propagate tasks ids between Freeze, Close and Verify Shard actions {pull}36630[#36630] * Always initialize the global checkpoint {pull}34381[#34381] Engine:: * Specialize pre-closing checks for engine implementations {pull}38702[#38702] * Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines {pull}37426[#37426] * Enable Bulk-Merge if all source remains {pull}37269[#37269] * Rename setting to enable mmap {pull}37070[#37070] (issue: {issue}36668[#36668]) * Add hybridfs store type {pull}36668[#36668] * Introduce time-based retention policy for soft-deletes {pull}34943[#34943] (issue: {issue}34908[#34908]) * handle AsyncAfterWriteAction failure on primary in the same way as failures on replicas {pull}31969[#31969] (issues: {issue}31716[#31716], {issue}31755[#31755]) Features/CAT APIs:: * Expose `search.throttled` on `_cat/indices` {pull}37073[#37073] (issue: {issue}34352[#34352]) Features/Features:: * Run Node deprecation checks locally (#38065) {pull}38250[#38250] (issue: {issue}38065[#38065]) Features/ILM:: * Ensure ILM policies run safely on leader indices {pull}38140[#38140] (issue: {issue}34648[#34648]) * Skip Shrink when numberOfShards not changed {pull}37953[#37953] (issue: {issue}33275[#33275]) * Inject Unfollow before Rollover and Shrink {pull}37625[#37625] (issue: {issue}34648[#34648]) * Add set_priority action to ILM {pull}37397[#37397] (issue: {issue}36905[#36905]) * [ILM] Add Freeze Action {pull}36910[#36910] (issue: {issue}34630[#34630]) Features/Indices APIs:: * New mapping signature and mapping string source fixed. {pull}37401[#37401] Features/Ingest:: * ingest: compile mustache template only if field includes '{{'' {pull}37207[#37207] (issue: {issue}37120[#37120]) * Move ingest-geoip default databases out of config {pull}36949[#36949] (issue: {issue}36898[#36898]) * Make the ingest-geoip databases even lazier to load {pull}36679[#36679] * Updates the grok patterns to be consistent with the logstash {pull}27181[#27181] Features/Java High Level REST Client:: * HLRC: Fix strict setting exception handling {pull}37247[#37247] (issue: {issue}37090[#37090]) * HLRC: Use nonblocking entity for requests {pull}32249[#32249] Features/Monitoring:: * Adding mapping for hostname field {pull}37288[#37288] Features/Stats:: * Stats: Add JVM dns cache expiration config to JvmInfo {pull}36372[#36372] Features/Watcher:: * Move watcher to use seq# and primary term for concurrency control {pull}37977[#37977] (issues: {issue}10708[#10708], {issue}37872[#37872]) * Use ILM for Watcher history deletion {pull}37443[#37443] (issue: {issue}32041[#32041]) * Watcher: Add whitelist to HttpClient {pull}36817[#36817] (issue: {issue}29937[#29937]) Infra/Core:: * fix a few versionAdded values in ElasticsearchExceptions {pull}37877[#37877] * Add simple method to write collection of writeables {pull}37448[#37448] (issue: {issue}37398[#37398]) * Date/Time parsing: Use java time API instead of exception handling {pull}37222[#37222] * [API] spelling: interruptible {pull}37049[#37049] (issue: {issue}37035[#37035]) Infra/Logging:: * Trim the JSON source in indexing slow logs {pull}38081[#38081] (issue: {issue}38080[#38080]) * Optimize warning header de-duplication {pull}37725[#37725] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597], {issue}37622[#37622]) * Remove warn-date from warning headers {pull}37622[#37622] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597]) * Add some deprecation optimizations {pull}37597[#37597] (issues: {issue}35754[#35754], {issue}37530[#37530]) * Only update response headers if we have a new one {pull}37590[#37590] (issues: {issue}35754[#35754], {issue}37530[#37530]) Infra/Packaging:: * Add OS/architecture classifier to distributions {pull}37881[#37881] * Change file descriptor limit to 65535 {pull}37537[#37537] (issue: {issue}35839[#35839]) * Exit batch files explictly using ERRORLEVEL {pull}29583[#29583] (issue: {issue}29582[#29582]) Infra/Scripting:: * Add getZone to JodaCompatibleZonedDateTime {pull}37084[#37084] * [Painless] Add boxed type to boxed type casts for method/return {pull}36571[#36571] Infra/Settings:: * Separate out validation of groups of settings {pull}34184[#34184] License:: * Handle malformed license signatures {pull}37137[#37137] (issue: {issue}35340[#35340]) Machine Learning:: * Move ML Optimistic Concurrency Control to Seq No {pull}38278[#38278] (issues: {issue}10708[#10708], {issue}36148[#36148]) * [ML] Add explanation so far to file structure finder exceptions {pull}38191[#38191] (issue: {issue}29821[#29821]) * ML: Add reason field in JobTaskState {pull}38029[#38029] (issue: {issue}34431[#34431]) * [ML] Add _meta information to all ML indices {pull}37964[#37964] * ML: Add upgrade mode docs, hlrc, and fix bug {pull}37942[#37942] * [ML] Tighten up use of aliases rather than concrete indices {pull}37874[#37874] * ML: Add support for single bucket aggs in Datafeeds {pull}37544[#37544] (issue: {issue}36838[#36838]) * [ML] Create the ML annotations index {pull}36731[#36731] (issues: {issue}26034[#26034], {issue}33376[#33376]) * [ML] Merge the Jindex master feature branch {pull}36702[#36702] (issue: {issue}32905[#32905]) * [FEATURE][ML] Add cluster setting to enable/disable config migration {pull}36700[#36700] (issue: {issue}32905[#32905]) Mapping:: * Log document id when MapperParsingException occurs {pull}37800[#37800] (issue: {issue}37658[#37658]) * [API] spelling: unknown {pull}37056[#37056] (issue: {issue}37035[#37035]) * Make SourceToParse immutable {pull}36971[#36971] * Use index-prefix fields for terms of length min_chars - 1 {pull}36703[#36703] Network:: * Enable TLSv1.3 by default for JDKs with support {pull}38103[#38103] (issue: {issue}32276[#32276]) Recovery:: * SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence {pull}37691[#37691] (issue: {issue}33888[#33888]) * Make prepare engine step of recovery source non-blocking {pull}37573[#37573] (issue: {issue}37174[#37174]) * Make recovery source send operations non-blocking {pull}37503[#37503] (issue: {issue}37458[#37458]) * Prepare to make send translog of recovery non-blocking {pull}37458[#37458] (issue: {issue}37291[#37291]) * Make finalize step of recovery source non-blocking {pull}37388[#37388] (issue: {issue}37291[#37291]) * Make recovery source partially non-blocking {pull}37291[#37291] (issue: {issue}36195[#36195]) * Do not mutate RecoveryResponse {pull}37204[#37204] (issue: {issue}37174[#37174]) * Don't block on peer recovery on the target side {pull}37076[#37076] (issue: {issue}36195[#36195]) * Reduce recovery time with compress or secure transport {pull}36981[#36981] (issue: {issue}33844[#33844]) * Translog corruption marker {pull}33415[#33415] (issue: {issue}31389[#31389]) Rollup:: * Replace the TreeMap in the composite aggregation {pull}36675[#36675] SQL:: * SQL: Allow look-ahead resolution of aliases for WHERE clause {pull}38450[#38450] (issue: {issue}29983[#29983]) * SQL: Implement CURRENT_DATE {pull}38175[#38175] (issue: {issue}38160[#38160]) * SQL: Generate relevant error message when grouping functions are not used in GROUP BY {pull}38017[#38017] (issue: {issue}37952[#37952]) * SQL: Skip the nested and object field types in case of an ODBC request {pull}37948[#37948] (issue: {issue}37801[#37801]) * SQL: Add protocol tests and remove jdbc_type from drivers response {pull}37516[#37516] (issues: {issue}36635[#36635], {issue}36882[#36882]) * SQL: Remove slightly used meta commands {pull}37506[#37506] (issue: {issue}37409[#37409]) * SQL: Describe aliases as views {pull}37496[#37496] (issue: {issue}37422[#37422]) * SQL: Make `FULL` non-reserved keyword in the grammar {pull}37377[#37377] (issue: {issue}37376[#37376]) * SQL: Use declared source for error messages {pull}37161[#37161] * SQL: Improve error message when unable to translate to ES query DSL {pull}37129[#37129] (issue: {issue}37040[#37040]) * [API] spelling: subtract {pull}37055[#37055] (issue: {issue}37035[#37035]) * [API] spelling: similar {pull}37054[#37054] (issue: {issue}37035[#37035]) * [API] spelling: input {pull}37048[#37048] (issue: {issue}37035[#37035]) * SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg {pull}36933[#36933] (issue: {issue}36903[#36903]) * SQL: Preserve original source for each expression {pull}36912[#36912] (issue: {issue}36894[#36894]) * SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms {pull}36753[#36753] (issue: {issue}36740[#36740]) * SQL: Fix wrong appliance of StackOverflow limit for IN {pull}36724[#36724] (issue: {issue}36592[#36592]) Search:: * Tie break on cluster alias when merging shard search failures {pull}38715[#38715] (issue: {issue}38672[#38672]) * Add finalReduce flag to SearchRequest {pull}38104[#38104] (issues: {issue}37000[#37000], {issue}37838[#37838]) * Streamline skip_unavailable handling {pull}37672[#37672] (issue: {issue}32125[#32125]) * Expose sequence number and primary terms in search responses {pull}37639[#37639] * Add support for merging multiple search responses into one {pull}37566[#37566] (issue: {issue}32125[#32125]) * Allow field types to optimize phrase prefix queries {pull}37436[#37436] (issue: {issue}31921[#31921]) * Add support for providing absolute start time to SearchRequest {pull}37142[#37142] (issue: {issue}32125[#32125]) * Ensure that local cluster alias is never treated as remote {pull}37121[#37121] (issues: {issue}32125[#32125], {issue}36997[#36997]) * [API] spelling: cacheable {pull}37047[#37047] (issue: {issue}37035[#37035]) * Add ability to suggest shard_size on coord node rewrite {pull}37017[#37017] (issues: {issue}32125[#32125], {issue}36997[#36997], {issue}37000[#37000]) * Skip final reduction if SearchRequest holds a cluster alias {pull}37000[#37000] (issues: {issue}32125[#32125], {issue}36997[#36997]) * Add support for local cluster alias to SearchRequest {pull}36997[#36997] (issue: {issue}32125[#32125]) * Use SearchRequest copy constructor in ExpandSearchPhase {pull}36772[#36772] (issue: {issue}36641[#36641]) * Add raw sort values to SearchSortValues transport serialization {pull}36617[#36617] (issue: {issue}32125[#32125]) Security:: * Move CAS operations in TokenService to sequence numbers {pull}38311[#38311] (issues: {issue}10708[#10708], {issue}37872[#37872]) * Cleanup construction of interceptors {pull}38294[#38294] * Add passphrase support to elasticsearch-keystore {pull}37472[#37472] (issue: {issue}32691[#32691]) Snapshot/Restore:: * RestoreService should update primary terms when restoring shards of existing indices {pull}38177[#38177] (issue: {issue}33888[#33888]) * Allow open indices to be restored {pull}37733[#37733] * Create specific exception for when snapshots are in progress {pull}37550[#37550] (issue: {issue}37541[#37541]) * SNAPSHOT: Make Atomic Blob Writes Mandatory {pull}37168[#37168] (issues: {issue}37011[#37011], {issue}37066[#37066]) * SNAPSHOT: Speed up HDFS Repository Writes {pull}37069[#37069] * Implement Atomic Blob Writes for HDFS Repository {pull}37066[#37066] (issue: {issue}37011[#37011]) * [API] spelling: repositories {pull}37053[#37053] (issue: {issue}37035[#37035]) * SNAPSHOT: Use CancellableThreads to Abort {pull}35901[#35901] (issue: {issue}21759[#21759]) * WIP: S3 client encryption {pull}30513[#30513] (issues: {issue}11128[#11128], {issue}16843[#16843]) Suggesters:: * Remove unused empty constructors from suggestions classes {pull}37295[#37295] * [API] spelling: likelihood {pull}37052[#37052] (issue: {issue}37035[#37035]) ZenDiscovery:: * Add elasticsearch-node detach-cluster tool {pull}37979[#37979] * Deprecate minimum_master_nodes {pull}37868[#37868] * Step down as master when configured out of voting configuration {pull}37802[#37802] (issue: {issue}37712[#37712]) * Enforce cluster UUIDs {pull}37775[#37775] * Bubble exceptions up in ClusterApplierService {pull}37729[#37729] * Use m_m_nodes from Zen1 master for Zen2 bootstrap {pull}37701[#37701] * Add tool elasticsearch-node unsafe-bootstrap {pull}37696[#37696] * Report terms and version if cluster does not form {pull}37473[#37473] * Bootstrap a Zen2 cluster once quorum is discovered {pull}37463[#37463] * Zen2: Add join validation {pull}37203[#37203] * Publish cluster states in chunks {pull}36973[#36973] [[bug-7.0.0-beta1]] [float] === Bug fixes Aggregations:: * Don't load global ordinals with the `map` execution_hint {pull}37833[#37833] (issue: {issue}37705[#37705]) * Issue #37303 - Invalid variance fix {pull}37384[#37384] (issue: {issue}37303[#37303]) Allocation:: * Fix _host based require filters {pull}38173[#38173] * ALLOC: Fail Stale Primary Alloc. Req. without Data {pull}37226[#37226] (issue: {issue}37098[#37098]) Audit:: * Fix NPE in Logfile Audit Filter {pull}38120[#38120] (issue: {issue}38097[#38097]) Authentication:: * Enhance parsing of StatusCode in SAML Responses {pull}38628[#38628] * Limit token expiry to 1 hour maximum {pull}38244[#38244] * Fix expired token message in Exception header {pull}37196[#37196] * Fix NPE in CachingUsernamePasswordRealm {pull}36953[#36953] (issue: {issue}36951[#36951]) CCR:: * Prevent CCR recovery from missing documents {pull}38237[#38237] * Fix file reading in ccr restore service {pull}38117[#38117] * Correct argument names in update mapping/settings from leader {pull}38063[#38063] * Ensure changes requests return the latest mapping version {pull}37633[#37633] * Do not set fatal exception when shard follow task is stopped. {pull}37603[#37603] * Add fatal_exception field for ccr stats in monitoring mapping {pull}37563[#37563] * Do not add index event listener if CCR disabled {pull}37432[#37432] * When removing an AutoFollower also mark it as removed. {pull}37402[#37402] (issue: {issue}36761[#36761]) * [CCR] Make shard follow tasks more resilient for restarts {pull}37239[#37239] (issue: {issue}37231[#37231]) * [CCR] Resume follow Api should not require a request body {pull}37217[#37217] (issue: {issue}37022[#37022]) * [CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled {pull}36886[#36886] (issue: {issue}33007[#33007]) * Remote cluster license checker and no license info. {pull}36837[#36837] (issue: {issue}36815[#36815]) * Make CCR resilient against missing remote cluster connections {pull}36682[#36682] (issues: {issue}36255[#36255], {issue}36667[#36667]) * [CCR] AutoFollowCoordinator and follower index already created {pull}36540[#36540] (issue: {issue}33007[#33007]) CRUD:: * Fix Reindex from remote query logic {pull}36908[#36908] * Synchronize WriteReplicaResult callbacks {pull}36770[#36770] Distributed:: * TransportVerifyShardBeforeCloseAction should force a flush {pull}38401[#38401] (issues: {issue}33888[#33888], {issue}37961[#37961]) * Fix limit on retaining sequence number {pull}37992[#37992] (issue: {issue}37165[#37165]) * Close Index API should force a flush if a sync is needed {pull}37961[#37961] (issues: {issue}33888[#33888], {issue}37426[#37426]) * Force Refresh Listeners when Acquiring all Operation Permits {pull}36835[#36835] * Replaced the word 'shards' with 'replicas' in an error message. (#36234) {pull}36275[#36275] (issue: {issue}36234[#36234]) Engine:: * Subclass NIOFSDirectory instead of using FileSwitchDirectory {pull}37140[#37140] (issues: {issue}36668[#36668], {issue}37111[#37111]) Features/ILM:: * Preserve ILM operation mode when creating new lifecycles {pull}38134[#38134] (issues: {issue}38229[#38229], {issue}38230[#38230]) * Retry ILM steps that fail due to SnapshotInProgressException {pull}37624[#37624] (issues: {issue}37541[#37541], {issue}37552[#37552]) * Remove `indexing_complete` when removing policy {pull}36620[#36620] Features/Indices APIs:: * Reject delete index requests with a body {pull}37501[#37501] (issue: {issue}8217[#8217]) * Fix duplicate phrase in shrink/split error message {pull}36734[#36734] (issue: {issue}36729[#36729]) * Get Aliases with wildcard exclusion expression {pull}34230[#34230] (issues: {issue}33518[#33518], {issue}33805[#33805], {issue}34144[#34144]) Features/Ingest:: * Support unknown fields in ingest pipeline map configuration {pull}38352[#38352] (issue: {issue}36938[#36938]) * Ingest node - user_agent, move device parsing to an object {pull}38115[#38115] (issues: {issue}37329[#37329], {issue}38094[#38094]) * ingest: fix on_failure with Drop processor {pull}36686[#36686] (issue: {issue}36151[#36151]) * ingest: support default pipelines + bulk upserts {pull}36618[#36618] (issue: {issue}36219[#36219]) Features/Java High Level REST Client:: * Update IndexTemplateMetaData to allow unknown fields {pull}38448[#38448] (issue: {issue}36938[#36938]) * `if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API {pull}38411[#38411] * Update Rollup Caps to allow unknown fields {pull}38339[#38339] (issue: {issue}36938[#36938]) * Fix ILM explain response to allow unknown fields {pull}38054[#38054] (issue: {issue}36938[#36938]) * Fix ILM status to allow unknown fields {pull}38043[#38043] (issue: {issue}36938[#36938]) * Fix ILM Lifecycle Policy to allow unknown fields {pull}38041[#38041] (issue: {issue}36938[#36938]) * Update authenticate to allow unknown fields {pull}37713[#37713] (issue: {issue}36938[#36938]) * Update verify repository to allow unknown fields {pull}37619[#37619] (issue: {issue}36938[#36938]) * Update get users to allow unknown fields {pull}37593[#37593] (issue: {issue}36938[#36938]) * Update Execute Watch to allow unknown fields {pull}37498[#37498] (issue: {issue}36938[#36938]) * Update Put Watch to allow unknown fields {pull}37494[#37494] (issue: {issue}36938[#36938]) * Update Delete Watch to allow unknown fields {pull}37435[#37435] (issue: {issue}36938[#36938]) * Fix rest reindex test for IPv4 addresses {pull}37310[#37310] * Fix weighted_avg parser not found for RestHighLevelClient {pull}37027[#37027] (issue: {issue}36861[#36861]) Features/Java Low Level REST Client:: * Fix potential IllegalCapacityException in LLRC when selecting nodes {pull}37821[#37821] Features/Monitoring:: * Allow built-in monitoring_user role to call GET _xpack API {pull}38060[#38060] (issue: {issue}37970[#37970]) Features/Watcher:: * Support merge nested Map in list for JIRA configurations {pull}37634[#37634] (issue: {issue}30068[#30068]) * Watcher accounts constructed lazily {pull}36656[#36656] * Ensures watch definitions are valid json {pull}30692[#30692] (issue: {issue}29746[#29746]) Geo:: * Fix GeoHash PrefixTree BWC {pull}38584[#38584] (issue: {issue}38494[#38494]) * Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297]) Infra/Core:: * Bubble-up exceptions from scheduler {pull}38317[#38317] (issue: {issue}38014[#38014]) * Core: Revert back to joda's multi date formatters {pull}36814[#36814] (issues: {issue}36447[#36447], {issue}36602[#36602]) * Propagate Errors in executors to uncaught exception handler {pull}36137[#36137] (issue: {issue}28667[#28667]) Infra/Packaging:: * Remove NOREPLACE for /etc/elasticsearch in rpm and deb {pull}37839[#37839] * Packaging: Update marker used to allow ELASTIC_PASSWORD {pull}37243[#37243] (issue: {issue}37240[#37240]) * Packaging: Remove permission editing in postinst {pull}37242[#37242] (issue: {issue}37143[#37143]) Infra/REST API:: * Reject all requests that have an unconsumed body {pull}37504[#37504] (issues: {issue}30792[#30792], {issue}37501[#37501], {issue}8217[#8217]) Infra/Scripting:: * Fix Painless void return bug {pull}38046[#38046] Infra/Settings:: * Fix setting by time unit {pull}37192[#37192] * Fix handling of fractional byte size value settings {pull}37172[#37172] * Fix handling of fractional time value settings {pull}37171[#37171] Machine Learning:: * [ML] Report index unavailable instead of waiting for lazy node {pull}38423[#38423] * ML: Fix error race condition on stop _all datafeeds and close _all jobs {pull}38113[#38113] (issue: {issue}37959[#37959]) * [ML] Update ML results mappings on process start {pull}37706[#37706] (issue: {issue}37607[#37607]) * [ML] Prevent submit after autodetect worker is stopped {pull}37700[#37700] (issue: {issue}37108[#37108]) * [ML] Fix ML datafeed CCS with wildcarded cluster name {pull}37470[#37470] (issue: {issue}36228[#36228]) * [ML] Update error message for process update {pull}37363[#37363] * [ML] Wait for autodetect to be ready in the datafeed {pull}37349[#37349] (issues: {issue}36810[#36810], {issue}37227[#37227]) * [ML] Stop datafeeds running when their jobs are stale {pull}37227[#37227] (issue: {issue}36810[#36810]) * [ML] Order GET job stats response by job id {pull}36841[#36841] (issue: {issue}36683[#36683]) * [ML] Make GetJobStats work with arbitrary wildcards and groups {pull}36683[#36683] (issue: {issue}34745[#34745]) Mapping:: * Treat put-mapping calls with `_doc` as a top-level key as typed calls. {pull}38032[#38032] * Correct deprec log in RestGetFieldMappingAction {pull}37843[#37843] (issue: {issue}37667[#37667]) * Restore a noop _all metadata field for 6x indices {pull}37808[#37808] (issue: {issue}37429[#37429]) * Make sure PutMappingRequest accepts content types other than JSON. {pull}37720[#37720] * Make sure to use the resolved type in DocumentMapperService#extractMappings. {pull}37451[#37451] (issue: {issue}36811[#36811]) * MAPPING: Improve Precision for scaled_float {pull}37169[#37169] (issue: {issue}32570[#32570]) * Make sure to accept empty unnested mappings in create index requests. {pull}37089[#37089] * Stop automatically nesting mappings in index creation requests. {pull}36924[#36924] * Rewrite SourceToParse with resolved docType {pull}36921[#36921] (issues: {issue}35790[#35790], {issue}36769[#36769]) Network:: * Reload SSL context on file change for LDAP {pull}36937[#36937] (issues: {issue}30509[#30509], {issue}36923[#36923]) * Do not resolve addresses in remote connection info {pull}36671[#36671] (issue: {issue}35658[#35658]) Ranking:: * QueryRescorer should keep the window size when rewriting {pull}36836[#36836] Recovery:: * RecoveryMonitor#lastSeenAccessTime should be volatile {pull}36781[#36781] Rollup:: * Fix Rollup's metadata parser {pull}36791[#36791] (issue: {issue}36726[#36726]) * Fix rollup search statistics {pull}36674[#36674] SQL:: * SQL: Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308]) * SQL: Relax StackOverflow circuit breaker for constants {pull}38572[#38572] (issue: {issue}38571[#38571]) * SQL: Fix issue with IN not resolving to underlying keyword field {pull}38440[#38440] (issue: {issue}38424[#38424]) * SQL: change the Intervals milliseconds precision to 3 digits {pull}38297[#38297] (issue: {issue}37423[#37423]) * SQL: Fix esType for DATETIME/DATE and INTERVALS {pull}38179[#38179] (issue: {issue}38051[#38051]) * SQL: Added SSL configuration options tests {pull}37875[#37875] (issue: {issue}37711[#37711]) * SQL: Fix casting from date to numeric type to use millis {pull}37869[#37869] (issue: {issue}37655[#37655]) * SQL: Fix BasicFormatter NPE {pull}37804[#37804] * SQL: Return Intervals in SQL format for CLI {pull}37602[#37602] (issues: {issue}29970[#29970], {issue}36186[#36186], {issue}36432[#36432]) * SQL: fix object extraction from sources {pull}37502[#37502] (issue: {issue}37364[#37364]) * SQL: Fix issue with field names containing "." {pull}37364[#37364] (issue: {issue}37128[#37128]) * SQL: Fix bug regarding alias fields with dots {pull}37279[#37279] (issue: {issue}37224[#37224]) * SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {pull}37254[#37254] (issue: {issue}30285[#30285]) * SQL: fix COUNT DISTINCT filtering {pull}37176[#37176] (issue: {issue}37086[#37086]) * SQL: Fix issue with wrong NULL optimization {pull}37124[#37124] (issue: {issue}35872[#35872]) * SQL: Fix issue with complex expression as args of PERCENTILE/_RANK {pull}37102[#37102] (issue: {issue}37099[#37099]) * SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless {pull}37024[#37024] (issue: {issue}37023[#37023]) * SQL: Fix bug regarding histograms usage in scripting {pull}36866[#36866] * SQL: Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980]) * SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756]) * SQL: Enhance Verifier to prevent aggregate or grouping functions from {pull}36799[#36799] (issue: {issue}36798[#36798]) * SQL: Fix translation of LIKE/RLIKE keywords {pull}36672[#36672] (issues: {issue}36039[#36039], {issue}36584[#36584]) * SQL: Scripting support for casting functions CAST and CONVERT {pull}36640[#36640] (issue: {issue}36061[#36061]) * SQL: Concat should be always not nullable {pull}36601[#36601] (issue: {issue}36169[#36169]) * SQL: Fix issue with complex HAVING and GROUP BY ordinal {pull}36594[#36594] (issue: {issue}36059[#36059]) Search:: * Look up connection using the right cluster alias when releasing contexts {pull}38570[#38570] * Fix fetch source option in expand search phase {pull}37908[#37908] (issue: {issue}23829[#23829]) * Change `rational` to `saturation` in script_score {pull}37766[#37766] (issue: {issue}37714[#37714]) * Throw if two inner_hits have the same name {pull}37645[#37645] (issue: {issue}37584[#37584]) * Ensure either success or failure path for SearchOperationListener is called {pull}37467[#37467] (issue: {issue}37185[#37185]) * `query_string` should use indexed prefixes {pull}36895[#36895] * Avoid duplicate types deprecation messages in search-related APIs. {pull}36802[#36802] Security:: * Fix exit code for Security CLI tools {pull}37956[#37956] (issue: {issue}37841[#37841]) * Fix potential NPE in UsersTool {pull}37660[#37660] Snapshot/Restore:: * Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {pull}38368[#38368] (issue: {issue}38226[#38226]) * Fix Two Races that Lead to Stuck Snapshots {pull}37686[#37686] (issues: {issue}32265[#32265], {issue}32348[#32348]) * Fix Race in Concurrent Snapshot Delete and Create {pull}37612[#37612] (issue: {issue}37581[#37581]) * Streamline S3 Repository- and Client-Settings {pull}37393[#37393] Suggesters:: * Fix duplicate removal when merging completion suggestions {pull}36996[#36996] (issue: {issue}35836[#35836]) Task Management:: * Un-assign persistent tasks as nodes exit the cluster {pull}37656[#37656] ZenDiscovery:: * Fix size of rolling-upgrade bootstrap config {pull}38031[#38031] * Always return metadata version if metadata is requested {pull}37674[#37674] * [Zen2] Elect freshest master in upgrade {pull}37122[#37122] (issue: {issue}40[#40]) * Fix cluster state persistence for single-node discovery {pull}36825[#36825] [[regression-7.0.0-beta1]] [float] === Regressions Infra/Core:: * Restore date aggregation performance in UTC case {pull}38221[#38221] (issue: {issue}37826[#37826]) * Speed up converting of temporal accessor to zoned date time {pull}37915[#37915] (issue: {issue}37826[#37826]) Mapping:: * Performance fix. Reduce deprecation calls for the same bulk request {pull}37415[#37415] (issue: {issue}37411[#37411]) [[upgrade-7.0.0-beta1]] [float] === Upgrades Engine:: * Upgrade to lucene-8.0.0-snapshot-83f9835. {pull}37668[#37668] Machine Learning:: * [ML] No need to add state doc mapping on job open in 7.x {pull}37759[#37759]