OpenSearch/docs/reference/release-notes/7.0.0-beta1.asciidoc

[[release-notes-7.0.0-beta1]]
== 7.0.0-beta1 Release Notes

coming[7.0.0-beta1]

Also see <<breaking-changes-7.0>>.

[[breaking-7.0.0-beta1]]
[float]
=== Breaking changes

Audit::
* Remove index audit output type {pull}37707[#37707] (issues: {issue}29881[#29881], {issue}37301[#37301])

Authentication::
* Remove bwc logic for token invalidation {pull}36893[#36893] (issue: {issue}36727[#36727])

Authorization::
* Remove implicit index monitor privilege {pull}37774[#37774]

CCR::
* Follow stats api should return a 404 when requesting stats for a non existing index {pull}37220[#37220] (issue: {issue}37021[#37021])

CRUD::
* Remove support for internal versioning for concurrency control {pull}38254[#38254] (issue: {issue}1078[#1078])

Features/Ingest::
* Add ECS schema for user-agent ingest processor (#37727) {pull}37984[#37984] (issues: {issue}37329[#37329], {issue}37727[#37727])
* Remove special handling for ingest plugins {pull}36967[#36967] (issues: {issue}36898[#36898], {issue}36956[#36956])

Features/Java Low Level REST Client::
* Drop support for the low-level REST client on JDK 7 {pull}38540[#38540] (issue: {issue}29607[#29607])

Features/Watcher::
* Remove Watcher Account "unsecure" settings {pull}36736[#36736] (issue: {issue}36403[#36403])

Infra/Logging::
* Elasticsearch json logging  {pull}36833[#36833] (issue: {issue}32850[#32850])

Infra/Packaging::
* Package ingest-user-agent as a module {pull}36956[#36956]
* Package ingest-geoip as a module {pull}36898[#36898]

Machine Learning::
* [ML] Remove types from datafeed {pull}36538[#36538] (issue: {issue}34265[#34265])

Mapping::
* Make sure to reject mappings with type _doc when include_type_name is false. {pull}38270[#38270] (issue: {issue}38266[#38266])
* Update the default for include_type_name to false. {pull}37285[#37285]
* Support 'include_type_name' in RestGetIndicesAction {pull}37149[#37149]

Network::
* Remove TLS 1.0 as a default SSL protocol {pull}37512[#37512] (issue: {issue}36021[#36021])
* Security: remove SSL settings fallback {pull}36846[#36846] (issue: {issue}29797[#29797])

Ranking::
* Forbid negative field boosts in analyzed queries {pull}37930[#37930] (issue: {issue}33309[#33309])

Search::
* Track total hits up to 10,000 by default {pull}37466[#37466] (issue: {issue}33028[#33028])
* Use mappings to format doc-value fields by default. {pull}30831[#30831] (issues: {issue}26948[#26948], {issue}29639[#29639])

Security::
* Remove heuristics that enable security on trial licenses {pull}38075[#38075] (issue: {issue}38009[#38009])

ZenDiscovery::
* Remove DiscoveryPlugin#getDiscoveryTypes {pull}38414[#38414] (issue: {issue}38410[#38410])



[[breaking-java-7.0.0-beta1]]
[float]
=== Breaking Java changes

Features/Java Low Level REST Client::
* Remove support for maxRetryTimeout from low-level REST client {pull}38085[#38085] (issues: {issue}25951[#25951], {issue}31834[#31834], {issue}33342[#33342])

Infra/Core::
* Handle scheduler exceptions {pull}38014[#38014] (issues: {issue}28667[#28667], {issue}36137[#36137], {issue}37708[#37708])



[[deprecation-7.0.0-beta1]]
[float]
=== Deprecations

Aggregations::
* Deprecate dots in aggregation names {pull}31468[#31468] (issues: {issue}17600[#17600], {issue}19040[#19040])

Analysis::
* [Analysis] Deprecate Standard Html Strip Analyzer in master {pull}26719[#26719] (issue: {issue}4704[#4704])

Audit::
* Deprecate index audit output type {pull}37301[#37301] (issue: {issue}29881[#29881])

Features/Indices APIs::
* Reject setting index.optimize_auto_generated_id after version 7.0.0 {pull}28895[#28895] (issue: {issue}27600[#27600])

Features/Ingest::
* Deprecate `_type` in simulate pipeline requests {pull}37949[#37949] (issue: {issue}37731[#37731])

Features/Java High Level REST Client::
* Deprecate HLRC security methods {pull}37883[#37883] (issues: {issue}36938[#36938], {issue}37540[#37540])
* Deprecate HLRC EmptyResponse used by security {pull}37540[#37540] (issue: {issue}36938[#36938])

Features/Watcher::
* Deprecate xpack.watcher.history.cleaner_service.enabled {pull}37782[#37782] (issue: {issue}32041[#32041])
* deprecate types for watcher {pull}37594[#37594] (issue: {issue}35190[#35190])

Infra/Core::
* Core: Deprecate negative epoch timestamps {pull}36793[#36793]
* Core: Deprecate use of scientific notation in epoch time parsing {pull}36691[#36691]

Infra/Scripting::
* Add types deprecation to script contexts {pull}37554[#37554]
* Deprecate _type from LeafDocLookup {pull}37491[#37491]
* Scripting: Remove deprecated params.ctx {pull}36848[#36848] (issue: {issue}34059[#34059])

Machine Learning::
* Adding ml_settings entry to HLRC and Docs for deprecation_info {pull}38118[#38118]
* [ML] Datafeed deprecation checks {pull}38026[#38026] (issue: {issue}37932[#37932])
* [ML] Remove "8" prefixes from file structure finder timestamp formats {pull}38016[#38016]
* [ML] Adjust structure finder for Joda to Java time migration {pull}37306[#37306]
* [ML] Resolve 7.0.0 TODOs in ML code {pull}36842[#36842] (issue: {issue}29963[#29963])

Mapping::
* Deprecate types in rollover index API {pull}38039[#38039] (issue: {issue}35190[#35190])
* Deprecate types in get field mapping API {pull}37667[#37667] (issue: {issue}35190[#35190])
*  Deprecate types in the put mapping API. {pull}37280[#37280] (issues: {issue}29453[#29453], {issue}37285[#37285])
* Support include_type_name in the field mapping and index template APIs. {pull}37210[#37210]
* Deprecate types in create index requests. {pull}37134[#37134] (issues: {issue}29453[#29453], {issue}37285[#37285])
* Deprecate use of the _type field in aggregations. {pull}37131[#37131] (issue: {issue}36802[#36802])
* Deprecate reference to _type in lookup queries {pull}37016[#37016] (issue: {issue}35190[#35190])
* Deprecate the document create endpoint. {pull}36863[#36863]
* Deprecate types in index API {pull}36575[#36575] (issues: {issue}35190[#35190], {issue}35790[#35790])
* Deprecate types in update APIs {pull}36225[#36225]

Search::
* Deprecate use of type in reindex request body {pull}36823[#36823]
* Add typless endpoints for get_source and exist_source {pull}36426[#36426]



[[feature-7.0.0-beta1]]
[float]
=== New features

Authentication::
* Add support for API keys to access Elasticsearch {pull}38291[#38291] (issue: {issue}34383[#34383])
* OIDC realm authentication flows {pull}37787[#37787]
* [WIP] OIDC Realm JWT+JWS related functionality {pull}37272[#37272] (issues: {issue}35339[#35339], {issue}37009[#37009])
* OpenID Connect Realm base functionality {pull}37009[#37009] (issue: {issue}35339[#35339])

Authorization::
* Allow custom authorization with an authorization engine  {pull}38358[#38358] (issues: {issue}32435[#32435], {issue}36245[#36245], {issue}37328[#37328], {issue}37495[#37495], {issue}37785[#37785], {issue}38137[#38137], {issue}38219[#38219])
* WIldcard IndicesPermissions don't cover .security {pull}36765[#36765]

CCR::
* Add ccr follow info api {pull}37408[#37408] (issue: {issue}37127[#37127])

Features/ILM::
* [ILM] Add unfollow action {pull}36970[#36970] (issue: {issue}34648[#34648])

Geo::
* geotile_grid implementation {pull}37842[#37842] (issue: {issue}30240[#30240])
* [GEO] Fork Lucene's LatLonShape Classes to local lucene package {pull}36794[#36794]
* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}36751[#36751] (issue: {issue}35320[#35320])
* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}35320[#35320] (issue: {issue}32039[#32039])

Machine Learning::
* ML: Adds set_upgrade_mode API endpoint {pull}37837[#37837]

Mapping::
* Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. {pull}37871[#37871] (issue: {issue}37773[#37773])
* Add nanosecond field mapper {pull}37755[#37755] (issues: {issue}27330[#27330], {issue}32601[#32601])

SQL::
* SQL: Allow sorting of groups by aggregates {pull}38042[#38042] (issue: {issue}35118[#35118])
* SQL: Implement FIRST/LAST aggregate functions {pull}37936[#37936] (issue: {issue}35639[#35639])
* SQL: Introduce SQL DATE data type {pull}37693[#37693] (issue: {issue}37340[#37340])

Search::
* Introduce ability to minimize round-trips in CCS {pull}37828[#37828] (issues: {issue}32125[#32125], {issue}37566[#37566])
* Add script filter to intervals {pull}36776[#36776]
* Add the ability to set the number of hits to track accurately {pull}36357[#36357] (issue: {issue}33028[#33028])
* Add a maximum search request size. {pull}26423[#26423]



[[enhancement-7.0.0-beta1]]
[float]
=== Enhancements

Aggregations::
* Add Composite to AggregationBuilders {pull}38207[#38207] (issue: {issue}38020[#38020])
* Allow nested fields in the composite aggregation {pull}37178[#37178] (issue: {issue}28611[#28611])
* Remove single shard optimization when suggesting shard_size {pull}37041[#37041] (issue: {issue}32125[#32125])
* Use List instead of priority queue for stable sorting in bucket sort aggregator {pull}36748[#36748] (issue: {issue}36322[#36322])
* Keys are compared in BucketSortPipelineAggregation so making key type… {pull}36407[#36407]

Allocation::
* Fail start on obsolete indices documentation {pull}37786[#37786] (issue: {issue}27073[#27073])
* Fail start on invalid index metadata {pull}37748[#37748] (issue: {issue}27073[#27073])
* Fail start of non-data node if node has data {pull}37347[#37347] (issue: {issue}27073[#27073])

Analysis::
* Allow word_delimiter_graph_filter to not adjust internal offsets {pull}36699[#36699] (issues: {issue}33710[#33710], {issue}34741[#34741])

Audit::
* Security Audit includes HTTP method for requests {pull}37322[#37322] (issue: {issue}29765[#29765])
* Add X-Forwarded-For to the logfile audit {pull}36427[#36427]

Authentication::
* Security: propagate auth result to listeners {pull}36900[#36900] (issue: {issue}30794[#30794])
* Security: reorder realms based on last success {pull}36878[#36878]
* Improve error message for 6.x style realm settings {pull}36876[#36876] (issues: {issue}30241[#30241], {issue}36026[#36026])
* Change missing authn message to not mention tokens {pull}36750[#36750]
* Invalidate Token API enhancements - HLRC {pull}36362[#36362] (issue: {issue}35388[#35388])
* Enhance Invalidate Token API {pull}35388[#35388] (issues: {issue}34556[#34556], {issue}35115[#35115])

Authorization::
* Add apm_user reserved role {pull}38206[#38206]
* Permission for restricted indices {pull}37577[#37577] (issue: {issue}34454[#34454])
* Remove kibana_user and kibana_dashboard_only_user index privileges {pull}37441[#37441]
* Create snapshot role {pull}35820[#35820] (issue: {issue}34454[#34454])

CCR::
* Concurrent file chunk fetching for CCR restore {pull}38495[#38495]
* Tighten mapping syncing in ccr remote restore {pull}38071[#38071] (issues: {issue}36879[#36879], {issue}37887[#37887])
* Do not allow put mapping on follower {pull}37675[#37675] (issue: {issue}30086[#30086])
* Added ccr to xpack usage infrastructure {pull}37256[#37256] (issue: {issue}37221[#37221])
* [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned {pull}37213[#37213]
* [CCR] Added auto_follow_exception.timestamp field to auto follow stats {pull}36947[#36947]
* [CCR] Add time since last auto follow fetch to auto follow stats {pull}36542[#36542] (issues: {issue}33007[#33007], {issue}35895[#35895])

CRUD::
* Add Seq# based optimistic concurrency control to UpdateRequest {pull}37872[#37872] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Introduce ssl settings to reindex from remote {pull}37527[#37527] (issues: {issue}29755[#29755], {issue}37287[#37287])
* Use Sequence number powered OCC for processing updates {pull}37308[#37308] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Document Seq No powered optimistic concurrency control {pull}37284[#37284] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Enable IPv6 URIs in reindex from remote {pull}36874[#36874]
* Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm  {pull}36757[#36757] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Expose Sequence Number based Optimistic Concurrency Control in the rest layer {pull}36721[#36721] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Add doc's sequence number + primary term to GetResult and use it for updates {pull}36680[#36680] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Add seq no powered optimistic locking support to the index and delete transport actions {pull}36619[#36619] (issues: {issue}10708[#10708], {issue}36148[#36148])
* Set acking timeout to 0 on dynamic mapping update {pull}31140[#31140] (issues: {issue}30672[#30672], {issue}30844[#30844])

Distributed::
* Recover retention leases during peer recovery {pull}38435[#38435] (issue: {issue}37165[#37165])
* Lift retention lease expiration to index shard {pull}38380[#38380] (issues: {issue}37165[#37165], {issue}37963[#37963], {issue}38070[#38070])
* Introduce retention lease background sync {pull}38262[#38262] (issue: {issue}37165[#37165])
* Allow shards of closed indices to be replicated as regular shards {pull}38024[#38024] (issue: {issue}33888[#33888])
* Expose retention leases in shard stats {pull}37991[#37991] (issue: {issue}37165[#37165])
* Introduce retention leases versioning {pull}37951[#37951] (issue: {issue}37165[#37165])
* Soft-deletes policy should always fetch latest leases {pull}37940[#37940] (issues: {issue}37165[#37165], {issue}37375[#37375])
* Sync retention leases on expiration {pull}37902[#37902] (issue: {issue}37165[#37165])
* Ignore shard started requests when primary term does not match {pull}37899[#37899] (issue: {issue}33888[#33888])
* Move update and delete by query to use seq# for optimistic concurrency control {pull}37857[#37857] (issues: {issue}10708[#10708], {issue}36148[#36148], {issue}37639[#37639])
* Introduce retention lease serialization {pull}37447[#37447] (issues: {issue}37165[#37165], {issue}37398[#37398])
* Add run under primary permit method {pull}37440[#37440] (issue: {issue}37398[#37398])
* Introduce retention lease syncing {pull}37398[#37398] (issue: {issue}37165[#37165])
* Introduce retention lease persistence {pull}37375[#37375] (issue: {issue}37165[#37165])
* Add validation for retention lease construction {pull}37312[#37312] (issue: {issue}37165[#37165])
* Introduce retention lease expiration {pull}37195[#37195] (issue: {issue}37165[#37165])
* Introduce shard history retention leases {pull}37167[#37167] (issue: {issue}37165[#37165])
* [Close Index API] Add unique UUID to ClusterBlock {pull}36775[#36775]
* [Close Index API] Mark shard copy as stale if needed during shard verification {pull}36755[#36755]
* [Close Index API] Propagate tasks ids between Freeze, Close and Verify Shard actions {pull}36630[#36630]
* Always initialize the global checkpoint {pull}34381[#34381]

Engine::
* Specialize pre-closing checks for engine implementations {pull}38702[#38702]
* Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines {pull}37426[#37426]
* Enable Bulk-Merge if all source remains {pull}37269[#37269]
* Rename setting to enable mmap {pull}37070[#37070] (issue: {issue}36668[#36668])
* Add hybridfs store type {pull}36668[#36668]
* Introduce time-based retention policy for soft-deletes {pull}34943[#34943] (issue: {issue}34908[#34908])
* handle AsyncAfterWriteAction failure on primary in the same way as failures on replicas  {pull}31969[#31969] (issues: {issue}31716[#31716], {issue}31755[#31755])

Features/CAT APIs::
* Expose `search.throttled` on `_cat/indices` {pull}37073[#37073] (issue: {issue}34352[#34352])

Features/Features::
* Run Node deprecation checks locally (#38065) {pull}38250[#38250] (issue: {issue}38065[#38065])

Features/ILM::
* Ensure ILM policies run safely on leader indices  {pull}38140[#38140] (issue: {issue}34648[#34648])
* Skip Shrink when numberOfShards not changed {pull}37953[#37953] (issue: {issue}33275[#33275])
* Inject Unfollow before Rollover and Shrink {pull}37625[#37625] (issue: {issue}34648[#34648])
* Add set_priority action to ILM {pull}37397[#37397] (issue: {issue}36905[#36905])
* [ILM] Add Freeze Action {pull}36910[#36910] (issue: {issue}34630[#34630])

Features/Indices APIs::
* New mapping signature and mapping string source fixed. {pull}37401[#37401]

Features/Ingest::
* ingest: compile mustache template only if field includes '{{'' {pull}37207[#37207] (issue: {issue}37120[#37120])
* Move ingest-geoip default databases out of config {pull}36949[#36949] (issue: {issue}36898[#36898])
* Make the ingest-geoip databases even lazier to load {pull}36679[#36679]
* Updates the grok patterns to be consistent with the logstash {pull}27181[#27181]

Features/Java High Level REST Client::
* HLRC: Fix strict setting exception handling {pull}37247[#37247] (issue: {issue}37090[#37090])
* HLRC: Use nonblocking entity for requests {pull}32249[#32249]

Features/Monitoring::
* Adding mapping for hostname field {pull}37288[#37288]

Features/Stats::
* Stats: Add JVM dns cache expiration config to JvmInfo {pull}36372[#36372]

Features/Watcher::
* Move watcher to use seq# and primary term for concurrency control {pull}37977[#37977] (issues: {issue}10708[#10708], {issue}37872[#37872])
* Use ILM for Watcher history deletion {pull}37443[#37443] (issue: {issue}32041[#32041])
* Watcher: Add whitelist to HttpClient {pull}36817[#36817] (issue: {issue}29937[#29937])

Infra/Core::
* fix a few versionAdded values in ElasticsearchExceptions {pull}37877[#37877]
* Add simple method to write collection of writeables {pull}37448[#37448] (issue: {issue}37398[#37398])
* Date/Time parsing: Use java time API instead of exception handling {pull}37222[#37222]
* [API] spelling: interruptible {pull}37049[#37049] (issue: {issue}37035[#37035])

Infra/Logging::
* Trim the JSON source in indexing slow logs {pull}38081[#38081] (issue: {issue}38080[#38080])
* Optimize warning header de-duplication {pull}37725[#37725] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597], {issue}37622[#37622])
* Remove warn-date from warning headers {pull}37622[#37622] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597])
* Add some deprecation optimizations {pull}37597[#37597] (issues: {issue}35754[#35754], {issue}37530[#37530])
* Only update response headers if we have a new one {pull}37590[#37590] (issues: {issue}35754[#35754], {issue}37530[#37530])

Infra/Packaging::
* Add OS/architecture classifier to distributions {pull}37881[#37881]
* Change file descriptor limit to 65535 {pull}37537[#37537] (issue: {issue}35839[#35839])
* Exit batch files explictly using ERRORLEVEL {pull}29583[#29583] (issue: {issue}29582[#29582])

Infra/Scripting::
* Add getZone to JodaCompatibleZonedDateTime {pull}37084[#37084]
* [Painless] Add boxed type to boxed type casts for method/return {pull}36571[#36571]

Infra/Settings::
* Separate out validation of groups of settings {pull}34184[#34184]

License::
* Handle malformed license signatures {pull}37137[#37137] (issue: {issue}35340[#35340])

Machine Learning::
* Move ML Optimistic Concurrency Control to Seq No {pull}38278[#38278] (issues: {issue}10708[#10708], {issue}36148[#36148])
* [ML] Add explanation so far to file structure finder exceptions {pull}38191[#38191] (issue: {issue}29821[#29821])
* ML: Add reason field in JobTaskState {pull}38029[#38029] (issue: {issue}34431[#34431])
* [ML] Add _meta information to all ML indices {pull}37964[#37964]
* ML: Add upgrade mode docs, hlrc, and fix bug {pull}37942[#37942]
* [ML] Tighten up use of aliases rather than concrete indices {pull}37874[#37874]
* ML: Add support for single bucket aggs in Datafeeds {pull}37544[#37544] (issue: {issue}36838[#36838])
* [ML] Create the ML annotations index {pull}36731[#36731] (issues: {issue}26034[#26034], {issue}33376[#33376])
* [ML] Merge the Jindex master feature branch {pull}36702[#36702] (issue: {issue}32905[#32905])
* [FEATURE][ML] Add cluster setting to enable/disable config  migration {pull}36700[#36700] (issue: {issue}32905[#32905])

Mapping::
* Log document id when MapperParsingException occurs {pull}37800[#37800] (issue: {issue}37658[#37658])
* [API] spelling: unknown {pull}37056[#37056] (issue: {issue}37035[#37035])
* Make SourceToParse immutable {pull}36971[#36971]
* Use index-prefix fields for terms of length min_chars - 1 {pull}36703[#36703]

Network::
* Enable TLSv1.3 by default for JDKs with support {pull}38103[#38103] (issue: {issue}32276[#32276])

Recovery::
* SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence {pull}37691[#37691] (issue: {issue}33888[#33888])
* Make prepare engine step of recovery source non-blocking {pull}37573[#37573] (issue: {issue}37174[#37174])
* Make recovery source send operations non-blocking {pull}37503[#37503] (issue: {issue}37458[#37458])
* Prepare to make send translog of recovery non-blocking {pull}37458[#37458] (issue: {issue}37291[#37291])
* Make finalize step of recovery source non-blocking {pull}37388[#37388] (issue: {issue}37291[#37291])
* Make recovery source partially non-blocking {pull}37291[#37291] (issue: {issue}36195[#36195])
* Do not mutate RecoveryResponse {pull}37204[#37204] (issue: {issue}37174[#37174])
* Don't block on peer recovery on the target side {pull}37076[#37076] (issue: {issue}36195[#36195])
* Reduce recovery time with compress or secure transport {pull}36981[#36981] (issue: {issue}33844[#33844])
* Translog corruption marker {pull}33415[#33415] (issue: {issue}31389[#31389])

Rollup::
* Replace the TreeMap in the composite aggregation {pull}36675[#36675]

SQL::
* SQL: Allow look-ahead resolution of aliases for WHERE clause {pull}38450[#38450] (issue: {issue}29983[#29983])
* SQL: Implement CURRENT_DATE {pull}38175[#38175] (issue: {issue}38160[#38160])
* SQL: Generate relevant error message when grouping functions are not used in GROUP BY {pull}38017[#38017] (issue: {issue}37952[#37952])
* SQL: Skip the nested and object field types in case of an ODBC request {pull}37948[#37948] (issue: {issue}37801[#37801])
* SQL: Add protocol tests and remove jdbc_type from drivers response {pull}37516[#37516] (issues: {issue}36635[#36635], {issue}36882[#36882])
* SQL: Remove slightly used meta commands {pull}37506[#37506] (issue: {issue}37409[#37409])
* SQL: Describe aliases as views {pull}37496[#37496] (issue: {issue}37422[#37422])
* SQL: Make `FULL` non-reserved keyword in the grammar {pull}37377[#37377] (issue: {issue}37376[#37376])
* SQL: Use declared source for error messages {pull}37161[#37161]
* SQL: Improve error message when unable to translate to ES query DSL {pull}37129[#37129] (issue: {issue}37040[#37040])
* [API] spelling: subtract {pull}37055[#37055] (issue: {issue}37035[#37035])
* [API] spelling: similar {pull}37054[#37054] (issue: {issue}37035[#37035])
* [API] spelling: input {pull}37048[#37048] (issue: {issue}37035[#37035])
* SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg {pull}36933[#36933] (issue: {issue}36903[#36903])
* SQL: Preserve original source for each expression {pull}36912[#36912] (issue: {issue}36894[#36894])
* SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms {pull}36753[#36753] (issue: {issue}36740[#36740])
* SQL: Fix wrong appliance of StackOverflow limit for IN {pull}36724[#36724] (issue: {issue}36592[#36592])

Search::
* Tie break on cluster alias when merging shard search failures {pull}38715[#38715] (issue: {issue}38672[#38672])
* Add finalReduce flag to SearchRequest {pull}38104[#38104] (issues: {issue}37000[#37000], {issue}37838[#37838])
* Streamline skip_unavailable handling {pull}37672[#37672] (issue: {issue}32125[#32125])
* Expose sequence number and primary terms in search responses {pull}37639[#37639]
* Add support for merging multiple search responses into one {pull}37566[#37566] (issue: {issue}32125[#32125])
* Allow field types to optimize phrase prefix queries {pull}37436[#37436] (issue: {issue}31921[#31921])
* Add support for providing absolute start time to SearchRequest {pull}37142[#37142] (issue: {issue}32125[#32125])
* Ensure that local cluster alias is never treated as remote {pull}37121[#37121] (issues: {issue}32125[#32125], {issue}36997[#36997])
* [API] spelling: cacheable {pull}37047[#37047] (issue: {issue}37035[#37035])
* Add ability to suggest shard_size on coord node rewrite {pull}37017[#37017] (issues: {issue}32125[#32125], {issue}36997[#36997], {issue}37000[#37000])
* Skip final reduction if SearchRequest holds a cluster alias {pull}37000[#37000] (issues: {issue}32125[#32125], {issue}36997[#36997])
* Add support for local cluster alias to SearchRequest {pull}36997[#36997] (issue: {issue}32125[#32125])
* Use SearchRequest copy constructor in ExpandSearchPhase {pull}36772[#36772] (issue: {issue}36641[#36641])
* Add raw sort values to SearchSortValues transport serialization {pull}36617[#36617] (issue: {issue}32125[#32125])

Security::
* Move CAS operations in TokenService to sequence numbers {pull}38311[#38311] (issues: {issue}10708[#10708], {issue}37872[#37872])
* Cleanup construction of interceptors {pull}38294[#38294]
* Add passphrase support to elasticsearch-keystore {pull}37472[#37472] (issue: {issue}32691[#32691])

Snapshot/Restore::
* RestoreService should update primary terms when restoring shards of existing indices {pull}38177[#38177] (issue: {issue}33888[#33888])
* Allow open indices to be restored {pull}37733[#37733]
* Create specific exception for when snapshots are in progress {pull}37550[#37550] (issue: {issue}37541[#37541])
* SNAPSHOT: Make Atomic Blob Writes Mandatory {pull}37168[#37168] (issues: {issue}37011[#37011], {issue}37066[#37066])
* SNAPSHOT: Speed up HDFS Repository Writes {pull}37069[#37069]
* Implement Atomic Blob Writes for HDFS Repository {pull}37066[#37066] (issue: {issue}37011[#37011])
* [API] spelling: repositories {pull}37053[#37053] (issue: {issue}37035[#37035])
* SNAPSHOT: Use CancellableThreads to Abort {pull}35901[#35901] (issue: {issue}21759[#21759])
* WIP: S3 client encryption {pull}30513[#30513] (issues: {issue}11128[#11128], {issue}16843[#16843])

Suggesters::
* Remove unused empty constructors from suggestions classes {pull}37295[#37295]
* [API] spelling: likelihood {pull}37052[#37052] (issue: {issue}37035[#37035])

ZenDiscovery::
* Add elasticsearch-node detach-cluster tool {pull}37979[#37979]
* Deprecate minimum_master_nodes {pull}37868[#37868]
* Step down as master when configured out of voting configuration {pull}37802[#37802] (issue: {issue}37712[#37712])
* Enforce cluster UUIDs {pull}37775[#37775]
* Bubble exceptions up in ClusterApplierService {pull}37729[#37729]
*  Use m_m_nodes from Zen1 master for Zen2 bootstrap {pull}37701[#37701]
*  Add tool elasticsearch-node unsafe-bootstrap {pull}37696[#37696]
* Report terms and version if cluster does not form {pull}37473[#37473]
* Bootstrap a Zen2 cluster once quorum is discovered {pull}37463[#37463]
* Zen2: Add join validation {pull}37203[#37203]
* Publish cluster states in chunks {pull}36973[#36973]



[[bug-7.0.0-beta1]]
[float]
=== Bug fixes

Aggregations::
* Don't load global ordinals with the `map` execution_hint {pull}37833[#37833] (issue: {issue}37705[#37705])
* Issue #37303 - Invalid variance fix {pull}37384[#37384] (issue: {issue}37303[#37303])

Allocation::
* Fix _host based require filters {pull}38173[#38173]
* ALLOC: Fail Stale Primary Alloc. Req. without Data {pull}37226[#37226] (issue: {issue}37098[#37098])

Audit::
* Fix NPE in Logfile Audit Filter {pull}38120[#38120] (issue: {issue}38097[#38097])

Authentication::
*  Enhance parsing of StatusCode in SAML Responses {pull}38628[#38628]
* Limit token expiry to 1 hour maximum {pull}38244[#38244]
* Fix expired token message in Exception header {pull}37196[#37196]
* Fix NPE in CachingUsernamePasswordRealm {pull}36953[#36953] (issue: {issue}36951[#36951])

CCR::
* Prevent CCR recovery from missing documents {pull}38237[#38237]
* Fix file reading in ccr restore service {pull}38117[#38117]
* Correct argument names in update mapping/settings from leader {pull}38063[#38063]
* Ensure changes requests return the latest mapping version {pull}37633[#37633]
* Do not set fatal exception when shard follow task is stopped. {pull}37603[#37603]
* Add fatal_exception field for ccr stats in monitoring mapping {pull}37563[#37563]
* Do not add index event listener if CCR disabled {pull}37432[#37432]
* When removing an AutoFollower also mark it as removed. {pull}37402[#37402] (issue: {issue}36761[#36761])
* [CCR] Make shard follow tasks more resilient for restarts {pull}37239[#37239] (issue: {issue}37231[#37231])
* [CCR] Resume follow Api should not require a request body {pull}37217[#37217] (issue: {issue}37022[#37022])
* [CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled {pull}36886[#36886] (issue: {issue}33007[#33007])
* Remote cluster license checker and no license info. {pull}36837[#36837] (issue: {issue}36815[#36815])
* Make CCR resilient against missing remote cluster connections {pull}36682[#36682] (issues: {issue}36255[#36255], {issue}36667[#36667])
* [CCR] AutoFollowCoordinator and follower index already created {pull}36540[#36540] (issue: {issue}33007[#33007])

CRUD::
* Fix Reindex from remote query logic {pull}36908[#36908]
* Synchronize WriteReplicaResult callbacks {pull}36770[#36770]

Distributed::
* TransportVerifyShardBeforeCloseAction should force a flush {pull}38401[#38401] (issues: {issue}33888[#33888], {issue}37961[#37961])
* Fix limit on retaining sequence number {pull}37992[#37992] (issue: {issue}37165[#37165])
* Close Index API should force a flush if a sync is needed {pull}37961[#37961] (issues: {issue}33888[#33888], {issue}37426[#37426])
* Force Refresh Listeners when Acquiring all Operation Permits {pull}36835[#36835]
* Replaced the word 'shards' with 'replicas' in an error message. (#36234) {pull}36275[#36275] (issue: {issue}36234[#36234])

Engine::
* Subclass NIOFSDirectory instead of using FileSwitchDirectory {pull}37140[#37140] (issues: {issue}36668[#36668], {issue}37111[#37111])

Features/ILM::
* Preserve ILM operation mode when creating new lifecycles {pull}38134[#38134] (issues: {issue}38229[#38229], {issue}38230[#38230])
* Retry ILM steps that fail due to SnapshotInProgressException {pull}37624[#37624] (issues: {issue}37541[#37541], {issue}37552[#37552])
* Remove `indexing_complete` when removing policy {pull}36620[#36620]

Features/Indices APIs::
* Reject delete index requests with a body {pull}37501[#37501] (issue: {issue}8217[#8217])
* Fix duplicate phrase in shrink/split error message {pull}36734[#36734] (issue: {issue}36729[#36729])
* Get Aliases with wildcard exclusion expression {pull}34230[#34230] (issues: {issue}33518[#33518], {issue}33805[#33805], {issue}34144[#34144])

Features/Ingest::
* Support unknown fields in ingest pipeline map configuration {pull}38352[#38352] (issue: {issue}36938[#36938])
* Ingest node - user_agent, move device parsing to an object {pull}38115[#38115] (issues: {issue}37329[#37329], {issue}38094[#38094])
* ingest: fix on_failure with Drop processor {pull}36686[#36686] (issue: {issue}36151[#36151])
* ingest: support default pipelines + bulk upserts {pull}36618[#36618] (issue: {issue}36219[#36219])

Features/Java High Level REST Client::
* Update IndexTemplateMetaData to allow unknown fields {pull}38448[#38448] (issue: {issue}36938[#36938])
* `if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API {pull}38411[#38411]
* Update Rollup Caps to allow unknown fields {pull}38339[#38339] (issue: {issue}36938[#36938])
* Fix ILM explain response to allow unknown fields {pull}38054[#38054] (issue: {issue}36938[#36938])
* Fix ILM status to allow unknown fields {pull}38043[#38043] (issue: {issue}36938[#36938])
* Fix ILM Lifecycle Policy to allow unknown fields {pull}38041[#38041] (issue: {issue}36938[#36938])
* Update authenticate to allow unknown fields {pull}37713[#37713] (issue: {issue}36938[#36938])
* Update verify repository to allow unknown fields {pull}37619[#37619] (issue: {issue}36938[#36938])
* Update get users to allow unknown fields {pull}37593[#37593] (issue: {issue}36938[#36938])
* Update Execute Watch to allow unknown fields {pull}37498[#37498] (issue: {issue}36938[#36938])
* Update Put Watch to allow unknown fields {pull}37494[#37494] (issue: {issue}36938[#36938])
* Update Delete Watch to allow unknown fields {pull}37435[#37435] (issue: {issue}36938[#36938])
* Fix rest reindex test for IPv4 addresses {pull}37310[#37310]
* Fix weighted_avg parser not found for RestHighLevelClient {pull}37027[#37027] (issue: {issue}36861[#36861])

Features/Java Low Level REST Client::
* Fix potential IllegalCapacityException in LLRC when selecting nodes {pull}37821[#37821]

Features/Monitoring::
* Allow built-in monitoring_user role to call GET _xpack API {pull}38060[#38060] (issue: {issue}37970[#37970])

Features/Watcher::
* Support merge nested Map in list for JIRA configurations {pull}37634[#37634] (issue: {issue}30068[#30068])
* Watcher accounts constructed lazily {pull}36656[#36656]
* Ensures watch definitions are valid json {pull}30692[#30692] (issue: {issue}29746[#29746])

Geo::
* Fix GeoHash PrefixTree BWC {pull}38584[#38584] (issue: {issue}38494[#38494])
* Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297])

Infra/Core::
* Bubble-up exceptions from scheduler {pull}38317[#38317] (issue: {issue}38014[#38014])
* Core: Revert back to joda's multi date formatters {pull}36814[#36814] (issues: {issue}36447[#36447], {issue}36602[#36602])
* Propagate Errors in executors to uncaught exception handler {pull}36137[#36137] (issue: {issue}28667[#28667])

Infra/Packaging::
* Remove NOREPLACE for /etc/elasticsearch in rpm and deb {pull}37839[#37839]
* Packaging: Update marker used to allow ELASTIC_PASSWORD {pull}37243[#37243] (issue: {issue}37240[#37240])
* Packaging: Remove permission editing in postinst {pull}37242[#37242] (issue: {issue}37143[#37143])

Infra/REST API::
* Reject all requests that have an unconsumed body {pull}37504[#37504] (issues: {issue}30792[#30792], {issue}37501[#37501], {issue}8217[#8217])

Infra/Scripting::
* Fix Painless void return bug {pull}38046[#38046]

Infra/Settings::
* Fix setting by time unit {pull}37192[#37192]
* Fix handling of fractional byte size value settings {pull}37172[#37172]
* Fix handling of fractional time value settings {pull}37171[#37171]

Machine Learning::
* [ML] Report index unavailable instead of waiting for lazy node {pull}38423[#38423]
* ML: Fix error race condition on stop _all datafeeds and close _all jobs {pull}38113[#38113] (issue: {issue}37959[#37959])
* [ML] Update ML results mappings on process start {pull}37706[#37706] (issue: {issue}37607[#37607])
* [ML] Prevent submit after autodetect worker is stopped {pull}37700[#37700] (issue: {issue}37108[#37108])
* [ML] Fix ML datafeed CCS with wildcarded cluster name {pull}37470[#37470] (issue: {issue}36228[#36228])
* [ML] Update error message for process update {pull}37363[#37363]
* [ML] Wait for autodetect to be ready in the datafeed {pull}37349[#37349] (issues: {issue}36810[#36810], {issue}37227[#37227])
* [ML] Stop datafeeds running when their jobs are stale {pull}37227[#37227] (issue: {issue}36810[#36810])
* [ML] Order GET job stats response by job id {pull}36841[#36841] (issue: {issue}36683[#36683])
* [ML] Make GetJobStats work with arbitrary wildcards and groups {pull}36683[#36683] (issue: {issue}34745[#34745])

Mapping::
* Treat put-mapping calls with `_doc` as a top-level key as typed calls. {pull}38032[#38032]
* Correct deprec log in RestGetFieldMappingAction {pull}37843[#37843] (issue: {issue}37667[#37667])
* Restore a noop _all metadata field for 6x indices {pull}37808[#37808] (issue: {issue}37429[#37429])
* Make sure PutMappingRequest accepts content types other than JSON. {pull}37720[#37720]
* Make sure to use the resolved type in DocumentMapperService#extractMappings. {pull}37451[#37451] (issue: {issue}36811[#36811])
* MAPPING: Improve Precision for scaled_float {pull}37169[#37169] (issue: {issue}32570[#32570])
* Make sure to accept empty unnested mappings in create index requests. {pull}37089[#37089]
* Stop automatically nesting mappings in index creation requests. {pull}36924[#36924]
* Rewrite SourceToParse with resolved docType {pull}36921[#36921] (issues: {issue}35790[#35790], {issue}36769[#36769])

Network::
* Reload SSL context on file change for LDAP {pull}36937[#36937] (issues: {issue}30509[#30509], {issue}36923[#36923])
* Do not resolve addresses in remote connection info {pull}36671[#36671] (issue: {issue}35658[#35658])

Ranking::
* QueryRescorer should keep the window size when rewriting {pull}36836[#36836]

Recovery::
* RecoveryMonitor#lastSeenAccessTime should be volatile {pull}36781[#36781]

Rollup::
* Fix Rollup's metadata parser {pull}36791[#36791] (issue: {issue}36726[#36726])
* Fix rollup search statistics {pull}36674[#36674]

SQL::
* SQL: Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308])
* SQL: Relax StackOverflow circuit breaker for constants {pull}38572[#38572] (issue: {issue}38571[#38571])
* SQL: Fix issue with IN not resolving to underlying keyword field {pull}38440[#38440] (issue: {issue}38424[#38424])
* SQL: change the Intervals milliseconds precision to 3 digits {pull}38297[#38297] (issue: {issue}37423[#37423])
* SQL: Fix esType for DATETIME/DATE and INTERVALS {pull}38179[#38179] (issue: {issue}38051[#38051])
* SQL: Added SSL configuration options tests {pull}37875[#37875] (issue: {issue}37711[#37711])
* SQL: Fix casting from date to numeric type to use millis {pull}37869[#37869] (issue: {issue}37655[#37655])
* SQL: Fix BasicFormatter NPE {pull}37804[#37804]
* SQL: Return Intervals in SQL format for CLI {pull}37602[#37602] (issues: {issue}29970[#29970], {issue}36186[#36186], {issue}36432[#36432])
* SQL: fix object extraction from sources {pull}37502[#37502] (issue: {issue}37364[#37364])
* SQL: Fix issue with field names containing "." {pull}37364[#37364] (issue: {issue}37128[#37128])
* SQL: Fix bug regarding alias fields with dots {pull}37279[#37279] (issue: {issue}37224[#37224])
* SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {pull}37254[#37254] (issue: {issue}30285[#30285])
* SQL: fix COUNT DISTINCT filtering {pull}37176[#37176] (issue: {issue}37086[#37086])
* SQL: Fix issue with wrong NULL optimization {pull}37124[#37124] (issue: {issue}35872[#35872])
* SQL: Fix issue with complex expression as args of PERCENTILE/_RANK {pull}37102[#37102] (issue: {issue}37099[#37099])
* SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless {pull}37024[#37024] (issue: {issue}37023[#37023])
* SQL: Fix bug regarding histograms usage in scripting {pull}36866[#36866]
* SQL: Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980])
* SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756])
* SQL: Enhance Verifier to prevent aggregate or grouping functions from {pull}36799[#36799] (issue: {issue}36798[#36798])
* SQL: Fix translation of LIKE/RLIKE keywords {pull}36672[#36672] (issues: {issue}36039[#36039], {issue}36584[#36584])
* SQL: Scripting support for casting functions CAST and CONVERT {pull}36640[#36640] (issue: {issue}36061[#36061])
* SQL: Concat should be always not nullable {pull}36601[#36601] (issue: {issue}36169[#36169])
* SQL: Fix issue with complex HAVING and GROUP BY ordinal {pull}36594[#36594] (issue: {issue}36059[#36059])

Search::
* Look up connection using the right cluster alias when releasing contexts {pull}38570[#38570]
* Fix fetch source option in expand search phase {pull}37908[#37908] (issue: {issue}23829[#23829])
* Change `rational` to `saturation` in script_score {pull}37766[#37766] (issue: {issue}37714[#37714])
* Throw if two inner_hits have the same name {pull}37645[#37645] (issue: {issue}37584[#37584])
* Ensure either success or failure path for SearchOperationListener is called {pull}37467[#37467] (issue: {issue}37185[#37185])
* `query_string` should use indexed prefixes {pull}36895[#36895]
* Avoid duplicate types deprecation messages in search-related APIs. {pull}36802[#36802]

Security::
* Fix exit code for Security CLI tools  {pull}37956[#37956] (issue: {issue}37841[#37841])
* Fix potential NPE in UsersTool {pull}37660[#37660]

Snapshot/Restore::
* Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {pull}38368[#38368] (issue: {issue}38226[#38226])
* Fix Two Races that Lead to Stuck Snapshots {pull}37686[#37686] (issues: {issue}32265[#32265], {issue}32348[#32348])
* Fix Race in Concurrent Snapshot Delete and Create {pull}37612[#37612] (issue: {issue}37581[#37581])
* Streamline S3 Repository- and Client-Settings {pull}37393[#37393]

Suggesters::
* Fix duplicate removal when merging completion suggestions {pull}36996[#36996] (issue: {issue}35836[#35836])

Task Management::
* Un-assign persistent tasks as nodes exit the cluster {pull}37656[#37656]

ZenDiscovery::
* Fix size of rolling-upgrade bootstrap config {pull}38031[#38031]
* Always return metadata version if metadata is requested {pull}37674[#37674]
* [Zen2] Elect freshest master in upgrade {pull}37122[#37122] (issue: {issue}40[#40])
* Fix cluster state persistence for single-node discovery {pull}36825[#36825]



[[regression-7.0.0-beta1]]
[float]
=== Regressions

Infra/Core::
* Restore date aggregation performance in UTC case {pull}38221[#38221] (issue: {issue}37826[#37826])
* Speed up converting of temporal accessor to zoned date time {pull}37915[#37915] (issue: {issue}37826[#37826])

Mapping::
* Performance fix. Reduce deprecation calls for the same bulk request {pull}37415[#37415] (issue: {issue}37411[#37411])



[[upgrade-7.0.0-beta1]]
[float]
=== Upgrades

Engine::
* Upgrade to lucene-8.0.0-snapshot-83f9835. {pull}37668[#37668]

Machine Learning::
* [ML] No need to add state doc mapping on job open in 7.x {pull}37759[#37759]