diff --git a/docs/reference/release-notes/7.0.0-beta1.asciidoc b/docs/reference/release-notes/7.0.0-beta1.asciidoc new file mode 100644 index 00000000000..e5b0eeac4db --- /dev/null +++ b/docs/reference/release-notes/7.0.0-beta1.asciidoc @@ -0,0 +1,691 @@ +[[release-notes-7.0.0-beta1]] +== 7.0.0-beta1 Release Notes + +coming[7.0.0-beta1] + +Also see <>. + +[[breaking-7.0.0-beta1]] +[float] +=== Breaking changes + +Audit:: +* Remove index audit output type {pull}37707[#37707] (issues: {issue}29881[#29881], {issue}37301[#37301]) + +Authentication:: +* Remove bwc logic for token invalidation {pull}36893[#36893] (issue: {issue}36727[#36727]) + +Authorization:: +* Remove implicit index monitor privilege {pull}37774[#37774] + +CCR:: +* Follow stats api should return a 404 when requesting stats for a non existing index {pull}37220[#37220] (issue: {issue}37021[#37021]) + +CRUD:: +* Remove support for internal versioning for concurrency control {pull}38254[#38254] (issue: {issue}1078[#1078]) + +Features/Ingest:: +* Add ECS schema for user-agent ingest processor (#37727) {pull}37984[#37984] (issues: {issue}37329[#37329], {issue}37727[#37727]) +* Remove special handling for ingest plugins {pull}36967[#36967] (issues: {issue}36898[#36898], {issue}36956[#36956]) + +Features/Java Low Level REST Client:: +* Drop support for the low-level REST client on JDK 7 {pull}38540[#38540] (issue: {issue}29607[#29607]) + +Features/Watcher:: +* Remove Watcher Account "unsecure" settings {pull}36736[#36736] (issue: {issue}36403[#36403]) + +Infra/Logging:: +* Elasticsearch json logging {pull}36833[#36833] (issue: {issue}32850[#32850]) + +Infra/Packaging:: +* Package ingest-user-agent as a module {pull}36956[#36956] +* Package ingest-geoip as a module {pull}36898[#36898] + +Machine Learning:: +* [ML] Remove types from datafeed {pull}36538[#36538] (issue: {issue}34265[#34265]) + +Mapping:: +* Make sure to reject mappings with type _doc when include_type_name is false. {pull}38270[#38270] (issue: {issue}38266[#38266]) +* Update the default for include_type_name to false. {pull}37285[#37285] +* Support 'include_type_name' in RestGetIndicesAction {pull}37149[#37149] + +Network:: +* Remove TLS 1.0 as a default SSL protocol {pull}37512[#37512] (issue: {issue}36021[#36021]) +* Security: remove SSL settings fallback {pull}36846[#36846] (issue: {issue}29797[#29797]) + +Ranking:: +* Forbid negative field boosts in analyzed queries {pull}37930[#37930] (issue: {issue}33309[#33309]) + +Search:: +* Track total hits up to 10,000 by default {pull}37466[#37466] (issue: {issue}33028[#33028]) +* Use mappings to format doc-value fields by default. {pull}30831[#30831] (issues: {issue}26948[#26948], {issue}29639[#29639]) + +Security:: +* Remove heuristics that enable security on trial licenses {pull}38075[#38075] (issue: {issue}38009[#38009]) + +ZenDiscovery:: +* Remove DiscoveryPlugin#getDiscoveryTypes {pull}38414[#38414] (issue: {issue}38410[#38410]) + + + +[[breaking-java-7.0.0-beta1]] +[float] +=== Breaking Java changes + +Features/Java Low Level REST Client:: +* Remove support for maxRetryTimeout from low-level REST client {pull}38085[#38085] (issues: {issue}25951[#25951], {issue}31834[#31834], {issue}33342[#33342]) + +Infra/Core:: +* Handle scheduler exceptions {pull}38014[#38014] (issues: {issue}28667[#28667], {issue}36137[#36137], {issue}37708[#37708]) + + + +[[deprecation-7.0.0-beta1]] +[float] +=== Deprecations + +Aggregations:: +* Deprecate dots in aggregation names {pull}31468[#31468] (issues: {issue}17600[#17600], {issue}19040[#19040]) + +Analysis:: +* [Analysis] Deprecate Standard Html Strip Analyzer in master {pull}26719[#26719] (issue: {issue}4704[#4704]) + +Audit:: +* Deprecate index audit output type {pull}37301[#37301] (issue: {issue}29881[#29881]) + +Features/Indices APIs:: +* Reject setting index.optimize_auto_generated_id after version 7.0.0 {pull}28895[#28895] (issue: {issue}27600[#27600]) + +Features/Ingest:: +* Deprecate `_type` in simulate pipeline requests {pull}37949[#37949] (issue: {issue}37731[#37731]) + +Features/Java High Level REST Client:: +* Deprecate HLRC security methods {pull}37883[#37883] (issues: {issue}36938[#36938], {issue}37540[#37540]) +* Deprecate HLRC EmptyResponse used by security {pull}37540[#37540] (issue: {issue}36938[#36938]) + +Features/Watcher:: +* Deprecate xpack.watcher.history.cleaner_service.enabled {pull}37782[#37782] (issue: {issue}32041[#32041]) +* deprecate types for watcher {pull}37594[#37594] (issue: {issue}35190[#35190]) + +Infra/Core:: +* Core: Deprecate negative epoch timestamps {pull}36793[#36793] +* Core: Deprecate use of scientific notation in epoch time parsing {pull}36691[#36691] + +Infra/Scripting:: +* Add types deprecation to script contexts {pull}37554[#37554] +* Deprecate _type from LeafDocLookup {pull}37491[#37491] +* Scripting: Remove deprecated params.ctx {pull}36848[#36848] (issue: {issue}34059[#34059]) + +Machine Learning:: +* Adding ml_settings entry to HLRC and Docs for deprecation_info {pull}38118[#38118] +* [ML] Datafeed deprecation checks {pull}38026[#38026] (issue: {issue}37932[#37932]) +* [ML] Remove "8" prefixes from file structure finder timestamp formats {pull}38016[#38016] +* [ML] Adjust structure finder for Joda to Java time migration {pull}37306[#37306] +* [ML] Resolve 7.0.0 TODOs in ML code {pull}36842[#36842] (issue: {issue}29963[#29963]) + +Mapping:: +* Deprecate types in rollover index API {pull}38039[#38039] (issue: {issue}35190[#35190]) +* Deprecate types in get field mapping API {pull}37667[#37667] (issue: {issue}35190[#35190]) +* Deprecate types in the put mapping API. {pull}37280[#37280] (issues: {issue}29453[#29453], {issue}37285[#37285]) +* Support include_type_name in the field mapping and index template APIs. {pull}37210[#37210] +* Deprecate types in create index requests. {pull}37134[#37134] (issues: {issue}29453[#29453], {issue}37285[#37285]) +* Deprecate use of the _type field in aggregations. {pull}37131[#37131] (issue: {issue}36802[#36802]) +* Deprecate reference to _type in lookup queries {pull}37016[#37016] (issue: {issue}35190[#35190]) +* Deprecate the document create endpoint. {pull}36863[#36863] +* Deprecate types in index API {pull}36575[#36575] (issues: {issue}35190[#35190], {issue}35790[#35790]) +* Deprecate types in update APIs {pull}36225[#36225] + +Search:: +* Deprecate use of type in reindex request body {pull}36823[#36823] +* Add typless endpoints for get_source and exist_source {pull}36426[#36426] + + + +[[feature-7.0.0-beta1]] +[float] +=== New features + +Authentication:: +* Add support for API keys to access Elasticsearch {pull}38291[#38291] (issue: {issue}34383[#34383]) +* OIDC realm authentication flows {pull}37787[#37787] +* [WIP] OIDC Realm JWT+JWS related functionality {pull}37272[#37272] (issues: {issue}35339[#35339], {issue}37009[#37009]) +* OpenID Connect Realm base functionality {pull}37009[#37009] (issue: {issue}35339[#35339]) + +Authorization:: +* Allow custom authorization with an authorization engine {pull}38358[#38358] (issues: {issue}32435[#32435], {issue}36245[#36245], {issue}37328[#37328], {issue}37495[#37495], {issue}37785[#37785], {issue}38137[#38137], {issue}38219[#38219]) +* WIldcard IndicesPermissions don't cover .security {pull}36765[#36765] + +CCR:: +* Add ccr follow info api {pull}37408[#37408] (issue: {issue}37127[#37127]) + +Features/ILM:: +* [ILM] Add unfollow action {pull}36970[#36970] (issue: {issue}34648[#34648]) + +Geo:: +* geotile_grid implementation {pull}37842[#37842] (issue: {issue}30240[#30240]) +* [GEO] Fork Lucene's LatLonShape Classes to local lucene package {pull}36794[#36794] +* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}36751[#36751] (issue: {issue}35320[#35320]) +* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach {pull}35320[#35320] (issue: {issue}32039[#32039]) + +Machine Learning:: +* ML: Adds set_upgrade_mode API endpoint {pull}37837[#37837] + +Mapping:: +* Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. {pull}37871[#37871] (issue: {issue}37773[#37773]) +* Add nanosecond field mapper {pull}37755[#37755] (issues: {issue}27330[#27330], {issue}32601[#32601]) + +SQL:: +* SQL: Allow sorting of groups by aggregates {pull}38042[#38042] (issue: {issue}35118[#35118]) +* SQL: Implement FIRST/LAST aggregate functions {pull}37936[#37936] (issue: {issue}35639[#35639]) +* SQL: Introduce SQL DATE data type {pull}37693[#37693] (issue: {issue}37340[#37340]) + +Search:: +* Introduce ability to minimize round-trips in CCS {pull}37828[#37828] (issues: {issue}32125[#32125], {issue}37566[#37566]) +* Add script filter to intervals {pull}36776[#36776] +* Add the ability to set the number of hits to track accurately {pull}36357[#36357] (issue: {issue}33028[#33028]) +* Add a maximum search request size. {pull}26423[#26423] + + + +[[enhancement-7.0.0-beta1]] +[float] +=== Enhancements + +Aggregations:: +* Add Composite to AggregationBuilders {pull}38207[#38207] (issue: {issue}38020[#38020]) +* Allow nested fields in the composite aggregation {pull}37178[#37178] (issue: {issue}28611[#28611]) +* Remove single shard optimization when suggesting shard_size {pull}37041[#37041] (issue: {issue}32125[#32125]) +* Use List instead of priority queue for stable sorting in bucket sort aggregator {pull}36748[#36748] (issue: {issue}36322[#36322]) +* Keys are compared in BucketSortPipelineAggregation so making key type… {pull}36407[#36407] + +Allocation:: +* Fail start on obsolete indices documentation {pull}37786[#37786] (issue: {issue}27073[#27073]) +* Fail start on invalid index metadata {pull}37748[#37748] (issue: {issue}27073[#27073]) +* Fail start of non-data node if node has data {pull}37347[#37347] (issue: {issue}27073[#27073]) + +Analysis:: +* Allow word_delimiter_graph_filter to not adjust internal offsets {pull}36699[#36699] (issues: {issue}33710[#33710], {issue}34741[#34741]) + +Audit:: +* Security Audit includes HTTP method for requests {pull}37322[#37322] (issue: {issue}29765[#29765]) +* Add X-Forwarded-For to the logfile audit {pull}36427[#36427] + +Authentication:: +* Security: propagate auth result to listeners {pull}36900[#36900] (issue: {issue}30794[#30794]) +* Security: reorder realms based on last success {pull}36878[#36878] +* Improve error message for 6.x style realm settings {pull}36876[#36876] (issues: {issue}30241[#30241], {issue}36026[#36026]) +* Change missing authn message to not mention tokens {pull}36750[#36750] +* Invalidate Token API enhancements - HLRC {pull}36362[#36362] (issue: {issue}35388[#35388]) +* Enhance Invalidate Token API {pull}35388[#35388] (issues: {issue}34556[#34556], {issue}35115[#35115]) + +Authorization:: +* Add apm_user reserved role {pull}38206[#38206] +* Permission for restricted indices {pull}37577[#37577] (issue: {issue}34454[#34454]) +* Remove kibana_user and kibana_dashboard_only_user index privileges {pull}37441[#37441] +* Create snapshot role {pull}35820[#35820] (issue: {issue}34454[#34454]) + +CCR:: +* Concurrent file chunk fetching for CCR restore {pull}38495[#38495] +* Tighten mapping syncing in ccr remote restore {pull}38071[#38071] (issues: {issue}36879[#36879], {issue}37887[#37887]) +* Do not allow put mapping on follower {pull}37675[#37675] (issue: {issue}30086[#30086]) +* Added ccr to xpack usage infrastructure {pull}37256[#37256] (issue: {issue}37221[#37221]) +* [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned {pull}37213[#37213] +* [CCR] Added auto_follow_exception.timestamp field to auto follow stats {pull}36947[#36947] +* [CCR] Add time since last auto follow fetch to auto follow stats {pull}36542[#36542] (issues: {issue}33007[#33007], {issue}35895[#35895]) + +CRUD:: +* Add Seq# based optimistic concurrency control to UpdateRequest {pull}37872[#37872] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Introduce ssl settings to reindex from remote {pull}37527[#37527] (issues: {issue}29755[#29755], {issue}37287[#37287]) +* Use Sequence number powered OCC for processing updates {pull}37308[#37308] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Document Seq No powered optimistic concurrency control {pull}37284[#37284] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Enable IPv6 URIs in reindex from remote {pull}36874[#36874] +* Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm {pull}36757[#36757] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Expose Sequence Number based Optimistic Concurrency Control in the rest layer {pull}36721[#36721] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Add doc's sequence number + primary term to GetResult and use it for updates {pull}36680[#36680] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Add seq no powered optimistic locking support to the index and delete transport actions {pull}36619[#36619] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* Set acking timeout to 0 on dynamic mapping update {pull}31140[#31140] (issues: {issue}30672[#30672], {issue}30844[#30844]) + +Distributed:: +* Recover retention leases during peer recovery {pull}38435[#38435] (issue: {issue}37165[#37165]) +* Lift retention lease expiration to index shard {pull}38380[#38380] (issues: {issue}37165[#37165], {issue}37963[#37963], {issue}38070[#38070]) +* Introduce retention lease background sync {pull}38262[#38262] (issue: {issue}37165[#37165]) +* Allow shards of closed indices to be replicated as regular shards {pull}38024[#38024] (issue: {issue}33888[#33888]) +* Expose retention leases in shard stats {pull}37991[#37991] (issue: {issue}37165[#37165]) +* Introduce retention leases versioning {pull}37951[#37951] (issue: {issue}37165[#37165]) +* Soft-deletes policy should always fetch latest leases {pull}37940[#37940] (issues: {issue}37165[#37165], {issue}37375[#37375]) +* Sync retention leases on expiration {pull}37902[#37902] (issue: {issue}37165[#37165]) +* Ignore shard started requests when primary term does not match {pull}37899[#37899] (issue: {issue}33888[#33888]) +* Move update and delete by query to use seq# for optimistic concurrency control {pull}37857[#37857] (issues: {issue}10708[#10708], {issue}36148[#36148], {issue}37639[#37639]) +* Introduce retention lease serialization {pull}37447[#37447] (issues: {issue}37165[#37165], {issue}37398[#37398]) +* Add run under primary permit method {pull}37440[#37440] (issue: {issue}37398[#37398]) +* Introduce retention lease syncing {pull}37398[#37398] (issue: {issue}37165[#37165]) +* Introduce retention lease persistence {pull}37375[#37375] (issue: {issue}37165[#37165]) +* Add validation for retention lease construction {pull}37312[#37312] (issue: {issue}37165[#37165]) +* Introduce retention lease expiration {pull}37195[#37195] (issue: {issue}37165[#37165]) +* Introduce shard history retention leases {pull}37167[#37167] (issue: {issue}37165[#37165]) +* [Close Index API] Add unique UUID to ClusterBlock {pull}36775[#36775] +* [Close Index API] Mark shard copy as stale if needed during shard verification {pull}36755[#36755] +* [Close Index API] Propagate tasks ids between Freeze, Close and Verify Shard actions {pull}36630[#36630] +* Always initialize the global checkpoint {pull}34381[#34381] + +Engine:: +* Specialize pre-closing checks for engine implementations {pull}38702[#38702] +* Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines {pull}37426[#37426] +* Enable Bulk-Merge if all source remains {pull}37269[#37269] +* Rename setting to enable mmap {pull}37070[#37070] (issue: {issue}36668[#36668]) +* Add hybridfs store type {pull}36668[#36668] +* Introduce time-based retention policy for soft-deletes {pull}34943[#34943] (issue: {issue}34908[#34908]) +* handle AsyncAfterWriteAction failure on primary in the same way as failures on replicas {pull}31969[#31969] (issues: {issue}31716[#31716], {issue}31755[#31755]) + +Features/CAT APIs:: +* Expose `search.throttled` on `_cat/indices` {pull}37073[#37073] (issue: {issue}34352[#34352]) + +Features/Features:: +* Run Node deprecation checks locally (#38065) {pull}38250[#38250] (issue: {issue}38065[#38065]) + +Features/ILM:: +* Ensure ILM policies run safely on leader indices {pull}38140[#38140] (issue: {issue}34648[#34648]) +* Skip Shrink when numberOfShards not changed {pull}37953[#37953] (issue: {issue}33275[#33275]) +* Inject Unfollow before Rollover and Shrink {pull}37625[#37625] (issue: {issue}34648[#34648]) +* Add set_priority action to ILM {pull}37397[#37397] (issue: {issue}36905[#36905]) +* [ILM] Add Freeze Action {pull}36910[#36910] (issue: {issue}34630[#34630]) + +Features/Indices APIs:: +* New mapping signature and mapping string source fixed. {pull}37401[#37401] + +Features/Ingest:: +* ingest: compile mustache template only if field includes '{{'' {pull}37207[#37207] (issue: {issue}37120[#37120]) +* Move ingest-geoip default databases out of config {pull}36949[#36949] (issue: {issue}36898[#36898]) +* Make the ingest-geoip databases even lazier to load {pull}36679[#36679] +* Updates the grok patterns to be consistent with the logstash {pull}27181[#27181] + +Features/Java High Level REST Client:: +* HLRC: Fix strict setting exception handling {pull}37247[#37247] (issue: {issue}37090[#37090]) +* HLRC: Use nonblocking entity for requests {pull}32249[#32249] + +Features/Monitoring:: +* Adding mapping for hostname field {pull}37288[#37288] + +Features/Stats:: +* Stats: Add JVM dns cache expiration config to JvmInfo {pull}36372[#36372] + +Features/Watcher:: +* Move watcher to use seq# and primary term for concurrency control {pull}37977[#37977] (issues: {issue}10708[#10708], {issue}37872[#37872]) +* Use ILM for Watcher history deletion {pull}37443[#37443] (issue: {issue}32041[#32041]) +* Watcher: Add whitelist to HttpClient {pull}36817[#36817] (issue: {issue}29937[#29937]) + +Infra/Core:: +* fix a few versionAdded values in ElasticsearchExceptions {pull}37877[#37877] +* Add simple method to write collection of writeables {pull}37448[#37448] (issue: {issue}37398[#37398]) +* Date/Time parsing: Use java time API instead of exception handling {pull}37222[#37222] +* [API] spelling: interruptible {pull}37049[#37049] (issue: {issue}37035[#37035]) + +Infra/Logging:: +* Trim the JSON source in indexing slow logs {pull}38081[#38081] (issue: {issue}38080[#38080]) +* Optimize warning header de-duplication {pull}37725[#37725] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597], {issue}37622[#37622]) +* Remove warn-date from warning headers {pull}37622[#37622] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597]) +* Add some deprecation optimizations {pull}37597[#37597] (issues: {issue}35754[#35754], {issue}37530[#37530]) +* Only update response headers if we have a new one {pull}37590[#37590] (issues: {issue}35754[#35754], {issue}37530[#37530]) + +Infra/Packaging:: +* Add OS/architecture classifier to distributions {pull}37881[#37881] +* Change file descriptor limit to 65535 {pull}37537[#37537] (issue: {issue}35839[#35839]) +* Exit batch files explictly using ERRORLEVEL {pull}29583[#29583] (issue: {issue}29582[#29582]) + +Infra/Scripting:: +* Add getZone to JodaCompatibleZonedDateTime {pull}37084[#37084] +* [Painless] Add boxed type to boxed type casts for method/return {pull}36571[#36571] + +Infra/Settings:: +* Separate out validation of groups of settings {pull}34184[#34184] + +License:: +* Handle malformed license signatures {pull}37137[#37137] (issue: {issue}35340[#35340]) + +Machine Learning:: +* Move ML Optimistic Concurrency Control to Seq No {pull}38278[#38278] (issues: {issue}10708[#10708], {issue}36148[#36148]) +* [ML] Add explanation so far to file structure finder exceptions {pull}38191[#38191] (issue: {issue}29821[#29821]) +* ML: Add reason field in JobTaskState {pull}38029[#38029] (issue: {issue}34431[#34431]) +* [ML] Add _meta information to all ML indices {pull}37964[#37964] +* ML: Add upgrade mode docs, hlrc, and fix bug {pull}37942[#37942] +* [ML] Tighten up use of aliases rather than concrete indices {pull}37874[#37874] +* ML: Add support for single bucket aggs in Datafeeds {pull}37544[#37544] (issue: {issue}36838[#36838]) +* [ML] Create the ML annotations index {pull}36731[#36731] (issues: {issue}26034[#26034], {issue}33376[#33376]) +* [ML] Merge the Jindex master feature branch {pull}36702[#36702] (issue: {issue}32905[#32905]) +* [FEATURE][ML] Add cluster setting to enable/disable config migration {pull}36700[#36700] (issue: {issue}32905[#32905]) + +Mapping:: +* Log document id when MapperParsingException occurs {pull}37800[#37800] (issue: {issue}37658[#37658]) +* [API] spelling: unknown {pull}37056[#37056] (issue: {issue}37035[#37035]) +* Make SourceToParse immutable {pull}36971[#36971] +* Use index-prefix fields for terms of length min_chars - 1 {pull}36703[#36703] + +Network:: +* Enable TLSv1.3 by default for JDKs with support {pull}38103[#38103] (issue: {issue}32276[#32276]) + +Recovery:: +* SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence {pull}37691[#37691] (issue: {issue}33888[#33888]) +* Make prepare engine step of recovery source non-blocking {pull}37573[#37573] (issue: {issue}37174[#37174]) +* Make recovery source send operations non-blocking {pull}37503[#37503] (issue: {issue}37458[#37458]) +* Prepare to make send translog of recovery non-blocking {pull}37458[#37458] (issue: {issue}37291[#37291]) +* Make finalize step of recovery source non-blocking {pull}37388[#37388] (issue: {issue}37291[#37291]) +* Make recovery source partially non-blocking {pull}37291[#37291] (issue: {issue}36195[#36195]) +* Do not mutate RecoveryResponse {pull}37204[#37204] (issue: {issue}37174[#37174]) +* Don't block on peer recovery on the target side {pull}37076[#37076] (issue: {issue}36195[#36195]) +* Reduce recovery time with compress or secure transport {pull}36981[#36981] (issue: {issue}33844[#33844]) +* Translog corruption marker {pull}33415[#33415] (issue: {issue}31389[#31389]) + +Rollup:: +* Replace the TreeMap in the composite aggregation {pull}36675[#36675] + +SQL:: +* SQL: Allow look-ahead resolution of aliases for WHERE clause {pull}38450[#38450] (issue: {issue}29983[#29983]) +* SQL: Implement CURRENT_DATE {pull}38175[#38175] (issue: {issue}38160[#38160]) +* SQL: Generate relevant error message when grouping functions are not used in GROUP BY {pull}38017[#38017] (issue: {issue}37952[#37952]) +* SQL: Skip the nested and object field types in case of an ODBC request {pull}37948[#37948] (issue: {issue}37801[#37801]) +* SQL: Add protocol tests and remove jdbc_type from drivers response {pull}37516[#37516] (issues: {issue}36635[#36635], {issue}36882[#36882]) +* SQL: Remove slightly used meta commands {pull}37506[#37506] (issue: {issue}37409[#37409]) +* SQL: Describe aliases as views {pull}37496[#37496] (issue: {issue}37422[#37422]) +* SQL: Make `FULL` non-reserved keyword in the grammar {pull}37377[#37377] (issue: {issue}37376[#37376]) +* SQL: Use declared source for error messages {pull}37161[#37161] +* SQL: Improve error message when unable to translate to ES query DSL {pull}37129[#37129] (issue: {issue}37040[#37040]) +* [API] spelling: subtract {pull}37055[#37055] (issue: {issue}37035[#37035]) +* [API] spelling: similar {pull}37054[#37054] (issue: {issue}37035[#37035]) +* [API] spelling: input {pull}37048[#37048] (issue: {issue}37035[#37035]) +* SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg {pull}36933[#36933] (issue: {issue}36903[#36903]) +* SQL: Preserve original source for each expression {pull}36912[#36912] (issue: {issue}36894[#36894]) +* SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms {pull}36753[#36753] (issue: {issue}36740[#36740]) +* SQL: Fix wrong appliance of StackOverflow limit for IN {pull}36724[#36724] (issue: {issue}36592[#36592]) + +Search:: +* Tie break on cluster alias when merging shard search failures {pull}38715[#38715] (issue: {issue}38672[#38672]) +* Add finalReduce flag to SearchRequest {pull}38104[#38104] (issues: {issue}37000[#37000], {issue}37838[#37838]) +* Streamline skip_unavailable handling {pull}37672[#37672] (issue: {issue}32125[#32125]) +* Expose sequence number and primary terms in search responses {pull}37639[#37639] +* Add support for merging multiple search responses into one {pull}37566[#37566] (issue: {issue}32125[#32125]) +* Allow field types to optimize phrase prefix queries {pull}37436[#37436] (issue: {issue}31921[#31921]) +* Add support for providing absolute start time to SearchRequest {pull}37142[#37142] (issue: {issue}32125[#32125]) +* Ensure that local cluster alias is never treated as remote {pull}37121[#37121] (issues: {issue}32125[#32125], {issue}36997[#36997]) +* [API] spelling: cacheable {pull}37047[#37047] (issue: {issue}37035[#37035]) +* Add ability to suggest shard_size on coord node rewrite {pull}37017[#37017] (issues: {issue}32125[#32125], {issue}36997[#36997], {issue}37000[#37000]) +* Skip final reduction if SearchRequest holds a cluster alias {pull}37000[#37000] (issues: {issue}32125[#32125], {issue}36997[#36997]) +* Add support for local cluster alias to SearchRequest {pull}36997[#36997] (issue: {issue}32125[#32125]) +* Use SearchRequest copy constructor in ExpandSearchPhase {pull}36772[#36772] (issue: {issue}36641[#36641]) +* Add raw sort values to SearchSortValues transport serialization {pull}36617[#36617] (issue: {issue}32125[#32125]) + +Security:: +* Move CAS operations in TokenService to sequence numbers {pull}38311[#38311] (issues: {issue}10708[#10708], {issue}37872[#37872]) +* Cleanup construction of interceptors {pull}38294[#38294] +* Add passphrase support to elasticsearch-keystore {pull}37472[#37472] (issue: {issue}32691[#32691]) + +Snapshot/Restore:: +* RestoreService should update primary terms when restoring shards of existing indices {pull}38177[#38177] (issue: {issue}33888[#33888]) +* Allow open indices to be restored {pull}37733[#37733] +* Create specific exception for when snapshots are in progress {pull}37550[#37550] (issue: {issue}37541[#37541]) +* SNAPSHOT: Make Atomic Blob Writes Mandatory {pull}37168[#37168] (issues: {issue}37011[#37011], {issue}37066[#37066]) +* SNAPSHOT: Speed up HDFS Repository Writes {pull}37069[#37069] +* Implement Atomic Blob Writes for HDFS Repository {pull}37066[#37066] (issue: {issue}37011[#37011]) +* [API] spelling: repositories {pull}37053[#37053] (issue: {issue}37035[#37035]) +* SNAPSHOT: Use CancellableThreads to Abort {pull}35901[#35901] (issue: {issue}21759[#21759]) +* WIP: S3 client encryption {pull}30513[#30513] (issues: {issue}11128[#11128], {issue}16843[#16843]) + +Suggesters:: +* Remove unused empty constructors from suggestions classes {pull}37295[#37295] +* [API] spelling: likelihood {pull}37052[#37052] (issue: {issue}37035[#37035]) + +ZenDiscovery:: +* Add elasticsearch-node detach-cluster tool {pull}37979[#37979] +* Deprecate minimum_master_nodes {pull}37868[#37868] +* Step down as master when configured out of voting configuration {pull}37802[#37802] (issue: {issue}37712[#37712]) +* Enforce cluster UUIDs {pull}37775[#37775] +* Bubble exceptions up in ClusterApplierService {pull}37729[#37729] +* Use m_m_nodes from Zen1 master for Zen2 bootstrap {pull}37701[#37701] +* Add tool elasticsearch-node unsafe-bootstrap {pull}37696[#37696] +* Report terms and version if cluster does not form {pull}37473[#37473] +* Bootstrap a Zen2 cluster once quorum is discovered {pull}37463[#37463] +* Zen2: Add join validation {pull}37203[#37203] +* Publish cluster states in chunks {pull}36973[#36973] + + + +[[bug-7.0.0-beta1]] +[float] +=== Bug fixes + +Aggregations:: +* Don't load global ordinals with the `map` execution_hint {pull}37833[#37833] (issue: {issue}37705[#37705]) +* Issue #37303 - Invalid variance fix {pull}37384[#37384] (issue: {issue}37303[#37303]) + +Allocation:: +* Fix _host based require filters {pull}38173[#38173] +* ALLOC: Fail Stale Primary Alloc. Req. without Data {pull}37226[#37226] (issue: {issue}37098[#37098]) + +Audit:: +* Fix NPE in Logfile Audit Filter {pull}38120[#38120] (issue: {issue}38097[#38097]) + +Authentication:: +* Enhance parsing of StatusCode in SAML Responses {pull}38628[#38628] +* Limit token expiry to 1 hour maximum {pull}38244[#38244] +* Fix expired token message in Exception header {pull}37196[#37196] +* Fix NPE in CachingUsernamePasswordRealm {pull}36953[#36953] (issue: {issue}36951[#36951]) + +CCR:: +* Prevent CCR recovery from missing documents {pull}38237[#38237] +* Fix file reading in ccr restore service {pull}38117[#38117] +* Correct argument names in update mapping/settings from leader {pull}38063[#38063] +* Ensure changes requests return the latest mapping version {pull}37633[#37633] +* Do not set fatal exception when shard follow task is stopped. {pull}37603[#37603] +* Add fatal_exception field for ccr stats in monitoring mapping {pull}37563[#37563] +* Do not add index event listener if CCR disabled {pull}37432[#37432] +* When removing an AutoFollower also mark it as removed. {pull}37402[#37402] (issue: {issue}36761[#36761]) +* [CCR] Make shard follow tasks more resilient for restarts {pull}37239[#37239] (issue: {issue}37231[#37231]) +* [CCR] Resume follow Api should not require a request body {pull}37217[#37217] (issue: {issue}37022[#37022]) +* [CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled {pull}36886[#36886] (issue: {issue}33007[#33007]) +* Remote cluster license checker and no license info. {pull}36837[#36837] (issue: {issue}36815[#36815]) +* Make CCR resilient against missing remote cluster connections {pull}36682[#36682] (issues: {issue}36255[#36255], {issue}36667[#36667]) +* [CCR] AutoFollowCoordinator and follower index already created {pull}36540[#36540] (issue: {issue}33007[#33007]) + +CRUD:: +* Fix Reindex from remote query logic {pull}36908[#36908] +* Synchronize WriteReplicaResult callbacks {pull}36770[#36770] + +Distributed:: +* TransportVerifyShardBeforeCloseAction should force a flush {pull}38401[#38401] (issues: {issue}33888[#33888], {issue}37961[#37961]) +* Fix limit on retaining sequence number {pull}37992[#37992] (issue: {issue}37165[#37165]) +* Close Index API should force a flush if a sync is needed {pull}37961[#37961] (issues: {issue}33888[#33888], {issue}37426[#37426]) +* Force Refresh Listeners when Acquiring all Operation Permits {pull}36835[#36835] +* Replaced the word 'shards' with 'replicas' in an error message. (#36234) {pull}36275[#36275] (issue: {issue}36234[#36234]) + +Engine:: +* Subclass NIOFSDirectory instead of using FileSwitchDirectory {pull}37140[#37140] (issues: {issue}36668[#36668], {issue}37111[#37111]) + +Features/ILM:: +* Preserve ILM operation mode when creating new lifecycles {pull}38134[#38134] (issues: {issue}38229[#38229], {issue}38230[#38230]) +* Retry ILM steps that fail due to SnapshotInProgressException {pull}37624[#37624] (issues: {issue}37541[#37541], {issue}37552[#37552]) +* Remove `indexing_complete` when removing policy {pull}36620[#36620] + +Features/Indices APIs:: +* Reject delete index requests with a body {pull}37501[#37501] (issue: {issue}8217[#8217]) +* Fix duplicate phrase in shrink/split error message {pull}36734[#36734] (issue: {issue}36729[#36729]) +* Get Aliases with wildcard exclusion expression {pull}34230[#34230] (issues: {issue}33518[#33518], {issue}33805[#33805], {issue}34144[#34144]) + +Features/Ingest:: +* Support unknown fields in ingest pipeline map configuration {pull}38352[#38352] (issue: {issue}36938[#36938]) +* Ingest node - user_agent, move device parsing to an object {pull}38115[#38115] (issues: {issue}37329[#37329], {issue}38094[#38094]) +* ingest: fix on_failure with Drop processor {pull}36686[#36686] (issue: {issue}36151[#36151]) +* ingest: support default pipelines + bulk upserts {pull}36618[#36618] (issue: {issue}36219[#36219]) + +Features/Java High Level REST Client:: +* Update IndexTemplateMetaData to allow unknown fields {pull}38448[#38448] (issue: {issue}36938[#36938]) +* `if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API {pull}38411[#38411] +* Update Rollup Caps to allow unknown fields {pull}38339[#38339] (issue: {issue}36938[#36938]) +* Fix ILM explain response to allow unknown fields {pull}38054[#38054] (issue: {issue}36938[#36938]) +* Fix ILM status to allow unknown fields {pull}38043[#38043] (issue: {issue}36938[#36938]) +* Fix ILM Lifecycle Policy to allow unknown fields {pull}38041[#38041] (issue: {issue}36938[#36938]) +* Update authenticate to allow unknown fields {pull}37713[#37713] (issue: {issue}36938[#36938]) +* Update verify repository to allow unknown fields {pull}37619[#37619] (issue: {issue}36938[#36938]) +* Update get users to allow unknown fields {pull}37593[#37593] (issue: {issue}36938[#36938]) +* Update Execute Watch to allow unknown fields {pull}37498[#37498] (issue: {issue}36938[#36938]) +* Update Put Watch to allow unknown fields {pull}37494[#37494] (issue: {issue}36938[#36938]) +* Update Delete Watch to allow unknown fields {pull}37435[#37435] (issue: {issue}36938[#36938]) +* Fix rest reindex test for IPv4 addresses {pull}37310[#37310] +* Fix weighted_avg parser not found for RestHighLevelClient {pull}37027[#37027] (issue: {issue}36861[#36861]) + +Features/Java Low Level REST Client:: +* Fix potential IllegalCapacityException in LLRC when selecting nodes {pull}37821[#37821] + +Features/Monitoring:: +* Allow built-in monitoring_user role to call GET _xpack API {pull}38060[#38060] (issue: {issue}37970[#37970]) + +Features/Watcher:: +* Support merge nested Map in list for JIRA configurations {pull}37634[#37634] (issue: {issue}30068[#30068]) +* Watcher accounts constructed lazily {pull}36656[#36656] +* Ensures watch definitions are valid json {pull}30692[#30692] (issue: {issue}29746[#29746]) + +Geo:: +* Fix GeoHash PrefixTree BWC {pull}38584[#38584] (issue: {issue}38494[#38494]) +* Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297]) + +Infra/Core:: +* Bubble-up exceptions from scheduler {pull}38317[#38317] (issue: {issue}38014[#38014]) +* Core: Revert back to joda's multi date formatters {pull}36814[#36814] (issues: {issue}36447[#36447], {issue}36602[#36602]) +* Propagate Errors in executors to uncaught exception handler {pull}36137[#36137] (issue: {issue}28667[#28667]) + +Infra/Packaging:: +* Remove NOREPLACE for /etc/elasticsearch in rpm and deb {pull}37839[#37839] +* Packaging: Update marker used to allow ELASTIC_PASSWORD {pull}37243[#37243] (issue: {issue}37240[#37240]) +* Packaging: Remove permission editing in postinst {pull}37242[#37242] (issue: {issue}37143[#37143]) + +Infra/REST API:: +* Reject all requests that have an unconsumed body {pull}37504[#37504] (issues: {issue}30792[#30792], {issue}37501[#37501], {issue}8217[#8217]) + +Infra/Scripting:: +* Fix Painless void return bug {pull}38046[#38046] + +Infra/Settings:: +* Fix setting by time unit {pull}37192[#37192] +* Fix handling of fractional byte size value settings {pull}37172[#37172] +* Fix handling of fractional time value settings {pull}37171[#37171] + +Machine Learning:: +* [ML] Report index unavailable instead of waiting for lazy node {pull}38423[#38423] +* ML: Fix error race condition on stop _all datafeeds and close _all jobs {pull}38113[#38113] (issue: {issue}37959[#37959]) +* [ML] Update ML results mappings on process start {pull}37706[#37706] (issue: {issue}37607[#37607]) +* [ML] Prevent submit after autodetect worker is stopped {pull}37700[#37700] (issue: {issue}37108[#37108]) +* [ML] Fix ML datafeed CCS with wildcarded cluster name {pull}37470[#37470] (issue: {issue}36228[#36228]) +* [ML] Update error message for process update {pull}37363[#37363] +* [ML] Wait for autodetect to be ready in the datafeed {pull}37349[#37349] (issues: {issue}36810[#36810], {issue}37227[#37227]) +* [ML] Stop datafeeds running when their jobs are stale {pull}37227[#37227] (issue: {issue}36810[#36810]) +* [ML] Order GET job stats response by job id {pull}36841[#36841] (issue: {issue}36683[#36683]) +* [ML] Make GetJobStats work with arbitrary wildcards and groups {pull}36683[#36683] (issue: {issue}34745[#34745]) + +Mapping:: +* Treat put-mapping calls with `_doc` as a top-level key as typed calls. {pull}38032[#38032] +* Correct deprec log in RestGetFieldMappingAction {pull}37843[#37843] (issue: {issue}37667[#37667]) +* Restore a noop _all metadata field for 6x indices {pull}37808[#37808] (issue: {issue}37429[#37429]) +* Make sure PutMappingRequest accepts content types other than JSON. {pull}37720[#37720] +* Make sure to use the resolved type in DocumentMapperService#extractMappings. {pull}37451[#37451] (issue: {issue}36811[#36811]) +* MAPPING: Improve Precision for scaled_float {pull}37169[#37169] (issue: {issue}32570[#32570]) +* Make sure to accept empty unnested mappings in create index requests. {pull}37089[#37089] +* Stop automatically nesting mappings in index creation requests. {pull}36924[#36924] +* Rewrite SourceToParse with resolved docType {pull}36921[#36921] (issues: {issue}35790[#35790], {issue}36769[#36769]) + +Network:: +* Reload SSL context on file change for LDAP {pull}36937[#36937] (issues: {issue}30509[#30509], {issue}36923[#36923]) +* Do not resolve addresses in remote connection info {pull}36671[#36671] (issue: {issue}35658[#35658]) + +Ranking:: +* QueryRescorer should keep the window size when rewriting {pull}36836[#36836] + +Recovery:: +* RecoveryMonitor#lastSeenAccessTime should be volatile {pull}36781[#36781] + +Rollup:: +* Fix Rollup's metadata parser {pull}36791[#36791] (issue: {issue}36726[#36726]) +* Fix rollup search statistics {pull}36674[#36674] + +SQL:: +* SQL: Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308]) +* SQL: Relax StackOverflow circuit breaker for constants {pull}38572[#38572] (issue: {issue}38571[#38571]) +* SQL: Fix issue with IN not resolving to underlying keyword field {pull}38440[#38440] (issue: {issue}38424[#38424]) +* SQL: change the Intervals milliseconds precision to 3 digits {pull}38297[#38297] (issue: {issue}37423[#37423]) +* SQL: Fix esType for DATETIME/DATE and INTERVALS {pull}38179[#38179] (issue: {issue}38051[#38051]) +* SQL: Added SSL configuration options tests {pull}37875[#37875] (issue: {issue}37711[#37711]) +* SQL: Fix casting from date to numeric type to use millis {pull}37869[#37869] (issue: {issue}37655[#37655]) +* SQL: Fix BasicFormatter NPE {pull}37804[#37804] +* SQL: Return Intervals in SQL format for CLI {pull}37602[#37602] (issues: {issue}29970[#29970], {issue}36186[#36186], {issue}36432[#36432]) +* SQL: fix object extraction from sources {pull}37502[#37502] (issue: {issue}37364[#37364]) +* SQL: Fix issue with field names containing "." {pull}37364[#37364] (issue: {issue}37128[#37128]) +* SQL: Fix bug regarding alias fields with dots {pull}37279[#37279] (issue: {issue}37224[#37224]) +* SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {pull}37254[#37254] (issue: {issue}30285[#30285]) +* SQL: fix COUNT DISTINCT filtering {pull}37176[#37176] (issue: {issue}37086[#37086]) +* SQL: Fix issue with wrong NULL optimization {pull}37124[#37124] (issue: {issue}35872[#35872]) +* SQL: Fix issue with complex expression as args of PERCENTILE/_RANK {pull}37102[#37102] (issue: {issue}37099[#37099]) +* SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless {pull}37024[#37024] (issue: {issue}37023[#37023]) +* SQL: Fix bug regarding histograms usage in scripting {pull}36866[#36866] +* SQL: Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980]) +* SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756]) +* SQL: Enhance Verifier to prevent aggregate or grouping functions from {pull}36799[#36799] (issue: {issue}36798[#36798]) +* SQL: Fix translation of LIKE/RLIKE keywords {pull}36672[#36672] (issues: {issue}36039[#36039], {issue}36584[#36584]) +* SQL: Scripting support for casting functions CAST and CONVERT {pull}36640[#36640] (issue: {issue}36061[#36061]) +* SQL: Concat should be always not nullable {pull}36601[#36601] (issue: {issue}36169[#36169]) +* SQL: Fix issue with complex HAVING and GROUP BY ordinal {pull}36594[#36594] (issue: {issue}36059[#36059]) + +Search:: +* Look up connection using the right cluster alias when releasing contexts {pull}38570[#38570] +* Fix fetch source option in expand search phase {pull}37908[#37908] (issue: {issue}23829[#23829]) +* Change `rational` to `saturation` in script_score {pull}37766[#37766] (issue: {issue}37714[#37714]) +* Throw if two inner_hits have the same name {pull}37645[#37645] (issue: {issue}37584[#37584]) +* Ensure either success or failure path for SearchOperationListener is called {pull}37467[#37467] (issue: {issue}37185[#37185]) +* `query_string` should use indexed prefixes {pull}36895[#36895] +* Avoid duplicate types deprecation messages in search-related APIs. {pull}36802[#36802] + +Security:: +* Fix exit code for Security CLI tools {pull}37956[#37956] (issue: {issue}37841[#37841]) +* Fix potential NPE in UsersTool {pull}37660[#37660] + +Snapshot/Restore:: +* Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {pull}38368[#38368] (issue: {issue}38226[#38226]) +* Fix Two Races that Lead to Stuck Snapshots {pull}37686[#37686] (issues: {issue}32265[#32265], {issue}32348[#32348]) +* Fix Race in Concurrent Snapshot Delete and Create {pull}37612[#37612] (issue: {issue}37581[#37581]) +* Streamline S3 Repository- and Client-Settings {pull}37393[#37393] + +Suggesters:: +* Fix duplicate removal when merging completion suggestions {pull}36996[#36996] (issue: {issue}35836[#35836]) + +Task Management:: +* Un-assign persistent tasks as nodes exit the cluster {pull}37656[#37656] + +ZenDiscovery:: +* Fix size of rolling-upgrade bootstrap config {pull}38031[#38031] +* Always return metadata version if metadata is requested {pull}37674[#37674] +* [Zen2] Elect freshest master in upgrade {pull}37122[#37122] (issue: {issue}40[#40]) +* Fix cluster state persistence for single-node discovery {pull}36825[#36825] + + + +[[regression-7.0.0-beta1]] +[float] +=== Regressions + +Infra/Core:: +* Restore date aggregation performance in UTC case {pull}38221[#38221] (issue: {issue}37826[#37826]) +* Speed up converting of temporal accessor to zoned date time {pull}37915[#37915] (issue: {issue}37826[#37826]) + +Mapping:: +* Performance fix. Reduce deprecation calls for the same bulk request {pull}37415[#37415] (issue: {issue}37411[#37411]) + + + +[[upgrade-7.0.0-beta1]] +[float] +=== Upgrades + +Engine:: +* Upgrade to lucene-8.0.0-snapshot-83f9835. {pull}37668[#37668] + +Machine Learning:: +* [ML] No need to add state doc mapping on job open in 7.x {pull}37759[#37759] + + +