[[release-notes-5.0.0-alpha1]] == 5.0.0-alpha1 Release Notes The changes listed below have been released for the first time in Elasticsearch 5.0.0-alpha1. Changes in this release which were first released in the 2.x series are listed in <>. Also see <>. IMPORTANT: This is an alpha release and is intended for _testing purposes only_. Indices created in this version will *not be compatible with Elasticsearch 5.0.0 GA*. Upgrading 5.0.0-alpha1 to any other version is not supported. [[breaking-5.0.0-alpha1]] [float] === Breaking changes Aggregations:: * getKeyAsString and key_as_string should be the same for terms aggregation on boolean field {pull}15393[#15393] Aliases:: * make get alias expand to open and closed indices by default {pull}15954[#15954] (issue: {issue}14982[#14982]) * Remove deprecated indices.get_aliases {pull}13906[#13906] Allocation:: * Simplify shard balancer interface {pull}17028[#17028] (issue: {issue}8954[#8954]) * Remove DisableAllocationDecider {pull}13313[#13313] Analysis:: * Simplify Analysis registration and configuration {pull}14355[#14355] CAT API:: * Add raw recovery progress to cat recovery API {pull}17064[#17064] (issue: {issue}17022[#17022]) * Remove host from cat nodes API {pull}16656[#16656] (issues: {issue}12959[#12959], {issue}16575[#16575]) * Using the accept header in the request instead of content-type in _cat API. {pull}14421[#14421] (issue: {issue}14195[#14195]) CRUD:: * Remove object notation for core types. {pull}15684[#15684] (issue: {issue}15388[#15388]) Cache:: * Refactor IndicesRequestCache to make it testable. {pull}16610[#16610] * Remove deprecated query cache settings {pull}15592[#15592] * Fold IndexCacheModule into IndexModule {pull}14293[#14293] Cluster:: * Remove memory section {pull}17278[#17278] (issues: {issue}12049[#12049], {issue}16756[#16756]) Core:: * Bootstrap does not set system properties {pull}17088[#17088] (issues: {issue}16579[#16579], {issue}16791[#16791]) * Add max number of processes check {pull}16919[#16919] * Add mlockall bootstrap check {pull}16909[#16909] * Remove es.useLinkedTransferQueue {pull}16786[#16786] * One log {pull}16703[#16703] (issue: {issue}16585[#16585]) Discovery:: * Include pings from client nodes in master election {pull}17329[#17329] (issue: {issue}17325[#17325]) Engine:: * Remove `index.compound_on_flush` setting and default to `true` {pull}15594[#15594] (issue: {issue}10778[#10778]) Fielddata:: * Remove "uninverted" and "binary" fielddata support for numeric and boolean fields. {pull}14082[#14082] Index APIs:: * Remove `GET` option for /_forcemerge {pull}15223[#15223] (issue: {issue}15165[#15165]) * Remove /_optimize REST API endpoint {pull}14226[#14226] (issue: {issue}13778[#13778]) Internal:: * Remove duplicate getters from DiscoveryNode and DiscoveryNodes {pull}17410[#17410] (issue: {issue}16963[#16963]) * Cli: Switch to jopt-simple {pull}17024[#17024] (issue: {issue}11564[#11564]) * Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation {pull}15776[#15776] * Remove NodeBuilder {pull}15354[#15354] * Fix IndexSearcherWrapper interface to not depend on the EngineConfig {pull}14654[#14654] * Cleanup query parsing and remove IndexQueryParserService {pull}14452[#14452] * Remove circular dependency between IndicesService and IndicesStore {pull}14285[#14285] * Remove guice injection from IndexStore and friends {pull}14279[#14279] * Replace IndicesLifecycle with a per-index IndexEventListener {pull}14217[#14217] (issue: {issue}13259[#13259]) * Simplify similarity module and friends {pull}13942[#13942] * Refactor SearchRequest to be parsed on the coordinating node {pull}13859[#13859] * Remove support for pre 2.0 indices {pull}13799[#13799] Java API:: * Remove the count api {pull}14166[#14166] (issue: {issue}13928[#13928]) * IdsQueryBuilder to accept only non null ids and types {pull}13937[#13937] Mapping:: * Change the field mapping index time boost into a query time boost. {pull}16900[#16900] * Deprecate string in favor of text/keyword. {pull}16877[#16877] * Term vector APIs should no longer update mappings {pull}16285[#16285] * [Mapping] Several MappingService cleanups {pull}16133[#16133] (issue: {issue}15924[#15924]) * Remove the `format` option of the `_source` field. {pull}15398[#15398] * Remove transform {pull}13657[#13657] (issue: {issue}12674[#12674]) Network:: * Remove ability to disable Netty gathering writes {pull}16774[#16774] (issue: {issue}7811[#7811]) Parent/Child:: * Removed `total` score mode in favour for `sum` score mode. {pull}17174[#17174] (issues: {issue}13470[#13470], {issue}17083[#17083]) * Cleanup ParentFieldMapper {pull}16045[#16045] * Several other parent/child cleanups {pull}13470[#13470] * Removed pre 2.x parent child implementation {pull}13376[#13376] Percolator:: * Change the percolate api to not dynamically add fields to mapping {pull}16077[#16077] (issue: {issue}15751[#15751]) Plugins:: * Enforce isolated mode for all plugins {pull}17276[#17276] * Rename bin/plugin in bin/elasticsearch-plugin {pull}16454[#16454] * Change the inner structure of the plugins zip {pull}16453[#16453] * Remove multicast plugin {pull}16326[#16326] (issue: {issue}16310[#16310]) * Plugins: Remove site plugins {pull}16038[#16038] * Don't use guice for QueryParsers {pull}15761[#15761] * Remove guice from the index level {pull}14518[#14518] * Remove shard-level injector {pull}13881[#13881] Query DSL:: * Remove the MissingQueryBuilder which was deprecated in 2.2.0. {pull}15364[#15364] (issue: {issue}14112[#14112]) * Remove NotQueryBuilder {pull}14204[#14204] (issue: {issue}13761[#13761]) * Function score query: remove deprecated support for boost_factor {pull}13510[#13510] * Remove support for deprecated queries. {pull}13418[#13418] (issue: {issue}13326[#13326]) REST:: * Limit the accepted length of the _id {pull}16036[#16036] (issue: {issue}16034[#16034]) Search:: * Remove deprecated reverse option from sorting {pull}17282[#17282] (issue: {issue}17047[#17047]) * Remove some deprecations {pull}14331[#14331] * Remove search exists api {pull}13911[#13911] (issues: {issue}13682[#13682], {issue}13910[#13910]) * Query refactoring: split parse phase into fromXContent and toQuery for all queries {pull}13788[#13788] (issue: {issue}10217[#10217]) * Remove the scan and count search types. {pull}13310[#13310] Search Refactoring:: * Refactored inner hits parsing and intoduced InnerHitBuilder {pull}17291[#17291] * Remove deprecated parameter from field sort builder. {pull}16573[#16573] (issue: {issue}16127[#16127]) * Remove support for query_binary and filter_binary {pull}14433[#14433] (issue: {issue}14308[#14308]) * Validate query api: move query parsing to the coordinating node {pull}14384[#14384] * Remove "query" query and fix related parsing bugs {pull}14304[#14304] (issue: {issue}13326[#13326]) Settings:: * Remove ability to specify arbitrary node attributes with `node.` prefix {pull}17402[#17402] (issue: {issue}17280[#17280]) * Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip {pull}17288[#17288] * Prevent index level setting from being configured on a node level {pull}17144[#17144] (issue: {issue}16799[#16799]) * Remove support for node.client setting {pull}16963[#16963] (issue: {issue}16565[#16565]) * Remove es.max-open-files flag {pull}16757[#16757] (issues: {issue}16506[#16506], {issue}483[#483]) * Enforce node level limits if node is started in production env {pull}16733[#16733] (issue: {issue}16727[#16727]) * Move remaining settings in NettyHttpServerTransport to the new infra {pull}16531[#16531] * Make settings validation strict {pull}16365[#16365] * Remove the ability to fsync on every operation and only schedule fsync task if really needed {pull}16257[#16257] (issue: {issue}16152[#16152]) * Script settings {pull}16197[#16197] * Remove index.flush_on_close entirely {pull}15977[#15977] * Restore chunksize of 512kb on recovery and remove configurability {pull}15235[#15235] (issue: {issue}15161[#15161]) * Remove ancient deprecated and alternative recovery settings {pull}15234[#15234] * Replace IndexSettings annotation with a full-fledged class {pull}14251[#14251] * Fix ping timeout settings inconsistencies {pull}13701[#13701] (issue: {issue}6579[#6579]) Similarities:: * Renames `default` similarity into `classic` {pull}15446[#15446] (issue: {issue}15102[#15102]) Snapshot/Restore:: * Fail closing or deleting indices during a full snapshot {pull}17021[#17021] (issue: {issue}16321[#16321]) Stats:: * Modify load average format {pull}15932[#15932] (issue: {issue}15907[#15907]) * Reintroduce five-minute and fifteen-minute load averages on Linux {pull}15907[#15907] (issues: {issue}12049[#12049], {issue}14741[#14741]) * Add system CPU percent to OS stats {pull}14741[#14741] Store:: * Standardize state format type for global and index level metadata {pull}17123[#17123] Suggesters:: * Remove suggest threadpool {pull}17304[#17304] (issue: {issue}17198[#17198]) * Remove suggest transport action {pull}17198[#17198] (issue: {issue}10217[#10217]) Term Vectors:: * Remove DFS support from TermVector API {pull}16452[#16452] Translog:: * Drop support for simple translog and hard-wire buffer to 8kb {pull}15574[#15574] * Simplify translog-based flush settings {pull}15573[#15573] Warmers:: * Remove query warmers and the warmer API. {pull}15614[#15614] (issue: {issue}15607[#15607]) [[deprecation-5.0.0-alpha1]] [float] === Deprecations Plugin Mapper Attachment:: * Deprecate mapper-attachments plugin {pull}16948[#16948] (issue: {issue}16910[#16910]) Search:: * Deprecate fuzzy query {pull}16211[#16211] (issues: {issue}15760[#15760], {issue}16121[#16121]) [[feature-5.0.0-alpha1]] [float] === New features Allocation:: * Add API to explain why a shard is or isn't assigned {pull}17305[#17305] (issue: {issue}14593[#14593]) Discovery:: * Add two phased commit to Cluster State publishing {pull}13062[#13062] Ingest:: * Merge feature/ingest branch into master branch {pull}16049[#16049] (issue: {issue}14049[#14049]) Mapping:: * Add a text field. {pull}16637[#16637] * Add a new `keyword` field. {pull}16589[#16589] Percolator:: * index the query terms from the percolator query {pull}13646[#13646] (issue: {issue}12664[#12664]) Plugin Ingest Attachment:: * Ingest: Add attachment processor {pull}16490[#16490] (issue: {issue}16303[#16303]) Plugin Mapper Attachment:: * Migrate mapper attachments plugin to main repository {pull}14605[#14605] Plugin Repository HDFS:: * HDFS Snapshot/Restore plugin {pull}15192[#15192] (issue: {issue}15191[#15191]) Query DSL:: * Adds a rewrite phase to queries on the shard level {pull}16870[#16870] (issue: {issue}9526[#9526]) Reindex API:: * Merge reindex to master {pull}16861[#16861] Scripting:: * Exceptions and Infinite Loop Checking {pull}15936[#15936] * Added a new scripting language (PlanA) {pull}15136[#15136] (issue: {issue}13084[#13084]) Search:: * Add `search_after` parameter in the SearchAPI {pull}16125[#16125] (issue: {issue}8192[#8192]) Settings:: * Add infrastructure to transactionally apply and reset dynamic settings {pull}15278[#15278] Stats:: * API for listing index file sizes {pull}16661[#16661] (issue: {issue}16131[#16131]) Suggesters:: * Add document-oriented completion suggester {pull}14410[#14410] (issue: {issue}10746[#10746]) Task Manager:: * Add task cancellation mechanism {pull}16320[#16320] * Make the Task object available to the action caller {pull}16033[#16033] * Task Management: Add framework for registering and communicating with tasks {pull}15347[#15347] (issue: {issue}15117[#15117]) [[enhancement-5.0.0-alpha1]] [float] === Enhancements Aggregations:: * Add tests and documentation for using `time_zone` in date range aggregation {pull}16955[#16955] (issue: {issue}10130[#10130]) * Fixes serialisation of Ranges {pull}16674[#16674] Allocation:: * Write shard state metadata as soon as shard is created / initializing {pull}16625[#16625] (issue: {issue}14739[#14739]) * Reuse existing allocation id for primary shard allocation {pull}16530[#16530] (issue: {issue}14739[#14739]) * Remove version in ShardRouting (now obsolete) {pull}16243[#16243] (issue: {issue}14739[#14739]) * Prefer nodes that previously held primary shard for primary shard allocation {pull}16096[#16096] (issue: {issue}14739[#14739]) * Extend reroute with an option to force assign stale primary shard copies {pull}15708[#15708] (issue: {issue}14739[#14739]) * Allocate primary shards based on allocation IDs {pull}15281[#15281] (issue: {issue}14739[#14739]) * Persist currently started allocation IDs to index metadata {pull}14964[#14964] (issue: {issue}14739[#14739]) * Use ObjectParser to parse AllocationID {pull}14962[#14962] (issue: {issue}14831[#14831]) * Persist allocation ID with shard state metadata on nodes {pull}14831[#14831] (issue: {issue}14739[#14739]) Analysis:: * Improve error message if resource files have illegal encoding {pull}17237[#17237] (issue: {issue}17212[#17212]) CAT API:: * Expose http address in cat/nodes {pull}16770[#16770] * [cat/recovery] Make recovery time a TimeValue() {pull}16743[#16743] (issue: {issue}9209[#9209]) * :CAT API: remove space at the end of a line {pull}15250[#15250] (issue: {issue}9464[#9464]) CRUD:: * CRUD: Allow to get and set ttl as a time value/string {pull}15047[#15047] Cache:: * Enable the indices request cache by default {pull}17162[#17162] (issues: {issue}16870[#16870], {issue}17134[#17134]) Cluster:: * Cluster Health should run on applied states, even if waitFor=0 {pull}17440[#17440] * Resolve index names to Index instances early {pull}17048[#17048] * Remove DiscoveryNode#shouldConnectTo method {pull}16898[#16898] (issue: {issue}16815[#16815]) * Fail demoted primary shards and retry request {pull}16415[#16415] (issue: {issue}14252[#14252]) * Illegal shard failure requests {pull}16275[#16275] * Shard failure requests for non-existent shards {pull}16089[#16089] (issue: {issue}14252[#14252]) * Add handling of channel failures when starting a shard {pull}16041[#16041] (issue: {issue}15895[#15895]) * Wait for new master when failing shard {pull}15748[#15748] (issue: {issue}14252[#14252]) * Master should wait on cluster state publication when failing a shard {pull}15468[#15468] (issue: {issue}14252[#14252]) * Split cluster state update tasks into roles {pull}14899[#14899] (issue: {issue}13627[#13627]) * Add timeout mechanism for sending shard failures {pull}14707[#14707] (issue: {issue}14252[#14252]) * Add listener mechanism for failures to send shard failed {pull}14295[#14295] (issue: {issue}14252[#14252]) Core:: * Remove PROTOTYPE from BulkItemResponse.Failure {pull}17433[#17433] (issue: {issue}17086[#17086]) * Throw an exception if Writeable.Reader reads null {pull}17332[#17332] * Remove PROTOTYPE from RescorerBuilders {pull}17330[#17330] * Port Primary Terms to master {pull}17044[#17044] (issues: {issue}14062[#14062], {issue}14651[#14651], {issue}17038[#17038]) * Use index UUID to lookup indices on IndicesService {pull}17001[#17001] * Add -XX+AlwaysPreTouch JVM flag {pull}16937[#16937] * Add max size virtual memory check {pull}16935[#16935] * Use and test relative time in TransportBulkAction {pull}16916[#16916] * Bump Elasticsearch version to 5.0.0-SNAPSHOT {pull}16862[#16862] * Assert that we can write in all data-path on startup {pull}16745[#16745] * Add G1GC check on startup {pull}16737[#16737] (issue: {issue}10740[#10740]) * Shards with heavy indexing should get more of the indexing buffer {pull}14121[#14121] * Remove and ban ImmutableMap {pull}13939[#13939] (issue: {issue}13224[#13224]) * Finish banning ImmutableSet {pull}13820[#13820] (issue: {issue}13224[#13224]) * Removes and bans ImmutableSet {pull}13754[#13754] (issue: {issue}13224[#13224]) * Remove and ban ImmutableMap#entrySet {pull}13724[#13724] * Forbid ForwardingSet {pull}13720[#13720] (issue: {issue}13224[#13224]) Discovery:: * Add a dedicate queue for incoming ClusterStates {pull}13303[#13303] (issue: {issue}13062[#13062]) Engine:: * Remove writeLockTimeout from InternalEngine {pull}16930[#16930] * Don't guard IndexShard#refresh calls by a check to isRefreshNeeded {pull}16118[#16118] * Never call a listerner under lock in InternalEngine {pull}15786[#15786] * Use System.nanoTime() to initialize Engine.lastWriteNanos {pull}14321[#14321] * Flush big merges automatically if shard is inactive {pull}14275[#14275] * Remove Engine.Create {pull}13955[#13955] * Remove the disabled autogenerated id optimization from InternalEngine {pull}13857[#13857] Exceptions:: * Fix typos in exception/assert/log messages in core module. {pull}16649[#16649] * Add field names to several mapping errors {pull}16508[#16508] (issue: {issue}16378[#16378]) * Add serialization support for more important IOExceptions {pull}15766[#15766] * Adds exception objects to log messages. {pull}14827[#14827] (issue: {issue}10021[#10021]) * Add stack traces to logged exceptions where missing {pull}13825[#13825] (issue: {issue}10021[#10021]) * Remove reflection hacks from ElasticsearchException {pull}13796[#13796] * Rename QueryParsingException to a more generic ParsingException {pull}13631[#13631] * Add *Exception(Throwable cause) constructors/ call where appropriate {pull}13544[#13544] (issue: {issue}10021[#10021]) Geo:: * Fix a potential parsing problem in GeoDistanceSortParser {pull}17111[#17111] * Geo: Add validation of shapes to ShapeBuilders {pull}15551[#15551] (issue: {issue}14416[#14416]) * Make remaining ShapeBuilders implement Writeable {pull}15010[#15010] (issue: {issue}14416[#14416]) * Geo: Remove internal `translated` flag from LineStringBuilder {pull}14969[#14969] * Make PointBuilder, CircleBuilder & EnvelopeBuilder implement Writable {pull}14933[#14933] (issue: {issue}14416[#14416]) * Merging BaseLineString and BasePolygonBuilder with subclass {pull}14887[#14887] (issue: {issue}14482[#14482]) * Moving static factory methods to ShapeBuilders {pull}14529[#14529] * Remove InternalLineStringBuilder and InternalPolygonBuilder {pull}14482[#14482] (issue: {issue}14416[#14416]) Highlighting:: * Switch Highlighting to ObjectParser {pull}17363[#17363] * Use HighlightBuilder in SearchSourceBuilder {pull}15376[#15376] (issue: {issue}15044[#15044]) * Joint parsing of common global Hightlighter and subfield parameters {pull}15368[#15368] (issue: {issue}15285[#15285]) * Enable HighlightBuilder to create SearchContextHighlight {pull}15324[#15324] * Add fromXContent method to HighlightBuilder {pull}15157[#15157] Ingest:: * add automatic type conversion support to ConvertProcessor {pull}17263[#17263] (issue: {issue}17139[#17139]) * Give the foreach processor access to the rest of the document {pull}17172[#17172] (issue: {issue}17147[#17147]) * Added ingest statistics to node stats API {pull}16915[#16915] * Add `ingest_took` to bulk response {pull}16876[#16876] * Add ingest info to node info API, which contains a list of available processors {pull}16865[#16865] * Use diffs for ingest metadata in cluster state {pull}16847[#16847] * hide null-valued metadata fields from WriteableIngestDocument#toXContent {pull}16557[#16557] * Ingest: use bulk thread pool for bulk request processing (was index before) {pull}16539[#16539] (issue: {issue}16503[#16503]) * Add foreach processor {pull}16432[#16432] * revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation {pull}16355[#16355] * Add processor tags to on_failure metadata in ingest pipeline {pull}16324[#16324] (issue: {issue}16202[#16202]) * catch processor/pipeline factory exceptions and return structured error responses {pull}16276[#16276] (issue: {issue}16010[#16010]) * Ingest: move get/put/delete pipeline methods to ClusterAdminClient {pull}16242[#16242] * Geoip processor: remove redundant latitude and longitude fields and make location an object with lat and lon subfields {pull}16173[#16173] Internal:: * Remove PROTOTYPE from MLT.Item {pull}17481[#17481] (issue: {issue}17085[#17085]) * Remove PROTOTYPE from VersionType {pull}17480[#17480] (issue: {issue}17085[#17085]) * Remove PROTOTYPEs from highlighting {pull}17466[#17466] (issue: {issue}17085[#17085]) * Remove PROTOTYPEs from ingest {pull}17434[#17434] (issue: {issue}17085[#17085]) * Start to rework query registration {pull}17424[#17424] * Factor out slow logs into Search and IndexingOperationListeners {pull}17398[#17398] * Remove PROTOTYPE from Suggesters {pull}17370[#17370] * Remove PROTOTYPE from SortBuilders {pull}17337[#17337] (issue: {issue}17085[#17085]) * Remove PROTOTYPE from ShapeBuilders {pull}17336[#17336] (issue: {issue}17085[#17085]) * Replace FieldStatsProvider with a method on MappedFieldType. {pull}17334[#17334] * Stop using PROTOTYPE in NamedWriteableRegistry {pull}17284[#17284] (issue: {issue}17085[#17085]) * Support scheduled commands in current context {pull}17077[#17077] * Thread limits {pull}17003[#17003] * Remove leniency from segments info integrity checks {pull}16985[#16985] (issue: {issue}16973[#16973]) * Rename SearchServiceTransportAction to SearchTransportService {pull}16880[#16880] * Decouple the TransportService and ClusterService {pull}16872[#16872] (issue: {issue}16788[#16788]) * Refactor bootstrap checks {pull}16844[#16844] (issues: {issue}16733[#16733], {issue}16835[#16835]) * Add LifecycleRunnable {pull}16752[#16752] * Hot inlined methods in your area {pull}16725[#16725] * Move IndicesQueryCache and IndicesRequestCache into IndicesService {pull}16603[#16603] * Forbid use of java.security.MessageDigest#clone() {pull}16543[#16543] (issue: {issue}16479[#16479]) * Make IndicesWarmer a private class of IndexService {pull}16470[#16470] * Simplify IndicesFieldDataCache and detach from guice {pull}16469[#16469] * Uppercase ells ('L') in long literals {pull}16329[#16329] (issue: {issue}16279[#16279]) * ShardId equality and hash code inconsistency {pull}16319[#16319] (issue: {issue}16217[#16217]) * Ensure all resources are closed on Node#close() {pull}16316[#16316] (issue: {issue}13685[#13685]) * Make index uuid available in Index, ShardRouting & ShardId {pull}16217[#16217] * Move RefreshTask into IndexService and use since task per index {pull}15933[#15933] * Make IndexingMemoryController private to IndicesService {pull}15877[#15877] * Cleanup IndexingOperationListeners infrastructure {pull}15875[#15875] * Remove and forbid use of j.u.c.ThreadLocalRandom {pull}15862[#15862] (issue: {issue}15294[#15294]) * Fix IntelliJ query builder type inference issues {pull}15429[#15429] * Remove and forbid use of Collections#shuffle(List) and Random#() {pull}15299[#15299] (issue: {issue}15287[#15287]) * Remove and forbid use of the type-unsafe empty Collections fields {pull}15187[#15187] * Move IndicesService.canDeleteShardContent to use IndexSettings {pull}15150[#15150] (issue: {issue}15059[#15059]) * Simplify MonitorService construction and detach from guice {pull}15035[#15035] * Use Supplier for StreamInput#readOptionalStreamable {pull}14806[#14806] * Add variable-length long encoding {pull}14780[#14780] * Extend usage of IndexSetting class {pull}14731[#14731] (issue: {issue}14251[#14251]) * Fold SimilarityModule into IndexModule {pull}14284[#14284] * Move to lucene BoostQuery {pull}14264[#14264] * Use built-in method for computing hash code of longs {pull}14213[#14213] * Refactor ShardFailure listener infrastructure {pull}14206[#14206] * Add methods for variable-length encoding integral arrays {pull}14087[#14087] * Fold IndexAliasesService into IndexService {pull}14044[#14044] * Remove unneeded Module abstractions {pull}13944[#13944] * Query refactoring: simplify IndexQueryParserService parse methods {pull}13938[#13938] (issue: {issue}13859[#13859]) * Remove and forbid use of com.google.common.collect.Iterators {pull}13916[#13916] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.ImmutableCollection {pull}13909[#13909] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.io.Resources {pull}13908[#13908] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.hash.* {pull}13907[#13907] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.net.InetAddresses {pull}13905[#13905] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.EvictingQueue {pull}13903[#13903] (issue: {issue}13224[#13224]) * Replace Guava cache with simple concurrent LRU cache {pull}13879[#13879] * Remove ClusterSerivce and IndexSettingsService dependency from IndexShard {pull}13853[#13853] * Start making RecoverySourceHandler unittestable {pull}13840[#13840] * Remove IndexService dep. from IndexShard {pull}13797[#13797] * Remove ES internal deletion policies in favour of Lucenes implementations {pull}13794[#13794] * Move ShardTermVectorService to be on indices level as TermVectorService {pull}13786[#13786] * Move ShardPercolateService creation into IndexShard {pull}13777[#13777] * Remove `ExpressionScriptCompilationException` and `ExpressionScriptExecutionException` {pull}13742[#13742] * Reduced the number of ClusterStateUpdateTask variants {pull}13735[#13735] * Add a BaseParser helper for stream parsing {pull}13615[#13615] * Remove and forbid use of com.google.common.primitives.Ints {pull}13596[#13596] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.math.LongMath {pull}13575[#13575] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.base.Joiner {pull}13572[#13572] (issue: {issue}13224[#13224]) * Replace and ban next batch of Guava classes {pull}13562[#13562] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.Iterables {pull}13559[#13559] (issue: {issue}13224[#13224]) * Replace LoadingCache usage with a simple ConcurrentHashMap {pull}13552[#13552] (issue: {issue}13224[#13224]) * Use Supplier instead of Reflection {pull}13545[#13545] * Remove and forbid use of com.google.common.base.Preconditions {pull}13540[#13540] (issue: {issue}13224[#13224]) * Remove and forbid use of guava Function, Charsets, Collections2 {pull}13533[#13533] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.ImmutableSortedMap {pull}13525[#13525] (issue: {issue}13224[#13224]) * Remove and forbid use of several com.google.common.util. classes {pull}13524[#13524] (issue: {issue}13224[#13224]) * Cleanup SearchRequest & SearchRequestBuilder {pull}13518[#13518] * Remove and forbid use of com.google.common.collect.Queues {pull}13498[#13498] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.base.Preconditions#checkNotNull {pull}13493[#13493] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.Sets {pull}13463[#13463] (issue: {issue}13224[#13224]) * Remove and forbid use of com.google.common.collect.Maps {pull}13438[#13438] (issue: {issue}13224[#13224]) * Remove use of underscore as an identifier {pull}13353[#13353] * Remove and forbid the use of com.google.common.base.Predicate(s)? {pull}13349[#13349] (issues: {issue}13224[#13224], {issue}13314[#13314]) * This commit removes com.google.common.io {pull}13302[#13302] (issue: {issue}13224[#13224]) Java API:: * Remove copy constructors from request classes and TransportMessage type {pull}16640[#16640] (issue: {issue}15776[#15776]) Mapping:: * Add a soft limit on the mapping depth. {pull}17400[#17400] * Disable fielddata on text fields by defaults. {pull}17386[#17386] * Add limit to total number of fields in mapping {pull}17357[#17357] * Make `parseMultiField` part of `parseField`. {pull}17313[#17313] * Automatically add a sub keyword field to string dynamic mappings. {pull}17188[#17188] * Remove friction from the mapping changes in 5.0. {pull}16991[#16991] * Rework norms parameters for 5.0. {pull}16987[#16987] * Moved dynamic field handling in doc parsing to end of parsing {pull}16798[#16798] * Remove the MapperBuilders utility class. {pull}16609[#16609] * Make the `index` property a boolean. {pull}16161[#16161] * Remove the ability to enable doc values with the `fielddata.format` setting. {pull}16147[#16147] * Be stricter about parsing boolean values in mappings. {pull}16146[#16146] * Fix default doc values to be enabled when a field is not indexed. {pull}16141[#16141] * Dynamically map floating-point numbers as floats instead of doubles. {pull}15319[#15319] (issue: {issue}13851[#13851]) * Simplify MetaDataMappingService. {pull}15217[#15217] * Remove MergeMappingException. {pull}15177[#15177] Packaging:: * Added RPM metadata {pull}17477[#17477] * Elasticsearch ownership for data, logs, and configs {pull}17197[#17197] (issue: {issue}12688[#12688]) * Fail early on JDK with compiler bug {pull}16418[#16418] (issues: {issue}16097[#16097], {issue}16362[#16362]) * Make security non-optional {pull}16176[#16176] * Remove RuntimePermission("accessDeclaredMembers") {pull}15378[#15378] * Remove Guava as a dependency {pull}14055[#14055] (issue: {issue}13224[#13224]) * Remove Guava as a dependency {pull}14054[#14054] (issue: {issue}13224[#13224]) Percolator:: * Add scoring support to the percolator query {pull}17385[#17385] (issue: {issue}13827[#13827]) * Add query extract support for the blended term query and the common terms query {pull}17347[#17347] * Add support for several span queries in ExtractQueryTermsService {pull}17323[#17323] * Add support for TermsQuery in ExtractQueryTermsService {pull}17316[#17316] * Replace percolate APIs with a percolator query {pull}16349[#16349] (issues: {issue}10741[#10741], {issue}11264[#11264], {issue}13176[#13176], {issue}13978[#13978], {issue}4317[#4317], {issue}7297[#7297]) Plugin Analysis Kuromoji:: * Add nbest options and NumberFilter {pull}17173[#17173] Plugin Discovery EC2:: * Add support for proxy authentication for s3 and ec2 {pull}15293[#15293] (issue: {issue}15268[#15268]) Plugin Ingest Attachment:: * Minor attachment processor improvements {pull}16574[#16574] Plugin Lang Painless:: * Painless Clean Up {pull}17428[#17428] * Make Painless a Module {pull}16755[#16755] * Minor Clean up {pull}16457[#16457] * Remove Extra String Concat Token {pull}16382[#16382] Plugin Mapper Attachment:: * minor attachments cleanups: IDE test support and EPUB format {pull}14626[#14626] Plugin Repository Azure:: * Support global `repositories.azure.` settings {pull}15141[#15141] (issue: {issue}13776[#13776]) * Add timeout settings (default to 5 minutes) {pull}15080[#15080] (issue: {issue}14277[#14277]) * Remove AbstractLegacyBlobContainer {pull}14650[#14650] (issue: {issue}13434[#13434]) Plugin Repository HDFS:: * merge current hdfs improvements to master {pull}15588[#15588] Plugin Repository S3:: * Check that S3 setting `buffer_size` is always lower than `chunk_size` {pull}17274[#17274] (issue: {issue}17244[#17244]) Plugins:: * PluginManager: Add xpack as official plugin {pull}17227[#17227] * CliTool: Cleanup and document Terminal {pull}16443[#16443] * Plugin cli: Improve maven coordinates detection {pull}16384[#16384] (issue: {issue}16376[#16376]) * Enforce plugin zip does not contain zip entries outside of the plugin dir {pull}16361[#16361] * CliTool: Allow unexpected exceptions to propagate {pull}16359[#16359] * Reduce complexity of plugin cli {pull}16336[#16336] * Remove Plugin.onIndexService. {pull}15029[#15029] (issue: {issue}14896[#14896]) * Open up QueryCache and SearcherWrapper extension points {pull}14303[#14303] Query DSL:: * An `exists` query on an object should query a single term. {pull}17186[#17186] (issue: {issue}17131[#17131]) * Function Score Query: make parsing stricter {pull}16617[#16617] (issue: {issue}16583[#16583]) * Parsers should throw exception on unknown objects {pull}14255[#14255] (issue: {issue}10974[#10974]) * UNICODE_CHARACTER_CLASS fix {pull}11598[#11598] (issue: {issue}10146[#10146]) Query Refactoring:: * Add infrastructure to rewrite query builders {pull}16599[#16599] * Switch geo validation to enum {pull}13672[#13672] (issue: {issue}13608[#13608]) REST:: * More robust handling of CORS HTTP Access Control {pull}16092[#16092] * Add option to exclude based on paths in XContent {pull}16017[#16017] Recovery:: * Recover broken IndexMetaData as closed {pull}17187[#17187] * Relocation source should be marked as relocating before starting recovery to primary relocation target {pull}16500[#16500] * Operation counter for IndexShard {pull}15956[#15956] (issue: {issue}15900[#15900]) * Primary relocation handoff {pull}15900[#15900] (issue: {issue}15532[#15532]) * Remove recovery threadpools and throttle outgoing recoveries on the master {pull}15372[#15372] * Refactor StoreRecoveryService to be a simple package private util class {pull}13766[#13766] Reindex API:: * Make reindex throttling dynamic {pull}17262[#17262] * Throttling support for reindex {pull}17039[#17039] * Add ingest pipeline support to reindex {pull}16932[#16932] Scripting:: * Skipping hidden files compilation for script service {pull}16286[#16286] (issue: {issue}15269[#15269]) * Rename Plan A to Painless {pull}16245[#16245] * Add plumbing for script compile-time parameters {pull}15464[#15464] * Factor mustache -> modules/lang-mustache {pull}15328[#15328] Search:: * Add a soft limit on the number of shards that can be queried in a single search request. {pull}17396[#17396] * Type filters should not have a performance impact when there is a single type. {pull}17350[#17350] * Store _all payloads on 1 byte instead of 4. {pull}16899[#16899] * Refuse to load fields from _source when using the `fields` option and support wildcards. {pull}15017[#15017] (issues: {issue}10783[#10783], {issue}14489[#14489]) * Add response into ClearScrollResponse {pull}13835[#13835] (issue: {issue}13817[#13817]) * Shuffle shards for _only_nodes + support multiple specifications like cluster API {pull}12575[#12575] (issues: {issue}12546[#12546], {issue}12700[#12700]) Search Refactoring:: * Remove RescoreParseElement {pull}17441[#17441] * Remove HighlighterParseElement {pull}17303[#17303] * Move top level parsing of sort element to SortBuilder {pull}17248[#17248] * Switch to using refactored SortBuilder instead of using BytesReference in serialization {pull}17205[#17205] (issues: {issue}17146[#17146], {issue}17257[#17257]) * Add build() method to SortBuilder implementations {pull}17146[#17146] (issue: {issue}10217[#10217]) * Refactoring of Suggestions {pull}17096[#17096] (issue: {issue}10217[#10217]) * Move sort `order` field up into SortBuilder {pull}17035[#17035] * Moves SortParser:parse(...) to only require QueryShardContext {pull}16999[#16999] (issue: {issue}15178[#15178]) * Change internal representation of suggesters {pull}16873[#16873] * Make GeoDistanceSortBuilder serializable, 2nd try {pull}16572[#16572] (issues: {issue}15178[#15178], {issue}16151[#16151]) * Move missing() from SortBuilder interface to class {pull}16225[#16225] (issues: {issue}15178[#15178], {issue}16151[#16151]) * Remove deprecated parameters from ScriptSortBuilder {pull}16153[#16153] (issue: {issue}15178[#15178]) * Refactor GeoSortBuilder {pull}16151[#16151] (issue: {issue}15178[#15178]) * Refactor FieldSortBuilder {pull}16127[#16127] (issue: {issue}15178[#15178]) * Make sort order enum writable. {pull}16124[#16124] (issue: {issue}15178[#15178]) * Make DistanceUnit writable. {pull}16122[#16122] (issue: {issue}15178[#15178]) * RescoreBuilder: Add parsing and creating of RescoreSearchContext {pull}16014[#16014] (issue: {issue}15559[#15559]) * Make RescoreBuilder and nested QueryRescorer Writable {pull}15953[#15953] (issue: {issue}15559[#15559]) * Explain api: move query parsing to the coordinating node {pull}14270[#14270] * Switch query parsers to use ParseField {pull}14249[#14249] (issue: {issue}8964[#8964]) * Refactoring of Aggregations {pull}14136[#14136] Settings:: * Add guard against null-valued settings {pull}17310[#17310] (issue: {issue}17292[#17292]) * Useful error message for null property placeholder {pull}17293[#17293] (issue: {issue}17292[#17292]) * Archive cluster level settings if unknown or broken {pull}17246[#17246] * Improve error message if setting is not found {pull}17230[#17230] * Improve upgrade experience of node level index settings {pull}17223[#17223] (issue: {issue}17187[#17187]) * Settings with complex matchers should not overlap {pull}16754[#16754] * Moves GCE settings to the new infra {pull}16722[#16722] (issue: {issue}16720[#16720]) * Add filtering support within Setting class {pull}16629[#16629] (issue: {issue}16598[#16598]) * Migrate AWS settings to new settings infrastructure {pull}16602[#16602] (issue: {issue}16293[#16293]) * Remove `gateway.initial_meta` and always rely on min master nodes {pull}16446[#16446] * Rewrite SettingsFilter to be immutable {pull}16425[#16425] * Simplify azure settings {pull}16363[#16363] * Convert PageCacheRecycler settings {pull}16341[#16341] * Monitor settings {pull}16313[#16313] * Cut over tribe node settings to new settings infra {pull}16311[#16311] * Convert multcast plugin settings to the new infra {pull}16295[#16295] * Convert `request.headers.*` to the new settings infra {pull}16292[#16292] * Migrate Azure settings to new settings infrastructure {pull}16291[#16291] * Validate logger settings and allow them to be reset via API {pull}16289[#16289] * Switch NodeEnvironment's settings to new settings {pull}16273[#16273] * Simplify AutoCreateIndex and add more tests {pull}16270[#16270] * Convert several pending settings {pull}16269[#16269] * Migrate query caching settings to the new settings infra. {pull}16267[#16267] * Convert `action.auto_create_index` and `action.master.force_local` to the new settings infra {pull}16263[#16263] * Convert `cluster.routing.allocation.type` and `processors` to the new settings infra. {pull}16238[#16238] * Validate tribe node settings on startup {pull}16237[#16237] * Move node.client, node.data, node.master, node.local and node.mode to new settings infra {pull}16230[#16230] * Moved http settings to the new settings infrastructure {pull}16188[#16188] * Migrate network service to the new infra {pull}16187[#16187] * Convert client.transport settings to new infra {pull}16183[#16183] * Move discovery.* settings to new Setting infrastructure {pull}16182[#16182] * Change over to o.e.common.settings.Setting for http settings {pull}16181[#16181] * Convert "path.*" and "pidfile" to new settings infra {pull}16180[#16180] * Migrate repository settings to the new settings API {pull}16178[#16178] * Convert "indices.*" settings to new infra. {pull}16177[#16177] * Migrate gateway settings to the new settings API. {pull}16175[#16175] * Convert several node and test level settings {pull}16172[#16172] * Run Metadata upgrade tool on every version {pull}16168[#16168] * Check for invalid index settings on metadata upgrade {pull}16156[#16156] * Validate the settings key if it's simple chars separated by `.` {pull}16120[#16120] * Validate known global settings on startup {pull}16091[#16091] * Cut over all index scope settings to the new setting infrastrucuture {pull}16054[#16054] (issues: {issue}12790[#12790], {issue}12854[#12854], {issue}16032[#16032], {issue}6732[#6732]) * Remove updatability of `index.flush_on_close` {pull}15964[#15964] (issue: {issue}15955[#15955]) * Move all dynamic settings and their config classes to the index level {pull}15955[#15955] (issue: {issue}6732[#6732]) * Always require units for bytes and time settings {pull}15948[#15948] (issue: {issue}11437[#11437]) * Make MetaData parsing less lenient. {pull}15828[#15828] * Move async translog sync logic into IndexService {pull}15584[#15584] * Remove `index.merge.scheduler.notify_on_failure` and default to `true` {pull}15572[#15572] (issue: {issue}15570[#15570]) * Remove cache concurrency level settings that no longer apply {pull}14210[#14210] (issues: {issue}13224[#13224], {issue}13717[#13717], {issue}7836[#7836]) Similarities:: * Defining a global default similarity {pull}16682[#16682] (issue: {issue}16594[#16594]) Stats:: * Normalize unavailable load average {pull}16061[#16061] (issues: {issue}12049[#12049], {issue}14741[#14741], {issue}15907[#15907], {issue}15932[#15932], {issue}15934[#15934]) * Add load averages to OS stats on FreeBSD {pull}15934[#15934] (issue: {issue}15917[#15917]) * Expose pending cluster state queue size in node stats {pull}14040[#14040] (issue: {issue}13610[#13610]) Store:: * Remove support for legacy checksums {pull}16931[#16931] * Rename index folder to index_uuid {pull}16442[#16442] (issues: {issue}13264[#13264], {issue}13265[#13265], {issue}14512[#14512], {issue}14932[#14932], {issue}15853[#15853]) Task Manager:: * Add ability to group tasks by common parent {pull}17341[#17341] * Add start time and duration to tasks {pull}16829[#16829] * Combine node name and task id into single string task id {pull}16744[#16744] * Add task status {pull}16356[#16356] (issue: {issue}16344[#16344]) * Extend tracking of parent tasks to master node, replication and broadcast actions {pull}15931[#15931] Translog:: * Remove ChannelReference and simplify Views {pull}15898[#15898] * Simplify TranslogWriter to always write to a stream {pull}15771[#15771] * Remove TranslogService and fold it into synchronous IndexShard API {pull}13707[#13707] [[bug-5.0.0-alpha1]] [float] === Bug fixes Aggregations:: * Fixes the defaults for `keyed` in the percentiles aggregations {pull}17217[#17217] * Correct typo in class name of StatsAggregator {pull}15264[#15264] (issue: {issue}14730[#14730]) Allocation:: * Replica shards must be failed before primary shards {pull}15686[#15686] CRUD:: * Prevent TransportReplicationAction to route request based on stale local routing table {pull}16274[#16274] (issues: {issue}12573[#12573], {issue}12574[#12574]) * Resolves the conflict between alias routing and parent routing by applying the alias routing and ignoring the parent routing. {pull}15371[#15371] (issue: {issue}3068[#3068]) Cluster:: * Shard state action channel exceptions {pull}16057[#16057] (issue: {issue}15748[#15748]) Core:: * Handle RejectedExecution gracefully in TransportService during shutdown {pull}16965[#16965] Geo:: * Fix multi-field support for GeoPoint types {pull}15702[#15702] (issue: {issue}15701[#15701]) * Enforce distance in distance query is > 0 [ISSUE] {pull}15135[#15135] Ingest:: * Handle regex parsing errors in Gsub and Grok Processors {pull}17260[#17260] * add on_failure exception metadata to ingest document for verbose simulate {pull}16562[#16562] * The IngestDocument copy constructor should make a deep copy {pull}16248[#16248] (issue: {issue}16246[#16246]) Internal:: * Enable unmap hack for java 9 {pull}16986[#16986] (issue: {issue}1[#1]) * Fix issues with failed cache loads {pull}14315[#14315] * Allow parser to move on the START_OBJECT token when parsing search source {pull}14145[#14145] * Ensure searcher is release if wrapping fails {pull}14107[#14107] * Avoid deadlocks in Cache#computeIfAbsent {pull}14091[#14091] (issue: {issue}14090[#14090]) Java API:: * Fix potential NPE in SearchSourceBuilder {pull}16905[#16905] (issue: {issue}16902[#16902]) Mapping:: * Make dynamic template parsing less lenient. {pull}17249[#17249] * Fix dynamic mapper when its parent already has an update {pull}17065[#17065] * Fix copy_to when the target is a dynamic object field. {pull}15216[#15216] (issues: {issue}111237[#111237], {issue}11237[#11237]) * Preserve existing mappings on batch mapping updates {pull}15130[#15130] (issues: {issue}14899[#14899], {issue}15129[#15129]) Packaging:: * Set MAX_OPEN_FILES to 65536 {pull}17431[#17431] (issue: {issue}17430[#17430]) * [windows] Service command still had positional start command {pull}17391[#17391] * Do not pass double-dash arguments on startup {pull}17087[#17087] (issue: {issue}17084[#17084]) Percolator:: * Let PercolatorQuery's explain use the two phase iterator {pull}17315[#17315] (issue: {issue}17314[#17314]) Plugin Store SMB:: * Fix calling ensureOpen() on the wrong directory (master forwardport) {pull}16395[#16395] (issue: {issue}16383[#16383]) Plugins:: * CliTool: Messages printed in Terminal should have percent char escaped {pull}16367[#16367] Query DSL:: * Resolve string dates and date math to millis before evaluating for rewrite in range query {pull}17239[#17239] * `constant_score` query should throw error on more than one filter {pull}17135[#17135] (issue: {issue}17126[#17126]) * Single IPv4 addresses in IP field term queries {pull}16068[#16068] (issue: {issue}16058[#16058]) * Make strategy optional in GeoShapeQueryBuilder readFrom and writeTo {pull}13963[#13963] Query Refactoring:: * Query refactoring: set has_parent & has_child types context properly {pull}13863[#13863] * Make sure equivalent geohashCellQueries are equal after toQuery called {pull}13792[#13792] Recovery:: * Invoke `IndexingOperationListeners` also when recovering from store or remote {pull}17406[#17406] * Prevent interruption while store checks lucene files for consistency {pull}16308[#16308] * Mark shard as recovering on the cluster state thread {pull}14276[#14276] (issues: {issue}13766[#13766], {issue}14115[#14115]) Search:: * Fix for search after {pull}16271[#16271] * Do not be lenient when parsing CIDRs {pull}14874[#14874] (issue: {issue}14862[#14862]) Settings:: * Register bootstrap settings {pull}16513[#16513] * Add settings filtering to node info requests {pull}16445[#16445] * Ban write access to system properties {pull}14914[#14914] Task Manager:: * Take filterNodeIds into consideration while sending task requests to nodes {pull}17081[#17081] Translog:: * Move translog recover outside of the engine {pull}17422[#17422] * Mark shard active during recovery; push settings after engine finally inits {pull}16250[#16250] (issues: {issue}14121[#14121], {issue}16209[#16209]) [[upgrade-5.0.0-alpha1]] [float] === Upgrades Core:: * Upgrade to lucene-6.0.0-f0aa4fc. {pull}17075[#17075] * upgrade to lucene 6.0.0-snapshot-bea235f {pull}16964[#16964] * Upgrade to Jackson 2.7.1 {pull}16801[#16801] (issue: {issue}16294[#16294]) Ingest:: * Update MaxMind geoip2 version to 2.6 {pull}16837[#16837] (issue: {issue}16801[#16801]) Internal:: * Bump master (3.0-snapshot) to java 8 {pull}13314[#13314] Search Templates:: * Update mustache.java to version 0.9.1 {pull}14053[#14053] (issue: {issue}13224[#13224])