The DiscoveryNodes.Delta was changed in #28197. Previous/Master nodes
are now always set in the `Delta` (before the change they were set only
if the master changed) and the `masterChanged()` method is now based on
object equality and nodes ephemeral ids (before the change it was based
on nodes id).
This commit adapts the DiscoveryNodesTests.testDeltas() to reflect the
changes.
We introduced a single commit assertion when opening an index but create
a new translog. However, this assertion is not held in this situation.
1. A replica with two commits c1 and c2 starts peer-recovery with c1
2. The recovery is sequence-based recovery but the primary is before 6.2 so
it sent true for “createNewTranslog”
3. Replica opens engine and create translog. We expect "open index and
create translog" have 1 commit but we have c1 and c2.
This commit makes sure to assert this iff the index was created on 6.2+.
This introduces a settings updater that allows to specify a list of
settings. Whenever one of those settings changes, the whole block of
settings is passed to the consumer.
This also fixes an issue with affix settings, when used in combination
with group settings, which could result in no found settings when used
to get a setting for a namespace.
Lastly logging has been slightly changed, so that filtered settings now
only log the setting key.
Another bug has been fixed for the mock log appender, which did not
work, when checking for the exact message.
Closes#28047
* es/master:
Remove redundant argument for buildConfiguration of s3 plugin (#28281)
Completely remove Painless Type from AnalyzerCaster in favor of Java Class. (#28329)
Fix spelling error
Reindex: Wait for deletion in test
Reindex: log more on rare test failure
Ensure we protect Collections obtained from scripts from self-referencing (#28335)
[Docs] Fix asciidoc style in composite agg docs
Adds the ability to specify a format on composite date_histogram source (#28310)
Provide a better error message for the case when all shards failed (#28333)
[Test] Re-Add integer_range and date_range field types for query builder tests (#28171)
Added Put Mapping API to high-level Rest client (#27869)
Revert change that does not return all indices if a specific alias is requested via get alias api. (#28294)
Painless: Replace Painless Type with Java Class during Casts (#27847)
Notify affixMap settings when any under the registered prefix matches (#28317)
Self referencing maps can cause SOE if they are iterated ie. in their toString methods. This chance adds some protected to the usage of those collections.
This commit adds the ability to specify a date format on the `date_histogram` composite source.
If the format is defined, the key for the source is returned as a formatted date.
Closes#27923
The tests for those field types were removed in #26549 because the range mapper
was moved to a module, but later this mapper was moved back to core in #27854.
This change adds back those two field types like before to the general setup in
AbstractQueryTestCase and adds some specifics to the RangeQueryBuilder and
TermsQueryBuilder tests. Also adding back an integration test in SearchQueryIT that
has been removed before but that can be kept with the mapper back in core now.
Relates to #28147
* Notify affixMap settings when any under the registered prefix matches
Previously if an affixMap setting was registered, and then a completely
different setting was applied, the affixMap update consumer would be notified
with an empty map. This caused settings that were previously set to be unset in
local state in a consumer that assumed it would only be called when the affixMap
setting was changed.
This commit changes the behavior so if a prefix `foo.` is registered, any
setting under the prefix will have the update consumer notified if there are
changes starting with `foo.`.
Resolves#28316
* Add unit test
* Address feedback
* master:
Trim down usages of `ShardOperationFailedException` interface (#28312)
Do not return all indices if a specific alias is requested via get aliases api.
[Test] Lower bwc version for rank-eval rest tests
CountedBitSet doesn't need to extend BitSet. (#28239)
Calculate sum in Kahan summation algorithm in aggregations (#27807) (#27848)
Remove the `update_all_types` option. (#28288)
Add information when master node left to DiscoveryNodes' shortSummary() (#28197)
Provide explanation of dangling indices, fixes#26008 (#26999)
In many cases we use the `ShardOperationFailedException` interface to abstract an exception that can only be of one type, namely `DefaultShardOperationException`. There is no need to use the interface in such cases, the concrete type should be used instead. That has the additional advantage of simplifying parsing such exceptions back from rest responses for the high-level REST client
If a get alias api call requests a specific alias pattern then
indices not having any matching aliases should not be included in the response.
Closes#27763
* es/master: (38 commits)
Build: Add pom generation to meta plugins (#28321)
Add 6.3 version constant to master
Minor improvements to translog docs (#28237)
[Docs] Remove typo in painless-getting-started.asciidoc
Build: Fix meta plugin usage in integ test clusters (#28307)
Painless: Add spi jar that will be published for extending whitelists (#28302)
mistyping in one of the highlighting examples comment -> content (#28139)
Documents applicability of term query to range type (#28166)
Build: Omit dependency licenses check for elasticsearch deps (#28304)
Clean up commits when global checkpoint advanced (#28140)
Implement socket and server ChannelContexts (#28275)
Plugins: Fix meta plugins to install bundled plugins with their real name (#28285)
Build: Fix meta plugin integ test installation (#28286)
Modify Abstract transport tests to use impls (#28270)
Fork Groovy compiler onto compile Java home
[Docs] Update tophits-aggregation.asciidoc (#28273)
Docs: match between snippet to its description (#28296)
[TEST] fix RequestTests#testSearch in case search source is not set
REST high-level client: remove index suffix from indices client method names (#28263)
Fix simple_query_string on invalid input (#28219)
...
Today we keep multiple index commits based on the current global
checkpoint, but only clean up unneeded index commits when we have a new
index commit. However, we can release the old index commits earlier once
the global checkpoint has advanced enough. This commit makes an engine
revisit the index deletion policy whenever a new global checkpoint value
is persisted and advanced enough.
Relates #10708
This change converts any exception that occurs during the parsing of
a simple_query_string to a match_no_docs query (instead of a null query)
when leniency is activated.
Closes#28204
This commit adds an extension point for client actions to action
plugins. This is useful for plugins to expose the client-side actions
without exposing the server-side implementations to the client. The
default implementation, of course, delegates to extracting the
client-side action from the server-side implementation.
Relates #28280
MixedClusterClientYamlTestSuiteIT sometimes fails when executing the
indices.stats/13_fields/* REST tests. It does not reproduce locally
but the execution logs show that it failed when a shard is relocating
during the set up execution. This commit change the set up so that it
now waits for all shards to be active before executing the tests.
closes#26732, #27146
* master:
Fix third-party audit tasks on JDK 8
Remove duplicated javadoc `fieldType` param
Handle 5.6.6 and 6.1.2 release
Introduce multi-release JAR
Move the multi-get response tests to server
Require JDK 9 for compilation (#28071)
Revert "[Docs] Fix Java Api index administration usage (#28133)"
Revert "[Docs] Fix base directory to include for put_mapping.asciidoc"
Added multi get api to the high level rest client.
[Docs] Clarify numeric datatype ranges (#28240)
[Docs] Fix base directory to include for put_mapping.asciidoc
Open engine should keep only starting commit (#28228)
This commit introduces the ability for the core Elasticsearch JAR to be
a multi-release JAR containing code that is compiled for JDK 8 and code
that is compiled for JDK 9. At runtime, a JDK 8 JVM will ignore the JDK
9 compiled classfiles, and a JDK 9 JVM will use the JDK 9 compiled
classfiles instead of the JDK 8 compiled classfiles. With this work, we
utilize the new JDK 9 API for obtaining the PID of the running JVM,
instead of relying on a hack.
For now, we want to keep IDEs on JDK 8 so when the build is in an IDE we
ignore the JDK 9 source set (as otherwise the IDE would give compilation
errors). However, with this change, running Gradle from the command-line
now requires JAVA_HOME and JAVA_9_HOME to be set. This will require
follow-up work in our CI infrastructure and our release builds to
accommodate this change.
Relates #28051
Keeping unsafe commits when opening an engine can be problematic because
these commits are not safe at the recovering time but they can suddenly
become safe in the future. The following issues can happen if unsafe
commits are kept oninit.
1. Replica can use unsafe commit in peer-recovery. This happens when a
replica with a safe commit c1 (max_seqno=1) and an unsafe commit c2
(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new
document (seqno=2) is added without flushing, the global checkpoint is
advanced to 2; and the replica recovers again, it will use the unsafe
commit c2 (max_seqno=2 <= gcp=2) as the starting commit for sequenced
based recovery even the commit c2 contains a stale operation and the
document (with seqno=2) will not be replicated to the replica.
2. Min translog gen for recovery can go backwards in peer-recovery. This
happens when a replica with a safe commit c1 (local_checkpoint=1,
recovery_translog_gen=1) and an unsafe commit c2 (local_checkpoint=2,
recovery_translog_gen=2). The replica recovers from a primary, and keeps
c2 as the last commit, then sets last_translog_gen to 2. Flushing a new
commit on the replica will cause exception as the new last commit c3
will have recovery_translog_gen=1. The recovery translog generation of a
commit is calculated based on the current local checkpoint. The local
checkpoint of c3 is 1 while the local checkpoint of c2 is 2.
3. Commit without translog can be used for recovery. An old index, which
was created before multiple-commits is introduced (v6.2), may not have a
safe commit. If that index has a snapshotted commit without translog and
an unsafe commit, the policy can consider the snapshotted commit as a
safe commit for recovery even the commit does not have translog.
These issues can be avoided if the combined deletion policy keeps only
the starting commit onInit.
Relates #27804
Relates #28181
* es/master: (30 commits)
[Docs] Fix Java Api index administration usage (#28133)
Fix eclipse build. (#28236)
Never return null from Strings.tokenizeToStringArray (#28224)
Fallback to TransportMasterNodeAction for cluster health retries (#28195)
[Docs] Changes to ingest.asciidoc (#28212)
TEST: Update logging for testAckedIndexing
[GEO] Add WKT Support to GeoBoundingBoxQueryBuilder
Painless: Add whitelist extensions (#28161)
Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter (#28225)
Avoid doing redundant work when checking for self references. (#26927)
Fix casts in HotThreads. (#27578)
Ignore the `-snapshot` suffix when comparing the Lucene version in the build and the docs. (#27927)
Allow update of `eager_global_ordinals` on `_parent`. (#28014)
Fix NPE on composite aggregation with sub-aggregations that need scores (#28129)
`MockTcpTransport` to connect asynchronously (#28203)
Fix synonym phrase query expansion for cross_fields parsing (#28045)
Introduce elasticsearch-core jar (#28191)
#28218: Update the Lucene version for 6.2.0 after backport
upgrade to lucene 7.2.1 (#28218)
[Docs] Fix an error in painless-types.asciidoc (#28221)
...
The Java API documentation for index administration currenty is wrong because
the PutMappingRequestBuilder#setSource(Object... source) and
CreateIndexRequestBuilder#addMapping(String type, Object... source) methods
delegate to methods that check that the input arguments are valid key/value
pairs:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html
This changes the docs so the java api code examples are included from
documentation integration tests so we detect compile and runtime issues earlier.
Closes#28131
This method has a different contract than all the other methods in this class, returning null instead of an empty array when receiving a null input. While switching over some methods from delimitedListToStringArray to this method tokenizeToStringArray, this resulted in unexpected nulls in some places of our code.
Relates #28213
Currently we test all maps, arrays or iterables. However, in the case that maps
contain sub maps for instance, we will test the sub maps again even though the
work has already been done for the top-level map.
Relates #26907
A bug introduced in #24407 currently prevents `eager_global_ordinals` from
being updated. This new approach should fix the issue while still allowing
mapping updates to not specify the `_parent` field if it doesn't need
updating, which was the goal of #24407.
The composite aggregation defers the collection of sub-aggregations to a second pass that visits documents only if they
appear in the top buckets. Though the scorer for sub-aggregations is not set on this second pass and generates an NPE if any sub-aggregation
tries to access the score. This change creates a scorer for the second pass and makes sure that sub-aggs can use it safely to check the score of
the collected documents.
The method `initiateChannel` on `TcpTransport` is explicit in that
channels can be connect asynchronously. All production implementations
do connect asynchronously. Only the blocking `MockTcpTransport`
connects in a synchronous manner. This avoids testing some of the
blocking code in `TcpTransport` that waits on connections to complete.
Additionally, it requires a more extensive method signature than
required for other transports.
This commit modifies the `MockTcpTransport` to make these connections
asynchronously on a different thread. Additionally, it simplifies that
`initiateChannel` method signature.
* Fix synonym phrase query expansion for cross_fields parsing
The `cross_fields` mode for query parser ignores phrase query generated by multi-word synonyms.
In such case only the first field of each analyzer group is kept. This change fixes this issue
by expanding the phrase query for each analyzer group to **all** fields using a disjunction max query.
