Commit Graph

2934 Commits

Author SHA1 Message Date
David Kyle 256588d773 Mute IndexStatsIT#testFilterCacheStats
See https://github.com/elastic/elasticsearch/issues/32506
2019-05-09 13:49:47 +01:00
Jim Ferenczi b7c7ca8f09 Fix IAE on cross_fields query introduced in 7.0.1 (#41938)
If the max doc in the index is greater than the minimum total term frequency
among the requested fields we need to adjust max doc to be equal to the min ttf.
This was removed by mistake when fixing #41125.

Closes #41934
2019-05-09 14:25:46 +02:00
Alan Woodward 309e4a11b5 Cut AnalyzeResponse over to Writeable (#41915)
This commit makes AnalyzeResponse and its various helper classes implement
Writeable. The classes are also now immutable.

Relates to #34389
2019-05-09 13:09:23 +01:00
Jim Ferenczi a329aaec90 Fix assertion error when caching the result of a search in a read-only index (#41900)
The ReadOnlyEngine wraps its reader with a SoftDeletesDirectoryReaderWrapper if soft deletes
are enabled. However the wrapping is done on top of the ElasticsearchDirectoryReader and that
trips assertion later on since the cache key of these directories are different. This commit
changes the order of the wrapping to put the ElasticsearchDirectoryReader first in order to
ensure that it is always retrieved first when we unwrap the directory.

Closes #41795
2019-05-09 08:59:52 +02:00
Benjamin Trent edd6438e34
mute test related to #41967 (#41968) 2019-05-08 15:03:28 -05:00
William Brafford a2b7871f9f
Allow unknown task time in QueueResizingEsTPE (#41957)
* Allow unknown task time in QueueResizingEsTPE

The afterExecute method previously asserted that a TimedRunnable task
must have a positive execution time. However, the code in TimedRunnable
returns a value of -1 when a task time is unknown. Here, we expand the
logic in the assertion to allow for that possibility, and we don't
update our task time average if the value is negative.

* Add a failure flag to TimedRunnable

In order to be sure that a task has an execution time of -1 because of
a failure, I'm adding a failure flag boolean to the TimedRunnable class.
If execution time is negative for some other reason, an assertion will
fail.

Backport of #41810
Fixes #41448
2019-05-08 14:15:22 -04:00
David Roberts 452ee55cdb Make ISO8601 date parser accept timezone when time does not have seconds (#41896)
Prior to this change the ISO8601 date parser would only
parse an optional timezone if seconds were specified.
This change moves the timezone to the same level of
optional components as hour, so that timestamps without
minutes or seconds may optionally contain a timezone.
It also adds a unit test to cover all the supported
formats.
2019-05-08 13:50:53 +01:00
Yannick Welsch 957046dad0 Allow IDEA test runner to control number of test iterations (#41653)
Allows configuring the number of test iterations via IntelliJ's config dialog, instead of having to add it
manually via the tests.iters system property.
2019-05-08 13:57:29 +02:00
Armin Braun 5c824f3993
Reenable testCloseOrDeleteIndexDuringSnapshot (#41892)
* Relates #39828
2019-05-08 13:10:19 +02:00
Jim Ferenczi ca3d881716 Always set terminated_early if terminate_after is set in the search request (#40839)
* terminated_early should always be set in the response with terminate_after

Today we set `terminated_early` to true in the response if the query terminated
early due to `terminate_after`. However if `terminate_after` is smaller than
the number of documents in a shard we don't set the flag in the response indicating
that the query was exhaustive. This change fixes this disprepancy by setting
terminated_early to false in the response if the number of documents that match
the query is smaller than the provided `terminate_after` value.

Closes #33949
2019-05-08 12:26:38 +02:00
David Turner 4c909e93bb
Reject port ranges in `discovery.seed_hosts` (#41905)
Today Elasticsearch accepts, but silently ignores, port ranges in the
`discovery.seed_hosts` setting:

```
discovery.seed_hosts: 10.1.2.3:9300-9400
```

Silently ignoring part of a setting like this is trappy. With this change we
reject seed host addresses of this form.

Closes #40786
Backport of #41404
2019-05-08 08:34:32 +01:00
David Turner 935f70c05e Handle serialization exceptions during publication (#41781)
Today if an exception is thrown when serializing a cluster state during
publication then the master enters a poisoned state where it cannot publish any
more cluster states, but nor does it stand down as master, yielding repeated
exceptions of the following form:

```
failed to commit cluster state version [12345]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publishing failed
        at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1045) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.0.0.jar:7.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: cannot start publishing next value before accepting previous one
        at org.elasticsearch.cluster.coordination.CoordinationState.handleClientValue(CoordinationState.java:280) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1030) ~[elasticsearch-7.0.0.jar:7.0.0]
        ... 11 more
```

This is because it already created the publication request using
`CoordinationState#handleClientValue()` but then it fails before accepting it.
This commit addresses this by performing the serialization before calling
`handleClientValue()`.

Relates #41090, which was the source of such a serialization exception.
2019-05-07 17:53:12 +01:00
Alan Woodward 4cca1e8fff Correct spelling of MockLogAppender.PatternSeenEventExpectation (#41893)
The class was called PatternSeenEventExcpectation. This commit
is a straight class rename to correct the spelling.
2019-05-07 17:28:51 +01:00
Ryan Ernst e9e4bae683 Fix fractional seconds for strict_date_optional_time (#41871)
The fractional seconds portion of strict_date_optional_time was
accidentally copied from the printer, which always prints at least 3
fractional digits. This commit fixes the formatter to allow 1 or 2
fractional seconds.

closes #41633
2019-05-07 09:09:30 -07:00
Henning Andersen f068a22f5f SeqNo CAS linearizability (#38561)
Add a test that stresses concurrent writes using ifSeqno/ifPrimaryTerm to do CAS style updates. Use linearizability checker to verify linearizability. Linearizability of successful CAS'es is guaranteed.

Changed linearizability checker to allow collecting history concurrently.

Changed unresponsive network simulation to wake up immediately when network disruption is cleared to ensure tests proceed in a timely manner (and this also seems more likely to provoke issues).
2019-05-07 14:04:38 +02:00
Jim Ferenczi 70bf432fa8 Fix full text queries test that start with now (#41854)
Full text queries that start with now are not cacheable if they target a date field.
However we assume in the query builder tests that all queries are cacheable and this assumption
fails when the random generated query string starts with "now". This fails twice in several years
since the probability that a random string starts with "now" is low but this commit ensures that
 isCacheable is correctly checked for full text queries that fall into this edge case.

 Closes #41847
2019-05-06 19:08:30 +02:00
Przemyslaw Gomulka 79b7ce8697
Fix javadoc in WrapperQueryBuilder backport(41641) #41849
missing brackets in javadoc
backports #41641
2019-05-06 17:55:11 +02:00
Henning Andersen 227d5e15fb ReadOnlyEngine assertion fix (#41842)
Fixed the assertion that maxSeqNo == globalCheckpoint to actually check
against the global checkpoint.
2019-05-06 16:11:38 +02:00
Hicham Mallah 4a88da70c5 Add index name to cluster block exception (#41489)
Updates the error message to reveal the index name that is causing it.

Closes #40870
2019-05-04 19:11:59 -04:00
Nhat Nguyen c7924014fa
Verify consistency of version and source in disruption tests (#41614) (#41661)
With this change, we will verify the consistency of version and source
(besides id, seq_no, and term) of live documents between shard copies
at the end of disruption tests.
2019-05-03 18:47:14 -04:00
Nhat Nguyen e61469aae6 Noop peer recoveries on closed index (#41400)
If users close an index to change some non-dynamic index settings, then the current implementation forces replicas of that closed index to copy over segment files from the primary. With this change, we make peer recoveries of closed index skip both phases.

Relates #33888

Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2019-05-03 12:07:37 -04:00
Issam EL-ATIF 23706d4cdf Update error message for allowed characters in aggregation names (#41573)
Exception message thrown when specifying illegal characters did
no accurately described the allowed characters.  This updates the 
error message to reflect reality (any character except [, ] and >)
2019-05-03 11:55:09 -04:00
Jason Tedor 03c959f188
Upgrade keystore on package install (#41755)
When Elasticsearch is run from a package installation, the running
process does not have permissions to write to the keystore. This is
because of the root:root ownership of /etc/elasticsearch. This is why we
create the keystore if it does not exist during package installation. If
the keystore needs to be upgraded, that is currently done by the running
Elasticsearch process. Yet, as just mentioned, the Elasticsearch process
would not have permissions to do that during runtime. Instead, this
needs to be done during package upgrade. This commit adds an upgrade
command to the keystore CLI for this purpose, and that is invoked during
package upgrade if the keystore already exists. This ensures that we are
always on the latest keystore format before the Elasticsearch process is
invoked, and therefore no upgrade would be needed then. While this bug
has always existed, we have not heard of reports of it in practice. Yet,
this bug becomes a lot more likely with a recent change to the format of
the keystore to remove the distinction between file and string entries.
2019-05-03 10:34:30 -04:00
David Turner 873d0020a5 Reject null customs at build time (#41782)
Today you can add a null `Custom` to the cluster state or its metadata, but
attempting to publish such a cluster state will fail. Unfortunately, the
publication-time failure gives very little information about the source of the
problem. This change causes the failure to manifest earlier and adds
information about which `Custom` was null in order to simplify the
investigation.

Relates #41090.
2019-05-03 14:52:32 +02:00
Jack Conradson 025619bbf1 Improve error message for ln/log with negative results in function score
This changes the error message for a negative result in a function score when 
using the ln modifier to suggest using ln1p or ln2p when a negative result 
occurs in a function score and for the log modifier to suggest using log1p or 
log2p.

This relates to #41509
2019-05-02 16:31:25 -07:00
Jason Tedor d0f071236a
Simplify filtering addresses on interfaces (#41758)
This commit is a refactoring of how we filter addresses on
interfaces. In particular, we refactor all of these methods into a
common private method. We also change the order of logic to first check
if an address matches our filter and then check if the interface is
up. This is to possibly avoid problems we are seeing where devices are
flapping up and down while we are checking for loopback addresses. We do
not expect the loopback device to flap up and down so by reversing the
logic here we avoid that problem on CI machines. Finally, we expand the
error message when this does occur so that we know which device is
flapping.
2019-05-02 16:36:27 -04:00
Colin Goodheart-Smithe ab9154005b
Adds version 6.7.3 2019-05-02 17:36:23 +01:00
Tim Brooks b4bcbf9f64
Support http read timeouts for transport-nio (#41466)
This is related to #27260. Currently there is a setting
http.read_timeout that allows users to define a read timeout for the
http transport. This commit implements support for this functionality
with the transport-nio plugin. The behavior here is that a repeating
task will be scheduled for the interval defined. If there have been
no requests received since the last run and there are no inflight
requests, the channel will be closed.
2019-05-02 09:48:52 -06:00
David Turner b189596631 Add details to BulkShardRequest#getDescription() (#41711)
Today a bulk shard request appears as follows in the detailed task list:

    requests[42], index[my_index]

This change adds the shard index and refresh policy too:

    requests[42], index[my_index][2], refresh[IMMEDIATE]
2019-05-02 08:29:25 +02:00
Andy Bristol b9e44288d3 mute NodeTests#testCloseOnInterruptibleTask
For #41448
2019-05-01 13:24:22 -07:00
Jason Tedor 39b0b5809d
Fix minimum compatible version after 6.8
This commit fixes the minimum compatible version after the introduction
of 6.8.
2019-05-01 16:21:13 -04:00
Jay Modi 7f7eb7b679 Add version 7.0.2 to 7.x branch (#41715) 2019-05-01 15:23:53 -04:00
Jason Tedor f08ac103ee
Add 6.8 version constant
This commit adds the 6.8 version constant to the 7.x branch.
2019-05-01 13:38:58 -04:00
Jason Tedor 7f3ab4524f
Bump 7.x branch to version 7.2.0
This commit adds the 7.2.0 version constant to the 7.x branch, and bumps
BWC logic accordingly.
2019-05-01 13:38:57 -04:00
Henning Andersen c6abe74dd6
Close and acquire commit during reset engine fix (#41584) (#41709)
If closing a shard while resetting engine,
IndexEventListener.afterIndexShardClosed would be called while there is
still an active IndexWriter on the shard. For integration tests, this
leads to an exception during check index called from MockFSIndexStore
.Listener. Fixed.

Relates to #38561
2019-05-01 15:22:24 +02:00
Jason Tedor 26c72c96bd
Fix imports in KeyStoreWrapperTests
This commit addresses a checkstyle violation in KeyStoreWrapperTests,
removing a leftover import.
2019-05-01 07:21:23 -04:00
Jason Tedor 0b46a62f6b
Drop distinction in entries for keystore (#41701)
Today we allow adding entries from a file or from a string, yet we
internally maintain this distinction such that if you try to add a value
from a file for a setting that expects a string or add a value from a
string for a setting that expects a file, you will have a bad time. This
causes a pain for operators such that for each setting they need to know
this difference. Yet, we do not need to maintain this distinction
internally as they are bytes after all. This commit removes that
distinction and includes logic to upgrade legacy keystores.
2019-05-01 07:02:04 -04:00
Nhat Nguyen 887f3f2c83 Simplify initialization of max_seq_no of updates (#41161)
Today we choose to initialize max_seq_no_of_updates on primaries only so
we can deal with a situation where a primary is on an old node (before
6.5) which does not have MUS while replicas on new nodes (6.5+).
However, this strategy is quite complex and can lead to bugs (for
example #40249) since we have to assign a correct value (not too low) to
MSU in all possible situations (before recovering from translog,
restoring history on promotion, and handing off relocation).

Fortunately, we don't have to deal with this BWC in 7.0+ since all nodes
in the cluster should have MSU. This change simplifies the
initialization of MSU by always assigning it a correct value in the
constructor of Engine regardless of whether it's a replica or primary.

Relates #33842
2019-04-30 15:14:52 -04:00
Igor Motov 10ab838106
Geo: Add GeoJson parser to libs/geo classes (#41575) (#41657)
Adds GeoJson parser for Geometry classes defined in libs/geo.

Relates #40908 and #29872
2019-04-29 19:43:31 -04:00
Alan Woodward a01f451ef7 Limit complexity of IntervalQueryBuilderTests#testRandomSource() (#41538)
IntervalsSources can throw IllegalArgumentExceptions if they would produce
too many disjunctions.  To mitigate against this when building random
sources, we limit the depth of the randomly generated source to four
nested sources

Fixes #41402
2019-04-29 13:31:19 +01:00
Dan Hermann b23709b178
Applies the same naming restrictions to repositories as to snapshots except that leading underscores and uppercase characters are permitted. (#41585)
Fixes #40817.
2019-04-29 07:31:01 -05:00
Armin Braun 6e51b6f96d
Add Repository Consistency Assertion to SnapshotResiliencyTests (#41631)
* Add Repository Consistency Assertion to SnapshotResiliencyTests (#40857)

* Add Repository Consistency Assertion to SnapshotResiliencyTests

* Add some quick validation on not leaving behind any dangling metadata or dangling indices to the snapshot resiliency tests
   * Added todo about expanding this assertion further

* Fix SnapshotResiliencyTest Repo Consistency Check (#41332)

* Fix SnapshotResiliencyTest Repo Consistency Check

* Due to the random creation of an empty `extra0` file by the Lucene mockFS we see broken tests because we use the existence of an index folder in assertions and the index deletion doesn't go through if there are extra files in an index folder
  * Fixed by removing the `extra0` file and resulting empty directory trees before asserting repo consistency
* Closes #41326

* Reenable SnapshotResiliency Test (#41437)

This was fixed in https://github.com/elastic/elasticsearch/pull/41332
but I forgot to reenable the test.

* fix compile on java8
2019-04-29 12:01:58 +02:00
Nhat Nguyen 615a0211f0 Recovery should not indefinitely retry on mapping error (#41099)
A stuck peer recovery in #40913 reveals that we indefinitely retry on
new cluster states if indexing translog operations hits a mapper
exception. We should not wait and retry if the mapping on the target is
as recent as the mapping that the primary used to index the replaying
operations.

Relates #40913
2019-04-27 10:55:08 -04:00
Michael Morello 75283294f5 Fix multi-node parsing in voting config exclusions REST API (#41588)
Fixes an issue where multiple nodes where not properly parsed in the voting config exclusions REST API.

Closes #41587
2019-04-27 12:20:03 +02:00
Nick Knize 113b24be4b Refactor GeoHashUtils (#40869)
This commit refactors GeoHashUtils class into a new Geohash utility class located in the ES geo library. The intent is to not only better control what geo methods are whitelisted for painless scripting but to clean up the geo utility API in general.
2019-04-26 10:06:36 -05:00
Armin Braun aad33121d8
Async Snapshot Repository Deletes (#40144) (#41571)
Motivated by slow snapshot deletes reported in e.g. #39656 and the fact that these likely are a contributing factor to repositories accumulating stale files over time when deletes fail to finish in time and are interrupted before they can complete.

* Makes snapshot deletion async and parallelizes some steps of the delete process that can be safely run concurrently via the snapshot thread poll
   * I did not take the biggest potential speedup step here and parallelize the shard file deletion because that's probably better handled by moving to bulk deletes where possible (and can still be parallelized via the snapshot pool where it isn't). Also, I wanted to keep the size of the PR manageable.
* See https://github.com/elastic/elasticsearch/pull/39656#issuecomment-470492106
* Also, as a side effect this gives the `SnapshotResiliencyTests` a little more coverage for master failover scenarios (since parallel access to a blob store repository during deletes is now possible since a delete isn't a single task anymore).
* By adding a `ThreadPool` reference to the repository this also lays the groundwork to parallelizing shard snapshot uploads to improve the situation reported in #39657
2019-04-26 15:36:09 +02:00
Armin Braun 7824f60a34
Simplify Snapshot Resiliency Test (#40930) (#41565)
* Thanks to #39793 dynamic mapping updates don't contain blocking operations anymore so we don't have to manually put the mapping in this test and can keep it a little simpler
2019-04-26 10:59:09 +02:00
Christoph Büscher 078936b8f5 Remove search analyzers from DocumentFieldMappers (#41484)
These references seem to be unused except for tests and should be removed to
keep the places we store analyzers limited.
2019-04-26 09:48:48 +02:00
Armin Braun 6a24fd3f26
Add Restore Operation to SnapshotResiliencyTests (#40634) (#41546)
* Add Restore Operation to SnapshotResiliencyTests

* Expand the successful snapshot test case to also include restoring the snapshop
  * Add indexing of documents as well to be able to meaningfully verify the restore
* This is part of the larger effort to test eventually consistent blob stores in #39504
2019-04-26 09:04:34 +02:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00