Commit Graph

25786 Commits

Author SHA1 Message Date
Christoph Büscher a773d46c69 Remove deprecated `minimum_number_should_match` in BoolQueryBuilder
After deprecating getters and setters and the query DSL parameter in 5.x,
support for `minimum_number_should_match` can be removed entirely. Also
consolidated comments with the ones on 5.x branch and added an entry to the
migration docs.
2017-01-03 15:14:33 +01:00
Jason Tedor 96ba45e310 Fix stale comment in Netty4Utils
We previously named the thread using a frame from the stack trace, but
this was removed to simplify the code here. However, the comment
explaining this was left behind and this commit cleans that up.
2017-01-03 08:15:57 -05:00
Daniel Mitterdorfer 1ed64f0551 Eliminate unneccessary declaration of IOException
With this commit we remove the declaration of IOException from
assertWarnings and modify all call sites.

Checked with @javanna
2017-01-03 12:36:28 +01:00
Christoph Büscher 16d79842ac Remove getters and setters for "minimumNumberShouldMatch" in BoolQueryBuilder
Currently we have getters an setters for both "minimumNumberShouldMatch" and
"minimumShouldMatch", which both access the same internal value
(minimumShouldMatch). Since we only document the `minimum_should_match`
parameter for the query DSL, I think we can deprecate the other getters and
setters for 5.x and remove with 6.0, also deprecating the
`minimum_number_should_match` query DSL parameter.
2017-01-03 11:29:04 +01:00
Tim Vernum 6ad5486e6b Implement Comparable in Version (#22378)
Supports using streams to calculate min/max of a collection of Versions, etc.
2017-01-03 12:20:17 +11:00
Ali Beyad 38427c1df0 [TEST] don't wait for all cluster info in the explain API, just assert
an upper and lower bound
2017-01-02 18:31:17 -05:00
Ali Beyad 91917d6e91 [TEST] mute backwards compatability tests for explain API until 5.2
snapshot builds can be published again
2017-01-02 18:26:19 -05:00
Ali Beyad 49298c16a9 [TEST] fix explain API awaiting info explanation check 2017-01-02 18:18:09 -05:00
Ali Beyad 47907b7093 [TEST] fix explain API test to allow for either awaiting info state or
no valid shard copy
2017-01-02 15:24:01 -05:00
Jason Tedor 0d40608dc8 Remove leftover checkstyle suppression
This commit removes a leftover checkstyle suppression for a source file
that was temporarily forked into the codebase to hack around a bug in
Log4j. When that source file was removed, the suppression was left
behind.
2017-01-02 14:05:09 -05:00
Ali Beyad 20ab4be59f Cluster Explain API uses the allocation process to explain shard allocation decisions (#22182)
This PR completes the refactoring of the cluster allocation explain API and improves it in the following two high-level ways:

 1. The explain API now uses the same allocators that the AllocationService uses to make shard allocation decisions. Prior to this PR, the explain API would run the deciders against each node for the shard in question, but this was not executed on the same code path as the allocators, and many of the scenarios in shard allocation were not captured due to not executing through the same code paths as the allocators.

 2. The APIs have changed, both on the Java and JSON level, to accurately capture the decisions made by the system. The APIs also now report on shard moving and rebalancing decisions, whereas the previous API did not report decisions for moving shards which cannot remain on their current node or rebalancing shards to form a more balanced cluster.

Note: this change affects plugin developers who may have a custom implementation of the ShardsAllocator interface. The method weighShards has been removed and no longer has any utility. In order to support the new explain API, however, a custom implementation of ShardsAllocator must now implement ShardAllocationDecision decideShardAllocation(ShardRouting shard, RoutingAllocation allocation) which provides a decision and explanation for allocating a single shard. For implementations that do not support explaining a single shard allocation via the cluster allocation explain API, this method can simply return an UnsupportedOperationException.
2017-01-02 12:28:32 -06:00
javanna a3918ad094 Remove unused ParseFieldMatcher#match method 2016-12-31 09:24:44 +01:00
javanna cd6b569286 Remove some usages of ParseFieldMatcher in favour of using ParseField directly
Relates to #19552
Relates to #22130
2016-12-31 09:24:44 +01:00
Igor Motov f985638bba Add a generic way of checking version before serializing custom cluster object
In #22313 we added a check that prevents the SnapshotDeletionsInProgress custom cluster state objects from being sent to older elasticsearch nodes. This commits make this check generic and available to other cluster state custom objects if needed.
2016-12-30 14:27:09 -05:00
javanna 74acffaae9 fix compiler warning on access to static field using `this` 2016-12-30 18:57:47 +01:00
javanna df2acb3d9d Remove some more usages of ParseFieldMatcher in favour of using ParseField directly
Relates to #19552
Relates to #22130
2016-12-30 18:57:47 +01:00
javanna 6c54cbade4 Remove some more usages of ParseFieldMatcher in favour of using ParseField directly
Relates to #19552
Relates to #22130
2016-12-30 18:57:47 +01:00
javanna 45d010e874 Remove some usages of ParseFieldMatcher in favour of using ParseField directly
Relates to #19552
Relates to #22130
2016-12-30 18:57:47 +01:00
Dave Richardson eaefb5f99b Descriptions for scripting enabled (#22388)
The `Script source settings` section currently states that `false` means scripting is ENABLED.
The other sections seem to indicate that `false` means scripting is DISABLED.

If the current documentation is correct, that would imply that `inline` and `stored` scripting are ENABLED by default, which seems to conflict with all the other sections in the document.
2016-12-30 12:50:17 -05:00
Adrien Grand 00de5b83bd The percentage of deleted docs needs to be strictly over 10% for deleted docs to be expunged. 2016-12-30 11:18:02 +01:00
Adrien Grand f1d7721932 Fix TermsAggregatorTests to not use LuceneTestCase.newSearcher since it needs a DirectoryReader. 2016-12-30 10:12:24 +01:00
Adrien Grand f89bb18a5d Dynamic `date` fields should use the `format` that was used to detect it is a date. (#22174)
Unless the dynamic templates define an explicit format in the mapping
definition: in that case the explicit mapping should have precedence.

Closes #9410
2016-12-30 09:48:24 +01:00
Adrien Grand 3f805d68cb Add the ability to set an analyzer on keyword fields. (#21919)
This adds a new `normalizer` property to `keyword` fields that pre-processes the
field value prior to indexing, but without altering the `_source`. Note that
only the normalization components that work on a per-character basis are
applied, so for instance stemming filters will be ignored while lowercasing or
ascii folding will be applied.

Closes #18064
2016-12-30 09:36:10 +01:00
Paweł Bobruk 117b63ed41 Docs fix native script usage in 5.x (#22362) 2016-12-29 09:41:26 -08:00
Yannick Welsch 816e1c6cc4 Free shard resources when recovery reset is cancelled
Resetting a recovery consists of resetting the old recovery target and replacing it by a new recovery target object. This is done on the Cancellable threads of
the new recovery target. If the new recovery target is already cancelled before or while this happens, for example due to shard closing or recovery source
changing, we have to make sure that the old recovery target object frees all shard resources.

Relates to #22325
2016-12-29 17:05:47 +01:00
Clinton Gormley 3999e5ba6b Docs: Added link from bool and constant score query to filter context
Closes #22353
2016-12-29 11:05:28 +01:00
Yannick Welsch 6e6d9eb255 Use a fresh recovery id when retrying recoveries (#22325)
Recoveries are tracked on the target node using RecoveryTarget objects that are kept in a RecoveriesCollection. Each recovery has a unique id that is communicated from the recovery target to the source so that it can call back to the target and execute actions using the right recovery context. In case of a network disconnect, recoveries are retried. At the moment, the same recovery id is reused for the restarted recovery. This can lead to confusion though if the disconnect is unilateral and the recovery source continues with the recovery process. If the target reuses the same recovery id while doing a second attempt, there might be two concurrent recoveries running on the source for the same target.

This commit changes the recovery retry process to use a fresh recovery id. It also waits for the first recovery attempt to be fully finished (all resources locally freed) to further prevent concurrent access to the shard. Finally, in case of primary relocation, it also fails a second recovery attempt if the first attempt moved past the finalization step, as the relocation source can then be moved to RELOCATED state and start indexing as primary into the target shard (see TransportReplicationAction). Resetting the target shard in this state could mean that indexing is halted until the recovery retry attempt is completed and could also destroy existing documents indexed and acknowledged before the reset.

Relates to #22043
2016-12-29 10:58:15 +01:00
Igor Motov ca90d9ea82 Remove PROTO-based custom cluster state components
Switches custom cluster state components from PROTO-based de-serialization to named objects based de-serialization
2016-12-28 13:32:35 -05:00
Itamar Syn-Hershko dea8cee70f Fixing a second "second" reference in docs (#22345) 2016-12-28 18:50:10 +01:00
Martijn van Groningen 9ccdd3303d percolator: Fix NPE in percolator's 'now' range check for percolator queries with range queries.
Closes #22355
2016-12-27 22:56:01 +01:00
Jim Ferenczi 02d4cbfeea Fix bwc integ test that tries to perform a term aggs on a scaled_float. This is broken when a node with version prior to 5.2.0 is used with another node > 5.2.0. This is because scaled_float fields are considered as longs in version < 5.2.0. This is fixed in 5.2.0 where scaled_float are recognized as doubles. 2016-12-27 21:52:27 +01:00
Tal Levy e6fb3a5d95 fix index out of bounds error in KV Processor (#22288)
- checks for index-out-of-bounds
- added unit tests for failed `field_split` and `value_split` scenarios

missed this test in #22272.
2016-12-27 10:57:11 -08:00
Jim Ferenczi e7444f7d77 Fix scaled_float numeric type in aggregations (#22351)
`scaled_float` should be used as DOUBLE in aggregations but currently they are used as LONG.
This change fixes this issue and adds a simple it test for it.

Fixes #22350
2016-12-27 09:23:22 +01:00
Adrien Grand 3cb164b22e Fix IndexShardTests.testDocStats. 2016-12-26 20:19:11 +01:00
Adrien Grand 2127db27a3 Add trace logging to CircuitBreakerServiceIT.testParentChecking. 2016-12-26 16:05:27 +01:00
Adrien Grand 2d81750a13 Make ESTestCase resilient to initialization errors. 2016-12-26 14:55:22 +01:00
Adrien Grand f80165c374 Fix LineLength issues. 2016-12-26 11:22:09 +01:00
Adrien Grand d89757b848 Fix mutate function to always actually modify the failure object. 2016-12-26 10:34:50 +01:00
Ali Beyad 1cb5dc42ff Updates SnapshotDeletionsInProgress version number introduced to 5.2.0 2016-12-25 19:28:01 -05:00
Ali Beyad 8261bd358a Synchronize snapshot deletions on the cluster state (#22313)
Before, snapshot/restore would synchronize all operations on the cluster
state except for deleting snapshots.  This meant that only one
snapshot/restore operation would be allowed in the cluster at any given
time, except for deletions - there could be two or more snapshot
deletions running at the same time, or a deletion could be running,
unbeknowest to the rest of the cluster, and thus a snapshot or restore
would be allowed at the same time as the snapshot deletion was still in
progress.  This could cause any number of synchronization issues,
including the situation where a snapshot that was deleted could reappear
in the index-N file, even though its data was no longer present in the
repository.

This commit introduces a new custom type to the cluster state to
represent deletions in progress.  Now, another deletion cannot start if
a deletion is currently in progress.  Similarily, a snapshot or restore
cannot be started if a deletion is currently in progress.  In each case,
if attempting to run another snapshot/restore operation while a deletion
is in progress, a ConcurrentSnapshotExecutionException will be thrown.
This is the same exception thrown if trying to snapshot while another
snapshot is in progress, or restore while a snapshot is in progress.

Closes #19957
2016-12-25 19:00:20 -05:00
Jason Tedor 5185a9734d Add comment clarifying property setting in client
This commit adds a comment clarifying why we do not catch a security
exception in the pre-built transport client.
2016-12-23 18:14:36 -05:00
Jason Tedor d5c18bf5c9 Fix doc stats test when deleting all docs
This commit fixes an issue with IndexShardTests#testDocStats when the
number of deleted docs is equal to the number of docs. In this case,
Luence will remove the underlying segment tripping an assertion on the
number of deleted docs.
2016-12-23 15:20:42 -05:00
Jason Tedor 6deb5283db Fix delete op serialization format constant
The delete op serizliation format constant for 5.x was off by one. This
commit fixes this, and cleans up the handling of these formats.
2016-12-23 11:50:43 -05:00
David Pilato 4fba1c562f Merge pull request #22235 from dadoonet/doc/dbq-java-api
Add documentation for Delete By Query Java API
2016-12-23 16:04:19 +01:00
Jason Tedor 2713549533 Use reader for doc stats
Today we try to pull stats from index writer but we do not get a
consistent view of stats. Under heavy indexing, this inconsistency can
be very skewed indeed. In particular, it can lead to the number of
deleted docs being reported as negative and this leads to serialization
issues. Instead, we should provide a consistent view of the stats by
using an index reader.

Relates #22317
2016-12-23 09:44:56 -05:00
Jay Modi 1243abfecc build: do not use cached snapshots for backwards compatibility tests
The backwards compatibility tests rely on gradle's built-in mechanisms for resolving dependencies
to get the zip of the older version we test against. By default, this will cache snapshots for
24 hours, which can lead to unexpected failures in CI. This change makes the special configurations
for backwards compatibility always update their snapshots by setting the amount of time to cache
to 0 seconds.
2016-12-23 08:17:16 -05:00
Boaz Leskes c2baa5f213 TransportService should capture listener before spawning background notification task
Not doing this made it difficult to establish a happens before relationship between connecting to a node and adding a listeners. Causing test code like this to fail sproadically:

```
        // connection to reuse
        handleA.transportService.connectToNode(handleB.node);

        // install a listener to check that no new connections are made
        handleA.transportService.addConnectionListener(new TransportConnectionListener() {
            @Override
            public void onConnectionOpened(DiscoveryNode node) {
                fail("should not open any connections. got [" + node + "]");
            }
        });

```

relates to #22277
2016-12-23 13:55:12 +01:00
Jason Tedor ddf4a463f3 Reject invalid test logging annotations
Today we silently ignore invalid test logging annotations. This commit
rejects these annotations, failing the processing of the annotation and
aborting the test.
2016-12-23 07:51:35 -05:00
Jason Tedor 432ec54347 Apply logging levels in hierarchical order
This commit adds a test for applying logging levels in hierarchical
order, and addresses an issue with restoring the logging levels at the
end of a test or suite.
2016-12-23 07:51:19 -05:00
Yannick Welsch baea17b53f Separate cluster update tasks that are published from those that are not (#21912)
This commit factors out the cluster state update tasks that are published (ClusterStateUpdateTask) from those that are not (LocalClusterUpdateTask), serving as a basis for future refactorings to separate the publishing mechanism out of ClusterService.
2016-12-23 12:23:52 +01:00