Commit Graph

8420 Commits

Author SHA1 Message Date
Igor Motov 6002b41b5f Add StreamInput.readEnum and StreamOutput.writeEnum (#24475)
Implements the common enum serialization/deserialization pattern for enumeration on the StreamInput/StreamOutput.
2017-05-04 12:22:54 -04:00
Jason Tedor de65f51d34 Simplify file store
Today we go to heroic lengths to workaround bugs in the JDK or around
issues like BSD jails to get information about the underlying file
store. For example, we went to lengths to work around a JDK bug where
the file store returned would incorrectly report whether or not a path
is writable in certain situations in Windows operating
systems. Another bug prevented getting file store information on
Windows on a virtual drive on Windows. We no longer need to work
around these bugs, we could simply try to write to disk and let an I/O
exception arise if we could not write to the disk or take advantage of
the fact that these bugs are fixed in recent releases of the JDK
(e.g., the file store bug is fixed since 8u72). Additionally, we
collected information about all file stores on the system which meant
that if the user had a stale NFS mount, Elasticsearch could hang and
fail on startup if that mount point was not available. Finally, we
collected information through Lucene about whether or not a disk was a
spinning disk versus an SSD, information that we do not need since we
assume SSDs by default. This commit takes into consideration that we
simply do not need this heroic effort, we do not need information
about all file stores, and we do not need information about whether or
not a disk spins to greatly simplfy file store handling.

Relates #24402
2017-05-04 11:19:41 -04:00
Jason Tedor cb46e97a04 Fix reschedule async fsync test
This commit fixes the reschedule async fsync test in index service
tests. This test was passing for the wrong reason. Namely, the test was
trying to set translog durability to async, fire off an indexing
request, and then assert that the translog eventually got fsynced. The
problem here is that in the course of issuing the indexing request, a
mapping update is trigger. The mapping update triggers the index
settings to be refreshed. Since the test did not issue a cluster state
update to change the durability from request to async but instead did
this directly through index service, the mapping update flops the
durability back to async. This means that when the indexing request
executes, an fsync is performed after the request and the assertoin that
an fsync is not needed passes but for the wrong reason (in short: the
test wanted it to pass because an async fsync fired, but instead it
passed because a request async fired). This commit fixes this by going
through the index settings API so that a proper cluster state update is
triggered and so the mapping update does not flop the durability back to
request.
2017-05-04 10:41:20 -04:00
Jason Tedor 50b617f73a Remove global checkpoint assertion in index shard
Due to races, this assertion in index shard can be wrong. This commit
removes that assertion and adjusts the explanatory comment.
2017-05-04 10:33:42 -04:00
Adrien Grand 977016ba25 Do not index `_type` when there is at most one type. (#24363)
This change makes `_type` behave pretty much like `_index` when
`index.mapping.single_type` is true.
2017-05-04 16:29:35 +02:00
Jason Tedor 1fc777b6e3 Change logging level on reroute test
We are still chasing a test failure here and increasing the logging
level stopped the failures. We have a theory as to what is going on so
this commit reduces the logging level to hopefully trigger the failure
again and give us the logging that we need to confirm the theory.
2017-05-04 09:23:19 -04:00
Jim Ferenczi c4002e5ca4 Bump the Lucene version for ES V5_4_1 2017-05-04 14:50:05 +02:00
Simon Willnauer 8356df0846 [TEST] Add a test that alias requests are dense for all indices 2017-05-04 14:29:59 +02:00
Jim Ferenczi 01872f3649 Add version for 5.4.1 and bwc indices for 5.4.0 2017-05-04 13:29:03 +02:00
Simon Willnauer c4eea85713 Use V_5_5_UNRELEASED constant consitently on both request and response 2017-05-04 11:52:07 +02:00
Simon Willnauer 14e57bf9f8 Add cross cluster support to `_field_caps` (#24463)
To support kibana this commit adds an internal optimization
to support the cross cluster syntax for indices on the `_field_caps`
API.

Closes #24334
2017-05-04 11:44:54 +02:00
Adrien Grand d0e71510ad Enforce at most one type. (#24428)
This is a follow-up to #24317, which did the hard work but was merged in such a
way that it exposes the setting while still allowing indices to have multiple
types by default in order to give time to people who test against master to
add this setting to their index settings.
2017-05-04 10:27:50 +02:00
Jason Tedor 4862544934 Increase logging in IndexRecoveryIT#rerouteTest
This test started failing but the logging here is insufficient to
discern what is happening. This commit increases the logging level on
this test until the failure can be understood.
2017-05-03 20:11:41 -04:00
javanna 62c37339b8 Merge branch 'master' into feature/client_aggs_parsing 2017-05-03 22:04:06 +02:00
Igor Motov cacba6bc46 Allow plugins to upgrade templates and index metadata on startup (#24379)
The UpgraderPlugin adds two additional extension points called during cluster upgrade and when old indices are introduced into the cluster state during initial recovery, restore from a snapshot or as a dangling index. One extension point allows plugin to update old templates and another extension points allows to update/check index metadata.
2017-05-03 12:51:41 -04:00
Tim Brooks 855b64b0ee Add non-dispatching listenable action future (#24412)
Currently the only implementation of `ListenableActionFuture` requires
dispatching listener execution to a thread pool. This commit renames
that variant to `DispatchingListenableActionFuture` and converts
`AbstractListenableActionFuture` to be a variant that does not require
dispatching. That class is now named `PlainListenableActionFuture`.
2017-05-03 10:30:54 -05:00
Yannick Welsch 7ecb79a8e1 Remove DiscoveryNodesProvider interface (#24461)
The DiscoveryNodesProvider interface provides an unnecessary abstraction and is just used in conjunction with the existing PingContextProvider interface. This commit removes it.
2017-05-03 16:29:51 +02:00
Luca Cavanna 144f96eaeb Open/Close index api to allow_no_indices by default (#24401)
Open/Close index api have allow_no_indices set to false by default, while delete index has it set to true. The flag controls where a wildcard expression that matches no indices will be ignored or an error will be thrown instead. This commit aligns open/close default behaviour to that of delete index.
2017-05-03 16:22:26 +02:00
Jason Tedor 16af0a9ce2 Remove unnecessary field from UnicastZenPing
This field was only ever used in the constructor, where it was set and
then passed around. As such, there's no need for it to be a field and we
can remove it.
2017-05-03 09:22:24 -04:00
Tanguy Leroux 2ac90b3de9 Add parsing methods for InternalDateHistogram and InternalHistogram (#24213) 2017-05-03 13:27:20 +02:00
Jason Tedor 070963658b Block global checkpoint advances when recovering
After a replica shard finishes recovery, it will be marked as active and
its local checkpoint will be considered in the calculation of the global
checkpoint on the primary. If there were operations in flight during
recovery, when the replica is activated its local checkpoint could be
lagging behind the global checkpoint on the primary. This means that
when the replica shard is activated, we can end up in a situtaion where
a global checkpoint update would want to move the global checkpoint
backwards, violating an invariant of the system. This only arises if a
background global checkpoint sync executes, which today is only a
scheduled operation and might be delayed until the in-flight operations
complete and the replica catches up to the primary. Yet, we are going to
move to inlining global checkpoints which will cause this issue to be
more likely to manifest. Additionally, the global checkpoint on the
replica, which is the local knowledge on the replica updated under the
mandate of the primary, could be higher than the local checkpoint on the
replica, again violating an invariant of the system. This commit
addresses these issues by blocking global checkpoint on the primary when
a replica shard is finalizing recovery. While we have blocked global
checkpoint advancement, recovery on the replica shard will not be
considered complete until its local checkpoint advances to the blocked
global checkpoint.

Relates #24404
2017-05-03 06:48:09 -04:00
Simon Willnauer c99cc8a896 Preserve cluster alias throughout search execution to lookup nodes by cluster and ID (#24438)
today we only lookup nodes by their ID but never by the (clusterAlias, nodeId) tuple.
This could in theory lead to lookups on the wrong cluster if there are 2 clusters
with a node that has the same ID. It's very unlikely to happen but we now can clearly
disambiguate between clusters and their nodes.
2017-05-03 12:03:30 +02:00
javanna 194742b3f4 [TEST] Remove unnecessary usages of ListenableActionFuture 2017-05-03 11:42:06 +02:00
Luca Cavanna 92bfd16c58 Java api: ActionRequestBuilder#execute to return a PlainActionFuture (#24415)
This change makes the request builder code-path same as `Client#execute`. The request builder used to return a `ListenableActionFuture` when calling execute, which allows to associate listeners with the returned future. For async execution though it is recommended to use the `execute` method that accepts an `ActionListener`, like users would do when using `Client#execute`.

Relates to #24412
Relates to #9201
2017-05-03 11:20:53 +02:00
Luca Cavanna fbd793d9e6 Introduce SearchResponseSections base class (#24442)
SearchResponseSections is the common part extracted from InternalSearchResponse that can be shared between high level REST and elasticsearch. The only bits left out are around serialization which is not supported. This way it can accept Aggregations as a constructor argument, without requiring InternalAggregations, as the high level REST client uses its own objects for aggs parsing rather than internal ones.

This change also makes Aggregations implement ToXContent, and Aggregation extend ToXContent. Especially the latter is suboptimal but the best solution that allows to share as much code as possible between core and the client, that doesn't require messing with generics and making the api complicated. Also it doesn't have downsides as all of the current implementations of Aggregation do implement ToXContent already.
2017-05-03 10:45:31 +02:00
Yannick Welsch be19ccef57 Discard stale node responses from async shard fetching (#24434)
Async shard fetching only uses the node id to correlate responses to requests. This can lead to a situation where a response from an earlier request is mistaken as response from a new request when a node is restarted. This commit adds unique round ids to correlate responses to requests.
2017-05-03 09:47:21 +02:00
Jason Tedor 4a578e9c71 Show JVM arguments
We often want the JVM arguments used for a running instance of
Elasticsearch. It sure would be nice if these came as part of the nodes
API, or any API that includes JVM info. This commit causes these
arguments to be displayed.

Relates #24450
2017-05-02 22:25:52 -04:00
Jason Tedor 4289e206ae Log JVM arguments on startup
These can be very useful to have, let's have them at our fingertips in
the logs (in case the JVM dies and we can no longer retrieve them, at
least they are here).

Relates #24451
2017-05-02 21:33:24 -04:00
Nik Everett 732741dd8d Build that java api docs from a test (#24354)
We've had `QueryDSLDocumentationTests` for a while but it had a very
hopeful comment at the top about how we want to make sure that the
example in the query-dsl docs match up with the test but we never
had anything that made *sure* that they did. This changes that!

Now the examples from the query-dsl docs are all built from the
`QueryDSLDocumentationTests`. All except for the percolator example
because that is hard to do as it stands now.

To make this easier this change moves `QueryDSLDocumentationTests`
from core and into the high level rest client. This is useful for
two reasons:
1. We expect the high level rest client to be able to use the builders.
2. The code that builds that docs doesn't check out all of
Elasticsearch. It only checks out certain directories. Since we're
already including snippets from that directory we don't have to
make any changes to that process.

Closes #24320
2017-05-02 13:00:56 -04:00
Simon Willnauer 2f9e9460d4 Move RemoteClusterService into TransportService (#24424)
TransportService and RemoteClusterService are closely coupled already today
and to simplify remote cluster integration down the road it can be a direct
dependency of TransportService. This change moves RemoteClusterService into
TransportService with the goal to make it a hidden implementation detail
of TransportService in followup changes.
2017-05-02 18:09:32 +02:00
Simon Willnauer 691ec68a3c Extract a common base class to allow services to listen to remote cluster config updates (#24367)
RemoteClusterService is an internal service that should not necessarily be exposed
to plugins or other parts of the system. Yet, for cluster name parsing for instance
it is crucial to reuse some code that is used for the RemoteClusterService. This
change extracts a base class that allows to share the settings related code as well
as cluster settings updates to `search.remote.*` to be observed by other services.
2017-05-02 16:02:36 +02:00
Ali Beyad 1435c23df2 Adds check to snapshot repository incompatible-snapshots blob to delete
a pre-existing one before attempting to overwrite it.
2017-05-02 09:56:05 -04:00
javanna 60866da4b7 Merge branch 'master' into feature/client_aggs_parsing 2017-05-02 12:48:30 +02:00
Colin Goodheart-Smithe 7b7cc488da
Fixes checkstyle errors 2017-05-02 11:37:36 +01:00
Winston Ewert c1ba4fdcb4 Allow scripted metric agg to access `_score` (#24295)
* Fixes #24259

Corrects the ScriptedMetricAggregator so that the script can have
access to scores during the map stage.

* Restored original tests. Added seperate test.

As requested, I've restored the non-score dependant tests, and added the
score dependent metric as a seperate test.
2017-05-02 11:24:23 +01:00
Tanguy Leroux ad3c042fc4 [Test] Add unit tests for HDR/TDigest PercentilesAggregators (#24245)
Relates to #22278
2017-05-02 10:55:45 +02:00
David Pilato 186b401fd8 Upgrade to JUnit 4.12 (#23877)
* Upgrade to JUnit 4.12
* Add permission to junit 4.12 and remove junit4-ant specific permission
2017-05-02 09:58:21 +02:00
Ali Beyad 9878aae144 Ensure every repository has an incompatible-snapshots blob (#24403)
In #22267, we introduced the notion of incompatible snapshots in a
repository, and they were stored in a root-level blob named
`incompatible-snapshots`.  If there were no incompatible snapshots in
the repository, then there was no `incompatible-snapshots` blob.

However, this causes a problem for some cloud-based repositories,
because if the blob does not exist, the cloud-based repositories may
attempt to keep retrying the read of a non-existent blob with
expontential backoff until giving up.  This causes performance issues
(and potential timeouts) on snapshot operations because getting the
`incompatible-snapshots` is part of getting the repository data (see
RepositoryData#getRepositoryData()).

This commit fixes the issue by creating an empty
`incompatible-snapshots` blob in the repository if one does not exist.
2017-05-01 15:09:21 -04:00
Nik Everett 8f1fe51f34 Test: more logging for refresh listeners many threads test
This test fails from time to time but doesn't give enough information
for me to figure out what caused the assertion to fail.

Relates to #24418
2017-05-01 14:50:13 -04:00
Luca Cavanna 91fbb0ba28 Move IndicesAliasesRequest#concreteAliases to TransportIndicesAliasesAction (#24400)
This method has to do with how the transport action may or may not resolve wildcards expressions to aliases names. It is only needed in TransportIndicesAliasesAction and for this reason it should be a private method in it rather than part of a request class which is also part of the Java API and later in the high level REST client.
2017-05-01 19:59:06 +02:00
Nik Everett 62712bf653 Test: extra debugging for refresh listeners stats
This test failed in CI but didn't give us enough information to
debug it:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+nfs/223/consoleFull

This turns on more debugging and make the test fail more quickly
if something is obviously wrong.
2017-05-01 11:46:55 -04:00
Koen De Groote 0fef5acd01 Cleanup collections construction
This commit cleans up some cases where a list or map was being
constructed, and then an existing collection was copied into the new
collection. The clean is to instead use an appropriate constructor to
directly copy the existing collection in during collection
construction. The advantage of this is that the new collection is sized
appropriately.

Relates #24409
2017-04-30 21:26:51 -04:00
Yannick Welsch 86aab98fde [TEST] Fix race condition in ZenDiscoveryIT.testDiscoveryStats
With #24236, the master now uses the pending queue when publishing to itself. This means that a cluster state update is put into the pending queue,
then sent to the ClusterApplierService to be applied. After it has been applied, it is marked as processed and removed from the pending queue.
ensureGreen is implemented as a cluster health action that waits on certain conditions, which will lead to a cluster state update task to be submitted
on the master. When this task gets to run and the conditions are not satisfied yet, it register a cluster state observer. This observer is registered
on the ClusterApplierService and waits on cluster state change events. ClusterApplierService first notifies the observer and then the discovery
layer. This means that there is a small time frame where ensureGreen can complete and call the node stats to find the pending queue still containing
the last cluster state update.

Closes #24388
2017-04-28 20:42:56 +02:00
Yannick Welsch 9c55bca8fb Fix node failure detection race when updating cluster state
Failure detection should only be updated in ZenDiscovery after the current state has been updated to prevent a race condition
with handleLeaveRequest and handleNodeFailure as those check the current state to determine whether the failure is to be handled by this node.
2017-04-28 19:52:27 +02:00
olcbean 55daf743d7 Open and close index to honour allow_no_indices option (#24222)
Open/close index API when executed providing an index expressions that matched no indices, threw an error even when allow_no_indices was set to true. The APIs should rather honour the option and behave as a no-op in that case.

Closes #24031
2017-04-28 17:41:05 +02:00
Christoph Büscher 4d14143ac6 Add parsing for InternalGeoCentroid (#24371) 2017-04-28 17:08:48 +02:00
Christoph Büscher eb002340d7 Don't render InternalGeoCentroid in static helper method
This was introduced by a previous commit but is not necessary.
2017-04-28 16:55:07 +02:00
Christoph Büscher db07a34718 Merge branch 'master' into feature/client_aggs_parsing 2017-04-28 16:42:55 +02:00
Christoph Büscher 16a7cbe463 Add `count` value to rest output of `geo_centroid` (#24387)
Currently we don't write the count value to the geo_centroid aggregation rest response,
but it is provided via the java api and the count() method in the GeoCentroid interface. 
We should add this parameter to the rest output and also provide it via the getProperty()
method.
2017-04-28 16:25:22 +02:00
Nik Everett e3b7b88756 Fix compilation in Ecipse (#24391)
Eclipse doesn't allow extra semicolons after an import statement:
```
import foo.Bar;;   // <-- syntax error!
```

Here is the Eclipse bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=425140
which the Eclipse folks closed as "the spec doesn't allow these
semicolons so why should we?" Which is fair. Here is the bug
against javac for allowing them:
https://bugs.openjdk.java.net/browse/JDK-8027682
which hasn't been touched since 2013 without explanation. There
is, however, a rather educations mailing list thread:
http://mail.openjdk.java.net/pipermail/compiler-dev/2013-August/006956.html

which contains gems like, "In general, it is better/simpler to
change javac to conform to the spec. (Except when it is not.)"

I suspect the reason this hasn't been fixed is:
```
FWIW, if we change javac such that the set of programs accepted by javac
is changed, we have an process (currently Oracle internal) to get
approval for such a change.   So, we would not simply change javac on a
whim to meet the spec; we would at least have other eyes looking at the
behavioral change to determine if it is "acceptable".
```
from http://mail.openjdk.java.net/pipermail/compiler-dev/2013-August/006973.html
2017-04-28 09:52:14 -04:00
Guillaume Le Floch 382a617d34 Handle multiple aliases in _cat/aliases api (#23698)
The alias parameter was documented as a list in our rest-spec, yet only the first value out of a list was getting read and processed. This commit adds support for multiple aliases to _cat/aliases

Closes #23661
2017-04-28 15:21:44 +02:00
Christoph Büscher 0d5cca8feb Fixing checktyle error for modifier order 2017-04-28 14:19:00 +02:00
Kunal Kapoor a5bd2012b6 Added validation for upserd request (#24282)
The version on an update request is a syntactic sugar
for get of a specific version, doc merge and a version
index. This changes it to reject requests with both
upsert and a version.
If the upsert index request is versioned, we also
reject the op.
2017-04-28 08:02:09 -04:00
Christoph Büscher a6c38b8f8a Add parsing for InternalGeoBounds (#24365) 2017-04-28 12:10:16 +02:00
Yannick Welsch a72db191f2 Weaken assertion in ZenDiscovery.publish
The previous commit (35f78d098a) introduced an assertion in ZenDiscovery that was overly restrictive - it could trip when a cluster state that was
successfully published would not be applied locally because a master with a better cluster state came along in the meantime.
2017-04-28 11:33:53 +02:00
Yannick Welsch 35f78d098a Separate publishing from applying cluster states (#24236)
Separates cluster state publishing from applying cluster states:

- ClusterService is split into two classes MasterService and ClusterApplierService. MasterService has the responsibility to calculate cluster state updates for actions that want to change the cluster state (create index, update shard routing table, etc.). ClusterApplierService has the responsibility to apply cluster states that have been successfully published and invokes the cluster state appliers and listeners.
- ClusterApplierService keeps track of the last applied state, but MasterService is stateless and uses the last cluster state that is provided by the discovery module to calculate the next prospective state. The ClusterService class is still kept around, which now just delegates actions to ClusterApplierService and MasterService.
- The discovery implementation is now responsible for managing the last cluster state that is used by the consensus layer and the master service. It also exposes the initial cluster state which is used by the ClusterApplierService. The discovery implementation is also responsible for adding the right cluster-level blocks to the initial state.
- NoneDiscovery has been renamed to TribeDiscovery as it is exclusively used by TribeService. It adds the tribe blocks to the initial state.
- ZenDiscovery is synchronized on state changes to the last cluster state that is used by the consensus layer and the master service, and does not submit cluster state update tasks anymore to make changes to the disco state (except when becoming master).

Control flow for cluster state updates is now as follows:

- State updates are sent to MasterService
- MasterService gets the latest committed cluster state from the discovery implementation and calculates the next cluster state to publish
- MasterService submits the new prospective cluster state to the discovery implementation for publishing
- Discovery implementation publishes cluster states to all nodes and, once the state is committed, asks the ClusterApplierService to apply the newly committed state.
- ClusterApplierService applies state to local node.
2017-04-28 09:34:31 +02:00
Toby McLaughlin e4bb360ae0 Fix typo in node environment exception message
This commit fixes a typo in an exception message when trying to create a
node environment.

Relates #24381
2017-04-28 00:48:52 -04:00
Christoph Büscher 9cb3666f0a Merge branch 'master' into feature/client_aggs_parsing 2017-04-28 00:41:16 +02:00
Zachary Tong 350573290f Agg builder accessibility fixes (#24323)
- Getters for DateHisto `interval` and `offset` should return a
long, not double
- Add getter for the filter in a FilterAgg
- Add getters for subaggs / pipelines in base AggregationBuilder
2017-04-27 16:50:59 -04:00
Ryan Ernst cdcc75dd2a Plugins: Add support for platform specific plugins (#24265)
This commit adds support for plugins having a platform specific variant.
It also adds unit tests for all official and maven urls.
2017-04-27 11:27:29 -07:00
Ali Beyad 2facc42a55 Change snapshot status error to use generic SnapshotException (#24355)
Changes the snapshot status read exception from the (misleading)
IndexShardRestoreFailedException to the generic SnapshotException

Closes #24225
2017-04-27 09:36:26 -04:00
Yannick Welsch 2fa1c9fff1 Provide target allocation id as part of start recovery request (#24333)
This makes it possible for the recovery source to verify that it is talking to the shard it thinks it is talking to.

Closes #24167
2017-04-27 14:45:44 +02:00
Tim Vernum 65f90b25e0 Pass Context to ConstructingObjectParser's function (#24230)
Allow the `Context` to be used in the builder function used within ConstructingObjectParser.
This facilitates scenarios where a constructor argument comes from a URL parameter, or from document id.
2017-04-27 20:26:10 +10:00
Christoph Büscher 93415cdfb4 Add parsing for InternalPercentilesBucket (#24330) 2017-04-27 11:04:57 +02:00
Clinton Gormley 8a8410b5ce Added bwc indices for v2.4.5 2017-04-27 10:30:53 +02:00
Ryan Ernst 4a5c3c5a4a Test: Write node ports file before starting tribe service (#24351)
The tribe service can take a while to initialize, depending on how many cluster it needs to connect to. This change moves writing the ports file used by tests to before the tribe service is started.
2017-04-27 09:59:54 +02:00
Adrien Grand 1be2800120 Only allow one type on 7.0 indices (#24317)
This adds the `index.mapping.single_type` setting, which enforces that indices
have at most one type when it is true. The default value is true for 6.0+ indices
and false for old indices.

Relates #15613
2017-04-27 08:43:20 +02:00
Koen De Groote 7f9d84cb1a The parseObject method in DocumentParse can be void. There is no point in the code that actually expects the return, plus the variable created for it was never actually used. (#24350) 2017-04-26 16:54:23 -06:00
Ali Beyad d387dcfd6c [TEST] fixes NPE in RoutingTableTests 2017-04-26 18:39:26 -04:00
Christoph Büscher b46c39ea2e Fixed another sneaky LineLength violation 2017-04-27 00:02:08 +02:00
Christoph Büscher 1f221154a6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-26 23:09:45 +02:00
Christoph Büscher e2b95f9758 Fix order of static field declarations for parser 2017-04-26 23:07:49 +02:00
Ali Beyad 0e74f5ddb1 [TEST] fixes shard count of source shard index in a restore shrink index
test
2017-04-26 16:34:53 -04:00
Christoph Büscher 774368d2ac Add parsing for InternalStatsBucket and InternalExtendedStatsBucket (#24312) 2017-04-26 22:32:30 +02:00
Nik Everett bc45d10e82 Remove most usages of 1-arg Script ctor (#24325)
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.

I've also factored out some copy and pasted script generation
code into a single place. I would have had to change that code
to use `mockScript` anyway, so it was easier to perform the
refactor.

Relates to #16314
2017-04-26 16:04:38 -04:00
Luca Cavanna 149629fec6 Cross Cluster Search: propagate original indices per cluster (#24328)
In case of a Cross Cluster Search, the coordinating node should split the original indices per cluster, and send over to each cluster only its own set of original indices, rather than the set taken from the original search request which contains all the indices.

In fact, each remote cluster should not be aware of the indices belonging to other remote clusters.
2017-04-26 21:45:49 +02:00
Yannick Welsch b7bf651738 [TEST] Fix cluster forming in testDynamicUpdateMinimumMasterNodes
This test can run into a split-brain situation as minimum_master_nodes is not properly set. To prevent this, make sure that at least one of the two
master nodes that are initially started has minimum_master_nodes correctly set.
2017-04-26 21:13:27 +02:00
Martijn van Groningen ebe98f9d62
test: don't randomly wrap index reader 2017-04-26 21:07:56 +02:00
Ali Beyad 0e52e3420e Fixes restore of a shrunken index when initial recovery node is gone (#24322)
When an index is shrunk using the shrink APIs, the shrink operation adds
some internal index settings to the shrink index, for example
`index.shrink.source.name|uuid` to denote the source index, as well as
`index.routing.allocation.initial_recovery._id` to denote the node on
which all shards for the source index resided when the shrunken index
was created.  However, this presents a problem when taking a snapshot of
the shrunken index and restoring it to a cluster where the initial
recovery node is not present, or restoring to the same cluster where the
initial recovery node is offline or decomissioned.  The restore
operation fails to allocate the shard in the shrunken index to a node
when the initial recovery node is not present, and a restore type of
recovery will *not* go through the PrimaryShardAllocator, meaning that
it will not have the chance to force allocate the primary to a node in
the cluster.  Rather, restore initiated shard allocation goes through
the BalancedShardAllocator which does not attempt to force allocate a
primary.

This commit fixes the aforementioned problem by not requiring allocation
to occur on the initial recovery node when the recovery type is a
restore of a snapshot.  This commit also ensures that the internal
shrink index settings are recognized and not archived (which can trip an
assertion in the restore scenario).

Closes #24257
2017-04-26 14:48:10 -04:00
Koen De Groote 3187ed73fc Removal of dead code in ScriptedMetricAggregationBuilder (#24346)
This code removes a few lines of dead code from ScriptedMetricAggregationBuilder. 
Just completely dead code, it adds things to a Set that is then not used in any way.
2017-04-26 14:44:03 -04:00
Koen De Groote 4c0eb35c22 Removal of dead code from SnapshotsService (#24347)
This code removes a few lines of dead code from SnapshotsService. 
Looks like a forgotten remnant of a past implementation.
2017-04-26 14:32:35 -04:00
Nik Everett 7c3efb829b Move char filters into analysis-common (#24261)
Another step down the road to dropping the
lucene-analyzers-common dependency from core.

Note that this removes some tests that no longer compile from
core. I played around with adding them to the analysis-common
module where they would compile but we already test these in
the tests generated from the example usage in the documentation.

I'm not super happy with the way that `requriesAnalysisSettings`
works with regards to plugins. I think it'd be fairly bug-prone
for plugin authors to use. But I'm making it visible as is for
now and I'll rethink later.

A part of #23658
2017-04-26 13:25:34 -04:00
Christoph Büscher db1b243343 InternalPercentilesBucket should not rely on ordered percents array (#24336)
Currently InternalPercentilesBucket#percentile() relies on the percent array passed in
to be in sorted order. This changes the aggregation to store an internal lookup table that
is constructed from the percent/percentiles arrays passed in that can be used to look up 
the percentile values.

Closes #24331
2017-04-26 19:15:48 +02:00
Yannick Welsch 91b61ce569 [TEST] Do a reroute with retry_failed after a bridge partition on testAckedIndexing
In case of a bridge partition, shard allocation can fail "index.allocation.max_retries" times if the master is the super-connected node and recovery
source and target are on opposite sides of the bridge. This commit adds a reroute with retry_failed after healing the network partition so that the
ensureGreen check succeeds.
2017-04-26 16:08:16 +02:00
Jay Modi 7f8fe8b81d StreamInput throws exceptions instead of using assertions (#24294)
StreamInput has methods such as readVInt that perform sanity checks on the data using assertions,
which will catch bad data in tests but provide no safety when running as a node without assertions
enabled. The use of assertions also make testing with invalid data difficult since we would need
to handle assertion errors in the code using the stream input and errors like this should not be
something we try to catch. This commit introduces a flag that will throw an IOException instead of
using an assertion.
2017-04-26 07:23:07 -04:00
Martijn van Groningen c17de49a6d
[percolator] Fix memory leak when percolator uses bitset or field data cache.
The percolator doesn't close the IndexReader of the memory index any more.
Prior to 2.x the percolator had its own SearchContext (PercolatorContext) that did this,
but that was removed when the percolator was refactored as part of the 5.0 release.

I think an alternative way to fix this is to let percolator not use the bitset and fielddata caches,
that way we prevent the memory leak.

Closes #24108
2017-04-26 11:08:15 +02:00
Christoph Büscher fede0bab2c Correcting 140 character line length 2017-04-26 11:04:33 +02:00
Koen De Groote 3c845727f8 Replace alternating regex with character classes
This commit replaces two alternating regular expressions (that is,
regular expressions that consist of the form a|b where a and b are
characters) with the equivalent regular expression rewritten as a
character class (that is, [ab]) The reason this is an improvement is
because a|b involves backtracking while [ab] does not.

Relates #24316
2017-04-25 22:15:00 -04:00
Christoph Büscher 0e31a21db2 Add parsing for InternalExtendedStats (#24284) 2017-04-25 23:55:24 +02:00
Guillaume Le Floch 739cb35d1b Allow passing single scrollID in clear scroll API body (#24242)
* Allow single scrollId in string format

Closes #24233
2017-04-25 13:43:21 +02:00
Christoph Büscher 768420db55 Add parsing for InternalStats (#24239) 2017-04-25 13:42:15 +02:00
Koen De Groote 88de33d43d Minor changes to collection creation from enums (#24274)
These changes are mainly cosmetic with minor perf advantages drawn from checkstyle.
2017-04-25 13:13:55 +02:00
Ryan Ernst 6ebf08759b Templates: Add compileTemplate method to ScriptService for template consumers (#24280)
This commit adds a compileTemplate method to the ScriptService.
Eventually this will be used to easily cutover all consumers to a new
TemplateService.

relates #16314
2017-04-24 15:45:20 -07:00
Christoph Büscher 1cb34b5eba Merge branch 'master' into feature/client_aggs_parsing 2017-04-24 18:42:41 +02:00
Christoph Büscher 026bf2e3ee Remove getCountAsString() from InternalStats and Stats interface (#24291)
The `count` value in the stats aggregation represents a simple doc count
that doesn't require a formatted version. We didn't render an "as_string"
version for count in the rest response, so the method should also be
removed in favour of just using String.valueOf(getCount()) if a string
version of the count is needed.

Closes #24287
2017-04-24 18:40:57 +02:00
Ali Beyad c5b6f52ecc Fixes maintaining the shards a snapshot is waiting on (#24289)
There was a bug in the calculation of the shards that a snapshot must
wait on, due to their relocating or initializing, before the snapshot
can proceed safely to snapshot the shard data.  In this bug, an
incorrect key was used to look up the index of the waiting shards,
resulting in the fact that each index would have at most one shard in
the waiting state causing the snapshot to pause.  This could be
problematic if there are more than one shard in the relocating or
initializing state, which would result in a snapshot prematurely
starting because it thinks its only waiting on one relocating or
initializing shard (when in fact there could be more than one).  While
not a common case and likely rare in practice, it is still problematic.

This commit fixes the issue by ensuring the correct key is used to look
up the waiting indices map as it is being built up, so the list of
waiting shards for each index (those shards that are relocating or
initializing) are aggregated for a given index instead of overwritten.
2017-04-24 10:59:08 -04:00
Martijn van Groningen dabbf5d4f4
[TEST] Added unittests for InternalGeoCentroid
Relates to #22278
2017-04-24 16:57:25 +02:00
Nilabh Sagar 373edee29a Provide informative error message in case of unknown suggestion context. (#24241)
Provide a list of available contexts when you send an unknown context to the completion suggester.
2017-04-24 10:35:14 -04:00
Jason Tedor 1500beafc7 Check for default.path.data included in path.data
If the user explicitly configured path.data to include
default.path.data, then we should not fail the node if we find indices
in default.path.data. This commit addresses this.

Relates #24285
2017-04-24 09:31:54 -04:00
Jason Tedor a7947b404b Fix hash code for AliasFilter
This commit fixes the hash code for AliasFilter as the previous
implementation was neglecting to take into consideration the fact that
the aliases field is an array and thus a deep hash code of it should be
computed rather than a shallow hash code on the reference.

Relates #24286
2017-04-24 09:06:36 -04:00
Christoph Büscher 5541debd5a Merge branch 'master' into feature/client_aggs_parsing 2017-04-24 14:48:20 +02:00
Yannick Welsch 7c395070e2 [TEST] Wait for tribe node to be fully connected before shutting it down
The tribe was being shutdown by the test while a publishing round (that adds the tribe node to a cluster) is not completed yet (i.e. the node itself
knows that it became part of the cluster, and the test shuts the tribe node down, but another node has not applied the cluster state yet, which makes
that node hang while trying to connect to the node that is shutting down (due to connect_timeout being 30 seconds), delaying publishing for 30
seconds, and subsequently tripping an assertion when another tribe instance wants to join.

Relates to #23695
2017-04-24 12:27:41 +02:00
Colin Goodheart-Smithe 6d6a230f70
Makes StoredScriptSource implement ToXContentObject 2017-04-24 10:20:15 +01:00
Colin Goodheart-Smithe d4a6ba8ec9 No longer add illegal content type option to stored search templates (#24251)
When parsing StoredSearchScript we were adding a Content type option that was forbidden (by a check that threw an exception) by the parser thats used to parse the template when we read it from the cluster state. This was stopping Elastisearch from starting after stored search templates had been added.

This change no longer adds the content type option to the StoredScriptSource object when parsing from the put search template request.  This is safe because the StoredScriptSource content is always JSON when its stored in the cluster state since we do a conversion to JSON before this point.

Also removes the check for the content type in the options when parsing StoredScriptSource so users who already have stored scripts can start Elasticsearch.

Closes #24227
2017-04-22 13:37:04 -04:00
Ryan Ernst 473e98981b Scripts: Remove unnecessary executable shortcut (#24264)
ScriptService has two executable methods, one which takes a
CompiledScript, which is similar to search, and one that takes a raw
Script and both compiles and returns an ExecutableScript for it. The
latter is not needed, and the call sites which used one or the other
were mixed. This commit removes the extra executable method in favor of
callers first calling compile, then executable.
2017-04-21 17:53:03 -07:00
Ryan Ernst aadc33d260 Scripts: Remove unwrap method from executable scripts (#24263)
The unwrap method was leftover from support javascript and python. Since
those languages are removed in 6.0, this commit removes the unwrap
feature from scripts.
2017-04-21 17:50:22 -07:00
Nik Everett 447f307ebb Fix _bulk response when it can't create an index (#24048)
Before #22488 when an index couldn't be created during a `_bulk`
operation we'd do all the *other* actions and return the index
creation error on each failing action. In #22488 we accidentally
changed it so that we now reject the entire bulk request if a single
action cannot create an index that it must create to run. This
gets reverts to the old behavior while still keeping the nicer
error messages. Instead of failing the entire request we now only
fail the portions of the request that can't work because the index
doesn't exist.

Closes #24028
2017-04-21 18:56:04 -04:00
Jason Tedor fe91c72151 Use a marker file when removing a plugin
Today when removing a plugin, we attempt to move the plugin directory to
a temporary directory and then delete that directory from the
filesystem. We do this to avoid a plugin being in a half-removed
state. We previously tried an atomic move, and fell back to a non-atomic
move if that failed. Atomic moves can fail on union filesystems when the
plugin directory is not in the top layer of the
filesystem. Interestingly, the regular move can fail as well. This is
because when the JDK is executing such a move, it first tries to rename
the source directory to the target directory and if this fails with
EXDEV (as in the case of an atomic move failing), it falls back to
copying the source to the target, and then attempts to rmdir the
source. The bug here is that the JDK never deleted the contents of the
source so the rmdir will always fail (except in the case of an empty
directory).

Given all this silliness, we were inspired to find a different
strategy. The strategy is simple. We will add a marker file to the
plugin directory that indicates the plugin is in a state of
removal. This file will be the last file out the door during removal. If
this file exists during startup, we fail startup.

Relates #24252
2017-04-21 15:50:44 -04:00
Simon Willnauer 2ca7072b24 Fill missing sequence IDs up to max sequence ID when recovering from store (#24238)
Today we might promote a primary and recover from store where after translog
recovery the local checkpoint is still behind the maximum sequence ID seen.
To fill the holes in the sequence ID history this PR adds a utility method
that fills up all missing sequence IDs up to the maximum seen sequence ID
with no-ops.

Relates to #10708
2017-04-21 20:28:00 +02:00
Ryan Ernst ba48674695 Build: Move plugin cli and tests to distribution tool (#24220)
The plugin cli currently resides inside the elasticsearch jar. This
commit moves it into a plugin-cli jar. This is change alone is a no-op;
it does not change anything about what is loaded at runtime. But it will
allow easier testing (with fixtures in the future to test ES or maven
installation), as well as eventually not loading these classes when
starting elasticsearch.
2017-04-21 09:25:58 -07:00
Boaz Leskes badb2be066 Peer Recovery: remove maxUnsafeAutoIdTimestamp hand off (#24243)
With #24149 , it is now stored in the Lucene commit and is implicitly transferred in the file phase of the recovery.
2017-04-21 17:31:50 +02:00
Ali Beyad 63e5aff5d6 Adds version 5.3.2 and backwards compatibility indices for 5.3.1 2017-04-21 10:48:41 -04:00
Tanguy Leroux adef5e227a Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-21 15:46:18 +02:00
Tanguy Leroux 480bf0996d Add utility method to parse named XContent objects with typed prefix (#24240)
This commit adds a XContentParserUtils.parseTypedKeysObject() method
that can be used to parse named XContent objects identified by a field
name containing a type identifier, a delimiter and the name of the
object to parse.
2017-04-21 15:41:27 +02:00
Tanguy Leroux 251b6d452b MultiBucketsAggregation.Bucket should not extend Writeable (#24216)
The MultiBucketsAggregation.Bucket interface extends Writeable, forcing
all implementation classes to implement writeTo(). This commit removes
the Writeable from the interface and move it down to the InternalBucket
implementation.
2017-04-21 15:29:53 +02:00
Yannick Welsch c2deb1c81d Don't expose cleaned-up tasks as pending in PrioritizedEsThreadPoolExecutor (#24237)
Changes in #24102 exposed the following oddity: PrioritizedEsThreadPoolExecutor.getPending() can return Pending entries where pending.task == null. This can happen for example when tasks are added to the pending list while they are in the clean up phase, i.e. TieBreakingPrioritizedRunnable#runAndClean has run already, but afterExecute has not removed the task yet. Instead of safeguarding consumers of the API (as was done before #24102) this changes the executor to not count these tasks as pending at all.
2017-04-21 15:25:19 +02:00
Christoph Büscher 4a69b658cd Merge branch 'master' into feature/client_aggs_parsing 2017-04-21 12:18:06 +02:00
Colin Goodheart-Smithe 3c7c4bc824 Adds declareNamedObjects methods to ConstructingObjectParser (#24219)
* Adds declareNamedObjects methods to ConstructingObjectParser

* Addresses review comments
2017-04-21 09:50:30 +01:00
Christoph Büscher c8ad26edc9 Tests: Extend InternalStatsTests (#24212)
Currently we don't test for count = 0 which will make a difference when adding
tests for parsing for the high level rest client. Also min/max/sum should also
be tested with negative values and on a larger range.
2017-04-21 10:38:09 +02:00
Adrien Grand 81b64ed587 IndicesQueryCache should delegate the scorerSupplier method. (#24209)
Otherwise the range improvements that we did on range queries would not work.
This is similar to https://issues.apache.org/jira/browse/LUCENE-7749.
2017-04-21 10:33:02 +02:00
Adrien Grand f322f537e4 Speed up parsing of large `terms` queries. (#24210)
The addition of the normalization feature on keywords slowed down the parsing
of large `terms` queries since all terms now have to go through normalization.
However this can be avoided in the default case that the analyzer is a
`keyword` analyzer since all that normalization will do is a UTF8 conversion.
Using `Analyzer.normalize` for that is a bit overkill and could be skipped.
2017-04-21 10:32:33 +02:00
Jim Ferenczi a4365971a0 [TEST] make sure that the random query_string query generator defines a default_field or a list of fields 2017-04-21 02:56:26 +02:00
Fabien Baligand 4a45579506 token_count type : add an option to count tokens (fix #23227) (#24175)
Add option "enable_position_increments" with default value true.
If option is set to false, indexed value is the number of tokens
(not position increments count)
2017-04-21 00:53:28 +02:00
javanna d9916f20a6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-20 23:03:58 +02:00
Jim Ferenczi 525101b64d Query string default field (#24214)
Currently any `query_string` query that use a wildcard field with no matching field is rewritten with the `_all` field.

For instance:
````
#creating test doc
PUT testing/t/1
{
  "test": {
    "field_one": "hello",
    "field_two": "world"
  }
}
#searching abc.* (does not exist) -> hit
GET testing/t/_search
{
  "query": {
    "query_string": {
      "fields": [
        "abc.*"
      ],
      "query": "hello"
    }
  }
}
````

This bug first appeared in 5.0 after the query refactoring and impacts only users that use `_all` as default field.
Indices created in 6.x will not have this problem since `_all` is deactivated in this version.

This change fixes this bug by returning a MatchNoDocsQuery for any term that expand to an empty list of field.
2017-04-20 22:12:20 +02:00
Luca Cavanna 82c678b5c7 Make Aggregations an abstract class rather than an interface (#24184)
Some of the base methods that don't have to do with reduce phase and serialization can be moved to the base class which is no longer an interface. This will be reusable by the high level REST client further on the road. Also it simplify things as having an interface with a single implementor is not that helpful.
2017-04-20 21:31:34 +02:00
Areek Zillur 077a6c3ee7 [TEST] ensure expected sequence no and version are set when index/delete engine operation has a document failure 2017-04-20 13:38:52 -04:00
Yannick Welsch 22e0795990 Extract batch executor out of cluster service (#24102)
Refactoring that extracts the task batching functionality from ClusterService and makes it a reusable component that can be tested in isolation.
2017-04-20 17:28:43 +02:00
Tanguy Leroux c8fc30a999 [Test] Expose AbstractPercentilesTestCase.randomPercents() 2017-04-20 13:36:16 +02:00
Tanguy Leroux 35946a13d0 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/InternalPercentilesTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/hdr/InternalHDRPercentilesTests.java
2017-04-20 13:35:21 +02:00
Tanguy Leroux d0df1ed193 [Test] Always check the XContent equivalent when parsing aggregations (#24208)
In InternalAggregationTestCase, we can check that the internal aggregation and the parsed aggregation always produce the same XContent even if the original internal aggregation has been shuffled or not.
2017-04-20 13:22:50 +02:00
Tanguy Leroux 55a879ee8d Align behavior or HDR percentiles iterator with percentile() method (#24206) 2017-04-20 12:37:33 +02:00
Christoph Büscher 6e22a1e9ba Add parsing for InternalBucketMetricValue (#24182) 2017-04-20 11:49:09 +02:00
Tanguy Leroux 2b14db9b70 Remove @Repeat(iterations = 1000) in tests 2017-04-20 10:44:12 +02:00
Tanguy Leroux 11da77388a Add parsing methods for Percentiles aggregations (#24183) 2017-04-20 10:16:51 +02:00
Nik Everett caf376c8af Start building analysis-common module (#23614)
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Jason Tedor 4796557a30 Add primary term to doc write response
This commit adds the primary term to the doc write response.

Relates #24171
2017-04-19 14:44:22 -04:00
Ryan Ernst c7e9231a86 Plugins: Remove leniency for missing plugins dir (#24173)
This leniency was left in after plugin installer refactoring for 2.0
because some tests still relied on it. However, the need for this
leniency no longer exists.
2017-04-19 09:09:34 -07:00
Christoph Büscher e12339a683 Merge branch 'master' into feature/client_aggs_parsing 2017-04-19 16:37:22 +02:00
Christoph Büscher a9657a5a09 Add BucketMetricValue interface (#24188)
Unlike other implementations of InternalNumericMetricsAggregation.SingleValue,
the InternalBucketMetricValue aggregation currently doesn't implement a
specialized interface that exposes the `keys()` method. This change adds this so
that clients can access the keys via the interface.
2017-04-19 16:27:33 +02:00
Jim Ferenczi f05af0a382 Enable index-time sorting (#24055)
This change adds an index setting to define how the documents should be sorted inside each Segment.
It allows any numeric, date, boolean or keyword field inside a mapping to be used to sort the index on disk.
It is not allowed to use a `nested` fields inside an index that defines an index sorting since `nested` fields relies on the original sort of the index.
This change does not add early termination capabilities in the search layer. This will be added in a follow up.

Relates #6720
2017-04-19 14:36:11 +02:00
Christoph Büscher 4562c8a345 Add parsing for InternalSimpleValue and InternalDerivative (#24162) 2017-04-19 12:58:26 +02:00
Tanguy Leroux bf5cfabe04 Fix checkstyle violation 2017-04-19 10:23:25 +02:00
Tanguy Leroux 5717ac3cc6 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/DocValueFormat.java
#	core/src/test/java/org/elasticsearch/search/aggregations/InternalAggregationTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalMaxTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvgTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/min/InternalMinTests.java
2017-04-19 10:12:11 +02:00
Boaz Leskes 8758c541b3 ElectMasterService.hasEnoughMasterNodes should return false if no masters were found
This is a regression introduced in #20063
2017-04-19 09:52:06 +02:00
Tanguy Leroux 741c031384 [Test] Add unit tests for InternalHDRPercentilesTests (#24157)
Related to #22278
2017-04-19 09:37:01 +02:00
Areek Zillur 4f773e2dbb Replicate write failures (#23314)
* Replicate write failures

Currently, when a primary write operation fails after generating
a sequence number, the failure is not communicated to the replicas.
Ideally, every operation which generates a sequence number on primary
should be recorded in all replicas.

In this change, a sequence number is associated with write operation
failure. When a failure with an assinged seqence number arrives at a
replica, the failure cause and sequence number is recorded in the translog
and the sequence number is marked as completed via executing `Engine.noOp`
on the replica engine.

* use zlong to serialize seq_no

* Incorporate feedback

* track write failures in translog as a noop in primary

* Add tests for replicating write failures.

Test that document failure (w/ seq no generated) are recorded
as no-op in the translog for primary and replica shards

* Update to master

* update shouldExecuteOnReplica comment

* rename indexshard noop to markSeqNoAsNoOp

* remove redundant conditional

* Consolidate possible replica action for bulk item request
depanding on it's primary execution

* remove bulk shard result abstraction

* fix failure handling logic for bwc

* add more tests

* minor fix

* cleanup

* incorporate feedback

* incorporate feedback

* add assert to remove handling noop primary response when 5.0 nodes are not supported
2017-04-19 01:23:54 -04:00
Jason Tedor 9e0ebc5965 Rename variable in translog simple commit test
This commit renames a variable for clarity in the translog simple commit
test.
2017-04-18 23:43:25 -04:00
Jason Tedor 20181dd0ad Strengthen translog commit with open view test
This commit strengthens an assertion in the translog commit with open
view test.
2017-04-18 23:41:55 -04:00
Jason Tedor 180d1f2219 Stronger check in translog prepare and commit test
This commit strengthens an assertion in the translog prepare commit and
commit test.
2017-04-18 23:37:54 -04:00
Jason Tedor 23b224a5a9 Fix translog prepare commit and commit test
This test was terribly, horribly, no goodly, and badly broken it's
amazing it ever passed so this commit fixes it.
2017-04-18 23:32:47 -04:00
Boaz Leskes edff30f82a Engine: store maxUnsafeAutoIdTimestamp in commit (#24149)
The `maxUnsafeAutoIdTimestamp` timestamp is a safety marker guaranteeing that no retried-indexing operation with a higher auto gen id timestamp was process by the engine. This allows us to safely process documents without checking if they were seen before.

Currently this property is maintained in memory and is handed off from the primary to any replica during the recovery process.

This commit takes a more natural approach and stores it in the lucene commit, using the same semantics (no retry op with a higher time stamp is part of this commit). This means that the knowledge is transferred during the file copy and also means that we don't need to worry about crazy situations where an original append only request arrives at the engine after a retry was processed *and* the engine was restarted.
2017-04-18 20:11:32 +02:00
Christoph Büscher 210e101f6d Minor changes in ParsedCardinality 2017-04-18 18:09:45 +02:00
Christoph Büscher bc646cf7ad Adding parsing for InternalValueCount 2017-04-18 18:09:39 +02:00
Christoph Büscher 5f96972b04 Adding parsing for InternalAvg 2017-04-18 17:48:11 +02:00
Christoph Büscher 695b2858f4 Adding parsing for InternalSum 2017-04-18 17:44:05 +02:00
Christoph Büscher 75fdc9449f Adding parsing for InternalMax and InternalMin 2017-04-18 17:38:44 +02:00
Simon Willnauer ab9884b2e9 Remove leniency when merging fetched hits in a search response phase (#24158)
Today when we merge hits we have a hard check to prevent AIOOB exceptions
that simply skips an expected search hit. This can only happen if there is a
bug in the code which should be turned into a hard exception or an assertion
triggered. This change adds an assertion an removes the lenient check for the
fetched hits.
2017-04-18 17:19:57 +02:00
Tanguy Leroux 829dd068d6 [Test] Use appropriate DocValueFormats in Aggregations tests (#24155)
Some aggregations (like Min, Max etc) use a wrong DocValueFormat in
tests (like IP or GeoHash). We should not test aggregations that expect
a numeric value with a DocValueFormat like IP. Such wrong DocValueFormat
can also prevent the aggregation to be rendered as ToXContent, and this
will be an issue for the High Level Rest Client tests which expect to be
able to parse back aggregations.
2017-04-18 17:03:32 +02:00
Tanguy Leroux c1ba6997ff AbstractParsedPercentiles should use Percentile class (#24160)
Now the Percentile interface has been merged with the InternalPercentile
class in core (#24154) the AbstractParsedPercentiles should use it.

This commit also changes InternalPercentilesRanksTestCase so that it now
tests the iterator obtained from parsed percentiles ranks aggregations.

Adding this new test raised an issue in the iterators where key and
value are "swapped" in internal implementations when building the
iterators (see InternalTDigestPercentileRanks.Iter constructor that
accepts the `keys` as the first parameter named `values`, each key
being mapped to the `value` field of Percentile class). This is because
 percentiles ranks aggs inverts percentiles/values compared to the
 percentiles aggs.

* Add assume in InternalAggregationTestCase

* Update after Luca review
2017-04-18 16:54:17 +02:00
Christoph Büscher 8f540346a9 Tests: Fixing typo in class name of InternalGlobalTests
Renaming from InternalGlogbalTests -> InternalGlobalTests
2017-04-18 16:27:15 +02:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Tanguy Leroux 67a9696e55 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-18 14:55:58 +02:00
Tanguy Leroux f217eb8ad8 Merge Percentile class with interface (#24154)
This commit merges the Percentile interface with the InternalPercentile
class, as we don't need to maintain both.
2017-04-18 14:47:18 +02:00
Martijn van Groningen edada2581e
[TEST] Added unittests for InternalSampler 2017-04-18 14:31:58 +02:00
Yannick Welsch 0b2cb68f6f [TEST] Randomly add and remove no_master blocks in IndicesClusterStateServiceRandomUpdatesTests
Checks that IndicesClusterStateService stays consistent with incoming cluster states that contain no_master blocks (especially
discovery.zen.no_master_block=all which disables state persistence). In particular this checks that active shards which have no in-memory data
structures on a node are failed.
2017-04-18 14:27:54 +02:00
Martijn van Groningen ac41fb2c4a
[TEST] Added test for GeoCentroidAggregator and
made constructors of GeoCentroidAggregator, GeoCentroidAggregatorFactory and InternalGeoCentroid package protected.
2017-04-18 13:54:31 +02:00
Tanguy Leroux c0036d8516 Add parsing for percentiles ranks (#23974)
This commit adds the logic for parsing the percentiles ranks aggregations.
2017-04-18 10:19:30 +02:00
Tanguy Leroux 81dbdb239f [Test] Add unit tests for InternalTDigestPercentilesTests (#24090) 2017-04-18 09:48:35 +02:00
Chris Earle 12c8423ec9 Warn on not enough masters during election (#20063)
This changes the trace level logging to warn, and adds the needed number to the message as well.

My fear is that it may get noisy, but this is an issue that you want to be noisy.
2017-04-17 22:18:28 -04:00
Jason Tedor 34eda1a1a8 Do not set path.data in environment if not set
When preparing the final settings in the environment, we unconditionally
set path.data even if path.data was not explicitly set. This confounds
detection for whether or not path.data was explicitly set, and this is
trappy. This commit adds logic to only set path.data in the final
settings if path.data was explicitly set, and provides a test case that
fails without this logic.

Relates #24132
2017-04-17 10:43:13 -04:00
Jason Tedor f7ebe9d18f Preserve multiple translog generations
Today when a flush is performed, the translog is committed and if there
are no outstanding views, only the current translog generation is
preserved. Yet for the purpose of sequence numbers, we need stronger
guarantees than this. This commit migrates the preservation of translog
generations to keep the minimum generation that would be needed to
recover after the local checkpoint.

Relates #24015
2017-04-17 08:51:54 -04:00
Jason Tedor 8033c576b7 Detect remnants of path.data/default.path.data bug
In Elasticsearch 5.3.0 a bug was introduced in the merging of default
settings when the target setting existed as an array. When this bug
concerns path.data and default.path.data, we ended up in a situation
where the paths specified in both settings would be used to write index
data. Since our packaging sets default.path.data, users that configure
multiple data paths via an array and use the packaging are subject to
having shards land in paths in default.path.data when that is very
likely not what they intended.

This commit is an attempt to rectify this situation. If path.data and
default.path.data are configured, we check for the presence of indices
there. If we find any, we log messages explaining the situation and fail
the node.

Relates #24099
2017-04-17 07:03:46 -04:00
jaymode a8be0a5836
Cat APIs should not close the stream obtained from the channel
The cat APIs and rest tables would obtain a stream from the RestChannel, which happened to be a
ReleasableBytesStreamOutput. These APIs used the stream to write content to, closed the stream,
and then tried to send a response. After #23941 was merged, closing the stream meant that the bytes
were released for use elsewhere. This caused occasional corruption of the response when the bytes
were used prior to the response being sent.

This commit changes these two usages to wrap the stream obtained from the channel in a flush on
close stream so that the bytes are still reserved until the message is sent.
2017-04-15 14:57:00 -04:00
Jason Tedor cd8e059885 Do not produce empty IDs in simple versioning test
Empty IDs are rejected during indexing, so we should not randomly
produce them during tests. This commit modifies the simple versioning
tests to no longer produce empty IDs.
2017-04-15 12:15:45 -04:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates #24118
2017-04-15 10:36:03 -04:00
Boaz Leskes ecf81688fb Use sequence numbers to identify out of order delivery in replicas & recovery (#24060)
Internal indexing requests in Elasticsearch may be processed out of order and repeatedly. This is important during recovery and due to concurrency in replicating requests between primary and replicas. As such, a replica/recovering shard needs to be able to identify that an incoming request contains information that is old and thus need not be processed. The current logic is based on external version. This is sadly not sufficient. This PR moves the logic to rely on sequences numbers and primary terms which give the semantics we need.

Relates to #10708
2017-04-14 21:46:17 +02:00
Jason Tedor 09efdc3151 Improve performance of extracting warning value
When building headers for a REST response, we de-duplicate the warning
headers based on the actual warning value. The current implementation of
this uses a capturing regular expression that is prone to excessive
backtracking. In cases a request involves a large number of warnings,
this extraction can be a severe performance penalty. An example where
this can arise is a bulk indexing request that utilizes a deprecated
feature (e.g., using deprecated forms of boolean values). This commit is
an attempt to address this performance regression. We already know the
format of the warning header, so we do not need to use a regular
expression to parse it but rather can parse it by hand to extract the
warning value. This gains back the vast majority of the performance lost
due to the usage of a deprecated feature. There is still a performance
loss due to logging the deprecation message but we do not address that
concern in this commit.

Relates #24114
2017-04-14 12:18:00 -04:00
Jay Modi 30ab8739a6 Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23941)
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only
release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use
the channel builder and in turn the bytes stream output that is managed by the channel.

Note, this commit differs from 6bfecdf921 in that it updates
ReleasableBytesStreamOutput to handle the case of the BigArray decreasing in size, which changes
the reference to the BigArray. When the reference is changed, the releasable needs to be updated
otherwise there could be a leak of bytes and corruption of data in unrelated streams.

This reverts commit afd45c1432, which reverted #23572.
2017-04-14 10:50:31 -04:00
javanna 5ccb4a0bbd fix typo in ParsedCardinality comment and add //norelease comment on DocValueFormat dep 2017-04-14 11:22:49 +02:00
Yannick Welsch e3aa2a89f9 [TEST] Wait in OldIndexBackwardsCompatibilityIT for cluster to be fully initialized
There are test failures that suggest that the import of dangling indices is happening too early, before the dangling indices are ready to be consumed.
This commit adds an ensureGreen() at the end of cluster initialization to make sure that no cluster state updates are happening while the dangling
indices are prepared on-disk.
2017-04-14 11:02:55 +02:00
Ali Beyad 5e54c0261a [TEST] fixes InternalTopHitsTests test to initialize the SearchHits
maxScore to Float.NaN if there is no max score, as that is what Lucene's
TopDocs does
2017-04-13 18:27:42 -04:00
Igor Motov cce321a560 Task Management: Make TaskInfo parsing forwards compatible (#24073)
TaskInfo is stored as a part of TaskResult and therefore can be read by nodes with an older version. If we add any additional information to TaskInfo (for #23250, for example), nodes with an older version should be able to ignore it, otherwise they will not be able to read TaskResults stored by newer nodes.
2017-04-13 16:16:01 -04:00
Tim Brooks ffaac5a08a Simplify BulkProcessor handling and retry logic (#24051)
This commit collapses the SyncBulkRequestHandler and
AsyncBulkRequestHandler into a single BulkRequestHandler. The new
handler executes a bulk request and awaits for the completion if the
BulkProcessor was configured with a concurrentRequests setting of 0.
Otherwise the execution happens asynchronously.

As part of this change the Retry class has been refactored.
withSyncBackoff and withAsyncBackoff have been replaced with two
versions of withBackoff. One method takes a listener that will be
called on completion. The other method returns a future that will been
complete on request completion.
2017-04-13 14:48:52 -05:00
Jason Tedor 99e0268e0a Remove support for default settings
Today Elasticsearch allows default settings to be used only if the
actual setting is not set. These settings are trappy, and the complexity
invites bugs. This commit removes support for default settings with the
exception of default.path.data, default.path.conf, and default.path.logs
which are maintainted to support packaging. A follow-up will remove
support for these as well.

Relates #24093
2017-04-13 14:25:45 -04:00
Jason Tedor 32b2caad42 Correct handling of default and array settings
In Elasticsearch 5.3.0 a bug was introduced in the merging of default
settings when the target setting existed as an array. This arose due to
the fact that when a target setting is an array, the setting key is
broken into key.0, key.1, ..., key.n, one for each element of the
array. When settings are replaced by default.key, we are looking for the
target key but not the target key.0. This leads to key, and key.0, ...,
key.n being present in the constructed settings object. This commit
addresses two issues here. The first is that we fix the merging of the
keys so that when we try to merge default.key, we also check for the
presence of the flattened keys. The second is that when we try to get a
setting value as an array from a settings object, we check whether or
not the backing map contains the top-level key as well as the flattened
keys. This latter check would have caught the first bug. For kicks, we
add some tests.

Relates #24074
2017-04-13 06:34:58 -04:00
Tanguy Leroux 7f730c9489 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-13 09:24:03 +02:00
Ryan Ernst fb3a281755 Build: Switch jna dependency to an elastic version (#24081)
This new version of jna is rebuilt from the official release of jna, but
with native libs linked against older glibc in order to support all
platforms elasticsearch supports.

closes #23640
2017-04-13 00:17:50 -07:00
Boaz Leskes 215a9b2df9 fix CategoryContextMappingTests compilation bugs 2017-04-13 09:15:10 +02:00
Boaz Leskes 342e745fc7 testConcurrentGetAndSetOnPrimary - fix a race condition between indexing and updating value map
Currently the map can be lagging behind what's actually in lucene causes assertions about adding/removing values to fail
2017-04-13 09:03:09 +02:00
Nilabh Sagar ec421974b9 Allow different data types for category in Context suggester (#23491)
The "category" in context suggester could be String, Number or Boolean. However with the changes in version 5 this is failing and only accepting String. This will have problem for existing users of Elasticsearch if they choose to migrate to higher version; as their existing Mapping and query will fail as mentioned in a bug #22358

This PR fixes the above mentioned issue and allows user to migrate seamlessly.

Closes #22358
2017-04-12 23:43:29 -07:00
Ryan Ernst c19044ddf6 Restrict build info loading to ES jar, not any jar (#24049)
This change makes the build info initialization only try to load a jar
manifest if it is the elasticsearch jar. Anything else (eg a repackaged
ES for use of transport client in an uber jar) will contain "Unknown"
for the build info as it does for tests currently.

fixes #21955
2017-04-12 23:22:43 -07:00
Jason Tedor 12b46bdbc4 Remove more hidden file leniency from plugins
This commit removes one more instance of leniency from the plugin
service which skips hidden files in the plugins directory.

Relates #23982
2017-04-12 22:23:42 -04:00
Jason Tedor edd16fa27e Register error listener in evil logger tests
This test needs an error listener registered since we configure logging
here.
2017-04-12 21:23:05 -04:00
Jason Tedor a1c2fe9e3a Detect using logging before configuration
It can easily happen that we touch a logger before logging is configured
due to chains of static intializers and other such scenarios. This
commit adds detection for this mechanism that will fail startup if we
touch a logger before logging is configured. This is a bug that will
cause builds to fail.

Relates #24076
2017-04-12 21:13:08 -04:00
Nik Everett 31c8903492 Add version constant for 5.5 (#24075)
This is required in master now that #24071 is in or else we fail during BWC testing because the 5.x branch contains 5.5 but the build thinks it should contain 5.4.
2017-04-12 16:29:47 -04:00
Zachary Tong 1fd50bc54d Add unit tests for NestedAggregator (#24054)
Add unit tests for NestedAggregator, change class visibilities

Relates to #22278
2017-04-12 15:59:51 -04:00
Nik Everett e99f90fb46 Add more debugging information to rethrottles
I'm still trying to track down failures like:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/1180/console

It looks like a task is hanging but I'm not sure why. So this
adds more logging for next time.
2017-04-12 08:37:31 -04:00
Christoph Büscher cf89fb86b5 Merge branch 'master' into feature/client_aggs_parsing 2017-04-12 12:00:40 +02:00
Christoph Büscher 356532816a Adding ParsedCardinality (#23973)
Adding parsing of InternalCardinality xContent output. Parsing method will return a new
implementation of the Cardinality interface, ParsedCardinality.
2017-04-12 11:58:57 +02:00
Christoph Büscher 1847bbac4d Tests: Use random analyzer only on string fields in Match/MultiMatchBuilderTests
Currently we can run into test errors by accidently using e.g. a "simple"
analyzer on a numeric field which might lead to number parsing errors. While
these errors are correct, we should avoid these combinations in our regular
tests.
2017-04-12 11:32:48 +02:00
Ryan Ernst 1207103b6d S3 Repository: Eagerly load static settings (#23910)
The S3 repostiory has many levels of settings it looks at to create a
repository, and these settings were read at repository creation time.
This meant secure settings like access and secret keys had to be
available after node construction. This change makes setting loading for
every except repository level settings eager, so that secure settings
can be stashed, and the keystore can once again be closed after
bootstrapping the node is complete.
2017-04-11 15:42:56 -07:00
Jason Tedor b4c3bb5d21 Reject duplicate settings on the command line
Today Elasticsearch and other CLI tools that rely on environment aware
command leniently accept duplicate settings with the last one
winning. This commit removes this leniency.

Relates #24053
2017-04-11 18:30:05 -04:00
Tim Brooks cf6b03c8f4 Wildcard cluster names for cross cluster search (#23985)
This is related to #23893. This commit allows users to use wilcards for
cluster names when executing a cross cluster search.

So instead of defining every cluster such as:

GET one:*,two:*,three:*/_search

A user could just search:

GET *:*/_search

As ":" characters are currently allowed in index names, if the text
up to the first ":" does not match a defined cluster name, the entire
string is treated as an index name.
2017-04-11 13:56:26 -05:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Simon Willnauer e30a275bfe Add a dedicated TransportRemoteInfoAction for consistency (#24040)
All our actions that are invoked from rest actions have corresponding
transport actions. This adds the transport action for RestRemoteClusterInfoAction
for consistency.

Relates to #23969
2017-04-11 14:40:37 +02:00
Yannick Welsch 88a54f14c7 Trigger replica recovery restarts by master when primary relocation completes (#23926)
When a primary relocation completes while there are ongoing replica recoveries, the recoveries for these replicas need to be restarted (as a new primary is in charge of replicating changes). Before this commit, the need for a recovery restart was detected by the data nodes that had the replicas, by checking on each cluster state update if the recovery process had completed before the recovery source changed. That code had a race, however, which could lead to a not-fully recovered shard exposing itself as started (see #23904).

This commit takes a different approach: When the primary relocation completes and the master updates the cluster state to move the primary shard from relocating to started, it will reinitialize all initializing replica shards, by giving them a fresh allocation id. Data nodes that have the replica shard will simply detect that the allocation id changed and restart the recovery process (instead of trying to determine the need to restart based on ongoing recoveries).

Note: Removal of the code in IndicesClusterStateService that checks whether the recovery source has changed will not be backported to the 5.x branch. This ensures backward compatibility for the situation where the master node is older and does not have the code changes that have been introduced in this PR.

Closes #23904
2017-04-11 11:21:57 +02:00
Colin Goodheart-Smithe 0114f0061c Removes version 2.x constants from Version (#24011)
* Removes version 2.x constants from Version

Closes #21887

* Addresses review comments
2017-04-11 08:31:22 +01:00
Simon Willnauer f22e0dc30b Add cross-cluster search remote cluster info API (#23969)
This commit adds an API to discover information like seed nodes,
http addresses and connection status of a configured remote cluster.

Closes #23925
2017-04-11 09:24:40 +02:00
Nik Everett 16a2048416 Remove real time from tests (#24025)
The `AsyncBulkByScrollActionTests` were brittle because they used the
current time. That was a mistake. This removes the current time from
the test, instead adding it to the parameters passed in to the
appropriate methods. This means that we take the current time slightly
earlier in all cases, but that shouldn't make a difference.

Closes #24005

Example failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+nfs/161/consoleFull
2017-04-10 17:55:02 -04:00
Ryan Ernst 65f7a76630 Settings: Add secure file setting to keystore (#24001)
Some systems like GCE rely on a plaintext file containing credentials.
Rather than extract the information out of that credentials file and
store each peace individually in the keystore, it is cleaner to just
store the entire file.

This commit adds support to the keystore wrapper for secure file
settings. These are settings that contain an entire file that would
normally be stored on the local filesystem. Retrieving the file returns
an input stream to the file contents. This also adds a `add-file`
command to the keystore cli.

In order to support both strings and files as values for settings, the
metadata format of the keystore has also been updated (with backcompat)
to keep a map of setting name to type.
2017-04-10 13:10:42 -07:00
Simon Willnauer a61fb3f708 Remote support for lucene versions without checksums (#24021)
We are still carrying some legacy code that deals with lucene indices
that don't have checksums. Yet, we do not support these indices
for a while now, in fact since version 5.0 such an index is not supported
anymore. This commit removes all the special handling and leniency involved.
2017-04-10 18:16:34 +02:00
Martijn van Groningen 887f3ed8dc
inner_hits: Replace `NestedChildrenQuery` with `ParentChildrenBlockJoinQuery`.
Closes #24009
2017-04-10 17:36:45 +02:00
Lee Hinman 53d4d747a6 Mark IndexWithShadowReplicasIT as AwaitsFix
Relates to #24007 and #23906
2017-04-10 09:32:20 -06:00
Simon Willnauer 040b86a76b Set shard count limit to unlimited (#24012)
Now that we have incremental reduce functions for topN and aggregations
we can set the default for `action.search.shard_count.limit` to unlimited.
This still allows users to restrict these settings while by default we executed
across all shards matching the search requests index pattern.
2017-04-10 17:09:21 +02:00
javanna f538d7b8d6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-10 14:46:57 +02:00
Luca Cavanna 2c545c064d Move getProperty method out of MultiBucketsAggregation.Bucket interface (#23988)
The getProperty method is an internal method needed to run pipeline aggregations and retrieve info by path from the aggs tree. It is not needed in the MultiBucketsAggregation.Bucket interface, which is returned to users running aggregations from the transport client. The method is moved to the InternalMultiBucketAggregation class as that's where it belongs.
2017-04-10 13:35:01 +02:00
Luca Cavanna 93f159429f Remove getProperty method from Aggregations interface and impl (#23972)
The `getProperty` method is an internal method needed to run pipeline aggregations and retrieve info by path from the aggs tree. It is not needed in the `Aggregations` interface, which is returned to users running aggregations from the transport client. Furthermore, the method is currenty unused by pipeline aggs too, as only InternalAggregation#getProperty is used. It can then be removed
2017-04-10 12:31:45 +02:00
Luca Cavanna b283c8b768 Move aggs CommonFields and TYPED_KEYS_DELIMITER from InternalAggregation to Aggregation (#23987)
These will be shared between internal objects and objects exposed through high level REST client, so they should be moved from internal classes.
2017-04-10 12:30:02 +02:00
Luca Cavanna 9db8a266e6 Un-deprecate NamedXContentRegistry.Entry constructor that takes a context (#23986)
We deprecated this method in the past because we thought it was a temporary thing that could go away over time. We radically trimmed down the usages of a context while parsing when we got rid of the ParseFieldMatcher, but the usages that are left are legit and we will hardly get rid of them. Also, working on aggs parsing we will need a context to carry around the aggregation name that gets parsed through XContentParser#namedObject .
2017-04-10 12:28:56 +02:00
Yannick Welsch 12471c4f76 [TEST] Fix wait condition on testMultipleNodesShutdownNonMasterNodes
After two nodes are being stopped and two more are joining the cluster, we first have to wait on the cluster to consist of the right nodes before
waiting on green status, otherwise we might get a green status for a cluster with dead nodes.
2017-04-10 11:38:56 +02:00
Jim Ferenczi 9b3c85dd88 Deprecate _field_stats endpoint (#23914)
_field_stats has evolved quite a lot to become a multi purpose API capable of retrieving the field capabilities and the min/max value for a field.
In the mean time a more focused API called `_field_caps` has been added, this enpoint is a good replacement for _field_stats since he can
retrieve the field capabilities by just looking at the field mapping (no lookup in the index structures).
Also the recent improvement made to range queries makes the _field_stats API obsolete since this queries are now rewritten per shard based on the min/max found for the field.
This means that a range query that does not match any document in a shard can return quickly and can be cached efficiently.
For these reasons this change deprecates _field_stats. The deprecation should happen in 5.4 but we won't remove this API in 6.x yet which is why
 this PR is made directly to 6.0.
 The rest tests have also been adapted to not throw an error while this change is backported to 5.4.
2017-04-10 10:10:16 +02:00
Simon Willnauer 1f40f8a2d2 Introduce incremental reduction of TopDocs (#23946)
This commit adds support for incremental top N reduction if the number of
expected shards in the search request is high enough. The changes here
also clean up more code in SearchPhaseController to make the separation
between values that are the same on each search result and values that
are per response. The reduced search phase result doesn't hold an arbitrary
result to obtain values like `from`, `size` or sort values which is now
cleanly encapsulated.
2017-04-10 09:37:52 +02:00
Boaz Leskes b636ca79d5 Engine: version logic on replicas should not be hard coded (#23998)
The refactoring in #23711 hardcoded version logic for replica to assume monotonic versions. Sadly that's wrong for `FORCE` and `VERSION_GTE`. Instead we should use the methods in VersionType to detect conflicts.

Note - once replicas use sequence numbers for out of order delivery, this logic goes away.
2017-04-09 22:04:12 +02:00
Boaz Leskes f0df5e64d8 InternalEngineTests: fix a potential NPE in assertOpsOnPrimary
assertOpsOnPrimary may inherit a situation where the document exist but it doesn't the last indexed value.
This cloud cause an NPE.
2017-04-09 21:21:00 +02:00
Jason Tedor 61c5976aee Upgrade to Log4j 2.8.2
This commit upgrades the Log4j dependencies from version 2.7 to version
2.8.2. This release includes a fix for a case where Log4j could lose
exceptions in the presence of a security manager.

Relates #23995
2017-04-09 07:19:16 -04:00
Jason Tedor 5c8d5677a4 Suppress ExtrasFS in plugins service tests
The ExtrasFS filesystem creates extra directories when creating temp
directories during tests to ensure that Lucene does not care about extra
files. These extra files get in our way in the plugins service tests
because some of these tests are counting only on certain directories
existing. This commit suppresses the ExtrasFS filesystem for the plugins
service tests, and fixes a test that was passing for the wrong reason
(because of the existence of an extra directory from ExtrasFS).
2017-04-08 20:42:18 -04:00
Jason Tedor 9056e0cb49 Remove hidden file leniency from plugin service
This commit removes some leniency from the plugin service which skips
hidden files in the plugins directory. We really want to ensure the
integrity of the plugin folder, so hasta la vista leniency.

Relates #23982
2017-04-08 18:22:44 -04:00
javanna 12e8a45de7 remove some more TODOs from ParsedAggregation 2017-04-08 00:41:32 +02:00
javanna 9e7b020578 Remove TODO on un-deprecating NamedContentRegistry.Entry ctor that takes a context 2017-04-08 00:28:49 +02:00
javanna 8464b4755f [TEST] replace FareRestRequest usage with ToXContent.MapParams 2017-04-08 00:18:56 +02:00
Ryan Ernst 73b8aad9a3 Settings: Disallow secure setting to exist in normal settings (#23976)
This commit removes the "legacy" feature of secure settings, which setup
a parallel setting that was a fallback in the insecure
elasticsearch.yml. This was previously used to allow the new secure
setting name to be that of the old setting name, but is now not in use
due to other refactorings. It is much cleaner to just have all secure
settings use new setting names. If in the future we want to reuse the
previous setting name, once support for the insecure settings have been
removed, we can then rename the secure setting.  This also adds a test
for the behavior.
2017-04-07 14:18:06 -07:00
javanna 306ef086c5 Align ParsedAggregation meta to InternalAggregation behaviour
Empty meta gets printed out, which means that if the request contains an empty meta object, that is returned with the response as well. On the other hand null, meaning when the object is not in the request, is not printed out. ParsedAggregation used to not print out empty metadata, and didn't allow the null value. Aligned behaviour to the existing behaviour from InternalAggregation.
2017-04-07 21:54:46 +02:00
Simon Willnauer 0c465b1931 Add comment why we check for null fetch results during merge 2017-04-07 21:00:19 +02:00
Jason Tedor 457a76c1c6 Fix import order in Spawner
The imports are not in alphabetical order in Spawner.java and this is a
crime that is rectified by this commit.
2017-04-07 14:52:22 -04:00
javanna 39e791291e Merge branch 'master' into feature/client_aggs_parsing 2017-04-07 15:43:32 +02:00
Yannick Welsch a3cceb8a00 [TEST] Fix testMultipleNodesShutdownNonMasterNodes to wait for the right nodes to rejoin the cluster
This test was sporadically failing for the following reason:
- 4 nodes (nodes 0, 1, 2, and 3) running with `minimum_master_nodes` set to 3
- we stop 2 nodes (node 0 and 3)
- wait for cluster block to be in place on all nodes
- start 2 nodes (node 4 and node 5) and do a `prepareHealth().setWaitForNodes("4")`
- then do a search request

The search request runs into the `ClusterBlockException` as the `prepareHealth().setWaitForNodes("4")` check succeeds on a cluster state that has
nodes 1, 2, 3, and 4, i.e., only one of the two new nodes has joined the cluster and only one of the two dead nodes was removed by the master
(removing the dead nodes only happens after there are again `minimum_master_nodes` nodes in the cluster).

This commit fixes the issue by reusing a method from InternalTestCluster that checks that the right nodes have rejoined the cluster.
2017-04-07 15:26:21 +02:00
Jim Ferenczi 0821fa23ff Restore special case for wilcard on _all query to rewrite to a match all query (#23967)
This change restores the rewrite to a match all query that we used to apply on wildcard query * on the query_string parser before #23433.
2017-04-07 15:15:43 +02:00
Yannick Welsch 8522b43ce7 [TEST] Take cluster state batching into account in testNodeFailuresAreProcessedOnce
The test assumes that two nodes leaving the cluster results in two cluster state updates on the master, which is invalidated by cluster state
batching.
2017-04-07 14:43:38 +02:00
Christoph Büscher 4f94aa8a6a Tests: Fix highlighter fields order in TopHitsTests (#23968)
Shuffling xContent breaks the order of the highlighter fields in the
internal list if the highlighter doesn't use the array syntax. In other tests we
avoid shuffling this json level, but since this is done in the base test for
aggregations we should ensure the highlight builder uses the array syntax here.
2017-04-07 14:24:32 +02:00
javanna 420fa8c400 Add ParsedAggregation as base Aggregation impl for high level client
ParsedAggregation is the base Aggregation implementation for the high level client, which parses aggs responses into java objects.
2017-04-07 11:11:06 +02:00
Luca Cavanna e156dbaf42 Move getProperty method out of Aggregation interface (#23949)
The `getProperty` method is an internal method needed to run pipeline aggregations and retrieve info by path from the aggs tree. It is not needed in the `Aggregation` interface, which is  returned to users running aggregations from the transport client. The method is moved to the InternalAggregation class as that's where it belongs.
2017-04-07 10:55:35 +02:00
Luca Cavanna 13cf8aaa52 [TEST] fix shuffling of xContent keys (#23929)
ESTestCase has methods to shuffle xContent keys given a builder or a parser. Shuffling wasn't actually doing what was expected but rather reordering the keys in their natural ordering, hence the output was always the same at every run. Corrected that and added tests, also fixed a couple of tests that were affected by this fix.
2017-04-07 10:20:32 +02:00
Ali Beyad 480cfe3fe0 Fixes snapshot status on failed snapshots (#23833)
If a snapshot is taken on multiple indices, and some of them are "good"
indices that don't contain any corruption or failures, and some of them
are "bad" indices that contain missing shards or corrupted shards, and
if the snapshot request is set to partial=false (meaning don't take a
snapshot if there are any failures), then the good indices will not be
snapshotted either.  Previously, when getting the status of such a
snapshot, a 500 error would be thrown, because the snap-*.dat blob for
the shards in the good index could not be found.

This commit fixes the problem by reporting shards of good indices as
failed due to a failed snapshot, instead of throwing the
NoSuchFileException.

Closes #23716
2017-04-06 20:54:21 -04:00
Jay Modi 495bf21b46 Preserve response headers when creating an index (#23950)
This commit preserves the response headers when creating an index and updating settings for an
index.

Closes #23947
2017-04-06 20:38:09 +01:00
Jim Ferenczi 042f7566e8 update Version.V_5_3_1_UNRELEASED to the latest bugfix release of Lucene:6_4_2 2017-04-06 10:03:17 +02:00
Jim Ferenczi 38009efedd Disable graph analysis at query time for shingle and cjk filters producing tokens of different size (#23920)
This change disables graph analysis of token streams containing a shingle or a cjk filters that produce shingle or ngram of different size. The graph analysis is disabled for phrase and boolean queries.

Closes #23918
2017-04-06 08:55:00 +02:00
Tim Brooks 5b1fbe5e6c Decouple BulkProcessor from client implementation (#23373)
This commit modifies the BulkProcessor to be decoupled from the
client implementation. Instead it just takes a
BiConsumer<BulkRequest, ActionListener<BulkResponse>> that executes
the BulkRequest.
2017-04-05 12:12:43 -05:00
Lee Hinman 0257a7b97a Only re-parse operation if a mapping update was needed
When executing an index operation on the primary shard,
`TransportShardBulkAction` first parses the document, sees if there are any
mapping updates that needs to be applied, and then updates the mapping on the
master node. It then re-parses the document to make sure that the mappings have
been applied and propagated.

This adds a check that skips the second parsing of the document in the event
there was not a mapping update applied in the first case.

Fixes a performance regression introduced in #23665
2017-04-05 09:29:44 -06:00
Adrien Grand d5d0f140d6 The `filter` and `significant_terms` aggregations should parse the `filter` as a filter, not as a query. (#23797)
This is important for some queries like `bool`, which are parsed differently
depending on whether we want to get a query or a filter.
2017-04-05 16:46:21 +02:00
Simon Willnauer adccdbb3cf Simplify sorted top docs merging in SearchPhaseController (#23881)
Today we have several code paths to merge top docs based on the number of
search results returned from the shards. If there is a only a single shard
holding any hits we go a different code path with quite some complexity while
if there are more than one the code is basically duplicated to safe the
creation of a dense array of top docs which can be large if there are many results.
This commit removes the need of the dense array and in-turn the justification for
the optimization. This commit introduces a single code path to merge top docs.
2017-04-05 14:49:35 +02:00
Boaz Leskes 75b4f408e0 Refactor InternalEngine's index/delete flow for better clarity (#23711)
The InternalEngine Index/Delete methods (plus satellites like version loading from Lucene) have accumulated some cruft over the years making it hard to clearly the code flows for various use cases (primary indexing/recovery/replicas etc). This PR refactors those methods for better readability. The methods are broken up into smaller sub methods, albeit at the price of less code I reused.

To support the refactoring I have considerably beefed up the versioning tests.

This PR is a spin-off from #23543 , which made it clear this is needed.
2017-04-05 14:43:01 +02:00
Boaz Leskes c89fdd938e ZenDiscovery - only validate min_master_nodes values if local node is master (#23915)
The purpose of this validation is to make sure that the master doesn't step down
due to a change in master nodes, which also means that there is no way to revert
an accidental change. Since we validate using the current cluster state (and
not the one from which the settings come from) we have to be careful and only
validate if the local node is already a master. Doing so all the time causes
subtle issues. For example, a node that joins a cluster has no nodes in its
current cluster state. When it receives a cluster state from the master with
a dynamic minimum master nodes setting int it, we must make sure we don't reject it.

Closes #23695
2017-04-05 14:31:32 +02:00
Jason Tedor 24127bf416 Remove hardcoded ports from SingleNodeDiscoveryIT
SingleNodeDiscoveryIT uses a hardcoded port for the purpose of binding
two nodes within the limited port range that an unconfigured unicast zen
ping hosts list would try to discover another node on. This commit at
least removes this hardcoding for the first node to come up, although
still tries to bind the second node to the limited port range after the
first node has bound.
2017-04-05 08:17:33 -04:00
Luca Cavanna 318d365b12 [TEST] make sure that fromXContent doesn't rely on keys ordering (#23901)
We shuffle the keys before we parse our responses for the high level client so that we make sure we never rely on keys ordering.
2017-04-05 11:12:34 +02:00
Jason Tedor afd45c1432 Revert "Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23572)"
This reverts commit 6bfecdf921.
2017-04-04 20:33:51 -04:00
Jay Modi 6bfecdf921 Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23572)
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use the channel builder and in turn the bytes stream output that is managed by the channel.
2017-04-04 17:01:30 +01:00
Jason Tedor 3136ed1490 Rename random ASCII helper methods
This commit renames the random ASCII helper methods in ESTestCase. This
is because this method ultimately uses the random ASCII methods from
randomized runner, but these methods actually only produce random
strings generated from [a-zA-Z].

Relates #23886
2017-04-04 11:04:18 -04:00
Jason Tedor a01f77210a Fix Javadocs for BootstrapChecks#enforceLimits
This commit adds a description for a parameter that was added to
BootstrapChecks#enforceLimits(BoundTransportAddress, String) without the
Javadocs having been updated.
2017-04-04 09:42:19 -04:00
Jason Tedor 51b5dbffb7 Disable bootstrap checks for single-node discovery
While there are use-cases where a single-node is in production, there
are also use-cases for starting a single-node that binds transport to an
external interface where the node is not in production (for example, for
testing the transport client against a node started in a Docker
container). It's tricky to balance the desire to always enforce the
bootstrap checks when a node might be in production with the need for
the community to perform testing in situations that would trip the
bootstrap checks. This commit enables some flexibility for these
users. By setting the discovery type to "single-node", we disable the
bootstrap checks independently of how transport is bound. While this
sounds like a hole in the bootstrap checks, the bootstrap checks can
already be avoided in the single-node use-case by binding only HTTP but
not transport. For users that are genuinely in production on a
single-node use-case with transport bound to an external use-case, they
can set the system property "es.enable.bootstrap.checks" to force
running the bootstrap checks. It would be a mistake for them not to do
this.

Relates #23598
2017-04-04 09:39:04 -04:00
Jim Ferenczi c14be20744 Add unit tests for the missing aggregator (#23895)
* Add unit tests for the missing aggregator

Relates #22278
2017-04-04 14:37:33 +02:00
Jim Ferenczi a04350f0dd Add a property to mark setting as final (#23872)
This change adds a setting property that sets the value of a setting as final.
Updating a final setting is prohibited in any context, for instance an index setting
marked as final must be set at index creation and will refuse any update even if the index is closed.
This change also marks the setting `index.number_of_shards` as Final and the special casing for refusing the updates on this setting has been removed.
2017-04-04 12:35:48 +02:00
Jason Tedor 71293a89bf Introduce single-node discovery
This commit adds a single node discovery type. With this discovery type,
a node will elect itself as master and never form a cluster with another
node.

Relates #23595
2017-04-04 03:02:58 -04:00
Jason Tedor 3bd2efa177 Await termination after shutting down executors
When terminating an executor service or a thread pool, we first
shutdown. Then, we do a timed await termination. If the await
termination fails because there are still tasks running, we then
shutdown now. However, this method does not wait for actively executing
tasks to terminate, so we should again wait for termination of these
tasks before returning. This commit does that.

Relates #23889
2017-04-04 03:01:00 -04:00
Jason Tedor 6234a49fb3 Fix initialization issue in ElasticsearchException
If a test touches ElasticsearchExceptionHandle before the class
initialzer for ElasticsearchException has run, a circular class
initialization problem can arise. Namely, the class initializer for
ElasticsearchExceptionHandle depends on the class initializer for
ElasticsearchExceptionHandle which depends on the class initializer for
all the classes that extend ElasticsearchException, but these classes
can not be loaded because ElasticsearchException has not finished its
class initializer. There are tests that can trigger this before
ElasticsearchException has been loaded due to an unlucky ordering of
test execution. This commit addresses this issue by making
ElasticsearchExceptionHandle private, and then exposing methods that
provide the necessary values from ElasticsearchExceptionHandle. Touching
these methods will force the class initializer for
ElasticsearchException to run first.
2017-04-04 00:33:00 -04:00
Boaz Leskes 48b0121f60 SpecificMasterNodesIT shouldn't use autoMinMasterNodes
as it tweaks the `discovery.initial_state_timeout` setting.
2017-04-03 16:23:17 +02:00
Boaz Leskes 40eb68c95a testRestorePersistentSettings doesn't to mess with discovery settings 2017-04-03 16:23:17 +02:00
Colin Goodheart-Smithe 8482503f9b
Adds tests for cardinality and filter aggregations
Relates to #22278
2017-04-03 10:09:27 +01:00
Colin Goodheart-Smithe cad4fcd9c9
Revert "Adds tests for cardinality and filter aggregations (#23826)"
This reverts commit 058869ed54.
2017-04-03 09:45:16 +01:00
Colin Goodheart-Smithe 058869ed54 Adds tests for cardinality and filter aggregations (#23826)
* Adds tests for cardinality and filter aggregations

Relates to #22278

* addresses review comments
2017-04-03 09:39:03 +01:00
Jim Ferenczi 7316b663e2 Replace custom sort field with SortedSetSortField and SortedNumericSortField when possible (#23827)
Currently for field sorting we always use a custom sort field and a custom comparator source.
Though for numeric fields this custom sort field could be replaced with a standard SortedNumericSortField unless
the field is nested especially since we removed the FieldData for numerics.
We can also use a SortedSetSortField for string sort based on doc_values when the field is not nested.

This change replaces IndexFieldData#comparatorSource with IndexFieldData#sortField that returns a Sorted{Set,Numeric}SortField when possible or a custom
 sort field when the field sort spec is not handled by the SortedSortFields.
2017-04-03 09:57:26 +02:00
Simon Willnauer bdb1cabe71 Prevent nodes from joining if newer indices exist in the cluster (#23843)
Today we prevent nodes from joining when indices exists that are too old.
Yet, the opposite can happen too since lucene / elasticsearch is not forward
compatible when it gets to indices we won't let nodes join the cluster once
there are indices in the clusterstate that are newer than the nodes version.
This prevents forward compatibility issues which we never test against. Yet,
this will not prevent rolling restarts or anything like this since indices
are always created with the minimum node version in the cluster such that an index
can only get the version of the higher nodes once all nodes are upgraded to this version.
2017-04-03 09:52:09 +02:00
Simon Willnauer 998eeb7687 Synchronized CollapseTopFieldDocs with lucenes relatives (#23854)
TopDocs et.al. got additional parameters to incrementally reduce
top docs. In order to add incremental reduction `CollapseTopFieldDocs`
needs to have the same properties.
2017-04-03 09:50:44 +02:00
Jason Tedor 7082baaed9 Stricter parsing of remote node attribute
This commit enables stricter parsing of the remote node attribute,
instead of leniently parsing values that are not "true" as false.
2017-04-01 13:18:46 -04:00
Jason Tedor 38b3fec885 Fix cross-cluster remote node gateway attributes
Remote nodes in cross-cluster search can be marked as eligible for
acting a gateway node via a remote node attribute setting. For example,
if search.remote.node.attr is set to "gateway", only nodes that have
node.attr.gateway set to "true" can be connected to for cross-cluster
search. Unfortunately, there is a bug in the handling of these
attributes due to the use of a dangerous method
Boolean#getBoolean(String) which obtains the system property with
specified name as a boolean. We are not looking at system properties
here, but node settings. This commit fixes this situation, and adds a
test. A follow-up will ban the use of Boolean#getBoolean.

Relates #23863
2017-04-01 13:04:51 -04:00
Jim Ferenczi ee68e75332 FieldCapabilitiesRequest should implements Replaceable since it accepts index patterns 2017-03-31 20:21:06 +02:00
Alexander Reelsen f720767cbc Cleanup: Remove unused FieldMappers class (#23851)
This class is unused, so it can be removed.
2017-03-31 18:26:59 +02:00
Nik Everett ba62229f47 Fix FieldCapabilities compilation in Eclipse (#23855)
Eclipse can't deal with the generics, maybe the fixed but
unreleased https://bugs.eclipse.org/bugs/show_bug.cgi?id=511750
2017-03-31 12:10:15 -04:00
Tanguy Leroux 28099162ab Cluster stats should not render empty http/transport types (#23735)
This commit changes the ClusterStatsNodes.NetworkTypes so that is does
not print out empty field names when no Transport or HTTP type is defined:

```
{
"network_types": {
        ...
        "http_types": {
          "": 2
        }
      }
}
```

is now rendered as:

```
{
"network_types": {
        ...
        "http_types": {
        }
      }
}
```
2017-03-31 17:13:27 +02:00
Simon Willnauer 135eae42b9 Cleanup SearchPhaseController interface (#23844)
SearchPhaseController is tighly coupled to AtomicArray which makes
non-dense representations of results very difficult. This commit removes
the coupling and cuts over to Collection rather than List to ensure no
order or random access lookup is implied.
2017-03-31 16:25:15 +02:00
Jim Ferenczi a8250b26e7 Add FieldCapabilities (_field_caps) API (#23007)
This change introduces a new API called `_field_caps` that allows to retrieve the capabilities of specific fields.

Example:

````
GET t,s,v,w/_field_caps?fields=field1,field2
````
... returns:
````
{
   "fields": {
      "field1": {
         "string": {
            "searchable": true,
            "aggregatable": true
         }
      },
      "field2": {
         "keyword": {
            "searchable": false,
            "aggregatable": true,
            "non_searchable_indices": ["t"]
            "indices": ["t", "s"]
         },
         "long": {
            "searchable": true,
            "aggregatable": false,
            "non_aggregatable_indices": ["v"]
            "indices": ["v", "w"]
         }
      }
   }
}
````

In this example `field1` have the same type `text` across the requested indices `t`, `s`, `v`, `w`.
Conversely `field2` is defined with two conflicting types `keyword` and `long`.
Note that `_field_caps` does not treat this case as an error but rather return the list of unique types seen for this field.
2017-03-31 15:34:46 +02:00
Colin Goodheart-Smithe 9f66b8cd38 Improves disabled fielddata error message (#23841)
Closes #22768
2017-03-31 10:01:07 +01:00
Simon Willnauer 5badf68bd9 Add infrastructure to mark contexts as system contexts (#23830)
Today we have no way to mark an execution as internal. This commit adds
a simple thread context header that allows executing code in a system context.
This allows intercepting code can make better decisions down the road when
it gets to authentication.
2017-03-31 10:47:10 +02:00
Tim Brooks 5fa80a6521 Pass exception from sendMessage to listener (#23559)
This commit changes the listener passed to sendMessage from a Runnable
to a ActionListener.

This change also removes IOException from the sendMessage signature.
That signature is misleading as it allows implementers to assume an
exception will be thrown in case of failure. That does not happen due
to Netty's async nature.
2017-03-30 15:08:23 -05:00
Jason Tedor 48357e43d3 Honor update request timeout
When executing an update request, the request timeout is not transferred
to the index/delete request executed on behalf of the update
request. This leads to update requests not timing out when they should
(e.g., if not all shards are available when the request specifies
wait_for_shards=all with a small timeout). This commit causes the
index/delete requests to honor the update request timeout.

Relates #23825
2017-03-30 14:38:34 -04:00
Christoph Büscher b92371a4dc Tests: Add base tests for InternalSimpleValue and InternalDerivative (#23799)
As an addition to #22278 we should probably also have base tests for InternalSimpleValue
and InternalDerivative.
2017-03-30 20:23:49 +02:00
Simon Willnauer 4125f012b9 Streamline shard index availability in all SearchPhaseResults (#23788)
Today we have the shard target and the target request ID available in SearchPhaseResults.
Yet, the coordinating node maintains a shard index to reference the request, response tuples
internally which is also used in many other classes to reference back from fetch results to
query results etc. Today this shard index is implicitly passed via the index in AtomicArray
which causes an undesirable dependency on this interface.
This commit moves the shard index into the SearchPhaseResult and removes some dependencies
on AtomicArray. Further removals will follow in the future. The most important refactoring here
is the removal of AtomicArray.Entry which used to be created for every element in the atomic array
to maintain the shard index during result processing. This caused an unnecessary indirection, dependency
and potentially thousands of unnecessary objects in every search phase.
2017-03-30 14:32:42 +02:00
Jim Ferenczi 3b559e01be Fixed sliced search tests that rely on BytesRef.hashCode output 2017-03-30 10:14:41 +02:00
David Causse a49e1c0062 Use a fixed seed for computing term hashCode in TermsSliceQuery (#23795)
I think this query should not use the hashCode provided BytesRef#hashCode().
It uses StringHelper#GOOD_FAST_HASH_SEED which is initialized in a static
block to System.currentTimeMillis().
Running this query on different replicas may return inconsistent results.

Using a fixed seed should guaranty that the docs are sliced consistently
accross replicas.

Fixes #23096
2017-03-30 10:10:32 +02:00
Lee Hinman c8081bde91 Further refactor and extend testing for `TransportShardBulkAction`
This moves `updateReplicaRequest` to `createPrimaryResponse` and separates the
translog updating to be a separate function so that the function purpose is more
easily understood (and testable).

It also separates the logic for `MappingUpdatePerformer` into two functions,
`updateMappingsIfNeeded` and `verifyMappings` so they don't do too much in a
single function. This allows finer-grained error testing for when a mapping
fails to parse or be applied.

Finally, it separates parsing and version validation for
`executeIndexRequestOnReplica` into a separate
method (`prepareIndexOperationOnReplica`) and adds a test for it.

Relates to #23359
2017-03-29 10:56:51 -06:00
Jason Tedor 72824609df Add lower bound for translog generation threshold
The translog already occupies 43 bytes on disk when empty. If the
translog generation threshold is below this, the flush thread can get
stuck in an infinite loop repeatedly rolling the generation. This commit
adds a lower bound on the translog generation to avoid this problem,
however we keep the lower bound small for convenience in testing.

Relates #23779
2017-03-28 14:11:50 -04:00
Ali Beyad 2d3c2a4800 Adds backwards compatibility index and repository for v5.3.0 2017-03-28 14:02:07 -04:00
Ali Beyad c675d92a56 Adds v5.3.1 to the version constants 2017-03-28 13:04:28 -04:00
Ali Beyad 2120086d82 Adds pattern keyword marker filter support (#23600)
This commit adds support for the pattern keyword marker filter in
Lucene.  Previously, the keyword marker filter in Elasticsearch
supported specifying a keywords set or a path to a set of keywords.
This commit exposes the regular expression pattern based keyword marker
filter also available in Lucene, so that any token matching the pattern
specified by the `keywords_pattern` setting is excluded from being
stemmed by any stemming filters.

Closes #4877
2017-03-28 11:13:34 -04:00
Dimitris Athanasiou 34f116eae3 Require explicit query in _delete_by_query API (#23632)
As the query of a search request defaults to match_all,
calling _delete_by_query without an explicit query may
result in deleting all data.

In order to protect users against falling into that
pitfall, this commit adds a check to require the explicit
setting of a query.

Closes #23629
2017-03-28 15:44:57 +01:00
Ali Beyad 8359dd05c9 Adds boolean similarity to Elasticsearch (#23637)
This commit adds the boolean similarity scoring from Lucene to
Elasticsearch.  The boolean similarity provides a means to specify that
a field should not be scored with typical full-text ranking algorithms,
but rather just whether the query terms match the document or not.
Boolean similarity scores a query term equal to its query boost only.
Boolean similarity is available as a default similarity option and thus
a field can be specified to have boolean similarity by declaring in its
mapping:
    "similarity": "boolean"

Closes #6731
2017-03-28 10:17:23 -04:00
Stuart Neivandt 3caf887632 Improve error handling for epoch format parser with time zone (#23689)
Change the error response when using a non UTF timezone for range queries with epoch_millis
or epoch_second formats to an illegal argument exception. The goal is to provide a better 
explanation of why the query has failed. The current behavior is to respond with a parse exception.

Closes #22621
2017-03-28 14:45:20 +02:00
Jason Tedor 742d929b56 Validate top-level keys when parsing mget requests
Today, when parsing mget requests, we silently ignore keys in the top
level that do not match "docs" or "ids". This commit addresses this
situation by throwing an exception if any other key occurs here, and
providing the names of valid keys.

Relates #23746
2017-03-28 08:27:31 -04:00
Jason Tedor 4f2dfb6819 Fix serialization for plugin info
This commit fixes the serialization for plugin info. Namely, the
serialization incorrectly specified the backwards compatibility version
as strictly after version 5.4.0, whereas it should be on or after
version 5.4.0.
2017-03-27 21:04:31 -04:00
Jason Tedor b54a9e9c83 Introduce translog generation rolling
This commit introduces a maximum size for a translog generation and
automatically rolls the translog when a generation exceeds the threshold
into a new generation. This threshold is configurable per index and
defaults to sixty-four megabytes. We introduce this constraint as
sequence numbers will require keeping around more than the current
generation (to ensure that we can rollback to the global
checkpoint). Without keeping the size of generations under control,
having to keep old generations around could consume excessive disk
space. A follow-up will enable commits to trim previous generations
based on the global checkpoint.

Relates #23606
2017-03-27 16:43:54 -04:00
Jason Tedor defd0452e7 Modify permissions dialog for plugins
This commit modifies the handling of plugins that require special
permissions to cover a case that was not previously covered.

Relates #23742
2017-03-27 15:52:45 -04:00
Christoph Büscher fc8cb417e7 FuzzyQueryBuilder should error when parsing array of values (#23762)
Closes #23759
2017-03-27 17:02:01 +02:00
Marios Trivyzas 4f694a3312 Remove obsolete index setting `index.version.minimum_compatible`. (#23593) 2017-03-27 15:59:48 +02:00
Jim Ferenczi 0e95c90e9f Upgrade to Lucene 6.5.0 (#23750) 2017-03-27 15:57:54 +02:00
Jason Tedor a6c4234575 Add early-access check
The OpenJDK project provides early-access builds of upcoming
releases. These early-access builds are not suitable for
production. These builds sometimes end up on systems due to aggressive
packaging (e.g., Ubuntu). This commit adds a bootstrap check to ensure
these early-access builds are not being used in production.

Relates #23743
2017-03-24 14:52:50 -04:00
Christoph Büscher 396785ccb1 Tests: Lower expected precision for InternalAvgTests
Closes #23723
2017-03-24 12:18:07 +01:00
Christoph Büscher e7a8e69900 Test: Check that parsing SearchHit without _type/_id works (#23715)
The hit object can be very small e.g. when using "stored_fields": ["_none_"],
this adds a test that checks that we can still parse back the object.

* also check type/id null
2017-03-24 10:14:52 +01:00
Luca Cavanna c379b9bdc8 Use ParseField for aggs CommonFields rather than String (#23717)
With this change we remove a TODO from CommonFields. Also this will be useful when parsing aggs response for the high level REST client.
2017-03-23 17:20:40 +01:00
AdityaJNair 63757efe9c Remove DocumentMapper#parse(String index, String type, String id, BytesReference source) (#23706)
Removed `parse(String index, String type, String id, BytesReference source)` in DocumentMapper.java and replaced all of its use in Test files with `parse(SourceToParse source)`.

`parse(String index, String type, String id, BytesReference source)` was only used in test files and never in the main code so it was removed. All of the test files that used it was then modified to use `parse(SourceToParse source)` method that existing in DocumentMapper.java
2017-03-23 11:01:09 -04:00
Boaz Leskes 6577503b37 AbstractSearchAsyncAction: fix potential NPE in debug logging 2017-03-23 09:00:38 +01:00
Ali Beyad 2df39689fc Fixes snapshot deletion handling on in-progress snapshot failure (#23703)
This commit fixes an issue manifested in the
SharedClusterSnapshotRestoreIT#testGetSnapshotsRequest where a delete
request on a snapshot encounters an in-progress snapshot, so it first
tries to abort the snapshot.  During the aborting process, an exception
is thrown which is handled by the snapshot listener's onSnapshotFailure
method.  This method retries the delete snapshot request, only to
encounter that the snapshot is missing, throwing an exception.  It is
possible that the snapshot failure resulted in the snapshot never having
been written to the repository, and hence, there is nothing to delete.
This commit handles the SnapshotMissingException by logging it and
notifying the listener of the missing snapshot.

Closes #23663
2017-03-22 21:02:23 -04:00
Igor Motov f927a2708d Make it possible to validate a query on all shards instead of a single random shard (#23697)
This is especially useful when we rewrite the query because the result of the rewrite can be very different on different shards. See #18254 for example.
2017-03-22 17:39:21 -04:00
Stefan Gorgiovski 798c19dd7f Deprecate request_cache for clear-cache (#23638)
It is called `request` now.
2017-03-22 08:28:04 -04:00
Luca Cavanna c6b881b53e Share XContent rendering code in terms aggs (#23680)
The output of the different implementations of terms aggs is always very similar. The toXContent methods for each of those classes though was duplicating almost the same code multiple times. This commit centralizes the code for rendering XContent to a single place, which can be reused from the different terms aggs implementations.
2017-03-22 12:28:13 +01:00
Ryan Ernst b31ed6a75c Plugins: Add plugin cli specific exit codes (#23599)
We currently use POSIX exit codes in all of our CLIs. However, posix
only suggests these exit codes are standard across tools. It does not
prescribe particular uses for codes outside of that range. This commit
adds 2 exit codes specific to plugin installation to make distinguishing
an incorrectly built plugin and a plugin already existing easier.

closes #15295
2017-03-21 13:56:00 -07:00
Ryan Ernst 111e703cde Plugins: Output better error message when existing plugin is incompatible (#23562)
This commit catches the underlying failure when trying to list plugin
information when a plugin is incompatible with the current version of
elasticsearch. This could happen when elasticsearch is upgraded but old
plugins still exist. With this change, all plugins will be output,
instead of failing at the first out of date plugin.

closes #20691
2017-03-21 13:45:27 -07:00
Nik Everett bc65be2a65 Reindex: wait for cleanup before responding (#23677)
Changes reindex and friends to wait until the entire request has
been "cleaned up" before responding. "Clean up" in this context
is clearing the scroll and (for reindex-from-remote) shutting
down the client. Failures to clean up are still only logged, not
returned to the user.

Closes #23653
2017-03-21 15:33:39 -04:00
Ryan Ernst f8453aca57 Packaging: Remove classpath ordering hack (#23596)
After the removal of the joda time hack we used to have, we can cleanup
the codebase handling in security, jarhell and plugins to be more picky
about uniqueness. This was originally in #18959 which was never merged.

closes #18959
2017-03-21 12:12:16 -07:00
Ali Beyad e72d287382 [TEST] Propertly cleans up failing restore test
The SharedClusterSnapshotRestoreIT#testDataFileCorruptionDuringRestore
would fail sporadically because it tried to simulate restoring a
corrupted index.  The test would wait until the restore is finished (and
marked as failed) before exiting.  However, in the background, the
cluster still continues to retry allocation of the failed shards,
despite the restore operation being marked as completed, which in turn
generates cluster states to process.  The end of every ESIntegTestCase
verifies that none of the nodes have any pending cluster states to
process.  Hence, this check sometimes fails on this particular test.

This commit solves the issue by ensuring the index is deleted before
exiting the test.
2017-03-21 14:02:42 -04:00
Christoph Büscher 889f0cbc40 Add unit tests for ReverseNestedAggregator (#23651)
Relates to #22278
2017-03-21 13:11:25 +01:00
Jason Tedor 7b17689458 Search took time should use a relative clock
Search took time uses an absolute clock to measure elapsed time, and
then tries to deal with the complexities of using an absolute clock for
this purpose. Instead, we should use a high-precision monotonic relative
clock that is designed exactly for measuring elapsed time. This commit
modifies the search infrastructure to use a relative clock for measuring
took time, but still provides an absolute clock for the components of
search that require a real clock (e.g., index name expression
resolution, etc.).

Relates #23662
2017-03-20 18:48:51 -04:00
Ali Beyad ce08594008 Adds toString() to snapshot operations in progress
A better toString() is added for snapshot operations in progress in the
cluster state and logging has been increased to help debug
SharedClusterSnapshotRestoreIT tests.
2017-03-20 16:45:23 -04:00
Nikiforos Botis e8b915d010 Comment and blank line cleanups (#23647) 2017-03-20 09:36:33 -04:00
Jordan Kiang d010cad503 Fix MapperService StackOverflowError (#23605)
MapperService#parentTypes is rewrapped in an UnmodifiableSet in MapperService#internalMerge every time the cluster state is updated. After thousands of updates the collection is wrapped so deeply that calling a method on it generates a StackOverflowError.

Closes #23604
2017-03-20 03:53:35 -07:00
Jason Tedor 2eafe8310e Format RemovePluginCommand to 100-column limit
This commit formats RemovePluginCommand.java to the 100-column limit and
removes this file from the list of suppressions.
2017-03-19 22:50:13 -04:00
Alex Lattas f9d6924f7d Add Javadocs for RemovePluginCommand#execute
This commit adds Javadocs for RemovePluginCommand#execute, the actual
implementation of the remove plugin command.

Relates #23644
2017-03-19 22:11:50 -04:00
Jason Tedor b23adb6d15 Avoid overflow when computing total FS stats
When adding filesystem stats from individual filesystems, free and
available can overflow. This commit guards against this by adjusting
these situations to Long.MAX_VALUE.

Relates #23641
2017-03-18 20:02:18 -04:00
Jason Tedor 44d75db9e2 Upgrade from JNA 4.2.2 to JNA 4.4.0
This commit upgrades the JNA dependency from version 4.2.2 to version
4.4.0.

Relates #23636
2017-03-17 21:06:16 -04:00
Igor Motov 1bd66136d7 Task Manager should be able to support non-transport tasks (#23619)
Currently the task manager is tied to the transport and can only create tasks based on TransportRequests. This commit enables task manager to support tasks created by non-transport services such as the persistent tasks service.
2017-03-17 19:29:18 -04:00
Jim Ferenczi b8c352fc3f Add support for fragment_length in the unified highlighter (#23431)
* Add support for fragment_length in the unified highlighter

This commit introduce a new break iterator (a BoundedBreakIterator) designed for the unified highlighter
 that is able to limit the size of fragments produced by generic break iterator like `sentence`.
The `unified` highlighter now supports `boundary_scanner` which can `words` or `sentence`.
The `sentence` mode will use the bounded break iterator in order to limit the size of the sentence to `fragment_length`.
When sentences bigger than `fragment_length` are produced, this mode will break the sentence at the next word boundary **after**
 `fragment_length` is reached.
2017-03-17 18:10:13 +01:00
Jason Tedor c462d7d486 Clear the interrupt flag before joining
This commit changes the method for checking the interrupt status of a
thread that is intentionally interrupted during
AdapterActionFutureTests#testInteruption. Namely, we want to check and
clear the interrupt status before joining on the interrupting thread. If
we do not clear the status, when we lose a race where the interrupting
thread is not yet finished, an interrupted exception will be thrown when
we try to join on it. Clearing the interrupted status on the main thread
addresses this issue.
2017-03-17 12:46:23 -04:00
Jason Tedor 90929f77ca Adapter action future should restore interrupts
When a thread blocking on an adapter action future is interrupted, we
throw an illegal state exception. This is documented, but it is rude to
not restore the interrupt flag. This commit restores the interrupt flag
in this situation, and adds a test.

Relates #23618
2017-03-17 01:05:48 -04:00
Christoph Büscher 96a92da682 CompletionSuggestionContext#toQuery() should also consider text if prefix/regex missing (#23451)
In cases where the user specifies only the `text` option on the top level
suggest element (either via REST or the java api), this gets transferred to the
`text` property in the SuggestionSearchContext. CompletionSuggestionContext
currently requires prefix or regex to be specified, otherwise errors. We should
use the global `text` property as a fallback if neither prefix nor regex is provided.

Closes to #23340
2017-03-16 21:36:18 +01:00
Ryan Ernst cb16ed1e26 Fix num docs to be positive in bucket deferring collector test 2017-03-15 15:43:07 -07:00
Ryan Ernst d808e751f4 Mapping: Fix NPE with scaled floats stats when field is not indexed (#23528)
This fixes an NPE in finding scaled float stats. The type of min/max
methods on the wrapped long stats returns a boxed type, but in the case
this is null, the unbox done for the FieldStats.Double ctor primitive
types will cause the NPE. These methods would have null for min/max when
the field exists, but does not actually have points values.

fixes #23487
2017-03-15 15:14:32 -07:00
Jason Tedor f7b8128f92 Enable explicitly enforcing bootstrap checks
This commit adds a system property that enables end-users to explicitly
enforce the bootstrap checks, independently of the binding of the
transport protocol. This can be useful for single-node production
systems that do not bind the transport protocol (and thus the bootstrap
checks would not be enforced).

Relates #23585
2017-03-15 10:36:17 -07:00
Sönke Liebau 326d6456fe Minor spelling corrections in documentation of ClusterState (#23592) 2017-03-15 10:20:18 -07:00
Boaz Leskes c0cafa786b UnicastZenPing shouldn't ping the address of the local node (#23567)
Pinging the local node address doesn't really add to discovering other nodes. It just pollutes the logs with unneeded information.
2017-03-14 07:02:42 -07:00
Jay Modi 3200da0327 Provide a method to retrieve a closeable char[] from a SecureString (#23389)
This change adds a new method that returns the underlying char[] of a SecureString and the ability
to clone the SecureString so that the original SecureString is not vulnerable to modification.
Closing the cloned SecureString will wipe the char[] that backs the clone but the original SecureString remains unaffected.

Additionally, while making a separate change I found that SecureSettings will fail when diff is called on them and there
is no fallback setting. Given the idea behind SecureSetting, I think that diff should just be a no-op and I have
implemented this here as well.
2017-03-13 19:50:55 -07:00
Christoph Büscher a8117a2d77 Tests: fix GeoHashGridAggregatorTests expectations (#23556)
Currently GeoHashGridAggregatorTests#testWithSeveralDocs increases the expected
document count per hash for each geo point added to a document. When points
added to the same doc fall into one bucket (one hash cell) the document should
only be counted once.

Closes #23555
2017-03-13 09:54:50 -07:00
Christoph Büscher 21dcd4f4ca Tests: Check that GetResponse.toString() outputs json xcontent (#23545) 2017-03-13 09:54:29 -07:00
Martijn van Groningen 78a48b102f
[INNER HITS] Changed DisMaxQueryBuilder to extract inner hits from leaf queries.
Closes #23482
2017-03-12 16:21:03 -07:00
Martijn van Groningen b01070a390
[TEST] Added unit tests for diversified sampler aggregator. 2017-03-12 16:14:47 -07:00
Jason Tedor c51ef0b2ca Honor max concurrent searches in multi-search
A previous change to the multi-search request execution to avoid stack
overflows regressed on limiting the number of concurrent search requests
from a batched multi-search request. In particular, the replacement of
the tail-recursive call with a loop could asynchronously fire off all of
the remaining search requests in the batch while max concurrent search
requests are already executing. This commit attempts to address this
issue by taking a more careful approach to the initial problem of
recurisve calls. The cause of the initial problem was due to possibility
of individual requests completing on the same thread as invoked the
search action execution. This can happen, for example, in cases when an
individual request does not resolve to any shards. To address this
problem, when an individual request completes we check if it completed
on the same thread as fired off the request. In this case, we loop and
otherwise safely recurse. Sadly, there was a unit test to check that the
maximum number of concurrent search requests was not exceeded, but that
test was broken while modifying the test to reproduce a case that led to
the possibility of stack overflow. As such, we randomize whether or not
search actions execute on the same thread as the thread that invoked the
action.

Relates #23538
2017-03-12 00:45:40 -08:00
Jason Tedor 2a26ae1d6a Fall back to non-atomic move when removing plugins
When plugins are installed on a union filesystem (for example, inside a
Docker container), removing them can fail because we attempt an atomic
move which will not work if the plugin is not installed in the top
layer. This commit modifies removing a plugin to fall back to a
non-atomic move in cases when the underlying filesystem does not support
atomic moves.

Relates #23548
2017-03-11 19:46:01 -08:00
Jason Tedor 3d82549d8e Avoid stack overflow in multi-search
Today when handling a multi-search request, we asynchornously execute as
many search requests as the minimum of the number of search requests in
the multi-search request and the maximum number of concurrent
requests. When these search requests return, we poll more search
requests from a queue of search requests from the original multi-search
request. The implementation of this was recursive, and if the number of
requests in the multi-search request was large, a stack overflow could
arise due to the recursive invocation. This commit replaces this
recursive implementation with a simple iterative implementation.

Relates #23527
2017-03-09 15:19:05 -08:00
Ryan Ernst 9488985775 Test: Upgrade randomized runner to 2.5.0 (#23513)
This commit upgrades to the newest version of randomized runner. There
is a new additional check that allows ensuring the working directory
for each child jvm is empty. By default, this check will fail the test
run. However, for elasticsearch, we default to wipe the directory. For
example, if you previously told the runner to not wipe the directory, in
order to investigate a failure, the wipe option will delete this data
upon re-running the test.
2017-03-09 11:56:43 -08:00
Jason Tedor ae6331f27e Handle existence of cgroup version 2 hierarchy
When parsing the control groups to which the Elasticsearch process
belongs, we extract a map from subsystems to paths by parsing
/proc/self/cgroup. This file contains colon-delimited entries of the
form hierarchy-ID:subsystem-list:cgroup-path. For control group version
1 hierarchies, the subsystem-list is a comma-delimited list of the
subsystems for that hierarchy. For control group version 2 hierarchies
(which can only exist on Linux kernels since version 4.5), the
subsystem-list is an empty string. The previous parsing of
/proc/self/cgroup incorrectly accounted for this possibility (a +
instead of a * in a regular expression). This commit addresses this
issue, adds a test case that covers this possibility, and simplifies the
code that parses /proc/self/cgroup.

Relates #23493
2017-03-06 14:12:26 -08:00
Tanguy Leroux 1ceb6d04c7 [Test] Fix BulkResponseTests 2017-03-03 09:34:10 +01:00
Jay Modi 01502893eb HTTP transport stashes the ThreadContext instead of the RestController (#23456)
Previously, the RestController would stash the context prior to copying headers. However, there could be deprecation
log messages logged and in turn warning headers being added to the context prior to the stashing of the context. These
headers in the context would then be removed from the request and also leaked back into the calling thread's context.

This change moves the stashing of the context to the HttpTransport so that the network threads' context isn't
accidentally populated with warning headers and to ensure the headers added early on in the RestController are not
excluded from the response.
2017-03-02 14:44:01 -05:00
Ali Beyad 577d2a6a1d Adds cluster state size to /_cluster/state response (#23440)
This commit adds the size of the cluster state to the response for the
get cluster state API call (GET /_cluster/state).  The size that is
returned is the size of the full cluster state in bytes when compressed.
This is the same size of the full cluster state when serialized to
transmit over the network.  Specifying the ?human flag displays the
compressed size in a more human friendly manner.  Note that even if the
cluster state request filters items from the cluster state (so a subset
of the cluster state is returned), the size that is returned is the
compressed size of the entire cluster state.

Closes #3415
2017-03-02 14:20:29 -05:00
Christoph Büscher d02b6f58fa Tests: Adapt ExistsQueryBuilderTests to changes in ExistQueryBuilder#toQuery() (#23462)
Recent changes in the Lucene query that the ExistsQueryBuilder creates broke
this test.
2017-03-02 18:27:30 +01:00
Kunal Kapoor 32d292b3c2 Added types options to DeleteByQueryRequest (#23265)
Add types setter and getter to `DeleteByQueryRequest`, which delegate to the inner `SearchRequest`.
2017-03-02 16:52:21 +01:00
Jim Ferenczi 6519e1207c Fix query_string_query to transform "foo:*" in an exists query on the field name (#23433)
Currently "foo:*" is parsed as prefix query on the field `foo` unless the field is defined in `default_field` or `fields`.
This commit fixes this behavior, "foo:*" is now rewritten to an exists query on the field name.
This change also removes the assumption that "_all:*" should return all docs.

relates #23356
2017-03-02 16:26:27 +01:00
Tanguy Leroux 15c936ec02 Correctly parse BulkItemResponse.Failure's status (#23432)
Today the status is lost when parsing back a BulkItemResponse.Failure. This commit changes the BulkItemResponse.Failure parsing method so that it correctly instantiates a failure with the parsed status instead of realying on the parsed ElasticsearchException (that always return an internal server error status).
2017-03-02 12:37:07 +01:00
Jim Ferenczi 22870cc6c0 Fix centroids equality tests in TDigestState#equals 2017-03-02 11:27:04 +01:00
Tanguy Leroux 5a668c4add Tests: Add unit test for SignificantLongTerms and SignificantStringTerms (#23428)
Relates to #22278
2017-03-02 10:48:29 +01:00
Jim Ferenczi 1228084c1c Fix tests on InternalAggregation that rely on equals/hashCode. 2017-03-02 10:38:16 +01:00
Martijn van Groningen 1d3f6c463c
[INGEST] Added processor type and tag header to error when processor type isn't available on node.
[INGEST] Accumulate any potential other processor parse errors before failing instead of failing upon first processor parsing error.
2017-03-02 08:37:03 +01:00
Nik Everett a54daade33 Tests InternalSingleBucketAggregation subclasses (#23388)
Adds a common base class for testing subclasses of
`InternalSingleBucketAggregation`. They are so similar they
call into question the utility of having all of these classes.
We maybe could just use `InternalSingleBucketAggregation` in
all those cases.... But for now, let's test the classes!

Relates to #22278
2017-03-01 16:50:52 -05:00
Tanguy Leroux e71d9c1960 Tests: Add unit test for InternalDateHistogram (#23402)
Relates to #22278
2017-03-01 16:08:16 +01:00
Adrien Grand 64c90346c6 Avoid adding unnecessary nested filters when ranges are used. (#23427)
The code was testing `PointRangeQuery` however we now use the
`IndexOrDocValuesQuery` in field mappers. This makes the test generate queries
through mappers so that we test the actual queries that would be generated.
2017-03-01 14:34:59 +01:00
Adrien Grand 3134d6b520 Add unit tests to percentile ranks aggregations. (#23240)
Relates #22278
2017-03-01 13:57:40 +01:00
Yannick Welsch c7edaba4e8 [TEST] Fix race condition when blocking cluster state processing during primary relocation
Two tests were periodically failing. What both tests are doing is starting a relocation of a shard from one node to another. Once the
recovery process is started, the test blocks cluster state processing on the relocation target using the BlockClusterStateProcessing disruption. The test then indefinitely
waits for the relocation to complete. The stack dump shows that the relocation is stuck in the PeerRecoveryTargetService.waitForClusterState method, waiting for the relocation target node to have at least the same cluster state
version as the relocation source.

The reason why it gets stuck is the following race:
1) The test code executes a reroute command that relocates a shard from one node to another
2) Relocation target node starts applying the clusterstate with relocation info, starting the recovery process.
4) Recovery is super fast and quickly goes to the waitForClusterState method, which wants to ensure that the cluster state that is
applied on the relocation target is at least as new as the one on the relocation source. The relocation source has already applied the
cluster state but the relocation target is still in the process of applying it. The waitForClusterState method thus uses a
ClusterObserver to wait for the next cluster state. Internally this means submitting a task with priority HIGH to the cluster service.
5) Cluster state application is about to finish on the relocation target. As one of the last steps, it acks to the master which makes the
reroute command return successfully.
6) The test code then blocks cluster state processing on the relocation target by submitting a cluster state update task (with priority
IMMEDIATE) that blocks execution.

If the task that is submitted in step 6 is handled before the one in step 4 by ClusterService's thread executor, cluster state
processing becomes blocked and prevents the cluster state observer from observing the applied cluster state.
2017-03-01 11:55:48 +01:00
Martijn van Groningen 524d7f592d
[TEST] Added unit tests for GeoHashGridAggregator and InternalGeoHashGrid
Part of #22278
2017-03-01 10:33:57 +01:00
Adrien Grand b388389ada Remove support for the include/pattern syntax. (#23141)
Relates #22933
2017-03-01 10:00:38 +01:00
Lee Hinman fd991f32f9 Refactor TransportShardBulkAction and add unit tests
This refactors the `TransportShardBulkAction` to split it appart and make it
unit-testable, and then it also adds unit tests that use these methods.

In particular, this makes `executeBulkItemRequest` shorter and more readable
2017-02-28 19:53:18 -07:00
Jason Tedor 7ce06aeb8c Fix date format in warning headers
This commit fixes the date format in warning headers. There is some
confusion around whether or not RFC 1123 requires two-digit
days. However, the warning header specification very clearly relies on a
format that requires two-digit days. This commit removes the usage of
RFC 1123 date/time format from Java 8, which allows for one-digit days,
in favor of a format that forces two-digit days (it's otherwise
identical to RFC 1123 format, it is just fixed width).

Relates #23418
2017-02-28 20:28:07 -05:00
Ryan Ernst 019263d664 Revert "Internal: Change version constant names for already released versions (#23416)"
This reverts commit dc0e93ed62.
2017-02-28 14:45:13 -08:00
Ryan Ernst dc0e93ed62 Internal: Change version constant names for already released versions (#23416)
We have many version constants in master that have already been
released, but are still marked (by naming convention) as unreleased.
This commit renames those version constants.
2017-02-28 13:05:44 -08:00
Christoph Büscher 98b7023318 Fix checkstyle LineLength issues in UpdateShardAllocationSettingsIT 2017-02-28 20:14:28 +01:00
David Pilato 0ae6bc36fa Create version constants for next bug fix version v5.2.3 2017-02-28 19:11:48 +01:00
Ali Beyad b58eb5f051 [TEST] removes unused SameShardRouting test 2017-02-28 12:58:27 -05:00
Ali Beyad 5e2e45cad9 Makes the same_shard host dyanamically updatable (#23397)
Previously, cluster.routing.allocation.same_shard.host was not a dynamic
setting and could not be updated after startup.  This commit changes the
behavior to allow the setting to be dynamically updatable.  The
documentation already states that the setting is dynamic so no
documentation changes are required.

Closes #22992
2017-02-28 12:48:54 -05:00
Christoph Büscher 5be7f6a76f Tests: fixing line length limit in ScriptedMetricAggregatorTests 2017-02-28 16:56:43 +01:00
Christoph Büscher a522deb6b5 Tests: Add unit test for InternalScriptedMetricAggregator (#23404)
Relates to #22278
2017-02-28 16:43:12 +01:00
Colin Goodheart-Smithe 406d2f7a64
Fixes the per term error in the terms aggregation
When multiple reduce phases were needed the per term error got lost in subsequent reduces in some situations:

When a previous reduce phase had calculated a non-zero error for a particular bucket we were not accounting for this error in subsequent reduce phases and instead were relying on the overall error for the agg which meant we were implicitly assuming that all shards that made up that aggregation had returned the term. This is plainly not true so we need to make sure the per term error for the aggregation is used when calcualting the error for that term in the new reduced aggregation.
2017-02-28 13:56:35 +00:00
Jim Ferenczi 17a0b4e69c Fix merge scheduler test that depends on the number of processors 2017-02-28 11:43:35 +01:00
Jim Ferenczi 0c03d0056c #23391: simplify setting fallback (missing in the squash commit) 2017-02-28 11:34:02 +01:00
Jim Ferenczi d27a55866c Fix merge scheduler config settings (#23391)
Change Setting#get(Settings, Settings) to fallback only if the setting is present in the secondary.
This is needed to fix setting that relies on other settings.
Replace IT with uts.
2017-02-28 10:21:27 +01:00
Christoph Büscher 084cb38207 Tests: Add unit test for InternalScriptedMetric (#23330)
Relates to #22278
2017-02-28 09:40:11 +01:00
Jason Tedor c61cc5f617 Fix deprecation escaping tests
This commit fixes an off-by-one error in the deprecation escaping tests.
2017-02-27 17:19:13 -05:00
Martijn van Groningen 73fb945980
Changed TaskOperationFailure#getCause() return type from Trowable to Exception. 2017-02-27 21:31:19 +01:00
Jim Ferenczi 5c84640126 Upgrade to lucene-6.5.0-snapshot-d00c5ca (#23385)
Lucene upgrade
2017-02-27 18:39:04 +01:00
Jason Tedor 577e6a5e14 Correct warning header to be compliant
The warning header used by Elasticsearch for delivering deprecation
warnings has a specific format (RFC 7234, section 5.5). The format
specifies that the warning header should be of the form

    warn-code warn-agent warn-text [warn-date]

Here, the warn-code is a three-digit code which communicates various
meanings. The warn-agent is a string used to identify the source of the
warning (either a host:port combination, or some other identifier). The
warn-text is quoted string which conveys the semantic meaning of the
warning. The warn-date is an optional quoted date that can be in a few
different formats.

This commit corrects the warning header within Elasticsearch to follow
this specification. We use the warn-code 299 which means a
"miscellaneous persistent warning." For the warn-agent, we use the
version of Elasticsearch that produced the warning. The warn-text is
unchanged from what we deliver today, but is wrapped in quotes as
specified (this is important as a problem that exists today is that
multiple warnings can not be split by comma to obtain the individual
warnings as the warnings might themselves contain commas). For the
warn-date, we use the RFC 1123 format.

Relates #23275
2017-02-27 12:14:21 -05:00
Luca Cavanna 2fb0466f66 Convert suggestion response parsing to use NamedXContentRegistry (#23355)
We recently added parsing code to parse suggesters responses into java api objects. This was done using a switch based on the type of the returned suggestion. We can now replace the switch with using NamedXContentRegistry, which will also be used for aggs parsing.
2017-02-27 15:42:25 +01:00
Colin Goodheart-Smithe 1ceaef0de6
Fixes terms error count for multiple reduce phases
Previously when multiple reduces occured for the terms aggregation we would add up the errors for the aggregations but would not take into account the errors that had already been calculated for the previous reduce phases.

This change corrects that by adding the previously created errors to the new error value.

Closes #23286
2017-02-27 13:44:18 +00:00
javanna 261f31f5b7 [TEST] move filters aggs wrapper query builder rewriting test to integ tests
This test makes little sense when sent from the REST layer, as WrapperQueryBuilder is supposed to be used from the Java api. Also, providing the inner query as base64 string will work only for string formats and break for binary formats like SMILE and CBOR, whcih doesn't play well with randomizing content type in our REST tests
2017-02-27 12:27:03 +01:00
Simon Willnauer b8e2d12b23 Factor out filling of TopDocs in SearchPhaseController (#23380)
Previously this code was duplicated across the 3 different topdocs variants
we have. It also had no real unittest (where we tested with holes in the results)
which caused a sneaky bug where the comparison used `result.size()` vs `results.size()`
causing several NPEs downstream. This change adds a static method to fill the top docs
that is shared across all variants and adds a unittest that would have caught the issue
very quickly.

Closes #19356
Closes #23357
2017-02-27 11:44:41 +01:00
Christoph Büscher 641c88dc29 Prevent negative `from` parameter in SearchSourceBuilder (#23358)
This prevents later errors like the one reported in #23324 and throws an
IllegalArgumentException early instead.
2017-02-27 09:45:10 +01:00
Boaz Leskes 396b8b371c reduce the number of iterations in testPrimaryRelocationWhileIndexing and flush every 5
Without flushing the translog doubles it's size on every recovery
2017-02-26 19:15:33 +01:00
Boaz Leskes 9088ddd09c rollback unneeded change in testNotifyOnDisconnect 2017-02-26 12:30:57 +01:00
Boaz Leskes aa49ba949c disable sampling in testNotifyOnDisconnect
the background activity confuses the test
2017-02-26 12:20:34 +01:00
Ryan Ernst 48548f6c3d CLI: Fix prompting for yes/no to handle console returning null (#23320)
Console.readText may return null in certain cases. This commit fixes a
bug in Terminal.promptYesNo which assumed a non-null return value.  It
also adds a test for this, and modifies mock terminal to be able to
handle null input values.
2017-02-24 20:20:17 -08:00
Ali Beyad 9a9259184a Handle snapshot repository's missing index.latest
If index.latest does not exist, the repository is on an older
version so simply return the empty repository's generation id.
2017-02-24 10:14:22 -05:00
Christoph Büscher 9270f8a873 Adding equals/hashCode to MainResponse (#23352) 2017-02-24 14:58:03 +01:00
Jay Modi 5490cb52b0 Always restore the ThreadContext for operations delayed due to a block (#23349)
The IndexShardOperationsLock has a mechanism to delay operations if there is currently a block on the lock. These
delayed operations are executed when the block is released and are executed by a different thread. When the different
thread executes the operations, the ThreadContext is that of the thread that was blocking operations. In order to
preserve the ThreadContext, we need to store it and wrap the listener when the operation is delayed.
2017-02-24 08:21:04 -05:00
Christoph Büscher b9eb1bba65 Add unit tests for ParentToChildAggregator (#23305)
Adds unit tests for the `children` aggregation.
This change also add the ability to mock Mapperservice in subtests of
AggregatorTestCase.
2017-02-24 10:29:31 +01:00
Jim Ferenczi 63bdd01eb7 Expose WordDelimiterGraphTokenFilter (#23327)
This change exposes the new Lucene graph based word delimiter token filter in the analysis filters.
Unlike the `word_delimiter` this token filter named `word_delimiter_graph` correctly handles multi terms expansion at query time.

Closes #23104
2017-02-24 00:53:38 +01:00
Shai Erera eeac6d27f2 Add BreakIteratorBoundaryScanner support for FVH (#23248)
This commit adds a boundary_scanner property to the search highlight
request so the user can specify different boundary scanners:

* `chars` (default,  current behavior)
* `word` Use a WordBreakIterator
* `sentence` Use a SentenceBreakIterator

This commit also adds "boundary_scanner_locale" to define which locale
should be used when scanning the text.
2017-02-23 23:32:22 +01:00
Ali Beyad 25a9a7ee3a Prioritize listing index-N blobs over index.latest in reading snapshots (#23333)
There are two ways to determine the latest index-N blob that contains
the truth of the contents of the repository: (1) list all index-N blobs
and figure out what the latest value of N is, and (2) read the
index.latest blob, which contains the latest value of N explicitely.
Note that the index.latest blob is not written atomically and can be
re-written, as opposed to the index-N blobs which are never re-written
(to create an updated index blob, index-{N+1} is written).

Previously, the latest index-N was determined by first trying to read
the index.latest blob and if that blob was missing (it was deleted
before being re-written and in between deleting it and re-writing it,
the system crashed), then all index-N blobs were listed to pick the
highest N value.

For non-read-only repositories, this could produce race conditions with
the file system.  In particular, it is possible that the index.latest
blob is being read in order to serve a read request (e.g. get snapshots)
and while doing so, an attempt is made to delete the index.latest blob
and re-write it in order to finalize a snapshot operation.  On some file
systems (e.g. Windows), it is forbidden to delete a file while it is
open for reading by another process/thread.

This commit changes the priority so that figuring out the latest index-N
blob is first done by listing all index-N blobs and determining the
latest N value.  If that values because the repository does not
support listing blobs (e.g. the URL repository), then the index.latest
blob is read.  This is safe because in read-only repositories that do
not support listing blobs, the index.latest blob is never deleted and
then re-written, so the aforementioned issue does not arise.
2017-02-23 15:44:12 -05:00
sabi0 09b3c7f270 Do not create String instances in 'Strings' methods accepting StringBuilder (#22907) 2017-02-23 10:57:34 -08:00
Christoph Büscher 8b1b152e91 Remove abstract InternalMetricsAggregation class (#23326)
This class doesn't seem to do much other than to group together
certain types of aggregations.
2017-02-23 18:03:40 +01:00
Simon Willnauer 2f3f9b9961 Remove unnecessary result sorting in SearchPhaseController (#23321)
In oder to use lucene's utilities to merge top docs the results
need to be passed in a dense array where the index corresponds to the shard index in
the result list. Yet, we were sorting results before merging them just to order them
in the incoming order again for the above mentioned reason. This change removes the
obsolet sort and prevents unnecessary materializing of results.
2017-02-23 13:48:54 +01:00
Simon Willnauer 771fd1f4ea Fix SamplerAggregatorTests to have stable and predictable docIds
Closes #23315
2017-02-23 08:08:38 +01:00
Ryan Ernst 18f57c05cf Script: Fix value of `ctx._now` to be current epoch time in milliseconds (#23175)
In update scripts, `ctx._now` uses the same milliseconds value used by the
rest of the system to calculate deltas. However, that time is not
actually epoch milliseconds, as it is derived from `System.nanoTime()`.
This change reworks the estimated time thread in ThreadPool which this
time is based on to make available both the relative time, as well as
absolute milliseconds (epoch) which may be used with calendar system. It
also renames the EstimatedTimeThread to a more apt CachedTimeThread.

closes #23169
2017-02-22 15:11:02 -08:00
Lee Hinman 77d641216a Handle long overflow when adding paths' totals
From #23093, we fixed the issue where a filesystem can be so large that it
overflows and returns a negative number. However, there is another issue when
adding a path as a sub-path to another `FsInfo.Path` object, when adding the
totals the values can still overflow.

This adds the same safety to return `Long.MAX_VALUE` instead of the negative
number, as well as a test exercising the logic.
2017-02-22 13:04:34 -07:00
Yannick Welsch 0f88f21535 Don't set local node on cluster state used for node join validation (#23311)
When a node wants to join a cluster, it sends a join request to the master. The master then sends a join validation request to the node. This checks that the node can deserialize the current cluster state that exists on the master and that it can thus handle all the indices that are currently in the cluster (see #21830).

The current code can trip an assertion as it does not take the cluster state as is but sets itself as the local node on the cluster state. This can result in an inconsistent DiscoveryNodes object as the local node is not yet part of the cluster state and a node with same id but different address can still exist in the cluster state. Also another node with the same address but different id can exist in the cluster state if multiple nodes are run on the same machine and ports have been swapped after node crashes/restarts.
2017-02-22 20:27:27 +01:00
Lee Hinman 6f1ed8a3d1
[TEST] Add additional logging to IndicesStoreIntegrationIT.testIndexCleanup 2017-02-22 10:11:05 -07:00
Luca Cavanna 495b24655b Update indices settings api to support CBOR and SMILE format (#23309)
Also expand testing on the different ways to provide index settings and remove dead code around ability to provide settings as query string parameters

Closes #23242
2017-02-22 17:51:10 +01:00
javanna f2acf466aa Convert script/template objects to json format
Elasticsearch accepts multiple content-type formats, hence scripts can be stored/provided in json, yaml, cbor or smile. Yet the format that should be used internally is json. This is a problem mainly around search templates, as they only support json out of the four content-types, so instead of maintaining the content-type of the request we should rather convert the scripts/templates to json.

 Binary formats were not previously supported. If you stored a template in yaml format, you'd get back an error "No encoder found for MIME type [application/yaml]" when trying to execute it. With this commit the request content-type is independent from the template, which always gets converted to json internally. That is transparent to users and doesn't affect the content type of the response obtained when executing the template.
2017-02-22 16:20:53 +01:00
Simon Willnauer 5c1924ad19 Remove BWC layer for number of reduce phases (#23303)
Both PRs below have been backported to 5.4 such that we can enable
BWC tests of this feature as well as remove version dependend serialization
for search request / responses.

Relates to #23288
Relates to #23253
2017-02-22 15:03:09 +01:00
mms-programming d31e41547a Handle BlobPath's trailing separator case (#23091) 2017-02-22 09:04:55 +01:00
Areek Zillur 148be11f26 Make document write requests immutable (#23038)
* Make document write requests immutable

Previously, write requests were mutated at the
transport level to update request version, version type
and sequence no before replication.
Now that all write requests go through the shard bulk
transport action, we can use the primary response stored
in item level bulk requests to pass the updated version,
seqence no. to replicas.

* incorporate feedback

* minor cleanup

* Add bwc test to ensure correct index version propagates to replica

* Fix bwc for propagating write operation versions

* Add assertion on replica request version type

* fix tests using internal version type for replica op

* Fix assertions to assert version type in replica and recovery

* add bwc tests for version checks in concurrent indexing

* incorporate feedback
2017-02-21 17:41:22 -05:00
Simon Willnauer ca38e88148 Remote assertion that relies on all shards being successful
The assertion that if there are buffered aggs at least one incremental
reduce phase should have happened doens't hold if there are shard failure.
This commit removes this assertion.

Relates to #23288
2017-02-21 22:41:49 +01:00
Nik Everett 7475175957 Adds unit test for sampler aggregation (#23243)
* Adds unit test for sampler aggregation

Relates to #22278
2017-02-21 12:51:47 -05:00
Jim Ferenczi 0ff6356b7e Revert "Never reduce the same agg twice"
This change reverts 5e4ba4a60e
Incremental reduction of aggs should also work with a single aggregation now that InternalTopHits.equals
 is fixed.
2017-02-21 18:48:28 +01:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Jim Ferenczi 1ba9770037 Fix comparaison of double in InternalTopHits
InternalTopHits uses "==" to compare hit scores and fails when score is NaN.
This commit changes the comparaison to always use Double.compare.

Relates #23253
2017-02-21 18:18:44 +01:00
Simon Willnauer 5e4ba4a60e Never reduce the same agg twice
Some randomization caused reduction of the same agg multiple times
which causes issues on some aggregations.

Relates to #23253
2017-02-21 17:55:44 +01:00
Simon Willnauer 489f38918d Fix incremental reduce randomization in base tests cases
We can and should randomly reduce down to a single result before
we passing the aggs to the final reduce. This commit changes the logic
to do that and ensures we don't trip the assertions the previous imple tripped.

Relates to #23253
2017-02-21 17:13:46 +01:00
Nik Everett 74c33823ab Comment 2017-02-21 10:43:29 -05:00
Nik Everett 0dee1f85e6 Remove closeAgg 2017-02-21 10:31:42 -05:00
Tanguy Leroux 3a0fc526bb UpdateRequest implements ToXContent (#23289)
This commit changes UpdateRequest so that it implements the ToXContentObject interface.
2017-02-21 15:20:15 +01:00
Jim Ferenczi cc865cbc96 Add unit tests for stats and extended stats aggregations (#23287)
Add tests for InternalStats, InternalExtendedStats and StatsAggregator/ExtendedStatsAggregator

Relates #22278
2017-02-21 15:14:54 +01:00
Simon Willnauer f933f80902 First step towards incremental reduction of query responses (#23253)
Today all query results are buffered up until we received responses of
all shards. This can hold on to a significant amount of memory if the number of
shards is large. This commit adds a first step towards incrementally reducing
aggregations results if a, per search request, configurable amount of responses
are received. If enough query results have been received and buffered all so-far
received aggregation responses will be reduced and released to be GCed.
2017-02-21 13:02:48 +01:00
Tanguy Leroux 39ed76c58b Add parsing method to bulk response (#23234)
This commit adds the `fromXContent()` parsing method to BulkResponse.
2017-02-21 10:49:40 +01:00
Tanguy Leroux c88eb00b83 Add javadoc for DocWriteResponse.Builders (#23267) 2017-02-21 10:19:01 +01:00
Martin Scholz 24bf18b610 Upgrade HDRHistogram to 2.1.9 (#23254) 2017-02-21 08:50:26 +01:00
Martin Scholz 3e292d5245 Migrate TermsQuery to TermInSetQuery (#23229) 2017-02-21 08:49:43 +01:00
Jim Ferenczi 1ff5b318be Fix for IpRangeAggregatorTests#testRanges
Handle null from/to ranges.

Closes #23272
2017-02-20 21:16:14 +01:00
Jason Tedor 4c2bd5feab Introduce sequence-number-aware translog
Today, the relationship between Lucene and the translog is rather
simple: every document not in Lucene is guaranteed to be in the
translog. We need a stronger guarantee from the translog though, namely
that it can replay all operations after a certain sequence number. For
this to be possible, the translog has to made sequence-number aware. As
a first step, we introduce the min and max sequence numbers into the
translog so that each generation knows the possible range of operations
contained in the generation. This will enable future work to keep around
all generations containing operations after a certain sequence number
(e.g., the global checkpoint).

Relates #22822
2017-02-20 15:05:24 -05:00
Jason Tedor 15f5810774 Mark IP range aggregator test as awaits fix
This test reliably fails with the seed 4AC319F8A6B0329B.
2017-02-20 14:42:16 -05:00
Christoph Büscher ea7deace5d Adding fromXContent to Suggest and Suggestion class (#23226)
A follow up to #23202, this adds parsing from xContent and tests to the four Suggestion implementations
and the top level suggest element to be used later when parsing the entire SearchResponse.
2017-02-20 15:45:10 +01:00
Christoph Büscher ea9d51114c Tests: Add unit test for InternalChildren (#23261)
Relates to #22278
2017-02-20 14:02:56 +01:00
Jim Ferenczi 76d6b872dd Add unit tests for GeoBoundsAggregator/InternalGeoBounds (#23259)
* Add unit tests for GeoBoundsAggregator/InternalGeoBounds

Relates #22278
2017-02-20 12:04:30 +01:00
Jim Ferenczi 69b1463f7c Add unit tests for BinaryRangeAggregator/InternalBinaryRange (#23255)
* Add unit tests for BinaryRangeAggregator/InternalBinaryRange

Relates #22278
2017-02-20 11:55:48 +01:00
Tanguy Leroux 872412f645 [Tests] Cleans up DocWriteResponse parsing tests (#23233)
This commit cleans up some parsing tests added from the High Level Rest Client: IndexResponseTests, DeleteResponseTests, UpdateResponseTests, BulkItemResponseTests.

These tests are now more uniform with the others test-from-to-XContent tests we have, they now shuffle the XContent fields before parsing, the asserting method for parsed objects does not used a Map<String, Object> anymore, and buggy equals/hasCode methods in ShardInfo and ShardInfo.Failure have been removed.
2017-02-20 09:45:33 +01:00
Nik Everett d9c37ce195 Adds unit test for sampler aggregation
Relates to #22278
2017-02-17 16:16:04 -05:00
Nik Everett d1de9574ea Checkstyle: Fix link lengths in sampler aggregation 2017-02-17 15:03:57 -05:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Adrien Grand 3bd1d46fc7 Add unit tests for terms aggregation objects. (#23149)
Relates #22278
2017-02-17 18:01:40 +01:00
javanna 578853f264 Remove stale comment about setting routing before parent
Order does not matter anymore since we merged #15371
2017-02-17 17:10:53 +01:00
Yuhao Bi 576e698613 Minor fix of _cat output (#23211) (#23213)
One line was missing a trailing "\n"
2017-02-17 10:46:20 +01:00
Jason Tedor 00a8b8799f Fix control group pattern
The file /proc/self/cgroup lists the control groups to which the process
belongs. This file is a colon separated list of three fields:
 1. a hierarchy ID number
 2. a comma-separated list of hierarchies
 3. the pathname of the control group in the hierarchy

The regex pattern for this contains a bug for the second field. It
allows one or two entries in the comma-separated list, but not
more. This commit fixes the pattern to allow one or more entires in the
comma-separated list.

Relates #23219
2017-02-16 15:31:18 -05:00
Christoph Büscher 268d15ec4c Adding fromXContent to Suggestion.Entry and subclasses (#23202)
This adds parsing from xContent to Suggestion.Entry and its subclasses for Terms-, Phrase-
and CompletionSuggestion.Entry.
2017-02-16 17:59:55 +01:00
markharwood 1cd1ff6010 Test fix - faulty assumptions about when exceptions are thrown in relation to number of failing shards. (#23205)
Search exceptions are thrown only when all shards report failure. Fix changes assertion logic to reflect this.

Closes #23203
2017-02-16 13:48:17 +00:00
Jason Tedor 0a5917d182 Fix get HEAD requests
Get HEAD requests incorrectly return a content-length header of 0. This
commit addresses this by removing the special handling for get HEAD
requests, and just relying on the general mechanism that exists for
handling HEAD requests in the REST layer.

Relates #23186
2017-02-15 13:07:29 -05:00
Christoph Büscher 458ca09e70 Fix checkstyle issue with modifier order in DocWriteResponse 2017-02-15 17:53:39 +01:00
Tanguy Leroux e8d669f50c Add parsing methods to BulkItemResponse (#22859)
This commit adds a parsing method to the BulkItemResponse class. In order to do that, the way DocWriteResponses are parsed has to be changed: ConstructingObjectParser/ObjectParser is removed in favor of a simpler and more readable way to parse these objects.

DocWriteResponse now provides the parseInnerToXContent() method that can be used by subclasses (IndexResponse, UpdateReponse and DeleteResponse) to parse the current token/field and potentially update a DocWriteResponseBuilder. The DocWriteResponseBuilder is a simple POJO used
to contain parsed values. It can be passed around from one parsing method to another parsing method. For example, this is what is done in IndexResponse: a IndexResponseBuilder is created in IndexResponse.fromXContent(), it get passed to IndexResponse.parseXContentFields() that
parses fields specific to IndexResponse (like "created") and updates the context, delegating to DocWriteResponse.parseInnerToXContent() the parsing of any other field. Once all XContent is parsed, IndexResponse.fromXContent() uses the method
IndexResponseBuilder.build() to create the new instance of IndexResponse.

This behavior allow to reuse parsing code among the class hierarchy while keeping the current behavior. It also allows other objects like BulkItemResponse to reuse the same parsing code to parse DocWriteResponses.

Finally, IndexResponseTests, UpdateResponseTests and DeleteResponseTests have been updated to introduce some random shuffling of fields before the XContent is parsed in order to ensure that the parsing code does not rely on field order.
2017-02-15 17:33:10 +01:00