Commit Graph

25585 Commits

Author SHA1 Message Date
Nik Everett 49bdd29f91 Consolidate more parser creation into ESTestCase
This will make it easier to add the forthcoming required argument,
`NamedXContentRegistry`.
2016-12-13 20:28:41 -05:00
Jason Tedor 0d195f1afa Avoid corruption when deserializing booleans
Today we write 0x00 or 0x01 for false or true when serializing a boolean
(and 0x02 for null when serializing an optional boolean) but we
deserialize any non-zero byte to true (except when deserializing an
optional boolean in which case we deserialize 0x02 to null, 0x01 to
true, and any other non-zero byte to false). This too easily allows
corruption into the stream. Instead, we should mark the stream as
corrupted and stop deserializing. This catches when we try to
deserialize something as a boolean that is not a boolean.

Relates #22152
2016-12-13 20:10:05 -05:00
Jason Tedor 510ad7b9c7 Add shutdown hook for closing CLI commands
This commit enables CLI commands to be closeable and installs a runtime
shutdown hook to ensure that if the JVM shuts down (as opposed to
aborting) the close method is called.

It is not enough to wrap uses of commands in main methods in
try-with-resources blocks as these will not run if, say, the virtual
machine is terminated in response to SIGINT, or system shutdown event.

Relates #22126
2016-12-13 19:10:11 -05:00
Nik Everett 872984d21a Continue consolidating `XContentParser` construction in tests (#22145)
Consolidate more parser creation in tests

Moves more parser creation in tests to the `createParser` methods
in `ESTestCase`.
2016-12-13 17:22:39 -05:00
Simon Willnauer b7bcb5bb3a [TEST] Skip term / int partitioning tests in bwc tests pre 5.2.0 2016-12-13 22:20:44 +01:00
Tal Levy f56097b57a Fixes GrokProcessor's ignorance of named-captures with same name. (#22131)
Grok was originally ignoring potential matches to named-capture groups
larger than one. For example, If you had two patterns containing the
same named field, but only the second pattern matched, it would fail to
pick this up.

This PR fixes this by exploring all potential places where a
named-capture was used and chooses the first one that matched.

Fixes #22117.
2016-12-13 13:19:55 -08:00
Jason Tedor 9ad49faa86 Fix for loop that is bad and should feel bad
This commit fixes a for loop that reverses the order of shard stats
coming off the wire, and is really hard to read anyway (with the
post-increment in the loop initializer).

Relates #22150
2016-12-13 15:27:42 -05:00
Simon Willnauer 7a9b667e98 Introduce a low level protocol handshake (#22094)
Today we rely on the version that the API user passes in together with the DiscoveryNode. This commit introduces a low level handshake where nodes exchange their version to be used with the transport protocol that is executed every time a connection to a node is established. This, on the one hand allows to change the wire protocol based on the version we are talking to even without a full cluster restart. Today we would need to carry on a BWC layer across major versions but with a handshake we can rely on the fact that the latest version of the previous minor executes a handshake and uses the latest protocol version across all communication with the N+1 version nodes.

This change is yet fully backwards compatible, a followup PR will remove the BWC in 6.0 once this has been back-ported to the 5.x branch
2016-12-13 21:06:23 +01:00
Nik Everett ce86405394 Start to centralize creation of XContentParser in tests (#22096)
Starts to centralize creation of the `XContentParser` in
`protected final` methods on `ESTestCase`. The idea is to enable
adding `NamedXContentRegistry` relatively easily by giving tests
a single place they can override to define the
`NamedXContentRegistry`. Since `NamedXContentRegistry` doesn't
exist yet neither does the override point.

This doesn't attempt to migrate all the tests to calling the
new methods to build the parsers. I wanted to make this so we
could review the concept and then I'll merge a followup to
migrate the tests.
2016-12-13 11:22:15 -05:00
markharwood 4c6d17a176 Added tests for toXContent and fromXContent for IncludeExclude class.
New REST test revealed an issue with inconsistent hashing in partitioned
term tests which is also fixed in this change.

Closes #22102
2016-12-13 15:23:09 +00:00
Clinton Gormley 28397c9594 Added version 2.4.3 plus bwc indices 2016-12-13 16:15:52 +01:00
javanna 478cd78047 Remove unused import from RequestLogger
Not only was StringJoiner unused, it's also a class only available in java 1.8, which is a problem given that the REST client has minimum java required set to 1.7
2016-12-13 15:01:51 +01:00
Adrien Grand 049fd3991c Remove `AggregationContext`. (#22124)
This class is just a wrapper around `SearchContext`, so let's use
`SearchContext` directly. The change is mechanical, except the
`ValuesSourceConfig` class, where I moved the logic to get a `ValuesSource`
given a config.
2016-12-13 09:09:40 +01:00
Lee Hinman a4e8b5d952 Throw an exception on unrecognized "match_mapping_type"
When using dynamic templates, ES will now throw an exception if a
`match_mapping_type` is used that doesn't correspond to an actual type.

Relates to #17285
2016-12-12 09:59:48 -07:00
Simon Willnauer b667ff46c4 Allow plugins to install bootstrap checks (#22110)
Plugins also have the need to provide better OOTB experience by configuring
defaults unless the plugin is used in _production_ mode. This change exposes
the bootstrap check infrastructure as part of the plugin API to allow plugins
to specify / install their own bootstrap checks if necessary.
2016-12-12 17:35:00 +01:00
Matt Sicker 32fcb7220e Clarify first search API explanation of parameters (#21958)
Added info about the sort parameter and included the pretty parameter in the example.
2016-12-12 15:00:53 +01:00
yutaono abf718832d add missing url-access-control link (#21960) 2016-12-12 14:56:52 +01:00
Peter Dyson 7904028fef Merge pull request #22105 from geekpete/master
Fix Fuziness typo in multimatch error message and associated tests
2016-12-12 23:13:20 +10:00
Luca Cavanna 6d987a9b69 Remove support for empty queries (#22092)
Our query DSL supports empty queries (`{}`), which have a different meaning depending on the query that holds it, either ignored, match_all or match_none. We deprecated the support for empty queries in 5.0, where we log a deprecation warning wherever they are used.

The way we supported it once we moved query parsing to the coordinating node was having an Optional<QueryBuilder> return type in all of our parse methods (called fromXContent). See #17624. The central place for this was QueryParseContext#parseInnerQueryBuilder. We can now remove all the optional return types and simply throw an exception whenever an empty query is found.
2016-12-12 12:37:12 +01:00
Luca Cavanna 9be778c5e5 Warn log deprecation warnings received from server (#21895)
The warnings get printed out in a single line e.g. WARNING: request [DELETE http://localhost:9200/index/type/_api] returned 3 warnings:[this is warning number 0],[this is warning number 1],[this is warning number 2]
2016-12-12 12:11:42 +01:00
Luca Cavanna a42bee5d60 Don't output empty ext object in SearchSourceBuilder#toXContent (#22093)
We shouldn't output an empty ext object if no ext sections have been set to the SearchSourceBuilder.

Closes #20969
2016-12-12 12:10:23 +01:00
Luca Cavanna 73cf002293 Un-deprecate fuzzy query (#22088)
When we decided to deprecate and remove fuzzy query in #15760, we didn't realize we would take away the possibililty for uses to use a fuzzy query as part of a span query, which is not possible using match query. This means we have to go back and un-deprecate fuzzy query, which will not be removed.

Closes #15760
2016-12-12 12:09:16 +01:00
David Pilato b0ab3237d3 Merge branch 'fix/22053-azure-readonly' 2016-12-12 11:59:01 +01:00
Peter Dyson d339814f77 Fix Fuziness typo in multimatch error message and associated tests 2016-12-12 11:33:31 +10:00
Artur Nowosielski 726f5dccc0 Rewrite filter queries in FiltersAggregationBuilder (#22076)
Queries must be rewritten before the query phase executes otherwise non-executable queries like `wrapper` query or `terms`  will fail or queries that require resources like script service can't access these service unless rewritten.

Relates to #21303
2016-12-11 14:37:12 +01:00
Simon Willnauer 20ff703e07 Fix IncludeExclude parsing
`include` / `exclude` in terms / sig-terms aggs seems completely broken
and massively untested. This commit makes the TermsTests pass again that
randomly use `include` / `exclude`. This class must be tested individually
and we need real integ tests that use xcontent that use this feature.
2016-12-11 09:55:53 +01:00
javanna e6b10ca4db Restore proper bwcVersion in qa/backwards gradle build file 2016-12-10 21:11:05 +01:00
javanna 6003cbfb64 fix typos in qa/backwards gradle build file 2016-12-10 21:09:49 +01:00
Nik Everett 7a74a41a0c Fix test for changed message
Message is only changed in 5.2.0 so we shouldn't assert on it
if we're running with any nodes less than that version.
2016-12-10 10:35:14 -05:00
Yannick Welsch 68f30cae0c [TEST] Unmute testRestoreUnsupportedSnapshots 2016-12-10 13:49:47 +01:00
Yannick Welsch 4831632a6f [TEST] Handle legacy snapshots as if they don't exist anymore
An earlier commit removed BWC for pre-5.0 snapshots, which also meant removing the capability to load pre-5.0 snapshots. In 6.0, such snapshots are now
invisible and must be treated by the BWC tests in that way.
2016-12-10 13:49:47 +01:00
Yannick Welsch 4ad85c38c3 Throw NoSuchFileException to correctly adhere to readBlob contract
URLBlobContainer can in certain situations throw a FileNotFoundException. To fulfill the contract of the readBlob method it should throw a NoSuchFileException instead when the given blob cannot be found.
2016-12-10 13:49:47 +01:00
Simon Willnauer 01d67e09b9 Detach handshake from connect to node (#22037)
Today we connect and publish the nodes connection before we execute a
handshake with the node we connect to. In the case of connecting to a node
that won't pass the handshake this connection is already `published` and other
code paths can use it. This commit detaches the connection and the publish of the
connection such that `TransportService` can do a handshake before actually connect
and publish the connection.
2016-12-10 10:03:26 +01:00
Nik Everett 3adefb7b4a Begin centralizing XContentParser creation into RestRequest (#22041)
To get #22003 in cleanly we need to centralize as much `XContentParser` creation as possible into `RestRequest`. That'll mean we have to plumb the `NamedXContentRegistry` into fewer places.

This removes `RestAction.hasBody`, `RestAction.guessBodyContentType`, and `RestActions.getRestContent`, moving callers over to `RestRequest.hasContentOrSourceParam`, `RestRequest.contentOrSourceParam`, and `RestRequest.contentOrSourceParamParser` and `RestRequest.withContentOrSourceParamParserOrNull`. The idea is to use `withContentOrSourceParamParserOrNull` if you need to handle requests without any sort of body content and to use `contentOrSourceParamParser` otherwise.

I believe the vast majority of this PR to be purely mechanical but I know I've made the following behavioral change (I'll add more if I think of more):
* If you make a request to an endpoint that requires a request body and has cut over to the new APIs instead of getting `Failed to derive xcontent` you'll get `Body required`.
* Template parsing is now non-strict by default. This is important because we need to be able to deprecate things without requests failing.
2016-12-09 20:23:02 -05:00
Nik Everett ddade1b5ac Improve the error message if task and node isn't found (#22062)
Improves the error message returned when looking up a task that
belongs to a node that is no longer part of the cluster. The new
error message tells the user that the node isn't part of the cluster.
This is useful because if you start a task and the node goes down
there isn't a record of the task at all. This hints to the user that
the task might have died with the node.

Relates to #22027
2016-12-09 15:50:46 -05:00
Matthew Haugen ca8d89af5a Correct typo in "detect_noop" documentation (#22039)
The documentation reads:

> You can disable this behavior by setting "detect_noop": false like this:

Followed by a code example, that originally set `"detect_noop": true`.

Please correct me if I got the change backwards (i.e. the paragraph should be changed to `true`), but this seems like it makes the most sense.
2016-12-09 15:24:08 -05:00
Yannick Welsch fca4f92fee Fix BWC condition on REST test
Adds a missing skip section to a REST test that was forgotten in #21998
2016-12-09 19:05:00 +01:00
Igor Motov 93b5e55660 Restores the original default format of search slow log
In 5.0, the search slow log switched to the multi-line format with no option to get back to the origin single-line format that was used prior to 5.0 by default. This commit removes the reformat option from the search slow log and returns the search slow log back to the single-line format.

Closes #21711
2016-12-09 12:38:28 -05:00
Yannick Welsch b20b160a5e Allow flush/force_merge/upgrade on shard marked as relocated (#22078)
A shard that is locally marked as relocated, but where the relocation target shard has not been activated yet by the master, can still receive index operations, which in return can lead to flushes being triggered. Flushing is currently (wrongly) prohibited on shards marked as relocated, which makes the flushing process go into an endless retry loop and log warnings until the shard is closed. This commit fixes this situation by allowing flush, force_merge and upgrade operations to run on shards that are marked as relocated.
2016-12-09 17:56:40 +01:00
Nik Everett bcef1e7452 Better error message when _parent isn't an object (#21987)
If you make a mistake and specify a mapping like:
```
{
  "parent": {
    "properties": {}
  },
  "child": {
    "_parent": "parent",
    "properties": {}
  }
}
```

then the error message you get back amounts to
`Failed to parse mapping for [child]: can't cast a String to a Map`.
Since it doens't tell you *which* string can't be cast to a map you
have to dig through the stack trace to figure out what to fix. This
replaces the error message with:
```
Failed to parse mapping [child]: [_parent] must be an object containing [type]
```
so you can tell that the problem is with the `parent` field.
2016-12-09 11:33:31 -05:00
Nik Everett fc2060ba7e Don't close rest client from its callback (#22061)
If you try to close the rest client inside one of its callbacks then
it blocks itself. The thread pool switches the status to one that
requests a shutdown and then waits for the pool to shutdown. When
another thread attempts to honor the shutdown request it waits
for all the threads in the pool to finish what they are working on.
Thus thread a is waiting on thread b while thread b is waiting
on thread a. It isn't quite that simple, but it is close.

Relates to #22027
2016-12-09 10:39:51 -05:00
Yannick Welsch a724f4eb61 Don't update nodes list when stepping down as master (#22049)
This commit simplifies the node update logic so that nodes are never removed from the cluster state when the cluster state is not published.
2016-12-09 14:55:48 +01:00
Christoph Büscher 2592ff86ce Add fromXContent to InternalNestedIdentity
This adds a fromXContent method and unit test to InternalNestedIdentity so we can parse it as part of a search response. This is part of the preparation for parsing search responses on the client side.
2016-12-09 14:52:06 +01:00
Yannick Welsch db0660a7ea Reject external versioning and explicit version numbers on create (#21998)
Fixes an issue where indexing requests with operation type "create" auto-convert external versioning to internal versioning and silently ignore the version number instead of failing with an error message.
2016-12-09 14:21:22 +01:00
Matias Anaya beb794cb0f Fix typo in percolated-query.asciidoc (#21991) 2016-12-09 13:45:57 +01:00
Michael McCandless 613a1a6a18 Add stored binary fields to static backwards compatibility indices tests (#22054)
Add stored binary fields to static backwards compatibility indices tests
2016-12-09 05:32:40 -05:00
Adrien Grand 6714e02bef Mute RestoreBackwardsCompatIT.testRestoreUnsupportedSnapshots. 2016-12-09 10:41:07 +01:00
Juan Paulo Gutierrez acc39b1496 Fixed typo (#22066) 2016-12-09 10:34:29 +01:00
Adrien Grand 787519ee4c Fix `other_bucket` on the `filters` agg to be enabled if a key is set. (#21994)
Closes #21951
2016-12-09 09:48:48 +01:00
Adrien Grand 1bdf4a2c5b Partition-based include-exclude does not implement equals/hashcode/serialization correctly. (#22051) 2016-12-09 09:48:16 +01:00