Commit Graph

7129 Commits

Author SHA1 Message Date
Christoph Büscher bc22c86d14 SuggestionBuilder doesn't need to extend ToXContentToBytes
This changes the class from extending the abstract class to implementing the
ToXContent interface only. The former could lead to unexpected behaviour when
trying to display the object, since the "toString()" method inherited from
ToXContentToBytes would create an error message because the SuggestionBuilders
toXContent() methods don't render complete json objects.
2016-12-20 14:57:28 +01:00
Tanguy Leroux 290326e73e Add fromXContent() methods for ReplicationResponse (#22196)
This commit adds the parsing fromXContent() methods to the ReplicationResponse.ShardInfo and ReplicationResponse.ShardInfo.Failure classes.
2016-12-20 09:29:11 +01:00
Ryan Ernst 850f51db01 Internal: Refactor SettingCommand into EnvironmentAwareCommand (#22175)
* Internal: Refactor SettingCommand into EnvironmentAwareCommand

This change renames and changes the behavior of SettingCommand to have
its primary method take in a fully initialized Environment for
elasticsearch instead of just a map of settings. All of the subclasses
of SettingCommand already did this at some point, so this just removes
duplication.
2016-12-19 15:23:44 -08:00
Nik Everett e508f2ef6a Fix java 9 build
We removed a cast we needed to appease Java 9. I've recreated it
in simpler form and left a comment about why we need it.
2016-12-19 17:34:09 -05:00
Alexander Lin 0ab3cbe3a3 Adds percent-encoding for Location headers (#21057)
This should cause unicode elements in the location header to be percent-encoded, instead of being left alone.

Closes #21016
2016-12-19 15:56:09 -05:00
Nik Everett 40b80ae104 Fix line length 2016-12-19 15:07:14 -05:00
Nik Everett 2e1d152fc0 Sub-fields should not accept `include_in_all` parameter (#21971)
Fail to update mapping when multifield has `include_in_all`.

Closes #21710
2016-12-19 15:07:00 -05:00
Grzegorz Gajos f6b6e4e376 Added ability to remove pipelines via wildcards (#22149) (#22191)
This commit is adding an ability to remove pipelines with wildcards.
2016-12-19 10:59:59 -08:00
javanna 5dae10db11 [TEST] add warnings check to ESTestCase
We are currenlty checking that no deprecation warnings are emitted in our query tests. That can be moved to ESTestCase (disabled in ESIntegTestCase) as it allows us to easily catch where our tests use deprecated features and assert on the expected warnings.
2016-12-19 19:39:56 +01:00
javanna 6a27628f12 Remove support for strict parsing mode
We return deprecation warnings as response headers, besides logging them. Strict parsing mode stayed around, but was only used in query tests, though we also introduced checks for deprecation warnings there that don't need strict parsing anymore (see #20993).

 We can then safely remove support for strict parsing mode. The final goal is to remove the ParseFieldMatcher class, but there are many many users of it. This commit prepares the field for the removal, by deprecating ParseFieldMatcher and making it effectively not needed. Strict parsing is removed from ParseFieldMatcher, and strict parsing is replaced in tests where needed with deprecation warnings checks.

 Note that the setting to enable strict parsing was never ported to the new settings infra hance it cannot be set in production. It is really only used in our own tests.

 Relates to #19552
2016-12-19 19:39:56 +01:00
javanna 38914f17ed [TEST] improve ElasticsearchAssertions#assertEquivalent for ToXContent
Rename the method to assertToXContentEquivalent to highlight that it's tailored to ToXContent comparisons.

Rather than parsing into a map and replacing byte[] in both those maps, add custom equality assertions that recursively walk maps and lists and call Arrays.equals whenever a byte[] is encountered.
2016-12-19 19:32:50 +01:00
javanna 04d929ff53 add inline comments on GetField binary values parsing 2016-12-19 19:32:50 +01:00
javanna 87d8764a32 [TEST] add unit test for XContentHelper#toXContent method 2016-12-19 17:53:42 +01:00
Luca Cavanna 3421e54a42 Add fromXContent method to GetResponse (#22082)
Moved field values `toXContent` logic to `GetField` (from `GetResult`), which outputs its own fields, and can also parse them now. Also added `fromXContent` to `GetResult` and `GetResponse`.

 The start object and end object for `GetResponse` output have been moved to `GetResult#toXContent`, from the corresponding rest action. This makes it possible to have `toXContent` and `fromXContent` completely symmetric, as parsing requires looping till an end object is found which is weird when the corresponding `toXContent` doesn't print that out.

This also introduces the foundation for testing retrieval of _source and stored field values.
2016-12-19 17:21:26 +01:00
Yannick Welsch 63af03a104 Atomic mapping updates across types (#22220)
This commit makes mapping updates atomic when multiple types in an index are updated. Mappings for an index are now applied in a single atomic operation, which also allows to optimize some of the cross-type updates and checks.
2016-12-19 14:39:50 +01:00
Yannick Welsch 1cabf66bd5 Use correct block levels for TRA subclasses (#22224)
Subclasses of TransportReplicationAction can currently chose to implement block levels for which the request will be blocked.
- Refresh/Flush was using the block level METADATA_WRITE although they don't operate at the cluster meta data level (but more like shard level meta data which is not represented in the block levels). Their level has been changed to null so that they can operate freely in the presence of blocks.
- GlobChkptSync was using WRITE although it does not make any changes to the actual documents of a shard. The level has been changed to null so that it can operate freely in the presence of blocks.
The commit also adds a check for closed indices in TRA so that the right exception is thrown if refresh/flush/checkpoint syncing is attempted on a closed index (before it was throwing an IndexNotFoundException, now it's throwing IndexClosedException).
2016-12-19 14:36:58 +01:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
Dimitris Athanasiou b58bbb9e48 Allow setting aggs after parsing them elsewhere (#22238)
This commit exposes public getters for the aggregations in
AggregatorFactories.Builder. The reason is that it allows to
parse the aggregation object from elsewhere (e.g. a plugin) and then
be able to get the aggregation builders in order to set them in
a SearchSourceBuilder.

The alternative would have been to expose a setter for the
AggregatorFactories.Builder object. But that would be making
the API a bit trappy.
2016-12-19 09:52:07 +00:00
Simon Willnauer ce5c094cda Speed up filter and prefix settings operations (#22249)
Today if a settings object has many keys ie. if somebody specifies
a gazillion synonym in-line (arrays are keys ending with ordinals) operations like
`Settings#getByPrefix` have a linear runtime. This can cause index creations to be
very slow producing lots of garbage at the same time. Yet, `Settings#getByPrefix` is called
quite frequently by group settings etc. which can cause heavy load on the system.

While it's not recommended to have synonym lists with 25k entries in-line these use-cases should
not have such a large impact on the cluster / node. This change introduces a view-like map
that filters based on the prefixes referencing the actual source map instead of copying all values
over and over again. A benchmark that adds a single key with 25k random synonyms between 2 and 5 chars
takes 16 seconds to get the synonym prefix 200 times while the filtered view takes 4 ms for the 200 iterations.

This relates to https://discuss.elastic.co/t/200-cpu-elasticsearch-5-index-creation-very-slow-with-a-huge-synonyms-list/69052
2016-12-19 10:48:38 +01:00
Adrien Grand 1ed2e18ded Fix MapperService.allEnabled(). (#22227)
It returns whether the last merged mapping has `_all` enabled rather than
whether any of the types has `_all` enabled.
2016-12-19 09:55:13 +01:00
Adrien Grand 96f1739c0d The `_all` default mapper is not completely configured. (#22236)
In some cases, it might happen that the `_all` field gets a field type that is
not totally configured, and in particular lacks analyzers. This is due to the
fact that `AllFieldMapper.TypeParser.getDefault` uses `Defaults.FIELD_TYPE` as
a default field type, which does not have any analyzers configured since it
does not know about the default analyzers.
2016-12-19 09:54:27 +01:00
Daniel Mitterdorfer 3ce7b119d2 Enable strict duplicate checks for all XContent types (#22225)
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).

We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.

Relates elastic/elasticsearch#19614
Relates elastic/elasticsearch#22073
2016-12-19 09:29:47 +01:00
Daniel Mitterdorfer 6327e35414 Change type of ingest doc meta-data field 'TIMESTAMP' to `Date` (#22234)
With this commit we change the data type of the 'TIMESTAMP'
meta-data field from a formatted date string to a plain
`java.util.Date` instance. The main reason for this change is
that our benchmarks have indicated that this contributes
significantly to the time spent in the ingest pipeline.

The overhead in terms of indexing throughput of the ingest
pipeline is about 15% and breaks down roughly as follows:

* 5% overhead caused by the conversion from `XContent` -> `Map`
* 5% overhead caused by the timestamp formatting
* 5% overhead caused by the conversion `Map` -> `XContent`

Relates #22074
2016-12-19 09:10:58 +01:00
Simon Willnauer ccfeac8dd5 Remove `doHandshake` test-only settings from TcpTransport (#22241)
In #22094 we introduce a test-only setting to simulate transport
impls that don't support handshakes. This commit implements the same logic
without a setting.
2016-12-18 09:26:53 +01:00
Boaz Leskes b78f7bc51d InternalEngine should use global checkpoint when committing the translog
relates to #22212
2016-12-18 08:05:59 +01:00
Jason Tedor 58d73bae74 Tighten sequence numbers recovery
This commit touches addresses issues related to recovery and sequence numbers:
 - A sequence number can be assigned and a Lucene commit created with a
   maximum sequence number at least as large as that sequence number,
   yet the operation corresponding to that sequence number can be
   missing from both the Lucene commit and the translog. This means that
   upon recovery the local checkpoint will be stuck at or below this
   missing sequence number. To address this, we force the local
   checkpoint to the maximum sequence number in the Lucene commit when
   opening the engine. Note that there can still be gaps in the history
   in the translog but we do not address those here.
 - The global checkpoint is transferred to the target shard at the end
   of peer recovery.
 - Additionally, we reenable the relocation integration tests.

Lastly, this work uncovered some bugs in the assignment of sequence
numbers on replica operations:
 - setting the sequence number on replica write requests was missing,
   very likely introduced as a result of resolving merge conflicts
 - handling operations that arrive out of order on a replica and have a
   version conflict with a previous operation were never marked as
   processed

Relates #22212
2016-12-17 09:20:46 -05:00
Simon Willnauer 1f3eb068d5 Add infrastructure to manage network connections outside of Transport/TransportService (#22194)
Some expert users like UnicastZenPing today establishes real connections to nodes during it's ping
phase that can be used by other parts of the system. Yet, this is potentially dangerous
and undesirable unless the nodes have been fully verified and should be connected to in the
case of a cluster state update or if we join a newly elected master. For use-cases like this, this change adds the infrastructure to manually handle connections that are not publicly available on the node ie. should not be managed by `Transport`/`TransportSerivce`
2016-12-17 11:49:57 +01:00
Simon Willnauer 0b338bf523 Cleanup random stats serialization code (#22223)
Some of our stats serialization code duplicates complicated seriazliation logic
or could use existing building blocks from StreamOutput/Input. This commit
cleans up some of the serialization code.
2016-12-17 11:45:55 +01:00
Ryan Ernst 9e5cedae23 Fix line lengths in renamed seccomp file 2016-12-16 22:18:56 -08:00
Jason Tedor f7d43132b2 Refer to system call filter instead of seccomp
Today in the codebase we refer to seccomp everywhere instead of system
call filter even if we are not specifically referring to Linux. This
commit is a purely mechanical change to refer to system call filter
where appropriate instead of the general seccomp, and only leaves
seccomp in place when actually referring to the Linux implementation.

Relates #22243
2016-12-16 18:30:19 -05:00
Jason Tedor 30806af6bd Rename bootstrap.seccomp to bootstrap.system_call_filter
We try to install a system call filter on various operating systems
(Linux, macOS, BSD, Solaris, and Windows) but the setting
(bootstrap.seccomp) to control this is named after the Linux
implementation (seccomp). This commit replaces this setting with
bootstrap.system_call_filter. For backwards compatibility reasons, we
fallback to bootstrap.seccomp and log a deprecation message if
bootstrap.seccomp is set. We intend to remove this fallback in
6.0.0. Note that now is the time to make this change it's likely that
most users are not making this setting anyway as prior to version 5.2.0
(currently unreleased) it was not necessary to configure anything to
enable a node to start up if the system call filter failed to install
(we marched on anyway) but starting in 5.2.0 it will be necessary in
this case.

Relates #22226
2016-12-16 18:22:54 -05:00
Luca Cavanna 2265be69d2 Deprecate XContentType auto detection methods in XContentFactory (#22181)
With recent changes to our parsing code we have drastically reduced the places where we auto-detect the content type from the input. The usage of these methods spread in our codebase for no reason, given that in most of the cases we know the content type upfront and we don't need any auto-detection mechanism. Deprecating these methods is a way to try and make sure that these methods are carefully used, and hopefully not introduced in newly written code.

We have yet to fix the REST layer to read the Content-Type header, which is the long term solution, but for now we just want to make sure that the usage of these methods doesn't spread any further.

Relates to #19388
2016-12-16 19:33:12 +01:00
Areek Zillur d44de0cecc Remove deprecated _suggest endpoint (#22203)
In #20305, _suggest endpoint was deprecated
in favour of using _search endpoint. This
commit removes the dedicated _suggest endpoint
entirely from master.
2016-12-16 12:06:02 -05:00
Masaru Hasegawa 7cfa6898bf Merge pull request #22215 from masaruh/skip_empty_boost
Don't print empty indices_boost
2016-12-16 17:31:03 +09:00
Simon Willnauer 25b79cd46b Only notify handshake handler onClose if it can be successfully removed
Depending on how the connection is closed the `#onChannelClosed` callback
might be invoked more than once or the handler has been processed by the response
of the handshake already. This commit only notifies the handler if was removed from
the pending map.
2016-12-16 09:06:57 +01:00
Masaru Hasegawa 6fe83fb524 Don't print empty indices_boost 2016-12-16 16:19:34 +09:00
Masaru Hasegawa a0185c83a7 Merge pull request #21393 from masaruh/alias_boost
Resolve index names in indices_boost
2016-12-16 15:07:51 +09:00
Nik Everett 61597f2c20 Send error_trace by default when testing (#22195)
Sends the `error_trace` parameter with all requests sent by the
yaml test framework, including the doc snippet tests. This can be
overridden by settings `error_trace: false`. While this drift's
core's handling of the yaml tests from the client's slightly this
should only be a problem for tests that rely on the default value,
both of which I've fixed by setting the value explicitly.

This also escapes `\n` and `\t` in the `Stash dump on failure` so
the `stack_trace` is more readable.

Also fixes `RestUpdateSettingsAction` to not think of the `error_trace`
parameter as a setting.
2016-12-15 13:35:14 -05:00
Boaz Leskes b6cbcc49ba ClusterService should expose "applied" cluster states (i.e., remove ClusterStateStatus) (#21817)
`ClusterService` is responsible of updating the cluster state on every node (as a response to an API call on the master and when non-masters receive a new state from the master). When a new cluster state is processed, it is made visible via the `ClusterService#state` method and is sent to series of listeners. Those listeners come in two flavours - one is to change the state of the node in response to the new cluster state (call these cluster state appliers), the other is to start a secondary process. Examples for the later include an indexing operation waiting for a shard to be started or a master node action waiting for a master to be elected. 

The fact that we expose the state before applying it means that samplers of the cluster state had to worry about two things - working based on a stale CS and working based on a future, i.e., "being applied" CS. The `ClusterStateStatus` was used to allow distinguishing between the two. Working with a stale cluster state is not avoidable. How this PR changes things to make sure consumers don't need to worry about future CS, removing the need for the status and simplifying the waiting logic.

This change does come with a price as "cluster state appliers" can't sample the cluster state from `ClusterService` whenever they want as the cluster state isn't exposed yet. However, recent clean ups made this is situation easier and this PR takes the last steps to remove such sampling. This also helps clarify the "information flow" and helps component separation (and thus potential unit testing). It also adds an assertion that will trigger if the cluster state is sampled by such listeners. 

Note that there are still many "appliers" that could be made a simpler, unrestricted "listener" but this can be done in smaller bits in the future. The commit also makes it clear what the `appliers` and what the `listeners` are by using dedicated interfaces.

Also, since I had to change the listener types I went ahead and changed the data structure for temporary/timeout listeners (used for the observer) so addition and removal won't be an O(n) operation.
2016-12-15 17:06:25 +01:00
Tanguy Leroux 391d3a20f3 Add unit tests for toXContent methods in ReplicationResponse (#22188)
This commit adds unit tests for the toXContent() methods of the inner classes  ReplicationResponse.ShardInfo and ReplicationResponse.ShardInfo.Failure.
2016-12-15 16:12:33 +01:00
Aaron Spiegel 80d3d790ae Fix handling of segment file sizes in stats API
This commit addresses an issue in the stats APIs where
include_segment_file_sizes was not being consumed leading to requests
containing this parameter being rejected.

Relates #21879
2016-12-15 07:29:11 -05:00
Simon Willnauer ef610636b6 Remove TCP handshake BWC from master (#22151)
Since #22094 has been back-ported to 5.2 we can remove all BWC layers from master since all supported version will handle handshake requests.

Relates to #22094
2016-12-15 12:47:01 +01:00
Simon Willnauer d27a12510b Handle race-condition when connection is closed before handshake listener was added
Today sending a message on a closed channel doesn't throw an exception. The channel
might just swallow the exception and informs the internal async exception handler
that a channel got disconnected. This change adds a safety check that we fail
the handshake if we registered a handler but the channel has been closed already
for instance due to a reset by peer.
2016-12-15 12:41:50 +01:00
Jim Ferenczi 3005366b13 Fix boost_mode propagation when the function score query builder is rewritten (#22172)
This change fixes the cloning of the FunctionScoreQueryBuilder when the inner query or functions are rewritten.

Fixes #22138
2016-12-15 11:08:17 +01:00
Yannick Welsch b9600c7891 Only update DocumentMapper if field type changes (#22165)
Merging mappings ensures that fields are used consistently across mapping types. Disabling norms for a specific field in one mapping type for example also disables norms for the same field in other mapping types of that index. The logic that ensures this while merging mappings currently always creates a fresh document mapper for all existing mapping types, even if no change occurred. Creating such a fresh document mapper does not come for free though as it involves recompressing the source. Making a mapping change to one type of an index with 100 types will thus re-serialize and recompress all 100 types, independent of any changes made to those types.

This commit fixes the update logic to only create a new DocumentMapper if a field type actually changes.
2016-12-15 09:20:28 +01:00
makeyang 4c79f36abb 1. add license declaration in class header
2. remove Created by declaration
3. fix typo method name from testExceptionForCopyToInMultiFields to testExceptionForIncludeInAllInMultiFields
4. fix typo method name from createMappinmgWithIncludeInAllInMultiField to createMappingWithIncludeInAllInMultiField
5. use expectThrows rather than try catch according to nik9000's comments
2016-12-15 14:31:18 +08:00
Areek Zillur cdd5fbe3a1 Deprecate _suggest endpoint in favour of _search (#20305)
* Replace _suggest endpoint to _search in docs

In 5.0, the _suggest endpoint is just sugar for _search
with suggestions specified. Users should move away from
using the _suggest endpoint, as it is marked as deprecated in 5.x and
will be removed in 6.0

* update docs to use _search endpoint instead of _suggest

* Add deprecation logging to RestSuggestAction

* Use search endpoint instead of suggest endpoint in rest tests
2016-12-14 21:49:53 -05:00
Simon Willnauer 80d6539e9c Handle connection close / reset events gracefully during handshake (#22178)
Low level handshake code doesn't handle situations gracefully if the connection
is concurrently closed or reset by peer. This commit adds the relevant code to
fail the handshake if the connection is closed.
2016-12-14 23:04:14 +01:00
Nik Everett 749039ad4f Consolidate the last easy parser construction (#22095)
Moves the last of the "easy" parser construction into
`RestRequest`, this time with a new method
`RestRequest#contentParser`. The rest of the production
code that builds `XContentParser` isn't "easy" because it is
exposed in the Transport Client API (a Builder) object.
2016-12-14 15:41:25 -05:00
Boaz Leskes bf65a69bbf Enforce min master nodes in test cluster (#22065)
In order to start clusters with min master nodes set without setting `discovery.initial_state_timeout`, #21846 has changed the way we start nodes. Instead to the previous serial start up, we now always start the nodes in an async fashion (internally). This means that starting a cluster is unsafe without `min_master_nodes` being set. We should therefore make it mandatory.
2016-12-14 20:14:16 +01:00
Nik Everett 1eddff822a Centralize remaining XContentParser creation in tests
Now we're ready to add `NamedXContentRegistry`.
2016-12-14 11:52:09 -05:00
Tanguy Leroux a8b238ebad Add equals() and hashCode() methods to ReplicationResponse (#22168)
This commit adds the equals() and hashCode() methods to ReplicationResponse.ShardInfo and ReplicationResponse.ShardInfo.Failure classes.
2016-12-14 17:37:52 +01:00
Tanguy Leroux 135f60502c Add parsing method for ElasticsearchException (#22143)
ElasticsearchException is used in various response objects like IndexResponse, DeleteResponse or BulkItemResponse.Failure. It would be helpful to the High Level Rest Client to be able to parse these exceptions back.

This commit adds the fromXContent() method to the ElasticsearchException object. This method does not return the original (wrapped or unwrapped) exception but always returns a ElasticsearchException that serves as a simple POJO for all types of exceptions. The parsed ElasticsearchException's message will be composed of the original exception type (ex: illegal_argument_exception) concatenated with the original reason to help users/clients to known and handle the error.
2016-12-14 16:11:44 +01:00
Yannick Welsch a511fb9ce6 Prefer joining node with conflicting transport address when becoming master (#22134)
PR #22049 changed the node update logic to never remove nodes from the cluster state when the cluster state is not published. This led to an issue electing a master (#22120) based on nodes with same transport address (but different node id) as previous nodes. The joining nodes should take precedence over conflicting ones. Note that this only applies to the action of becoming master. If a master is established and a node joins an existing master, it will be rejected if there is another node with same transport address.
2016-12-14 15:04:24 +01:00
Jim Ferenczi 23d11ed729 Fix numeric terms aggregations with includes/excludes and minDocCount=0 (#22141)
For minDocCount=0 the numeric terms aggregator should also check the includes/excludes when buckets with empty count are added to the result.
This change fixes this bug and adds a test for it.

Fixes #22140
2016-12-14 12:37:11 +01:00
Adrien Grand 149ef74b26 Fix `missing` on aggs on `boolean` fields. (#22135)
The creation of the `ValuesSource` used to pass `DateTimeZone.UTC` as a time
zone all the time in case of empty fields in spite of the fact that all doc
value formats but the date one reject this parameter.

This commit centralizes the creation of the `ValuesSource` and adds unit tests
to it.

Closes #22009
2016-12-14 10:03:09 +01:00
Daniel Mitterdorfer 7e5058037b Enable strict duplicate checks for JSON content
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.

As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.

Closes #19614
2016-12-14 09:35:53 +01:00
Nik Everett 49bdd29f91 Consolidate more parser creation into ESTestCase
This will make it easier to add the forthcoming required argument,
`NamedXContentRegistry`.
2016-12-13 20:28:41 -05:00
Jason Tedor 0d195f1afa Avoid corruption when deserializing booleans
Today we write 0x00 or 0x01 for false or true when serializing a boolean
(and 0x02 for null when serializing an optional boolean) but we
deserialize any non-zero byte to true (except when deserializing an
optional boolean in which case we deserialize 0x02 to null, 0x01 to
true, and any other non-zero byte to false). This too easily allows
corruption into the stream. Instead, we should mark the stream as
corrupted and stop deserializing. This catches when we try to
deserialize something as a boolean that is not a boolean.

Relates #22152
2016-12-13 20:10:05 -05:00
Jason Tedor 510ad7b9c7 Add shutdown hook for closing CLI commands
This commit enables CLI commands to be closeable and installs a runtime
shutdown hook to ensure that if the JVM shuts down (as opposed to
aborting) the close method is called.

It is not enough to wrap uses of commands in main methods in
try-with-resources blocks as these will not run if, say, the virtual
machine is terminated in response to SIGINT, or system shutdown event.

Relates #22126
2016-12-13 19:10:11 -05:00
Nik Everett 872984d21a Continue consolidating `XContentParser` construction in tests (#22145)
Consolidate more parser creation in tests

Moves more parser creation in tests to the `createParser` methods
in `ESTestCase`.
2016-12-13 17:22:39 -05:00
Jason Tedor 9ad49faa86 Fix for loop that is bad and should feel bad
This commit fixes a for loop that reverses the order of shard stats
coming off the wire, and is really hard to read anyway (with the
post-increment in the loop initializer).

Relates #22150
2016-12-13 15:27:42 -05:00
Simon Willnauer 7a9b667e98 Introduce a low level protocol handshake (#22094)
Today we rely on the version that the API user passes in together with the DiscoveryNode. This commit introduces a low level handshake where nodes exchange their version to be used with the transport protocol that is executed every time a connection to a node is established. This, on the one hand allows to change the wire protocol based on the version we are talking to even without a full cluster restart. Today we would need to carry on a BWC layer across major versions but with a handshake we can rely on the fact that the latest version of the previous minor executes a handshake and uses the latest protocol version across all communication with the N+1 version nodes.

This change is yet fully backwards compatible, a followup PR will remove the BWC in 6.0 once this has been back-ported to the 5.x branch
2016-12-13 21:06:23 +01:00
Nik Everett ce86405394 Start to centralize creation of XContentParser in tests (#22096)
Starts to centralize creation of the `XContentParser` in
`protected final` methods on `ESTestCase`. The idea is to enable
adding `NamedXContentRegistry` relatively easily by giving tests
a single place they can override to define the
`NamedXContentRegistry`. Since `NamedXContentRegistry` doesn't
exist yet neither does the override point.

This doesn't attempt to migrate all the tests to calling the
new methods to build the parsers. I wanted to make this so we
could review the concept and then I'll merge a followup to
migrate the tests.
2016-12-13 11:22:15 -05:00
markharwood 4c6d17a176 Added tests for toXContent and fromXContent for IncludeExclude class.
New REST test revealed an issue with inconsistent hashing in partitioned
term tests which is also fixed in this change.

Closes #22102
2016-12-13 15:23:09 +00:00
Clinton Gormley 28397c9594 Added version 2.4.3 plus bwc indices 2016-12-13 16:15:52 +01:00
Adrien Grand 049fd3991c Remove `AggregationContext`. (#22124)
This class is just a wrapper around `SearchContext`, so let's use
`SearchContext` directly. The change is mechanical, except the
`ValuesSourceConfig` class, where I moved the logic to get a `ValuesSource`
given a config.
2016-12-13 09:09:40 +01:00
Lee Hinman a4e8b5d952 Throw an exception on unrecognized "match_mapping_type"
When using dynamic templates, ES will now throw an exception if a
`match_mapping_type` is used that doesn't correspond to an actual type.

Relates to #17285
2016-12-12 09:59:48 -07:00
Simon Willnauer b667ff46c4 Allow plugins to install bootstrap checks (#22110)
Plugins also have the need to provide better OOTB experience by configuring
defaults unless the plugin is used in _production_ mode. This change exposes
the bootstrap check infrastructure as part of the plugin API to allow plugins
to specify / install their own bootstrap checks if necessary.
2016-12-12 17:35:00 +01:00
Peter Dyson 7904028fef Merge pull request #22105 from geekpete/master
Fix Fuziness typo in multimatch error message and associated tests
2016-12-12 23:13:20 +10:00
Luca Cavanna 6d987a9b69 Remove support for empty queries (#22092)
Our query DSL supports empty queries (`{}`), which have a different meaning depending on the query that holds it, either ignored, match_all or match_none. We deprecated the support for empty queries in 5.0, where we log a deprecation warning wherever they are used.

The way we supported it once we moved query parsing to the coordinating node was having an Optional<QueryBuilder> return type in all of our parse methods (called fromXContent). See #17624. The central place for this was QueryParseContext#parseInnerQueryBuilder. We can now remove all the optional return types and simply throw an exception whenever an empty query is found.
2016-12-12 12:37:12 +01:00
Luca Cavanna a42bee5d60 Don't output empty ext object in SearchSourceBuilder#toXContent (#22093)
We shouldn't output an empty ext object if no ext sections have been set to the SearchSourceBuilder.

Closes #20969
2016-12-12 12:10:23 +01:00
Luca Cavanna 73cf002293 Un-deprecate fuzzy query (#22088)
When we decided to deprecate and remove fuzzy query in #15760, we didn't realize we would take away the possibililty for uses to use a fuzzy query as part of a span query, which is not possible using match query. This means we have to go back and un-deprecate fuzzy query, which will not be removed.

Closes #15760
2016-12-12 12:09:16 +01:00
Peter Dyson d339814f77 Fix Fuziness typo in multimatch error message and associated tests 2016-12-12 11:33:31 +10:00
Artur Nowosielski 726f5dccc0 Rewrite filter queries in FiltersAggregationBuilder (#22076)
Queries must be rewritten before the query phase executes otherwise non-executable queries like `wrapper` query or `terms`  will fail or queries that require resources like script service can't access these service unless rewritten.

Relates to #21303
2016-12-11 14:37:12 +01:00
Masaru Hasegawa 3df2a086d4 Resolve index names in indices_boost
This change allows specifying alias/wildcard expression in indices_boost.
And added another format for specifying indices_boost. It accepts array of index name and boost pair.
If an index is included in multiple aliases/wildcard expressions, the first match will be used.
With new format, old format is marked as deprecated.

Closes #4756
2016-12-11 21:41:49 +09:00
Simon Willnauer 20ff703e07 Fix IncludeExclude parsing
`include` / `exclude` in terms / sig-terms aggs seems completely broken
and massively untested. This commit makes the TermsTests pass again that
randomly use `include` / `exclude`. This class must be tested individually
and we need real integ tests that use xcontent that use this feature.
2016-12-11 09:55:53 +01:00
Yannick Welsch 68f30cae0c [TEST] Unmute testRestoreUnsupportedSnapshots 2016-12-10 13:49:47 +01:00
Yannick Welsch 4831632a6f [TEST] Handle legacy snapshots as if they don't exist anymore
An earlier commit removed BWC for pre-5.0 snapshots, which also meant removing the capability to load pre-5.0 snapshots. In 6.0, such snapshots are now
invisible and must be treated by the BWC tests in that way.
2016-12-10 13:49:47 +01:00
Yannick Welsch 4ad85c38c3 Throw NoSuchFileException to correctly adhere to readBlob contract
URLBlobContainer can in certain situations throw a FileNotFoundException. To fulfill the contract of the readBlob method it should throw a NoSuchFileException instead when the given blob cannot be found.
2016-12-10 13:49:47 +01:00
Simon Willnauer 01d67e09b9 Detach handshake from connect to node (#22037)
Today we connect and publish the nodes connection before we execute a
handshake with the node we connect to. In the case of connecting to a node
that won't pass the handshake this connection is already `published` and other
code paths can use it. This commit detaches the connection and the publish of the
connection such that `TransportService` can do a handshake before actually connect
and publish the connection.
2016-12-10 10:03:26 +01:00
Nik Everett 3adefb7b4a Begin centralizing XContentParser creation into RestRequest (#22041)
To get #22003 in cleanly we need to centralize as much `XContentParser` creation as possible into `RestRequest`. That'll mean we have to plumb the `NamedXContentRegistry` into fewer places.

This removes `RestAction.hasBody`, `RestAction.guessBodyContentType`, and `RestActions.getRestContent`, moving callers over to `RestRequest.hasContentOrSourceParam`, `RestRequest.contentOrSourceParam`, and `RestRequest.contentOrSourceParamParser` and `RestRequest.withContentOrSourceParamParserOrNull`. The idea is to use `withContentOrSourceParamParserOrNull` if you need to handle requests without any sort of body content and to use `contentOrSourceParamParser` otherwise.

I believe the vast majority of this PR to be purely mechanical but I know I've made the following behavioral change (I'll add more if I think of more):
* If you make a request to an endpoint that requires a request body and has cut over to the new APIs instead of getting `Failed to derive xcontent` you'll get `Body required`.
* Template parsing is now non-strict by default. This is important because we need to be able to deprecate things without requests failing.
2016-12-09 20:23:02 -05:00
Nik Everett ddade1b5ac Improve the error message if task and node isn't found (#22062)
Improves the error message returned when looking up a task that
belongs to a node that is no longer part of the cluster. The new
error message tells the user that the node isn't part of the cluster.
This is useful because if you start a task and the node goes down
there isn't a record of the task at all. This hints to the user that
the task might have died with the node.

Relates to #22027
2016-12-09 15:50:46 -05:00
Igor Motov 93b5e55660 Restores the original default format of search slow log
In 5.0, the search slow log switched to the multi-line format with no option to get back to the origin single-line format that was used prior to 5.0 by default. This commit removes the reformat option from the search slow log and returns the search slow log back to the single-line format.

Closes #21711
2016-12-09 12:38:28 -05:00
Yannick Welsch b20b160a5e Allow flush/force_merge/upgrade on shard marked as relocated (#22078)
A shard that is locally marked as relocated, but where the relocation target shard has not been activated yet by the master, can still receive index operations, which in return can lead to flushes being triggered. Flushing is currently (wrongly) prohibited on shards marked as relocated, which makes the flushing process go into an endless retry loop and log warnings until the shard is closed. This commit fixes this situation by allowing flush, force_merge and upgrade operations to run on shards that are marked as relocated.
2016-12-09 17:56:40 +01:00
Nik Everett bcef1e7452 Better error message when _parent isn't an object (#21987)
If you make a mistake and specify a mapping like:
```
{
  "parent": {
    "properties": {}
  },
  "child": {
    "_parent": "parent",
    "properties": {}
  }
}
```

then the error message you get back amounts to
`Failed to parse mapping for [child]: can't cast a String to a Map`.
Since it doens't tell you *which* string can't be cast to a map you
have to dig through the stack trace to figure out what to fix. This
replaces the error message with:
```
Failed to parse mapping [child]: [_parent] must be an object containing [type]
```
so you can tell that the problem is with the `parent` field.
2016-12-09 11:33:31 -05:00
Yannick Welsch a724f4eb61 Don't update nodes list when stepping down as master (#22049)
This commit simplifies the node update logic so that nodes are never removed from the cluster state when the cluster state is not published.
2016-12-09 14:55:48 +01:00
Christoph Büscher 2592ff86ce Add fromXContent to InternalNestedIdentity
This adds a fromXContent method and unit test to InternalNestedIdentity so we can parse it as part of a search response. This is part of the preparation for parsing search responses on the client side.
2016-12-09 14:52:06 +01:00
Yannick Welsch db0660a7ea Reject external versioning and explicit version numbers on create (#21998)
Fixes an issue where indexing requests with operation type "create" auto-convert external versioning to internal versioning and silently ignore the version number instead of failing with an error message.
2016-12-09 14:21:22 +01:00
Michael McCandless 613a1a6a18 Add stored binary fields to static backwards compatibility indices tests (#22054)
Add stored binary fields to static backwards compatibility indices tests
2016-12-09 05:32:40 -05:00
Adrien Grand 6714e02bef Mute RestoreBackwardsCompatIT.testRestoreUnsupportedSnapshots. 2016-12-09 10:41:07 +01:00
Adrien Grand 787519ee4c Fix `other_bucket` on the `filters` agg to be enabled if a key is set. (#21994)
Closes #21951
2016-12-09 09:48:48 +01:00
Adrien Grand 1bdf4a2c5b Partition-based include-exclude does not implement equals/hashcode/serialization correctly. (#22051) 2016-12-09 09:48:16 +01:00
Adrien Grand 9524c81af9 Document the `locale` option of the `date` field. (#22050)
This also adds another level of protection against using the default locale.
Relates to https://discuss.elastic.co/t/mapping-for-12h-date-format/68433/3.
2016-12-09 09:45:53 +01:00
Adrien Grand 36f598138a Start using `ObjectParser` for aggs. (#22048)
This is an attempt to start moving aggs parsing to `ObjectParser`. There is
still A LOT to do, but ObjectParser is way better than the way aggregations
parsing works today. For instance in most cases, we reject numbers that are
provided as strings, which we are supposed to accept since some client languages
(looking at you Perl) cannot make sure to use the appropriate types.

Relates to #22009
2016-12-09 09:45:16 +01:00
Ryan Ernst b1cef5fdf8 Remove 2.0 prerelease version constants (#22004)
* Remove 2.0 prerelease version constants

This is a start to addressing #21887. This removes:
* pre 2.0 snapshot format support
* automatic units addition to cluster settings
* bwc check for delete by query in pre 2.0 indexes
2016-12-08 21:48:35 -08:00
Igor Motov 7f79c99e9a Add descriptions to bulk tasks
Related to #21768
2016-12-08 21:59:52 -05:00
Lee Hinman ef64d230e7 Merge remote-tracking branch 'dakrone/index-seq-id-and-primary-term' 2016-12-08 19:47:21 -07:00
Lee Hinman ee22a477df Add internal _primary_term doc values field, fix _seq_no indexing
This adds the `_primary_term` field internally to the mappings. This field is
populated with the current shard's primary term.

It is intended to be used for collision resolution when two document copies have
the same sequence id, therefore, doc_values for the field are stored but the
filed itself is not indexed.

This also fixes the `_seq_no` field so that doc_values are retrievable (they
were previously stored but irretrievable) and changes the `stats` implementation
to more efficiently use the points API to retrieve the min/max instead of
iterating on each doc_value value. Additionally, even though we intend to be
able to search on the field, it was previously not searchable. This commit makes
it searchable.

There is no user-visible `_primary_term` field. Instead, the fields are
updated by calling:

```java
index.parsedDoc().updateSeqID(seqNum, primaryTerm);
```

This includes example methods in `Versions` and `Engine` for retrieving the
sequence id values from the index (see `Engine.getSequenceID`) that are only
used in unit tests. These will be extended/replaced by actual implementations
once we make use of sequence numbers as a conflict resolution measure.

Relates to #10708
Supercedes #21480

P.S. As a side effect of this commit, `SlowCompositeReaderWrapper` cannot be
used for documents that contain `_seq_no` because it is a Point value and SCRW
cannot wrap documents with points, so the tests have been updated to loop
through the `LeafReaderContext`s now instead.
2016-12-08 19:47:03 -07:00
Jason Tedor c9882dd1a0 Avoid NPE in NodeService#stats if HTTP is disabled
This commit adds safety against an NPE if HTTP stats are requested but
HTTP is disabled on a node.

Relates #22060
2016-12-08 19:59:02 -05:00