Commit Graph

2248 Commits

Author SHA1 Message Date
Armin Braun c5b3ac5578
SNAPSHOTS: Allow Parallel Restore Operations (#36397)
* Enable parallel restore operations
* Add uuid to restore in progress entries to uniquely identify them
* Adjust restore in progress entries to be a map in cluster state
* Added tests for:
   * Parallel restore from two different snapshots
   * Parallel restore from a single snapshot to different indices to test uuid identifiers are correctly used by `RestoreService` and routing allocator
   * Parallel restore with waiting for completion to test transport actions correctly use uuid identifiers
2018-12-14 11:39:23 +01:00
Nhat Nguyen 1ae53e6b20
TEST: Disable merges in testRebuildLocalCheckpointTracker (#36625)
In this test, we keep track of a list of index commits then verify that
we reload exactly every operation from the safe commit. If a background
merge is triggered, then we might have a new index commit which is not
recorded in the tracking list. This change disables merges in the test.

Closes #36470
2018-12-14 05:25:36 -05:00
Christoph Büscher 573b6325e7
[Tests] Fix edge case in SimpleQueryStringBuilderTests (#36611)
A previous fix of a similar problem in #35201 wasn't general enough, we also
need to catch cases where the randomly generated query string starts with some
version of "now" and hits a date field.

Closes #36595
2018-12-14 11:10:45 +01:00
Daniel Mitterdorfer b7ba2fa74b
Fix line length for VersionTests
Relates #34884
Relates #36585
2018-12-14 10:07:27 +01:00
Daniel Mitterdorfer 42f094952b
Fix line length for IndexLifecycleActionIT
Relates #34884
Relates #36586
2018-12-14 10:00:37 +01:00
Ryan Ernst 254d1e8f22
Core: Add backcompat for joda time formats (#36531)
This commit adds deprecation warnings when using format specifiers with
joda data formats that will change with java time. It also adds the "8"
prefix which may be used to force the new java time format parsing.
2018-12-13 12:26:51 -08:00
Christoph Büscher b33ff16d62
Remove deprecated `useDisMax` from MultiMatchQuery (#36488)
The getters and setters for useDisMax() have been deprecated since at least 6.0,
also there hasn't been any reference to the query parameter in the
documentation. Removing it from the builder and tests and replacing it with
`tieBreaker(1.0f)` where necessary.
2018-12-13 20:13:16 +01:00
Tanguy Leroux 1f6f236d44 Merge branch 'master' into close-index-api-refactoring 2018-12-13 17:40:06 +01:00
Tanguy Leroux 8e5dd20efb
[Close Index API] Refactor MetaDataIndexStateService (#36354)
The commit changes how indices are closed in the MetaDataIndexStateService. 
It now uses a 3 steps process where writes are blocked on indices to be closed, 
then some verifications are done on shards using the TransportVerifyShardBeforeCloseAction 
added in #36249, and finally indices states are moved to CLOSE and their routing 
tables removed.

The closing process also takes care of using the pre-7.0 way to close indices if the 
cluster contains mixed version of nodes and a node does not support the TransportVerifyShardBeforeCloseAction. It also closes unassigned indices.

Related to #33888
2018-12-13 17:36:23 +01:00
Tal Levy b820d7c617
fix MultiValuesSourceFieldConfig toXContent (#36525)
This commit turns MultiValuesSourceFieldConfig into a proper
ToXContentObject for easy testing and verification of its
to/from XContent methods.

Closes #36474.
2018-12-13 08:17:36 -08:00
Tim Brooks f998e04c34
Remove usages of `MockTcpTransport` from zen tests (#36579)
We are attempting to replace the usage of the `MockTcpTransport` with
the `MockNioTransport`. This commit replaces usages of
`MockTcpTransport` in two zen test cases.
2018-12-13 09:10:18 -07:00
Jason Tedor 2afa7faefd
Override the JVM DNS cache policy (#36570)
When a security manager is present, the JVM will cache positive hostname
lookups indefinitely. This can be problematic, especially in the modern
world with cloud services where DNS addresses can change, or
environments using Docker containers where IP addresses could be
considered ephemeral. This behavior impacts cluster discovery,
cross-cluster replication and cross-cluster search, reindex from remote,
snapshot repositories, webhooks in Watcher, external authentication
mechanisms, and the Elastic Stack Monitoring Service. The experience of
watching a DNS lookup change yet not be reflected within Elasticsearch
is a poor experience for users. The reason the JVM has this is guard
against DNS cache posioning attacks. Yet, there is already a defense in
the modern world against such attacks: TLS. With proper certificate
validation, even if a resolver falls prey to a DNS cache poisoning
attack, using TLS would neuter the attack. Therefore we have a policy
with dubious security value that significantly impacts usability. As
such we make the usability/security tradeoff towards usability, since
the security risks are very low. This commit introduces new system
properties that Elasticsearch observes to override the JVM DNS cache
policy.
2018-12-13 10:23:45 -05:00
Tanguy Leroux 40953d70ca Merge branch 'master' into close-index-api-refactoring 2018-12-13 15:45:56 +01:00
Igor Motov 7446f75346
Geo: Adds a name of the field to geopoint parsing errors (#36529)
Adds the field name and type to geo_point parsing errors.

Closes #15965
2018-12-13 17:32:45 +04:00
Armin Braun 088d3f39c3
MINOR: Cleanup Dead Code in o.e.c.metadata Package (#36587) 2018-12-13 12:54:22 +01:00
David Roberts 13cb0fb98b
Periodically try to reassign unassigned persistent tasks (#36069)
Previously persistent task assignment was checked in the
following situations:

- Persistent tasks are changed
- A node joins or leaves the cluster
- The routing table is changed
- Custom metadata in the cluster state is changed
- A new master node is elected

However, there could be situations when a persistent
task that could not be assigned to a node could become
assignable due to some other change, such as memory
usage on the nodes.

This change adds a timed recheck of persistent task
assignment to account for such situations.  The timer
is suspended while checks triggered by cluster state
changes are in-flight to avoid adding burden to an
already busy cluster.

Closes #35792
2018-12-13 09:15:27 +00:00
Armin Braun 34d7cc13c1
SETTINGS: Correctly Identify Noop Updates (#36560)
* We should compare the target value with the to be applied value before interpreting the update as a change
* This speeds up the test failing in #36496 considerably by preventing state updates on noop setting updates
2018-12-13 10:02:54 +01:00
Boaz Leskes f6b5d7e013
Add sequence numbers based optimistic concurrency control support to Engine (#36467)
This commit add support to engine operations for resolving and verifying the sequence number and
primary term of the last modification to a document before performing an operation. This is
infrastructure to move our (optimistic concurrency control)[http://en.wikipedia.org/wiki/Optimistic_concurrency_control] API to use sequence numbers instead of internal versioning.

Relates #36148 
Relates #10708
2018-12-13 08:08:40 +01:00
Tal Levy cd1bec3a06
[refactor] add Environment in BootstrapContext (#36573)
There are certain BootstrapCheck checks that may need access environment-specific
values. Watcher's EncryptSensitiveDataBootstrapCheck passes in the node's environment
via a constructor to bypass the shortcoming in BootstrapContext. This commit
pulls in the node's environment into BootstrapContext.

Another case is found in #36519, where it is useful to check the state of the
data-path. Since PathUtils.get and Paths.get are forbidden APIs, we rely on
the environment to retrieve references to things like node data paths.

This means that the BootstrapContext will have the same Settings used in the
Environment, which currently differs from the Node's settings.
2018-12-12 21:07:21 -08:00
Mayya Sharipova d40037c91e
Deprecate uses of _type as a field name in queries (#36503) 2018-12-12 21:21:53 -05:00
Tal Levy 9c1cdea839
[cleanup] remove deprecated references to dataWithClusterFiles (#36574)
data files under the cluster name subdirectory has been deprecated and was
meant to be removed in 6.0. This commit removes some leftover referrences to
these paths.
2018-12-12 16:28:11 -08:00
Julie Tibshirani 71a39d10be
Make sure that BWC tests run successfully, even with types deprecation messages. (#36511) 2018-12-12 12:57:32 -08:00
Alan Woodward 9ac7359643
Update lucene to snapshot-7e4555a2fd (#36563)
Includes the following:

* Reversion of doc-values changes in LUCENE-8374; we are interested in seeing if this 
  has an effect on benchmarks for node-stats and index-stats
* More improvements to docvalues updates
2018-12-12 20:18:32 +00:00
Julie Tibshirani 33152f648f
Fix some inconsistencies in the types deprecation code. (#36517)
* Make sure to test conversion for both typed and typeless HLRC requests.
* Update a few more statements to deprecatedAndMaybeLog.
* Make sure Rest*SearchTemplateActionTests extend RestActionTestCase.
2018-12-12 10:38:02 -08:00
Tim Brooks 7f612d5dd8
Always compress based on the settings (#36522)
Currently TransportRequestOptions allows specific requests to request
compression. This commit removes this and always compresses based on the
settings. Additionally, it removes TransportResponseOptions as they
are unused.

This closes #36399.
2018-12-12 09:39:15 -07:00
Armin Braun 02d0f163a4
MINOR: Cleanup Dead Code in Snapshots Package (#36528) 2018-12-12 16:48:34 +01:00
Tanguy Leroux dbcafbefc9 Merge branch 'master' into close-index-api-refactoring 2018-12-12 16:45:31 +01:00
Simon Willnauer ff5dd14753
Fix test failures related to file corruption (#36530)
* Fix CorruptFileIT to also take last DV generation into account

We currently only prune old .liv generations. With soft_deletes it's important
to also prune DV generations.

* Fix CorruptionUtils to skip the footer bytes after the checksum is read.

Today we read a broken checksum since we also checksum the 8 footer bytes that include
the checksum algorithm and the footer magic.

Closes #36526
2018-12-12 16:21:02 +01:00
Tim Brooks 3a56bb0924
Do not zero out pages for bytes output stream (#36523)
Currently, the `BytesStreamOutput` always zeros out the underlying byte
pages when they are acquired. This should not be necessary as the stream
overwrites the underlying bytes as serialization occurs.
2018-12-12 07:55:17 -07:00
Tim Brooks e63d52af63
Move page size constants to PageCacheRecycler (#36524)
`PageCacheRecycler` is the class that creates and holds pages of arrays
for various uses. `BigArrays` is just one user of these pages. This
commit moves the constants that define the page sizes for the recycler
to be on the recycler class.
2018-12-12 07:00:50 -07:00
Jason Tedor bdb1e0e04e
Make node field in JoinRequest private (#36405)
This commit moves the node field in the JoinRequest object to be a
private field, adding a dedicated accessor. This is a minor breaking
change in that it is no longer possible for all callers to overwrite
this field, but that is a feature.
2018-12-12 08:50:21 -05:00
David Turner aa43e0b2cc
[Zen2] Migrate no-master-block integration tests (#36502)
This change follows up on #36478 by migrating the affected integration tests to
use Zen2.
2018-12-12 12:52:34 +00:00
Ryan Ernst c4f4378006
Core: Rework multi date formatter merging (#36447)
This commit moves the MergedDateFormatter to a package private class and
reworks joda DateFormatter instances to use that instead of a single
DateTimeFormatter with multiple parsers. This will allow the java and
joda multi formats to share the same format parsing method in a
followup.
2018-12-11 23:47:44 -08:00
Nhat Nguyen 1bb6f844fe AwaitsFix testCorruptFileThenSnapshotAndRestore
Tracked at #36526
2018-12-12 02:03:23 -05:00
Nhat Nguyen 3fb5a12b30 Upgrade to Lucene-8.0.0-snapshot-61e448666d (#36518)
Includes:
- LUCENE-8602: Share TermsEnum if possible while applying DV updates
2018-12-12 06:47:40 +01:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later (#36141)
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates #33222

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
Tim Brooks 797f985067
Add version to handshake requests (#36171)
Currently our handshake requests do not include a version. This is
unfortunate as we cannot rely on the stream version since it is not the
sending node's version. Instead it is the minimum compatibility version.
The handshake request is currently empty and we do nothing with it. This
should allow us to add data to the request without breaking backwards
compatibility.

This commit adds the version to the handshake request. Additionally, it
allows "future data" to be added to the request. This allows nodes to craft
a version compatible response. And will properly handle additional data in
future handshake requests. The proper handling of "future data" is useful
as this is the only request where we do not know the other node's version.

Finally, it renames the TcpTransportHandshaker to
TransportHandshaker.
2018-12-11 16:09:28 -07:00
Mayya Sharipova 2f18325384
Deprecate types in update_by_query and delete_by_query (#36365)
Relates to #35190
2018-12-11 17:09:59 -05:00
Luca Cavanna dafea3cc23
Minor search controller changes (#36479)
This commit contains a few minor changes to our search code:

- adjust the visibility of a couple of methods in our search code to package private from public or protected.
- make some of the `SearchPhaseController` methods static where possible
- rename one of the `SearchPhaseController#reducedQueryPhase` methods (used only for scroll requests) to `reducedScrollQueryPhase` without the `isScrollRequest` argument which was always set to `true`
- replace leniency in `SearchPhaseController#setShardIndex` with an assert to make sure that we never set the shard index twice
- remove two null checks where the checked field can never be null
- resolve an unchecked warning
- replace `List#toArray` invocation that creates an array providing the true size with array creation of length 0
- correct a couple of typos in comments
2018-12-11 20:24:29 +01:00
Luca Cavanna fb18b35347
Add default methods to DocValueFormat (#36480)
The different `DocValueFormat` implementors throw `UnsupportedOperationException` for methods that they don't support. That is perfectly fine, and quite common as not all implementors support all of the possible formats. This makes it hard though to trace back which implementors support which formats as they all implement the same methods.

This commit introduces default methods in the `DocValueFormat` interface so that all methods throw `UnsupportedOperationException` by default. This way implementors can override only the methods that they specifically support.
2018-12-11 20:19:01 +01:00
Tim Brooks 790f8102e9
Modify `BigArrays` to take name of circuit breaker (#36461)
This commit modifies BigArrays to take a circuit breaker name and
the circuit breaking service. The default instance of BigArrays that
is passed around everywhere always uses the request breaker. At the
network level, we want to be using the inflight request breaker. So this
change will allow that.

Additionally, as this change moves away from a single instance of
BigArrays, the class is modified to not be a Releasable anymore.
Releasing big arrays was always dispatching to the PageCacheRecycler,
so this change makes the PageCacheRecycler the class that needs to be
managed and torn-down.

Finally, this commit closes #31435 be making the serialization of
transport messages use the inflight request breaker. With this change,
we no longer push the global BigArrays instnace to the network level.
2018-12-11 11:55:41 -07:00
Martijn van Groningen 405a38131b
added 6.5.4 version 2018-12-11 19:38:59 +01:00
David Turner 21d91f1268
[Zen2] Respect the no_master_block setting (#36478)
Today the Zen2 coordinator only applies a write block when there is no known
elected master, ignoring the `discovery.zen.no_master_block` setting. This
commit resolves this, applying the correct block according to the configuration
instead.
2018-12-11 16:40:19 +00:00
Nhat Nguyen 084e06e481
Require soft-deletes when access changes snapshot (#36446)
Today we do not enforce soft-deletes when accessing the Lucene changes
snapshot. This might lead to internal errors because we assume
soft-deletes are enabled in that code path.
2018-12-11 11:18:10 -05:00
Andrey Ershov 8b821706cc
Switch more tests to zen2 (#36367)
1. CCR tests work without any changes
2. `testDanglingIndices` require changes the source code (added TODO).
3. `testIndexDeletionWhenNodeRejoins` because it's using just two
nodes, adding the node to exclusions is needed on restart.
4. `testCorruptTranslogTruncationOfReplica` starts dedicated master
one, because otherwise, the cluster does not form, if nodes are stopped
and one node is started back.
5. `testResolvePath` needs TEST cluster, because all nodes are stopped
at the end of the test and it's not possible to perform checks needed
by SUITE cluster.
6. `SnapshotDisruptionIT`. Without changes, the test fails because Zen2
retries snapshot creation as soon as network partition heals. This
results into the race between creating snapshot and test cleanup logic
(deleting index). Zen1 on the
other hand, also schedules retry, but it takes some time after network
partition heals, so cleanup logic executes latter and test passes. The
check that snapshot is eventually created is added to
the end of the test.
2018-12-11 17:12:17 +01:00
David Turner c3a6d1998a
Log [initial_master_nodes] on formation failure (#36466)
Today we log a slightly cryptic "cluster bootstrapping is disabled on this
node" message if bootstrapping hasn't been configured. Since there is today
only one way to bootstrap the cluster it seems preferable to spell out exactly
which setting is missing.
2018-12-11 12:53:05 +00:00
Christian Schneider 00eadd93eb converting ForbiddenPatternsTask to .java (#36194)
* converting ForbiddenPatternsTask to java impl & unit tests
2018-12-11 14:15:44 +02:00
Yu d01b30acba lower fielddata circuit breaker's default limit (#27162)
* Lower fielddata circuit breaker default limit

Lower fielddata circuit breaker default limit from 60% to 40% as we have
moved to doc_values for most of the cases.

* merge master in

* update tests

* update docs
2018-12-11 11:30:58 +01:00
Yannick Welsch 71cf6faaad
Only turn to follower when term bumping on follower check (#36449)
Deals with a situation where a follower becomes disconnected from the leader, but only for such a
short time where it becomes candidate and puts up a NO_MASTER_BLOCK, but then receives a
follower check from the leader. If the leader does not notice the node disconnecting, it is important
for the node not to be turned back into a follower but try and join the leader again.

We still should prefer the node into a follower on a follower check when this follower check triggers
a term bump as this can help during a leader election to quickly have a leader turn all other nodes
into followers, even before the leader has had the chance to transfer a possibly very large cluster
state.

Closes #36428
2018-12-11 11:22:23 +01:00
Yannick Welsch d8e3d97a7d
Add discovery types to cluster stats (#36442)
Adds information about the used discovery types to the cluster stats, similar as we have for the network types.
2018-12-11 10:35:58 +01:00
David Turner 2bf0583fec
Exclude Zen2 fault detection traffic from tracer (#36444)
Today we exclude `internal:discovery/zen/fd*` from the transport tracer by
default. This commit excludes the new fault detection mechanism similarly.
2018-12-11 09:02:01 +00:00
Julie Tibshirani 87831051dc
Deprecate types in explain requests. (#35611)
The following updates were made:
- Add a new untyped endpoint `{index}/_explain/{id}`.
- Add deprecation warnings to Rest*Action, plus tests in Rest*ActionTests.
- For each REST yml test, make sure there is one version without types, and another legacy version that retains types (called *_with_types.yml).
- Deprecate relevant methods on the Java HLRC requests/ responses.
- Update documentation (for both the REST API and Java HLRC).
2018-12-10 19:45:13 -08:00
Ryan Ernst a0da390df2
Scripting: Switch watcher to use joda bwc time objects (#35966)
This commit converts the watcher execution context to use the joda
compat java time objects. It also again removes the joda methods from
the painless whitelist.
2018-12-10 17:29:25 -08:00
Julie Tibshirani 99f89cd3b4
Deprecate types in get, exists, and multi get. (#35930)
For each API, the following updates were made:
- Add deprecation warnings to `Rest*Action`, plus tests in `Rest*ActionTests`.
- For each REST yml test, make sure there is one version without types, and another legacy version that retains types (called *_with_types.yml).
- Deprecate relevant methods on the Java HLRC requests/ responses.
- Update documentation (for both the REST API and Java HLRC).
2018-12-10 17:22:48 -08:00
Nhat Nguyen 2a7edca59f
Upgrade to Lucene-8.0.0-snapshot-ef61b547b1 (#36450)
Includes:

- LUCENE-8598: Improve field updates packed values
- LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates
2018-12-10 16:33:49 -05:00
Jernej Klancic d615add1b1 Add pipeline parent validation for auto date histogram (#35670)
Allow `auto_date_histogram` as a valid parent agg for derivative, 
cumulative sum, moving average, moving function and serial 
differencing pipeline aggregations.

Since all these aggs share the same requirement (sequentially
ordered parent aggs), this commit also refactors to share
the same validation code so that any newly added aggs won't
be forgotten.

Closes #35578
2018-12-10 16:02:49 -05:00
Jim Ferenczi 75392adf60
[TEST] Convert SearchHitsTests to AbstractStreamableXContentTestCase (#36313)
This change adds a way to provide the content type of the rest serialization
tests when creating random instances. This is used by SearchHitsTests to ensure
that the internal members of the class are created with the same xContentType
and that equals can be used to compare an instances created from an XContent
view.
2018-12-10 20:41:20 +01:00
David Turner ca3f5c1e2e
Cancel GetDiscoveredNodesAction when bootstrapped (#36423)
Today the `GetDiscoveredNodesAction` waits, possibly indefinitely, to discover
enough nodes to bootstrap the cluster. However it is possible that the cluster
forms before a node has discovered the expected collection of nodes, in which
case the action will wait indefinitely despite the fact that it is no longer
required.

This commit changes the behaviour so that the action fails once a node receives
a cluster state with a nonempty configuration, indicating that the cluster has
been successfully bootstrapped and therefore the `GetDiscoveredNodesAction`
need wait no longer.

Relates #36380 and #36381; reverts 558f4ec278.
2018-12-10 17:23:03 +00:00
Ryan Ernst 1a36046421
Core: Convert FormatDateTimeFormatter to DateFormatter (#36396)
This commit creates JodaDateFormatter to replace
FormatDateTimeFormatter. It converts all uses of the old class
to DateFormatter to allow a future change to use JavaDateFormatter
when appropriate.
2018-12-10 08:06:41 -08:00
Yannick Welsch f79e602d9f Mute ClusterDisruptionIT.testSendingShardFailure
Relates to #36428
2018-12-10 13:53:43 +01:00
Igor Suhorukov 0d9e3adc95 Loop replace with Collection.removeIf() (#36351) 2018-12-10 11:26:24 +01:00
Yannick Welsch 6e6e63d01d
Zen2: Move all mixed-version REST tests to Zen2 (#36398)
Moves all remaining (rolling-upgrade and mixed-version) REST tests to use Zen2. To avoid adding
extra configuration, it relies on Zen2 being set as the default discovery type. This required a few
smaller changes in other tests. I've removed AzureMinimumMasterNodesTests which tests Zen1
functionality and dates from a time where host providers were not configurable and each cloud
plugin had its own discovery.type, subclassing the ZenDiscovery class. I've also adapted a few tests
which were unnecessarily adding addTestZenDiscovery = false for the same legacy reasons. Finally,
this also moves the unconfigured-node-name REST test to Zen2, testing the auto-bootstrapping
functionality in development mode when no discovery configuration is provided.
2018-12-10 11:00:57 +01:00
Nhat Nguyen 902d6f579a
Exposed engine must have all ops below gcp during rollback (#36159)
Today we expose a new engine immediately during Lucene rollback. The new
engine is started with a safe commit which might not include all
acknowledged operation. With this change, we won't expose the new engine
until it has recovered from the local translog.

Note that this solution is not complete since it's able to reserve only
acknowledged operations before the global checkpoint. This is because we
replay translog up to the global checkpoint during rollback. A per-doc
Lucene rollback would solve this issue entirely.

Relates #32867
2018-12-08 21:27:44 -05:00
Nhat Nguyen d41cf6ac9f
Use delCount of SegmentInfos to calculate numDocs (#36323)
Today, we iterate the bitset of hardLiveDocs to calculate the number of
live docs. This calculation might be expensive if we enable soft-deletes
(by default) for old indices whose soft-deletes was disabled previously
and had hard-deletes.

Once soft-deletes is enabled, we no longer hard-update or hard-delete
documents directly. We have hard-deletes in two scenarios: (1) from old
segments where soft-deletes was disabled, (2) when IndexWriter hits
non-aborted exceptions. These two cases, IW flushes SegmentInfos before
exposing the hard-deletes; thus we can use the hard-delete count of
SegmentInfos.
2018-12-08 21:24:36 -05:00
Tim Brooks cc3872d934
Reduce composite references at network level (#36400)
Creating `CompositeBytesReference` has more overhead than a single
`ByteBufferReference`. Many of our messages will be contained to a
single `ByteBuffer`. This commit avoids creating composite instances
when there is 0 or 1  underlying `ByteBuffers`.
2018-12-08 14:24:38 -07:00
David Turner 9f86e996fe
[Zen2] Support rolling upgrades from Zen1 (#35737)
We support rolling upgrades from Zen1 by keeping the master as a Zen1 node
until there are no more Zen1 nodes in the cluster, using the following
principles:

- Zen1 nodes will never vote for Zen2 nodes
- Zen2 nodes will, while not bootstrapped, vote for Zen1 nodes
- Zen2 nodes that were previously part of a mixed cluster will automatically
  (and unsafely) bootstrap themselves when the last Zen1 node leaves.
2018-12-08 07:33:35 +00:00
Ryan Ernst a27f2efca5
Core: Converge FormatDateTimeFormatter and DateFormatter apis (#36390)
This commit makes FormatDateTimeFormatter and DateFormatter apis close
to each other, so that the former can be removed in favor of the latter.
This PR does not change the uses of FormatDateTimeFormatter yet, so that
that future change can be purely mechanical.
2018-12-07 17:23:41 -08:00
Julie Tibshirani b15d1aebcf
For msearch templates, make sure to use the right name for deprecation logging. (#36344) 2018-12-07 14:50:47 -08:00
Tim Brooks 8a53f2b464
Implement basic `CcrRepository` restore (#36287)
This is related to #35975. It implements a basic restore functionality
for the CcrRepository. When the restore process is kicked off, it
configures the new index as expected for a follower index. This means
that the index has a different uuid, the version is not incremented, and
the Ccr metadata is installed.

When the restore shard method is called, an empty shard is initialized.
2018-12-07 15:27:04 -07:00
Ryan Ernst a998f4dec6
Core: Remove parseDefaulting from DateFormatter (#36386)
This commit removes the parseDefaulting method from DateFormatter,
bringing it more inline with the joda equivalent
FormatDateTimeFormatter. This method was only needed for the java
time implementation of DateMathParser. Instead, a DateFormatter now
returns an implementation of DateMathParser for the given format,
allowing the java time implementation to construct the appropriate date
math parser internally.
2018-12-07 12:40:14 -08:00
Julie Tibshirani 51e1d40dca
Small improvements related to types deprecation. (#36328)
* Make sure to use deprecatedAndMaybeLog for types deprecation messages.
* Introduce a common base class for Rest*Action tests.
2018-12-07 11:21:24 -08:00
Tim Brooks 5556204f81
Use MockNioTransport in MockTransportService (#36346)
The default transport used in the MockTransportService is the
MockTcpTransport. This commit changes that to be the
MockNioTransport.
2018-12-07 11:17:11 -07:00
David Turner 9d417984bd
[Zen2] Add warning if cluster fails to form fast enough (#35993)
* Add warning if cluster fails to form fast enough

Today if a leader is not discovered or elected then nodes are essentially
silent at INFO and above, and log copiously at DEBUG and below. A short delay
when electing a leader is not unusual, for instance if other nodes have not yet
started, but a persistent failure to elect a leader is a problem worthy of log
messages in the default configuration.

With this change, while there is no leader each node outputs a WARN-level log
message every 10 seconds (by default) indicating as such, describing the
current discovery state and the current quorum(s).

* Add note about whether the discovered nodes form a quorum or not

* Introduce separate ClusterFormationFailureHelper

... and back out the unnecessary changes elsewhere

* It can be volatile
2018-12-07 17:23:20 +00:00
Nhat Nguyen f2df0a5be4
Remove LocalCheckpointTracker#resetCheckpoint (#34667)
In #34474, we added a new assertion to ensure that the
LocalCheckpointTracker is always consistent with Lucene index. However,
we reset LocalCheckpoinTracker in testDedupByPrimaryTerm cause this
assertion to be violated.

This commit removes resetCheckpoint from LocalCheckpointTracker and
rewrites testDedupByPrimaryTerm without resetting the local checkpoint.

Relates #34474
2018-12-07 12:22:20 -05:00
Jim Ferenczi a53e8653f2
Add support for inlined user dictionary in Nori (#36123)
Add support for inlined user dictionary in Nori

This change adds a new option called `user_dictionary_rules` to the
Nori a tokenizer`. It can be used to set additional tokenization rules
to the Korean tokenizer directly in the settings (instead of using a file).

Closes #35842
2018-12-07 15:26:08 +01:00
David Turner c32e4fb83f
[Zen2] Best-effort cluster formation if unconfigured (#36215)
In real deployments it is important that clusters are properly configured to
avoid accidentally forming multiple independent clusters at cluster
bootstrapping time. However we also expect to be able to unpack Elasticsearch
and start up one or more nodes without any up-front configuration, and have
them do their best to find each other and form a cluster after a few seconds.

This change adds a delayed automatic bootstrapping process to nodes that start
up with no relevant settings set to support the desired out-of-the-box
experience without compromising safety in properly-configured deployments.
2018-12-07 12:47:09 +00:00
Christoph Büscher 879397df70
Remove deprecated methods in SimpleQueryStringBuilder (#36341)
The `useAllFields`methods have already been deprecated in 6.0 and are save to remove in 7.0.
2018-12-07 11:15:16 +01:00
Christoph Büscher 562479b5a5
Remove deprecated methods in MoreLikeThisQueryBuilder (#36340)
The ids method is deprecated since 6.0 (at least) and only used in a few tests
internally, so we can safely remove it.
2018-12-07 11:14:41 +01:00
Adrien Grand 266b9bc796
Inner hits fail to propagate doc-value format. (#36310)
If you pass a doc-value format to inner hits, it gets ignored.
2018-12-07 09:57:12 +01:00
Alexander Reelsen 6a987415f8
Scripting: Properly support no-offset date formatting (#36316)
The conversion of timezones in JodaCompatibleZonedDateTime from joda to
java time requires the use of the DateUtils class to cater for corner
cases.

Closes #36306
2018-12-07 09:40:58 +01:00
Tanguy Leroux 593d8637c4 Merge branch 'master' into close-index-api-refactoring 2018-12-07 09:31:35 +01:00
Tanguy Leroux 3ca885e9ac
[Close Index API] Add TransportShardCloseAction for pre-closing verifications (#36249)
This pull request adds the TransportShardCloseAction which is a 
transport replication action that acquires all index shard permits for
 its execution. This action will be used in the future by the 
MetaDataIndexStateService in a new index closing process, where 
we need to execute some sanity checks before closing an index.

The action executes the following verifications on the primary and replicas:
 * there is no other on going operation active on the shard
 * the data node holding the shard knows that the index is blocked for writes
 * the shard's max sequence number is equal to the global checkpoint

When the verifications are done and successful, the shard is flushed.

Relates #33888
2018-12-07 09:30:27 +01:00
Tanguy Leroux 9691d0508c
Remove unused class DelegatingActionListener (#36257) 2018-12-07 09:20:24 +01:00
Armin Braun e6d190613f
[ZEN2] Use Zen2 in REST Tests (#36300) 2018-12-07 09:15:11 +01:00
Nhat Nguyen 10feb75eb7
Upgrade to Lucene-8.0.0-snapshot-aaa64d70159 (#36335)
Includes:

LUCENE-8594: DV update are broken for updates on new field
LUCENE-8590: Optimize DocValues update datastructures
LUCENE-8593: Specialize single value numeric DV updates

Relates #36286
2018-12-06 20:33:25 -05:00
Nhat Nguyen a7edf1ed1f
TEST: Ensure a manual flush always occur (#36188)
The test testLookupSeqNoByIdInLucene fails because it expects if any
change should be visible after a flush. However, that flush might be
ignored if the waitIfOngoing parameter is false (the default value), and
there is an ongoing flush triggered after merge is running.

Closes #35823
2018-12-06 18:37:54 -05:00
Ryan Ernst 37b3fc383f
Build: Use explicit deps on test tasks for check (#36325)
This commit moves back to use explicit dependsOn for test tasks on
check. Not all tasks extending RandomizedTestingTask should be run by
check directly.
2018-12-06 14:13:49 -08:00
Julie Tibshirani 3f3cde41d3
Deprecate types in termvector and mtermvector requests. (#36182)
* Add deprecation warnings to `Rest*TermVectorsAction`, plus tests in `Rest*TermVectorsActionTests`.
* Deprecate relevant methods on the Java HLRC requests/ responses.
* Update documentation (for both the REST API and Java HLRC).
* For each REST yml test, create one version without types, and another legacy version that retains types (called *_with_types.yml).
2018-12-06 10:23:15 -08:00
Armin Braun 96ee6e0d2a
TEST: Reenable RemoveCorruptedShardDataCommandIT.testCorruptIndex
* Merging the zen2 branch has made it impossible to reproduce the test failure here, likely as a result of the state now being written atomically
* Closes #36189
2018-12-06 17:55:40 +01:00
Christoph Büscher 65e15f07fb Add comments about need for explicit cast
Some IDEs (specifically Eclipse 4.8.0) needs some explicit casting for type
inference. This was added in a recent change but we should also have comments so
we remember why this needs to be there (at least for now).
2018-12-06 17:40:59 +01:00
David Turner ed1c5a0241
Introduce `zen2` discovery type (#36298)
With this change it is now possible to start a node running Zen2.
2018-12-06 16:20:08 +00:00
Andrey Ershov f0340d6d32
[Zen2] Storage layer WriteStateException propagation (#36052)
Currently, we only log that WriteStateException has occurred, in
GatewayMetaState.

This PR goal is to re-throw WriteStateException to upper layers.
If dirty flag is set to false, we wrap WriteStateException in
UncheckedIOException, we prefer unchecked exception not to add
explicit throws in the multitude of layers.
If dirty flag is set to true - the world is broken. And we need to
halt the JVM. Instead of explicit halting in GatewayMetaState, we
prefer to throw IOError, which will be subsequently handled by
ElasticsearchUncaughtExceptionHandler and JVM will be halted.

This PR also adds tests for WriteStateException.
2018-12-06 17:02:02 +01:00
Adrien Grand b08cffad48
Fix line length offenders in the o.e.search package. (#36223)
Relates #34884
2018-12-06 16:31:12 +01:00
Christoph Büscher f5fa1f4746
Help Eclipse with type inference for functions (#36301)
The Eclipse IDE java compiler seems to need some special hints about what types
some functions used in the tests return. Correcting this for some test that were
newly merged to master.
2018-12-06 16:04:25 +01:00
Armin Braun 98e66c6afe
SNAPSHOT: Increase Timeout to Stabilize Test (#36294)
* 30s is pretty close here, most runs take 25s of the timeout making the test unstable
* closes #36283
2018-12-06 13:43:28 +01:00
Yannick Welsch ee05ef1312 Merge branch 'zen2' 2018-12-06 08:31:46 +01:00
Jim Ferenczi 3ac64c1ec2 Fix total hits serialization of the search response (#36290)
This change removes the custom serialization of the total hits
and reuses the shard's serialization of Lucene#read/writeTopDocs
in the client code. This also removes the incorrect assertion that
trips randomly in bwc tests.

Closes #36284
2018-12-05 22:02:22 -08:00
Christoph Büscher 75635251ab
Make sure test don't use Math.random for reproducability (#36241)
Currently we use Math.random() in a few places in the tests which makes these
tests not reproducable with the random seed mechanism that comes with
ESTestCase. The change removes those instances.
2018-12-06 00:07:15 +01:00
Yannick Welsch 38d2c093ad Fix compilation 2018-12-05 23:44:33 +01:00
Jake Landis 190ac8e9bf
ingest: support default pipeline through an alias (#36231)
This commit allows writes that go through an alias to use the default
pipeline defined on the backing index.

Fixes #35817
2018-12-05 16:25:50 -06:00
Yannick Welsch a0ae1cc987 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 23:13:12 +01:00
Yannick Welsch 03d0ea91ef
Zen2: Rename tombstones to exclusions (#36226)
Renames the withdrawal / tombstones APIs to voting configuration exclusions.
2018-12-05 23:12:28 +01:00
Andrey Ershov 5d6602120f [Zen2] Hide not recovered state (#36224)
This commit hides ClusterStates that have a STATE_NOT_RECOVERED_BLOCK from
ClusterStateAppliers. This is needed, because some appliers, such as IngestService, rely on
the fact, that cluster states with STATE_NOT_RECOVERED_BLOCK won't contain anything useful.
Once the state is recovered it's fully available for the appliers. This commit also switches many of
the remaining tests that require state persistence/recovery from Zen1 to Zen2.
2018-12-05 23:11:20 +01:00
jaymode 2d03eeb632
Test: mute testDataNodeRestartWithBusyMasterDuringSnapshot
This commit mutes DedicatedClusterSnapshotRestoreIT#
testDataNodeRestartWithBusyMasterDuringSnapshot which fails reproducibly.

See #36283
2018-12-05 14:06:38 -07:00
Jay Modi dba3f4ec9f
add version 6.5.3 (#36268) 2018-12-05 12:15:09 -07:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Mark Tozzi ce7b1886b8
Unit test for very large percentile aggs (#36122)
Closes #19528

 * Port a test Colin wrote for the TDigest library to validate TDigests storing over 2*MAXINT values.  This appears to have been fixed in version 3.2 of TDigest, which Elasticsearch has been using for some time, so no changes were necessary to resolve this issue.
2018-12-05 10:56:48 -05:00
Andrey Ershov e8289167ee Fix FAIL_ON_METHOD in MetaDataStateFormatTests
fail method should be computed before eval, not inside it
2018-12-05 14:59:41 +01:00
Yannick Welsch b20497560c Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 14:06:38 +01:00
Yannick Welsch 0b9efff5cb
Zen2: Persist cluster states the old way on non-master-eligible nodes (#36247)
The shard deletion logic (triggered by IndicesStore), which also leads to index metadata deletion on
non-master-eligible data nodes, currently races against the new cluster state persistence logic
triggered by accepting cluster states. One thread is writing the index metadata while another one is
deleting the index metadata, leading to exceptions and assertions tripping (see below). The solution
proposed by this PR is to move the cluster state persistence of non-master-eligible nodes back to
the cluster applier service, just as it used to be for Zen1. This ensures that the index metadata
deletion logic, which is triggered by the shard deletion logic, runs on the same thread on which we
persist the cluster state.
2018-12-05 14:04:45 +01:00
Alan Woodward 73ceaad03a
Update to lucene-8.0.0-snapshot-c78429a554 (#36212)
Includes:

* A fix for a bug in Intervals.or() (https://issues.apache.org/jira/browse/LUCENE-8586)
* The ability to disable offset mangling in WordDelimiterGraphFilter
        (https://issues.apache.org/jira/browse/LUCENE-8509)
* BM25Similarity no longer multiplies scores by k1 + 1
2018-12-05 12:43:56 +00:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 (#36107)
Closes #35435

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Martijn van Groningen 11935cd480
Replace Streamable w/ Writeable in BaseTasksResponse and subclasses (#36176)
This commit replaces usages of Streamable with Writeable for the
BaseTasksResponse / TransportTasksAction classes and subclasses of
these classes.

Note that where possible response fields were made final.

Relates to #34389
2018-12-05 13:14:10 +01:00
Yannick Welsch 42457b5960 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 11:39:38 +01:00
Adrien Grand 4c2e0d76f3 Fix test failures when using the `termvectors` endpoint with a type name. 2018-12-05 11:28:12 +01:00
Tim Brooks 8bde608979
Register CcrRepository based on settings update (#36086)
This commit adds an empty CcrRepository snapshot/restore repository.
When a new cluster is registered in the remote cluster settings, a new
CcrRepository is registered for that cluster.

This is implemented using a new concept of "internal repositories".
RepositoryPlugin now allows implementations to return factories for
"internal repositories". The "internal repositories" are different from
normal repositories in that they cannot be registered through the
external repository api. Additionally, "internal repositories" are local
to a node and are not stored in the cluster state.

The repository will be unregistered if the remote cluster is removed.
2018-12-04 14:36:50 -07:00
Yannick Welsch 70c361ea5a Merge remote-tracking branch 'elastic/master' into zen2 2018-12-04 21:26:11 +01:00
Armin Braun 3c54b413ad
SNAPSHOT: Repo Creation out of ClusterStateTask (#36157)
* Move `createRepository` call out of cluster state tasks
    * Now only `RepositoriesService#applyClusterState` manipulates `this.repositories`
* Closes #9488
2018-12-04 20:53:51 +01:00
Adrien Grand d24b40f688
Make typeless APIs usable with indices whose type name is different from `_doc` (#35790)
This commit makes `document`, `update`, `explain`, `termvectors` and `mapping`
typeless APIs work on indices that have a type whose name is not `_doc`.
Unfortunately, this needs to be a bit of a hack since I didn't want calls with
random type names to see documents with the type name that the user had chosen
upon type creation.

The `explain` and `termvectors` do not support being called without a type for
now so the test is just using `_doc` as a type for now, we will need to fix
tests later but this shouldn't require further changes server-side since passing
`_doc` as a type name is what typeless APIs do internally anyway.

Relates #35190
2018-12-04 19:22:17 +01:00
Yannick Welsch 9026f98aca Remove trace logging from CoordinatorTests 2018-12-04 18:53:32 +01:00
Adrien Grand 0df08dd458
Set Lucene version upon index creation. (#36038)
It is important that all shards of a given index have the same
`indexCreatedVersionMajor` to Lucene, or eg. merging those shards is going to
be considered illegal. At the moment, we use the latest Lucene version when
creating a shard, which could cause shards to have different created versions
eg. in case of forced allocation. This commit makes sure to reuse the
appropriate Lucene version in order to avoid such issues.

Closes #33826
2018-12-04 17:53:20 +01:00
Nhat Nguyen b59deb573e
Always set soft-deletes field of IndexWriterConfig (#36196)
Today we configure the soft-deletes field iff soft-deletes enabled.
Although this choice was correct, it prevents an engine with
soft-deletes disabled from opening a Lucene index with soft-deletes.
Moreover, this change should not have any side-effect if a Lucene index
does not have any soft-deletes.

Relates #36141
2018-12-04 11:15:34 -05:00
Andrey Ershov 35e3d77e2c
[Zen2] Implement state recovery (#36013)
This commit implements proper metadata recovery for Zen2.

GatewayService is responsible for the recovery. In Zen1 GatewayService
creates an instance of Gateway, that is used to reach out to other cluster
nodes, get their state and calculate the most up-to-date state based on
versions. After that Gateway performs upgrade and archival of
 ClusterSettings and closes bad indices. Then recovered state is passed to GatewayService.GatewayRecoveryListener that mixes up current state
and restored state, removes state not recovered block, creates the
routing table and performs re-routing.

In Zen2 we should perform this kind of logic on cluster startup, except
mixing state (because there is nothing to mix) and opening routing table.

This commit refactors out all `ClusterUpdate` functions in a separate class
`ClusterStateUpdaters`, which is used by `Gateway` and `GatewayService`
in case of Zen1, and by `GatewayMetaState` and `GatewayService` in case of
Zen2.

This commit also switches all integration tests that are already using Zen2 from
InMemoryPersistedState to GatewayMetaState.
2018-12-04 14:45:45 +01:00
Tanguy Leroux 5d684ca473
Combine the execution of an exclusive replica operation with primary term update (#36116)
This commit changes how an operation which requires all index shard 
operations permits is executed when a primary term update is required: 
the operation and the update are combined so that the operation is 
executed after the primary term update under the same blocking 
operation.

Closes #35850

Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2018-12-04 14:39:31 +01:00
Yannick Welsch 80ee7943c9 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-04 09:37:09 +01:00
David Turner 034c7655b7
[Zen2] Reduce cluster scope in NodeDisconnectIT (#36168)
This test suite can stop all the shared master-eligible nodes, which breaks the
cluster since any non-shared master-eligible nodes are stopped first in the
reset process between tests.

Since this test suite can leave the cluster in this somewhat broken state, it
seems best that it uses a new cluster for each test.
2018-12-04 07:48:56 +00:00
Andy Bristol 01b8f99c17 [test] mute RemoveCorruptedShardDataCommandIT 2018-12-03 16:07:16 -08:00
João Barbosa d27aa72b17 Added soft limit to open scroll contexts #25244 (#36009)
This change adds a soft limit to open scroll contexts that can be controlled with the dynamic cluster setting `search.max_open_scroll_context` (defaults to 500).
2018-12-03 19:57:10 +01:00
Julie Tibshirani 59ee8b5c69
Remove the deprecated _termvector endpoint. (#36131) 2018-12-03 10:22:42 -08:00
David Turner c01aecb4b1
[Zen2] Do not probe non-master nodes back (#36160)
Today if a node `A` sends a peers request to another node `B` then `B` will
react by sending a peers request back to `A`. However if `A` is not
master-eligible then this reaction is pointless and fails with an exception
saying `non-master-eligible node found`, adding noise to the logs. This change
suppresses this response to non-master-eligible nodes.
2018-12-03 17:19:18 +00:00
Armin Braun 433a506d06
SNAPSHOT: Improve Resilience SnapshotShardService (#36113)
* Resolve the index in the snapshotting thread
* Added test for routing table - snapshot state mismatch
2018-12-03 16:39:29 +01:00
Nhat Nguyen 9c1c46a02f TEST: Adjust min_retained_seq_no expectation
min_retained_seq_no is non-negative, however, if the number of retained
operations is greater than 0, then the expectation may be negative.
2018-12-03 08:59:13 -05:00
Luca Cavanna b5cae0af58
Enforce max_buckets limit only in the final reduction phase (#36152)
Given that we check the max buckets limit on each shard when collecting the buckets, and that non final reduction cannot add buckets (see #35921), there is no point in counting and checking the number of buckets as part of non final reduction phases.

Such check is still needed though in the final reduction phases to make sure that the number of returned buckets is not above the allowed threshold.

Relates somehow to #32125 as we will make use of non final reduction phases in CCS alternate execution mode and that increases the chance that this check trips for nothing when reducing aggs in each remote cluster.
2018-12-03 13:55:18 +01:00
Boaz Leskes 36ddca7d0c Disable merges in testReuseInFileBasedPeerRecovery
The test assumes lucene files don't change.

Closes #35772
2018-12-03 13:45:19 +01:00
Jim Ferenczi 74aca756b8
Remove the distinction between query and filter context in QueryBuilders (#35354)
When building a query Lucene distinguishes two cases, queries that require to produce a score and queries that only need to match. We cloned this mechanism in the QueryBuilders in order to be able to produce different queries based on whether they need to produce a score or not. However the only case in es that require this distinction is the BoolQueryBuilder that sets a different minimum_should_match when a `bool` query is built in a filter context..
This behavior doesn't seem right because it makes the matching of `should` clauses different when the score is not required.

Closes #35293
2018-12-03 11:49:11 +01:00
Armin Braun 328d022ddd
MINOR: Some Cleanups around Store (#36139)
* Moved method `canOpenIndex` is only used in tests -> moved to test CP
* Simplify `org.elasticsearch.index.store.Store#renameTempFilesSafe`
* Delete some dead methods
2018-12-03 11:21:42 +01:00
Armin Braun f763037b03
MINOR: BlobstoreRepository Cleanups (#36140)
* Removed redundant private getter
* Removed unused `version` field
2018-12-03 11:11:10 +01:00
Armin Braun 9c49aacbcf
MINOR: Remove Dead Code in QueryCache (#36147) 2018-12-03 10:02:35 +01:00
Martijn van Groningen 43773a32a4
Replace Streamable w/ Writeable in BaseTasksRequest and subclasses (#35854)
* Replace Streamable w/ Writeable in BaseTasksRequest and subclasses

This commit replaces usages of Streamable with Writeable for the
BaseTasksRequest / TransportTasksAction classes and subclasses of
these classes.

Relates to #34389
2018-12-03 08:04:29 +01:00
Armin Braun 9c0a429709
TESTS: Fix IndexStatsIT#testFilterCacheStats (#36143)
* Test randomly failed because of background merges
   * Fixed by force merging down to a single segment
* Closes #32506
2018-12-03 06:16:12 +01:00
David Turner 8bb1952975
Fix NodeJoinTests again (#36133)
In #36033 we removed a catch block because we thought we were preventing
exceptions by avoiding concurrent elections, missing the obvious fact that some
joins are supposed to be failing.

As a quick fix the catch was reinstated in 3a5dab6d8e
but this change adds finesse by only catching exceptions from the joins that we
expect to fail. It also inlines an always-false parameter to `initialState()`.
2018-12-01 09:54:01 +00:00
David Turner 9cc416bc46 Weaken assertion in PeerFinder
It can be inactive with no leader if it's handling an incoming PeersRequest
before being activated for the first time.
2018-12-01 07:20:19 +00:00
David Turner 3a5dab6d8e Reinstate catch removed in error in #36033 2018-12-01 07:10:19 +00:00
Julie Tibshirani 0e1ddfd825
Deprecate types in document delete requests. (#36087)
* Make sure to use _doc as a type name in the CRUD HLRC tests.
* Deprecate types in document delete requests.
2018-11-30 15:11:29 -08:00
Julie Tibshirani 98b290637d
Deprecate the _termvector endpoint. (#36098) 2018-11-30 13:11:58 -08:00
Nik Everett df56f0734e
Tasks: Retry if task can't be written (#35054)
Adds about a minute worth of backoffs and retries to saving task
results so it is *much* more likely that a busy cluster won't lose task
results. This isn't an ideal solution to losing task results, but it is
an incremental improvement. If all of the retries fail when still log
the task result, but that is far from ideal.

Closes #33764
2018-11-30 16:06:58 -05:00
Luca Cavanna 0ebc17743a
Histogram aggs: add empty buckets only in the final reduce step (#35921)
Empty buckets don't need to be added when performing an incremental reduction step, they can be added later in the final reduction step. This will allow us to later remove the max buckets limit when performing non final reduction.
2018-11-30 20:33:09 +01:00
Tim Brooks ea7ea51050
Make `TcpTransport#openConnection` fully async (#36095)
This is a follow-up to #35144. That commit made the underlying
connection opening process in TcpTransport asynchronous. However the
method still blocked on the process being complete before returning.
This commit moves the blocking to the ConnectionManager level. This is
another step towards the top-level TransportService api being async.
2018-11-30 11:30:42 -07:00
Armin Braun 986bf52d1f
[Zen2] Allow Setting a List of Bootstrap Nodes to Wait for (#35847) 2018-11-30 18:53:08 +01:00
Tim Brooks da100c5479
Remove `Lifecycle` from `ConnectionManager` (#36092)
Prior to #35441 `ConnectionManager` had a `Lifecycle` object to support
the ping runnable. After that commit, the connection amanger only needs
the existing `AtomicBoolean` to indicate if it is running.
2018-11-30 09:04:32 -07:00
Luca Cavanna 43ea498f2f [TEST] Reduce number of buckets created in InternalDateHistogramTests
New that we test with min_doc_count set to 0 as well, we may end up generating a lot more buckets. This commit adjusts the min bound and max bound, as well as the offset for each randomly generated agg instance so that we don't end up hitting the 10.000 max buckets limit.

Relates to #36064
2018-11-30 14:16:32 +01:00
Luca Cavanna e3eb05b14b
[TEST] Increase InternalDateHistogramTests coverage (#36064)
In this test we were randomizing different values but minDocCount was hardcoded to 1. It's important to test other values, especially `0` as it's the default. The test needed some adapting in the way buckets are randomly generated: all aggs need to share the same interval, minDocCount and emptyBucketInfo. Also assertions need to take into account that more (or less) buckets are expected depending on minDocCount.
2018-11-30 11:21:46 +01:00
Adrien Grand fa3d365ee8
Fix CompositeBytesReference#slice to not throw AIOOBE with legal offsets. (#35955)
CompositeBytesReference#slice has two bugs:
 - One that makes it fail if the reference is empty and an empty slice is
   created, this is #35950 and is fixed by special-casing empty-slices.
 - One performance bug that makes it always create a composite slice when
   creating a slice that ends on a boundary, this is fixed by computing `limit`
   as the index of the sub reference that holds the last element rather than
   the next element after the slice.

Closes #35950
2018-11-30 10:32:46 +01:00
Jake Landis f8636e58f9
Support content type `application/x-ndjson` in DeprecationRestHandler (#36025)
org.elasticsearch.rest.RestController#hasContentType checks to see if the
RestHandler supports the `application/x-ndjson` Content-Type. DeprecationRestHandler
is a wrapper around the real RestHandler, and prior to this change
would always return `false` due to the interface's default supportsContentStream().
This prevents API's that use multi-line JSON from properly being deprecated
resulting in an HTTP 406 error.

This change ensures that the DeprecationRestHandler honors the
supportsContentStream() of the wrapped RestHandler.

Relates to #35958
2018-11-29 11:45:45 -06:00
Jim Ferenczi 8a7f3f75f3
Add support for rest_total_hits_as_int (#36051)
The support for rest_total_hits_as_int has already been merged to 6x
in #35848 so this change adds this new option to master. The plan was
to add this new option as part of #35848 but we've decided to wait a few
days before merging this breaking change so this commit just handles
the new option as a noop exactly like 6x for now. This will allow
users to migrate to this parameter before #35848 is merged.

Relates #33028
2018-11-29 18:36:16 +01:00
Tim Brooks c305f9dc03
Make keepalive pings bidirectional and optimizable (#35441)
This is related to #34405 and a follow-up to #34753. It makes a number
of changes to our current keepalive pings.

The ping interval configuration is moved to the ConnectionProfile.

The server channel now responds to pings. This makes the keepalive
pings bidirectional.

On the client-side, the pings can now be optimized away. What this
means is that if the channel has received a message or sent a message
since the last pinging round, the ping is not sent for this round.
2018-11-29 08:55:53 -07:00
Jim Ferenczi ecd29089a8
Cache the score of the parent document in the nested agg (#36019)
The nested agg can defer the collection of children if it is nested
under another aggregation. In such case accessing the score in the children
aggregation throws an error because the scorer has already advanced to the next
parent. This change fixes this error by caching the score of the parent in the
nested aggregation. Children aggregations that work on nested documents will be
able to access the _score. Also note that the _score in this case is always the
parent's score, there is no way to retrieve the score of a nested docs in aggregations.

Closes #35985
Closes #34555
2018-11-29 14:35:25 +01:00
Armin Braun 48dc6c3442
[Zen2] Implement Tombstone REST APIs (#36007)
* [Zen2] Implement Tombstone REST APIs

* Adds REST API for withdrawing votes and clearing vote withdrawls
* Tests added to Netty4 module since we need a real Network impl. for Http endpoints
2018-11-29 14:34:10 +01:00
David Turner 7f257187af
[Zen2] Update default for USE_ZEN2 to true (#35998)
Today the default for USE_ZEN2 is false and it is overridden in many places. By
defaulting it to true we can be sure that the only places in which Zen2 does
not work are those in which it is explicitly set to false.
2018-11-29 12:18:35 +00:00
David Turner 277ccba3bd
[Zen2] fix NodeJoinTests#testConcurrentJoining() (#36033)
Today we sometimes create a setup in which the node is a quorum on its own,
which allows it to win a pre-voting round and schedule an election essentially
at will, causing it to discard all the joins it just received and fail the
test. This change excludes this case, preventing stray elections from ruining
things.
2018-11-29 12:11:08 +00:00
David Turner 87408b04d4
[Zen2] Only elect master-eligible nodes (#35996)
Today any node can win an election. However, the whole point of
master-eligibility is that master-ineligible nodes should not be elected as the
leader; furthermore master-ineligible nodes do not have any outgoing STATE
channels so cannot publish cluster states, so their leadership is ineffective
and disruptive.

This change ensures that the elected leader is master-eligible by preventing
master-ineligible nodes from scheduling an election.
2018-11-29 12:10:43 +00:00
Alan Woodward a646f85a99
Ensure TokenFilters only produce single tokens when parsing synonyms (#34331)
A number of tokenfilters can produce multiple tokens at the same position.  This
is a problem when using token chains to parse synonym files, as the SynonymMap
requires that there are no stacked tokens in its input.

This commit ensures that when used to parse synonyms, these tokenfilters either produce
a single version of their input token, or that they throw an error when mappings are 
generated.  In indexes created in elasticsearch 6.x deprecation warnings are emitted in place 
of the error. 

* asciifolding and cjk_bigram produce only the folded or bigrammed token
* decompounders, synonyms and keyword_repeat are skipped
* n-grams, word-delimiter-filter, multiplexer, fingerprint and phonetic throw errors

Fixes #34298
2018-11-29 10:35:38 +00:00
Tanguy Leroux 0967620641
ActiveShardCount should not fail when closing the index (#35936)
The ActiveShardCount is used by cluster state observers to wait for a 
given number of shards to be active before returning to the caller. The 
current implementation does not work when an index is closed while an 
observer is waiting on shards to be active. In this case, a NPE is thrown 
and the observer is never notified that the shards won't become active.

This commit fixes the ActiveShardCount.enoughShardsActive() so that it 
does not fail when an index is closed, similarly to what is done when an 
index is deleted.
2018-11-29 09:08:30 +01:00
Alpar Torok e0a678f0c4
Remove version.qualified from MainResponse (#35412)
The fully qualified version will be returned as `version.number`
2018-11-29 08:41:39 +02:00
Ryan Ernst afd42df15f
Core: Deguice RepositoriesService (#36016)
This commit moves the RepositoriesService to be created outside of
guice.
2018-11-28 21:20:44 -08:00
Jim Ferenczi 9ca3a06475
Remove custom QueryBuilder#analyzeGraphPhrase (#35983)
Now that https://issues.apache.org/jira/browse/LUCENE-8479 is fixed
we can remove the custom implementation of QueryBuilder#analyzeGraphPhrase
in the match QueryBuilder.
2018-11-28 20:15:27 +01:00
Luca Cavanna 4b85769d24
Increase InternalHistogramTests coverage (#36004)
In `InternalHistogramTests` we were randomizing different values but `minDocCount` was hardcoded to `1`. It's important to test other values, especially `0` as it's the default. To make this possible, the test needed some adapting in the way buckets are randomly generated: all aggs need to share the same `interval`, `minDocCount` and `emptyBucketInfo`. Also assertions need to take into account that more (or less) buckets are expected depending on `minDocCount`.

This was originated by #35921 and its need to test adding empty buckets as part of the reduce phase.

Also relates to #26856 as one more key comparison needed to use `Double.compare` to properly handle `NaN` values, which was triggered by the increased test coverage.
2018-11-28 20:06:40 +01:00
Nik Everett 0588dad80b
Tasks: Only require task permissions (#35667)
Right now using the `GET /_tasks/<taskid>` API and causing a task to opt
in to saving its result after being completed requires permissions on
the `.tasks` index. When we built this we thought that that was fine,
but we've since moved towards not leaking details like "persisting task
results after the task is completed is done by saving them into an index
named `.tasks`." A more modern way of doing this would be to save the
tasks into the index "under the hood" and to have APIs to manage the
saved tasks. This is the first step down that road: it drops the
requirement to have permissions to interact with the `.tasks` index when
fetching task statuses and when persisting statuses beyond the lifetime
of the task.

In particular, this moves the concept of the "origin" of an action into
a more prominent place in the Elasticsearch server. The origin of an
action is ignored by the server, but the security plugin uses the origin
to make requests on behalf of a user in such a way that the user need
not have permissions to perform these actions. It *can* be made to be
fairly precise. More specifically, we can create an internal user just
for the tasks API that just has permission to interact with the `.tasks`
index. This change doesn't do that, instead, it uses the ubiquitus
"xpack" user which has most permissions because it is simpler. Adding
the tasks user is something I'd like to get to in a follow up change.

Instead, the majority of this change is about moving the "origin"
concept from the security portion of x-pack into the server. This should
allow any code to use the origin. To keep the change managable I've also
opted to deprecate rather than remove the "origin" helpers in the
security code. Removing them is almost entirely mechanical and I'd like
to that in a follow up as well.

Relates to #35573
2018-11-28 09:28:27 -05:00
Christoph Büscher 51a7dc54ec
Fix custom AUTO issue with Fuzziness#toXContent (#35807)
Currently when a Fuzziness instance with custom AUTO distance values gets
written to XContent, the customized lower and upper distance values are ommited
and can consequently not be parsed back. This changes this to write the String
including the optional custom values when writing to XContent and fixes the
tests that should have caught this in the first place, e.g. by adding the custom
low and high distance values to the equality check.
2018-11-28 15:07:11 +01:00
Andrey Ershov 0b45fb98b9
[Zen2] Generate coordinationMetaData with different configs (#35991)
This PR fixes test failure, which is caused by equal randomly generated lastAcceptedConfiguration and lastCommittedConfguration.
2018-11-28 14:49:07 +01:00
Yannick Welsch 5f0c036183 Disable testDeleteCreateInOneBulk on Zen2
This test needs adaptation to run with Zen2
2018-11-28 13:34:48 +01:00
Christoph Büscher 2f547bac65
Remove deprecated methods from QueryStringQueryBuilder (#35912)
This change removes the deprecated useDisMax() and useAllFields() methods from
the QueryStringQueryBuilder and related tests. The disMax parameter has already
been a no-op since 6.0 and also the useAllFields has been deprecated since 6.0
and there is a direct replacement via defaultField.
2018-11-28 11:09:03 +01:00
Jeff Hajewski 49087f16f5 Adds deprecation logging to ScriptDocValues#getValues. (#34279)
`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

Closes #22919
2018-11-27 14:30:13 -05:00
Julie Tibshirani 25c416b12d
Deprecate types in search and multi search templates. (#35669)
This PR adds deprecation warnings to the relevant `Rest*Action` classes, plus tests in `Rest*ActionTests`. No updates to REST tests, the Java HLRC, or documentation were necessary, since they didn't make use of types.
2018-11-27 10:19:19 -08:00
Simon Willnauer ad1f0dccd4
Validate metdata on `_msearch` (#35938)
MultiSearchRequests issues through `_msearch` now validate all keys
in the metadata section. Previously unknown keys were ignored
while now an exception is thrown.

Closes #35869
2018-11-27 17:08:24 +01:00
Tim Brooks cc1fa799c8
Remove `TcpChannel#setSoLinger` method (#35924)
This commit removes the dedicated `setSoLinger` method. This simplifies
the `TcpChannel` interface. This method has very little effect as the
SO_LINGER is not set prior to the channels being closed in the abstract
transport test case. We still will set SO_LINGER on the
`MockNioTransport`. However we can do this manually.
2018-11-27 09:08:14 -07:00
Christophe Bismuth adc0b560c0 Raise a 404 exception when document source is not found (#33384) (#34083)
This pull request makes the `RestGetSourceAction` return a `ResourceNotFoundException` with a proper JSON response when source or document itself is missing (see issue #33384).

Here is below a sample JSON output:

```
{
  "error": {
    "root_cause": [
      {
        "type": "resource_not_found_exception",
        "reason": "Source not found [index1]/[_doc]/[1]"
      }
    ],
    "type": "resource_not_found_exception",
    "reason": "Source not found [index1]/[_doc]/[1]"
  },
  "status": 404
}
```
2018-11-27 10:35:45 -05:00
Andrey Ershov 0e283f9670
[Zen2] PersistedState interface implementation (#35819)
Today GatewayMetaState is capable of atomically storing MetaData to
disk. We've also moved fields that are needed to be persisted in Zen2
from ClusterState to ClusterState.MetaData.CoordinationMetaData.

This commit implements PersistedState interface.

version and currentTerm are persisted as a part of Manifest.
GatewayMetaState now implements both ClusterStateApplier and
PersistedState interfaces. We started with two descendants
Zen1GatewayMetaState and Zen2GatewayMetaState, but it turned
out to be not easy to glue it.
GatewayMetaState now constructs previousClusterState (including
MetaData) and previousManifest inside the constructor so that all
PersistedState methods are usable as soon as GatewayMetaState
instance is constructed. Also, loadMetaData is renamed to
getMetaData, because it just returns
previousClusterState.metaData().
Sadly, we don't have access to localNode (obtained from 
TransportService in the constructor, so getLastAcceptedState
should be called, after setLocalNode method is invoked.
Currently, when deciding whether to write IndexMetaData to disk,
we're comparing current IndexMetaData version and received
IndexMetaData version. This is not safe in Zen2 if the term has changed.
So updateClusterState now accepts incremental write
method parameter. When it's set to false, we always write
IndexMetaData to disk.
Things that are not covered by GatewayMetaStateTests are covered
by GatewayMetaStatePersistedStateTests.
This commit also adds an option to use GatewayMetaState instead of
InMemoryPersistedState in TestZenDiscovery. However, by default
InMemoryPersistedState is used and only one test in PersistedStateIT
used GatewayMetaState. In order to use it for other tests, proper
state recovery should be implemented.
2018-11-27 15:04:52 +01:00
Martijn van Groningen 447e5d212a
Changed versions in serialization code after backporting #35535 2018-11-27 08:00:06 +01:00
Gordon Brown 119835decd
Always enforce cluster-wide shard limit (#34892)
This removes the option to run a cluster without enforcing the
cluster-wide shard limit, making strict enforcement the default and only
behavior.  The limit can still be adjusted as desired using the cluster
settings API.
2018-11-26 17:05:12 -07:00
Igor Motov 663563f64b
Geo: better handling of malformed geo_points (#35554)
Improves handling of malformed geo_points when `ignore_malformed` is
set to true

Closes #35419
2018-11-26 09:44:42 -10:00
Jim Ferenczi 900caa20ef
Handles exists query in composite aggs (#35758)
This commit adds the support for exists query in the sorted execution mode
of the `composite` aggregation. We'll execute the aggregation from the sorted
points and use early termination if the main query is an `exists` query over the
first source of the `composite` aggregation.
2018-11-26 19:08:14 +01:00
Christophe Bismuth b95a4db6e6 Throw a parsing exception when boost is set in span_or query (#28390) (#34112) 2018-11-26 12:15:59 -05:00
Simon Willnauer ca9b2b9931
Repsect indices options on _msearch (#35887)
Today we don't respect the indices options when they are passed
as request parameters to the `_msearch` endpoint. This is unintuitive
and doesn't cause any errors. This changes uses the top-level indices
options as the defaults for each sub search-request.

Closes #35851
2018-11-26 14:26:39 +01:00
Christophe Bismuth 04ebc63e34 RoutingMissingException in more like this (#33974)
More like this query allows to provide identifiers of documents to be retrieved as like/unlike items. 
It can happen that at retrieval time an error is thrown, for instance caused by missing routing value when `_routing` is set required in the mapping. 
Instead of ignoring such error and returning no documents for the query, the error should be re-thrown and returned to users. As part of this 
change also mget and mtermvectors are unified in the way they throw such exception like it happens in other places, so that a `RoutingMissingException` is raised.

Closes #29678
2018-11-26 13:57:57 +01:00
Tanguy Leroux 9bdbba23f8 [Tests] Fix IndexShardTests.testAcquirePrimaryAllOperationsPermits()
This test fails on CI because of an inappropriate assertion, which is
I think a leftover and has no real value.
2018-11-26 13:44:12 +01:00
Luca Cavanna e44390ac20
InitialSearchPhase minor cleanups (#35864)
This commit simplifies  the throttling logic in InitialSearchPhase and removes some asserts from it. Also, a few formatting changes are applied to its code and surrounding classes.
2018-11-26 13:42:41 +01:00
David Turner a68a46450b
[Zen2] Add lag detector (#35685)
A publication can succeed and complete before all nodes have applied the
published state and acknowledged it, thanks to the publication timeout; however
we need every node eventually either to apply the published state (or a later
state) or be removed from the cluster. This change introduces the LagDetector
which achieves this liveness property by removing any lagging nodes from the
cluster.
2018-11-26 10:52:49 +00:00
iverase 401b814d1a [CI] Muting method testOperationPermitOnReplicaShards in IndexShardTests
Relates to #35850
2018-11-26 09:33:23 +01:00
Martijn van Groningen 7624734f14
Added wait_for_metadata_version parameter to cluster state api. (#35535)
The `wait_for_metadata_version` parameter will instruct the cluster state
api to only return a cluster state until the metadata's version is equal or
greater than the version specified in `wait_for_metadata_version`. If  
the specified `wait_for_timeout` has expired then a timed out response 
is returned. (a response with no cluster state and wait for timed out flag set to true)
In  the case metadata's version is equal or higher than  `wait_for_metadata_version`
then the api will immediately return.

This feature is useful to avoid external components from constantly
polling the cluster state to whether somethings have changed in the
cluster state's metadata.
2018-11-26 08:50:08 +01:00
Simon Willnauer 4711c5cdf3
Always return false from `refreshNeeded` on ReadOnlyEngine (#35837)
Acquiring a searcher is unnecessary to determine if a refresh is
necessary since read-only engines never refresh.

Closes #35785
2018-11-24 09:25:42 +01:00
Simon Willnauer e46e44ce38
Wrap can_match reader with ElasticsearchDirectoryReader (#35857)
Code that operates on-top of the engine requires all readers returned to be
unwrapped into ElasticsearchDirectoryReader. The special reader
the FrozenEngine uses wasn't wrapped.
2018-11-24 09:23:53 +01:00
Andrey Ershov f47636b254
[Zen2] Introduce VotingTombstone class (#35832)
Today voting tombstones are stored in CoordinationMetaData as
Set<DiscoveryNode>.
DiscoveryNode is not a lightweight object and have a lot of fields.
It also has toXContent method, but no fromXContent method and the
output of toXContent is not enough to re-create DiscoveryNode
object.
And votingTombstone set should be persisted as a part of MetaData.
On the other hand, the only thing required from the tombstone is the
nodeId.
This PR adds VotingTombstone class for voting tombstones, which
consists of two fields for now - nodeId and nodeName. It could be
extended/shrank in the future if needed.
This PR also resolves TODO's related to the voting tombstones xcontent
story.
Example of CoordinationMetaData.toXContent with voting tombstones:

{
  "term": 1,
  "last_committed_config": [
    "fkwLdOBvXSlgRTBfgNAL",
    "tmQiPGHvUxXzPkkCDSJo",
    "HhOmtQBZAThpHIGWhxpz",
    "qZHWGpoDNPYRNIiqKsDl"
  ],
  "last_accepted_config": [
    "lhqacKmriwhHGFZcvqbx",
    "MYysmBuROkvJRlDcusyd"
  ],
  "voting_tombstones": [
    {
      "node_id": "McjbZbRkEz",
      "node_name": "pdKIWeNJUO"
    },
    {
      "node_id": "cpXkVibGwo",
      "node_name": "UnCvFgdVsc"
    },
    {
      "node_id": "EylRNOztbc",
      "node_name": "ohOhkbMWZX"
    }
  ]
}
2018-11-23 18:34:06 +01:00
Yannick Welsch 51d2e986c5 Remove BWC conditions after backport of #35731
This PR was backported to 6.x, so the extra BWC conditions are not needed anymore
2018-11-23 17:11:06 +01:00
Adrien Grand 5b370316a6
Remove some legacy code from when indices could have multiple types. (#35815)
This code is only necessary up to indices created with version 5.x while 7.0
only supports indices created with 6.x or 7.0.
2018-11-23 15:15:26 +01:00
Yannick Welsch 2970abfce9
Add read-only repository verification (#35731)
Adds a verification mode for read-only repositories. It also makes the extra bucket check on
repository creation obsolete, which fixes #35703.
2018-11-23 14:45:05 +01:00
Christoph Büscher 88d862e69f [CI] Muting two methods in IndexShardTests
Relates to #35850
2018-11-23 14:29:26 +01:00
David Turner d01436de3c
Copy checkpoint atomically when rolling generation (#35407)
Today when rolling a transog generation we copy the checkpoint from
`translog.ckp` to `translog-nnnn.ckp` using a simple `Files.copy()` followed by
appropriate `fsync()` calls. The copy operation is not atomic, so if we crash
at the wrong moment we can leave an incomplete checkpoint file on disk. In
practice the checkpoint is so small that it's either empty or fully written.
However, we do not correctly handle the case where it's empty when the node
restarts.

In contrast, in `recoverFromFiles()` we _do_ copy the checkpoint atomically.
This commit extracts the atomic copy operation from `recoverFromFiles()` and
re-uses it in `rollGeneration()`.
2018-11-23 08:43:34 +00:00
Jim Ferenczi be69a774df
Fix analyzed prefix query in query_string (#35756)
This change fixes analyzed prefix queries in `query_string` to be ignored
if all terms are removed during the analysis.

Closes #31702
2018-11-23 09:42:23 +01:00
Tanguy Leroux 2e37f17a7d
Expose all permits acquisition in IndexShard and TransportReplicationAction (#35540)
This pull request exposes two new methods in the IndexShard and 
TransportReplicationAction classes in order to allow transport replication 
actions to acquire all index shard operation permits for their execution.

It first adds the acquireAllPrimaryOperationPermits() and the 
acquireAllReplicaOperationsPermits() methods to the IndexShard class 
which allow to acquire all operations permits on a shard while exposing 
a Releasable. It also refactors the TransportReplicationAction class to 
expose two protected methods (acquirePrimaryOperationPermit() and 
acquireReplicaOperationPermit()) that can be overridden when a transport 
replication action requires the acquisition of all permits on primary and/or 
replica shard during execution.

Finally, it adds a TransportReplicationAllPermitsAcquisitionTests which
 illustrates how a transport replication action can grab all permits before 
adding a cluster block in the cluster state, making subsequent operations 
that requires a single permit to fail).

Related to elastic #33888
2018-11-23 09:26:38 +01:00
Dimitris Athanasiou 43d6ec8bcd
Remove unnecessary throws IOException in CompressedXContent.string() (#35821) 2018-11-22 15:08:46 +00:00
Jim Ferenczi e37a0ef844
Upgrade to lucene-8.0.0-snapshot-67cdd21996 (#35816) 2018-11-22 15:42:59 +01:00
Albert Zaharovits 4fc911a129 Mute test InternalEngineTests
Relates #35823
2018-11-22 15:34:53 +02:00
Tanguy Leroux f9f7261d60 Revert "Revert "[RCI] Check blocks while having index shard permit in TransportReplicationAction (#35332)""
This reverts commit d3d7c01
2018-11-22 12:13:19 +01:00
Mayya Sharipova b6014d971c
Forbid negative scores in functon_score query (#35709)
* Forbid negative scores in functon_score query

- Throw an exception when scores are negative in field_value_factor
function
- Throw an exception when scores are negative in script_score
function

Relates to #33309
2018-11-22 06:08:48 -05:00
Tanguy Leroux 11052b75c7
TransportResyncReplicationAction should not honour blocks (#35795)
After #35332 has been merged, we noticed some test failures like #35597 
in which one or more replica shards failed to be promoted as primaries 
because the primary replica re-synchronization never succeed.

After some digging it appeared that the execution of the resync action was 
blocked because of the presence of a global cluster block in the cluster state 
(in this case, the "no master" block), making the resync action to fail when 
executed on the primary.

Until #35332 such failures never happened because the 
TransportResyncReplicationAction is skipping the reroute phase, the only 
place where blocks were checked. Now with #35332 blocks are checked 
during reroute and also during the execution of the transport replication 
action on the primary. After some internal discussion, we decided that the TransportResyncReplicationAction should never be blocked. This action is 
part of the replica to primary promotion and makes sure that replicas are in 
sync and should not be blocked when the cluster state has no master or 
when the index is read only.

This commit changes the TransportResyncReplicationAction to make obvious 
that it does not honor blocks. It also adds a simple test that fails if the resync 
action is blocked during the primary action execution.

Closes #35597
2018-11-22 10:50:12 +01:00
David Turner cfdf666672
[Zen2] Fix test failures in diff-based publishing (#35684)
`testIncompatibleDiffResendsFullState` sometimes makes a 2-node cluster and
then partitions one of the nodes from the leader, which makes the leader stand
down.  Then when the partition is removed the cluster re-forms but does so by
sending full cluster states, not diffs, causing the test to fail.

Additionally `testDiffBasedPublishing` sometimes fails if a publication is
delivered out-of-order, wiping out a fresher last-received cluster state with a
less-fresh one. This is fixed here by passing the received cluster state to the
coordinator before recording it as the last-received one, relying on the
coordinator's freshness checks.
2018-11-22 09:08:52 +00:00
Igor Motov 39789d0a10
GEO: More robust handling of ignore_malformed in geoshape parsing (#35603)
Adds an XContent sub parser class that can to wrap another
XContent parser at the beginning of an object and allow skiping
all children in case of the parsing failure. It also uses this
subparser to ignore the rest of the GeoJson shape if the 
parsing fails and we need to ignore the geoshape due to the 
ignore_malformed flag.

Supersedes #34498

Closes #34047
2018-11-21 11:04:01 -10:00
Armin Braun 1a5553d495
MINOR: Cleanup Runnables in SnapshotsService (#35796)
* Simplify complex `Runnable` by moving to `AbstractRunnable`
2018-11-21 19:58:51 +01:00
Nick Knize 3bee25cb70
[GEO] Add support to ShapeBuilders for building Lucene geometry (#35707)
* [GEO] Add support to ShapeBuilders for building Lucene geometry

This commit adds support for building lucene geometry from the ShapeBuilders.
This is needed for integrating LatLonShape as the primary indexing approach
for geo_shape field types. All unit and integration tests are updated to
add randomization for testing both jts/s4j shapes and lucene shapes.
2018-11-21 11:15:01 -06:00
Yannick Welsch c816347253 Disable testClusterJoinDespiteOfPublishingIssues for Zen2
This test is failing sometimes with Zen2 due to the lack of lag detection.
Zen1 does not have this problem as it only considers a join as valid if the
corresponding cluster state update is successfully published and committed
on the joining node.
2018-11-21 17:21:44 +01:00
Andrey Ershov a056bd8c1c
[Zen2] Move ClusterState fields to be persisted to ClusterState.MetaData (#35625)
Today we have a way to atomically persist global MetaData and
IndexMetaData to disk when new ClusterState is received. All other
ClusterState fields are not persisted.
However, there are other parts of ClusterState that should be
persisted, namely:

version
term
lastCommittedConfiguration
lastAcceptedConfiguration
votingTombstones

version is changed frequently, other fields are not. We decided
to group term, lastCommittedConfiguration,
lastAcceptedConfiguration and votingTombstones into
CoordinationMetaData class and make CoordinationMetaData a field
inside MetaData.
MetaData.toXContent and MetaData.fromXContent should take care of
CoordinationMetaData.
version stays as a top level field in ClusterState and will be
persisted as part of Manifest in a follow-up commit.
Also MetaData.isGlobalStateEquals should be extended to include
coordinationMetaData in comparison.

This commit favors exposing getters, such as getTerm directly in
ClusterState to avoid massive code changes.

An example of CoordinationMetaState.toXContent:

{
  "term": 1,
  "last_committed_config": [
    "TiIuBcbBtpuXyDDVHXeD",
    "ZIAoVbkjjLPLUuYLaTkw"
  ],
  "last_accepted_config": [
    "OwkXbXZNOZPJqccdFHdz",
    "LouzsGYwmQzpeQMrboZe",
    "fCKGRZdjLTqzXAqPUtGL",
    "pLoxshjpJXwDhbgjfYJy",
    "SjINLwFIlIEFZCbjrSFo",
    "MDkVncJEVyZLJktopWje"
  ]
}
2018-11-21 17:03:26 +01:00
Andrey Ershov 6ac0cb1842 Merge branch master into zen2
2 types of conflicts during the merge:
1) Line length fix
2) Classes no longer extend AbstractComponent
2018-11-21 15:36:49 +01:00
Yannick Welsch 8939a7894f
Zen2: Move disruption tests to Zen2 (#35724)
- Moves disruption tests to Zen2
- Registers a few missing settings
- Removes .put(TestZenDiscovery.USE_ZEN2.getKey(), true) from tests where Zen2 is now enabled
by default through the parent test class
- Moves QuorumGatewayIT back to Zen1, as it is not stable with Zen2 as it currently relies on
dangling indices due to the lack of proper CS persistence, which triggers secondary failures
2018-11-21 14:43:33 +01:00
Armin Braun bdf632b6f9
SNAPSHOTING+MINOR: Simplify SnapshortShardService (#35769) 2018-11-21 13:50:17 +01:00
Christoph Büscher 6638708b56
Remove deprecated QueryStringQueryBuilder#splitOnWhiteSpace (#35763)
This parameter has been deprecated and was ignored since 6.0, so its Java API
methods can be removed.
2018-11-21 10:29:08 +01:00
Christoph Büscher 5847f8379c
Move ScoreAccessor to test-framework (#35766)
This class is only used by RandomScoreFunctionIT and the MockScriptEngine, so it
shouldn't be part of the server codebase.
2018-11-21 10:28:31 +01:00
Christoph Büscher ff03443ab9
Fix problem with MatchNoDocsQuery in disjunction queries (#35726)
Queries across multiple fields generate MatchNoDocsQuerys for fields that are
unmapped. In certain situation this can lead to erroneous behaviour,
for example when an umapped field is used in a query_string query across
several fields. If some of the tokens in the query string get eliminated by an
analyzer on the mapped fields, the same token will currently generate
MatchNoDocsQuerys combined into a disjunction, which in turn
leads to no matches in the overall query. Instead we should simply not add
MatchNoDocsQuerys to those disjunctions.

Closes #34708
2018-11-21 03:49:49 +01:00
Mayya Sharipova 643bb20137
Add a new query type - ScriptScoreQuery (#34533)
* Add a new query type - ScriptScoreQuery

script_score query uses script to calculate document scores.
Added as a substitute for function_score with an intentation
to deprecate function_scoreq query.

```http
GET /_search
{
    "query": {
        "script_score" : {
            "query": {
                "match": { "message": "elasticsearch" }
            },
            "script" : {
              "source": "Math.log(2 + doc['likes'].value)"
            },
            "min_score" : 2
        }
    }
}
```

Add several functions to painless to be used inside script_score:

double rational(double, double)
double sigmoid(double, double, double)
double randomNotReproducible() 
double randomReproducible(String, int) 

double decayGeoLinear(String, String, String, double, GeoPoint)
double decayGeoExp(String, String, String, double, GeoPoint)
double decayGeoGauss(String, String, String, double, GeoPoint)

double decayNumericLinear(String, String, String, double, double)
double decayNumericExp(String, String, String, double, double)
double decayNumericGauss(String, String, String, double, double)

double decayDateLinear(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateExp(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateGauss(String, String, String, double, JodaCompatibleZonedDateTime)

Date functions only works on dates in  the default format and default time zone
2018-11-20 16:10:06 -05:00
Ryan Ernst c07ad67718 Add 6.5.2 version 2018-11-20 12:33:26 -08:00
Gordon Brown 4bda469861
Fix line lengths in misc other files (#35650) 2018-11-20 12:29:48 -07:00
Vladimir Dolzhenko 953c8586df
Fix PrimaryAllocationIT#testForceStaleReplicaToBePromotedToPrimary (#35728)
Closes 35497
2018-11-20 20:05:23 +01:00
Christoph Büscher e91f404d16
Remove remains of 'auto_generate_phrase_queries' (#35735)
This parameter in the `query_string` query was deprecated in 6.0 and ignored
since then. Its API methods and remaining uses can be removed in the upcoming
major version.

Relates to #35734
2018-11-20 16:07:17 +01:00
Jim Ferenczi a5f5ceb869
Remove remaining line length violations for o.e.index (#35652)
This commit removes line length violations in the classes under
org.elasticsearch.index.
2018-11-20 08:09:14 +01:00
Simon Willnauer 29ef442841
Add a `_freeze` / `_unfreeze` API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Benjamin Trent 7a779a9988
[TEST] fixing ClusterApplierService#testClusterStateUpdateLogging() (#35714) 2018-11-19 14:23:49 -06:00
Yannick Welsch 47ada69c46
Zen2: Move most integration tests to Zen2 (#35678)
Zen2 is now feature-complete enough to run most ESIntegTestCase tests. The changes in this PR
are as follows:
- ClusterSettingsIT is adapted to not be Zen1 specific anymore (it was using Zen1 settings).
- Some of the integration tests require persistent storage of the cluster state, which is not fully
implemented yet (see #33958). These tests keep running with Zen1 for now but will be switched
over as soon as that is fully implemented.
- Some very few integration tests are not running yet with Zen2 for other reasons, depending on
some of the other open points in #32006.
2018-11-19 21:15:29 +01:00
Andrey Ershov f9ecd0c49e
[Zen2] Write manifest file (#35049)
Elasticsearch node is responsible for storing cluster metadata. 
There are 2 types of metadata: global metadata and index metadata. 
`GatewayMetaState` implements `ClusterStateApplier` and receives all 
`ClusterStateChanged` events and is responsible for storing modified 
metadata to disk. 

When new `ClusterStateChanged` event is received, `GatewayMetaState` 
checks if global metadata has changed and if it's the case writes new 
global metadata to disk. After that `GatewayMetaState` checks if index 
metadata has changed or there are new indices assigned to this node and 
if it's the case writes new index metadata to disk. Atomicity of global 
metadata and index metadata writes is ensured by `MetaDataStateFormat` 
class.

Unfortunately, there is no atomicity when more than one metadata changes
(global and index, or metadata for two indices). And atomicity is 
important for Zen2 correctness.
This commit adds atomicity by adding a notion of manifest file, 
represented by `MetaState` class. `MetaState` contains pointers to
current metadata.
More precisely, it stores global state generation as long and map from 
`Index` to index metadata generation as long. Atomicity of writes for 
manifest file is ensured by `MetaStateFormat` class.

The algorithm of writing changes to the disk would be the following:

1. Write global metadata state file to disk and remember
it's generation.
2. For each new/changed index write state file to disk and remember
it's generation. For each not-changed index use generation from 
previous manifest file. If index is removed or this node is no longer
responsible for this index - forget about the index.
3. Create `MetaState` object using previously remembered generations and
write it to disk.
4. Remove old state files for global metadata, indices metadata and 
manifest.

Additonally new implementation relies on enhanced `MetaDataStateFormat` 
failure semantics, `applyClusterState` throws IOException, whose 
descendant `WriteStateException` could be (and should be in Zen2) 
explicitly handled.
2018-11-19 19:49:44 +01:00
Gordon Brown b2057138a7
Remove AbstractComponent from AbstractLifecycleComponent (#35560)
AbstractLifecycleComponent now no longer extends AbstractComponent. In
order to accomplish this, many, many classes now instantiate their own
logger.
2018-11-19 09:51:32 -07:00
Nik Everett 036c15d8a3
Build: Fix official plugins list (#35661)
The list of official plugins accidentally included `qa` projects like,
well, `qa` and `amazon-ec2`. This changes the mechanism that we use to
build the list and adds a test to catch this.

Closes #35623
2018-11-19 10:17:05 -05:00
Ekal Golas 450db7fcf7 [Tests] Fix slowness of AutoDateHistogramAggregatorTests (#35072)
Randomize test assertion and test set size instead of asserting on an
exhaustive list of dates with fixed test set size. Also refactor common 
objects used to avoid recreating them, avoid date to string conversion
and reduce duplicate test code

Closes #33181
2018-11-19 15:21:01 +01:00
Martijn van Groningen c468d928b8
Remove line length violations in o.e.indices package (#35647)
Relates to #34884
2018-11-19 07:36:24 +01:00
Lee Hinman 22933ebcd5
Fix line length for org.elasticsearch.action.* files (#35607)
This removes the checkstyle suppressions for things in the `action` package.

Relates to #34884
2018-11-16 14:11:39 -07:00
Arthur Gavlyukovskiy 022726011c Remove use of AbstractComponent in server (#35444)
Removed extending of AbstractComponent and changed logger usage to
explicit declaration. Abstract classes still have logger
declaration using this.getClass() in order to show implementation class
name in its logs.

See #34488
2018-11-16 16:10:32 -05:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
Jack Conradson d62bbca56d
[Scripting] Use Number as a return value for BucketAggregationScript (#35653)
This change fixes #35351. Users were no longer able to return types of numbers other than doubles for bucket aggregation scripts. This change reverts to the previous behavior of being able to return any type of number and having it converted to a double outside of the script.
2018-11-16 12:33:01 -08:00
Jernej Klancic baf33b3162 Removes AbstractComponent from several classes (#35566)
Removes inhertiting from AbstractComponent for some classes (mostly
in the plugins module).

Relates to #34488
2018-11-16 20:50:18 +01:00
Julie Tibshirani 40ba4de5e6
Deprecate types in validate query requests. (#35575) 2018-11-16 08:59:04 -08:00
Tanguy Leroux d3d7c0158e Revert "[RCI] Check blocks while having index shard permit in TransportReplicationAction (#35332)"
This reverts commit 31567cefb4.
2018-11-16 15:38:51 +01:00
Tom Callahan 76b77dbaa4
Remove remaining line length violations for o.e.action.admin.cluster (#35156)
This inserts newlines in order to reduce line lengths in the
o.e.action.admin.cluster package to 140 characters or less.  This
also remves the checkstyle suppressions for affected files.

Relates #34884, #34923
2018-11-16 09:37:42 -05:00
Christoph Büscher 2da4bc85cf
Remove usages of CharSequence in Sets (#35501)
The javadocs of the CharSequence interface state that not all of its
implementations define the general contracts of the Object#equals and
Object#hashCode methods, therefore it is dangerous to use different CharSequence
instances as elements in a set or as keys in a map. While we probably mostly use
Strings in sets, in some places this is not enforced. To prevent this from
accidentally happening, this change replaces all occurances of Set<CharSequence>
which are currently mostly used in the completion suggester code with the more
concrete usage of Set<String>.
2018-11-16 09:35:22 +01:00
Jim Ferenczi 903d27a8fd
Fix phrase_slop in query_string query (#35533)
This change ensures that custom `phrase_slop`s are correctly handled
in the `query_string` query.

Fixes #35125
2018-11-16 09:08:07 +01:00
Lee Hinman ce35d049e9 [TEST] Fix ClusterApplierServiceTests.testClusterStateUpdateLogging
This changes the test to not use a `CountDownlatch`, instead adding an assertion
for the final logging message and waiting until the `MockAppender` has seen it
before proceeding.

Resolves #23739
2018-11-15 14:15:23 -07:00
David Turner 86ef041539
[Zen2] Introduce ClusterBootstrapService (#35488)
Today, the bootstrapping of a Zen2 cluster is driven externally, requiring
something else to wait for discovery to converge and then to inject the initial
configuration. This is hard to use in some situations, such as REST tests.

This change introduces the `ClusterBootstrapService` which brings the bootstrap
retry logic within each node and allows it to be controlled via an (unsafe)
node setting.
2018-11-15 20:09:22 +00:00
Jim Ferenczi c7a2c6d549
Handle IndexOrDocValuesQuery in composite aggregation (#35392)
The `composite` aggregation can optimize its execution when the query
is a `match_all` or a `range` over the field that is used in the first source
of the aggregation. However we only check for instances of `PointRangeQuery` whereas
the range query builder creates an  `IndexOrDocValuesQuery`. This means that
today the optimization does not apply to `range` query even if the code could handle it.
This change fixes this issue by extracting the index query inside `IndexOrDocValuesQuery`.
2018-11-15 17:52:06 +01:00
David Turner 928ab6a5b1
Suppress CachedTimeThread in hot threads output (#35558)
In #23175 we renamed `ThreadPool$EstimatedTimeThread` to
`ThreadPool$CachedTimeThread` but did not update the corresponding entry in
`HotThreads#isIdleThread`. This commit addresses this.
2018-11-15 08:25:09 +00:00
Tanguy Leroux c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
David Turner 135c3f0f07 Merge branch 'master' into zen2 2018-11-15 08:24:26 +00:00
Tanguy Leroux 5cd9b518ac
Fix usages of XContentParserUtils.ensureExpectedToken() (#35530)
Some usages of XContentParserUtils.ensureExpectedToken() are inverting
the expected and actual tokens, resulting in wrong error messages.
2018-11-15 09:18:32 +01:00
Hendrik Muhs fc774a3776
add ES 6.5.1 (on master) (#35549)
add ES 6.5.1
2018-11-14 21:08:21 +01:00
Tanguy Leroux c8c8ce2374
Extract RunOnce into a dedicated class (#35489)
This commit extracts the static inner class RunOnce from 
WorkerBulkByScrollTaskState so that it can be reused at 
other places.
2018-11-14 17:33:04 +01:00
Andrey Ershov 045fdd0d3b Merge master into zen2 2018-11-14 15:37:13 +03:00
Tanguy Leroux bbe50e7a86
Remove LoggingRunnable class (#35486)
This commit removes the unused LoggingRunnable class.
2018-11-14 10:12:25 +01:00
Tanguy Leroux 31567cefb4
[RCI] Check blocks while having index shard permit in TransportReplicationAction (#35332)
Today, the TransportReplicationAction checks the global level blocks and 
the index level blocks before routing the operation to the primary, in the 
ReroutePhase, and it happens at the very beginning of the transport 
replication action execution. For the upcoming rework of the Close Index 
API and in order to deal with primary relocation, we'll need to also check 
for blocks before executing the operation on the primary (while holding a 
permit) but before routing to the new primary.

This pull request change the AsyncPrimaryAction so that it checks for 
replication action's blocks before executing the operation locally or before 
routing the primary action to the newly primary shard. The check is done 
while holding a PrimaryShardReference.

Related to #33888
2018-11-14 09:43:55 +01:00
Hendrik Muhs 5c84708ee5 test: expose error message on failure 2018-11-14 08:25:41 +01:00
Christoph Büscher d8b1c23e1d
Remove Comparable interface from ScoreAccessor (#35519)
The way ScoreAccessor implements `compareTo()` is problematic because it doesn't
completely follow the Comparable contract, specificaly symmetry (if x is a
ScoreAccessor and y any Number then x.comparTo(y) works, but y.compareTo(x)
generally does not even compile). Fortunately we don't seem to use the fact that
ScoreAccessor is a Comparable anywhere, so we can simply remove it.
2018-11-14 05:58:05 +01:00
David Turner 229637fd7e
[Zen2] Remove duplicate discovered peers (#35505)
Today the `PeerFinder` probes each address it obtains, identifies the node to
which it just connected, and then returns all such nodes. However, this can
lead to duplicates if a node manages to connect to another node via two
distinct addresses.  This causes bootstrapping to fail since
`BootstrapConfiguration#resolve` forbids duplicates.

This change alters the behaviour of the `PeerFinder` to remove duplicates in
this situation.
2018-11-13 22:30:36 +00:00
Vladimir Dolzhenko 9728119b82 [CI] AllocationIdIT testFailedRecoveryOnAllocateStalePrimaryRequiresAnotherAllocateStalePrimary failure
Closes #35504
2018-11-13 20:57:50 +01:00
David Turner 8e40a2bbe2
[Zen2] Introduce vote withdrawal (#35446)
If shutting down half or more of the master-eligible nodes, their votes must
first be explicitly withdrawn to ensure that the cluster doesn't lose its
quorum. This works via _voting tombstones_, stored in the cluster state, which
tell the reconfigurator to remove nodes from the voting configuration.

This change introduces voting tombstones to the cluster state, together with
transport APIs for adding and removing them, and makes use of these APIs in
`InternalTestCluster` to support tests which remove at least half of the
master-eligible nodes at once (e.g. shrinking from two master-eligible nodes to
one).
2018-11-13 19:32:32 +00:00
David Turner 0e1a12122c Merge branch 'master' into zen2 2018-11-13 15:25:35 +00:00
David Turner fbd3cab410
[Zen2] Remove AbstractComponent usage (#35483)
AbstractComponent was deprecated in #35140 and is looking like it will be
removed at some point by #34888. Today all it does is provide a logger. This
change removes the usages of AbstractComponent that live solely in the zen2
feature branch to avoid some future merge pain, and replaces it where necessary
with some directly-created loggers.
2018-11-13 15:20:49 +00:00
Simon Willnauer 3229dfc4de
Allow efficient can_match phases on frozen indices (#35431)
This change adds a special caching reader that caches all relevant
values for a range query to rewrite correctly in a can_match phase
without actually opening the underlying directory reader. This
allows frozen indices to be filtered with can_match and in-turn
searched with wildcards in a efficient way since it allows us to
exclude shards that won't match based on their date-ranges without
opening their directory readers.

Relates to #34352
Depends on #34357
2018-11-13 14:53:55 +01:00
Christoph Büscher 0a6614a03a
Correct implemented interface of ParsedReverseNested (#35455)
The ParsedReverseNested implementation should implement the ReverseNested
interface and not the Nested interface. Although this is an empty marker
interface it is confusing and can lead to casting errors. Also adding a test to
check that both ParsedNested and ParsedReverseNested implement the correct
interface.

Closes #35449
2018-11-13 10:34:29 +01:00
Jason Tedor a18b599d64
Handle OS pretty name on old OS without OS release (#35453)
Some very old ancient versions of Linux do not have /etc/os-release. For
example, old Red Hat-like OS. This commit adds a fallback for handling
pretty name for these OS.
2018-11-12 19:31:12 -05:00
Tim Brooks 71cfb730f6
Register remote cluster compress setting (#35464)
This is a follow up to #35357. That commit failed to register the new
cluster.remote.cluster_name.transport.compress setting with
`ClusterSettings`. This commit fixes that.
2018-11-12 16:07:42 -07:00
Igor Motov e7896bcefc
Geo: enables coerce support in WKT polygon parser (#35414)
WKT parser now automatically closes open polygons similar to GeoJSON
parser if coerce flag in mapping is set to true.

Closes to #35059
2018-11-12 09:40:04 -10:00
Jason Tedor 40ca62c298
Address handling of OS pretty name on some OS (#35451)
Some OS (e.g., Oracle Linux Server 6.9) have a trailing space at the end
of the PRETTY_NAME line in /etc/os-release. This commit addresses this
by accounting for this trailing space when extracting the pretty name.
2018-11-12 14:27:57 -05:00
Yannick Welsch d2ff01af13
Zen2: Add basic Zen1 transport-level BWC (#35443)
Implements serialization compatibility between Zen1 and Zen2 transport action, allowing a Zen1 node to join a fully formed Zen2 cluster and vice-versa.
2018-11-12 19:31:10 +01:00
Nick Knize 2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00
Yannick Welsch fe29b18c26 Fix compilation 2018-11-12 11:05:11 +01:00
Yannick Welsch 4e6c58c942 Merge remote-tracking branch 'elastic/master' into zen2 2018-11-12 10:03:59 +01:00
Christoph Büscher 09cac321e7
Upgrade to Joda 2.10.1 (#35410)
This version contains a bugfix that allows us to reenable one of our muted tests
in DateTimeUnitTests.

Closes #33749
2018-11-12 10:02:41 +01:00
Tim Brooks ba478827ad
Improve MockTcpTransport memory usage (#35402)
The MockTcpTransport is not friendly in regards to memory usage. It must
allocate multiple byte arrays for every message. This improves the
memory situation by failing fast if the message is improperly formatted.
Additionally, it uses reusable big arrays for at least half of the
allocated byte arrays.
2018-11-09 10:12:49 -07:00
David Turner f69a5c9b3c Fix compile error introduced by conflict in previous two commits 2018-11-09 15:50:11 +00:00
Jim Ferenczi 7054e289fa
Add trace log of the request for the query and fetch phases (#34479)
This change adds a logger for the query and fetch phases that prints all requests
before their execution at the trace level. This will help debugging cases where an issue
occurs during the execution since only completed queries are logged by the slow logs.
2018-11-09 09:41:51 +01:00
Tim Brooks bccc99c2be
Fix TcpTransport compression test (#35396)
This commit fixes an assertion in the TcpTransportTests compresssion
test.
2018-11-08 18:04:48 -07:00
Tim Brooks 93c2c604e5
Move compression config to ConnectionProfile (#35357)
This is related to #34483. It introduces a namespaced setting for
compression that allows users to configure compression on a per remote
cluster basis. The transport.tcp.compress remains as a fallback
setting. If transport.tcp.compress is set to true, then all requests
and responses are compressed. If it is set to false, only requests to
clusters based on the cluster.remote.cluster_name.transport.compress
setting are compressed. However, after this change regardless of any
local settings, responses will be compressed if the request that is
received was compressed.
2018-11-08 10:37:59 -07:00
Jason Tedor 5c2a5f2e37
Adjust BWC version on OS pretty name
This commit adjusts the BWC version the OS pretty name field on OsInfo
now that this field has been backported to the 6.x development branch.
2018-11-08 12:24:10 -05:00
Jason Tedor 730ec1ddfb
Add more detailed OS name on Linux (#35352)
Today our OS information returned in node stats only returns a
high-level name of the OS (e.g., "Linux"). Yet, for some uses this is
too high-level and knowing at a finer level of granularity the
underlying OS can be useful. This commit extracts the pretty name on
Linux from /etc/os-release. This pretty name usually includes the Linux
vendor and the Linux vendor version number (e.g., Fedora 28).
2018-11-08 12:16:58 -05:00
Yannick Welsch c315ead0ac
Zen2: Add diff-based publishing (#35290)
Enables diff-based publishing, which is an optimization where only the changing parts of the cluster
state are published to the nodes in the cluster, falling back to full cluster state publishing if the
receiver does not have the previous cluster state.
2018-11-08 17:16:09 +01:00
David Turner 6885a7cb0f
Introduce transport API for cluster bootstrapping (#34961)
- Introduces a transport API for bootstrapping a Zen2 cluster
- Introduces a transport API for requesting the set of nodes that a
  master-eligible node has discovered and for waiting until this comprises the
  expected number of nodes.
- Alters ESIntegTestCase to use these APIs when forming a cluster, rather than
  injecting the initial configuration directly.
2018-11-08 16:09:37 +00:00
Christoph Büscher 113af7996c
Make limit on number of expanded fields configurable (#35284)
Currently we introduced a hard limit of 1024 to the number of fields a query can
be expanded to in #26541. Instead of using a hard limit, we should make this
configurable. This change removes the hard limit check and uses the existing
`max_clause_count` setting instead.

Closes #34778
2018-11-08 17:04:40 +01:00
Daniel Mitterdorfer 6980feddd2
Remove unused class MemoryCircuitBreaker
The class `MemoryCircuitBreaker` is unused so we remove all its traces
from the code base.

Relates #35367
2018-11-08 15:33:24 +01:00
David Turner 77789a733d Merge branch 'master' into 2018-11-08-merge-master 2018-11-08 13:38:18 +00:00
Alpar Torok 518e0de078 Mute test #35365 2018-11-08 12:27:40 +02:00
Christoph Büscher 14b811446f
Preserve `date_histogram` format when aggregating on unmapped fields (#35254)
Currently when aggregating on an unmapped date field (e.g. using a
date_histogram) we don't preserve the aggregations `format` setting but instead
use the default format. This can lead to loosing the aggregations `format` when
aggregating over several indices where some of them contain unmapped date fields
and are encountered first in the reduce phase.

Related to #31760
2018-11-08 10:22:25 +01:00
Jim Ferenczi 891fdda68e
Allow unmapped fields in composite aggregations (#35331)
Today the `composite` aggregation throws an error if a source targets an
unmapped field and `missing_bucket` is set to false. Documents without a
value for a source cannot produce any bucket if `missing_bucket` is not
activated so the error is a shortcut to say that the response will be empty.
However this is not consistent with the `terms` aggregation which accepts
unmapped field by default even if the response is also guaranteed to be empty.
This commit removes this restriction, if a source contains an unmapped field
we now return an empty response (no buckets).

Closes #35317
2018-11-08 09:30:52 +01:00
Tanguy Leroux 1703a61fec
[RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListener<Releasable>) (#34902)
The current implementation of asyncBlockOperations() can be used to
execute some code once all indexing operations permits have been acquired,
 then releases all permits immediately after the code execution. This
 immediate release is not suitable for treatments that need to keep all
 permits over multiple execution steps.

This commit adds a new asyncBlockOperations() that exposes a Releasable,
 making it possible to acquire all permits and only release them all
 when needed by closing the Releasable. The existing blockOperations() 
method has been modified to delegate permit acquisition/releasing to this new
method.

Relates to #33888
2018-11-08 09:23:33 +01:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Ryan Ernst a4d979cfc8 Scripting: Add back lookup vars in score script (#34833)
The lookup vars under params (namely _fields and _source) were
inadvertently removed when scoring scripts were converted to using
script contexts. This commit adds them back, along with deprecation
warnings for those that should not be used.
2018-11-07 15:09:09 -08:00
Nhat Nguyen ed8732b161
Use soft-deleted docs to resolve strategy for engine operation (#35230)
A CCR test failure shows that the approach in #34474 is flawed.
Restoring the LocalCheckpointTracker from an index commit can cause both
FollowingEngine and InternalEngine to incorrectly ignore some deletes.

Here is a small scenario illustrating the problem:

1. Delete doc with seq=1 => engine will add a delete tombstone to Lucene

2. Flush a commit consisting of only the delete tombstone

3. Index doc with seq=0  => engine will add that doc to Lucene but soft-deleted

4. Restart an engine with the commit (step 2); the engine will fill its
LocalCheckpointTracker with the delete tombstone in the commit

5. Replay the local translog in reverse order: index#0 then delete#1

6. When process index#0, an engine will add it into Lucene as a live doc
and advance the local checkpoint to 1 (seq#1 was restored from the
commit - step 4).

7. When process delete#1, an engine will skip it because seq_no=1 is
less than or equal to the local checkpoint.

We should have zero document after recovering from translog, but here we
have one.

Since all operations after the local checkpoint of the safe commit are
retained, we should find them if the look-up considers also soft-deleted
documents. This PR fills the disparity between the version map and the
local checkpoint tracker by taking soft-deleted documents into account
while resolving strategy for engine operations.

Relates #34474
Relates #33656
2018-11-07 15:26:30 -05:00
Martijn van Groningen 8de3c6e618
Ignore date ranges containing 'now' when pre-processing a percolator query (#35160)
Today when a percolator query contains a date range then the query
analyzer extracts that range, so that at search time the `percolate` query
can exclude percolator queries efficiently that are never going to match.

The problem is that if 'now' is used it is evaluated at index time.
So the idea is to rewrite date ranges with 'now' to a match all query, 
so that the query analyzer can't extract it and the `percolate` query 
is  then able to evaluate 'now' at query time.
2018-11-07 20:41:27 +01:00
Simon Willnauer 0cc0fd2d15
Add a frozen engine implementation (#34357)
This change adds a `frozen` engine that allows lazily open a directory reader
on a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader
that also allows to release and reset the underlying index readers after any and before
secondary search phases.

Relates to #34352
2018-11-07 20:23:35 +01:00
Vladimir Dolzhenko f789d49fb3
Put a fake allocation id on allocate stale primary command (#34140)
removes fake allocation id after recovery is done

Relates to #33432
2018-11-07 20:18:11 +01:00
Simon Willnauer 2131e119d7
Apply `ignore_throttled` also to concrete indices (#35335)
Today we only apply `ingore_throttled` to expansions from wildcards,
date math expressions and aliases. Yet, this is tricky since we might
have resolved certain expressions in pre-filter steps like security.
It's more consistent to apply this logic to all expressions including
concrete indices.

Relates to #34354
2018-11-07 18:43:27 +01:00
Alpar Torok 5ae03195d3
Make version field names more meaningful (#35334)
* Consolidate the name of the qualified build version

* Field name in response should not be redundant
2018-11-07 18:36:02 +02:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Vladimir Dolzhenko a467a816bc
Fix DeleteRequest / GetRequest / UpdateRequest / ExplainRequest validation for null and/or empty id/type (#35314)
Closes #35297
2018-11-07 12:52:11 +01:00
Tim Brooks f395b1eace
Open node connections asynchronously (#35144)
This is related to #29023. Additionally at other points we have
discussed a preference for removing the need to unnecessarily block
threads for opening new node connections. This commit lays the groudwork
for this by opening connections asynchronously at the transport level.
We still block, however, this work will make it possible to eventually
remove all blocking on new connections out of the TransportService
and Transport.
2018-11-06 17:58:20 -07:00
Julie Tibshirani ec53288fc0
Remove include_type_name from the relevant APIs. (#35192)
We've decided that the bulk, delete, get, index, update, and search APIs should not
contain this request parameter, and we will instead accept both typed and typeless calls.
2018-11-06 14:33:48 -08:00