Commit Graph

44285 Commits

Author SHA1 Message Date
Christoph Büscher 15510da2af
Mute SharedClusterSnapshotRestoreIT#testAbortedSnapshotDuringInitDoesNotStart (#38304) 2019-02-04 10:41:35 +01:00
Boaz Leskes ff13a43144
Move ML Optimistic Concurrency Control to Seq No (#38278)
This commit moves the usage of internal versioning for CAS operations to use sequence numbers and primary terms

Relates to #36148
Relates to #10708
2019-02-04 10:41:08 +01:00
David Turner 1d82a6d9f9
Deprecate unused Zen1 settings (#38289)
Today the following settings in the `discovery.zen` namespace are still used:

- `discovery.zen.no_master_block`
- `discovery.zen.hosts_provider`
- `discovery.zen.ping.unicast.concurrent_connects`
- `discovery.zen.ping.unicast.hosts.resolve_timeout`
- `discovery.zen.ping.unicast.hosts`

This commit deprecates all other settings in this namespace so that they can be
removed in the next major version.
2019-02-04 08:52:08 +00:00
Armin Braun 4561f425db
Remove Redundandant Loop in SnapshotShardsService (#38283)
* This was a merge mistake on my end I think, obviously we only need to loop over the shards once not twice here to find those that we missed in INIT state
2019-02-04 09:06:39 +01:00
Alpar Torok d58e899d45
Remove empty service files (#38192) 2019-02-04 10:05:04 +02:00
Tim Vernum 0164acb0a7
Cleanup construction of interceptors (#38294)
It would be beneficial to apply some of the request interceptors even
when features are disabled. This change reworks the way we build that
list so that the interceptors we always want to use are constructed
outside of the settings check.
2019-02-04 17:27:41 +11:00
Costin Leau 75f0750ff7
SQL: Remove exceptions from Analyzer (#38260)
Instead of throwing an exception, use an unresolved attribute to pass
the message to the Verifier.
Additionally improve the parser to save the extended source for the
Aggregate and OrderBy.

Close #38208
2019-02-03 22:32:16 +02:00
Costin Leau a088155f4d
SQL: Move metrics tracking inside PlanExecutor (#38259)
Move metrics in one place, from the transport layer inside the
PlanExecutor
Remove unused class

Close #38258
2019-02-03 22:31:35 +02:00
Christoph Büscher 820029522b
Mute DateProcessorTests#testJodaPatternLocale (#38265)
Only fails on FIPS 8, muting this selectively.
2019-02-03 19:52:53 +01:00
Jason Tedor d2cc1459a3
Fix ordering problem in add or renew lease test (#38280)
We have to set the primary term before we add a retention lease,
otherwise we can not assert the correct primary term.
2019-02-03 12:54:31 -05:00
Christoph Büscher 6ca7a913ea
Mute ReplicationTrackerRetentionLeaseTests#testAddOrRenewRetentionLease (#38275) 2019-02-03 12:54:13 +01:00
Albert Zaharovits 3c1544d259
Fix NPE in Logfile Audit Filter (#38120)
The culprit in #38097 is an `IndicesRequest` that has no indices,
but instead of `request.indices()` returning `null` or `String[0]`
it returned `String[] {null}` . This tripped the audit filter.

I have addressed this in two ways:
1. `request.indices()` returning `String[] {null}` is treated as `null`
    or `String[0]`, i.e. no indices
2. `null` values among the roles and indices lists, which are
    unexpected, will never again stumble the audit filter; `null` values
    are treated as special values that will not match any policy,
    i.e. their events will always be printed.

Closes #38097
2019-02-03 10:34:17 +02:00
Armin Braun 89d7c57bd9
Fix Incorrect Transport Response Handler Type (#38264)
* Fix Incorrect Transport Response Handler Type
* The response type here is not empty and was always wrong but this only became visible now that 0a604e3b24 was introduced
   * As a result of 0a604e3b24 we started actually handling the response
of this request and logging/handling exceptions before that we simply dropped the classcast exception here quietly using the empty response handler
* fix busy assert not handling `Exception`
* Closes #38226
* Closes #38256
2019-02-03 08:48:15 +01:00
Nhat Nguyen 0861dc3581
Mute testCanRunUnsafeBootstrapAfterErroneousDetachWithoutLoosingMetaData (#38268)
Tracked at #38267
2019-02-02 20:02:21 -05:00
Andrei Stefan 6968f0925b
SQL: Generate relevant error message when grouping functions are not used in GROUP BY (#38017)
* Add checks for Grouping functions restriction to be placed inside GROUP BY
* Fixed bug where GROUP BY HISTOGRAM (not using alias) wasn't recognized
properly in the Verifier due to functions equality not working correctly.
2019-02-02 22:05:47 +02:00
Nhat Nguyen 75abb5b8a6
Adapt LLRest warning exception in FullClusterRestartIT (#38253)
We now throw a WarningFailureException instead of ResponseException if
there's any warning in a response. This change leads to the failures of
testSnapshotRestore in the BWC builds for the last two days.

Relates #37247
2019-02-02 12:09:14 -05:00
Christoph Büscher 50cdc61874
Mute DedicatedClusterSnapshotRestoreIT#testRestoreShrinkIndex (#38257) 2019-02-02 13:46:29 +01:00
David Turner c311062476
Add CoordinatorTests for empty unicast hosts list (#38209)
Today we have DiscoveryDisruptionIT tests for checking that discovery can still
work once the cluster has formed, even if the cluster is misconfigured and only
has a single master-eligible node in its unicast hosts list. In fact with Zen2
we can go one better: we do not need any nodes in the unicast hosts list,
because nodes also use the contents of the last-committed cluster state for
discovery. Additionally, the DiscoveryDisruptionIT tests were failing due to
the overenthusiastic fault-detection timeouts.

This commit replaces these tests with deterministic `CoordinatorTests` that
verify the same behaviour. It also removes some duplication by extracting a
test method called `testFollowerCheckerAfterMasterReelection()`

Closes #37687
2019-02-02 07:54:56 +00:00
Nhat Nguyen 80d3092292
Fix primary term in testAddOrRenewRetentionLease (#38239)
We should increase primary term before renewing leases; otherwise, the
term of the latest RetentionLeases will be lower than the current term.

Relates #37951
2019-02-02 02:38:53 -05:00
Nhat Nguyen 1ec04dff43
FIx testReplicaIgnoresOlderRetentionLeasesVersion (#38246)
If the innerLength is 0, the version won't be increased; then there will
be two RetentionLeases with the same term and version, but their leases
are different.

Relates #37951
Closes #38245
2019-02-02 02:37:37 -05:00
Gordon Brown 475a045192
Mute tests in SSLConfigurationReloaderTests (#38248)
Specifically `testReloadingTrustStore` and `testReloadingPEMTrustConfig`
2019-02-01 21:00:58 -07:00
Gordon Brown 7a1e89c7ed
Ensure ILM policies run safely on leader indices (#38140)
Adds a Step to the Shrink and Delete actions which prevents those
actions from running on a leader index - all follower indices must first
unfollow the leader index before these actions can run. This prevents
the loss of history before follower indices are ready, which might
otherwise result in the loss of data.
2019-02-01 20:46:12 -07:00
Nhat Nguyen 8bee5b8e06
Mute testAddOrRenewRetentionLease (#38240)
Relates #38239
2019-02-01 21:27:10 -05:00
Boaz Leskes f6e06a2b19 Adapt minimum versions for seq# powered operations in Watch related requests and UpdateRequest (#38231)
After backporting #37977, #37857 and #37872
2019-02-01 20:37:16 -05:00
Costin Leau 783c9ed372
SQL: Allow sorting of groups by aggregates (#38042)
Introduce client-side sorting of groups based on aggregate
functions. To allow this, the Analyzer has been extended to push down
to underlying Aggregate, aggregate function and the Querier has been
extended to identify the case and consume the results in order and sort
them based on the given columns.
The underlying QueryContainer has been slightly modified to allow a view
of the underlying values being extracted as the columns used for sorting
might not be requested by the user.

The PR also adds minor tweaks, mainly related to tree output.

Close #35118
2019-02-02 01:38:25 +02:00
Jack Conradson 630889baec
Remove extraneous test from Painless lambda tests (#38111)
This test has been awaiting a fix that isn't currently relevant because incoming
lambda parameters are read-only. If this ever changes a new set of tests can
be added that are up-to-date.
2019-02-01 15:10:59 -08:00
Jason Tedor f181e17038
Introduce retention leases versioning (#37951)
Because concurrent sync requests from a primary to its replicas could be
in flight, it can be the case that an older retention leases collection
arrives and is processed on the replica after a newer retention leases
collection has arrived and been processed. Without a defense, in this
case the replica would overwrite the newer retention leases with the
older retention leases. This commit addresses this issue by introducing
a versioning scheme to retention leases. This versioning scheme is used
to resolve out-of-order processing on the replica. We persist this
version into Lucene and restore it on recovery. The encoding of
retention leases is starting to get a little ugly. We can consider
addressing this in a follow-up.
2019-02-01 17:19:19 -05:00
Ioannis Kakavas 78a65c340d
Correctly disable tests for FIPS JVMs (#38214)
Replace assertFalse with assumeFalse

Resolves: #38212
2019-02-01 23:56:35 +02:00
Nhat Nguyen 9c39dea7ae
AwaitsFix testAbortedSnapshotDuringInitDoesNotStart (#38227)
Tracked at #38226
2019-02-01 16:24:02 -05:00
Tal Levy bae656dcea
Preserve ILM operation mode when creating new lifecycles (#38134)
There was a bug where creating a new policy would start
the ILM service, even if it was stopped. This change ensures
that there is no change to the existing operation mode
2019-02-01 13:16:34 -08:00
Nhat Nguyen 3ecdfe1060
Enable trace log in FollowerFailOverIT (#38148)
This suite still fails one per week sometimes with a worrying assertion.
Sadly we are still unable to find the actual source.

Expected: <SeqNoStats{maxSeqNo=229, localCheckpoint=86, globalCheckpoint=86}>
but: was   <SeqNoStats{maxSeqNo=229, localCheckpoint=-1, globalCheckpoint=86}>

This change enables trace log in the suite so we will have a better
picture if this fails again.

Relates #3333
2019-02-01 15:44:39 -05:00
Armin Braun 03a1d21070
SnapshotShardsService Simplifications (#38025)
* Instead of replacing the `shardSnapshots` field, we mutate it, explicitly removing entries from it in only a single spot
* Decreased the amount of indirection by moving all logic for starting a snapshot's newly discovered shard tasks into `startNewShards` (saves us two maps (keyed by snapshot) and iterations over them)
2019-02-01 20:46:14 +01:00
Julie Tibshirani c2e9d13ebd
Default include_type_name to false in the yml test harness. (#38058)
This PR removes the temporary change we made to the yml test harness in #37285
to automatically set `include_type_name` to `true` in index creation requests
if it's not already specified. This is possible now that the vast majority of
index creation requests were updated to be typeless in #37611. A few additional
tests also needed updating here.

Additionally, this PR updates the test harness to set `include_type_name` to
`false` in index creation requests when communicating with 6.x nodes. This
mirrors the logic added in #37611 to allow for typeless document write requests
in test set-up code. With this update in place, we can remove many references
to `include_type_name: false` from the yml tests.
2019-02-01 11:44:13 -08:00
Boaz Leskes 9350da98a7
Disable bwc preparing to backport of#37977, #37857 and #37872 (#38126) 2019-02-01 20:28:08 +01:00
Benjamin Trent a70f54fc77
Adding ml_settings entry to HLRC and Docs for deprecation_info (#38118) 2019-02-01 12:45:28 -06:00
Nhat Nguyen f64b20383e
Replace awaitBusy with assertBusy in atLeastDocsIndexed (#38190)
Unlike assertBusy, awaitBusy does not retry if the code-block throws an
AssertionError. A refresh in atLeastDocsIndexed can fail because we call
this method while we are closing some node in FollowerFailOverIT.
2019-02-01 13:31:17 -05:00
Luca Cavanna ee57420de6
Adjust SearchRequest version checks (#38181)
The finalReduce flag is now supported on 6.x too, hence we need to update the version checks in master.
2019-02-01 19:23:13 +01:00
Nhat Nguyen 70235838d1
AwaitsFix testClientSucceedsWithVerificationDisabled (#38213)
Tracked at #38212
2019-02-01 12:50:07 -05:00
Andrey Ershov 04dc41b99e
Zen2ify RareClusterStateIT (#38184)
In Zen 1 there are commit timeout and publish timeout and these
settings could be changed on-the-fly.

In Zen 2, there is only commit timeout and this setting is static.
RareClusterStateIT is actively using these settings and the fact, they
are dynamic.

This commit adds cancelCommitedPublication method to Coordinator to
be used by tests. This method will cancel current committed publication
if there is any.
When there is BlockClusterStateProcessing on the non-master node, the
publication will be accepted and committed, but not yet applied. So we
can use the method above to cancel it.

Also, this commit replaces callback + AtomicReference with ActionFuture,
which makes test code easier to read.
2019-02-01 18:18:11 +01:00
Benjamin Trent 5db305023d
ML: Fix error race condition on stop _all datafeeds and close _all jobs (#38113)
* ML: Ignore when task is not found for _all

* Addressing PR comments

* Update TransportStopDatafeedAction.java
2019-02-01 11:16:35 -06:00
Nhat Nguyen f5f3cb8f4c
AwaitsFix PUT mapping with _doc on an index that has types (#38204)
Tracked at #38202
2019-02-01 12:00:43 -05:00
Shaunak Kashyap cc7c42d7e2
Allow built-in monitoring_user role to call GET _xpack API (#38060)
This PR adds the `monitor/xpack/info` cluster-level privilege to the built-in `monitoring_user` role.

This privilege is required for the Monitoring UI to call the `GET _xpack API` on the Monitoring Cluster. It needs to do this in order to determine the license of the Monitoring Cluster, which further determines whether Cluster Alerts are shown to the user or not.

Resolves #37970.
2019-02-01 08:56:34 -08:00
Nick Knize 603cdf40f1
Update geo_shape docs to include unsupported features (#38138)
There are a two major features that are not yet supported by BKD Backed geo_shape: MultiPoint queries, and CONTAINS relation. It is important we are explicitly clear in the documentation that using the new approach may not work for users that depend on these features. This commit adds an IMPORTANT NOTE section to geo_shape docs that explicitly highlights these missing features and what should be done if they are an absolute necessity.
2019-02-01 10:41:41 -06:00
David Roberts 1fa413a16d
[ML] Remove "8" prefixes from file structure finder timestamp formats (#38016)
In 7.x Java timestamp formats are the default timestamp format and
there is no need to prefix them with "8".  (The "8" prefix was used
in 6.7 to distinguish Java timestamp formats from Joda timestamp
formats.)

This change removes the "8" prefixes from timestamp formats in the
output of the ML file structure finder.
2019-02-01 15:36:04 +00:00
Luca Cavanna 5c58c2508e
Disable bwc tests while backporting #38104 (#38182)
Relates to #38180
2019-02-01 16:34:24 +01:00
Jay Modi 2ca22209cd
Enable TLSv1.3 by default for JDKs with support (#38103)
This commit enables the use of TLSv1.3 with security by enabling us to
properly map `TLSv1.3` in the supported protocols setting to the
algorithm for a SSLContext. Additionally, we also enable TLSv1.3 by
default on JDKs that support it.

An issue was uncovered with the MockWebServer when TLSv1.3 is used that
ultimately winds up in an endless loop when the client does not trust
the server's certificate. Due to this, SSLConfigurationReloaderTests
has been pinned to TLSv1.2.

Closes #32276
2019-02-01 08:34:11 -07:00
Yannick Welsch 025bf28405
Fix _host based require filters (#38173)
Using index.routing.allocation.require._host does not correctly work because the boolean logic in
filter matching is broken (DiscoveryNodeFilters.match(...) will return false) when
opType ==OpType.AND
2019-02-01 16:02:37 +01:00
Tanguy Leroux da6269b456
RestoreService should update primary terms when restoring shards of existing indices (#38177)
When restoring shards of existing indices, the RestoreService also 
restores the values of primary terms stored in the snapshot index 
metadata. The primary terms are not updated and could potentially 
conflict with current index primary terms if the restored primary terms 
are lower than the existing ones.

This situation is likely to happen with replicated closed indices 
(because primary terms are increased when the index is transitioning 
from open to closed state, and the snapshotted primary terms are the
 one at the time the index was opened) (see #38024) and maybe also 
with CCR.

This commit changes the RestoreService so that it updates the primary 
terms using the maximum value between the snapshotted values and 
the existing values.

Related to #33888
2019-02-01 15:59:11 +01:00
Desmond Vehar c1c4abae10 Throw if two inner_hits have the same name (#37645)
This change throws an error if two inner_hits have the same name

Closes #37584
2019-02-01 15:53:50 +01:00
Alexander Reelsen 35ed137684
Ensure joda compatibility in custom date formats (#38171)
If custom date formats are used, there may be combinations that the new
performat DateFormatters.from() method has not covered yet. This adds a
few such corner cases and ensures the tests are correctly commented
out.
2019-02-01 15:42:56 +01:00