Commit Graph

43212 Commits

Author SHA1 Message Date
Simon Willnauer ff5dd14753
Fix test failures related to file corruption (#36530)
* Fix CorruptFileIT to also take last DV generation into account

We currently only prune old .liv generations. With soft_deletes it's important
to also prune DV generations.

* Fix CorruptionUtils to skip the footer bytes after the checksum is read.

Today we read a broken checksum since we also checksum the 8 footer bytes that include
the checksum algorithm and the footer magic.

Closes #36526
2018-12-12 16:21:02 +01:00
Tim Brooks 3a56bb0924
Do not zero out pages for bytes output stream (#36523)
Currently, the `BytesStreamOutput` always zeros out the underlying byte
pages when they are acquired. This should not be necessary as the stream
overwrites the underlying bytes as serialization occurs.
2018-12-12 07:55:17 -07:00
Tim Brooks e63d52af63
Move page size constants to PageCacheRecycler (#36524)
`PageCacheRecycler` is the class that creates and holds pages of arrays
for various uses. `BigArrays` is just one user of these pages. This
commit moves the constants that define the page sizes for the recycler
to be on the recycler class.
2018-12-12 07:00:50 -07:00
Jason Tedor bdb1e0e04e
Make node field in JoinRequest private (#36405)
This commit moves the node field in the JoinRequest object to be a
private field, adding a dedicated accessor. This is a minor breaking
change in that it is no longer possible for all callers to overwrite
this field, but that is a feature.
2018-12-12 08:50:21 -05:00
Max Penet dabe29e783 [Docs] Adding community clojure client built on java low level rest client (#36484) 2018-12-12 13:57:53 +01:00
David Turner aa43e0b2cc
[Zen2] Migrate no-master-block integration tests (#36502)
This change follows up on #36478 by migrating the affected integration tests to
use Zen2.
2018-12-12 12:52:34 +00:00
Martijn van Groningen 883940ad92
[CCR] Change AutofollowCoordinator to use wait_for_metadata_version (#36264)
Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from #35895
Relates to #33007
2018-12-12 12:47:24 +01:00
Tim Vernum c6de68c3b9
Fix realm config in SAML QA test (#36515)
An issue was introduced due to the merge of authorization_realms with
the change to use Affix Settings for realms.
The ".type" setting no longer exists as the type is now part of the
setting key.
2018-12-12 21:35:56 +11:00
Alpar Torok 562f09603b
plugin install: don't print download progress in batch mode (#36361)
* Don't print download progress in batch mode

With this change we will no longer provide the progress bar in batch
mode.
Assuming that this is mode is mainly for consumption by tools which
will serialize the output, we shouldn't print a progress bar to be
for every percentile.

* PR review
2018-12-12 12:24:39 +02:00
Alpar Torok f920f571d1
Fix possible missing class error (#36491)
- Use the `testRuntime` classpath when loading the tests instead of
compile
- simplify task action
- improove error message when classes are missing
2018-12-12 12:24:17 +02:00
Alpar Torok 09a9e2236c
Third party audit improovements (#36167)
- fix up to date checks to ignore elasticsearch jars. We were not scanning them but these still triggered a rebuild.
- add tests to assert correct behavior and up to date checks.
- make the task less verbose with `-i` and include the output only on errors.
2018-12-12 12:04:36 +02:00
Alpar Torok c00d0fc814
Test fixtures improovements (#36037)
* Upgrae plugin to latest and expose udp
* Explicit check for windows
* Rename the properties for the port numbers
* Tasks for pre and pos container actions
2018-12-12 12:00:47 +02:00
Martijn van Groningen 4a825e2e86
[CCR] Clean followed leader index UUIDs in auto follow metadata (#36408)
The auto follow coordinator keeps track of the UUIDs of indices that it has followed. The index UUID strings need to be cleaned up in the case that these indices are removed in the remote cluster.

Relates to #33007
2018-12-12 09:55:37 +01:00
Ryan Ernst c4f4378006
Core: Rework multi date formatter merging (#36447)
This commit moves the MergedDateFormatter to a package private class and
reworks joda DateFormatter instances to use that instead of a single
DateTimeFormatter with multiple parsers. This will allow the java and
joda multi formats to share the same format parsing method in a
followup.
2018-12-11 23:47:44 -08:00
Nhat Nguyen 1bb6f844fe AwaitsFix testCorruptFileThenSnapshotAndRestore
Tracked at #36526
2018-12-12 02:03:23 -05:00
Nhat Nguyen 3fb5a12b30 Upgrade to Lucene-8.0.0-snapshot-61e448666d (#36518)
Includes:
- LUCENE-8602: Share TermsEnum if possible while applying DV updates
2018-12-12 06:47:40 +01:00
Tim Vernum 143f151185
HLRC: Implement get-user-privileges API (#36292)
This adds the _security/user/_privileges API to the High
Level Rest Client.

This also makes some changes to the Java model for the Role APIs
in order to better accommodate the GetPrivileges API
2018-12-12 15:12:49 +11:00
Nik Everett 03daad9812
Re-deprecate xpack rollup endpoints (#36451)
Redeprecates the `/_xpack/rollup` endpoints in favor of `/_rollup`.

When we cleanup the rollup in a cluster containing 6.x nodes we need to
use `/_xpack/rollup` instead of `/_rollup` because the 6.x nodes don't
know about `/_rollup`. In those cases we must ignore the deprecation
warnings that the 7.0 node will return for the end point.

Closes #36044
2018-12-11 19:43:17 -05:00
Andrei Stefan de373060fb
SQL: non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions (#36358)
* Renamed DAY_OF_WEEK and WEEK_OF_YEAR functions to their ISO version and
added the same functions with different functionality.
* Rewritten the datetime functions documentation to follow the format of the other
functions documentation pages.
2018-12-12 02:29:02 +02:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later (#36141)
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates #33222

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
Tim Brooks 797f985067
Add version to handshake requests (#36171)
Currently our handshake requests do not include a version. This is
unfortunate as we cannot rely on the stream version since it is not the
sending node's version. Instead it is the minimum compatibility version.
The handshake request is currently empty and we do nothing with it. This
should allow us to add data to the request without breaking backwards
compatibility.

This commit adds the version to the handshake request. Additionally, it
allows "future data" to be added to the request. This allows nodes to craft
a version compatible response. And will properly handle additional data in
future handshake requests. The proper handling of "future data" is useful
as this is the only request where we do not know the other node's version.

Finally, it renames the TcpTransportHandshaker to
TransportHandshaker.
2018-12-11 16:09:28 -07:00
lcawl 55743aac47 [DOCS] Adds placeholder for alpha2 release notes 2018-12-11 14:26:41 -08:00
Mayya Sharipova 2f18325384
Deprecate types in update_by_query and delete_by_query (#36365)
Relates to #35190
2018-12-11 17:09:59 -05:00
Jack Conradson 8e988f6c06
[Painless] Add def to boxed type casts (#36506)
This adds casts for the def type to all standard boxed types. Prior to this certain 
casts such as def [long/Long] -> Double would fail which does not follow the 
goals of the Painless casting model to remove the need for explicit boxing. This 
also creates symmetry with the casts for the newly created bridge methods 
being called at run-time.
2018-12-11 14:06:38 -08:00
docteurjay c1e3bdd42d docs: Remove issue key from jira documentation (#36136)
This commit removes an incorrect "issues" key from the documentation.
see: https://developer.atlassian.com/server/jira/platform/rest-apis/

Fixes #36128
2018-12-11 15:17:15 -06:00
Nhat Nguyen f23701406b CCR/TEST: Enable soft-deletes in ShardChangesActionTests
Relates #36446
2018-12-11 15:00:09 -05:00
Jason Tedor f2e18a6bf6
Add check for minimum required Docker version (#36497)
Our Docker build uses a multi-stage Docker build. This requires Docker
version 17.05 or greater. Without an explicit check here, the build
fails in a mysterious way such as "invalid reference format" that is
hard to track down (Google searches for "Docker invalid reference
format" do not turn up anything useful). This commit refactors our
existing Docker checks, and adds a new one for the minimum Docker
version.
2018-12-11 14:33:53 -05:00
Luca Cavanna dafea3cc23
Minor search controller changes (#36479)
This commit contains a few minor changes to our search code:

- adjust the visibility of a couple of methods in our search code to package private from public or protected.
- make some of the `SearchPhaseController` methods static where possible
- rename one of the `SearchPhaseController#reducedQueryPhase` methods (used only for scroll requests) to `reducedScrollQueryPhase` without the `isScrollRequest` argument which was always set to `true`
- replace leniency in `SearchPhaseController#setShardIndex` with an assert to make sure that we never set the shard index twice
- remove two null checks where the checked field can never be null
- resolve an unchecked warning
- replace `List#toArray` invocation that creates an array providing the true size with array creation of length 0
- correct a couple of typos in comments
2018-12-11 20:24:29 +01:00
Luca Cavanna fb18b35347
Add default methods to DocValueFormat (#36480)
The different `DocValueFormat` implementors throw `UnsupportedOperationException` for methods that they don't support. That is perfectly fine, and quite common as not all implementors support all of the possible formats. This makes it hard though to trace back which implementors support which formats as they all implement the same methods.

This commit introduces default methods in the `DocValueFormat` interface so that all methods throw `UnsupportedOperationException` by default. This way implementors can override only the methods that they specifically support.
2018-12-11 20:19:01 +01:00
Julie Tibshirani eb733f404a Fix the mixed cluster REST test explain/11_basic_with_types. 2018-12-11 11:03:26 -08:00
Tim Brooks 790f8102e9
Modify `BigArrays` to take name of circuit breaker (#36461)
This commit modifies BigArrays to take a circuit breaker name and
the circuit breaking service. The default instance of BigArrays that
is passed around everywhere always uses the request breaker. At the
network level, we want to be using the inflight request breaker. So this
change will allow that.

Additionally, as this change moves away from a single instance of
BigArrays, the class is modified to not be a Releasable anymore.
Releasing big arrays was always dispatching to the PageCacheRecycler,
so this change makes the PageCacheRecycler the class that needs to be
managed and torn-down.

Finally, this commit closes #31435 be making the serialization of
transport messages use the inflight request breaker. With this change,
we no longer push the global BigArrays instnace to the network level.
2018-12-11 11:55:41 -07:00
James Baiera a07fb312fc
Move LoggedExec to minimumRuntime source set (#36453)
This moves LoggedExec to the minimumRuntime source set so that it correctly 
compiles for earlier Java versions, namely Java 8.
2018-12-11 13:49:24 -05:00
Martijn van Groningen 405a38131b
added 6.5.4 version 2018-12-11 19:38:59 +01:00
Christoph Büscher c595a73c70 Add test logging for #35644
Adding logging to confirm the suspicion that there might be another old index
lingering around in the test cluster in cases where this test occasionally
fails.
2018-12-11 19:07:32 +01:00
markharwood a9eccbcd02
Tests- added helper methods to ESRestTestCase for checking warnings (#36443)
Added helper methods to ESRestTestCase for checking warnings in mixed and current-version-only clusters.
This is supported by a new VersionSpecificWarningsHandler class with associated unit test.
Closes #36251
2018-12-11 17:30:15 +00:00
Andrei Stefan eead8a144b
SQL: move requests' parameters to requests JSON body (#36149) 2018-12-11 19:17:51 +02:00
David Turner 21d91f1268
[Zen2] Respect the no_master_block setting (#36478)
Today the Zen2 coordinator only applies a write block when there is no known
elected master, ignoring the `discovery.zen.no_master_block` setting. This
commit resolves this, applying the correct block according to the configuration
instead.
2018-12-11 16:40:19 +00:00
Nhat Nguyen 084e06e481
Require soft-deletes when access changes snapshot (#36446)
Today we do not enforce soft-deletes when accessing the Lucene changes
snapshot. This might lead to internal errors because we assume
soft-deletes are enabled in that code path.
2018-12-11 11:18:10 -05:00
Andrey Ershov 8b821706cc
Switch more tests to zen2 (#36367)
1. CCR tests work without any changes
2. `testDanglingIndices` require changes the source code (added TODO).
3. `testIndexDeletionWhenNodeRejoins` because it's using just two
nodes, adding the node to exclusions is needed on restart.
4. `testCorruptTranslogTruncationOfReplica` starts dedicated master
one, because otherwise, the cluster does not form, if nodes are stopped
and one node is started back.
5. `testResolvePath` needs TEST cluster, because all nodes are stopped
at the end of the test and it's not possible to perform checks needed
by SUITE cluster.
6. `SnapshotDisruptionIT`. Without changes, the test fails because Zen2
retries snapshot creation as soon as network partition heals. This
results into the race between creating snapshot and test cleanup logic
(deleting index). Zen1 on the
other hand, also schedules retry, but it takes some time after network
partition heals, so cleanup logic executes latter and test passes. The
check that snapshot is eventually created is added to
the end of the test.
2018-12-11 17:12:17 +01:00
Jack Conradson 13b1f19772
[Painless] Add extensive tests for def to primitive casts (#36455)
This adds tests for each possible cast of def to a primitive type both implicit 
and explicit. This also fixes a minor bug where we were only checking the type 
of a def to be Number in some explicit casts. This does not work because it 
allows possible unintended casts from BigInteger and BigDecimal to primitive 
types since they both extend Number but are not included as part of the 
Painless casting model.
2018-12-11 08:03:08 -08:00
Gordon Brown 6481f2e380
Add setting to bypass Rollover action (#36235)
Adds a setting that indicates that an index is done indexing, set by ILM
when the Rollover action completes. This indicates that the Rollover
action should be skipped in any future invocations, as long as the index
is no longer the write index for its alias.

This enables 1) an index with a policy that involves the Rollover action
to have the policy removed and switched to another one without use of
the move-to-step API, and 2) integrations with Beats and CCR.
2018-12-11 08:53:05 -07:00
Jason Tedor c556b9ef19
Try running CI against Zulu (#36391)
This commit adds Zulu to the CI rotation. This commit only means that we
are going to try running CI against Zulu, it does not carry any
implications of support in the past, now, or in the future.
2018-12-11 10:48:14 -05:00
debadair c9e03e6ead
[DOCS] Reworked the shard allocation filtering info. (#36456)
* [DOCS] Reworked the shard allocation filtering info. Closes #36079

* Added multiple index allocation settings example back.

* Removed extraneous space
2018-12-11 07:44:57 -08:00
David Turner c3a6d1998a
Log [initial_master_nodes] on formation failure (#36466)
Today we log a slightly cryptic "cluster bootstrapping is disabled on this
node" message if bootstrapping hasn't been configured. Since there is today
only one way to bootstrap the cluster it seems preferable to spell out exactly
which setting is missing.
2018-12-11 12:53:05 +00:00
Christian Schneider 00eadd93eb converting ForbiddenPatternsTask to .java (#36194)
* converting ForbiddenPatternsTask to java impl & unit tests
2018-12-11 14:15:44 +02:00
Martijn van Groningen 5f7524bb89
fixed typo 2018-12-11 13:13:30 +01:00
Marios Trivyzas 031d0566b0
SQL: [docs] Add `beta[]` markup to all SQL pages (#36471)
Closes: #36437
2018-12-11 13:25:29 +02:00
Yu d01b30acba lower fielddata circuit breaker's default limit (#27162)
* Lower fielddata circuit breaker default limit

Lower fielddata circuit breaker default limit from 60% to 40% as we have
moved to doc_values for most of the cases.

* merge master in

* update tests

* update docs
2018-12-11 11:30:58 +01:00
Martijn van Groningen c635904a1b
muted mixed clusters explain/11_basic_with_types/Explain body without query element} test 2018-12-11 11:28:47 +01:00
Yannick Welsch 71cf6faaad
Only turn to follower when term bumping on follower check (#36449)
Deals with a situation where a follower becomes disconnected from the leader, but only for such a
short time where it becomes candidate and puts up a NO_MASTER_BLOCK, but then receives a
follower check from the leader. If the leader does not notice the node disconnecting, it is important
for the node not to be turned back into a follower but try and join the leader again.

We still should prefer the node into a follower on a follower check when this follower check triggers
a term bump as this can help during a leader election to quickly have a leader turn all other nodes
into followers, even before the leader has had the chance to transfer a possibly very large cluster
state.

Closes #36428
2018-12-11 11:22:23 +01:00