Commit Graph

43373 Commits

Author SHA1 Message Date
Tal Levy cd1bec3a06
[refactor] add Environment in BootstrapContext (#36573)
There are certain BootstrapCheck checks that may need access environment-specific
values. Watcher's EncryptSensitiveDataBootstrapCheck passes in the node's environment
via a constructor to bypass the shortcoming in BootstrapContext. This commit
pulls in the node's environment into BootstrapContext.

Another case is found in #36519, where it is useful to check the state of the
data-path. Since PathUtils.get and Paths.get are forbidden APIs, we rely on
the environment to retrieve references to things like node data paths.

This means that the BootstrapContext will have the same Settings used in the
Environment, which currently differs from the Node's settings.
2018-12-12 21:07:21 -08:00
Mayya Sharipova d40037c91e
Deprecate uses of _type as a field name in queries (#36503) 2018-12-12 21:21:53 -05:00
Mayya Sharipova b5d532f9e3
Vector field (#33022)
1. Dense vector

PUT dindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "dense_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT dinex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [ 0.5, 10, 6 ]
}

2. Sparse vector

PUT sindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "sparse_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT sindex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : {"1": 0.5, "99": -0.5,  "5": 1}
}
2018-12-12 21:20:53 -05:00
Tal Levy 9c1cdea839
[cleanup] remove deprecated references to dataWithClusterFiles (#36574)
data files under the cluster name subdirectory has been deprecated and was
meant to be removed in 6.0. This commit removes some leftover referrences to
these paths.
2018-12-12 16:28:11 -08:00
Gordon Brown 6a824322fc
Improve error message for deleting in-use policy (#36457)
The error message used when attempting to delete a lifecycle policy that
is in use previously only included one index which was using the policy.
It now includes all indices using that policy.
2018-12-12 14:57:48 -07:00
Julie Tibshirani 71a39d10be
Make sure that BWC tests run successfully, even with types deprecation messages. (#36511) 2018-12-12 12:57:32 -08:00
Alan Woodward 9ac7359643
Update lucene to snapshot-7e4555a2fd (#36563)
Includes the following:

* Reversion of doc-values changes in LUCENE-8374; we are interested in seeing if this 
  has an effect on benchmarks for node-stats and index-stats
* More improvements to docvalues updates
2018-12-12 20:18:32 +00:00
Jason Tedor 4a8cd45cca
Avoid blocking non-reproducible randomness in test (#36561)
The security documentation test uses
SecureRandom#getStrongInstance. This defaults to
securerandom.strongAlgorithms=NativePRNGBlocking:SUN,DRBG:SUN which
means a blocking implementation that reads from /dev/random. This means
that this test can stall if the entropy on the machine is
exhausted. Anyway, it also means that the randomness is
non-reproducible, a thing that we try to avoid in tests. This commit
switches to a boring randomness source to avoid the blocking, and to
keep the test reproducible.
2018-12-12 15:11:28 -05:00
Julie Tibshirani 33152f648f
Fix some inconsistencies in the types deprecation code. (#36517)
* Make sure to test conversion for both typed and typeless HLRC requests.
* Update a few more statements to deprecatedAndMaybeLog.
* Make sure Rest*SearchTemplateActionTests extend RestActionTestCase.
2018-12-12 10:38:02 -08:00
Tim Brooks 7f612d5dd8
Always compress based on the settings (#36522)
Currently TransportRequestOptions allows specific requests to request
compression. This commit removes this and always compresses based on the
settings. Additionally, it removes TransportResponseOptions as they
are unused.

This closes #36399.
2018-12-12 09:39:15 -07:00
Armin Braun 02d0f163a4
MINOR: Cleanup Dead Code in Snapshots Package (#36528) 2018-12-12 16:48:34 +01:00
Simon Willnauer ff5dd14753
Fix test failures related to file corruption (#36530)
* Fix CorruptFileIT to also take last DV generation into account

We currently only prune old .liv generations. With soft_deletes it's important
to also prune DV generations.

* Fix CorruptionUtils to skip the footer bytes after the checksum is read.

Today we read a broken checksum since we also checksum the 8 footer bytes that include
the checksum algorithm and the footer magic.

Closes #36526
2018-12-12 16:21:02 +01:00
Tim Brooks 3a56bb0924
Do not zero out pages for bytes output stream (#36523)
Currently, the `BytesStreamOutput` always zeros out the underlying byte
pages when they are acquired. This should not be necessary as the stream
overwrites the underlying bytes as serialization occurs.
2018-12-12 07:55:17 -07:00
Tim Brooks e63d52af63
Move page size constants to PageCacheRecycler (#36524)
`PageCacheRecycler` is the class that creates and holds pages of arrays
for various uses. `BigArrays` is just one user of these pages. This
commit moves the constants that define the page sizes for the recycler
to be on the recycler class.
2018-12-12 07:00:50 -07:00
Jason Tedor bdb1e0e04e
Make node field in JoinRequest private (#36405)
This commit moves the node field in the JoinRequest object to be a
private field, adding a dedicated accessor. This is a minor breaking
change in that it is no longer possible for all callers to overwrite
this field, but that is a feature.
2018-12-12 08:50:21 -05:00
Max Penet dabe29e783 [Docs] Adding community clojure client built on java low level rest client (#36484) 2018-12-12 13:57:53 +01:00
David Turner aa43e0b2cc
[Zen2] Migrate no-master-block integration tests (#36502)
This change follows up on #36478 by migrating the affected integration tests to
use Zen2.
2018-12-12 12:52:34 +00:00
Martijn van Groningen 883940ad92
[CCR] Change AutofollowCoordinator to use wait_for_metadata_version (#36264)
Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from #35895
Relates to #33007
2018-12-12 12:47:24 +01:00
Tim Vernum c6de68c3b9
Fix realm config in SAML QA test (#36515)
An issue was introduced due to the merge of authorization_realms with
the change to use Affix Settings for realms.
The ".type" setting no longer exists as the type is now part of the
setting key.
2018-12-12 21:35:56 +11:00
Alpar Torok 562f09603b
plugin install: don't print download progress in batch mode (#36361)
* Don't print download progress in batch mode

With this change we will no longer provide the progress bar in batch
mode.
Assuming that this is mode is mainly for consumption by tools which
will serialize the output, we shouldn't print a progress bar to be
for every percentile.

* PR review
2018-12-12 12:24:39 +02:00
Alpar Torok f920f571d1
Fix possible missing class error (#36491)
- Use the `testRuntime` classpath when loading the tests instead of
compile
- simplify task action
- improove error message when classes are missing
2018-12-12 12:24:17 +02:00
Alpar Torok 09a9e2236c
Third party audit improovements (#36167)
- fix up to date checks to ignore elasticsearch jars. We were not scanning them but these still triggered a rebuild.
- add tests to assert correct behavior and up to date checks.
- make the task less verbose with `-i` and include the output only on errors.
2018-12-12 12:04:36 +02:00
Alpar Torok c00d0fc814
Test fixtures improovements (#36037)
* Upgrae plugin to latest and expose udp
* Explicit check for windows
* Rename the properties for the port numbers
* Tasks for pre and pos container actions
2018-12-12 12:00:47 +02:00
Martijn van Groningen 4a825e2e86
[CCR] Clean followed leader index UUIDs in auto follow metadata (#36408)
The auto follow coordinator keeps track of the UUIDs of indices that it has followed. The index UUID strings need to be cleaned up in the case that these indices are removed in the remote cluster.

Relates to #33007
2018-12-12 09:55:37 +01:00
Ryan Ernst c4f4378006
Core: Rework multi date formatter merging (#36447)
This commit moves the MergedDateFormatter to a package private class and
reworks joda DateFormatter instances to use that instead of a single
DateTimeFormatter with multiple parsers. This will allow the java and
joda multi formats to share the same format parsing method in a
followup.
2018-12-11 23:47:44 -08:00
Nhat Nguyen 1bb6f844fe AwaitsFix testCorruptFileThenSnapshotAndRestore
Tracked at #36526
2018-12-12 02:03:23 -05:00
Nhat Nguyen 3fb5a12b30 Upgrade to Lucene-8.0.0-snapshot-61e448666d (#36518)
Includes:
- LUCENE-8602: Share TermsEnum if possible while applying DV updates
2018-12-12 06:47:40 +01:00
Tim Vernum 143f151185
HLRC: Implement get-user-privileges API (#36292)
This adds the _security/user/_privileges API to the High
Level Rest Client.

This also makes some changes to the Java model for the Role APIs
in order to better accommodate the GetPrivileges API
2018-12-12 15:12:49 +11:00
Nik Everett 03daad9812
Re-deprecate xpack rollup endpoints (#36451)
Redeprecates the `/_xpack/rollup` endpoints in favor of `/_rollup`.

When we cleanup the rollup in a cluster containing 6.x nodes we need to
use `/_xpack/rollup` instead of `/_rollup` because the 6.x nodes don't
know about `/_rollup`. In those cases we must ignore the deprecation
warnings that the 7.0 node will return for the end point.

Closes #36044
2018-12-11 19:43:17 -05:00
Andrei Stefan de373060fb
SQL: non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions (#36358)
* Renamed DAY_OF_WEEK and WEEK_OF_YEAR functions to their ISO version and
added the same functions with different functionality.
* Rewritten the datetime functions documentation to follow the format of the other
functions documentation pages.
2018-12-12 02:29:02 +02:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later (#36141)
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates #33222

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
Tim Brooks 797f985067
Add version to handshake requests (#36171)
Currently our handshake requests do not include a version. This is
unfortunate as we cannot rely on the stream version since it is not the
sending node's version. Instead it is the minimum compatibility version.
The handshake request is currently empty and we do nothing with it. This
should allow us to add data to the request without breaking backwards
compatibility.

This commit adds the version to the handshake request. Additionally, it
allows "future data" to be added to the request. This allows nodes to craft
a version compatible response. And will properly handle additional data in
future handshake requests. The proper handling of "future data" is useful
as this is the only request where we do not know the other node's version.

Finally, it renames the TcpTransportHandshaker to
TransportHandshaker.
2018-12-11 16:09:28 -07:00
lcawl 55743aac47 [DOCS] Adds placeholder for alpha2 release notes 2018-12-11 14:26:41 -08:00
Mayya Sharipova 2f18325384
Deprecate types in update_by_query and delete_by_query (#36365)
Relates to #35190
2018-12-11 17:09:59 -05:00
Jack Conradson 8e988f6c06
[Painless] Add def to boxed type casts (#36506)
This adds casts for the def type to all standard boxed types. Prior to this certain 
casts such as def [long/Long] -> Double would fail which does not follow the 
goals of the Painless casting model to remove the need for explicit boxing. This 
also creates symmetry with the casts for the newly created bridge methods 
being called at run-time.
2018-12-11 14:06:38 -08:00
docteurjay c1e3bdd42d docs: Remove issue key from jira documentation (#36136)
This commit removes an incorrect "issues" key from the documentation.
see: https://developer.atlassian.com/server/jira/platform/rest-apis/

Fixes #36128
2018-12-11 15:17:15 -06:00
Nhat Nguyen f23701406b CCR/TEST: Enable soft-deletes in ShardChangesActionTests
Relates #36446
2018-12-11 15:00:09 -05:00
Jason Tedor f2e18a6bf6
Add check for minimum required Docker version (#36497)
Our Docker build uses a multi-stage Docker build. This requires Docker
version 17.05 or greater. Without an explicit check here, the build
fails in a mysterious way such as "invalid reference format" that is
hard to track down (Google searches for "Docker invalid reference
format" do not turn up anything useful). This commit refactors our
existing Docker checks, and adds a new one for the minimum Docker
version.
2018-12-11 14:33:53 -05:00
Luca Cavanna dafea3cc23
Minor search controller changes (#36479)
This commit contains a few minor changes to our search code:

- adjust the visibility of a couple of methods in our search code to package private from public or protected.
- make some of the `SearchPhaseController` methods static where possible
- rename one of the `SearchPhaseController#reducedQueryPhase` methods (used only for scroll requests) to `reducedScrollQueryPhase` without the `isScrollRequest` argument which was always set to `true`
- replace leniency in `SearchPhaseController#setShardIndex` with an assert to make sure that we never set the shard index twice
- remove two null checks where the checked field can never be null
- resolve an unchecked warning
- replace `List#toArray` invocation that creates an array providing the true size with array creation of length 0
- correct a couple of typos in comments
2018-12-11 20:24:29 +01:00
Luca Cavanna fb18b35347
Add default methods to DocValueFormat (#36480)
The different `DocValueFormat` implementors throw `UnsupportedOperationException` for methods that they don't support. That is perfectly fine, and quite common as not all implementors support all of the possible formats. This makes it hard though to trace back which implementors support which formats as they all implement the same methods.

This commit introduces default methods in the `DocValueFormat` interface so that all methods throw `UnsupportedOperationException` by default. This way implementors can override only the methods that they specifically support.
2018-12-11 20:19:01 +01:00
Julie Tibshirani eb733f404a Fix the mixed cluster REST test explain/11_basic_with_types. 2018-12-11 11:03:26 -08:00
Tim Brooks 790f8102e9
Modify `BigArrays` to take name of circuit breaker (#36461)
This commit modifies BigArrays to take a circuit breaker name and
the circuit breaking service. The default instance of BigArrays that
is passed around everywhere always uses the request breaker. At the
network level, we want to be using the inflight request breaker. So this
change will allow that.

Additionally, as this change moves away from a single instance of
BigArrays, the class is modified to not be a Releasable anymore.
Releasing big arrays was always dispatching to the PageCacheRecycler,
so this change makes the PageCacheRecycler the class that needs to be
managed and torn-down.

Finally, this commit closes #31435 be making the serialization of
transport messages use the inflight request breaker. With this change,
we no longer push the global BigArrays instnace to the network level.
2018-12-11 11:55:41 -07:00
James Baiera a07fb312fc
Move LoggedExec to minimumRuntime source set (#36453)
This moves LoggedExec to the minimumRuntime source set so that it correctly 
compiles for earlier Java versions, namely Java 8.
2018-12-11 13:49:24 -05:00
Martijn van Groningen 405a38131b
added 6.5.4 version 2018-12-11 19:38:59 +01:00
Christoph Büscher c595a73c70 Add test logging for #35644
Adding logging to confirm the suspicion that there might be another old index
lingering around in the test cluster in cases where this test occasionally
fails.
2018-12-11 19:07:32 +01:00
markharwood a9eccbcd02
Tests- added helper methods to ESRestTestCase for checking warnings (#36443)
Added helper methods to ESRestTestCase for checking warnings in mixed and current-version-only clusters.
This is supported by a new VersionSpecificWarningsHandler class with associated unit test.
Closes #36251
2018-12-11 17:30:15 +00:00
Andrei Stefan eead8a144b
SQL: move requests' parameters to requests JSON body (#36149) 2018-12-11 19:17:51 +02:00
David Turner 21d91f1268
[Zen2] Respect the no_master_block setting (#36478)
Today the Zen2 coordinator only applies a write block when there is no known
elected master, ignoring the `discovery.zen.no_master_block` setting. This
commit resolves this, applying the correct block according to the configuration
instead.
2018-12-11 16:40:19 +00:00
Nhat Nguyen 084e06e481
Require soft-deletes when access changes snapshot (#36446)
Today we do not enforce soft-deletes when accessing the Lucene changes
snapshot. This might lead to internal errors because we assume
soft-deletes are enabled in that code path.
2018-12-11 11:18:10 -05:00
Andrey Ershov 8b821706cc
Switch more tests to zen2 (#36367)
1. CCR tests work without any changes
2. `testDanglingIndices` require changes the source code (added TODO).
3. `testIndexDeletionWhenNodeRejoins` because it's using just two
nodes, adding the node to exclusions is needed on restart.
4. `testCorruptTranslogTruncationOfReplica` starts dedicated master
one, because otherwise, the cluster does not form, if nodes are stopped
and one node is started back.
5. `testResolvePath` needs TEST cluster, because all nodes are stopped
at the end of the test and it's not possible to perform checks needed
by SUITE cluster.
6. `SnapshotDisruptionIT`. Without changes, the test fails because Zen2
retries snapshot creation as soon as network partition heals. This
results into the race between creating snapshot and test cleanup logic
(deleting index). Zen1 on the
other hand, also schedules retry, but it takes some time after network
partition heals, so cleanup logic executes latter and test passes. The
check that snapshot is eventually created is added to
the end of the test.
2018-12-11 17:12:17 +01:00