Commit Graph

51115 Commits

Author SHA1 Message Date
Julie Tibshirani f75e3b4551 Remove an unused method on LeafDocLookup. 2020-04-15 16:58:49 -07:00
Jason Tedor cad1a3b0ad
Fix imports in CCRFeatureSet
This commit fixes some imports that were mixed up during a
backport. Because, backports.
2020-04-15 19:37:25 -04:00
Jason Tedor a18faacf1b
Make feature usage version aware (#55246)
Today we indiscriminately serialize these independent of the version on
the stream, even though the other side might not understand a new
feature set usage that we have added. For example, if we add feature set
usage in 7.7 for EQL, in a mixed cluster context if a request is sent to
an old coordinating node, but the master is a new version, then it would
attempt to serialize the usage information for the new feature back to
the old coordinating node, who will blow up on the unrecognized named
writeable. This commit addresses this by making feature usage version
aware, and only serializing those that the other side would understand.
2020-04-15 19:24:47 -04:00
Nik Everett a2ff2331bf
Add explict tests for auto_date_histogram (backport of #54819) (#55262)
`auto_date_histogram`'s reduction behavior is fairly complex and we have
some fairly complex testing logic for it but it is super difficult to
look at that testing logic and say "ah, that is what it does in this
case". This adds some tests explicit (non-randomized) tests of the
reduction logic that *should* be easier to read.
2020-04-15 19:09:46 -04:00
William Brafford 2ba3be9db6
Remove deprecated third-party methods from tests (#55255) (#55269)
I've noticed that a lot of our tests are using deprecated static methods
from the Hamcrest matchers. While this is not a big deal in any
objective sense, it seems like a small good thing to reduce compilation
warnings and be ready for a new release of the matcher library if we
need to upgrade. I've also switched a few other methods in tests that
have drop-in replacements.
2020-04-15 17:54:47 -04:00
Stéphane Campinas 6ef1c64760 SearchService#canMatch takes into consideration the alias filter (#55120)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-15 22:33:40 +02:00
Ryan Ernst 29b70733ae
Use task avoidance with forbidden apis (#55034)
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
2020-04-15 13:27:53 -07:00
Bogdan Pintea b88dd47de3 Docs: add the change log for 7.7 (#55019)
* Add the change log for 7.7

Add the change log for 7.7

* Update rel. notes to latest state (BC5)

Update the release notes to current state (i.e. BC5).

* Update docs/reference/release-notes/7.7.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-15 15:25:08 -04:00
Henning Andersen b3eb57a094 CCR: Test follow on top of closed index (#54956)
Added testing of following on top of a closed index.
This could for instance be the old leader index in
cases where leader and follower clusters have been
swapped.
2020-04-15 20:13:32 +02:00
Ryan Ernst b5974f3d41
Use standard base repo in SLES 12 (#55135)
The SLES 12 image's repos appear to no longer have any standard repos.
This commit adds the the standard SLES 12 SP3 repo, and forces expect to
be resolved as it has an issue with the version of tcl/tck (those
packages are thus downgraded to get expect installed).

closes #55048
2020-04-15 10:49:04 -07:00
Lisa Cawley f0b9578684 [DOCS] Removes transform performance note (#55177) 2020-04-15 10:42:52 -07:00
Mark Vieira 5d4bc8aea6
Mute ModelLoadingServiceTests.testMaxCachedLimitReached 2020-04-15 10:25:51 -07:00
Ignacio Vera eaf75fc7b6
disable BWC for lucene upgrade (#55242) (#55245) 2020-04-15 18:17:53 +02:00
Ignacio Vera a677b63daa
Upgrade to lucene 8.5.1 release (#55229) (#55235)
Upgrade to lucene 8.5.1 release that contains a bug fix for a bug that might introduce index corruption when deleting data from an index that was previously shrunk.
2020-04-15 17:35:42 +02:00
James Rodewig 4f2ab96f38 [DOCS] EQL: Document `indexOf` function (#55071) 2020-04-15 11:29:50 -04:00
James Rodewig 8d6f0f6a76 [DOCS] Document `max_concurrent_searches` default (#55116) 2020-04-15 10:04:23 -04:00
Benjamin Trent 8ff2cbf1a3
[7.x] [ML] adding prediction_field_type to inference config (#55128) (#55230)
* [ML] adding prediction_field_type to inference config (#55128)

Data frame analytics dynamically determines the classification field type. This field type then dictates the encoded JSON that is written to Elasticsearch. 

Inference needs to know about this field type so that it may provide the EXACT SAME predicted values as analytics. 

Here is added a new field `prediction_field_type` which indicates the desired type. Options are: `string` (DEFAULT), `number`, `boolean` (where close_to(1.0) == true, false otherwise). 

Analytics provides the default `prediction_field_type` when the model is created from the process.
2020-04-15 09:45:22 -04:00
Armin Braun 2f91e2aab7
Fix Race in Snapshot Abort (#54873) (#55233)
We can be a little more efficient when aborting a snapshot. Since we know the new repository
data after finalizing the aborted snapshot when can pass it down to the snapshot completion listeners.
This way, we don't have to fork off to the snapshot threadpool to get the repository data when the listener completes and can directly submit the delete task with high priority straight from the cluster state thread.
2020-04-15 15:42:15 +02:00
Jason Tedor acf2acba4a
Exclude JDK Contents/MacOS directory on Darwin (#55197)
This directory interferes with notarization and removing it before we
notarize allows us to have a properly notarized
distribution. Conceptually, this directory is only needed when building
a distribution to be installed by Installer (a so-called "pkg"). Since
we are not building such distributions, and this directory interferes
with notarization, we choose to exclude it here. We do this here, rather
than in our notarization process, to ensure that what we run through CI
for testing is also what we ship to the world.
2020-04-15 09:28:47 -04:00
Przemysław Witek b5fe565c89
Add log line that will help debug item failures during multi search request (#55220) (#55227) 2020-04-15 15:01:17 +02:00
Jake Landis 85139fad7e
[7.x] Advise a simpler curator migration (#54457) (#55188)
Advice for migrating from Curator should simply be to phase out curator managed indices, 
since curator will ignore ILM indices
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.7/ilm-and-curator.html#ilm-and-curator.

Co-authored-by: Jay Greenberg <PhaedrusTheGreek@users.noreply.github.com>
2020-04-15 07:55:31 -05:00
Armin Braun d8b43c6283
Make Snapshot Deletes Less Racy (#54765) (#55226)
Snapshot deletes should first check the cluster state for an in-progress snapshot
and try to abort it before checking the repository contents. This allows for atomically
checking and aborting a snapshot in the same cluster state update, removing all possible
races where a snapshot that is in-progress could not be found if it finishes between
checking the repository contents and the cluster state.
Also removes confusing races, where checking the cluster state off of the cluster state thread
finds an in-progress snapshot that is then not found in the cluster state update to abort it.
Finally, the logic to use the repository generation of the in-progress snapshot + 1 was error
prone because it would always fail the delete when the repository had a pending generation different from its safe generation when a snapshot started (leading to the snapshot finalizing at a
higher generation).

These issues (particularly that last point) can easily be reproduced by running `SLMSnapshotBlockingIntegTests` in a loop with current `master` (see #54766).

The snapshot resiliency test for concurrent snapshot creation and deletion was made to more
aggressively start the delete operation so that the above races would become visible.
Previously, the fact that deletes would never coincide with initializing snapshots resulted
in a number of the above races not reproducing.

This PR is the most consistent I could get snapshot deletes without changes to the state machine. The fact that aborted deletes will not put the delete operation in the cluster state before waiting for the snapshot to abort still allows for some possible (though practically very unlikely) races. These will be fixed by a state-machine change in upcoming work in #54705 (which will have a much simpler and clearer diff after this change).

Closes #54766
2020-04-15 14:47:16 +02:00
Rory Hunter 70616cd76a
Define aarch64 packaging test tasks (#55228)
Backport of #55073.

We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
2020-04-15 13:39:18 +01:00
Dan Hermann 30638a0b41
[7.x] Wipe data streams in each REST test (#55009) 2020-04-15 07:27:39 -05:00
Hendrik Muhs 9ec9866acb [Transform] simplify TransformConfigUpdate (#55224)
removes the unnecessary ToXContent method in TransformConfigUpdate
2020-04-15 13:22:50 +02:00
Armin Braun e164c9aaee
Remove Redundant Cluster State during Snapshot INIT + Master Failover (#54420) (#55208)
* Remove Redundant Cluster State during Snapshot INIT + Master Failover (#54420)

Similar to #54395 we know that a snapshot in INIT state has not
written anything to the repository yet. If we see one from a master
failover, there is no point in moving it to ABORTED before removing it
from the cluster state in a subsequent CS update.
Instead, we can simply remove its job from the CS the first time
we see it on master failover and be done with it.
2020-04-15 12:27:52 +02:00
Yannick Welsch d1123281b1 Use unlimited cache size by default (#55218)
Sets the default cache size for searchable snapshots to unlimited, which, for testing purposes,
is a better default than the 1GB that we currently have.
2020-04-15 12:20:51 +02:00
David Kyle bdf0eab78d
[7.x] Fix non-deterministic behaviour in ModelLoadingServiceTests (#55008) (#55213) 2020-04-15 11:09:12 +01:00
Ioannis Kakavas 0f51934bcf
[7.x] Add support for more named curves (#55179) (#55211)
We implicitly only supported the prime256v1 ( aka secp256r1 )
curve for the EC keys we read as PEM files to be used in any
SSL Context. We would not fail when trying to read a key
pair using a different curve but we would silently assume
that it was using `secp256r1` which would lead to strange
TLS handshake issues if the curve was actually another one.

This commit fixes that behavior in that it
supports parsing EC keys that use any of the named curves
defined in rfc5915 and rfc5480 making no assumptions about
whether the security provider in use supports them (JDK8 and
higher support all the curves defined in rfc5480).
2020-04-15 12:33:40 +03:00
Tomas Della Vedova 9872deace7
Yaml test: Fixed bad indentation (#55170) (#55206) 2020-04-15 11:13:52 +02:00
David Roberts 8c33cad2b2 [TEST] Lower ML model memory limit in HLRC datafeed tests (#55210)
The MachineLearningIT.testStopDatafeed test was creating 3
jobs each with the default model memory limit of 1GB.  This
meant that the test would not run on a machine with less than
10GB of RAM (due to the default ML memory percentage of 30%).

This change reduces the model memory limit for these jobs to
0.5GB, which means the test will run on a machine with only
5GB of RAM.

Relates to https://discuss.elastic.co/t/failed-ml-tests-when-running-the-gradle-check-task-against-unchanged-repo-code/227829
2020-04-15 09:44:07 +01:00
Armin Braun 48048646e7
Move Snapshot Status Related Method to Appropriate Places (#54558) (#55209)
* Move Snapshot Status Related Method to Appropriate Places

Lots of things living in `SnapshotsService` for no reason other than
that `SnapshotsService` provides the `RepositoriesService`.
Cleaning this up to directly use `RepositoriesService` in the relevant
transport actions and by that shortening the already very complex `SnapshotsService`.
2020-04-15 10:25:52 +02:00
Dimitris Athanasiou 4000138105
[7.x][ML] Add debug logging for outlier detection stop and restart integ test (#55169) (#55202)
To understand the failures in #55068

Backport of #55169
2020-04-15 10:40:38 +03:00
Lisa Cawley 2910d01179
[DOCS] Removes unshared sections from ml-shared.asciidoc (#55192) 2020-04-14 18:47:09 -07:00
Yang Wang f49354b7d7
Add migration notes for deprecating local parameter of get field mapping API (#55194)
This is a follow-up for #55099 to add migration notes about the deprecation of local parameter for get field mappings API.
2020-04-15 11:38:05 +10:00
Ryan Ernst 5243295788
Update gradle in vagrant to use jdk14 (#55130)
This commit updates the Java version used within vagrant tests by gradle
inside the VM to use JDK14.

closes #55044
2020-04-14 17:00:12 -07:00
Lee Hinman 36f6e542a2 Ignore ILM indices in the TerminalPolicyStep (#55184)
Prior to the change in #51631 indices were moved to the `TerminalPolicyStep` when their ILM actions
had completed. Once we switched ILM to stop in the last policy configured, these steps because
inaccessible from the policy's perspective. This meant that indices upgraded from ES prior to 7.7.0
could see the following error spammed in their logs every 10 minutes (by default) for every index in
this state:

```
[2020-04-14T15:52:23,764][ERROR][o.e.x.i.IndexLifecycleRunner] [midgar] current step [{"phase":"completed","action":"completed","name":"completed"}] for index [foo] with policy [full] is not recognized
```

This changes the runner to ignore these steps, which is what is desired anyway since the index is
already in the terminal phase.
2020-04-14 16:45:03 -06:00
Rory Hunter 18dc2f7330 Rename some Docker projects for consistency (#55150)
Apply the :distribution:archives naming convention to some of the Docker
sub-projects, so that we have a more consistent naming scheme.

Also, we've seen some examples of Docker packaging tests failing sporadically
when they try to clean up the temp directory, citing a not-empty
directory. Ensure that any running container is removed before cleaning
up the temp dir, in an effort to avoid this problem.
2020-04-14 22:09:05 +01:00
Igor Motov 1754e50cbd
[7.x] Add analytics plugin usage stats to _xpack/usage (#54911) (#55162)
Adds analytics plugin usage stats to _xpack/usage.

Closes #54847
2020-04-14 17:03:14 -04:00
Mark Vieira ce85063653
[7.x] Re-add origin url information to publish POM files (#55173) 2020-04-14 13:24:15 -07:00
Armin Braun f7467a7fe8
Fix Cluster Stabilization in SnapshotResiliencyTests (#55159) (#55168)
Just like in `AbstractCoordinatorTestCase` we can't just assume the cluster
is stable once all the cluster states align since stray follower/leader check
tasks could still hit us after a disconnect, causing future test operations to fail.
=> fixed by running all tasks in the possible time span of running into these
checks before validating that cluster states align on all nodes to prevent this
like we do in the coordinator tests.

Closes #55103
2020-04-14 19:22:26 +02:00
Albert Zaharovits 7f35b927d1
Preserve parent task id for ml transform (#55124)
This change ensures that internal client requests spawned by the
transform persistent task executor and that use the end user security
credentials, have the parent task id assigned. The objective here is
to permit auditing (as well as tracking for debugging purposes) of all
the end-user requests executed on its behalf by persistent tasks.
Because transform tasks already implements graceful shutdown of the
child tasks, this change does not interfere with that by opting out of
the persistent task cancellation of child tasks.

Relates #55046 #54943 #52314
Closes #54957
2020-04-14 18:43:47 +03:00
James Rodewig 12130843ca
[DOCS] Add maintenance releases to upgrade table (#55012)
Updates the supported upgrade path table in [Upgrade Elasticsearch][0]
to include a new row for maintenance releases. For example, this row
covers upgrading from 7.6.0 to 7.6.2.

The new table row only displays for releases greater than n.x.0. For
example, the new row will display for the 7.7.1 release but not the
7.7.0 release.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-upgrade.html
2020-04-14 11:28:55 -04:00
Andrei Dan d918ef0da9
[Tests] Enable searchable_snapshots for non-snapshot builds (#55151) (#55157)
Fixes https://github.com/elastic/elasticsearch/issues/55050

(cherry picked from commit 13391ceff1cbf6db69706c5f46127b6ff8850a1f)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-14 16:13:39 +01:00
James Rodewig 3fbd8b371f [DOCS] Use consistent line breaks in EQL function docs 2020-04-14 10:17:45 -04:00
Albert Zaharovits 5998486ce8
Refactor AuditTrail for TransportRequests instead of TransportMessage (#55141)
This commit refactors the `AuditTrail` to use the `TransportRequest` as a parameter
for all its audit methods, instead of the current `TransportMessage` super class.

The goal is to gain access to the `TransportRequest#parentTaskId` member,
so that it can be audited. The `parentTaskId` is used internally when spawning tasks
that handle transport requests; in this way tasks across nodes are related by the
same parent task.

Relates #52314
2020-04-14 16:53:59 +03:00
Yannick Welsch a610513ec7 Provide repository-level stats for searchable snapshots (#55051)
Provides basic repository-level stats that will allow us to get some insight into how many
requests are actually being made by the underlying SDK. Currently only tracks GET and LIST
calls for S3 repositories. Most of the code is unfortunately boiler plate to add a new endpoint
that will help us better understand some of the low-level dynamics of searchable snapshots.
2020-04-14 14:34:08 +02:00
Przemysław Witek d5bb574e1e
[7.x] Unassign DFA tasks in SetUpgradeModeAction (#54523) (#55143) 2020-04-14 14:09:02 +02:00
Igor Motov 8a669dc9b7
EQL: Add cascading search cancellation (#54843)
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to #49638
2020-04-14 08:06:02 -04:00
Alan Woodward 16ebbff3b6 Mute CancellableTasksIT (#55152)
Test failures are tracked in #55106
2020-04-14 12:55:20 +01:00