Commit Graph

4913 Commits

Author SHA1 Message Date
Tom Veasey 690099553c
[7.x][ML] Adds the class_assignment_objective parameter to classification (#53552)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-13 17:35:51 +00:00
Tim Vernum a8677499d7
[Backport] Add support for secondary authentication (#53530)
This change makes it possible to send secondary authentication
credentials to select endpoints that need to perform a single action
in the context of two users.

Typically this need arises when a server process needs to call an
endpoint that users should not (or might not) have direct access to,
but some part of that action must be performed using the logged-in
user's identity.

Backport of: #52093
2020-03-13 16:30:20 +11:00
Tim Vernum bac1740d44
Support authentication without anonymous user (#53528)
This change adds a new parameter to the authenticate methods in the
AuthenticationService to optionally exclude support for the anonymous
user (if an anonymous user exists).

Backport of: #52094
2020-03-13 14:27:29 +11:00
Jason Tedor f696360517
Fix SHAs for :x-pack:snapshot-tool
This commit fixes the SHA for jackson-databind in :x-pack:snapshot-tool.
2020-03-12 20:24:29 -04:00
Nik Everett 9dcd64c110
Preserve metric types in top_metrics (backport of #53288) (#53440)
This changes the `top_metrics` aggregation to return metrics in their
original type. Since it only supports numerics, that means that dates,
longs, and doubles will come back as stored, with their appropriate
formatter applied.
2020-03-12 17:17:09 -04:00
Jason Tedor 5b08ea84c9
Add deprecation check for listener thread pool (#53438)
This commit adds a deprecation check for the listener thread pool
settings as these will be removed in 8.0.0.
2020-03-12 14:32:41 -04:00
Jay Modi af36665b08
Deprecate the logstash enabled setting (#53487)
The setting, `xpack.logstash.enabled`, exists to enable or disable the
logstash extensions found within x-pack. In practice, this setting had
no effect on the functionality of the extension. Given this, the
setting is now deprecated in preparation for removal.

Backport of #53367
2020-03-12 10:18:39 -06:00
Dan Hermann 34adfd9611
Validate SSL settings at parse time (#49196) (#53473) 2020-03-12 10:14:51 -05:00
Aleksandr Maus 31d45b3c95
EQL: Improve query folder test suite (#53187) (#53476)
Related to https://github.com/elastic/elasticsearch/issues/52775
2020-03-12 10:58:07 -04:00
Yannick Welsch 48124807d5 Fix SourceOnlySnapshotIT (#53462)
The tests in this class had been failing for a while, but went unnoticed as not tested by CI (see #53442).

The reason the tests fail is that the can-match phase is smarter now, and filters out access to a non-existing field.

Closes #53442
2020-03-12 14:15:03 +01:00
Jason Tedor d8e70d4688
Enable deprecation checks for removed settings (#53317)
Today we do not have any infrastructure for adding a deprecation check
for settings that are removed. This commit enables this by adding such
infrastructure. Note that this infrastructure is unused in this commit,
which is deliberate. However, the primary target for this commit is 7.x
where this infrastructue will be used, in a follow-up.
2020-03-11 16:49:16 -04:00
Benjamin Trent 89668c5ea0
[ML][Inference] adds new default_field_map field to trained models (#53294) (#53419)
Adds a new `default_field_map` field to trained model config objects.

This allows the model creator to supply field map if it knows that there should be some map for inference to work directly against the training data.

The use case internally is having analytics jobs supply a field mapping for multi-field fields. This allows us to use the model "out of the box" on data where we trained on `foo.keyword` but the `_source` only references `foo`.
2020-03-11 13:49:39 -04:00
Lisa Cawley c408a34a21 [DOCS] Fixes link to custom realm examples (#53205) 2020-03-11 09:15:48 -07:00
Jay Modi 9a21a8abf2
Opt-in logstash plugin to formatting (#53413)
This change opts-in the logstash plugin for enforced formatting.

Backport of #53370
2020-03-11 09:58:37 -06:00
Nhat Nguyen 6665ebe7ab Harden search context id (#53143)
Using a Long alone is not strong enough for the id of search contexts
because we reset the id generator whenever a data node is restarted.
This can lead to two issues:

1. Fetch phase can fetch documents from another index
2. A scroll search can return documents from another index

This commit avoids these issues by adding a UUID to SearchContexId.
2020-03-11 11:48:11 -04:00
Przemysław Witek 8c4c19d310
Perform evaluation in multiple steps when necessary (#53295) (#53409) 2020-03-11 15:36:38 +01:00
Przemysław Witek 063957b7d8
Simplify "refresh" calls. (#53385) (#53393) 2020-03-11 12:26:11 +01:00
Dimitris Athanasiou cc7751eb16
[7.x][ML] Add ILM policy to ml stats indices (#53349) (#53392)
Adds a size based ILM policy to automatically
rollover ml stats indices.

Backport of #53349
2020-03-11 13:01:34 +02:00
Dimitris Athanasiou 0fd0516d0d
[7.x][ML] Rename data frame analytics maximum_number_trees to max_trees (#53300) (#53390)
Deprecates `maximum_number_trees` parameter of classification and
regression and replaces it with `max_trees`.

Backport of #53300
2020-03-11 12:45:27 +02:00
David Roberts 532a720e1b
[ML] Skeleton estimate_model_memory endpoint for anomaly detection (#53386)
This is a partial implementation of an endpoint for anomaly
detector model memory estimation.

It is not complete, lacking docs, HLRC and sensible numbers
for many anomaly detector configurations.  These will be
added in a followup PR in time for 7.7 feature freeze.

A skeleton endpoint is useful now because it allows work on
the UI side of the change to commence.  The skeleton endpoint
handles the same cases that the old UI code used to handle,
and produces very similar estimates for these cases.

Backport of #53333
2020-03-11 10:20:00 +00:00
Jake Landis 2ab502afc4
[7.x] Remove dead 'beats' code (#53312) (#53376) 2020-03-10 20:57:29 -05:00
Nhat Nguyen 24f114766f Fix doc_stats and segment_stats of ReadOnlyEngine (#53345)
We can't always have the same segment stats and doc stats between
InternalEngine and ReadOnlyEngine if there are some fully deleted
segments. ReadOnlyEngine always filters out them. InternalEngine,
however, will keep them if peer recovery retention leases exist or the
number of the retaining operations is non-zero.

This change reverts the fix in #51331 and uses the wrapped reader to
calculate the segment stats and doc stats. For the test, we need to
disable the extra retaining soft-deletes operations.

Closes #51303
2020-03-10 21:51:33 -04:00
Nhat Nguyen cad02d4a31 Increase timeout testFollowIndexMaxOperationSizeInBytes (#53014)
Replicating 1000 documents one by one (as we cap the request size at 
1 byte) can take more than 10 seconds on a slow CI.

Closes #52812
2020-03-10 21:51:33 -04:00
William Brafford 3494c73c8d
Mute failing tests (#53362) (#53363) 2020-03-10 16:01:31 -04:00
Przemko Robakowski 847ac9c7d7
Fix null config in SnapshotLifecyclePolicy.toRequest (#53328) (#53355)
This avoids NPE when executing SLM policy when no config was provided.

Related to #44465

Closes #53171

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-10 20:44:30 +01:00
Przemysław Witek d54d7f2be0
[7.x] Implement ILM policy for .ml-state* indices (#52356) (#53327) 2020-03-10 14:24:18 +01:00
Benjamin Trent 856d9bfbc1
[ML] fixing data frame analysis test when two jobs are started in succession quickly (#53192) (#53332)
A previous change (#53029) is causing analysis jobs to wait for certain indices to be made available. While this it is good for jobs to wait, they could fail early on _start. 

This change will cause the persistent task to continually retry node assignment when the failure is due to shards not being available.

If the shards are not available by the time `timeout` is reached by the predicate, it is treated as a _start failure and the task is canceled. 

For tasks seeking a new assignment after a node failure, that behavior is unchanged.


closes #53188
2020-03-10 08:30:47 -04:00
Hendrik Muhs 5912895838 [Transform] wait for transform templates in Rest integration t… (#53330)
add transform templates to the list of templates to be installed before
executing tests
2020-03-10 13:22:12 +01:00
Hendrik Muhs 696aa4ddaf
[7.x][Transform] add support for script in group_by (#53167) (#53324)
add the possibility to base the group_by on the output of a script.

closes #43152
backport #53167
2020-03-10 11:12:58 +01:00
Alan Woodward 5c861cfe6e Upgrade to final lucene 8.5.0 snapshot (#53293)
Lucene 8.5.0 release candidates are imminent. This commit upgrades master to use
the latest snapshot to check that there are no last-minute bugs or regressions.
2020-03-10 09:32:59 +00:00
Cauê Marcondes b68d7b1c33
giving kibana user privileges to create custom link index (#53221) (#53278) 2020-03-10 09:50:38 +01:00
Henning Andersen a4d481f2bb ILM Freeze step retry when not acknowledged (#53287)
A freeze operation can partially fail in multiple places, including the
close verification step. This left the index in an unfrozen but
partially closed state. Now throw an exception to retry the freeze step
instead.
2020-03-10 08:03:39 +01:00
Gordon Brown 1cb0a4399d
Fix Get Alias API handling of hidden indices with visible aliases (#53147)
This commit changes the Get Aliases API to include hidden indices by
default - this is slightly different from other APIs, but is necessary
to make this API work intuitively.
2020-03-09 16:16:29 -06:00
Przemko Robakowski f075d70cf8
[7.x] Avoid race condition in ILMHistorySotre (#53039) (#53094)
* Avoid race condition in ILMHistorySotre (#53039)

* Avoid race condition in ILMHistorySotre

This change modifies ILMHistoryStore to always apply correct settings and mappings,
even if template is deleted and not yet recreated. This ensures that ILM history index
is correctly managed by ILM and also fixes flaky history tests that were prone to
triggenring this race.

This commit also refactors and simplifies ILM history tests.

Closes #50353 and #52853

* Review comment

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* fixed tests

* backport #53306

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-09 22:24:15 +01:00
Bogdan Pintea 62c8ac9993
SQL: transfer version compatibility decision to the server (#53082) (#53302)
This commit adds a new request object field, "version", containing the version of the requesting client. This parameter is now accepted - and for certain clients required - by the server and the request is validated against it. Currently server's and client's versions still need to be equal in order for the request to be accepted. Relaxing this check is going to be part of future work. 

On the clients' side, the only check remaining is to ensure that the peer server is supporting version backwards compatibility (i.e. is on, or newer than a certain release).

(cherry picked from commit a8f413a20fb023bec83af0de1211a2936a7f558c)
2020-03-09 21:16:57 +01:00
Aleksandr Maus d064846416
EQL: Test infrastructure improvements (#53253) (#53297)
Update CommonEqlRestTestCase code to simplify making changes as requested.
Update EqlActionIT to simplify the test code as requested.
Replace Jackson parser with XContent in EqlActionIT.
Whitelist more EQL tests specs that are now supported.
2020-03-09 14:11:54 -04:00
Ross Wolf f5f922c6f6
EQL: Add IsNull/IsNotNull checks (#52791)
* EQL: Add IsNull/IsNotNull checks
* EQL: Simplify IsNull/IsNotNull optimization
* EQL: Split string tests over multiple lines
2020-03-09 10:41:04 -06:00
Jason Tedor 8ad0080a59
Fork CCR checkpoint listeners on CCR thread pool (#53265)
This commit moves the global checkpoint listeners used in CCR to the CCR
thread pool. This removes the last use of the listener thread pool in
the codebase.
2020-03-09 08:56:30 -04:00
Martijn van Groningen 7775ddbc9c
Verify watch_count before a test starts and not after a test.
This check was added as part of: 0f2d26bdca

Checking this before the test starts makes more sense, because
the watches index has then also be removed.

Relates to #53177
2020-03-09 07:45:44 +01:00
Jason Tedor 5e96d3e59a
Use given executor for global checkpoint listener (#53260)
Today when notifying a global checkpoint listener, we use the listener
thread pool. This commit turns this inside out so that the global
checkpoint listener must provide an executor on which to notify the
listener.
2020-03-08 13:51:05 -04:00
Lisa Cawley 341417613e
[7.x][DOCS] Adds common definitions for security settings (#51017) (#53242)
Co-Authored-By: Tim Vernum <tim@adjective.org>
2020-03-06 16:28:54 -08:00
Gordon Brown ff9b8bda63
Implement hidden aliases (#52547)
This commit introduces hidden aliases. These are similar to hidden
indices, in that they are not visible by default, unless explicitly
specified by name or by indicating that hidden indices/aliases are
desired.

The new alias property, `is_hidden` is implemented similarly to
`is_write_index`, except that it must be consistent across all indices
with a given alias - that is, all indices with a given alias must
specify the alias as either hidden, or all specify it as non-hidden,
either explicitly or by omitting the `is_hidden` property.
2020-03-06 16:02:38 -07:00
Ross Wolf d6813cb348
EQL: Convert wildcards to LIKE in analyzer (#51901)
* EQL: Convert wildcard comparisons to Like
* EQL: Simplify wildcard handling, update tests
* EQL: Lint fixes for Optimizer.java
2020-03-06 13:13:07 -07:00
Mayya Sharipova f96ad5c32d Mute testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows 2020-03-06 12:48:05 -05:00
Jay Modi a81460dbf5
Make watch history indices hidden (#52974)
This commit updates the template used for watch history indices with
the hidden index setting so that new indices will be created as hidden.

Relates #50251
Backport of #52962
2020-03-06 09:47:03 -07:00
Mark Vieira 09a3f45880
Mute ClassificationIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-06 07:38:04 -08:00
James Baiera 01f00df5cd
Mute RegressionIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet 2020-03-06 07:37:57 -08:00
Benjamin Trent 85d7112e78
[ML] Fixing datafeed bwc tests (#52959)
Datafeed bwc tests have been muted for some time in the 7.x. This is because of date_histogram interval deprecation warnings.

This commit fixes the tests as must as possible while still handling deprecation warnings.
2020-03-06 10:27:21 -05:00
Dimitris Athanasiou 9abf537527
[7.x][ML] Improve DF analytics audits and logging (#53179) (#53218)
Adds audits for when the job starts reindexing, loading data,
analyzing, writing results. Also adds some info logging.

Backport of #53179
2020-03-06 13:47:27 +02:00
Nhat Nguyen 5476a49833 Revert "upgrade to lucene-snapshot-fa75139efea (#53150) (#53151)"
This reverts commit 058113aa42.
2020-03-05 17:33:00 -05:00