Commit Graph

1340 Commits

Author SHA1 Message Date
Jim Ferenczi d6445fae4b Add a cluster setting to disallow loading fielddata on _id field (#49166)
This change adds a dynamic cluster setting named `indices.id_field_data.enabled`.
When set to `false` any attempt to load the fielddata for the `_id` field will fail
with an exception. The default value in this change is set to `false` in order to prevent
fielddata usage on this field for future versions but it will be set to `true` when backporting
to 7x. When the setting is set to true (manually or by default in 7x) the loading will also issue
a deprecation warning since we want to disallow fielddata entirely when https://github.com/elastic/elasticsearch/issues/26472
is implemented.

Closes #43599
2019-11-28 09:35:28 +01:00
Benjamin Trent b5d7c939f8
[7.x] [ML][Inference][HLRC] add GET _stats (#49562) (#49600)
* [ML][Inference][HLRC] add GET _stats (#49562)

* fixing for backport
2019-11-26 11:28:26 -05:00
Benjamin Trent 26a8ca00db
[7.x] [ML][Inference][HLRC] Delete trained model API (#49567) (#49585)
* [ML][Inference][HLRC] Delete trained model API (#49567)

* fixing for backport
2019-11-26 08:27:08 -05:00
Tim Brooks 416178c7c8
Enable simple remote connection strategy (#49561)
This commit back ports three commits related to enabling the simple
connection strategy.

Allow simple connection strategy to be configured (#49066)

Currently the simple connection strategy only exists in the code. It
cannot be configured. This commit moves in the direction of allowing it
to be configured. It introduces settings for the addresses and socket
count. Additionally it introduces new settings for the sniff strategy
so that the more generic number of connections and seed node settings
can be deprecated.

The simple settings are not yet registered as the registration is
dependent on follow-up work to validate the settings.

Ensure at least 1 seed configured in remote test (#49389)

This fixes #49384. Currently when we select a random subset of seed
nodes from a list, it is possible for 0 seeds to be selected. This test
depends on at least 1 seed being selected.

Add the simple strategy to cluster settings (#49414)

This is related to #49067. This commit adds the simple connection
strategy settings and strategy mode setting to the cluster settings
registry. With these changes, the simple connection mode can be used.
Additionally, it adds validation to ensure that settings cannot be
misconfigured.
2019-11-25 16:53:07 -07:00
jesinity c9eba17517 Fix HLRC parsing of CancelTasks response (#47017)
Adds support for proper cancel tasks parsing.

Closes #45414
2019-11-22 16:56:27 -06:00
Dimitris Athanasiou 8eaee7cbdc
[7.x][ML] Explain data frame analytics API (#49455) (#49504)
This commit replaces the _estimate_memory_usage API with
a new API, the _explain API.

The API consolidates information that is useful before
creating a data frame analytics job.

It includes:

- memory estimation
- field selection explanation

Memory estimation is moved here from what was previously
calculated in the _estimate_memory_usage API.

Field selection is a new feature that explains to the user
whether each available field was selected to be included or
not in the analysis. In the case it was not included, it also
explains the reason why.

Backport of #49455
2019-11-22 22:06:10 +02:00
Benjamin Trent ed787d06e8
[7.x] [ML][Inference][HLRC] GET trained models (#49464) (#49488)
* [ML][Inference][HLRC] GET trained models (#49464)

* fixing for backport
2019-11-22 09:24:06 -05:00
Tim Vernum 2e5f2dd1e1
Deprecate misconfigured SSL server config (#49280)
This commit adds a deprecation warning when starting
a node where either of the server contexts
(xpack.security.transport.ssl and xpack.security.http.ssl)
meet either of these conditions:

1. The server lacks a certificate/key pair (i.e. neither
   ssl.keystore.path not ssl.certificate are configured)
2. The server has some ssl configuration, but ssl.enabled is not
   specified. This new validation does not care whether ssl.enabled is
   true or false (though other validation might), it simply makes it
   an error to configure server SSL without being explicit about
   whether to enable that configuration.

Backport of: #45892
2019-11-22 12:14:55 +11:00
Benjamin Trent a7477ad7c3
[7.x] [ML][Inference] compressing model definition and lazy parsing (#49269) (#49446)
* [ML][Inference] compressing model definition and lazy parsing (#49269)

* [ML][Inference] compressing model definition and lazy parsing

* addressing PR comments

* adding commons io

* implementing simplified bounded stream

* adjusting for type inclusion
2019-11-21 15:32:32 -05:00
Benjamin Trent d41b2e3f38
[ML][Inference] allowing per-model licensing (#49398) (#49435)
* [ML][Inference] allowing per-model licensing

* changing to internal action + removing pre-mature opt
2019-11-21 09:46:34 -05:00
Przemysław Witek c7ac2011eb
[7.x] Implement accuracy metric for multiclass classification (#47772) (#49430) 2019-11-21 15:01:18 +01:00
Dimitris Athanasiou 543f5f4faf
[7.x][ML][HLRC] Add FAILED state for data frame analytics (#49326) (#49327)
Backport of #49326
2019-11-20 09:58:13 +02:00
Benjamin Trent eefe7688ce
[7.x][ML] ML Model Inference Ingest Processor (#49052) (#49257)
* [ML] ML Model Inference Ingest Processor (#49052)

* [ML][Inference] adds lazy model loader and inference (#47410)

This adds a couple of things:

- A model loader service that is accessible via transport calls. This service will load in models and cache them. They will stay loaded until a processor no longer references them
- A Model class and its first sub-class LocalModel. Used to cache model information and run inference.
- Transport action and handler for requests to infer against a local model
Related Feature PRs:

* [ML][Inference] Adjust inference configuration option API (#47812)

* [ML][Inference] adds logistic_regression output aggregator (#48075)

* [ML][Inference] Adding read/del trained models (#47882)

* [ML][Inference] Adding inference ingest processor (#47859)

* [ML][Inference] fixing classification inference for ensemble (#48463)

* [ML][Inference] Adding model memory estimations (#48323)

* [ML][Inference] adding more options to inference processor (#48545)

* [ML][Inference] handle string values better in feature extraction (#48584)

* [ML][Inference] Adding _stats endpoint for inference (#48492)

* [ML][Inference] add inference processors and trained models to usage (#47869)

* [ML][Inference] add new flag for optionally including model definition (#48718)

* [ML][Inference] adding license checks (#49056)

* [ML][Inference] Adding memory and compute estimates to inference (#48955)

* fixing version of indexed docs for model inference
2019-11-18 13:19:17 -05:00
Przemysław Witek 5f9965e4b8
Lower minimum model memory limit value from 1MB to 1kB. (#49227) (#49242) 2019-11-18 14:58:20 +01:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Alan Woodward f0e386d60d Fix ResizeResponseTests randomization 2019-11-13 11:50:04 +00:00
Alan Woodward 999d66fc87 Add client-side ResizeRequest and ResizeResponse classes (#48937)
Closes #48468
2019-11-13 10:34:11 +00:00
Michael Basnight bc23bc5146 Add delete alias to the HLRC (#48819)
The delete alias call is a rest only API call, but should still be added
to the rest client. This commit adds it as well as relevant tests.

Ref #47678
2019-11-12 11:02:53 -06:00
Benjamin Trent 46ab1db54f
[7.x] [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050) (#48958)
* [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)

[ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)

Related PR: https://github.com/elastic/ml-cpp/pull/809

* adjusting bwc version
2019-11-11 15:43:03 -05:00
Dan Hermann c85cf7a6de
Validate proxy base path at parse time (#47912) (#48825) 2019-11-04 09:51:13 -06:00
Michael Basnight ede1681c5a Remove old HLRC base test case (#48714)
This commit finishes cleaning up the AbstractHlrcXContentTestCase work
and removes this class. All classes that were using this are now using
the updated base class.

Ref #39745
2019-10-30 14:42:52 -05:00
Michael Basnight eee4cfaa8b Refactor HLRC transform stats test (#48708)
This test uses a deprecated base class, and this commit moves it over
to the new class.

Ref #39745
2019-10-30 14:42:37 -05:00
Michael Basnight d63e5772c0 Cleanup HLRC graph tests to use new test style (#48644)
The old graph tests were duplicated a lot and used a deprecated parent
class. This commit cleans that up and removes one of the duplicated
tests.

Ref #39745
2019-10-30 14:42:22 -05:00
Benjamin Trent c9ead80c31
[7.x] [ML][Inference] separating definition and config object storage (#48651) (#48695)
* [ML][Inference] separating definition and config object storage (#48651)

This separates out the `definition` object from being stored within the configuration object in the index. 

This allows us to gather the config object without decompressing a potentially large definition.

Additionally, `input` is moved to the TrainedModelConfig object and out of the definition. This is so the trained input fields are accessible outside the potentially large model definition.
2019-10-30 13:27:29 -04:00
Gordon Brown 4bd514715d
Increase timeout in ILM doc test slightly (#48606)
This assertBusy can occasionally time out on systems under heavy load,
such as CI, so this commit increases the timeout.
2019-10-29 11:12:41 -07:00
Christoph Büscher 09d68e7548
Support `search_type` in Rank Evaluation API (#48542) (#48631)
Adding support for the `search_type` request parameter to the Ranking Evaluation
API since this parameter can impact the ranking and the metric score and should
be choosen in the same way when evaluating the search as later in the real
search.

Closes #48503
2019-10-29 14:54:33 +01:00
Rory Hunter da4654527b
Improve resiliency to auto-formatting in client (#48617)
Backport of #48447. Make a number of changes so that code in the
`client` directory is more resilient to automatic formatting. This
covers:

* Literal JSON handling:
   * Reformatting multiline JSON to embed whitespace in the strings
   * Remove string concatenation where JSON fits on a single line
   * Use `String.format` for large documents with variable content
* Remove some erroneous doc refs in `QueryDSLDocumentationTests`
* Move some comments around to they aren't auto-formatted to a strange
  place
2019-10-29 10:40:54 +00:00
Mark Vieira e5c6440a4f
Simplify usage of Gradle Shadow plugin (#48478) (#48597)
This commit simplifies and standardizes our usage of the Gradle Shadow
plugin to conform more to plugin conventions. The custom "bundle" plugin
has been removed as it's not necessary and performs the same function
as the Shadow plugin's default behavior with existing configurations.

Additionally, this removes unnecessary creation of a "nodeps" artifact,
which is unnecessary because by default project dependencies will in
fact use the non-shadowed JAR unless explicitly depending on the
"shadow" configuration.

Finally, we've cleaned up the logic used for unit testing, so we are
now correctly testing against the shadow JAR when the plugin is applied.
This better represents a real-world scenario for consumers and provides
better test coverage for incorrectly declared dependencies.

(cherry picked from commit 3698131109c7e78bdd3a3340707e1c7b4740d310)
2019-10-28 12:11:55 -07:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Gordon Brown 5021410165
Retry on RepositoryException in SLM tests (#48548)
Due to a bug, GETing a snapshot can cause a RespositoryException to be
thrown. This error is transient and should be retried, rather than
causing the test to fail. This commit converts those
RepositoryExceptions into AssertionErrors so that they will be retried
in code wrapped in assertBusy.
2019-10-28 09:24:38 -07:00
Michael Basnight 1ba57dbe08 [Docs] add missing snapshot restore reference (#45256) 2019-10-28 09:55:10 -05:00
David Turner e821a22580 Mute SecurityDocumentationIT#testGetUsers - see #48440 2019-10-28 14:03:01 +01:00
Michael Basnight 5228956ecc Add slices to delete and update by query in HLRC (#48420)
The slices param was missing from both delete by query and update by
query in the HLRC request converters. This commit fixes the omission.
2019-10-25 15:23:17 -05:00
Tanguy Leroux 2861088a59 Mute RestClientMultipleHostsIntegTests.testCancelAsyncRequests (#48535)
Relates https://github.com/elastic/elasticsearch/issues/45577
2019-10-25 17:42:24 +02:00
Tim Brooks c0b545f325
Make BytesReference an interface (#48486)
BytesReference is currently an abstract class which is extended by
various implementations. This makes it very difficult to use the
delegation pattern. The implication of this is that our releasable
BytesReference is a PagedBytesReference type and cannot be used as a
generic releasable bytes reference that delegates to any reference type.
This commit makes BytesReference an interface and introduces an
AbstractBytesReference for common functionality.
2019-10-24 15:39:30 -06:00
Michael Basnight d49958cef3 Remove deprecated test from the HLRC tests (#48424)
The AbstractHlrcWriteableXContentTestCase was replaced by a better test
case a while ago, and this is the last two instances using it. They have
been converted and the test is now deleted.

Ref #39745
2019-10-24 14:02:04 -05:00
Michael Basnight c19379ef31 Remove random when using HLRC sync and async calls (#48211)
This commit removes the randomization used by every execute call in the
high level rest tests. Previously every execute call, which can be many
calls per single test, would rely on a random boolean to determine if
they should use the sync or async methods provided to the execute
method. This commit runs the tests twice, using two different clusters,
both of them providing the value one time via a sysprop. This ensures
that the whole suite of tests is run using the sync and async code
paths.

Closes #39667
2019-10-24 09:06:17 -05:00
Przemysław Witek aa29567e11
[7.x] Fix assignment
Backport of https://github.com/elastic/elasticsearch/pull/48216
2019-10-22 11:34:09 +02:00
Martijn van Groningen c09b62d5bf
Backport: also validate source index at put enrich policy time (#48311)
Backport of: #48254

This changes tests to create a valid
source index prior to creating the enrich policy.
2019-10-22 07:38:16 +02:00
Przemysław Witek 2db2b945ec
[7.x] Change format of MulticlassConfusionMatrix result to be more self-explanatory (#48174) (#48294) 2019-10-21 22:07:19 +02:00
Lee Hinman cc0c876a8d
fix incorrect comparison (#48208) (#48303)
* remove comparison of identical values

the comparison `tookInMillis == tookInMillis` is always true.

* add comparison between tookInMillis
2019-10-21 09:14:44 -06:00
Przemysław Witek 1a42e37070
[7.x] Default "prediction_field_name" to (dependent_variable + "_prediction") (#48232) (#48279) 2019-10-21 13:18:08 +02:00
Albert Zaharovits 69fc715bc3
Fix security origin for TokenService#findActiveTokensFor... (#47418) (#48280)
All internal searches (triggered by APIs) across the .security index
must be performed while "under the security origin". Otherwise,
the search is performed in the context of the caller which most
likely does not have privileges to search .security (hopefully).
This commit fixes this in the case of two methods in the
TokenService and corrects an overly done such context switch
in the ApiKeyService.

In addition, this makes all tests from the client/rest-high-level
module execute as an all mighty administrator,
but not a literal superuser.

Closes #47151
2019-10-21 13:15:05 +03:00
Martijn van Groningen 5cb44a414c
Fixed links in java docs for EnrichClient (#48233) 2019-10-18 16:26:49 +02:00
Benjamin Trent 876f4aafac
[ML] Add logistic_regression output aggregator (#48238) (#48244) 2019-10-18 10:08:17 -04:00
Przemysław Witek 28f68fa221
Make num_top_classes parameter's default value equal to 2 (#48119) (#48201) 2019-10-17 18:43:15 +02:00
Gordon Brown eb7969e8cc
Fix ILM HLRC Javadoc->Documentation links (#48083)
Several links from the ILM HLRC Javadoc to the online documentation were
not updated when the ILM HLRC documentation was written. This commit
fixes those links.
2019-10-17 09:59:40 -06:00
Stuart Tettemer 356eef00c8
Scripting: get context names REST API (#48026) (#48168)
Adds `GET /_script_context`, returning a `contexts` object with each
available context as a key whose value is an empty object. eg.
```
{
  "contexts": {
    "aggregation_selector": {},
    "aggs": {},
    "aggs_combine": {},
...
  }
}
```

refs: #47411
2019-10-17 09:08:55 -06:00
Benjamin Trent 0dddbb5b42
[ML] Parse and index inference model (#48016) (#48152)
This adds parsing an inference model as a possible
result of the analytics process. When we do parse such a model
we persist a `TrainedModelConfig` into the inference index
that contains additional metadata derived from the running job.
2019-10-16 15:46:20 -04:00
Martijn van Groningen aff0c9babc
This commits merges (#48040) the enrich-7.x feature branch,
which is backport merge and adds a new ingest processor, named enrich processor,
that allows document being ingested to be enriched with data from other indices.

Besides a new enrich processor, this PR adds several APIs to manage an enrich policy.
An enrich policy is in charge of making the data from other indices available to the enrich processor in an efficient manner.

Related to #32789
2019-10-15 17:31:45 +02:00