Commit Graph

50987 Commits

Author SHA1 Message Date
lcawl d3fa0ec730 [DOCS] Fixes typo in node settings 2020-04-02 16:01:54 -07:00
Lisa Cawley 98965116fe [DOCS] Clarify ML and transform settings on coordinating nodes (#54676) 2020-04-02 15:38:15 -07:00
Julie Tibshirani 5fb7602227
Disallow changing 'enabled' on the root mapper. (#54681)
In #33933 we disallowed changing the `enabled` parameter in object mappings.
However, the fix didn't cover the root object mapper. This PR adjusts the change
to also include the root mapper and clarifies the error message.
2020-04-02 15:28:48 -07:00
Dan Hermann 39c4ec6821
[7.x] Create first backing index when creating data stream 2020-04-02 17:19:35 -05:00
Benjamin Trent 6e73f67f3b
[ML] unmute categorization test for native backport (#54679) 2020-04-02 17:08:19 -04:00
Benjamin Trent 7fe38935f6
[ML] add training_percent to analytics process params (#54605) (#54678)
This adds training_percent parameter to the analytics process for Classification and Regression. This parameter is then used to give more accurate memory estimations.

See native side pr: elastic/ml-cpp#1111
2020-04-02 17:08:06 -04:00
Nik Everett 54ea4f4f50 Begin to drop pipeline aggs from the result tree (backport of #54311) (#54659)
Removes pipeline aggregations from the aggregation result tree as they
are no longer used. This stops us from building the pipeline aggregators
at all on data nodes except for backwards compatibility serialization.
This will save a tiny bit of space in the aggregation tree which is
lovely, but the biggest benefit is that it is a step towards simplifying
pipeline aggregators.

This only does about half of the work to remove the pipeline aggs from
the tree. Removing all of it would, well, double the size of the change
and make it harder to review.
2020-04-02 16:45:12 -04:00
Dan Hermann 547ab849b4
Test to enforce response to invalid data stream names (#54663) 2020-04-02 15:04:03 -05:00
Nik Everett cc6468a0cb
Fix BWC error on pipeline aggs (#54672)
I derped out on a last minute bug fix when backporting #54282 and it
only causes the tests to fail about half the time. So I didn't catch
it until after merging. Great! This fixes it.
2020-04-02 14:51:30 -04:00
David Turner 4e083cd97d indices.recovery.max_bytes_per_sec may be per-node (#54633)
The `indices.recovery.max_bytes_per_sec` recovery bandwidth limit can differ
between nodes if it is not set dynamically, but today this is not obvious. This
commit adds a paragraph to its documentation clarifying how to set different
bandwidth limits on each node.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-02 18:15:41 +01:00
Benjamin Trent 4a1610265f
[7.x] [ML] add new inference_config field to trained model config (#54421) (#54647)
* [ML] add new inference_config field to trained model config (#54421)

A new field called `inference_config` is now added to the trained model config object. This new field allows for default inference settings from analytics or some external model builder.

The inference processor can still override whatever is set as the default in the trained model config.

* fixing for backport
2020-04-02 12:25:10 -04:00
Jason Tedor 2113c1ffb6
Fix autoscaling internal cluster release tests
This commit addresses an issue with the autoscaling feature flag not
being registered in release builds of the internal cluster tests. This
commit addresses this by enabling the system property that is needed,
but only in release builds.
2020-04-02 11:48:02 -04:00
Benjamin Trent 65233383f6
[7.x] [ML] prefer secondary authorization header for data[feed|frame] authz (#54121) (#54645)
* [ML] prefer secondary authorization header for data[feed|frame] authz (#54121)

Secondary authorization headers are to be used to facilitate Kibana spaces support + ML jobs/datafeeds.

Now on PUT/Update/Preview datafeed, and PUT data frame analytics the secondary authorization is preferred over the primary (if provided).

closes https://github.com/elastic/elasticsearch/issues/53801

* fixing for backport
2020-04-02 11:20:25 -04:00
Zachary Tong 20d67720aa
Refactor Percentiles/Ranks aggregation builders and factories (#51887) (#54537)
- Consolidates HDR/TDigest factories into a single factory
- Consolidates most HDR/TDigest builder into an abstract builder
- Deprecates method(), compression(), numSigFig() in favor of a new
unified PercentileConfig object
- Disallows setting algo options that don't apply to current algo

The unified config method carries both the method and algo-specific
setting. This provides a mechanism to reject settings that apply
to the wrong algorithm.  For BWC the old methods are retained
but marked as deprecated, and can be removed in future versions.

Co-authored-by: Mark Tozzi <mark.tozzi@gmail.com>

Co-authored-by: Mark Tozzi <mark.tozzi@gmail.com>
2020-04-02 10:39:41 -04:00
Nik Everett a5adac0d1e
Fix pipeline agg serialization for ccs (backport of #54282) (#54468)
This fixes pipeline aggregations used in cross cluster search from an older
version of Elasticsearch to a newer version of Elasticsearch. I broke
this in #53730 when I was too aggressive in shutting off serialization
of pipeline aggs. In particular, this comes up when the coordinating
node is pre-7.8.0 and the gateway node is on or after 7.8.0.

The fix is another step down the line to remove pipeline aggregators
from the aggregation tree. Sort of. It create a new
`List<PipelineAggregator>` member in `InternalAggregation` *but* it is
only used for bwc serialization and it is fed by the mechanism
established in #53730 to read the pipelines from the
2020-04-02 10:35:40 -04:00
Jason Tedor 7467cc04ec
Remove toXContent from autoscaling request classess (#54643)
These methods are not needed, we were only following a pattern in the
rest of the codebase, but it's legacy from the HLRC sharing
request/response objects with the server.
2020-04-02 10:30:20 -04:00
Christoph Büscher 4882bb5cd8 [Tests] Remove unneeded test logging (#54634)
This logging was added to get better insight into #35644 
which was closed by #37302 and can be removed now.
2020-04-02 15:47:38 +02:00
David Roberts 4b4800e096
[ML] Take more care that normalize processes use unique named pipes (#54641)
When one of ML's normalize processes fails to connect to the JVM
quickly enough and another normalize process for the same job
starts shortly afterwards it is possible that their named pipes
can get mixed up.

This change avoids the risk of that by adding an incrementing
counter value into the named pipe names used for normalize
processes.

Backport of #54636
2020-04-02 14:25:31 +01:00
qiye de8e0200fe [DOCS] Correct `shape` field release in 7.5 release highlights (#54631)
The `shape` field was added in 7.4, not 7.3.
This corrects a small error in the 7.5 release highlights.
2020-04-02 09:19:40 -04:00
Benjamin Trent eb31be0e71
[7.x] [ML] add num_matches and preferred_to_categories to category defintion objects (#54214) (#54639)
* [ML] add num_matches and preferred_to_categories to category defintion objects (#54214)

This adds two new fields to category definitions.

- `num_matches` indicating how many documents have been seen by this category
- `preferred_to_categories` indicating which other categories this particular category supersedes when messages are categorized.

These fields are only guaranteed to be up to date after a `_flush` or `_close`

native change: https://github.com/elastic/ml-cpp/pull/1062

* adjusting for backport
2020-04-02 09:09:19 -04:00
Jason Tedor 54ecb009bb
Add delete autoscaling policy API (#54601)
This commit adds an API for deleting autoscaling policies.
2020-04-02 09:05:12 -04:00
Nik Everett b4feda84e8
Add scroll info to search task description (backport of #54606) (#54612)
Right now you can't tell from the task description whether or not the
search is a scroll. This adds that information to the description which
is super useful if you are trying to debug a cluster that is running out
of scroll contexts.
2020-04-02 09:04:49 -04:00
Jason Tedor 18b602280c
Add validation to the usage service (#54617)
Today the usage service can let in some issues, such as handlers that do
not have a name, where the errors do not manifest until later (calling
the usage API), or conflicting handlers with the same name. This commit
addresses this by adding some validation to the usage service.
2020-04-02 08:56:28 -04:00
Aleh Zasypkin 161eac1942
[7.x] Switch to the most recent Kibana configuration format and SAML/OIDC endpoints. (#54624) 2020-04-02 11:59:11 +02:00
Martijn Laarman 0ed20cc349 Rename cat.transform => cat.transforms (#54438)
* Rename cat.transform => cat.transforms

To match the url.

We typically prefer singular url nouns but _cat tends to use plural and
this API does in fact uses `/_cat/transforms`

* also rename the api in the spec and tests

(cherry picked from commit c495d220ac8fedba7f70f82387cd6d6a672b8b14)
2020-04-02 09:40:51 +02:00
István Zoltán Szabó 3a8e880fe6 [DOCS] Adds snippet comparing two indices to the painless examples (#54563) 2020-04-02 08:46:25 +02:00
Tim Vernum c40ec6a577
Turn on trace logging for failing test (#54623)
SamlIdentityProviderTests is failing with 409 conflicts that have not
been reproducible outside of CI.
This change turn on additional logging in this test to determine why
these conflict occur.

Relates: #54423
Backport of: #54475
2020-04-02 16:15:12 +11:00
Russ Cam 2978024375 Update rest API specs (#54252)
This commit updates the rest API specs to validate against a
JSON schema for the specifications. Most updates are to add
a description, whilst others fix typos and unify conventions
e.g. deprecations, descriptions, urls starting with /. The schema
conforms to draft-07 JSON schema.

(cherry picked from commit da37e01d32f9764c3937736ef0c7d3ab40af9a77)
2020-04-02 10:53:32 +10:00
Andy Bristol eb14635f1f
add tests to StatsAggregatorTests (#53768)
Adds tests for supported ValuesSourceTypes, unmapped fields, scripting,
and the missing param. The tests for unmapped fields and scripting are
migrated from the StatsIT integration test
2020-04-01 17:07:51 -07:00
Andy Bristol c87b830d06
migrate tests from MissingIT to agg tests (#53448)
Move the remaining tests for the missing aggregation into its
AggregatorTestCase out of its integration test and remove the IT
2020-04-01 17:05:44 -07:00
Russ Cam 110d9a7845 Correct description for mget API request URI index (#52305)
This commit corrects the description for the request URI index for the Multi Get (mget) API.
The index can only be a single index name (multiple or wildcard expressions not supported),
and acts as the index to use when "ids" are specified, or a document in the "docs" array does
not specify an index.

(cherry picked from commit aa4926ed7f91dfbf7973a01b1e4682e91dda11a9)
2020-04-02 09:08:34 +10:00
Russ Cam a2f59a2744 Add hidden value to expand_wildcards params (#54551)
This commit adds the hidden enum value to
all expand_wildcards params

(cherry picked from commit 581b8cdabe11444105edb62226b439ba4c7e908a)
2020-04-02 09:01:20 +10:00
Jake Landis d412fd1bbd
[7.x] Convert RestIntegTestTask to Java (#54528) (#54585)
This commit migrates the RestIntegTestTask from groovy to Java.
No changes to logic should be included, however the following changes
are needed:

* Move Fixture interface to Java (Java can not depend on Groovy classes)
* Support lazy evaluation of non-input System parameters (can not use Groovy strings)
* Use constants for system property names
* Remove dead System property pass through code (the build plugin does this already)
2020-04-01 17:33:30 -05:00
Andy Bristol ec76e7306e
supported field type tests for max agg (#53701)
Adds test hooks for testing supported ValuesSource types for the max
aggregation
2020-04-01 15:24:53 -07:00
Andy Bristol 5d0351ea00
add tests to SumAggregatorTests (#53568)
This adds tests for supported ValuesSourceTypes, unmapped fields,
scripting, and the missing param. The tests for unmapped fields and
scripting are migrated from the SumIT integration test
2020-04-01 15:24:21 -07:00
Andy Bristol 62a52465fc
aggregator and yaml tests for missing agg (#53214)
Tests for unmapped fields, the missing parameter, scripting, and correct
ValuesSource types in MissingAggregatorTests. Basic yaml tests for the
missing agg

For #42949
2020-04-01 15:23:08 -07:00
lcawl 949636944c [DOCS] Fixes shared attribute for feature importance 2020-04-01 14:52:08 -07:00
William Brafford 958e9d1b78
Refactor nodes stats request builders to match requests (#54363) (#54604)
* Refactor nodes stats request builders to match requests (#54363)

* Remove hard-coded setters from NodesInfoRequestBuilder

* Remove hard-coded setters from NodesStatsRequest

* Use static imports to reduce clutter

* Remove uses of old info APIs
2020-04-01 17:03:04 -04:00
Jason Tedor fd729a6509
Fix the name of an autoscaling policy test
The test name says it is testing the put autoscaling decision API, but
that is not right, since no such API exists (nor will exist). This
commit corrects the name of this test to reflect the fact that the test
is about the put autoscaling policy API.
2020-04-01 16:36:47 -04:00
Jason Tedor ccecb78c98
Rename the policy in put autoscaling policy docs
The put autoscaling policy docs use a "hot" policy as an
example. Instead, this commit changes the name of this policy to
"my_autoscaling_policy".
2020-04-01 16:32:03 -04:00
Gordon Brown f0cb8a56a9
Handle -1 gc_threshold settings explicitly (#54546)
Because -1 is technically a valid TimeValue (as a sentinel value), that is now
explicitly checked for when validating gc_thresholds. The tests are also
adjusted to test this case separately from other negative values.
2020-04-01 13:56:50 -06:00
Jason Tedor bd6b383926
Include autoscaling APIs in API docs (#54603)
This commit adds a top-level link to the autoscaling API reference page
to the API docs. Additionally, we add a conditional guard on the API
pages to only include them in development builds of the docs.
2020-04-01 15:44:13 -04:00
Mayya Sharipova bf4857d9e0
Search hit refactoring (#41656) (#54584)
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue #24422 to remove
dependency on MapperService to check if a field is metafield.

Relates to PR: #38373
Relates to issue #24422

Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
2020-04-01 15:19:00 -04:00
lcawl 2cd35bf696 [DOCS] Adds release highlights placeholder 2020-04-01 09:22:20 -07:00
Jason Tedor 8ed1a6cdb6
Use List.of convenience methods in 7.x autoscaling
We do not have access to JDK 9 collection convenience methods in 7.x
because we are compatible with JDK 8 there. Yet, we have recently added
a substitute for these convenience methods that even delegate to the
right places when running on JDK 9, to make backporting easier. This
commit utilizes these new methods in the autoscaling codebase.
2020-04-01 12:17:56 -04:00
Lisa Cawley f5ccf939d9 [DOCS] Clarifies API key breaking change (#54522) 2020-04-01 08:58:15 -07:00
James Rodewig 21abc311fd
[DOCS] Reformat `keyword_repeat` token filter (#54428) 2020-04-01 11:56:05 -04:00
James Rodewig 4982b720ef
[DOCS] EQL: Document `length` function (#54225) 2020-04-01 11:35:36 -04:00
Ioannis Kakavas c9ffa379ba
[7.x] Add end to end QA authentication test (#54215) (#54567)
Use the same ES cluster as both an SP and an IDP and perform
IDP initiated and SP initiated SSO. The REST client plays the role
of both the Cloud UI and Kibana in these flows

Backport of #54215

* fix compilation issues
2020-04-01 18:35:21 +03:00
Dan Hermann 1d0a9a1b27
Java8 implementations of collection initializers (#54183) 2020-04-01 10:27:18 -05:00