Commit Graph

54185 Commits

Author SHA1 Message Date
Henning Andersen ddd897f747 Fix test timeout for health on master failover (#63455)
testHealthOnMasterFailover could timeout on some of the health requests
in the case where an index is added, since the recovery leads to
extended test run time.

Closes #62690
2020-10-21 14:31:53 +02:00
Yang Wang 428fd7218c
Use asterisk instead of empty string to clear all cached entries (#63907) (#63989)
The officially supported way to clearing all entries from a cache is to use 
wildcard of either * or _all. Though empty string has the same effect, it was
never intended. Therefore the tests should not use empty string and this PR
changes them to use *.
2020-10-21 23:14:01 +11:00
Hendrik Muhs f2517678aa
[7.10][Transform] add support for unsigned_long data type (#63957)
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes #63871
backport #63940
2020-10-20 21:05:46 +02:00
Nik Everett 8d30766a7d
Fix scripted metric BWC serialization (backport of #63821) (#63897)
We had and an error when serializing fully reduced scripted metrics.
Small typo and sever lack of tests..... Anyway, this fixed the one
character typo and adds a bunch more tests.
2020-10-20 13:15:26 -04:00
István Zoltán Szabó 6f977fc6f7
[7.10][DOCS] Adds ML related release highlights to What's new. (#63934)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-20 15:46:31 +02:00
Ignacio Vera d0f5066310
Upgrade to lucene-8.7.0-snapshot-72d8528c3a6 (#63912) (#63928) (#63933) 2020-10-20 15:08:06 +02:00
Adam Locke e064db27b1
Move clone snapshot API page. (#63902) (#63936) 2020-10-20 09:03:56 -04:00
Tanguy Leroux b2e07076a0
Add snapshot shard size based test in DiskThresholdDeciderTests (#63913)
This commit adds a test in DiskThresholdDeciderTests that verifies
 the allocation of a snapshot recovery source based shard in the 
situation where the snapshot shard size was successfully provided 
by the SnapshotInfoService introduced in #61906 and when the 
service failed to provide the size.

Relates #61906
2020-10-20 14:59:00 +02:00
Benjamin Trent eff7f06ca6
[ML] fix inference binary classification predication label and feature importance (#63688) (#63930)
When calculating feature importance, the leaf values directly correlate the value of the importance.

Consequently, positive leaf values -> positive feature importance

negative leaf values -> negative feature importance.

It follows that for binary classification, this is done such that the importance relates to the leaf values, which relate directly to the "probability of class 1".

So, the feature importance calculated is always for the importance as it relates to class 1.

The inverse is the importance as it relates to class 0.
2020-10-20 08:50:15 -04:00
Rene Groeschke b5448f07f3
Fix resolveAllDependencies by removing refs to non existing configs (#63409) (#63594) (#63715)
* Fix concurrent modification on task realization
* Use taskprovider instead of relying on tasks in distribution setup
* Port more task references in :distribution to task provider
* Fix nullpointer in distribution setup
2020-10-20 12:43:17 +02:00
Jim Ferenczi 3423f214dd Composite aggregation must check live docs when the index is sorted (#63864)
This change ensures that the live docs are checked in the composite aggregator
when the index is sorted.
2020-10-20 11:40:28 +02:00
Armin Braun 1880bcdc09
Add REST Test for Snapshot Clone API (#63863) (#63881)
Adds snapshot clone REST tests and HLRC support for the API.
2020-10-20 09:48:03 +02:00
Mayya Sharipova 1287df4074 Fix max/min aggs for unsigned_long (#63904)
Max and min aggs were producing wrong results for unsigned_long field
if field was indexed. If field is indexed for max/min aggs instead of
field data, we use values from indexed Points, values of which
are derived using method pointReaderIfPossible. Before
UnsignedLongFieldType#pointReaderIfPossible was incorrectly
producing values, as it failed to shift them back to original
values.

This patch fixes method pointReaderIfPossible to produce
correct original values.

Relates to #60050
2020-10-19 15:59:55 -04:00
James Rodewig 3369216087 [DOCS] Fix typo 2020-10-19 14:44:43 -04:00
Nik Everett 5583db5a73
Fix broken parent and child aggregator (backport #63811) (#63892)
In #57892 I broke *some* sub-aggregations inside of the `parent` and
`child` aggregator, specifically any sub-aggregations that do work in
the `postCollect` phase. This fixes it by delaying the post collect
phase of aggs under `parent` and `child` until `beforeBuildingBuckets`
because, well, we haven't done *any* collection until after that phase.
2020-10-19 13:05:22 -04:00
Adam Locke c28c3422bb
[DOCS] [7.10] Combining important config settings into a single page (#63849) (#63883)
* [DOCS] Combining important config settings into a single page (#63849)

* Combining important config settings into a single page.

* Updating ids for two pages causing link errors and implementing redirects.

* Updating links to use IDs instead of xrefs.
2020-10-19 12:59:44 -04:00
James Rodewig af9e96d681
[DOCS] EQL: Update `allow_no_indices` default (#63748) (#63891)
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-10-19 12:31:22 -04:00
Rene Groeschke 34ccd3f0dc Explicitly use TLS 1.2 in Gradle (#63880) 2020-10-19 08:57:26 -07:00
Aref Razavi 2950818d44 Remove useless parentheses in bucket_key formula (#63868) 2020-10-19 11:53:33 +02:00
István Zoltán Szabó eef670f004
[DOCS] Clarifies destination index mappings for Transforms. (#63794) (#63804) 2020-10-19 10:03:08 +02:00
Julie Tibshirani f122b88bc5 Remove dependency from version plugin. 2020-10-18 14:09:32 -07:00
Julie Tibshirani ff312512c5 Mark the search 'fields' option as beta. (#63699)
We've identified two important enhancements that may affect the API. We expect
any API changes from these enhancements to be minor, but want to leave open the
possibility for small breaks. For example, we may end up returning unmapped
fields by default, or omitting nested fields from the root hit. The impact to
users should be quite small.

We're tracking the issues we need to resolve before removing the 'beta' label
here: #60985.
2020-10-18 14:09:27 -07:00
Lisa Cawley e6959d0d20
[7.10][DOCS] Add grant API key API and grant_api_key privilege (#63853) 2020-10-16 15:41:46 -07:00
Andrei Stefan 5f3c79d64b
Remove filter from QL's field_caps requests (#63840) (#63845)
(cherry picked from commit f009e6341d0fc0471f212d5a41c91e7aab77e006)
2020-10-17 01:36:26 +03:00
Jay Greenberg 3e5e54b884 [DOCS] Add monitoring note (#61793) 2020-10-16 13:36:28 -07:00
James Rodewig 07e4129328
[DOCS] Remove 7.9.2 coming tags (#62872) (#63830)
Removes coming tags from 7.9.2 Release Notes.

Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
2020-10-16 12:44:53 -04:00
James Rodewig 309bb9bb3a
[7.10] Release notes for 7.9.2 (#62664) (#63827)
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
2020-10-16 12:18:36 -04:00
James Rodewig df8070722a
[DOC] Review integration page (#63799) (#63816)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Aaron Bull Schaefer <abs@elastic.co>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
2020-10-16 11:40:09 -04:00
Mayya Sharipova c0c1a7a9a6 Apply boost only once for distance_feature query (#63767)
Currently if distance_feature query contains boost,
it incorrectly  gets applied twice: in AbstractQueryBuilder::toQuery and
we also pass this boost to Lucene's LongPoint.newDistanceFeatureQuery.
As a result we get incorrect scores.

This fixes this error to ensure that boost is applied only once.

Closes #63691
2020-10-16 10:02:55 -04:00
James Rodewig 179c6d4014
[DOCS] Fix role template snippet (#63774) (#63779) 2020-10-16 08:24:29 -04:00
Enrico Zimuel e724be5f41
Specify security.get_role_mapping and security.get_role name as list (#53207) (#63785)
This commit updates the rest API specs for

- security.get_role_mapping to specify name as a list type.
- security.get_role to specify name as a list type.

Co-authored-by: Russ Cam <russ.cam@elastic.co>
2020-10-16 14:22:37 +02:00
Ioannis Kakavas 364511395d
[7.10] Move RestRequestFilter to core (#63507)
Move RestRequestFilter to core so that Rest requests outside xpack can use 
it to filter fields and expand its usage.

Backport of #63507
2020-10-16 13:57:52 +03:00
Andrei Dan f4a856d423
Update 7.10 release notes after respin (#63725) (#63757)
Adds PRs diff to the release notes.

(cherry picked from commit 1ede4b332e5f87591710723e1a6ff9353384e2ff)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-10-16 09:06:08 +01:00
Tanguy Leroux 7ea44d20c3
Try to fix DiskThresholdDeciderIT (#63614) (#63721)
This is another attempt to fix #62326 as my previous 
attempts failed (#63112, #63385).
2020-10-16 09:20:54 +02:00
Jim Ferenczi 1d78bd0f72 Async search should retry updates on version conflict (#63652)
* Async search should retry updates on version conflict

The _async_search APIs can throw version conflict exception when the internal response
is updated concurrently. That can happen if the final response is written while the user
extends the expiration time. That scenario should be rare but it happened in Kibana for
several users so this change ensures that updates are retried at least 5 times. That
should resolve the transient errors for Kibana. This change also preserves the version
conflict exception in case the retry didn't work instead of returning a confusing 404.
This commit also ensures that we don't delete the response if the search was cancelled
internally and not deleted explicitly by the user.

Closes #63213
2020-10-16 08:49:02 +02:00
Albert Zaharovits f4e1e6893d Add view_index_metadata over metricbeat-* for monitoring agent (#63750)
The `remote_monitoring_agent` reserved role is extended to grant more privileges
over the metricbeat-* index pattern.
In addition to the index and create_index index privileges that it granted already,
it now also grants the view_index_metadata privilege.

Closes #63203
2020-10-16 02:13:55 +03:00
James Rodewig da84622764
[DOCS] Document `discovery.seed_hosts` only supports YAML sequences and arrays (#63746) (#63776) 2020-10-15 16:56:48 -04:00
Ryan Ernst d62981fb27
Remove opensuse 42 from vagrant tests (#63759)
Opensuse 42 has not worked in a while. The test image is unmaintained,
and cannot be launched. It was removed from CI packaging test runs, but
still remained in vagrant tests. This commit removes it from vagrant
tests.
2020-10-15 13:05:46 -07:00
Jay Modi ebdaeb2f9a
Ensure cancelled jobs do not continue to run (#63771)
This commit ensures that jobs within the SchedulerEngine do not
continue to run after they are cancelled. There was no synchronization
between the cancel method of an ActiveSchedule and the run method, so
an actively running schedule would go ahead and reschedule itself even
if the cancel method had been called.

This commit adds synchronization between cancelling and the scheduling
of the next run to ensure that the job is cancelled. In real life
scenarios this could manifest as a job running multiple times for
SLM. This could happen if a job had been triggered and was cancelled
prior to completing its run such as if the node was no longer the
master node or if SLM was stopping/stopped.

Closes #63754
Backport of #63762
2020-10-15 14:01:14 -06:00
Jay Modi 4d6daa6e40
Handle missing logstash index exceptions (#63753)
This commit updates the APIs in the logstash plugin to handle
IndexNotFoundExceptions that are returned by client calls. Until we
have the creation of this index in place, we need to handle this case
and not let the exception propagate out of the API.

Backport of #63698
2020-10-15 09:45:02 -06:00
Marios Trivyzas 1dbd3a90ae
EQL: [Tests] Use snapshot from 7.10
To be able to run the tests from 7.10 onwards use a snapshot created
with 7.10.

Follows: #63735
2020-10-15 17:28:52 +02:00
Bogdan Pintea 35f5c49e54
SQL: integer parameter validation in string functions (#63338) (#63728)
* SQL: integer parameter validation in string functions (#58923)

In insert, locate, substring function, when argument `start` or `length` is greater than Integer.MAX_INT OR less then Integer.MIN_INT + 1 (note that `start` need to minus 1), it causes overflow and leads to unexpected results.

* Add range checks for BinaryStringNumericProcessors

- Add range checks for Left, Right, Repeat.
- Minor refactorings on initial PR changes.

Co-authored-by: yinanwu <yinanwu@tencent.com>
(cherry picked from commit bf6dc58b93529f977d035a846d083b1c31867694)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-10-15 16:00:48 +02:00
James Rodewig 41ee8df334
[DOCS] Clarifies the effect of per-field boosting (#63733) (#63744)
The original description of per-field boosting is incorrect. Boosting a
field does not imply that it is more important relative to other fields.
It simply means that the score is multiplied by the supplied boost
value. Due to the differences in each field's term and document
statistics, it's not possible to imply relative importance of fields
based on the per-field boost value alone.

Co-authored-by: Josh Devins <josh.devins@elastic.co>
2020-10-15 09:46:36 -04:00
Marios Trivyzas 095f979060
EQL: [Tests] Add correctness integration tests (#63644) (#63735)
Add a new gradle module under eql/qa which runs and validates a set of
queries over a 4m event dataset (restored from a snapshot residing in a
gcs bucket). The results are providing by running the exact set of queries
with Python EQL against the same dataset.

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
(cherry picked from commit 1cf789e5fcfb0f364f665bfaac021e24a4c2f556)

Co-authored-by: Mark Vieira <portugee@gmail.com>
2020-10-15 15:28:26 +02:00
James Rodewig eb3ae9a204
[DOCS] Replace xref with external link (#63736) 2020-10-15 08:47:04 -04:00
Costin Leau 06eae58d40 EQL: Fix translation of bool fields (#63694)
This commit fixes two issues in dealing with bool fields in EQL:
- avoid simplifications of field == true expressions
- adding comparison to clauses on fields missing logic (where bool)

Fix #63693

(cherry picked from commit d10a5d0e842bbd4e0031834de948ceb24da3872b)
(cherry picked from commit 0227da3a275c7f22ff524d99d53e1a79146f9e28)
2020-10-15 14:38:31 +03:00
Andrei Dan 2cc0ccb110
Docs: sync breaking changes from release notes (#63479)
This syncs breaking changes from release notes

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-15 08:45:26 +01:00
Andrei Dan e011d0328b
Add release notes for 7.10.0 (#63460)
This adds the release notes for 7.10.0

Co-authored-by: David Roberts <dave.roberts@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-10-15 08:38:43 +01:00
Hendrik Muhs 65ef399563 [Transform] increase timeout in testStopWaitForCheckpoint (#63657)
increase the overall timeout by increasing the wait time after every retry.

fixes #63365
2020-10-15 08:51:33 +02:00
Jay Modi 822fea9889
Fix threadpool setting test for system_write (#63706)
This commit fixes the UpdateThreadPoolSettingsTests to be aware of the
hard limit on the maximum size of the system_write executor. This
executor has a hard limit that matches the write executor, which is
the number of allocated processors.

Closes #63131
Backport #63700
2020-10-14 14:57:43 -06:00