Commit Graph

54198 Commits

Author SHA1 Message Date
Ryan Ernst d62981fb27
Remove opensuse 42 from vagrant tests (#63759)
Opensuse 42 has not worked in a while. The test image is unmaintained,
and cannot be launched. It was removed from CI packaging test runs, but
still remained in vagrant tests. This commit removes it from vagrant
tests.
2020-10-15 13:05:46 -07:00
Jay Modi ebdaeb2f9a
Ensure cancelled jobs do not continue to run (#63771)
This commit ensures that jobs within the SchedulerEngine do not
continue to run after they are cancelled. There was no synchronization
between the cancel method of an ActiveSchedule and the run method, so
an actively running schedule would go ahead and reschedule itself even
if the cancel method had been called.

This commit adds synchronization between cancelling and the scheduling
of the next run to ensure that the job is cancelled. In real life
scenarios this could manifest as a job running multiple times for
SLM. This could happen if a job had been triggered and was cancelled
prior to completing its run such as if the node was no longer the
master node or if SLM was stopping/stopped.

Closes #63754
Backport of #63762
2020-10-15 14:01:14 -06:00
Jay Modi 4d6daa6e40
Handle missing logstash index exceptions (#63753)
This commit updates the APIs in the logstash plugin to handle
IndexNotFoundExceptions that are returned by client calls. Until we
have the creation of this index in place, we need to handle this case
and not let the exception propagate out of the API.

Backport of #63698
2020-10-15 09:45:02 -06:00
Marios Trivyzas 1dbd3a90ae
EQL: [Tests] Use snapshot from 7.10
To be able to run the tests from 7.10 onwards use a snapshot created
with 7.10.

Follows: #63735
2020-10-15 17:28:52 +02:00
Bogdan Pintea 35f5c49e54
SQL: integer parameter validation in string functions (#63338) (#63728)
* SQL: integer parameter validation in string functions (#58923)

In insert, locate, substring function, when argument `start` or `length` is greater than Integer.MAX_INT OR less then Integer.MIN_INT + 1 (note that `start` need to minus 1), it causes overflow and leads to unexpected results.

* Add range checks for BinaryStringNumericProcessors

- Add range checks for Left, Right, Repeat.
- Minor refactorings on initial PR changes.

Co-authored-by: yinanwu <yinanwu@tencent.com>
(cherry picked from commit bf6dc58b93529f977d035a846d083b1c31867694)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-10-15 16:00:48 +02:00
James Rodewig 41ee8df334
[DOCS] Clarifies the effect of per-field boosting (#63733) (#63744)
The original description of per-field boosting is incorrect. Boosting a
field does not imply that it is more important relative to other fields.
It simply means that the score is multiplied by the supplied boost
value. Due to the differences in each field's term and document
statistics, it's not possible to imply relative importance of fields
based on the per-field boost value alone.

Co-authored-by: Josh Devins <josh.devins@elastic.co>
2020-10-15 09:46:36 -04:00
Marios Trivyzas 095f979060
EQL: [Tests] Add correctness integration tests (#63644) (#63735)
Add a new gradle module under eql/qa which runs and validates a set of
queries over a 4m event dataset (restored from a snapshot residing in a
gcs bucket). The results are providing by running the exact set of queries
with Python EQL against the same dataset.

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
(cherry picked from commit 1cf789e5fcfb0f364f665bfaac021e24a4c2f556)

Co-authored-by: Mark Vieira <portugee@gmail.com>
2020-10-15 15:28:26 +02:00
James Rodewig eb3ae9a204
[DOCS] Replace xref with external link (#63736) 2020-10-15 08:47:04 -04:00
Costin Leau 06eae58d40 EQL: Fix translation of bool fields (#63694)
This commit fixes two issues in dealing with bool fields in EQL:
- avoid simplifications of field == true expressions
- adding comparison to clauses on fields missing logic (where bool)

Fix #63693

(cherry picked from commit d10a5d0e842bbd4e0031834de948ceb24da3872b)
(cherry picked from commit 0227da3a275c7f22ff524d99d53e1a79146f9e28)
2020-10-15 14:38:31 +03:00
Andrei Dan 2cc0ccb110
Docs: sync breaking changes from release notes (#63479)
This syncs breaking changes from release notes

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-15 08:45:26 +01:00
Andrei Dan e011d0328b
Add release notes for 7.10.0 (#63460)
This adds the release notes for 7.10.0

Co-authored-by: David Roberts <dave.roberts@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-10-15 08:38:43 +01:00
Hendrik Muhs 65ef399563 [Transform] increase timeout in testStopWaitForCheckpoint (#63657)
increase the overall timeout by increasing the wait time after every retry.

fixes #63365
2020-10-15 08:51:33 +02:00
Jay Modi 822fea9889
Fix threadpool setting test for system_write (#63706)
This commit fixes the UpdateThreadPoolSettingsTests to be aware of the
hard limit on the maximum size of the system_write executor. This
executor has a hard limit that matches the write executor, which is
the number of allocated processors.

Closes #63131
Backport #63700
2020-10-14 14:57:43 -06:00
Christoph Büscher fe75bf5cf3 [Docs] Correct default scroll_size for update by query (#63656)
The current _update_by_query documentation mentions a scroll_size default of 100 and later another default of 1000.
We use the default of 1000 defined in AbstractBulkByScrollRequest and this PR changes the documentation accordingly.

Closes #63637
2020-10-14 16:43:36 +02:00
James Rodewig bd7633482b
[DOCS] EQL: Remove `match` fn (#63271) (#63677) 2020-10-14 10:13:00 -04:00
James Rodewig 5fc25442c4 [DOCS] Reword EQL intro 2020-10-14 10:03:21 -04:00
James Rodewig ac2b668016
[DOCS] Fix AbstractDiffable typo (#59034) (#63668)
Co-authored-by: Howard <danielhuang@tencent.com>
2020-10-14 09:56:56 -04:00
James Rodewig f9adb36d7d
[DOCS] Update `ignore_unavailable` default for EQL search API (#63210) (#63674) 2020-10-14 09:56:46 -04:00
James Rodewig 5de05ba69c
[DOCS] EQL: Add `:` operator, remove wildcard operator (#63195) (#63666) 2020-10-14 09:31:20 -04:00
James Rodewig e4b4351a43
[DOCS] EQL: Remove Endgame EQL refs (#63636) (#63664) 2020-10-14 08:48:56 -04:00
Albert Zaharovits 2b7fbe9957 Add the missing apikey.* fields to the logfile audit layout for docker builds (#63609)
The layout pattern for the security audit for docker builds was missing the apiKey.* fields.
2020-10-14 13:58:41 +03:00
Tanguy Leroux 57b5715bf7
Add CCR repository test for snapshot shard size (#63649)
Following #61906 this commit adds two new integration tests
that verifies the sizes of snapshotted shards for CCR repositories.

Backport of #63590
2020-10-14 12:51:42 +02:00
Armin Braun 424b313784
Adapt Shard Generation Assertion for 7.x (#63625) (#63642)
In 7.x we can have `null` generations so we need to adjust the `assert`
accordingly.
See e.g. failure https://gradle-enterprise.elastic.co/s/dgypleytdotfu/tests/:server:internalClusterTest/org.elasticsearch.snapshots.ConcurrentSnapshotsIT/testConcurrentSnapshotWorksWithOldVersionRepo
2020-10-14 06:57:25 +02:00
Ryland Herrick 7e8769a666
EQL: make allow_no_indices true by default (#63573) (#63645)
* Allow all indices options variants
Irrespective of allow_no_indices value, throw VerificationException when
there is no index validated

Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
2020-10-14 03:41:04 +03:00
Nhat Nguyen 9015b50e1b
Check docs limit before indexing on primary (#63273)
Today indexing to a shard with 2147483519 documents will fail that
shard. We should check the number of documents and reject the write
requests instead.

Closes #51136
2020-10-13 17:39:08 -04:00
James Rodewig 8455cf8ff0 [DOCS] Remove unneeded word in EQL docs 2020-10-13 13:57:41 -04:00
James Rodewig 086df0636e
[DOCS] EQL: Document multi-value field support (#63622) (#63633) 2020-10-13 12:44:48 -04:00
Lee Hinman 7371e51583
[7.10] Add DiscoveryNodeRole compatibility role for bwc tier serialization (#63581) (#63613)
Backports the following commits to 7.10:

    Add DiscoveryNodeRole compatibility role for bwc tier serialization (#63581)
2020-10-13 09:17:15 -06:00
Andras Palinkas f9c8ae5f77 SQL: Improve parser error message for `ESCAPE` (#63616)
Mentions the list of wildchars in case a wildchar is used as an
`ESCAPE` character.

Relates #63428

(cherry picked from commit 74cbcf871e9593b3640e382ae6845168fd14966b)
2020-10-13 10:47:06 -04:00
Andras Palinkas 4aabc050a0 SQL: Escaped wildcard (*) not accepted in LIKE (#63428)
For a query like `SELECT name FROM test WHERE name LIKE ''%c*'` ES SQL
generates an error. `*` is not a special character in a `LIKE` construct
and it's expected to not needing to be escaped, so the previous query
should work as is.
In the LIKE pattern any `*` character was treated as invalid character
and the usage of `%` or `_` was suggested instead. But `*` is a valid,
acceptable non-wildcard on the right side of the `LIKE` operator.

Fix: #55108
(cherry picked from commit 190d9fe3deb31aed0d8f312007360625d4fff217)
2020-10-13 10:47:03 -04:00
Armin Braun f70391c6cc
Fix Broken Snapshot State Machine in Corner Case (#63534) (#63608)
This fixes a gap in testing and a bug that can occur in various forms:
When we would start a snapshot or clone related to a shard that was done
snapshotting/cloning but its overall operation was not yet finalized
at the time of starting the operation, we would base the operation off of
the wrong generation. This would not cause a corrupted repo, but would
cause the operation to be `PARTIAL`.
This commit fixes the state machine to take into account the correct generation
in this case.

Closes #63498
2020-10-13 16:05:34 +02:00
James Rodewig 845ccc2264
[DOCS] Fix dup word in ShardRouting hashcode method. (#63452) (#63583)
Co-authored-by: Howard <danielhuang@tencent.com>
2020-10-13 09:05:19 -04:00
Andrei Dan b3aa575f0d
Add release 7.10 highlights (#63468)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>
Co-authored-by: Adrien Grand <jpountz@gmail.com>
2020-10-13 13:25:08 +01:00
Rene Groeschke 26f58c8bdb
Fix build tools git tests on windows 7x (#63456)
* Add error logging for git commands in build-tools integTests
* Explicitly set user and email in git related func tests
2020-10-13 14:12:52 +02:00
Przemysław Witek acbd48f834
[ML] Allow setting num_top_classes to a special value -1 (#63587) (#63602) 2020-10-13 13:57:50 +02:00
Adrien Grand 652af26369
Add space efficiency release highlight to 7.10. (#63475)
This release highlight is brought in by Lucene 8.7.

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
2020-10-13 13:42:21 +02:00
István Zoltán Szabó 7139c1d7db
[DOCS] Adds AUC ROC classification metric to the API examples (#63563) (#63596) 2020-10-13 12:18:29 +02:00
Tanguy Leroux 8499924e51
InternalSnapshotsInfoService should also removed failed snapshot shard size infos (#63492) (#63592)
Relates #61906
2020-10-13 10:42:38 +02:00
Julie Tibshirani 9e52513c7b
Add support for missing value fetchers. (#63585)
This PR implements value fetching for the following field types:
* `text` phrase and prefix subfields
* `search_as_you_type`, plus its subfields
* `token_count`, which is implemented by fetching doc values

Supporting these types helps ensure that retrieving all fields through
`"fields": ["*"]` doesn't fail because of unsupported value fetchers.
2020-10-12 17:34:21 -07:00
David Roberts 3f210e2620 [ML] Load data streams plugin for ML internal cluster tests (#63560)
Now that deprecation logs get indexed to a data stream, if we
do not load the data stream plugin in our tests and any test
generates a deprecation log message then millions of exceptions
get logged, slowing down the tests to the extent that they can
fail.

This change loads the data streams plugin during the ML internal
cluster tests.  (It should already be present in external cluster
tests.)

Fixes #63548
2020-10-12 17:46:50 +01:00
Gordon Brown 5fe51b1bf4
Fix type name in testSystemIndexMetadataIsUpgraded (#63506)
This test used _doc as the mapping type name, which needs to be set
to doc for versions prior to 6.7.0. This commit fixes the test to use
the proper type name for the current BWC version.
2020-10-12 10:44:38 -06:00
James Rodewig 25ead6abb7
[DOCS] Fix typo in search profile docs (#63522) (#63559)
Co-authored-by: ondrejbardon <70531797+ondrejbardon@users.noreply.github.com>
2020-10-12 10:13:13 -04:00
James Rodewig 9c170706dd
[DOCS] EQL: Move to beta (#63284) (#63552) 2020-10-12 09:12:26 -04:00
Dimitris Athanasiou e1c418aac7
[7.10][ML] Validate dest pipeline exists on transform update (#63494) (#63549)
Adds validation that the dest pipeline exists when a transform
is updated. Refactors the pipeline check into the `SourceDestValidator`.

Fixes #59587

Backport of #63494
2020-10-12 15:41:35 +03:00
Tomas Della Vedova 8b07750a8b
Updated version string in yaml test (#63410) (#63533) 2020-10-12 11:19:36 +02:00
David Turner 6ddeaa4791 Third-party S3-compatible repos aren't unsupported (#63238)
Today in the `repository-s3` docs we say

> Other S3-compatible storage systems may also work with Elasticsearch,
> but these are not tested or supported.

Saying that they are explicitly not supported is a very strong
statement, implying that it is positively irresponsible to use anything
except S3 or Minio, even after extensive testing. S3-compatibility in
third-party systems has matured in recent years and users today report
success with a good number of them. In contrast, we effectively claim
support for any old NFS implementation when used with the
shared-filesystem repository.

This commit weakens this statement, removing the absolute claim of
unsupportedness and instead spelling out that the user is responsible
for ironing out any incompatibilities with the storage supplier.
2020-10-12 09:49:30 +01:00
Tim Brooks 56092b1a9f
Flush translog writer before adding new operation (#63505)
Currently we flush the Translog buffer when a new operation causes the
buffer to breach 1MB. This introduces a scenario where an exception is
thrown AFTER the writer has accepted the operation. To avoid this, this
commit flushes the Translog in an #add call before adding a new
operation.

This fixes #63299.
2020-10-09 10:02:55 -06:00
Ryan Ernst 01dde647a6
Remove glibc warning (#63447)
Now that the bundled jdk is switched to the Oracle JDK, the glibc
warning is no longer relevant. This commit removes the warning.
2020-10-08 14:45:43 -07:00
Lee Hinman 27f13267f0
Add release note highlights for data tiers (#63427)
This adds release note highlights for the data tiers formalization feature.

Relates to #60848
2020-10-08 14:39:09 -06:00
Julie Tibshirani ae2fc4118d Add factory methods for common value fetchers. (#63438)
This PR adds factory methods for the most common implementations:
* `SourceValueFetcher.identity` to pass through the source value untouched.
* `SourceValueFetcher.toString` to simply convert the source value to a string.
2020-10-08 12:14:53 -07:00