Commit Graph

50636 Commits

Author SHA1 Message Date
Nik Everett 7c9641ef9d
Simplify BucketedSort (#53199) (#53240)
Our lovely `BitArray` compactly stores "flags", lazilly growing its
underlying storage. It is super useful when you need to store one bit of
data for a zillion buckets or a documents or something. Usefully, it
defaults to `false`. But there is a wrinkle! If you ask it whether or
not a bit is set but it hasn't grown its underlying storage array
"around" that index then it'll throw an `ArrayIndexOutOfBoundsException`.
The per-document use cases tend to show up in order and don't tend to
mind this too much. But the use case in aggregations, the per-bucket use
case, does. Because buckets are collected out of order all the time.

This changes `BitArray` so it'll return `false` if the index is too big
for the underlying storage. After all, that index *can't* have been set
or else we would have grown the underlying array. Logically, I believe
this makes sense. And it makes my life easy. At the cost of three lines.

*but* this adds an extra test to every call to `get`. I think this is
likely ok because it is "very close" to an array index lookup that
already runs the same test. So I *think* it'll end up merged with the
array bounds check.
2020-03-06 15:27:51 -05:00
Ross Wolf d6813cb348
EQL: Convert wildcards to LIKE in analyzer (#51901)
* EQL: Convert wildcard comparisons to Like
* EQL: Simplify wildcard handling, update tests
* EQL: Lint fixes for Optimizer.java
2020-03-06 13:13:07 -07:00
Mayya Sharipova f96ad5c32d Mute testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows 2020-03-06 12:48:05 -05:00
Jay Modi a81460dbf5
Make watch history indices hidden (#52974)
This commit updates the template used for watch history indices with
the hidden index setting so that new indices will be created as hidden.

Relates #50251
Backport of #52962
2020-03-06 09:47:03 -07:00
Adam Canady a88d0c7ca3 [Docs] Correct examples for * and + in regexp-syntax.asciidoc (#53210) 2020-03-06 17:17:32 +01:00
Mark Vieira 09a3f45880
Mute ClassificationIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-06 07:38:04 -08:00
James Baiera 01f00df5cd
Mute RegressionIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet 2020-03-06 07:37:57 -08:00
Benjamin Trent 85d7112e78
[ML] Fixing datafeed bwc tests (#52959)
Datafeed bwc tests have been muted for some time in the 7.x. This is because of date_histogram interval deprecation warnings.

This commit fixes the tests as must as possible while still handling deprecation warnings.
2020-03-06 10:27:21 -05:00
István Zoltán Szabó bf3dcd4229 [DOCS] Adds deleting flag to the GET job stats API docs (#53223) 2020-03-06 16:04:20 +01:00
Christoph Büscher 9e561c2921 Fix AbstractBulkByScrollRequest slices parameter via Rest (#53068)
Currently the AbstractBulkByScrollRequest accepts slice values of 0 via its
`setSlices` method, denoting the "auto" slicing behaviour that is usable by
settting the "slices=auto" parameter on rest requests. When using the High Level
Rest Client, however, we send the 0 value as an integer, which is then rejected
as invalid by `AbstractBulkByScrollRequest#parseSlices`. Instead of making
parsing of the rest request more lenient, this PR opts for changing the
RequestConverter logic in the client to translate 0 values to "auto" on the rest
requests.

Closes #53044
2020-03-06 15:38:04 +01:00
William Brafford d145b5536f
Serialize NodesInfoRequest as a set of strings (#53140) (#53202)
For Node Info to be pluggable, NodesInfoRequest must be able to carry
arbitrary strings. This commit reworks the internals of that class to
use a set rather than hard-coded boolean fields.

NodesInfoRequest defaults to specifying all values. We test for
this behavior as we refactor and use random testing for the
various combinations of metrics.

Add backwards compatibility for transport requests.
2020-03-06 09:07:49 -05:00
James Rodewig 9bb9f63364 [DOCS] Note that `trim` filter doesn't change offsets (#53220)
The [word delimiter graph token filter docs][0] note that the `trim`
filter changes the length of tokens without changing their offsets.

This explicitly mentions that in the `trim` filter docs.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html
2020-03-06 07:32:35 -05:00
Dimitris Athanasiou 9abf537527
[7.x][ML] Improve DF analytics audits and logging (#53179) (#53218)
Adds audits for when the job starts reindexing, loading data,
analyzing, writing results. Also adds some info logging.

Backport of #53179
2020-03-06 13:47:27 +02:00
Marios Trivyzas 7ddbda4c20
Check for query cancellation during rewrite (#53166) (#53203)
With ExitableDirectoryReader in place, check for query cancellation
during QueryPhase#preProcess where the query rewriting takes place.

Follows: #52822

(cherry picked from commit 0d38626d8e6e9e2620a7a446b617a2ac42852461)
2020-03-06 11:04:01 +01:00
Alan Woodward c204137451 Deprecate BoolQueryBuilder's mustNot field (#53125)
The bool query builder in elasticsearch accepts both must_not and mustNot
fields. Given that leniency is abhorrent and must be eschewed, we should deprecate
the latter as it doesn't fit with the style of parameters elsewhere in the DSL.
2020-03-06 09:11:34 +00:00
Henning Andersen 2e924e4a83 Fix ClusterDisruptionIT.testAckedIndexing (#53169)
Use assertBusy when doing reroute after bridged disruption,
since it can return non-acked if a node is marked faulty
by follower check after disruption ended.

Closes #53064
2020-03-06 08:56:55 +01:00
Jake Landis 6a5d9195aa
[7.x] Ensure only plugin REST tests are run for plugins (#5318… (#53196)
This commit fixes ensures that for external builds
(e.g. plugin development) that the REST tests that are
copied are properly filtered to only include the API
by default.

The code prior to this change resulted in including both
the API and tests since the copy.include resulted as an
empty list by default since the stream is empty unless
explicitly configured.

related #52114
fixes #53183
2020-03-05 17:41:17 -06:00
Nhat Nguyen 5476a49833 Revert "upgrade to lucene-snapshot-fa75139efea (#53150) (#53151)"
This reverts commit 058113aa42.
2020-03-05 17:33:00 -05:00
Nhat Nguyen d456e8ffca Revert "Mute InternalEngineTests.testVersionOnPrimaryWithConcurrentRefresh"
This reverts commit 66788afa67.
2020-03-05 17:32:18 -05:00
Nhat Nguyen e9e209ae58 Revert "Mute InternalEngineTests.testRandomOperations"
This reverts commit d1cc2e68d5.
2020-03-05 17:32:11 -05:00
Nhat Nguyen dc78cc6131 Revert "Mute InternalEngineTests.testForceMergeWithSoftDeletesRetentionAndRecoverySource"
This reverts commit da8aac9e66.
2020-03-05 17:31:56 -05:00
Nhat Nguyen f11ae5fd14 Revert "Mute GatewayMetaStatePersistedStateTests.testDataOnlyNodePersistence"
This reverts commit 4452addf10.
2020-03-05 17:31:38 -05:00
Nik Everett f32e4583d1
Add `allowed_warnings` to yaml tests (backport of #53139) (#53173)
When we test backwards compatibility we often end up in a situation
where we *sometimes* get a warning, and sometimes don't. Like, we won't
get the warning if we're testing against an older version, but we will
in a newer one. Or we won't get the warning if the request randomly
lands on a node with an old version of the code. But we wouldn't if it
randomed into a node with newer code.

This adds `allowed_warnings` to our yaml test runner for those cases:
warnings declared this way are "allowed" but not "required".

Blocks #52959

Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
2020-03-05 17:11:54 -05:00
James Baiera 4452addf10 Mute GatewayMetaStatePersistedStateTests.testDataOnlyNodePersistence 2020-03-05 16:44:03 -05:00
James Baiera da8aac9e66 Mute InternalEngineTests.testForceMergeWithSoftDeletesRetentionAndRecoverySource 2020-03-05 15:55:50 -05:00
James Baiera d1cc2e68d5 Mute InternalEngineTests.testRandomOperations 2020-03-05 15:09:47 -05:00
James Baiera 66788afa67 Mute InternalEngineTests.testVersionOnPrimaryWithConcurrentRefresh 2020-03-05 15:09:47 -05:00
James Rodewig 4bc6d2dbec [DOCS] Correct link for Lucene StopFilter 2020-03-05 14:52:25 -05:00
Mayya Sharipova 7e2a9f58ee
script_score query errors on negative scores (#53133)
7.5 and 7.6 had a regression that allowed for
script_score queries to have negative scores.
We have corrected this regression in #52478.
This is an addition to #52478 that adds
a test and release notes.
2020-03-05 14:23:39 -05:00
Mark Vieira 8851fb2a08
Upgrade to Gradle 6.2.2 (#53136) 2020-03-05 09:27:03 -08:00
David Turner c2627aa22f Clarify futher the order for a rolling upgrade (#52964)
Expands the "master-ineligible then master-eligible" sentence into a list and
specifies that within these subsets the order doesn't matter.
2020-03-05 15:28:59 +00:00
István Zoltán Szabó 58ce56f6c8 [DOCS] Makes the naming convention of the DFA response objects coherent (#53172) 2020-03-05 16:26:57 +01:00
Benjamin Trent af0b1c2860
[ML] Fix minor race condition in dataframe analytics _stop (#53029) (#53164)
Tests have been periodically failing due to a race condition on checking a recently `STOPPED` task's state. The `.ml-state` index is not created until the task has already been transitioned to `STARTED`. This allows the `_start` API call to return. But, if a user (or test) immediately attempts to `_stop` that job, the job could stop and the task removed BEFORE the `.ml-state|stats` indices are created/updated.

This change moves towards the task cleaning up itself in its main execution thread. `stop` flips the flag of the task to `isStopping` and now we check `isStopping` at every necessary method. Allowing the task to gracefully stop.

closes #53007
2020-03-05 09:59:18 -05:00
Benjamin Trent 181ee3ae0b
[ML] specifying missing_field_value value and using it instead of empty_string (#53108) (#53165)
For analytics, we need a consistent way of indicating when a value is missing. Inheriting from anomaly detection, analysis sent `""` when a field is missing. This works fine with numbers, but the underlying analytics process actually treats `""` as a category in categorical values. 

Consequently, you end up with this situation in the resulting model
```
{
              "frequency_encoding" : {
                "field" : "RainToday",
                "feature_name" : "RainToday_frequency",
                "frequency_map" : {
                  "" : 0.009844409027270245,
                  "No" : 0.6472019970785184,
                  "Yes" : 0.6472019970785184
                }
              }
            }
```
For inference this is a problem, because inference will treat missing values as `null`. And thus not include them on the infer call against the model.

This PR takes advantage of our new `missing_field_value` option and supplies `\0` as the value.
2020-03-05 09:50:52 -05:00
István Zoltán Szabó 48707ec55a [DOCS] Expands GET DFA stat API docs with response objects. (#53107) 2020-03-05 15:31:55 +01:00
Aleksandr Maus 2dc872f052
EQL: Add HLRC for EQL stats (#53043) (#53148) 2020-03-05 09:20:38 -05:00
Adrien Grand 360ac1997f Fix test failures with the new `constant_keyword` field. (#53153)
This test failed because YAML tests randomly install an index template
that updates the default number of shards to 2.

Closes #53131
2020-03-05 14:29:13 +01:00
Marios Trivyzas 487d442760
Implement Exitable DirectoryReader (#52822) (#53162)
Implement an Exitable DirectoryReader that wraps the original
DirectoryReader so that when a search task is cancelled the
DirectoryReaders also stop their work fast. This is usuful for
expensive operations like wilcard/prefix queries where the
DirectoryReaders can spend lots of time and consume resources,
as previously their work wouldn't stop even though the original
search task was cancelled (e.g. because of timeout or dropped client
connection).

(cherry picked from commit 67acaf61f33bc5f54e26541514d07e375c202e03)
2020-03-05 14:17:31 +01:00
Nik Everett 28df7ae5ed
Support multiple metrics in `top_metrics` agg (backport of #52965) (#53163)
This adds support for returning multiple metrics to the `top_metrics`
agg. It looks like:
```
POST /test/_search?filter_path=aggregations
{
  "aggs": {
    "tm": {
      "top_metrics": {
        "metrics": [
          {"field": "v"},
          {"field": "m"}
        ],
        "sort": {"s": "desc"}
      }
    }
  }
}
```
2020-03-05 08:12:01 -05:00
David Roberts 01504df876 [TEST] Force close failed job before skipping test (#53128)
The assumption added in #52631 skips a problematic test
if it fails to create the required conditions for the
scenario it is supposed to be testing.  (This happens
very rarely.)

However, before skipping the test it needs to remove the
failed job it has created because the standard test
cleanup code treats failed jobs as fatal errors.

Closes #52608
2020-03-05 10:52:41 +00:00
Armin Braun 204c366a4e
Upgrade GCS SDK to 1.104.0 (#52839) (#53152)
Upgrading the GCS SDK to the most recent version.
Adjusting (i.e. improving) the REST mock accordingly.
This should significantly boost performance by pulling in
https://github.com/googleapis/java-core/issues/86 in some cases.
2020-03-05 11:18:18 +01:00
Alan Woodward 3cd4b97618 Remove UnknownNamedObjectException (#53105)
This was originally thrown from NamedXContentRegistry#parseNamedObject() but
that method now throws a NamedObjectNotFoundException, so this is unused.
2020-03-05 10:06:59 +00:00
James Rodewig e46bb54c7b
[DOCS] Document `any` keyword in EQL syntax (#52821) (#53157)
Adds documentation for the `any` keyword to the EQL syntax docs.

Includes:

* Definition of an event category and its relationship to the event
   category field.
* Example matching all event categories using `any` keyword
* Example using `any` with `where true`
2020-03-05 05:02:47 -05:00
Ignacio Vera 058113aa42
upgrade to lucene-snapshot-fa75139efea (#53150) (#53151) 2020-03-05 10:04:05 +01:00
Tanguy Leroux 52d4807f8d
Mute GoogleCloudStorageBlobStoreRepositoryTests on jdk8 (#53119)
Tests in GoogleCloudStorageBlobStoreRepositoryTests are known 
to be flaky on JDK 8 (#51446, #52430 ) and we suspect a JDK 
bug (https://bugs.openjdk.java.net/browse/JDK-8180754) that triggers
 some assertion on the server side logic that emulates the Google 
Cloud Storage service.

Sadly we were not able to reproduce the failures, even when using 
the same OS (Debian 9, Ubuntu 16.04) and JDK (Oracle Corporation 
1.8.0_241 [Java HotSpot(TM) 64-Bit Server VM 25.241-b07]) of 
almost all the test failures on CI. While we spent some time fixing 
code (#51933, #52431) to circumvent the JDK bug they are still flaky 
on JDK-8. This commit mute these tests for JDK-8 only.

Close ##52906
2020-03-05 09:18:05 +01:00
Lisa Cawley 859c6441b3 [DOCS] Adds PKI delegation.enabled example (#53030) 2020-03-04 14:59:45 -08:00
Nik Everett 302980e0c4
Remove some ceremony in agg parsing (#53078) (#53117)
With #50871 aggrgations should now be parsed directly by an
`ObjectParser` or `ConstructingObjectParser` without the need for the
ceremonial `parse` method. This removes 9 of those `parse` methods and
parses the aggregation directly from their `ObjectParser`.
2020-03-04 13:06:41 -05:00
Ross Wolf a5e82d7fd6
EQL: Add explicit 'any where ...' handling (#52526) 2020-03-04 10:11:03 -07:00
Tim Brooks f68917160e
Fix RemoteConnectionManager size() method (#52823)
Currently the remote connection manager will delegate the size() call to
the underlying cluster connection manager. This introduces the
possibility that call will return 1 before the nodeConnection method has
been triggered to add the connection to the remote connection list. This
can cause issues, as the ensureConnected method checks the connection
managers size and executes synchronously if the size is > 0. This leads
to a potential cluster not connected exception while we are still
waiting for the connection opened callback to be triggered.

This commit fixes this issue by using the remote connection manager's
size to report the connection manager's size.

Fixes #52029.
2020-03-04 09:53:22 -07:00
Yannick Welsch 8ab74fea58
[7.x] Add 7.6.2 as version (#53114) 2020-03-04 10:39:09 -06:00