Commit Graph

51498 Commits

Author SHA1 Message Date
Armin Braun 70b378cd1b
Upgrade GCS Dependency to 1.106.0 (#54092) (#54112)
* Upgrade GCS Dependency to 1.106.0 (#54092)

Upgrading GCS Dep + related dependencies as it seems some more retry bugs were fixed between .104 and .106
2020-03-25 19:05:01 +01:00
James Baiera b84c74cf70
Update the HDFS version used by HDFS Repo (#53693) (#54125) 2020-03-25 14:01:29 -04:00
Martijn Laarman 077bf52acc transform.cat should live in the cat namespace. (#54196)
* transform.cat should live in the cat namespace.

Similarly to to ml cat API's also living in the `cat` namespace.

Clients treat the `cat` namespace differently then other API's (return
types, content types). This introduces an exception to this rule.

* rename the specification file as well

(cherry picked from commit 0a98904b1a73a30bbaebc32bd16a238c8d03c329)
2020-03-25 18:16:01 +01:00
Bogdan Pintea 77da9dd040 Add version 7.8.0
Add version 7.8.0
2020-03-25 18:10:30 +01:00
James Rodewig 30a32040d3
[DOCS] EQL: Document `substring` function (#53867)
Adds documentation for the EQL `substring` function.

Supporting changes:

* Creates a new "EQL function reference" page
* Updates the title of the "EQL syntax reference" page for consistency
* Adds a brief "Functions" section to the EQL syntax docs
* Updates EQL limitations docs to state that only array functions are
  unsupported
2020-03-25 12:23:59 -04:00
Martijn Laarman 4c36b5daee Document known features on rest-api-spec tests (#52916)
* Document known features on rest-api-spec tests

Features dictate wheter a `rest-api-spec` test runner can execute a
test. This PR documents all the know features in the java implementation
of the runner.

* Apply suggestions from code review

Co-Authored-By: Luca Cavanna <javanna@users.noreply.github.com>

Co-authored-by: Luca Cavanna <javanna@users.noreply.github.com>
(cherry picked from commit fbe173723d0ad7ebb920cda855bce3fb758b04a6)
2020-03-25 16:29:53 +01:00
Mark Vieira 7728ccd920
Encore consistent compile options across all projects (#54120)
(cherry picked from commit ddd068a7e92dc140774598664efdc15155ab05c2)
2020-03-25 08:24:21 -07:00
jimczi 04fabead14 Fix small typo in SearchService#executeQueryPhase
Relates #54044
2020-03-25 16:16:30 +01:00
Armin Braun 32fa90c9ba
Fix ClusterHealthIT.testHealthOnMasterFailover (#54170) (#54177)
We can run into a state where there's no more events to wait for temporarily
but the cluster still isn't green. I added the wait for green flag to the request
so the assertion for green cluster health below doesn't fail.

Closes #53457
2020-03-25 14:33:31 +01:00
Nik Everett 16e4bd50e2 Add breaking change note for #53669 2020-03-25 09:31:14 -04:00
James Rodewig 74051d68a8
[DOCS] Reformat `keyword_marker` token filter (#54076)
Makes the following changes to the `keyword_marker` token filter docs:

* Rewrites description and adds Lucene link
* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer and filter example
2020-03-25 09:26:06 -04:00
Dimitris Athanasiou ba09a778dc
[7.x][ML] Unmute classification cardinality integ test (#54165) (#54173)
Adjusts test to work for new cardinality limit.

Backport of #54165
2020-03-25 15:00:34 +02:00
James Rodewig 2fdf6b2f96
[DOCS] Document missing data types for node stats API's response parameters (#53475)
Documents missing data types for several response parameters returned
by the node stats API.

Also adds several missing human-readable parameters returned by the API.
2020-03-25 08:42:58 -04:00
Benjamin Trent ef05a4f416
[ML] relaxing parameters on stratified split test (#54127) (#54168)
Relaxing the error rate a bit on two of the tests.
Ran 1000s of times locally and never had a failure after these changes. 

closes https://github.com/elastic/elasticsearch/issues/54122
2020-03-25 08:06:15 -04:00
Rory Hunter b066cc587d Cleanup after DockerTests suite completes (#54164)
Closes #54101. The Docker tests weren't cleanup up the created container
after the last test completed. Now they are.
2020-03-25 11:58:44 +00:00
Tanguy Leroux 3a3930c7ec
Mute TooManyJobsIT.testCloseFailedJob on 7.x (#54163)
Relates #54162
2020-03-25 12:44:41 +01:00
Armin Braun 0a70250201
Fix BlobStoreIncrementalityIT Assertion (#54149) (#54155)
We are using this assertion for identical shard snapshots
for situations where `snapshot1` wasn't the first snapshot
for the tested shard. Hence, we can't assume that it will
not share any files with previous snapshots.
This showed up in failing tests when `snapshot1` was equivalent
to a previous snapshot because no documents were deleted from the
repo randomly in the failing test but even if documents are deleted
there is no guarantee that no files will be shared.

=> I removed this assertion since its immaterial for what is tested
here anyway.

Closes #54034
2020-03-25 12:13:52 +01:00
Armin Braun 4271963462
Revert "Use Azure Bulk Deletes in Azure Repository (#53919)" (#54089) (#54111)
This reverts commit 23cccf088810b8416ed278571352393cc2de9523.
Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet.

Closes #54080
2020-03-25 12:13:25 +01:00
jimczi e380a5a8c3 Fix off-by one error in TransportSearchActionTests
Closes #54156
2020-03-25 11:41:02 +01:00
Tanguy Leroux b6e482295b
Mute TransportSearchActionTests.testShouldPreFilterSearchShards (#54158)
Relates #53873
Relates #54156
2020-03-25 11:25:06 +01:00
Tanguy Leroux 4a2db4651e
Mute ReadActionsTests (#54153)
Relates #53340
2020-03-25 10:35:58 +01:00
Ioannis Kakavas 7d4ae7d982
Upgrade Tika to 1.24 (#54130) (#54150)
Also updates commons-compress to 1.19, pdfbox to 2.0.19 and
POI to 4.1.2. Adds a compile dependency to commons-math3
3.6.1 and SparseBitSet 1.2
2020-03-25 11:03:26 +02:00
Jim Ferenczi 3b4751bdb7
Avoid I/O operations when rewriting shard search request (#54044) (#54139)
This commit ensures that we rewrite the shard request with a short-lived can_match searcher.
This is required for frozen indices since the high level rewrite is now performed on a network thread where we don't want to perform I/O.

Closes #53985
2020-03-25 09:02:36 +01:00
Jason Tedor 381d7586e4
Introduce formal role for remote cluster client (#54138)
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.

Relates #53924
2020-03-24 21:59:43 -04:00
Mark Vieira 75e80e207a
Upgrade Gradle Enterprise plugin to 3.2 (#54137) 2020-03-24 18:24:41 -07:00
Oliver Gupte 96f0c668a8
[APM] Allow kibana to collect APM telemetry in background task (#52917) (#54106)
* Required for elastic/kibana#50757.
Allows the kibana user to collect APM telemetry in a background task.

* removed unnecessary priviledges on `.ml-anomalies-*` for the `kibana_system` reserved role
2020-03-24 18:11:19 -07:00
Dan Hermann cb73de2eb7
Unmute data stream YML tests (#54100) 2020-03-24 19:11:13 -05:00
Mark Vieira c4d4af3123
Enable auto restart on debug elasticsearch run configuration 2020-03-24 16:35:45 -07:00
David Roberts 7667004b20
[ML] Add a model memory estimation endpoint for anomaly detection (#54129)
A new endpoint for estimating anomaly detection job
model memory requirements:

POST _ml/anomaly_detectors/estimate_model_memory

Backport of #53507
2020-03-24 22:55:11 +00:00
Ioannis Kakavas 7c0123d6f3
Add SAML IdP plugin for internal use (#54046) (#54124)
This change merges the "feature-internal-idp" branch into Elasticsearch.

This introduces a small identity-provider plugin as a child of the x-pack module.
This allows ES to act as a SAML IdP, for users who are authenticated against the
Elasticsearch cluster.

This feature is intended for internal use within Elastic Cloud environments
and is not supported for any other use case. It falls under an enterprise license tier.

The IdP is disabled by default.

Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Co-authored-by: Tim Vernum <tim.vernum@elastic.co>
2020-03-25 09:45:13 +11:00
Gordon Brown 82e041442e
Add version guards around Transform hidden index settings (#54036)
This commit ensures that the hidden index settings are only applied to the
Transform index templates when the cluster can support those settings.

Also unmutes the tests which were failing due to the previous behavior.
2020-03-24 15:52:56 -06:00
Jim Ferenczi 55f2e8bff0 [DOCS] Add 7.6.2 release notes (#53720)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-24 22:42:25 +01:00
Henning Andersen 7ce7aff66e Reindex negative TimeValue fix (#54057)
Reindex would use timeValueNanos(System.nanoTime()). The intended use
for TimeValue is as a duration, not as absolute time. In particular,
this could result in negative TimeValue's, being unsupported in #53913.
Modified to use the bare long nano-second value.
2020-03-24 22:29:09 +01:00
Przemko Robakowski fc498f625a
[7.x] Add validation for component templates (#54023) (#54118)
* Add validation for component templates (#54023)

* Add validation for component templates

This change adds validation to make sure that settings and mappings are correct in
component template. It's done the same way as in index templates - code is reused.

Reletes to #53101

* Fix checkstyle violation

* Update server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Adjusted to 7.7

* unused import fixed

* npe fixeD

* change exception type

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2020-03-24 22:20:34 +01:00
Jim Ferenczi 025857d949 Fix ShardSearchRequest cache key (#54071)
This commit ensures that we don't use the non-deterministic canReturnNullResponseIfMatchNoDocs
boolean in the cache key of the ShardSearchRequest. The value of this boolean has no influence
on the cacheability of the request.

Closes #32827
2020-03-24 22:15:52 +01:00
Ross Wolf 627ca03c72
EQL: Remove parser handling for functions (#54028)
* EQL: Remove parser handling for functions
* EQL: Comment out array functions in queries-unsupported.eql
2020-03-24 14:03:02 -06:00
Przemko Robakowski 5594d57727
/_cat/shards support path stats (#53461) (#54119)
* _cat/shards support path stats

* fix some style case

* fix some style case

* fix rest-api-spec cat.shards error

* fix rest-api-spec cat.shards bwc error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: weizijun <weizijun1989@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-24 20:46:05 +01:00
Costin Leau 68f74cf593
EQL: Fix custom scripting for functions (#53935) (#54114)
Improve separation of scripting between EQL and SQL by delegating common
methods to QL. The context detection is determined based on the package
to avoid having repetitive class hierarchies.
The Painless whitelists have been improved so that the declaring class
is used instead of the inherited one.

Relates #53688

(cherry picked from commit 6d46033e736c64ac9255c5d6964600d2a931430a)

EQL: Add Substring function with Python semantics (#53688)

Does not reuse substring from SQL due to the difference in semantics and
the accepted arguments.
Currently it is missing full integration tests as, due to the usage of
scripting, requires an actual integration test against a proper cluster
(and likely its own QA project).

(cherry picked from commit f58680bad33d5ce4139157a69a4d9f5f286bc3c4)
2020-03-24 20:54:19 +02:00
Tim Brooks b21b7fb09b
Allow proxy mode server name to be updated (#54107)
Currently there is a bug where the proxy strategy will not be rebuilt if
the server_name is dynamically updated. This commit fixes this issue.
2020-03-24 11:54:23 -06:00
David Turner 21afc788f8 Reduce log level for pipeline failure (#54097)
Today we log `failed to execute pipeline for a bulk request` at `ERROR` level
if an attempt to run an ingest pipeline fails. A failure here is commonly due
to an `EsRejectedExecutionException`. We also feed such failures back to the
client and record the rejection in the threadpool statistics.

In line with #51459 there is no need to log failures within actions so noisily
and with such urgency. It is better to leave it up to the client to react
accordingly. Typically an `EsRejectedExecutionException` should result in the
client backing off and retrying, so a failure here is not normally fatal enough
to justify an `ERROR` log at all.

This commit reduces the log level for this message to `DEBUG`.
2020-03-24 17:41:35 +00:00
markharwood 6a60f85bba
Wildcard field - add normalizer support (#53851) (#54109)
Backport support for normalisation to wildcard field

Closes #53603
2020-03-24 17:37:47 +00:00
Dimitris Athanasiou c141c1dd89
[7.x][ML] Stratified cross validation split for classification (#54087) (#54104)
As classification now works for multiple classes, randomly
picking training/test data frame rows is not good enough.
This commit introduces a stratified cross validation splitter
that maintains the proportion of the each class in the dataset
in the sample that is used for training the model.

Backport of #54087
2020-03-24 18:47:36 +02:00
Yannick Welsch e006d1f6cf Use special XContent registry for node tool (#54050)
Fixes an issue where the elasticsearch-node command-line tools would not work correctly
because PersistentTasksCustomMetaData contains named XContent from plugins. This PR
makes it so that the parsing for all custom metadata is skipped, even if the core system would
know how to handle it.

Closes #53549
2020-03-24 17:40:51 +01:00
Lisa Cawley 88b1b2f36f [DOCS] Adds transform security privileges (#53908) 2020-03-24 09:35:45 -07:00
Nik Everett 42be39177b
Remove ceremony declaring aggs (backport of #53990) (#54099)
This removes some more ceremony when declaring agg parsers. You no
longer need a static `parse` method, instead you can just make the
`PARSER` public in most cases.

There are still a few aggs with the `parse` method, but those `parse`
methods are a little more complex to untangle.
2020-03-24 12:29:52 -04:00
Tim Brooks caefa78513
Align remote info api with new settings (#54102)
Currently the remote info api has added a number of possible fields
(proxy, num_socket_connections, etc) that are available in proxy mode.
These fields are not aligned with what the settings are named. This
commit modifies this API to align with the settings.
2020-03-24 10:27:24 -06:00
Christoph Büscher 1c1730facd
Mask wildcard query special characters on keyword queries (#53127) (#53512)
Wildcard queries on keyword fields get normalized, however this normalization
step should exclude the two special characters * and ? in order to keep the
wildcard query itself intact.

Closes #46300
2020-03-24 17:22:29 +01:00
Luca Cavanna 6b457abbd3 Async search: prevent users from overriding pre_filter_shard_size (#54088)
Submit async search forces pre_filter_shard_size for the underlying search that it creates.
With this commit we also prevent users from overriding such default as part of request validation.
2020-03-24 17:06:04 +01:00
Luca Cavanna 3c67762f1b Async search response: output start and expiration time as time fields (#54084)
This commits makes start_time and expiration_time time fields, so that their date variant will be printed out when human readable output is requested.
2020-03-24 17:05:56 +01:00
Jim Ferenczi 0330bef409 Improve async search's tasks cancellation (#53799)
This commit adds an explicit cancellation of the search task if
the initial async search submit task is cancelled (connection closed by the user).
This was previously done through the cancellation of the parent task but we don't
handle grand-children cancellation yet so we have to manually cancel the search task
in order to ensure that shard actions are cancelled too.
This change can be considered as a workaround until #50990 is fixed.
2020-03-24 15:51:10 +01:00