Commit Graph

50369 Commits

Author SHA1 Message Date
Hendrik Muhs 5912895838 [Transform] wait for transform templates in Rest integration t… (#53330)
add transform templates to the list of templates to be installed before
executing tests
2020-03-10 13:22:12 +01:00
James Rodewig d503bf9a45 [DOCS] Make token graph diagrams consistent (#53331)
Updates the SVG for a token graph to make the layout consistent with
other graphs. This means moving the text directly above the edge lines.
Previously, the text was above the edge line.
2020-03-10 06:53:28 -04:00
Hendrik Muhs 696aa4ddaf
[7.x][Transform] add support for script in group_by (#53167) (#53324)
add the possibility to base the group_by on the output of a script.

closes #43152
backport #53167
2020-03-10 11:12:58 +01:00
Alan Woodward 5c861cfe6e Upgrade to final lucene 8.5.0 snapshot (#53293)
Lucene 8.5.0 release candidates are imminent. This commit upgrades master to use
the latest snapshot to check that there are no last-minute bugs or regressions.
2020-03-10 09:32:59 +00:00
James Rodewig 5e3df18d56 [DOCS] Adds Beats tip to EQL search docs (#53292)
Adds a tip admonition to the basic example in the EQL search docs.

This tip lets users know they can set up a Beat to automatically
index data in ES, rather than manually indexing using the bulk or index
APIs.
2020-03-10 05:16:18 -04:00
James Rodewig 641eb69520 [DOCS] Document `nodes` cluster stats (#52813)
Documents the `nodes` response parameters returned by the
`_cluster/stats` API.

Also adds collapsible attributes for the `indices` and `nodes`
sections.
2020-03-10 05:05:05 -04:00
Cauê Marcondes b68d7b1c33
giving kibana user privileges to create custom link index (#53221) (#53278) 2020-03-10 09:50:38 +01:00
Henning Andersen a4d481f2bb ILM Freeze step retry when not acknowledged (#53287)
A freeze operation can partially fail in multiple places, including the
close verification step. This left the index in an unfrozen but
partially closed state. Now throw an exception to retry the freeze step
instead.
2020-03-10 08:03:39 +01:00
Gordon Brown 1cb0a4399d
Fix Get Alias API handling of hidden indices with visible aliases (#53147)
This commit changes the Get Aliases API to include hidden indices by
default - this is slightly different from other APIs, but is necessary
to make this API work intuitively.
2020-03-09 16:16:29 -06:00
William Brafford 2bb4b96a7f
Serialize NodesStatsRequest as set of strings (#53235) (#53313)
* Add unit tests before refactoring

* Convert boolean fields to set of strings

In order to make nodes stats plugins pluggable, we need to make the
NodesStatsRequest class capable of carrying a flexible list of metrics
rather than a fixed list of boolean flags. This commit changes the
internal storage of the class without changing its serialization.

* Change serialization of NodesStatsRequest

* Set up BWC before merging

* Singularize enum name
2020-03-09 18:13:29 -04:00
Przemko Robakowski f075d70cf8
[7.x] Avoid race condition in ILMHistorySotre (#53039) (#53094)
* Avoid race condition in ILMHistorySotre (#53039)

* Avoid race condition in ILMHistorySotre

This change modifies ILMHistoryStore to always apply correct settings and mappings,
even if template is deleted and not yet recreated. This ensures that ILM history index
is correctly managed by ILM and also fixes flaky history tests that were prone to
triggenring this race.

This commit also refactors and simplifies ILM history tests.

Closes #50353 and #52853

* Review comment

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* fixed tests

* backport #53306

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-09 22:24:15 +01:00
Jason Tedor 1860c57147
Deprecate the listener thread pool (#53266)
The listener thread pool is being removed from use in the server
codebase. This commit deprecates configuring the listener thread pool.
2020-03-09 16:56:01 -04:00
Bogdan Pintea 62c8ac9993
SQL: transfer version compatibility decision to the server (#53082) (#53302)
This commit adds a new request object field, "version", containing the version of the requesting client. This parameter is now accepted - and for certain clients required - by the server and the request is validated against it. Currently server's and client's versions still need to be equal in order for the request to be accepted. Relaxing this check is going to be part of future work. 

On the clients' side, the only check remaining is to ensure that the peer server is supporting version backwards compatibility (i.e. is on, or newer than a certain release).

(cherry picked from commit a8f413a20fb023bec83af0de1211a2936a7f558c)
2020-03-09 21:16:57 +01:00
Aleksandr Maus d064846416
EQL: Test infrastructure improvements (#53253) (#53297)
Update CommonEqlRestTestCase code to simplify making changes as requested.
Update EqlActionIT to simplify the test code as requested.
Replace Jackson parser with XContent in EqlActionIT.
Whitelist more EQL tests specs that are now supported.
2020-03-09 14:11:54 -04:00
Julie Tibshirani c33afea9fb Small corrections to stored_fields docs. (#53247)
* Fix a reference to the 'field' option.
* Remove claim about detecting script fields.
* Specify that object fields will just be ignored.
2020-03-09 10:59:17 -07:00
Ross Wolf f5f922c6f6
EQL: Add IsNull/IsNotNull checks (#52791)
* EQL: Add IsNull/IsNotNull checks
* EQL: Simplify IsNull/IsNotNull optimization
* EQL: Split string tests over multiple lines
2020-03-09 10:41:04 -06:00
Dominik Riemer f78bd00f57 [DOCS] Update StreamPipes link in Integrations docs (#53087)
Updates the link to [StreamPipes][0] after transition to Apache Software Foundation.

[0]: https://streampipes.apache.org
2020-03-09 11:07:57 -04:00
David Turner b20f86e450 Clarify JavaDoc for DiscoveryNodes#resolveNodes (#53277)
Closes #52887
2020-03-09 14:44:29 +00:00
Greg Back 64ea94ac8e [DOCS] Update ES Python client doc links (#53092)
Replaces `rtfd.org` links with `readthedocs.org`, which is now the official URL.
2020-03-09 10:34:38 -04:00
Jason Tedor 8ad0080a59
Fork CCR checkpoint listeners on CCR thread pool (#53265)
This commit moves the global checkpoint listeners used in CCR to the CCR
thread pool. This removes the last use of the listener thread pool in
the codebase.
2020-03-09 08:56:30 -04:00
Romain Gonord 00657901ec [DOCS] Fix "Wait for Snapshot" link in ILM actions reference (#53269)
Corrects an anchor for the `Wait for Snapshot` action, which previously linked to the `Delete` action.
2020-03-09 08:40:36 -04:00
Anton Dollmaier 35c8226419 [DOCS] Fix parameter formatting for GeoHash grid agg docs (#53032)
Adds missing colon (`:`) to the parameter definition list.
2020-03-09 08:16:40 -04:00
David Turner 52ff341814 Deprecate passing settings in restore requests (#53268)
Today we accept a `settings` field in snapshot restore requests, but this field
is not used. This commit deprecates it.
2020-03-09 12:01:07 +00:00
Christoph Büscher 2fd954a3b7 Fix potential NPE in FuzzyTermsEnum (#53231)
Under certain circumstances SpanMultiTermQueryWrapper uses
SpanBooleanQueryRewriteWithMaxClause as its rewrite method, which in turn tries
to get a TermsEnum from the wrapped MultiTermQuery currently using a `null`
AttributeSource. While queries TermsQuery or subclasses of AutomatonQuery ignore
this argument, FuzzyQuery uses it to create a FuzzyTermsEnum which triggers an
NPE when the AttributeSource is not provided. This PR fixes this by supplying an
empty AttributeSource instead of a `null` value.

Closes #52894
2020-03-09 12:59:08 +01:00
James Rodewig 28cb4a167d
[DOCS] Reformat `word_delimiter_graph` token filter (#53170) (#53272)
Makes the following changes to the `word_delimiter_graph` token filter
docs:

* Updates the Lucene experimental admonition.
* Updates description
* Adds analyze snippet
* Adds custom analyzer and custom filter snippets
* Reorganizes and updates parameter list
* Expands and updates section re: differences between `word_delimiter`
  and `word_delimiter_graph`
2020-03-09 06:45:44 -04:00
István Zoltán Szabó aafc0409a9 [DOCS] Makes the description clearer on how to use aggregations in an anomaly detection job (#53103)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-03-09 09:49:59 +01:00
Martijn van Groningen 7775ddbc9c
Verify watch_count before a test starts and not after a test.
This check was added as part of: 0f2d26bdca

Checking this before the test starts makes more sense, because
the watches index has then also be removed.

Relates to #53177
2020-03-09 07:45:44 +01:00
Jason Tedor 5e96d3e59a
Use given executor for global checkpoint listener (#53260)
Today when notifying a global checkpoint listener, we use the listener
thread pool. This commit turns this inside out so that the global
checkpoint listener must provide an executor on which to notify the
listener.
2020-03-08 13:51:05 -04:00
Jason Tedor 79b67eb3ba
Drop action future that forks on listener executor (#53261)
This commit drops the dispatching listenable action future that forks to
the listener thread pool. This was previously used in the transport
client but is no longer used.
2020-03-08 12:36:09 -04:00
Jason Tedor a0b235888f
Avoid self-suppression on grouped action listener (#53262)
It can be that a failure is repeated to a grouped action listener. For
example, if the same exception such as a connect transport exception, is
the cause of repeated failures. Previously we were unconditionally
self-suppressing the exception into the first exception, but
self-supressing is not allowed. Thus, we would throw an exception and
the grouped action listener would never complete. This commit addresses
this by guarding against self-suppression.
2020-03-08 08:59:57 -04:00
Jason Tedor c5738ae312
Notify refresh listeners on the calling thread (#53259)
Today we notify refresh listeners by forking to the listener thread pool
and then serially notifying listeners on a thread there. Refreshes are
expensive though, so the expectation is that we are executing refreshes
on threads that can afford an expensive operation (e.g., not a network
thread) and as such, executing listeners that we expect to be cheap aon
the calling thread is okay. This commit removes the forking of notifying
refresh listeners to run directly on the calling thread that executed a
refresh.
2020-03-07 13:12:40 -05:00
Lisa Cawley 341417613e
[7.x][DOCS] Adds common definitions for security settings (#51017) (#53242)
Co-Authored-By: Tim Vernum <tim@adjective.org>
2020-03-06 16:28:54 -08:00
Gordon Brown ff9b8bda63
Implement hidden aliases (#52547)
This commit introduces hidden aliases. These are similar to hidden
indices, in that they are not visible by default, unless explicitly
specified by name or by indicating that hidden indices/aliases are
desired.

The new alias property, `is_hidden` is implemented similarly to
`is_write_index`, except that it must be consistent across all indices
with a given alias - that is, all indices with a given alias must
specify the alias as either hidden, or all specify it as non-hidden,
either explicitly or by omitting the `is_hidden` property.
2020-03-06 16:02:38 -07:00
Nik Everett 7c9641ef9d
Simplify BucketedSort (#53199) (#53240)
Our lovely `BitArray` compactly stores "flags", lazilly growing its
underlying storage. It is super useful when you need to store one bit of
data for a zillion buckets or a documents or something. Usefully, it
defaults to `false`. But there is a wrinkle! If you ask it whether or
not a bit is set but it hasn't grown its underlying storage array
"around" that index then it'll throw an `ArrayIndexOutOfBoundsException`.
The per-document use cases tend to show up in order and don't tend to
mind this too much. But the use case in aggregations, the per-bucket use
case, does. Because buckets are collected out of order all the time.

This changes `BitArray` so it'll return `false` if the index is too big
for the underlying storage. After all, that index *can't* have been set
or else we would have grown the underlying array. Logically, I believe
this makes sense. And it makes my life easy. At the cost of three lines.

*but* this adds an extra test to every call to `get`. I think this is
likely ok because it is "very close" to an array index lookup that
already runs the same test. So I *think* it'll end up merged with the
array bounds check.
2020-03-06 15:27:51 -05:00
Ross Wolf d6813cb348
EQL: Convert wildcards to LIKE in analyzer (#51901)
* EQL: Convert wildcard comparisons to Like
* EQL: Simplify wildcard handling, update tests
* EQL: Lint fixes for Optimizer.java
2020-03-06 13:13:07 -07:00
Mayya Sharipova f96ad5c32d Mute testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows 2020-03-06 12:48:05 -05:00
Jay Modi a81460dbf5
Make watch history indices hidden (#52974)
This commit updates the template used for watch history indices with
the hidden index setting so that new indices will be created as hidden.

Relates #50251
Backport of #52962
2020-03-06 09:47:03 -07:00
Adam Canady a88d0c7ca3 [Docs] Correct examples for * and + in regexp-syntax.asciidoc (#53210) 2020-03-06 17:17:32 +01:00
Mark Vieira 09a3f45880
Mute ClassificationIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-06 07:38:04 -08:00
James Baiera 01f00df5cd
Mute RegressionIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet 2020-03-06 07:37:57 -08:00
Benjamin Trent 85d7112e78
[ML] Fixing datafeed bwc tests (#52959)
Datafeed bwc tests have been muted for some time in the 7.x. This is because of date_histogram interval deprecation warnings.

This commit fixes the tests as must as possible while still handling deprecation warnings.
2020-03-06 10:27:21 -05:00
István Zoltán Szabó bf3dcd4229 [DOCS] Adds deleting flag to the GET job stats API docs (#53223) 2020-03-06 16:04:20 +01:00
Christoph Büscher 9e561c2921 Fix AbstractBulkByScrollRequest slices parameter via Rest (#53068)
Currently the AbstractBulkByScrollRequest accepts slice values of 0 via its
`setSlices` method, denoting the "auto" slicing behaviour that is usable by
settting the "slices=auto" parameter on rest requests. When using the High Level
Rest Client, however, we send the 0 value as an integer, which is then rejected
as invalid by `AbstractBulkByScrollRequest#parseSlices`. Instead of making
parsing of the rest request more lenient, this PR opts for changing the
RequestConverter logic in the client to translate 0 values to "auto" on the rest
requests.

Closes #53044
2020-03-06 15:38:04 +01:00
William Brafford d145b5536f
Serialize NodesInfoRequest as a set of strings (#53140) (#53202)
For Node Info to be pluggable, NodesInfoRequest must be able to carry
arbitrary strings. This commit reworks the internals of that class to
use a set rather than hard-coded boolean fields.

NodesInfoRequest defaults to specifying all values. We test for
this behavior as we refactor and use random testing for the
various combinations of metrics.

Add backwards compatibility for transport requests.
2020-03-06 09:07:49 -05:00
James Rodewig 9bb9f63364 [DOCS] Note that `trim` filter doesn't change offsets (#53220)
The [word delimiter graph token filter docs][0] note that the `trim`
filter changes the length of tokens without changing their offsets.

This explicitly mentions that in the `trim` filter docs.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html
2020-03-06 07:32:35 -05:00
Dimitris Athanasiou 9abf537527
[7.x][ML] Improve DF analytics audits and logging (#53179) (#53218)
Adds audits for when the job starts reindexing, loading data,
analyzing, writing results. Also adds some info logging.

Backport of #53179
2020-03-06 13:47:27 +02:00
Marios Trivyzas 7ddbda4c20
Check for query cancellation during rewrite (#53166) (#53203)
With ExitableDirectoryReader in place, check for query cancellation
during QueryPhase#preProcess where the query rewriting takes place.

Follows: #52822

(cherry picked from commit 0d38626d8e6e9e2620a7a446b617a2ac42852461)
2020-03-06 11:04:01 +01:00
Alan Woodward c204137451 Deprecate BoolQueryBuilder's mustNot field (#53125)
The bool query builder in elasticsearch accepts both must_not and mustNot
fields. Given that leniency is abhorrent and must be eschewed, we should deprecate
the latter as it doesn't fit with the style of parameters elsewhere in the DSL.
2020-03-06 09:11:34 +00:00
Henning Andersen 2e924e4a83 Fix ClusterDisruptionIT.testAckedIndexing (#53169)
Use assertBusy when doing reroute after bridged disruption,
since it can return non-acked if a node is marked faulty
by follower check after disruption ended.

Closes #53064
2020-03-06 08:56:55 +01:00
Jake Landis 6a5d9195aa
[7.x] Ensure only plugin REST tests are run for plugins (#5318… (#53196)
This commit fixes ensures that for external builds
(e.g. plugin development) that the REST tests that are
copied are properly filtered to only include the API
by default.

The code prior to this change resulted in including both
the API and tests since the copy.include resulted as an
empty list by default since the stream is empty unless
explicitly configured.

related #52114
fixes #53183
2020-03-05 17:41:17 -06:00