Commit Graph

51612 Commits

Author SHA1 Message Date
James Rodewig c16b1edae0 [DOCS] EQL: Fix whitespace in `stringContains` docs 2020-04-27 15:53:59 -04:00
Julie Tibshirani 4bfd65a375 Remove TODO around aggregating on _index.
The _index field can in fact be used in aggregations.
2020-04-27 12:48:20 -07:00
Ryan Ernst 70b499b7aa
Simplify java home verification (#55635)
* Simplify java home verification

At one time, all uses of java home were found through the getJavaHome
utility method on BuildPlugin. However, that was changed many
refactorings ago, but the complex support for registering a java home
version needed that fails at configuration time still exists. The only
remaining use of grabbing java home is within bwc tests, and must be at
runtime since that is when we have the checkout and know what version is
needed.

This commit consolidates the java home finding method into a utility
unassociated with BuildPlugin.

* fix checkstyle

* address feedback
2020-04-27 12:43:32 -07:00
Tal Levy 6ba5148ead
Add geo_shape support for the geo_centroid aggregation (#55602) (#55819)
this commit leverages the new geo_shape doc values
to register a new geo_centroid aggregator that works
on geo_shape field.
2020-04-27 12:16:10 -07:00
James Rodewig 8df5cff9c1 [DOCS] Correct stemmer token filters anchor 2020-04-27 14:57:59 -04:00
James Rodewig 5b8a18c756 [DOCS] Correct stemmer token filter anchor 2020-04-27 14:51:51 -04:00
Ioannis Kakavas ca5d677130
Mute-55816 (#55818)
See #55816
2020-04-27 21:26:02 +03:00
Hendrik Muhs 4b93f17b24 [Transform] improve TransformRestTestCase robustness (#55786)
handles/retries temporary SearchPhaseExecutionErrors

fixes #54810
2020-04-27 17:17:53 +02:00
Jake Landis 6f392cf5b9
[7.x] json spec - add description for searchable snapshots (#55746) (#55809) 2020-04-27 10:08:09 -05:00
Mark Tozzi 22a98ec279
Aggregation support for Value Scripts that change types (#54830) (#55752) 2020-04-27 09:57:05 -04:00
Jake Landis 7b4bacebb5
[7.x] fix the schema validation for scripts_painless_context (#55738) (#55751) 2020-04-27 08:39:56 -05:00
Jim Ferenczi b5916ac455 Ignore closed exception on refresh pending location listener (#55799)
This newly added listener should catch closed exceptions when
accessing the internal engine.

Closes #55792
2020-04-27 15:06:35 +02:00
Dimitris Athanasiou abab4c4d4f
[7.x][ML] Do not fail DFA task when it's stopped whilst reindexing (#55797) (#55800)
Adding to #55659, we missed another way we could set the task to
failed due to task cancellation. CI revealed that we might also
get a `SearchPhaseExecutionException` whose cause is a
`TaskCancelledException`. That exception is not wrapped so
unwrapping it will not return the underlying `TaskCancelledException`.
Thus to be complete in catching this, we also need to check the
error's cause.

Closes #55068

Backport of #55797
2020-04-27 16:03:57 +03:00
Dimitris Athanasiou 7f100c1196
[7.x][ML] Allow analytics process define its own progress phases (#55763) (#55791)
This is a continuation from #55580.

Now that we're parsing phase progresses from the analytics process
we change `ProgressTracker` to allow for custom phases between
the `loading_data` and `writing_results` phases. Each `DataFrameAnalysis`
may declare its own phases.

This commit sets things in place for the analytics process to start
reporting different phases per analysis type. However, this is
still preserving existing behaviour as all analyses currently
declare a single `analyzing` phase.

Backport of #55763
2020-04-27 13:30:05 +03:00
Armin Braun fe9904fbea
More Efficient Blobstore Metdata IO (#55777) (#55788)
No need to copy all these bytes multiple times, especially not when
writing a multiple MB global cluster state snapshot through this method.
2020-04-27 11:48:53 +02:00
Ioannis Kakavas d56f25acb4
Validate hashing algorithm in users tool (#55628) (#55734)
This change adds validation when running the users tool so that
if Elasticsearch is expected to run in a JVM that is configured to
be in FIPS 140 mode and the password hashing algorithm is not
compliant, we would throw an error.
Users tool uses the configuration from the node and this validation
would also happen upon node startup but users might be added in the
file realm before the node is started and we would have the
opportunity to notify the user of this misconfiguration.
The changes in #55544 make this much less probable to happen in 8
since the default algorithm will be compliant but this change can
act as a fallback in anycase and makes for a better user experience.
2020-04-27 12:23:41 +03:00
Ioannis Kakavas 38b55f06ba
Fix concurrent refresh of tokens (#55114) (#55733)
Our handling for concurrent refresh of access tokens suffered from
a race condition where:

1. Thread A has just finished with updating the existing token
document, but hasn't stored the new tokens in a new document
yet
2. Thread B attempts to refresh the same token and since the
original token document is marked as refreshed, it decrypts and
gets the new access token and refresh token and returns that to
the caller of the API.
3. The caller attempts to use the newly refreshed access token
immediately and gets an authentication error since thread A still
hasn't finished writing the document.

This commit changes the behavior so that Thread B, would first try
to do a Get request for the token document where it expects that
the access token it decrypted is stored(with exponential backoff )
and will not respond until it can verify that it reads it in the
tokens index. That ensures that we only ever return tokens in a
response if they are already valid and can be used immediately

It also adjusts TokenAuthIntegTests
to test authenticating with the tokens each thread receives,
which would fail without the fix.

Resolves: #54289
2020-04-27 12:23:17 +03:00
Adrien Grand 0753d9a35c
Exists queries to MatchNoneQueryBuilder when the field is unmapped (#55785)
Co-authored-by: Sivagurunathan Velayutham <sivadeva.93@gmail.com>

Closes #54062
2020-04-27 11:06:50 +02:00
Armin Braun 4403b69048
Fix NPE in Partial Snapshot Without Global State (#55776) (#55783)
We make sure to filter shard generations for indices that are missing
from the metadata when finalizing a partial snapshot (from concurrent index deletion)
but we failed to account for the case where we manually build a fake metadata instance
for snapshots without the global state.
Fixed this by handling missing indices by skipping, same way we do it for filtering the
shard generations.

Relates #50234
2020-04-27 10:07:09 +02:00
Nhat Nguyen 1a3f9e5a07 Return true for can_match on idle search shards (#55428)
With this change, we will always return true for can_match requests on
idle search shards; otherwise, some shards will never get refreshed if
all search requests perform the can_match phase (i.e., total shards >
pre_filter_shard_size).

Relates #27500
Relates #50043
2020-04-26 22:21:42 -04:00
David Roberts 3ba44a5af8
[ML] Adding failed_category_count to model_size_stats (#55761)
The failed_category_count statistic records the number of times
categorization wanted to create a new category but couldn't
because the job had reached its model_memory_limit.

Backport of #55716
2020-04-25 10:36:49 +01:00
Aleksandr Maus ad54cca823
EQL: implement math functions: add, divide, module, multiply, subtract (#55137) (#55737)
* EQL: implement math functions: add, divide, module, multiply, subtract
2020-04-24 15:52:27 -04:00
James Rodewig c1b0548db0
[DOCS] Document EQL search REST API (#52384) 2020-04-24 15:36:01 -04:00
James Rodewig 5981412bf7
[DOCS] EQL: Document `stringContains` function (#54968) 2020-04-24 15:09:05 -04:00
Nick Knize b0e8a8a4d1
[Backport] Refactor Spatial Field Mappers (#55696)
This commit refactors all spatial Field Mappers to a common
AbstractGeometryFieldMapper that implements shared parameter functionality
(e.g., ignore_malformed, ignore_z_value) and provides a common framework for
overriding type parsing, and building in xpack. Common shape functionality is
implemented in a new AbstractShapeGeometryFieldMapper that is reused and
overridden in GeoShapeFieldMapper, GeoShapeFieldMapperWithDocValues,
LegacyGeoShapeFieldMapper, and ShapeFieldMapper. This abstraction provides a
reusable foundation for adding new xpack features; such as coordinate reference
system support.
2020-04-24 14:05:16 -05:00
James Rodewig e4ebe55d04
[DOCS] EQL: Document `cidrMatch` function (#54216) (#55739) 2020-04-24 14:01:11 -04:00
Mark Tozzi 87b4979c24
[7.x] Make ValuesSourceRegistry immutable after initilization #55493 (#55697) 2020-04-24 13:33:38 -04:00
Jason Tedor 22a8b60187
Reduce code duplication in CCR non-compliance tests
This commit removes some code duplication in the CCR non-compliance
tests by refactoring an assertion method so that it can be used in both
tests that are present there.
2020-04-24 13:24:56 -04:00
Tanguy Leroux 41ddbd4188 Allow to prewarm the cache for searchable snapshot shards (#55322)
Relates #50999
2020-04-24 18:03:34 +02:00
Mark Vieira 75f8d8bfd3
Remove duplicate debug run configuration definition 2020-04-24 08:54:43 -07:00
Rory Hunter c2c14e8875
Use official checksums to verify Tini (#55717)
Firstly, backport the use of tini as the Docker entrypoint. This was supposed
to have been done following #50277, but was missed. It isn't a direct backport
as this branch will continue using root as the initial Docker user.

Secondly, backport #55491 to use the official checksums when downloading tini.
2020-04-24 16:47:24 +01:00
James Rodewig e0a8adb5b2
[DOCS] Reformat `stemmer` token filter (#55693)
Makes the following changes to the `stemmer` token filter docs:

* Adds detailed analyze example
* Rewrites parameter definitions
* Adds custom analyzer example
* Adds a `language` value for the `estonian` stemmer
* Reorders the `language` values to show recommended algorithms first,
  followed by other values alphabetically
2020-04-24 11:25:01 -04:00
James Rodewig 96285b90c1
[DOCS] Add stemming concept docs (#55156)
Adds conceptual documentation for stemming, including:

* An overview of why stemming is helpful in search
* Algorithmic vs. dictionary stemming
* Token filters used to control stemming, such as `stemmer_override`, `keyword_marker`, and `conditional`
2020-04-24 11:01:28 -04:00
Christoph Büscher f95a741ad3
[Docs] Fix fuzziness example in match-query.asciidoc (#55715)
The example looks the same as in the previous section although it should use the
"fuzziness" parameter. This seems to be okay on 6.8 and master and was probably
only forgotten to port to 7.x branches.
2020-04-24 16:21:40 +02:00
Dimitris Athanasiou 210b7f1b76
[7.x][ML] Remove parsing of old progress format in DF Analytics (#55711) (#55720)
Since #55580 we've introduced a new format for parsing progress
from the data frame analytics process. As the process is now
writing out progress in this new way, we can remove the parsing
of the old format.

Backport of #55711
2020-04-24 16:50:56 +03:00
David Turner aa9a2bce37 Avoid accidental contiguous read (#55713)
If we choose to read from two random positions that are 1024 bytes apart then
this counts as a contiguous read for stats purposes, failing this test. This
commit ensures that we always perform a non-contiguous read.
2020-04-24 11:50:31 +01:00
Rory Hunter 6d2a5378a0 Rename docker context artifacts to satisfy release-manager (#55692)
Our release tool expects artifacts to have a certain naming convention. Rename
the Docker context artifacts to match this convention.
2020-04-24 10:48:18 +01:00
David Turner de30550aea Relax elapsed time stats assertion (#55710)
`SearchableSnapshotDirectoryStatsTests#testCachedBytesReadsAndWrites` asserts
that each write takes one clock tick, but we now permit concurrent reads and
writes so each write might take longer. This commit relaxes the assertion to
match.

Closes #55707
2020-04-24 10:21:08 +01:00
Przemysław Witek c89917c799
Register DFA jobs on putAnalytics rather than via a separate method (#55458) (#55708) 2020-04-24 10:59:32 +02:00
Dimitris Athanasiou b8379872a7
[7.x][ML] Logs error when DFA task is set to failed (#55545) (#55668)
Also unmutes the integ test that stops and restarts
an outlier detection job with the hope of learning more
of the failure in #55068.

Backport of #55545

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-24 11:06:07 +03:00
Jim Ferenczi 0a6c74b7d3 AsyncSearchMaintenanceService should stop when closing a node (#55651)
This change turns the AsyncSearchMaintenanceService into an
AbstractLifecycleComponent and ensures that the service is
stopped when a node is closing.

Closes #55646
2020-04-24 09:38:40 +02:00
Hendrik Muhs b213209f0c [Rollup] improve stopping tests (#55666)
improve tests related to stopping using a client that answers and can be
synchronized with the test thread in order to test special situations

relates #55011
2020-04-24 08:48:36 +02:00
Jay Modi 30f8c326fe
Test: fix SSLReloadDuringStartupIntegTests (#55637)
This commit fixes reproducible test failures with the
SSLReloadDuringStartupIntegTests on the 7.x branch. The failures only
occur on 7.x due to the existence of the transport client and its usage
in our test infrastructure. This change removes the randomized usage of
transport clients when retrieving a client from a node in the internal
cluster. Transport clients do not support the reloading of files for
TLS configuration changes but if we build one from the nodes settings
and attempt to use it after the files have been changed, the client
will not know about the changes and the TLS connection will fail.

Closes #55524
2020-04-23 21:36:43 -06:00
Ryan Ernst 97c4b64fb1
Add isAllowed license utility (#55424) (#55700)
License state is currently made up of boolean methods that check whether
a particular feature is allowed by the current license state. Each new
feature must copy/past boiler plate code. While that has gotten easier
with utilities like isAllowedByLicense, this is still more cumbersome
than should be necessary. This commit adds a general purpose isAllowed
method which takes a new Feature enum, where each value of the enum
defines the minimum license mode and whether the license must be active
to be allowed. Only security features are converted in this PR, in order
to keep the commit size relatively small. The rest of the features will
be converted in a followup.
2020-04-23 16:28:28 -07:00
Zachary Tong 715c90bf7d Aggs must specify a `field` or `script` (or both) (#52226)
This adds a validation to VSParserHelper to ensure that a field or
script or both are specified by the user.  This is technically
required today already, but throws an exception much deeper
in the agg framework and has a very unintuitive error for the user
(as well as eating more resources instead of failing early)
2020-04-23 19:23:41 -04:00
jimczi c857adf603 Fix AsyncSearchTaskTests#testWithFetchFailures
Fix usage of a possible invalid random range [1, 0].

Relates #55688
2020-04-24 00:45:17 +02:00
Jim Ferenczi 31d1727698 Fix (de)serialization of async search failures (#55688)
The (de)serialization code of the async search response
cannot handle exceptions that extend ElasticsearchException (e.g. ScriptException).
This commit fixes this bug by serializing the error with the more generic
StreamInput#writeException.
2020-04-24 00:44:43 +02:00
Igor Motov 8c7ef2417f
Make AsyncSearchIndexService reusable (#55598)
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Supersedes #55119
Relates to #49638
2020-04-23 18:02:17 -04:00
Nick Knize 96a02089c2
Refactor GeoShape DocValues in spatial xpack (#55691)
This commit refactors geo_shape doc values, fielddata, and utility classes from
the single mapper package in x-pack spatial plugin to a package structure that
is consistent with the server module.
2020-04-23 15:32:23 -05:00
David Roberts 46be9959a0
[ML] Audit when unassigned datafeeds are stopped (#55667)
Previously audit messages were indexed when datafeeds that were
assigned to a node were stopped, but not datafeeds that were
unassigned at the time they were stopped.

This change adds auditing for the unassigned case.

Backport of #55656
2020-04-23 20:46:35 +01:00