Commit Graph

52769 Commits

Author SHA1 Message Date
David Roberts 9fdf1722e6
[TEST] Fix more allowed warnings for composable template rename (#57398)
Should have been done in #57232
2020-05-31 18:14:48 +01:00
Armin Braun 59570eaa7d
Fix Local Translog Recovery not Updating Safe Commit in Edge Case (#57350) (#57380)
In case the local checkpoint in the latest commit is less
than the last processed local checkpoint we would recover
0 ops and hence not commit again.
This would lead to the logic in `IndexShard#recoverLocallyUpToGlobalCheckpoint`
not seeing the latest local checkpoint when it reload the safe commit from the store
and thus cause inefficient recoveries because the recoveries would work from a
lower than possible local checkpoint.

Closes #57010
2020-05-30 09:28:50 +02:00
Nik Everett d6a3704932
Fold some of sig_terms into terms (backport of #57361) (#57386)
This merges the global-ordinals-based implementation for
`significant_terms` into the global-ordinals-based implementation of
`terms`, removing a bunch of copy and pasted code that is subtly
different across the two implementations and replacing it with an
explicit `ResultStrategy` with nice stuff like Javadoc.

The actual behavior is mostly unchanged, though I was able to remove a
redundant copy of bytes representing the string from the result
construction phase of `significant_terms`.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-29 22:51:11 -04:00
Nik Everett f52e779806
Fix casting of scaled_float in sorts (#57207) (#57385)
Previously we'd get a `ClassCastException` when you tried to use
`numeric_type` on `scaled_float`. Oops! This cleans up the CCE and moves
some code around so the casting actually works.
2020-05-29 18:06:04 -04:00
Nik Everett 07c76f2894
Update date_histogram docs (#56922) (#57387)
* Make it more clear that you can use `month` or `1M`.
* Explain rounding rules
* Consistently use "time zone" instead of "timezone". It looks like both
  are right but I see "time zone" much more. And the parameter in
  elasticsearch is `time_zone` so we may as well line up.

Closes #56760

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-05-29 17:40:40 -04:00
Nik Everett d5e86d7c4d
Small cleanups for terms aggregator (#57315) (#57381)
This includes a few small cleanups for the `TermsAggregatorFactory`:

1. Removes an unused `DeprecationLogger`
2. Moves the members to right above the ctor.
3. Merges some all of the heuristics for picking `SubAggCollectionMode`
   into a single method.
2020-05-29 16:59:35 -04:00
James Rodewig ebe433343f [DOCS] Correct link for ESMS redirect 2020-05-29 16:43:04 -04:00
Tomasz Elendt a7c36c8af5 Support multiple tokens on LHS in stemmer_override rules (#56113) (#56484)
This commit adds support for rules with multiple tokens on LHS, also
known as "contraction rules", into stemmer override token
filter. Contraction rules are handy into translating multiple
inflected words into the same root form. One side effect of this change is
that it brings stemmer override rules format closer to synonym rules
format so that it makes it easier to translate one into another.

This change also makes stemmer override rules parser more strict so
that it should catch more errors which were previously accepted.

Closes #56113
2020-05-29 22:34:31 +02:00
Nik Everett 4263c25b2f
Save memory when histogram agg is not on top (backport of #57277) (#57377)
This saves some memory when the `histogram` aggregation is not a top
level aggregation by dropping `asMultiBucketAggregator` in favor of
natively implementing multi-bucket storage in the aggregator. For the
most part this just uses the `LongKeyedBucketOrds` that we built the
first time we did this.
2020-05-29 15:07:37 -04:00
Nik Everett b15a304155 Expain when `gradle run` is ready
It isn't obvious.

Closes #57097
2020-05-29 14:56:50 -04:00
Benjamin Trent 34f1e0b6bb
[7.x] [ML] mark forecasts for force closed/failed jobs as failed (#57143) (#57374)
* [ML] mark forecasts for force closed/failed jobs as failed (#57143)

forecasts that are still running should be marked as failed/finished in the following scenarios:

- Job is force closed
- Job is re-assigned to another node.

Forecasts are not "resilient". Their execution does not continue after a node failure. Consequently, forecasts marked as STARTED or SCHEDULED should be flagged as failed. These forecasts can then be deleted.

Additionally, force closing a job kills the native task directly. This means that if a forecast was running, it is not allowed to complete and could still have the status of `STARTED` in the index.

relates to https://github.com/elastic/elasticsearch/issues/56419
2020-05-29 14:48:10 -04:00
Benjamin Trent 35d5126cea
[7.x] [ML] adds new for_export flag to GET _ml/inference API (#57351) (#57368)
* [ML] adds new for_export flag to GET _ml/inference API (#57351)

Adds a new boolean flag, `for_export` to the `GET _ml/inference/<model_id>` API.

This flag is useful for moving models between clusters.
2020-05-29 14:01:08 -04:00
James Rodewig 7dbf5baf60 [DOCS] Remove Elastic Stack Monitoring Service docs (#57371) 2020-05-29 13:16:57 -04:00
Gordon Brown 1d5e476256
Add expand_wildcards to _cat/indices and _cat/aliases docs (#56964)
This commit adds the `expand_wildcards` parameter documentation to the
`_cat/indices` and `_cat/aliases` docs, as those APIs now support
`expand_wildcards`. Additionally, clarifies the `expand_wildcards` docs with
respect to hidden indices.
2020-05-29 11:01:53 -06:00
Benjamin Trent 15aba60c02
[7.x] Add new circuitbreaker plugin and refactor CircuitBreakerService (#55695) (#57359)
* Add new circuitbreaker plugin and refactor CircuitBreakerService (#55695)

This commit lays the ground work for plugins supplying their own circuit breakers.

It adds a new interface: `CircuitBreakerPlugin`.

This interface provides methods for providing custom child CircuitBreaker objects. There are also facilities for allowing dynamic settings for the custom breakers.

With the refactor, circuit breakers are no longer replaced on setting changes. Instead, the two mutable settings themselves are `volatile`. Plugins that want to use their custom circuit breaker should keep a reference of their constructed breaker.
2020-05-29 12:13:46 -04:00
Mayya Sharipova aebb78bf5c Run sort optimization when from+size>0 (#57250) 2020-05-29 11:30:35 -04:00
Benjamin Trent c8374dc9f3
[ML] add max_model_memory parameter to forecast request (#57254) (#57355)
This adds a max_model_memory setting to forecast requests. 
This setting can take a string value that is formatted according to byte sizes (i.e. "50mb", "150mb").

The default value is `20mb`.

There is a HARD limit at `500mb` which will throw an error if used.

If the limit is larger than 40% the anomaly job's configured model limit, the forecast limit is reduced to be strictly lower than that value. This reduction is logged and audited.

related native change: https://github.com/elastic/ml-cpp/pull/1238

closes: https://github.com/elastic/elasticsearch/issues/56420
2020-05-29 11:16:08 -04:00
Armin Braun e4fd78f866
Remove Overly Strict Safety Mechnism in Shard Snapshot Logic (#57227) (#57362)
Unfortunately, we cannot have a safety mechnism like this where we throw whenever we find unreadable data in a shard.
This breaks in the case of an older ES version (without shard generations enabled) having failed to snapshot a shard snapshot after writing some data to its path and having finalized it for example.
Another example of where we can't support this check is the test I added, if we snapshot an index with a name that already exists in the repository and more shards than the existing index, fail doing that and then retry snapshotting it we will also see unexpected data in the path.

We could technically do deeper inspections on the unexpected data but I don't think it's worth it really. In the end if we are unable to read the data here it's broken anyway. By moving to a new `index-` blob in the shard directory I don't see us ever
corrupting existing data and since we (by virtue of moving to an empty generation) won't do any incremental work on top of potentially corrupt data we also do not risk creating broken snapshots going forward.
=> Just logging a warning in this very unlikely case is the best we can do I think
2020-05-29 16:41:57 +02:00
Marios Trivyzas b2651323fd
SQL: Implement TIME_PARSE function for parsing strings into TIME values (#55223) (#57342)
Implement TIME_PARSE(<time_str>, <pattern_str>) function
which allows to parse a time string according to the specified
pattern into a time object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Closes #54963

Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: Patrick Jiang(白泽) <patrickjiang0530@gmail.com>

(cherry picked from commit 1fe1188d449cad7d0782a202372edc52a4014135)
2020-05-29 15:48:37 +02:00
Dan Hermann 6b0d707671
[7.x] Do not report negative values for swap sizes (#57353) 2020-05-29 08:11:47 -05:00
Henning Andersen 8427d677e9
Reindex and friends fail nicely when max_docs < slices (#54901) (#57348)
When the parameter `max_docs` is less than `slices` in update_by_query,
delete_by_query or reindex API, `max_docs ` is set to 0 and we throw an
action_request_validation_exception with confused error message:
"maxDocs should be greater than 0...".
This change checks that whether `max_docs` is less than `slices` and
throw an illegal_argument_exception with clear message.

Relates to #52786.

Co-authored-by: bellengao <gbl_long@163.com>
2020-05-29 14:30:14 +02:00
Martijn van Groningen d8928b3f48
fixed allowed warnings in yaml test 2020-05-29 13:37:49 +02:00
Martijn van Groningen 04ef39da77
Change cluster info actions to be able to resolve data streams. (#57343)
Backport of #56878 to 7.x branch.

With this change the following APIs will be able to resolve data streams:
get index, get mappings and ilm explain APIs.

Relates to #53100
2020-05-29 12:17:53 +02:00
Dimitris Athanasiou 322f953060
[7.x][ML] Anomaly detection jobs should allow missing values for geo fields (#57300) (#57338)
Allows geo fields (`geo_point`, `geo_shape`) to have missing values.
Fixes a bug where such missing values would result in an error.

Closes #57299

Backport of #57300
2020-05-29 13:06:16 +03:00
Armin Braun be6fa72432
Fix GCS Mock Behavior for Missing Bucket (#57283) (#57310)
* Fix GCS Mock Behavior for Missing Bucket

We were throwing a 500 instead of a 404 for a missing bucket.
This would make yaml tests needlessly wait for multiple seconds, retrying
the 500 response with backoff, in the test checking behavior for missing buckets.
2020-05-29 10:01:20 +02:00
Ignacio Vera 75868ea915
Catch InputCoercionException thrown by Jackson parser (#57287) (#57330)
Jackson 2.10 library has added a new type of error that is thrown when a numeric value is out 
of range. This error should be catch and handle properly in case the flag ignore_malformed 
has been set to true.
2020-05-29 09:47:47 +02:00
Russ Cam 2a9073d4c1 Deprecate local param in get_mapping.json (#57265)
Relates: elastic/elasticsearch#55014

This commit deprecates the local param in get_mapping.json.
This parameter is a no-op and field mappings are always retrieved locally.

(cherry picked from commit 0b041cccd894f01d723fb2979f70c1cf279700a6)
2020-05-29 12:25:41 +10:00
Adam Locke 11ae062877
[7.6] [DOCS] Explain flood stage watermark (#57321)
* [7.7] [DOCS] Explain flood stage watermark

* Adding id to fix cross-linking issue.
2020-05-28 19:29:32 -04:00
Nik Everett b9fe10866e
Make global ords terms simpler to understand (backport of #57241) (#57311)
When the `terms` enum operates on non-numeric data it can collect it via
global ordinals. It actually has two separate collection strategies for,
one "dense" and one "remapping". Each of *those* strategies has two
"iteration" strategies that it uses to build buckets, depending on
whether or not we need buckets with `0` docs in them. Previously this
was done with several `null` checks and never really explained. This
change replaces those checks with two `CollectionStrategy` classes which
have good stuff like documentation.
2020-05-28 16:52:35 -04:00
Benjamin Trent 24d605e41e
[ML] fixing GET _ml/inference so size param is respected (#57303) (#57308)
`size` was previously ignored when grabbing full trained model configs. 

closes https://github.com/elastic/elasticsearch/issues/57298
2020-05-28 15:45:26 -04:00
Ryan Ernst 2aeb6ce7ad
Omit translog bwc test before 6.3.0 with default distro (#57266)
The new translog bwc test checks a corruption case before 6.3.0.
However, it needs to restart the old node to reproduce, which does not
currently work given how testclusters works when plugins are installed.
As a workaround, this commit omits creating bwc tests before 6.3.0 only
when the default distribution is used.

fixes #57252
2020-05-28 12:38:30 -07:00
Julie Tibshirani 36e5670f76 Unmute FullClusterRestartIT#testSearch. 2020-05-28 10:55:17 -07:00
Julie Tibshirani 10e1dc199d Revert "Remove unused logic from FieldNamesFieldMapper. (#56834)"
This reverts commit 343fb699a4.
2020-05-28 10:54:10 -07:00
Rene Groeschke 7a4a6360b7
Fix up-to-date checks for precommit related tasks (#57203) (#57291)
* Fix up-to-date checks for precommit related tasks

- Do not use lambdas for doFirst / doLast action declarations as this is not supported by gradle up-to-date check
- Use marker output folder for dependencies license task to make task incremental build compliant

* Tweak formatting
2020-05-28 17:12:17 +02:00
James Rodewig 47024ccb3c
[DOCS] Add verify snapshot repository API docs (#57253) (#57288) 2020-05-28 09:48:04 -04:00
James Rodewig 9277ce6f9e
[DOCS] Adds a doc value field example to Mapper Size docs (#57257) (#57280)
Changes:

* Updates snippet to include doc value field example.
* Fixes a broken link to inline script settings.
2020-05-28 09:26:49 -04:00
Martijn van Groningen 225ccd1cfa
Ensure template exists when creating data stream (#57275)
Backporting #56888 to 7.x branch.

Limit the creation of data streams only for namespaces that have a composable template with a data stream definition.

This way we ensure that mappings/settings have been specified and will be used at data stream creation and data stream rollover.

Also remove `timestamp_field` parameter from create data stream request and
let the create data stream api resolve the timestamp field
from the data stream definition snippet inside a composable template.

Relates to #53100
2020-05-28 15:08:25 +02:00
Rene Groeschke 51158a2d8b
Fix deprecation warning in ThirdpartyAuditTask (#57123) (#57224) 2020-05-28 14:41:42 +02:00
Marios Trivyzas fdac9e99fa
SQL: Fix unecessary evaluation for CASE/IIF (#57159) (#57262)
Previously, `CASE` and `IIF` when translated to painless scripts
(used in GROUP BY, HAVING, WHERE) a custom `caseFunction`
registered in the `InternalSqlScriptUtils` was used. This function
received and array of arbitrary length:
```[condition1, result1, condition2, result2, ... elseResult]```

Painless doesn't know of the context and therefore is evaluating
all conditions and results before invoking the `caseFunction` on them.
As a consequence, erroneous result expressions (i.e. division by 0)
where always evaluated despite of the guarding condition.

Replace the `caseFunction` with painless `<cond> ? <res1> : <res2>`
expressions to properly guard the result expressions and only evaluate
the one for which its guarding condition evaluates to true (or of course
the elseResult).

As a bonus, this approach includes performance benefits since we avoid
unnecessary evaluations of both conditions and result expressions.

Fixes: #49672
(cherry picked from commit 9584b345d89f797bfb658212b928b9812804f02f)
2020-05-28 11:30:14 +02:00
István Zoltán Szabó e1cab4feb4 [DOCS] Puts a link into the loss_function variable description (#56678) 2020-05-28 09:46:11 +02:00
Tim Vernum 408250dcc4
Fix smtp.ssl.trust setting for watcher email (#57268)
The ssl.trust setting for Watcher provides a list of hostnames that
should be automatically trusted for SSL hostname verification. It was
accidentally broken when we added the full ssl.* settings for email
notifications (see #45272)

This commit corrects this, so the setting is once again respected,
as long as none of the other ssl settings are configured for email
notifications.

Resolves: #52153
Backport of: #56090
2020-05-28 17:34:13 +10:00
Ryan Ernst 1b59e9ab22
Move test error reporting to java plugin (#57259)
This commit moves the global hook for reporting failed test cases to the
ElasticsearchJavaPlugin. It should always be applied for all java
projects since the Test class is what emits the failures logged.
2020-05-27 17:41:04 -07:00
Ryan Ernst 97353297dc
Move gradle version check to global build info plugin (#57255)
The gradle version check currently exists in BuildPlugin. However, there
is no reason to check this within every project. Instead, this commit
moves the check to the global build info, which is only applied to the
root project. Additionally, this commit removes the check from buildSrc
because it is not really necessary. The check exists really just for
external plugin authors since we use the gradle wrapper for our own
build.
2020-05-27 17:38:02 -07:00
Ryan Ernst fdb8573413
Convert remaining compilerJavaHome reference 2020-05-27 17:04:04 -07:00
Ryan Ernst beb1d0c338
Remove compiler java version flag (#57237)
This commit removes the compiler.java setting from the build. It was
originally added when Gradle was far behind support for the latest jdk,
but is no longer applicable as we don't have any need to update the
supported compile version before gradle supports the newer version. Note
that the runtime version changing support still exists here, this only
ensures we use the same jdk to compile as we use to run gradle.
2020-05-27 16:33:38 -07:00
Ryan Ernst 3ead0a183b
Upgrade bundled jdk to 14.0.1 (#57233) 2020-05-27 15:08:12 -07:00
David Roberts d139a79ef6
[7.x][ML] Fix monitoring if orphaned anomaly detector persistent tasks exist (#57240)
Since #51888 the ML job stats endpoint has returned entries for
jobs that have a persistent task but not job config. Such
orphaned tasks caused monitoring to fail.

This change ignores any such corrupt jobs for monitoring purposes.

Backport of #57235
2020-05-27 22:59:11 +01:00
Dan Hermann 2738998ebb
Limit _cat/indices test to versions with fix (#57244) (#57256) 2020-05-27 16:57:24 -05:00
James Baiera 3b73ce3112
Fix enrich coordinator to reject documents instead of deadlocking (#56247) (#57179)
This PR removes the blocking call to insert ingest documents into a queue in the
coordinator. It replaces it with an offer call which will throw a rejection exception
in the event that the queue is full. This prevents deadlocks of the write threads
when the queue fills to capacity and there are more than one enrich processors
in a pipeline.
2020-05-27 15:32:13 -04:00
Gordon Brown 2f561084f0
Mute FullClusterRestartIT.testSearch (#57249)
This test fails reliably on upgrades from 6.0.0 and 6.0.1. See #57245 for details.
2020-05-27 13:15:05 -06:00