Commit Graph

53518 Commits

Author SHA1 Message Date
Nik Everett 6d2cab9437
Stop runtime script from emitting too many values (#61938) (#62186)
This prevent `keyword` valued runtime scripts from emitting too many
values or values that take up too much space. Without this you can put
allocate a ton of memory with the script by sticking it into a tight
loop. Painless has some protections against this but:
1. I don't want to rely on them out of sheer paranoia
2. They don't really kick in when the script uses callbacks like we do
   anyway.

Relates to #59332
2020-09-09 14:47:24 -04:00
Lisa Cawley 1eb4595a29 [DOCS] Removes inference from trained model API text (#62125) 2020-09-09 10:13:32 -07:00
Nik Everett 1104d65465
Fix bug with terms' min_doc_count (#62130) (#62177)
The `global_ordinals` implementation of `terms` had a bug when
`min_doc_count: 0` that'd cause sub-aggregations to have array index out
of bounds exceptions. Ooops. My fault. This fixes the bug by assigning
ordinals to those buckets.

Closes #62084
2020-09-09 13:04:51 -04:00
Christoph Büscher 885051fc14 Correct command for docs snippets test (#62182)
The command for running individual test seems outdated. Using `integTestRunner`
produces an error while `integTest` seems to work.
2020-09-09 18:58:43 +02:00
Rene Groeschke 7696b39807
Update gradle enterprise plugin to 3.4.1 (#62165) (#62178) 2020-09-09 18:30:34 +02:00
Dan Hermann eeeb355adf
Configurable output format for date processor (#61324) (#62175) 2020-09-09 11:11:02 -05:00
Benjamin Trent e181e24d48
[ML] only persist progress if it has changed (#62123) (#62180)
* [ML] only persist progress if it has changed

We already search for the previously stored progress document.

For optimization purposes, and to prevent restoring the same
progress after a failed analytics job is stopped,
this commit does an equality check between the previously stored progress and current progress
If the progress has changed, persistence continues as normal.
2020-09-09 12:04:09 -04:00
James Rodewig f1522fcafc
[DOCS] Fix range query admon for clarity (#62163) (#62171) 2020-09-09 10:37:43 -04:00
James Rodewig 6db8ca4113
[DOCS] Split delete index template API docs (#62074) (#62169)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Co-authored-by: Zaeem <zaeemarshad@users.noreply.github.com>
2020-09-09 10:13:16 -04:00
Dan Hermann 0b1e2172e1
[7.x] Preserve grok pattern ordering and add sort option (#61671) (#62162) 2020-09-09 08:53:11 -05:00
William Brafford 8d8a918bcf
Avoid extra whitespace in parsed JVM options (#57769) (#62138)
The windows service script does a little munging of the parsed JVM
option string, converting whitespaces to semicolons. We recently added
an optional Java 14 JDK flag to our system JVM flags. On earlier JDKs,
the windows service batch script would encounter a double whitespace
when this option was missing and convert it into double semicolons.
Double semicolons, in turn, don't work in the arguments to the windows
service command, and led to a lot of JVM options being dropped,
including "es.path.conf", which is required for startup.

This commit puts in a double defense. First, it removes any empty-string
options from the system options list in the Java Options Parser code.
Second, it munges out double semicolons if they do appear in the parsed
option output.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-09-09 09:51:12 -04:00
Benjamin Trent 1b9dc0172a
[ML] adding feature_name and node size validation for tree models (#62096) (#62161)
When a tree model is provided, it is possible that it is a stump.
Meaning, it only has one node with no splits
This implies that the tree has no features. In this case,
having zero feature_names is appropriate. In any other case,
this should be considered a validation failure.

This commit adds the validation if there is more than 1 node,
that the feature_names in the model are non-empty.

closes #60759
2020-09-09 08:50:25 -04:00
Armin Braun 6710104673
Fix Creating NOOP Tasks on SNAPSHOT Pool (#62152) (#62157)
Fixing a few spots where NOOP tasks on the snapshot pool were created needlessly.
Especially when it comes to mixed master+data nodes and concurrent snapshots these
hurt delete operation performance needlessly.
2020-09-09 14:05:17 +02:00
Luca Cavanna b680d3fb29 Async search: don't track fetch failures (#62111)
Fetch failures are currently tracked byy AsyncSearchTask like ordinary shard failures. Though they should be treated differently or they end up causing weird scenarios like total=num_shards and successful=num_shards as the query phase ran fine yet the failed count would reflect the number of shards where fetch failed.

Given that partial results only include aggs for now and are complete even if fetch fails, we can ignore fetch failures in async search, as they will be anyways included in the response. They are in fact either received as a failure when all shards fail during fetch, or as part of the final response when only some shards fail during fetch.
2020-09-09 13:44:07 +02:00
Luca Cavanna fbf0967e20 QueryPhaseResultConsumer to call notifyPartialReduce (#62083)
As part of #60275 QueryPhaseResultConsumer ended up calling SearchProgressListener#onPartialReduce directly instead of notifyPartialReduce. That means we don't catch exceptions that may occur while executing the progress listener callback.

This commit fixes the call and adds a test for this scenario.
2020-09-09 13:44:07 +02:00
Luca Cavanna ad83261348 Print out search request as part of async search task description (#62057)
Currently, the async search task is the task that will be running through the whole execution of an async search. While the submit async search task prints out the search as part of its description, async search task doesn't while it should.

With this commit we address that while also making sure that the description highlights that the task is originated from an async search.

Also, we streamline the way the description is printed out by SearchTask so that it does not get forgotten in the future.
2020-09-09 13:44:07 +02:00
Rory Hunter b7fd7cf154
Write deprecation logs to a data stream (#61966)
Backport of #58924.

Closes #46106. Introduce a mechanism for writing deprecation logs to a data stream
as well as to disk.
2020-09-09 12:16:28 +01:00
David Kyle a4fb501a33
[ML] Extra exceptions for renamed fields in mapping tests (#62151)
Add renamed 'maximum_number_trees' fields to exceptions
2020-09-09 12:03:42 +01:00
markharwood 5a48895065
Wildcard field bug fix for prefix + term queries. Wildcard syntax should not be supported. (#62085) (#62154)
Wildcard field bug fix for term and prefix queries.
We now escape any * or ? characters in the search string before delegating to the main wildcardQuery() method.

Closes #62081
2020-09-09 10:46:52 +01:00
Dimitris Athanasiou 6c1700c343
[7.x][ML] Outlier detection mapping for nested feature influence (#62068) (#62150)
Adds mappings for outlier detection results.

Backport of #62068
2020-09-09 12:41:26 +03:00
Yang Wang 146b2e6b1a
[Test] Fix data-stream rest test failure (#62137) (#62144)
By enabling searchable snapshots for release builds.
2020-09-09 17:12:53 +10:00
Dimitris Athanasiou 14547b14fb
[ML] Add changes stats fields as exception in mappings upgrade test (#62121)
Following #61980 we need to exclude the replaced fields in the test
in the 7.x branch.
2020-09-09 10:09:35 +03:00
Yang Wang 18a08c0cf2
Mute MlMappingsUpgradeIT testMappingsUpgrade (#61909) (#62136)
For #61908

Co-authored-by: David Kyle <david.kyle@elastic.co>
2020-09-09 13:23:36 +10:00
Armin Braun ed4984a32e
Remove Redundant Stream Wrapping from Compression (#62017) (#62132)
In many cases we don't need a `StreamInput` or `StreamOutput`
wrapper around these streams so I this commit adjusts the API
to just normal streams and adds the wrapping where necessary.
2020-09-09 03:27:38 +02:00
Julie Tibshirani 2ca5f98e05 Small fixes to breaking changes docs.
* Move ngram and shingle changes to the analysis section.
* Add missing heading for field caps change.
2020-09-08 17:19:36 -07:00
Lisa Cawley 78b955eb86 [DOCS] Fix from and size descriptions for model APIs (#62128) 2020-09-08 12:56:36 -07:00
Nik Everett b8e9a7125f
Speed up empty highlighting many fields (backport of #61860) (#62122)
Kibana often highlights *everything* like this:
```
POST /_search
{
  "query": ...,
  "size": 500,
  "highlight": {
    "fields": {
      "*": { ... }
    }
  }
}
```

This can get slow when there are hundreds of mapped fields. I tested
this locally and unscientifically and it took a request from 20ms to
150ms when there are 100 fields. I've seen clusters with 2000 fields
where simple search go from 500ms to 1500ms just by turning on this sort
of highlighting. Even when the query is just a `range` that and the
fields are all numbers and stuff so it won't highlight anything.

This speeds up the `unified` highlighter in this case in a few ways:
1. Build the highlighting infrastructure once field rather than once pre
   document per field. This cuts out a *ton* of work analyzing the query
   over and over and over again.
2. Bail out of the highlighter before loading values if we can't produce
   any results.

Combined these take that local 150ms case down to 65ms. This is unlikely
to be really useful when there are only a few fetched docs and only a
few fields, but we often end up having many fields with many fetched
docs.
2020-09-08 15:49:50 -04:00
Alan Woodward 28fd4a2ae8 Convert RangeFieldMapper to parametrized form (#62058)
This also adds the ability to define a serialization check on Parameters, used
in this case to only serialize format and locale parameters if the mapper is a
date range.
2020-09-08 18:44:13 +01:00
Alan Woodward 5f05eef7e3 Convert some more mapping tests to MapperServiceTestCase (#62089)
We don't need to extend ESSingleNodeTestCase for all these tests.
2020-09-08 17:51:40 +01:00
James Rodewig 5bca671f57
[DOCS] Fix ILM read only link (#62113) (#62119) 2020-09-08 12:19:24 -04:00
James Rodewig cc5e01a242
[DOCS] Fix field caps API docs (#62110) (#62116) 2020-09-08 12:19:04 -04:00
Costin Leau 0f9532689f EQL: Propagate key constraints through the query (#62073)
Since join keys are common across all queries in a Join/Sequence, any
constraint applied on one query needs to be obeyed but all the other
queries.
This PR enhances the optimizer to propagate such constraints across
all queries so they get pushed down to the actual generated ES queries.

Fix #58937

(cherry picked from commit 4afa5debc199c132c07015bfae17952c40a21e5d)
2020-09-08 18:40:47 +03:00
Benjamin Trent 057bf3f7d5
[ML] setting require_alias to previous value on bulk index retry (#62103) (#62108)
Previous work has been done to prevent automatically creating a concrete index when an alias is desired.

This commit addresses a path where this check was not being done.

relates: #62064
2020-09-08 11:38:32 -04:00
Lisa Cawley f0e7d88699 [DOCS] Fix allow_no_match description for model APIs (#62008) 2020-09-08 08:15:16 -07:00
Tim Brooks 075271758e
Keep checkpoint file channel open across fsyncs (#61744)
Currently we open and close the checkpoint file channel for every fsync.
This file channel can be kept open for the lifecycle of a translog
writer. This avoids the overhead of opening the file, checking file
permissions, and closing the file on every fsync.
2020-09-08 08:54:53 -06:00
Adam Locke 8f27e9fa28
[DOCS] [7.8] Clarify HTTPS usage for create key API (#60858) (#62098)
* Update create-api-keys.asciidoc

* Adding note to create API keys for https

* Adding note for enabling TLS

* Add specific setting for ssl.enabled

* Incorporating review feedback
2020-09-08 10:23:43 -04:00
James Rodewig 97bba08ea6
[DOCS] Fix typo in Java API docs (#62095) (#62097) 2020-09-08 09:49:03 -04:00
Dimitris Athanasiou 41507cff48
[7.x][ML] Update mappings of ml stats index (#61980) (#62091)
- Adds missing mappings for `alpha`, `gamma`, and `lambda`.
- Corrects name of `soft_tree_depth_limit` and `soft_tree_depth_tolerance`.
- Removes unused `regularization_depth_penalty_multiplier`,
  `regularization_leaf_weight_penalty_multiplier` and
  `regularization_tree_size_penalty_multiplier`.

Backport of #61980
2020-09-08 16:41:57 +03:00
David Roberts b2636678b2 [ML] Add support for date_nanos fields in find_file_structure (#62048)
Now that #61324 is merged it is possible for the find_file_structure
endpoint to suggest using date_nanos fields for timestamps where
the timestamp format provides greater than millisecond accuracy.
2020-09-08 13:05:09 +01:00
Francisco Fernández Castaño 2bb5716b3d
Add repositories metering API (#62088)
This pull request adds a new set of APIs that allows tracking the number of requests performed
by the different registered repositories.

In order to avoid losing data, the repository statistics are archived after the repository is closed for
a configurable retention period `repositories.stats.archive.retention_period`. The API exposes the
statistics for the active repositories as well as the modified/closed repositories.

Backport of #60371
2020-09-08 14:01:04 +02:00
David Kyle fb6ee5b36d
[7.x] [ML] Assert mappings match templates in Upgrade tests (#61905)
At the end of the rolling upgrade tests check the mappings of the concrete
.ml and .transform-internal indices match the mappings in the templates.
When the templates change, the tests should prove that the mappings have
been updated in the new cluster.
2020-09-08 12:21:19 +01:00
Przemko Robakowski bb357f6aae
[7.x] Move internal index templates to composable templates (#61457) (#61661)
This change moves watcher, ILM history and SLM history templates to composable templates.
Versions are updated to reflect the switch. Only change to the templates themselves is added `_meta` to mark them as managed
2020-09-08 11:26:06 +02:00
Armin Braun ebd1569028
Fix testMasterFailOverWithQueuedDeletes (#62062) (#62078)
Fixing very rare corner case where the delete retry is slow.

Closes #62031
2020-09-08 10:35:06 +02:00
Andrei Stefan 7d5791b6bd
EQL: create the search request with a list of indices (#62005) (#62076)
* The query client uses an array of indices instead of the comma separated
version of the indices names

(cherry picked from commit 8ec4a768f4892a4a2faed25836cb333a9deb2ace)
2020-09-08 10:26:59 +03:00
Nhat Nguyen 6574d81c59 Fix testEnableSoftDeletesOnRestore
Relates #62018
2020-09-07 15:10:55 -04:00
David Kyle a5b24bf44c
Mute ClassificationIT (#62063)
testWithOnlyTrainingRowsAndTrainingPercentIsFifty_DependentVariableIsBoolean
For #60759
2020-09-07 16:10:48 +01:00
Nhat Nguyen bb0a583990
Allow enabling soft-deletes on restore from snapshot (#62018)
Closes #61969
2020-09-07 09:45:36 -04:00
Luca Cavanna 168b448a0f
Rename runtime_script field type to runtime (#62034)
We've had some discussions around the user experience when using runtime fields. Although we do plan on having multiple runtime fields implementation (e.g. grok, lookup etc.) which could be exposed as different field types, we decided to expose all runtime fields under the same `runtime` type. At the moment, the only implementation will be through scripts, hence a `script` must be specified. In the future, there will be other ways to generate values for runtime fields besides scripts.

This translates also to renaming the RuntimeScriptFieldMapper class to RuntimeFieldMapper .

Relates to #59332
2020-09-07 15:07:23 +02:00
Alan Woodward cbc9578cbd Remove SearchPhase interface (#62050)
The interface is never used as an abstraction - implementations are are called directly,
and most of them don't need to implement the preProcess method.
2020-09-07 13:45:43 +01:00
David Turner 3389d5ccb2 Introduce integ tests for high disk watermark (#60460)
An important goal of the disk threshold decider is to ensure that nodes
use less disk space than the high watermark, and to take action if a
node ever exceeds this watermark. Today we do not have any
integration-style tests of this high-level behaviour. This commit
introduces a small test harness that can adjust the apparent size of the
disk and verify that the disk threshold decider moves shards around in
response.

Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2020-09-07 14:39:39 +02:00