Commit Graph

52732 Commits

Author SHA1 Message Date
James Rodewig 1402f787f8
[DOCS] Add data streams to field caps API docs (#59326) (#59340) 2020-07-09 16:54:33 -04:00
James Rodewig 41345d4dd3
[DOCS] Add data streams to clear cache API docs (#59324) (#59339) 2020-07-09 16:54:04 -04:00
James Rodewig 77e227bf9b
[DOCS] Document custom routing support for data streams (#59323) (#59338) 2020-07-09 16:52:30 -04:00
James Rodewig ef74a68bcc
[DOCS] Document index aliases do not support data streams (#59321) (#59337) 2020-07-09 16:51:58 -04:00
Alan Woodward 4b9cbfca64 Remove test backported in error 2020-07-09 21:45:41 +01:00
Stuart Tettemer 94e213dd5f
Scripting: Per context stats in `script` in _nodes/stats (#59266)
Updated `_nodes/stats`:
 * Update `script` in `_node/stats` to include stats per context:

```
      "script": {
        "compilations": 1,
        "cache_evictions": 0,
        "compilation_limit_triggered": 0,
        "contexts":[
          {
            "context": "aggregation_selector",
            "compilations": 0,
            "cache_evictions": 0,
            "compilation_limit_triggered": 0
          },

```

Refs: #50152
Backport: #59625
2020-07-09 15:30:50 -05:00
Alan Woodward f4caadd239 MappedFieldType no longer requires equals/hashCode/clone (#59212)
With the removal of mapping types and the immutability of FieldTypeLookup in #58162, we no longer
have any cause to compare MappedFieldType instances. This means that we can remove all equals
and hashCode implementations, and in addition we no longer need the clone implementations which
were required for equals/hashcode testing. This greatly simplifies implementing new MappedFieldTypes,
which will be particularly useful for the runtime fields project.
2020-07-09 21:05:10 +01:00
Lisa Cawley 54483394ae
[DOCS] Clarify subscription requirements (#58958) (#59307) 2020-07-09 12:24:45 -07:00
Dan Hermann c7e977701a
Data stream support for async search 2020-07-09 13:12:04 -05:00
Dan Hermann c26d2b5fa5
Data stream support for indices shard stores API 2020-07-09 13:11:45 -05:00
Dan Hermann 34c50c045c
Data stream support for rank eval API 2020-07-09 13:11:29 -05:00
Dan Hermann b9fb12924b
Data stream support for EQL search 2020-07-09 13:10:44 -05:00
James Rodewig fca722cee1
[DOCS] Add x-pack tag to data stream docs (#59241) (#59299) 2020-07-09 13:12:38 -04:00
Dimitris Athanasiou b2243337d8
[7.x][ML] Data frame analytics max_num_threads setting (#59254) (#59308)
This adds a setting to data frame analytics jobs called
`max_number_threads`. The setting expects a positive integer.
When used the user specifies the max number of threads that may
be used by the analysis. Note that the actual number of threads
used is limited by the number of processors on the node where
the job is assigned. Also, the process may use a couple more threads
for operational functionality that is not the analysis itself.

This setting may also be updated for a stopped job.

More threads may reduce the time it takes to complete the job at the cost
of using more CPU.

Backport of #59254 and #57274
2020-07-09 19:15:46 +03:00
Nik Everett 28ef997953
Improve vwh's distant bucket handling (#59094) (#59248)
This modifies the `variable_width_histogram`'s distant bucket handling
to:
1. Properly handle integer overflows
2. Recalculate the average distance when new buckets are added on the
   ends. This should slow down the rate at which we build extra buckets
   as we build more of them.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-09 12:14:46 -04:00
Przemko Robakowski c870d6e570
[7.x] Restart tests with data streams (#58330) (#59303)
* Restart tests with data streams (#58330)
2020-07-09 17:52:20 +02:00
Costin Leau d9c1e531db EQL: Introduce until functionality (#59292)
Sequences now support until conditional, which prevents a match from
occurring if the until matches a document while doing look-ups.
Thus a sequence must complete before the until condition matches - if
any document within the sequence occurs at, or after, the until hit, the
sequence is discarded.

(cherry picked from commit 1ba1b9f0661aee655aa48cf9475ac61aaee2bfda)
2020-07-09 17:12:01 +03:00
Dimitris Athanasiou d07b11b86b
[7.x][ML] Perform test inference on java (#58877) (#59298)
Since we are able to load the inference model
and perform inference in java, we no longer need
to rely on the analytics process to be performing
test inference on the docs that were not used for
training. The benefit is that we do not need to
send test docs and fit them in memory of the c++
process.

Backport of #58877

Co-authored-by: Dimitris Athanasiou <dimitris@elastic.co>

Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
2020-07-09 16:30:49 +03:00
David Kyle 86555ec163
Remove unused function InferenceIndexConstants.mapping() (#59146) (#59158)
InferenceIndexConstants.mapping() is broken and unused.
2020-07-09 14:28:53 +01:00
Andrei Stefan d187b531ed
EQL: Give a name to all toml tests and enforce the naming of new tests (#59283) (#59295)
(cherry picked from commit c8ffe3c9237d3cdd90331795b8e37517155b7e91)
2020-07-09 16:20:29 +03:00
Rory Hunter 5debd09808 Dangling indices documentation (#58751)
Part of #48366. Add documentation for the dangling indices
API added in #58176.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-09 14:02:23 +01:00
David Kyle dbb9c802b1
Better error message when the model cannot be parsed due to its size (#59166) (#59209)
The actual cause can be lost in a long list of parse exceptions
this surfaces the cause when the problem is size.
2020-07-09 13:43:46 +01:00
David Kyle c5443f78ce
Add Inference Pipeline aggregation to HLRC (#59086) (#59250)
Adds InferencePipelineAggregationBuilder to the HLRC duplicating 
the server side classes
2020-07-09 13:38:45 +01:00
David Turner d56fc72ee5 Fix node health-check-related test failures (#59277)
In #52680 we introduced a new health check mechanism. This commit fixes
up some sporadic related test failures, and improves the behaviour of
the `FollowersChecker` slightly in the case that no retries are
configured.

Closes #59252
Closes #59172
2020-07-09 12:46:12 +01:00
David Turner c80a9e2ec2 Skip unnecessary directory iteration (#59007)
Today `NodeEnvironment#findAllShardIds` enumerates the index directories
in each data path in order to find one with a specific name. Since we
already know the name of the folder we seek we can construct the path
directly and avoid this directory listing. This commit does that.
2020-07-09 11:56:41 +01:00
Daniel Mitterdorfer 10ef4d2140
Mute testMaxRestoreBytesPerSecIsUsed (#59289)
Relates #59287
2020-07-09 12:52:17 +02:00
Alan Woodward 67a27e2b9d Add declarative parameters to FieldMappers (#58663)
The FieldMapper infrastructure currently has a bunch of shared parameters, many of which
are only applicable to a subset of the 41 mapper implementations we ship with. Merging,
parsing and serialization of these parameters are spread around the class hierarchy, with
much repetitive boilerplate code required. It would be much easier to reason about these
things if we could declare the parameter set of each FieldMapper directly in the implementing
class, and share the parsing, merging and serialization logic instead.

This commit is a first effort at introducing a declarative parameter style. It adds a new FieldMapper
subclass, ParametrizedFieldMapper, and refactors two mappers, Boolean and Binary, to use it.
Parameters are declared on Builder classes, with the declaration including the parameter name,
whether or not it is updateable, a default value, how to parse it from mappings, and how to
extract it from another mapper at merge time. Builders have a getParameters method, which
returns a list of the declared parameters; this is then used for parsing, merging and serialization.
Merging is achieved by constructing a new Builder from the existing Mapper, and merging in
values from the merging Mapper; conflicts are all caught at this point, and if none exist then a new,
merged, Mapper can be built from the Builder. This allows all values on the Mapper to be final.

Other mappers can be gradually migrated to this new style, and once they have all been refactored
we can merge ParametrizedFieldMapper and FieldMapper entirely.
2020-07-09 11:43:21 +01:00
Daniel Mitterdorfer daa48329ec
[TEST] Mute FollowerFailOverIT.testFailOverOnFollower (#58659) (#59286)
Relates #58534

Co-authored-by: Dimitris Athanasiou <dimitris@elastic.co>
2020-07-09 12:38:36 +02:00
Albert Zaharovits 2b7456db7f
Improve auditing of API key authentication #58928
1. Add the `apikey.id`, `apikey.name` and `authentication.type` fields
to the `access_granted`, `access_denied`, `authentication_success`, and
(some) `tampered_request` audit events. The `apikey.id` and `apikey.name`
are present only when authn using an API Key.
2. When authn with an API Key, the `user.realm` field now contains the effective
realm name of the user that created the key, instead of the synthetic value of
`_es_api_key`.
2020-07-09 13:26:18 +03:00
Dimitris Athanasiou d323f8d698
[ML] Add REST spec for the update data frame analytics endpoint (#59253) (#59281)
Closes #59148

Backport of #59253
2020-07-09 13:12:21 +03:00
Ignacio Vera 1ad00d1ceb
Add Support in geo_match enrichment policy for any type of geometry (#59276)
geo_match enrichment works currently only with points. This change adds the ability to
use any type of geometry.
2020-07-09 11:41:41 +02:00
Andrei Stefan c0e0bca84c
Remove search_after and implicit_join_key_field (#59232) (#59280)
(cherry picked from commit 6ede6c59eff321b9fedad30e19508b9e4f788b54)
2020-07-09 12:34:01 +03:00
Bogdan Pintea acfff7b896
Add sample versions of standard deviation and variance funcs (#59093) (#59274)
* Add sample versions of standard deviation and variance functions (#59093)

* Add STDDEV_SAMP, VAR_SAMP

This commit adds the sampling variations of the standard deviation and
variance agg functions.

(cherry picked from commit 8b29817b49e386215f29cb5b3356d0183fd5d9de)

* Fix: workaround for lack of Map#of() in Java8

Replace Map#of() with a HashMap static init.
2020-07-09 10:17:13 +02:00
Ignacio Vera 14ab35e323
Fix numerical error in CentroidCalculatorTests#testPolygonAsPoint (#59012) (#59272) 2020-07-09 08:42:07 +02:00
Ryan Ernst 04a531dcfa
Use consistent system jdk in 7.x packaging tests (#59257)
In 7.x, we have java 8 as minimum jdk version. In the past, for
packaging tests, we relied on the system to provide an alternative jdk
to be used by the no-jdk distributions. Master switched to using a build
provided jdk, but 7.x was stuck relying on the system because it needed
a java 8 jdk. The jdk download plugin was updated a while ago to support
jdk 8, and so this PR converts the distro tests to use the build
provided jdk just as master branch does.

Note also this fixes a failure that would sometimes occur on older jdks
in windows where the expected gc filename can be different.
2020-07-08 15:23:59 -07:00
Mark Vieira b41bbe2911
Parameterize windows packaging test script (#58980) 2020-07-08 13:50:20 -07:00
Adam Locke 96a06685cf
[7.x] [DOCS] Adding get snapshot api docs (#59238)
* [DOCS] Adding get snapshot API docs (#59098)

* Adding page for get snapshot API.

* Adding values for state and cleaning up some other formatting.

* Adding missing forward slash to GET request.

* Updating values for start_time and end_time in TESTRESPONSE.

* Swap "return" for "retrieve"

* Swap "return" for "retrieve" 2

* Change .snapshot to .response

* Adding response parameters and incorporating edits from review.

* Update response example to include repository info

* Change dash to underscore

* Add data type for snapshot in response

* Incorporating review comments and adding missing response definitions.

* Minor rewording in description.

* Removing multi-snapshot support for 7.x.

* Changing end_time value from build error.

* Removing .response from snippet testing.
2020-07-08 16:40:35 -04:00
Lee Hinman bb1c53a0f5
Allow warnings about 'global' template in upgrade tests (#59242)
These tests sometimes install a template so they can be compatible with older versions, but they run
amok of the occasionally installed "global" template which changes the default number of shards.

This commit adds `allowedWarnings` and allows these warnings to be present, but doesn't fail if they
are not (since the global template is only randomly installed).

Resolves #58807
Resolves #58258
2020-07-08 13:40:55 -06:00
Nhat Nguyen e50a0330ec Remove random of recovery chunk size setting
The recovery chunk size setting was injected in #58018, but too
aggressively and broke several tests. This change removes that
random injection.

Relates #58018
2020-07-08 15:29:37 -04:00
James Rodewig d2c5a4c5e9
[7.x] [DOCS] Update get data stream API response (#59197) (#59221)
Updates docs and snippets for changes made to the get data stream API with
PR #59128.
2020-07-08 14:04:14 -04:00
Nhat Nguyen 6a0f7411e2 Do not release safe commit with CancellableThreads (#59182)
We are leaking a FileChannel in #39585 if we release a safe commit with 
CancellableThreads. Although it is a bug in Lucene where we do not close
a FileChannel if we failed to create a NIOFSIndexInput, I think it's
safer if we release a safe commit using the generic thread pool instead.

Closes #39585
Relates #45409
2020-07-08 13:51:48 -04:00
Armin Braun cc3c8be0f1
Fix SLMSnapshotBlockingIntegTests.testSnapshotInProgress (#59218) (#59239)
Waiting `INIT` here is dead code in newer versions that don't use `INIT`
any longer and leads to nothing being written to the repository in older versions
if the snapshot is cancelled at the `INIT` step which then breaks repo consistency
checks.
Since we have other tests ensuring that snapshot abort works properly we can just remove
the wait for `INIT` here and backport this down to 7.8 to fix tests.

relates #59140
2020-07-08 19:13:01 +02:00
James Rodewig 838f717e5f
[DOCS] Add data streams to security docs (#59084) (#59237) 2020-07-08 12:53:56 -04:00
Nhat Nguyen 00c859bfca Fix testSendSnapshotSendsOps
We need to use a concurrent collection to keep track of the shipped operations
as they can arrive concurrently since #58018.

Relates #58018
2020-07-08 12:25:33 -04:00
James Rodewig 93a5eb0688
[DOCS] EQL: Document `size` limit for pipes (#59085) (#59236)
Changes:
* Documents the `size` default as `10`.
* Updates `size` param def to note its relation to pipes.
* Updates the `head` and `tail` pipe docs to modify sequences.
* Documents the `fetch_size` parameter.

Relates to #59014 and #59063
2020-07-08 12:22:57 -04:00
Martijn van Groningen 17bd559253
Fix the timestamp field of a data stream to @timestamp (#59210)
Backport of #59076 to 7.x branch.

The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.

Relates to #58642
Relates to #53100
Closes #58956
Closes #58583
2020-07-08 17:30:46 +02:00
Rene Groeschke 6a7ac3830a
Verify distro archives do not contain plain class files (#59073) (#59229)
This is a test to ensure we do not run into a regression like we did in
https://github.com/elastic/elasticsearch/issues/59031
2020-07-08 17:06:36 +02:00
James Rodewig b27de36b5d
[DOCS] EQL: Document `maxspan` keyword (#58931) (#59223) 2020-07-08 11:04:28 -04:00
James Rodewig 37be56ab97
[DOCS] EQL: Document unsupported var comparison (#58941) (#59224)
ES EQL queries do not support the comparison of a variable, such as
a field value, to another variable.

This adds a related para and example to the EQL syntax docs.
2020-07-08 11:04:05 -04:00
David Turner 6ffdb19a2a Clean searchable snapshots cache on startup (#59009)
Today we empty the searchable snapshots cache when cleanly closing a
shard, but leak cache files in some cases involving an unclean shutdown.
Such leaks are not permanent, they are cleaned up on shard relocation or
deletion, but they still might last for arbitrarily long until that
happens. This commit introduces a cleanup process that runs during node
startup to catch such leaks sooner.

Also, today we permit searchable snapshots to be held on custom data
paths, and store the corresponding cache files within the custom
location. Supporting this feature would make the cleanup process
significantly more complicated since it would require each node to parse
the index metadata for the shards it held before shutdown. Yet, this
feature is undocumented and offers minimal benefits to searchable
snapshots. Therefore with this commit we forbid custom data paths for
searchable snapshot shards.
2020-07-08 15:17:52 +01:00