Commit Graph

52358 Commits

Author SHA1 Message Date
Nik Everett 40850a780d
Fail variable_width_histogram that collects from many (#58619) (#58780)
Adds an explicit check to `variable_width_histogram` to stop it from
trying to collect from many buckets because it can't. I tried to make it
do so but that is more than an afternoon's project, sadly. So for now we
just disallow it.

Relates to #42035
2020-06-30 18:26:45 -04:00
James Rodewig 3aa08fbcde
[DOCS] Add data streams to API conventions (#58695) (#58785)
Updates the existing API conventions docs to make them aware of data
streams.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-30 17:33:35 -04:00
James Rodewig 046d9eeb41 [DOCS] Make `<target>` defs consistent 2020-06-30 15:55:37 -04:00
James Rodewig 416652cdd8
[DOCS] Clarify request formats for index API (#58768) (#58774) 2020-06-30 15:53:03 -04:00
James Rodewig e90c465640
[DOCS] Add data streams to cat APIs (#58699) (#58773) 2020-06-30 15:52:48 -04:00
James Rodewig 3778ca8c25
[DOCS] Add data streams to count API (#58771) (#58772) 2020-06-30 15:52:34 -04:00
Lee Hinman 74a78b3a7b
Mute AzureSearchableSnapshotsIT (#58775)
Relates to #58260
2020-06-30 13:30:51 -06:00
Dan Hermann cae49b0fd7
[7.x] Add data stream support to open index API (#58767) 2020-06-30 14:30:32 -05:00
Dan Hermann 22806c943d
Data stream support for ILM remove policy API (#58595) (#58770) 2020-06-30 14:03:19 -05:00
Dan Hermann a84ff81743
Data stream support for get field mappings API (#58488) (#58766) 2020-06-30 13:45:04 -05:00
Benjamin Trent a2331bc9d4
[Transform] fix bug in supporting boolean values in pivot (#58741) (#58760)
Since the underlying composite aggs support boolean mapped values for terms, transforms should also support them

closes #58697
2020-06-30 13:47:58 -04:00
James Rodewig 35830a7c12
[DOCS] Add data streams to get field mapping API docs (#58689) (#58759)
Updates the existing get field mapping API docs to make them aware of
data streams. Relates to #58488.
2020-06-30 13:24:41 -04:00
Martijn van Groningen adcef93a6c
Introduce new put mapping action for dynamic mapping updates. (#58746)
Backport of #58419

Mapping updates that originate from indexing a document with unmapped fields will use this new action
instead of the current put mapping action. This way on the security side, authorization logic
can easily determine whether a mapping update is automatically generated or a mapping update originates
from the put mapping api.

The new auto put mapping action is only used if all nodes are on the version that supports it.
2020-06-30 18:02:31 +02:00
Boice Huang 8c93f4e154 Sort document by internal doc id in FetchPhase to better use LRU cache (#57273)
This change sorts the docIdsToLoad once instead of in each sub-phase.
2020-06-30 17:06:09 +02:00
Julie Tibshirani ab65a57d70
Merge mappings for composable index templates (#58709)
This PR implements recursive mapping merging for composable index templates.

When creating an index, we perform the following:
* Add each component template mapping in order, merging each one in after the
last.
* Merge in the index template mappings (if present).
* Merge in the mappings on the index request itself (if present).

Some principles:
* All 'structural' changes are disallowed (but everything else is fine). An
object mapper can never be changed between `type: object` and `type: nested`. A
field mapper can never be changed to an object mapper, and vice versa.
* Generally, each section is merged recursively. This includes `object`
mappings, as well as root options like `dynamic_templates` and `meta`. Once we
reach 'leaf components' like field definitions, they always overwrite an
existing one instead of being merged.

Relates to #53101.
2020-06-30 08:01:37 -07:00
David Roberts d9e0e0bf95
[ML] Pass through the stop-on-warn setting for categorization jobs (#58738)
When per_partition_categorization.stop_on_warn is set for an analysis
config it is now passed through to the autodetect C++ process.

Also adds some end-to-end tests that exercise the functionality
added in elastic/ml-cpp#1356

Backport of #58632
2020-06-30 15:17:04 +01:00
James Rodewig 874ab36b14
[DOCS] Fix error in stop SLM API docs (#58747) (#58750) 2020-06-30 10:16:43 -04:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
James Rodewig 19190c529c [DOCS] Reword admon for index API and data streams 2020-06-30 09:54:24 -04:00
James Rodewig 770f9f11af [DOCS] Fix xref format in async EQL search docs 2020-06-30 09:37:47 -04:00
James Rodewig e5d5b9f5e8
[DOCS] Suppress searchable snapshots in releases (#58740) (#58742)
Fixes a searchable snapshot reference overlooked in #58652
2020-06-30 09:22:32 -04:00
Armin Braun b52a764143
Fix NPE in SnapshotService CS Application (#58680) (#58735)
In the unlikely corner case of deleting a relocation (hence `WAITING`) primary shard's
index during a partial snapshot, we would throw an NPE when checking if there's any external
changes to process.
2020-06-30 15:20:49 +02:00
James Rodewig d8731853a3
[DOCS] EQL: Document `head` and `tail` pipes (#58673) (#58739) 2020-06-30 09:12:54 -04:00
James Rodewig d33764583c
[7.x] [DOCS] Document delete/update by query for data streams (#58679) (#58706) 2020-06-30 08:35:13 -04:00
Alan Woodward a9677efb56 Mute DebMetadataTests#test05CheckLintian (#58733) 2020-06-30 13:30:52 +01:00
David Turner ceff00997d Suppress searchable snapshots docs in releases (#58652)
This commit adds conditional logic to the docs to avoid including any
docs on searchable snapshots in released versions.

Rework of #58556 which was reverted.
2020-06-30 13:13:09 +01:00
Przemysław Witek 9ea9b7bd3b
[7.x] Implement MSLE (MeanSquaredLogarithmicError) evaluation metric for regression analysis (#58684) (#58731) 2020-06-30 14:09:11 +02:00
Yannick Welsch b885cbff1a
Add index block api (#58716)
Adds an API for putting an index block in place, which also ensures for write blocks that, once successfully returning to
the user, all shards of the index are properly accounting for the block, for example that all in-flight writes to an index have
been completed after adding the write block.

This API allows coordinating more complex workflows, where it is crucial that an index is no longer receiving writes after
the API completes, useful for example when marking an index as read-only during an upgrade in order to reindex its
documents.
2020-06-30 14:06:52 +02:00
Benjamin Trent def5550df3
[ML] fix ml inference stats tests (#58690) (#58729) 2020-06-30 07:53:33 -04:00
Przemyslaw Gomulka 3923a10165
Exclude SystemV timezones from randomZone method (#58549) (#58655)
RandomZone test method returns a ZoneId from the set of ids supported by
java. The only difference between joda and java supported timezones are
SystemV* timezones.
These should be excluded from randomZone method as they would break
testing. They also do not bring much confidence when used in testing as
I suspect they are rarely used.
That exclude should be removed for simplification once joda support is removed.
2020-06-30 12:45:53 +02:00
Andrei Stefan 7b80ea7218
Fix release tests (#58713) (#58725)
(cherry picked from commit 7816c100612168bf46595c4813fe374bca2e7259)
2020-06-30 13:42:32 +03:00
Tanguy Leroux 4e03633a66
Differentiate base paths for searchable snapshots QA tests (#58664) (#58714)
This commit adds the BuildParams.testSeed to the repository base paths used 
in searchable snapshots QA tests. For S3 and GCS the test seed is added for 
coherency sake with other integration tests while it's required for Azure as 
Azure 3rd party tests are executed on CI simultaneously for regular and 
SAS token accounts.

Closes #58260
2020-06-30 10:18:33 +02:00
Tim Vernum dcc5a06dec
Display enterprise license as platinum in /_xpack (#58217)
The GET /_license endpoint displays "enterprise" licenses as
"platinum" by default so that old clients (including beats, kibana and
logstash) know to interpret this new license type as if it were a
platinum license.

However, this compatibility layer was not applied to the GET /_xpack/
endpoint which also displays a license type & mode.

This commit causes the _xpack API to mimic the _license API and treat
enterprise as platinum by default, with a new accept_enterprise
parameter that will cause the API to return the correct "enterprise"
value.

This BWC layer exists only for the 7.x branch.
This is a breaking change because, since 7.6, the _xpack API has
returned "enterprise" for enterprise licenses, but this has been found
to break old versions of beats and logstash so needs to be corrected.
2020-06-30 16:42:28 +10:00
James Rodewig 8341ebc061
[7.x] [DOCS] Reformats the update by query API. (#46199) (#58700)
Co-authored-by: debadair <debadair@elastic.co>
2020-06-29 17:50:32 -04:00
Dan Hermann 84513c7539
Document the prohibition on freezing data stream write indices (#58058) (#58705) 2020-06-29 16:34:36 -05:00
Ryan Ernst 35a703db16
Skip packaging test log file stashing once tests have failed (#58520)
When a packaging test completes, we move the elasticsearch logfile to
another location so that if the next test fails, we have just the
contents of the test that ran. However, once the test fails, we continue
trying to move this file in the subsequent calls to teardown when the
rest of the tests are skipped. This commit skips the teardown step once
we have marked the suite as failed.
2020-06-29 13:33:56 -07:00
Ryan Ernst 67a5a2bb60
Remove vagrant debug logging (#58629)
Previously we had odd issues in CI with vagrant which we attempted to
diagnose by enabling debug logging within our vagrant commands. However,
the debug logging didn't explain anything additional to the failures,
and it is extremely verbose. This commit removes the debug logging.
2020-06-29 13:33:07 -07:00
Adam Locke 719f2fb135
[DOCS] [7.x] Adding create index snapshot API docs (#58519) (#58692)
* Adding create index snapshot API page.
* Condense API description.
* Remove parameter from query.
* Add POST method and remove `-name` from the snapshot variable.
* Expand description of `<snapshot>`.
* Add data streams to introduction and expand the overall description.
* Add support for data streams.
* Add support for data streams.
* Add data stream and reference for "point-in-time view".
* Add data streams.
* Change `my_backup` to `my_repository`.
* Add description of boolean options for `wait_for_completion` parameter.
* Change command --> response
* Clarify `indices` parameter description
* Update `ignore-unavailable` parameter description
* Reword example description
* Remove "index" from API name
* Incorporating review comments from James R.
* Adding a much better request + response
* Clarify `include_global_state` description
* Incorporating additional edits.
* Changing my_backup to my_repository in example.
* Update snippet test to avoid failures
* Update TESTRESPONSE snippets
* Remove errant space
* Removing the  parameter per reviewer comments
2020-06-29 16:13:53 -04:00
Costin Leau 3a546f1f51 EQL: Introduce support for sequence maxspan (#58635)
EQL sequences can specify now a maximum time allowed for their span
(computed between the first and the last matching event).

(cherry picked from commit 747c3592244192a2e25a092f62aec91a899afc83)
2020-06-29 21:31:00 +03:00
Igor Motov 773f3574a9
Removes debug logging from RestEqlCancellationIT (#58676)
The test didn't fail since the fix in #58493. So, it's time to remove debug
logging and close the issue.

Closes #58270
2020-06-29 13:15:01 -04:00
Andrei Stefan 3cb8f54f28
EQL: case sensitivity aware integration testing (#58624) (#58672)
* EQL: case sensitivity aware integration testing (#58624)

* Add DataLoader
* Rewrite case sensitivity settings:
NULL -> run both case sensitive and insensitive tests
TRUE -> run case sensitive test only
FALSE -> run case insensitive test only
* Rename test_queries_supported
* Add more toml tests from the Python client

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
(cherry picked from commit 34d383421599f060a5c083b40df35f135de49e39)
2020-06-29 18:40:07 +03:00
Patrick Jiang(白泽) be20aacec3 Add `matchBoolPrefix` method to QueryBuilders (#58637) 2020-06-29 16:30:40 +02:00
James Rodewig 735a3f344d
[DOCS] EQL: Remove fields from EQL search response (#58667) (#58669) 2020-06-29 09:34:20 -04:00
Tanguy Leroux 73adcf4d44
SparseFileTracker.Gap should keep a reference to the corresponding Range (#58587) (#58665)
SparseFileTracker.Gap can keep a reference to the corresponding range it is about to fill,
it does not need to resolve the range each time onSuccess/onProgress/onFailure are 
called.

Relates #58477
2020-06-29 15:24:19 +02:00
Armin Braun ffcbf9ca0c
Fix Snapshots Capturing Incomplete Datastreams (#58630) (#58656)
Only snapshot datastreams that are recorded in `SnapshotInfo` and clean those
that aren't from the snapshotted metadata.
Do not restore all datastreams by default when restoring global metadata, use the same
mechanics used for indices here.

Closes #58544
2020-06-29 13:32:21 +02:00
István Zoltán Szabó 13aa8b8d9a [DOCS] Updates results_field description in the inference processor docs (#58554) 2020-06-29 13:15:15 +02:00
Armin Braun 95d85f29f8
Fix Snapshots Capturing Incomplete Datastreams (#58630) (#58656)
Only snapshot datastreams that are recorded in `SnapshotInfo` and clean those
that aren't from the snapshotted metadata.
Do not restore all datastreams by default when restoring global metadata, use the same
mechanics used for indices here.

Closes #58544
2020-06-29 12:51:40 +02:00
Armin Braun 4f2f257b12
Fix DataStream Handling on Restore of Global Metadata (#58631) (#58649)
When restoring a global metadata snapshot we were overwriting the correctly
adjusted data streams in the metadata when looping over all custom values.

Closes #58496
2020-06-29 10:58:41 +02:00
Przemysław Witek 3f7c45472e
[7.x] Introduce DataFrameAnalyticsConfig update API (#58302) (#58648) 2020-06-29 10:56:11 +02:00
David Turner 8f82ec0b19 Revert "Suppress searchable snapshots docs in releases (#58556)"
This reverts commit f0c0ee691a.
2020-06-29 09:21:58 +01:00