Commit Graph

52769 Commits

Author SHA1 Message Date
Russ Cam 7d34fa9b67 Update link to .NET BulkAllObservable 2020-07-01 19:54:30 +10:00
David Turner 3a234d2669
Account for remaining recovery in disk allocator (#58800)
Today the disk-based shard allocator accounts for incoming shards by
subtracting the estimated size of the incoming shard from the free space on the
node. This is an overly conservative estimate if the incoming shard has almost
finished its recovery since in that case it is already consuming most of the
disk space it needs.

This change adds to the shard stats a measure of how much larger each store is
expected to grow, computed from the ongoing recovery, and uses this to account
for the disk usage of incoming shards more accurately.

Backport of #58029 to 7.x

* Picky picky

* Missing type
2020-07-01 10:12:44 +01:00
David Kyle 27d52d4d23
Remove the Model interface (#58754) (#58803)
The Model interface was implemented by just one class and did not 
contribute to making the code more undertandable
2020-07-01 09:57:02 +01:00
Rene Groeschke 2e278eaf3c
Detangle JdkJarHellCheck from build tool building (#58601) (#58755)
* Detangle JdkJarHellCheck from build tool building

- allows building the tool with same runtime as es
- allows building build tools with newer runtime version and keep ThirdPartyAuditTask
running with minimum runtime to ensure we check against correct jre
- add jdkjarhell test jar setup into fixture
2020-07-01 10:41:01 +02:00
Dario Gieselaar 417f7062c5
[7.x] Add read privileges for annotations for apm_user (#58530) (#58781)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-01 09:04:57 +02:00
Yang Wang 3d49e62960
Support handling LogoutResponse from SAML idP (#56316) (#58792)
SAML idP sends back a LogoutResponse at the end of the logout workflow. It can be sent via either HTTP-Redirect binding or HTTP-POST binding. Currently, the HTTP-Redirect request is simply ignored by Kibana and never reaches ES. It does not cause any obvious issue and the workflow is completed normally from user's perspective.

The HTTP-POST request results in a 404 error because POST request is not accepted by Kibana's logout end-point. This causes a non-trivial issue because it renders an error page in user's browser. In addition, some resources do not seem to be fully cleaned up due to the error, e.g. the username will be pre-filled when trying to login again after the 404 error.

This PR solves both of the above issues from ES side with a new /_security/saml/complete_logout end-point. Changes are still needed on Kibana side to relay the messages.
2020-07-01 16:47:27 +10:00
Mark Vieira d81b455532
Fix comment formatting 2020-06-30 22:18:04 -07:00
Mark Vieira 119462dc85
Temporarily disable docker-compose test fixtures 2020-06-30 22:08:48 -07:00
Tim Vernum 9e49af03b7
Reenable test after backport (#58717)
This commit re-enables CCR rolling upgrade tests following the
backport of #58217 to 7.8 branch (7.8.1)
2020-07-01 11:50:30 +10:00
Lee Hinman efa0686a6c Fix template name in mapping composition yml test (#58788)
The warning was copied from elsewhere and just needed to use the correct template and index name.
2020-06-30 17:04:15 -06:00
Dan Hermann 1c2a726731
Data stream support for search shards API (#58486) (#58765) 2020-06-30 17:59:51 -05:00
Nik Everett 40850a780d
Fail variable_width_histogram that collects from many (#58619) (#58780)
Adds an explicit check to `variable_width_histogram` to stop it from
trying to collect from many buckets because it can't. I tried to make it
do so but that is more than an afternoon's project, sadly. So for now we
just disallow it.

Relates to #42035
2020-06-30 18:26:45 -04:00
James Rodewig 3aa08fbcde
[DOCS] Add data streams to API conventions (#58695) (#58785)
Updates the existing API conventions docs to make them aware of data
streams.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-30 17:33:35 -04:00
James Rodewig 046d9eeb41 [DOCS] Make `<target>` defs consistent 2020-06-30 15:55:37 -04:00
James Rodewig 416652cdd8
[DOCS] Clarify request formats for index API (#58768) (#58774) 2020-06-30 15:53:03 -04:00
James Rodewig e90c465640
[DOCS] Add data streams to cat APIs (#58699) (#58773) 2020-06-30 15:52:48 -04:00
James Rodewig 3778ca8c25
[DOCS] Add data streams to count API (#58771) (#58772) 2020-06-30 15:52:34 -04:00
Lee Hinman 74a78b3a7b
Mute AzureSearchableSnapshotsIT (#58775)
Relates to #58260
2020-06-30 13:30:51 -06:00
Dan Hermann cae49b0fd7
[7.x] Add data stream support to open index API (#58767) 2020-06-30 14:30:32 -05:00
Dan Hermann 22806c943d
Data stream support for ILM remove policy API (#58595) (#58770) 2020-06-30 14:03:19 -05:00
Dan Hermann a84ff81743
Data stream support for get field mappings API (#58488) (#58766) 2020-06-30 13:45:04 -05:00
Benjamin Trent a2331bc9d4
[Transform] fix bug in supporting boolean values in pivot (#58741) (#58760)
Since the underlying composite aggs support boolean mapped values for terms, transforms should also support them

closes #58697
2020-06-30 13:47:58 -04:00
James Rodewig 35830a7c12
[DOCS] Add data streams to get field mapping API docs (#58689) (#58759)
Updates the existing get field mapping API docs to make them aware of
data streams. Relates to #58488.
2020-06-30 13:24:41 -04:00
Martijn van Groningen adcef93a6c
Introduce new put mapping action for dynamic mapping updates. (#58746)
Backport of #58419

Mapping updates that originate from indexing a document with unmapped fields will use this new action
instead of the current put mapping action. This way on the security side, authorization logic
can easily determine whether a mapping update is automatically generated or a mapping update originates
from the put mapping api.

The new auto put mapping action is only used if all nodes are on the version that supports it.
2020-06-30 18:02:31 +02:00
Boice Huang 8c93f4e154 Sort document by internal doc id in FetchPhase to better use LRU cache (#57273)
This change sorts the docIdsToLoad once instead of in each sub-phase.
2020-06-30 17:06:09 +02:00
Julie Tibshirani ab65a57d70
Merge mappings for composable index templates (#58709)
This PR implements recursive mapping merging for composable index templates.

When creating an index, we perform the following:
* Add each component template mapping in order, merging each one in after the
last.
* Merge in the index template mappings (if present).
* Merge in the mappings on the index request itself (if present).

Some principles:
* All 'structural' changes are disallowed (but everything else is fine). An
object mapper can never be changed between `type: object` and `type: nested`. A
field mapper can never be changed to an object mapper, and vice versa.
* Generally, each section is merged recursively. This includes `object`
mappings, as well as root options like `dynamic_templates` and `meta`. Once we
reach 'leaf components' like field definitions, they always overwrite an
existing one instead of being merged.

Relates to #53101.
2020-06-30 08:01:37 -07:00
David Roberts d9e0e0bf95
[ML] Pass through the stop-on-warn setting for categorization jobs (#58738)
When per_partition_categorization.stop_on_warn is set for an analysis
config it is now passed through to the autodetect C++ process.

Also adds some end-to-end tests that exercise the functionality
added in elastic/ml-cpp#1356

Backport of #58632
2020-06-30 15:17:04 +01:00
James Rodewig 874ab36b14
[DOCS] Fix error in stop SLM API docs (#58747) (#58750) 2020-06-30 10:16:43 -04:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
James Rodewig 19190c529c [DOCS] Reword admon for index API and data streams 2020-06-30 09:54:24 -04:00
James Rodewig 770f9f11af [DOCS] Fix xref format in async EQL search docs 2020-06-30 09:37:47 -04:00
James Rodewig e5d5b9f5e8
[DOCS] Suppress searchable snapshots in releases (#58740) (#58742)
Fixes a searchable snapshot reference overlooked in #58652
2020-06-30 09:22:32 -04:00
Armin Braun b52a764143
Fix NPE in SnapshotService CS Application (#58680) (#58735)
In the unlikely corner case of deleting a relocation (hence `WAITING`) primary shard's
index during a partial snapshot, we would throw an NPE when checking if there's any external
changes to process.
2020-06-30 15:20:49 +02:00
James Rodewig d8731853a3
[DOCS] EQL: Document `head` and `tail` pipes (#58673) (#58739) 2020-06-30 09:12:54 -04:00
James Rodewig d33764583c
[7.x] [DOCS] Document delete/update by query for data streams (#58679) (#58706) 2020-06-30 08:35:13 -04:00
Alan Woodward a9677efb56 Mute DebMetadataTests#test05CheckLintian (#58733) 2020-06-30 13:30:52 +01:00
David Turner ceff00997d Suppress searchable snapshots docs in releases (#58652)
This commit adds conditional logic to the docs to avoid including any
docs on searchable snapshots in released versions.

Rework of #58556 which was reverted.
2020-06-30 13:13:09 +01:00
Przemysław Witek 9ea9b7bd3b
[7.x] Implement MSLE (MeanSquaredLogarithmicError) evaluation metric for regression analysis (#58684) (#58731) 2020-06-30 14:09:11 +02:00
Yannick Welsch b885cbff1a
Add index block api (#58716)
Adds an API for putting an index block in place, which also ensures for write blocks that, once successfully returning to
the user, all shards of the index are properly accounting for the block, for example that all in-flight writes to an index have
been completed after adding the write block.

This API allows coordinating more complex workflows, where it is crucial that an index is no longer receiving writes after
the API completes, useful for example when marking an index as read-only during an upgrade in order to reindex its
documents.
2020-06-30 14:06:52 +02:00
Benjamin Trent def5550df3
[ML] fix ml inference stats tests (#58690) (#58729) 2020-06-30 07:53:33 -04:00
Przemyslaw Gomulka 3923a10165
Exclude SystemV timezones from randomZone method (#58549) (#58655)
RandomZone test method returns a ZoneId from the set of ids supported by
java. The only difference between joda and java supported timezones are
SystemV* timezones.
These should be excluded from randomZone method as they would break
testing. They also do not bring much confidence when used in testing as
I suspect they are rarely used.
That exclude should be removed for simplification once joda support is removed.
2020-06-30 12:45:53 +02:00
Andrei Stefan 7b80ea7218
Fix release tests (#58713) (#58725)
(cherry picked from commit 7816c100612168bf46595c4813fe374bca2e7259)
2020-06-30 13:42:32 +03:00
Tanguy Leroux 4e03633a66
Differentiate base paths for searchable snapshots QA tests (#58664) (#58714)
This commit adds the BuildParams.testSeed to the repository base paths used 
in searchable snapshots QA tests. For S3 and GCS the test seed is added for 
coherency sake with other integration tests while it's required for Azure as 
Azure 3rd party tests are executed on CI simultaneously for regular and 
SAS token accounts.

Closes #58260
2020-06-30 10:18:33 +02:00
Tim Vernum dcc5a06dec
Display enterprise license as platinum in /_xpack (#58217)
The GET /_license endpoint displays "enterprise" licenses as
"platinum" by default so that old clients (including beats, kibana and
logstash) know to interpret this new license type as if it were a
platinum license.

However, this compatibility layer was not applied to the GET /_xpack/
endpoint which also displays a license type & mode.

This commit causes the _xpack API to mimic the _license API and treat
enterprise as platinum by default, with a new accept_enterprise
parameter that will cause the API to return the correct "enterprise"
value.

This BWC layer exists only for the 7.x branch.
This is a breaking change because, since 7.6, the _xpack API has
returned "enterprise" for enterprise licenses, but this has been found
to break old versions of beats and logstash so needs to be corrected.
2020-06-30 16:42:28 +10:00
James Rodewig 8341ebc061
[7.x] [DOCS] Reformats the update by query API. (#46199) (#58700)
Co-authored-by: debadair <debadair@elastic.co>
2020-06-29 17:50:32 -04:00
Dan Hermann 84513c7539
Document the prohibition on freezing data stream write indices (#58058) (#58705) 2020-06-29 16:34:36 -05:00
Ryan Ernst 35a703db16
Skip packaging test log file stashing once tests have failed (#58520)
When a packaging test completes, we move the elasticsearch logfile to
another location so that if the next test fails, we have just the
contents of the test that ran. However, once the test fails, we continue
trying to move this file in the subsequent calls to teardown when the
rest of the tests are skipped. This commit skips the teardown step once
we have marked the suite as failed.
2020-06-29 13:33:56 -07:00
Ryan Ernst 67a5a2bb60
Remove vagrant debug logging (#58629)
Previously we had odd issues in CI with vagrant which we attempted to
diagnose by enabling debug logging within our vagrant commands. However,
the debug logging didn't explain anything additional to the failures,
and it is extremely verbose. This commit removes the debug logging.
2020-06-29 13:33:07 -07:00
Adam Locke 719f2fb135
[DOCS] [7.x] Adding create index snapshot API docs (#58519) (#58692)
* Adding create index snapshot API page.
* Condense API description.
* Remove parameter from query.
* Add POST method and remove `-name` from the snapshot variable.
* Expand description of `<snapshot>`.
* Add data streams to introduction and expand the overall description.
* Add support for data streams.
* Add support for data streams.
* Add data stream and reference for "point-in-time view".
* Add data streams.
* Change `my_backup` to `my_repository`.
* Add description of boolean options for `wait_for_completion` parameter.
* Change command --> response
* Clarify `indices` parameter description
* Update `ignore-unavailable` parameter description
* Reword example description
* Remove "index" from API name
* Incorporating review comments from James R.
* Adding a much better request + response
* Clarify `include_global_state` description
* Incorporating additional edits.
* Changing my_backup to my_repository in example.
* Update snippet test to avoid failures
* Update TESTRESPONSE snippets
* Remove errant space
* Removing the  parameter per reviewer comments
2020-06-29 16:13:53 -04:00
Costin Leau 3a546f1f51 EQL: Introduce support for sequence maxspan (#58635)
EQL sequences can specify now a maximum time allowed for their span
(computed between the first and the last matching event).

(cherry picked from commit 747c3592244192a2e25a092f62aec91a899afc83)
2020-06-29 21:31:00 +03:00