Commit Graph

50847 Commits

Author SHA1 Message Date
C.J. Jameson 5ac8b795ba [DOCS] Clarify routing enforcement in docs (#53945)
Removes a mention of the `_doc` mapping type that's
no longer applicable now that mapping types are
removed/deprecated.
2020-03-23 12:57:07 -04:00
Lisa Cawley 5f45780577 [DOCS] Adds data nanos transform limitation (#53826) 2020-03-23 09:49:21 -07:00
Lisa Cawley 8b3ef4ac21 [DOCS] Add generated_dest_index to preview transform API (#53905) 2020-03-23 09:43:44 -07:00
Mark Vieira eda023e2c3
Don't include HLRC on downstream classpath twice (#53983) 2020-03-23 08:49:34 -07:00
Armin Braun 5b9864db2c
Better Incrementality for Snapshots of Unchanged Shards (#52182) (#53984)
Use sequence numbers and force merge UUID to determine whether a shard has changed or not instead before falling back to comparing files to get incremental snapshots on primary fail-over.
2020-03-23 16:43:41 +01:00
Tanguy Leroux 8b9d6e6dbb Increase ensureGreen() timeout in CloseWhileRelocatingShardsIT (#53981)
The test in CloseWhileRelocatingShardsIT failed recently 
multiple times (3) when waiting for initial indices to be 
become green. Looking at the execution logs from  #53544
 it appears at the very beginning of the test and when 
the WindowsFS file system is picked up (which is known 
to slow down tests).

This commit simply increases the timeout for the first 
ensureGreen() to 60 seconds. If the test continues to fail, 
we might want to test a larger timeout or disable 
WindowsFS for this test.

Closes #53544
2020-03-23 16:24:25 +01:00
Dimitris Athanasiou 3873510332
[7.x][ML] Refactor DFA custom processor to cross validation splitter (#53915) (#53956)
While `CustomProcessor` is generic and allows for flexibility, there
are new requirements that make cross validation a concept it's hard
to abstract behind custom processor. In particular, we would like to
add data_counts to the DFA jobs stats. Counting training VS. test
docs would be a useful statistic. We would also want to add a
different cross validation strategy for multiclass classification.

This commit renames custom processors to cross validation splitters
which allows for those enhancements without cryptically doing
things as a side effect of the abstract custom processing.

Backport of #53915
2020-03-23 17:15:14 +02:00
Alan Woodward 0c010e1bfc lang-expressions should use DoubleValuesSource, not ValueSource (#53661)
DoubleValuesSource is the type-safe replacement for ValueSource in the lucene
core. Most of elasticsearch has moved to use these, but lang-expressions is still
using the old version. This commit migrates lang-expressions as well.
2020-03-23 14:48:38 +00:00
Armin Braun 754d071c4e
Upgrade to AWS SDK 1.11.749 (#53962) (#53974)
Upgrading AWS SDK to v1.11.749.
Required building clients inside privileged contexts because some class loading that requires privileges now happens there and working around a new SDK bug in the S3 client builder.

Closes #53191
2020-03-23 15:31:29 +01:00
Marios Trivyzas af03200ad6
SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) (#47720) (#53972)
The function is extended to operate on intervals according to the PostgreSQL: https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes : #46632
(cherry picked from commit 2dc79505825fa75e0711dcfa8e9c69e8028fc979)

Co-authored-by: musteaf <gs_mustea@hotmail.com>
2020-03-23 15:05:16 +01:00
Dan Hermann 0528e2b06b
Document the deprecation of the 'auth.password' setting (#53822) 2020-03-23 08:58:25 -05:00
Armin Braun b51ea25a00
Use Azure Bulk Deletes in Azure Repository (#53919) (#53967)
Now that we upgraded the Azure SDK to 8.6.2 in #53865 we can make use of
bulk deletes.
2020-03-23 13:35:05 +01:00
Martijn van Groningen aef7b89219
Backport: initial data stream commit (#53959)
This commits adds a data stream feature flag, initial definition of a data stream and
the stubs for the data stream create, delete and get APIs. Also simple serialization
tests are added and a rest test to thest the data stream API stubs.

This is a large amount of code and mainly mechanical, but this commit should be
straightforward to review, because there isn't any real logic.

The data stream transport and rest action are behind the data stream feature flag and
are only intialized if the feature flag is enabled. The feature flag is enabled if
elasticsearch is build as snapshot or a release build and the
'es.datastreams_feature_flag_registered' is enabled.

The integ-test-zip sets the feature flag if building a release build, otherwise
rest tests would fail.

Relates to #53100
2020-03-23 12:58:09 +01:00
James Rodewig 1d1e177a62 [DOCS] Note doc links should be live in REST API JSON specs (#53871)
Downstream Elasticsearch clients, such as the Elaticsearch-JS client,
use the documentation links in our REST API JSON specifications to
create their docs.

Using a broken link or linking to yet-to-be-created doc pages can
break the docs build for these clients.

This PR adds a related note to the README for the REST API JSON Specs.
2020-03-23 07:42:03 -04:00
Armin Braun 69a35158ce
Fix Azure Repository with HTTPs Endpoint (#53903) (#53963)
Upgrading to 8.6.2 in #53865 broke running against HTTPs endpoints (and hence real azure)
because the https url connection needs the newly added permission to work.
2020-03-23 12:16:33 +01:00
Yannick Welsch 060c72c799 Only link fd* files during source-only snapshot (#53463)
Source-only snapshots currently create a second full source-only copy of the shard on disk to
support incrementality during upload. Given that stored fields are occupying a substantial part
of a shard's storage, this means that clusters with source-only snapshots can require up to
50% more local storage. Ideally we would only generate source-only parts of the shard for the
things that need to be uploaded (i.e. do incrementality checks on original file instead of
trimmed-down source-only versions), but that requires much bigger changes to the snapshot
infrastructure. This here is an attempt to dramatically cut down on the storage used by the
source-only copy of the shard by soft-linking the stored-fields files (fd*) instead of copying
them.

Relates #50231
2020-03-23 11:04:53 +01:00
István Zoltán Szabó 534e734ad1 [DOCS] Adds painless transform examples (#53274)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-23 10:39:47 +01:00
Przemyslaw Gomulka 5b86fc4315
remove accidentaly added files (#53955) 2020-03-23 10:29:25 +01:00
Przemyslaw Gomulka 015ad019d5
[docs] Known issue about joda patterns on 7.6 (#53957) 2020-03-23 10:28:55 +01:00
David Turner 0fb31d9e7a Allow static cluster.max_voting_config_exclusions (#53717)
Today we only read `cluster.max_voting_config_exclusions` from the dynamic
settings in the cluster metadata, ignoring any value set in
`elasticsearch.yml`. This commit addresses this.

Closes #53455
2020-03-23 08:38:12 +00:00
Ignacio Vera efd1838206
Handle properly indexing rectangles that crosses the dateline (#53810) (#53947)
When indexing a rectangle that crosses the dateline, we are currently not
handling it properly and we index a polygon that do not cross the dateline.
This changes generates two polygons wrapping the dateline.
2020-03-23 09:12:03 +01:00
Tim Vernum cde8725e3c
Create API Key on behalf of other user (#53943)
This change adds a "grant API key action"

   POST /_security/api_key/grant

that creates a new API key using the privileges of one user ("the
system user") to execute the action, but creates the API key with
the roles of the second user ("the end user").

This allows a system (such as Kibana) to create API keys representing
the identity and access of an authenticated user without requiring
that user to have permission to create API keys on their own.

This also creates a new QA project for security on trial licenses and runs
the API key tests there

Backport of: #52886
2020-03-23 18:50:07 +11:00
Przemyslaw Gomulka 412e163cf6
[Doc] migration guide joda (#51986)
The joda to java.time migration requires users to upgrade their mappings. We allow them to still use 6.x created indices with joda patterns in 7 but ask them to upgrade their patterns in 7.x.
This migration guide is to help them understand how they could be affected and what needs to be changed in their mappings.
closes #51614
closes #51236
2020-03-23 08:29:01 +01:00
Tim Vernum f003a419a5
Add exception metadata for disabled features (#53941)
This change adds a new exception with consistent metadata for when
security features are not enabled. This allows clients to be able to
tell that an API failed due to a configuration option, and respond
accordingly.

Relates: kibana#55255
Resolves: #52311, #47759

Backport of: #52811
2020-03-23 14:13:15 +11:00
Jason Tedor 27c8bcbbd1
Introduce aarch64 packaging (#53914) (#53926)
This commit introduces aarch64 packaging, including bundling an aarch64
JDK distribution. We had to make some interesting choices here:
 - ML binaries are not compiled for aarch64, so for now we disable ML on
   aarch64
 - depending on underlying page sizes, we have to disable class data
   sharing
2020-03-22 11:58:11 -04:00
David Roberts 076ba02e9c
[TEST] Mute transforms rolling upgrade tests (#53932)
Due to https://github.com/elastic/elasticsearch/issues/53931
2020-03-22 15:17:07 +00:00
Armin Braun 5d3cb46232
Fix StackOverflowError in CheckStyle for CRUDDocumentationIT (#53840) (#53922)
This file seems to randomly cause a stack overflow like other HLRC ITs that
already have this check suppressed on them.

Closes #37647
2020-03-21 20:35:42 +01:00
Stuart Tettemer d25c01a373
Scripting: Increase ingest script cache defaults (#53906)
* Adds ability for contexts to specify their own defaults.
* Context defaults are applied if no context-specific or
  general setting exists.
* See 070ea7e for settings keys.

* Increases the per-context default for the `ingest` context.
  * Cache size is doubled, 200 compared to default of 100
  * Cache expiration is unchanged at no expiration
  * Cache max compilation is quintupled, 375/5m instead of 75/5m

Backport of: 1b37d4b
Refs: #50152
2020-03-20 16:48:50 -06:00
Gordon Brown 10cabbbade
Transition Transforms to using hidden indices for notifcations index (#53773)
This commit changes the Transforms notifications index to be hidden
index, with a hidden alias.

This commit also removes the temporary hack in
MetaDataCreateIndexService that prevents deprecation warnings for known
dot-prefixed index names which are not hidden/system indices, as this
was the last index pattern to need that hack.
2020-03-20 15:40:58 -06:00
Ryan Ernst caa4e0dc18
Use boolean methods for allowed realm types in license state (#53456) (#53834)
In xpack the license state contains methods to determine whether a
particular feature is allowed to be used. The one exception is
allowsRealmTypes() which returns an enum of the types of realms allowed.
This change converts the enum values to boolean methods. There are 2
notable changes: NONE is removed as we always fall back to basic license
behavior, and NATIVE is not needed because it would always return true
since we should always have a basic license.
2020-03-20 14:30:31 -07:00
Aleksandr Maus fd0cdde38c
EQL: EqlActionIT improvements (#53780) (#53888)
Related to https://github.com/elastic/elasticsearch/issues/53598
2020-03-20 17:28:15 -04:00
Stuart Tettemer ac575b68a9
Scripting: Context script cache unlimited compile (#53769) (#53899)
* Adds "unlimited" compilation rate for context script caches
* `script.context.${CONTEXT}.max_compilations_rate` = `unlimited`
  disables compilation rate limiting for `${CONTEXT}`'s script
  cache

Refs: #50152
2020-03-20 15:14:30 -06:00
Lee Hinman 1f3de2fa7e
Set feature flags for IndexTemplatesV2 in top-level gradle file (#53898)
Resolves #53892
2020-03-20 14:52:22 -06:00
Gordon Brown f0674af132
Add isHidden to AliasActions equals/hashcode (#53700)
This commit adds the `isHidden` flag to the `equals` and
`hashCode` methods for `AliasActions`.
2020-03-20 13:59:40 -06:00
David Turner 879e26ec06 Describe STALE_STATE_CONFIG in ClusterFormationFH (#53878)
We mark cluster states persisted on master-ineligible nodes as
potentially-stale using the voting configuration `{STALE_STATE_CONFIG}` which
prevents these nodes from being elected as master if they are restarted as
master-eligible. Today we do not handle this special voting configuration
differently in the `ClusterFormationFailureHandler`, leading to a mysterious
message `an election requires a node with id [STALE_STATE_CONFIG]` if the
election does not succeed.

This commit adds a special case description for this situation to explain
better why this node cannot win an election.

Closes #53734
2020-03-20 20:02:51 +01:00
Mark Vieira 0cfe6d90cc
Mute async-search test 2020-03-20 11:35:24 -07:00
Luca Belluccini 3937854444 [DOCS] Clarify upgrade paths (#53417)
Unsupported upgrade paths:
- `6.8 to 7.0`.
Supported, but requires a full cluster restart:
- `6.0–6.7 directly to 7.x`
2020-03-20 18:05:07 +00:00
Igor Motov 88d50ec583
Fix random failures in InternalTopHitsTests#testReduceRandom (#53832)
The test was randomly and very rarely failing due to generating the same sort
key for multiple records, which was making order of these records in the results
nondeterministic. While investigating the test I also found that the data wasn't
generated in the way that matches the actual data. Normally, the order of
documents in hits and scoreDocs in InternalTopHits should be the same. However,
in the test only scoreDocs were sorted which was cause very confusing failure
messages. This commit fixes this issue as well.

Fixes #53676
2020-03-20 13:35:59 -04:00
Lisa Cawley c2af015ee7 [DOCS] Updates list of transform aggs (#53820) 2020-03-20 10:28:23 -07:00
Armin Braun 41301d74b0
Upgrade to Azure SDK 8.6.2 (#53865) (#53886)
This fixes some bugs around retrying and URL encoding and should enable a follow-up
that finally adds bulk deletes on Azure.
2020-03-20 18:27:02 +01:00
Rory Hunter 17de77fd0f
Convert Wildfly tests to run in Docker (#53159)
Backport of #53126.

Convert the Wildfly tests to run using Docker Compose. This drastically
simplifies the build setup, and will hopefully make the tests more
resilient.
2020-03-20 16:47:21 +00:00
Nik Everett c2a2fcb5a1
Clean up eclipse build (backport of #53831) (#53870)
Fixes up the "forbidden" warnings that you get when you import
Elasticsearch using "import gradle projects".

With this, and the manual step of switching circular project definitions
to warnings this gets most thing *compiling*.
2020-03-20 12:12:05 -04:00
Alan Woodward a70016a6d6 TermsLookup fields should be marked as Required in docs (#53784)
The terms-lookup section of our terms query docs currently state that the
index, id and path fields are optional. They should be marked instead
as required.
2020-03-20 15:39:41 +00:00
David Turner adfeb50a53 Use consistent threadpools in CoordinatorTests (#53868)
Today in the `CoordinatorTests` each node uses multiple threadpools. This is
mostly fine as they are almost completely stateless, except for the
`ThreadContext`: by using multiple threadpools we cannot make assertions that
the thread context is/isn't preserved as we expect. This commit consolidates
the threadpool instances in use so that each node uses just one.
2020-03-20 16:22:42 +01:00
Alan Woodward a3f21f24ea
Emit deprecation warning when TermsLookup contains a type (#53731)
TermsLookup in master no longer accepts a type parameter. We should emit
a deprecate warning in 7.x when a terms lookup requests includes type to prepare
users for its removal.

Relates to #41059
2020-03-20 15:11:31 +00:00
Luca Cavanna d486bdefdd [DOCS] correct async search note
The sort optimization kicks in whenever results are sorted by field.
2020-03-20 15:58:19 +01:00
Aleksandr Maus 83bef862e0
EQL: Extract query folder tests definitions into resources (#53802) (#53869) 2020-03-20 10:39:35 -04:00
Armin Braun 7d66c7e25c
Rethrow Failed Assertion in BlobStoreTestUtils (#53859) (#53863)
This fixes two issues:

1. Currently, the future here is never resolved on assertion error so a failing test would take a full minute
to complete until the future times out.
2. S3 tests overide this method to busy assert on this method. This only works if an assertion error makes it
to the calling thread.

Closes #53508
2020-03-20 15:31:48 +01:00
Luca Cavanna 03fca61fcb [DOCS] add docs for async search (#53675)
Relates to #49091

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-03-20 14:46:38 +01:00
James Rodewig 6c57681446 [DOCS] Add redirects for missing data stream API docs (#53866)
The following API spec files contain a link to a not-yet-created
docs page for the data stream APIs:

* [indices.create_data_stream.json][0]
* [indices.delete_data_stream.jsonn][1]
* [indices.get_data_streams.json][2]

The Elaticsearch-JS client uses these spec files to create their docs.
This created a broken link in the Elaticsearch-JS, which has broken
the docs build.

This PR adds a temporary redirect for the docs page. This redirect
should be removed when the actual API docs are added.

[0]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.create_data_stream.json
[1]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_stream.json
[2]: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_streams.json

Relates to #53558.

CC @martijnvg
2020-03-20 09:41:47 -04:00