Commit Graph

47126 Commits

Author SHA1 Message Date
Igor Motov cfc8d17bb4 Geo: refactor geo mapper and query builder (#44884)
Refactors out the indexing and query generation logic out of the
mapper and query builder into a separate unit-testable classes.
2019-07-26 16:48:31 -04:00
Yannick Welsch 1561ab5420 Guard open connection call in RemoteClusterConnection (#44921)
Fixes an issue where a call to openConnection was not properly guarded, allowing an exception
to bubble up to the uncaught exception handler, causing test failures.

Closes #44912
2019-07-26 22:27:45 +02:00
Lisa Cawley a041d1eacf [DOCS] Updates anomaly detection terminology (#44888) 2019-07-26 11:10:49 -07:00
Lisa Cawley cef375f883 [DOCS] Updates terms in machine learning datafeed APIs (#44883) 2019-07-26 10:48:28 -07:00
Gordon Brown d4b2d21339
Add option to filter ILM explain response (#44777)
In order to make it easier to interpret the output of the ILM Explain
API, this commit adds two request parameters to that API:

- `only_managed`, which causes the response to only contain indices
  which have `index.lifecycle.name` set
- `only_errors`, which causes the response to contain only indices in an
  ILM error state

"Error state" is defined as either being in the `ERROR` step or having
`index.lifecycle.name` set to a policy that does not exist.
2019-07-26 11:57:38 -04:00
Adrien Grand 97177a3b4f Update 7.3 release notes. 2019-07-26 17:47:35 +02:00
Tanguy Leroux e1b626b947 Ensure index is green in SimpleClusterStateIT.testIndicesOptions() (#44893)
SimpleClusterStateIT testIndicesOptions failed in #44817 because it tries to close 
an index at the beginning of the test. With random index settings, it is possible that 
the index has a high number of shards (10) and replicas (1), which means that on 
CI this index can take time to be fully allocated.

The close index request can fail in the case where replicas are still recovering operations. 
Thiscommit adds a simple ensureGreen() at the beginning of the test to be sure that all 
replicas are started before trying to close the index.

closes #44817
2019-07-26 17:07:53 +02:00
Armin Braun 1340ff19bc
Fix Test Failure in ScalingThreadPoolTests (#44898) (#44901)
* Due to #44894 some constellations log a deprecation warning here now
* Fixed by checking for that
2019-07-26 17:05:50 +02:00
Przemysław Witek 79121ea127
[7.x] Implement exponential average search time per hour statistics. (#44683) (#44897) 2019-07-26 15:56:34 +02:00
István Zoltán Szabó cd7ba9f302 [DOCS] Amends data frame analytics resources, GET, and PUT API docs (#44806)
This PR addresses the feedback in  https://github.com/elastic/ml-team/issues/175#issuecomment-512215731.

* Adds an example to `analyzed_fields`
* Includes `source` and `dest` objects inline in the resource page
* Lists `model_memory_limit` in the PUT API page
* Amends the `analysis` section in the resource page
* Removes Properties headings in subsections
2019-07-26 11:52:43 +02:00
Tanguy Leroux 8848fcfb22 Ensure cluster is stable in ShrinkIndexIT.testShrinkThenSplitWithFailedNode (#44860)
The test ShrinkIndexIT.testShrinkThenSplitWithFailedNode sometimes fails 
because the resize operation is not acknowledged (see #44736). This resize 
operation creates a new index "splitagain" and it results in a cluster state 
update (TransportResizeAction uses MetaDataCreateIndexService.createIndex() 
to create the resized index). This cluster state update is expected to be 
acknowledged by all nodes (see IndexCreationTask.onAllNodesAcked()) but 
this is not always true: the data node that was just stopped in the test before 
executing the resize operation might still be considered as a "faulty" node
 (and not yet removed from the cluster nodes) by the FollowersChecker. The 
cluster state is then acked on all nodes but one, and it results in a non 
acknowledged resize operation.

This commit adds an ensureStableCluster() check after stopping the node in 
the test. The goal is to ensure that the data node has been correctly removed 
from the cluster and that all nodes are fully connected to each before moving 
forward with the resize operation.

Closes #44736
2019-07-26 10:14:27 +02:00
Jason Tedor 6ea2b5dec0
Deprecate setting processors to more than available (#44889)
Today the processors setting is permitted to be set to more than the
number of processors available to the JVM. The processors setting
directly sizes the number of threads in the various thread pools, with
most of these sizes being a linear function in the number of
processors. It doesn't make any sense to set processors very high as the
overhead from context switching amongst all the threads will overwhelm,
and changing the setting does not control how many physical CPU
resources there are on which to schedule the additional threads. We have
to draw a line somewhere and this commit deprecates setting processors
to more than the number of available processors. This is the right place
to draw the line given the linear growth as a function of processors in
most of the thread pools, and that some are capped at the number of
available processors already.
2019-07-26 17:06:44 +09:00
Ioannis Kakavas ac131f986b Document xpack.security.authc.saml.realm for Kibana (#44705)
Since 7.3, it's possible to explicitly configure the SAML realm to
be used in Kibana's configuration. This in turn, eliminates the need
of properly setting `xpack.security.public.*` settings in Kibana
and largely simplifies relevant documentation.
This also changes `xpack.security.authProviders` to
`xpack.security.authc.providers` as the former was deprecated in
favor of the latter in 7.3 in Kibana
2019-07-26 09:38:49 +03:00
Ignacio Vera 821f6f893b
Upgrade to Lucene 8.2.0 release (#44859) (#44892) 2019-07-26 08:14:59 +02:00
Nhat Nguyen d128188c28 Return seq_no and primary_term in noop update (#44603)
With this change, we will return primary_term and seq_no of the current
document if an update is detected as a noop. We already return the
version; hence we should also return seq_no and primary_term.

Relates #42497
2019-07-25 19:16:56 -04:00
Mark Vieira aebfdf1477
Fix issue with Gradle daemons hanging indefinitely on shutdown (#44867) (#44878) 2019-07-25 15:33:51 -07:00
lcawl 03605169f3 [DOCS] Clarifies dataframe transform validations 2019-07-25 15:05:42 -07:00
Yannick Welsch bd8470e738 Asynchronously connect to remote clusters (#44825)
Refactors RemoteClusterConnection so that it no longer blockingly connects to remote clusters.

Relates to #40150
2019-07-25 22:59:59 +02:00
Yannick Welsch 0ce841915c Add Clone Index API (#44267)
Adds an API to clone an index. This is similar to the index split and shrink APIs, just with the
difference that the number of primary shards is kept the same. In case where the filesystem
provides hard-linking capabilities, this is a very cheap operation.

Indexing cloning can be done by running `POST my_source_index/_clone/my_target_index` and it
supports the same options as the split and shrink APIs.

Closes #44128
2019-07-25 22:02:28 +02:00
Mark Vieira a89860160b
Expose Elasticsearch API nullability information to Kotlin compiler. (#43912) (#44518)
This change allows the Kotlin compiler to type check methods annotated with the
org.elasticsearch.common.Nullable annotation in Elasticsearch Java
APIs as described in: https://kotlinlang.org/docs/reference/java-interop.html#jsr-305-support.

(cherry picked from commit 0d0485ad9cf10e16b75b862b023b42827c375599)
2019-07-25 12:16:38 -07:00
Lisa Cawley 21971feae8 [DOCS] Updates terms in machine learning calendar APIs (#44866) 2019-07-25 11:50:34 -07:00
Przemysław Witek 8bb8543fdf
Treat PostDataActionResponse.DataCounts.bucketCount as incremental rather than absolute (total). (#44803) (#44856) 2019-07-25 20:46:56 +02:00
Ryan Ernst 03dd22b56c Add missing ZonedDateTime methods for joda compat layer (#44829)
While joda no longer exists in the apis for 7.x, the compatibility layer
still exists with helper methods mimicking the behavior of joda for
ZonedDateTime objects returned for date fields in scripts. This layer
was originally intended to be removed in 7.0, but is now likely to exist
for the lifetime of 7.x.

This commit adds missing methods from ChronoZonedDateTime to the compat
class. These methods were not part of joda, but are needed to act like a
real ZonedDateTime.

relates #44411
2019-07-25 11:45:57 -07:00
Lisa Cawley c9909b09b5 [DOCS] Adds command reference for elasticsearch-croneval (#43946) 2019-07-25 11:41:05 -07:00
Julie Tibshirani acb7f599a3 Fix an NPE when requesting inner hits and _source is disabled. (#44836)
This PR makes two changes to FetchSourceSubPhase when _source is disabled and
we're in a nested context:
* If no source filters are provided, return early to avoid an NPE.
* If there are source filters, make sure to throw an exception.

The behavior was chosen to match what currently happens in a non-nested context.
2019-07-25 10:38:00 -07:00
lcawl 6a60fd6d30 [DOCS] Fixes terms in HLRC data frame transform APIs (#44838) (#44865) 2019-07-25 10:08:38 -07:00
Nicholas Knize 48757da6e1 [GEO] Fix GeoShapeQueryBuilder to check for valid spatial relations
Refactor left out the spatial strategy check in GeoShapeQueryBuilder.relation
setter method. This commit adds that check back in.
2019-07-25 11:32:13 -05:00
Nick Knize 133f848e9f [Geo] Refactor GeoShapeQueryBuilder to derive from AbstractGeometryQueryBuilder (#44780)
Refactors GeoShapeQueryBuilder to derive from a new AbstractGeometryQueryBuilder that provides common parsing and build logic for spatial geometries. This will allow development of custom geometry queries by extending AbstractGeometryQueryBuilder preventing duplication of common spatial query logic.
2019-07-25 11:32:13 -05:00
Lisa Cawley a79adca7e3 [DOCS] Updates terms in anomaly detection job APIs (#44839) 2019-07-25 09:06:52 -07:00
István Zoltán Szabó 4a31c426e6 [DOCS] Adds allow no datafeeds query param to the GET, GET stats and STOP datafeed APIs (#44499) 2019-07-25 17:08:01 +02:00
Hendrik Muhs 2ca6306452 do not assert on indexer state (#44854)
remove the unreliable check for the state change

fixes #44813
2019-07-25 16:39:24 +02:00
David Roberts b2e969f4ba
[ML-DataFrame] Remove ID field from data frame indexer stats (#44848)
This is a followup to #44350. The indexer stats used to
be persisted standalone, but now are only persisted as
part of a state-and-stats document. During the review
of #44350 it was decided that we'll stick with this
design, so there will never be a need for an indexer
stats object to store its transform ID as it is stored
on the enclosing document. This PR removes the indexer
stats document ID.

Backport of #44768
2019-07-25 15:19:32 +01:00
Albert Zaharovits af937b14ae
SecurityIndexManager handle RuntimeEx while reading mapping (#44409)
Fixes exception handling while reading and parsing `.security-*`
mappings and templates.
2019-07-25 16:52:21 +03:00
Armin Braun 383d7b7713
Cleanup Dead Code in Index Creation (#44784) (#44822)
* Cleanup Dead Code in Index Creation
* This is all unused and the state of a create request is always `OPEN`
2019-07-25 10:50:04 +02:00
Przemysław Witek 53f409e5ae
Add result_type field to TimingStats and DatafeedTimingStats documents (#44812) (#44841) 2019-07-25 10:11:55 +02:00
Yannick Welsch e0d4544ef6 Close connection manager on current thread in RemoteClusterConnection (#44805)
The problem is that RemoteClusterConnection closes the connection manager asynchronously, which races with the threadpool being shutdown at the end of the test.

Closes #44339
Closes #44610
2019-07-25 09:34:41 +02:00
Tanguy Leroux 9944e193f9
[7.x] Clean up ShardFollowTasks for deleted indices (#44702) (#44790)
Deleting a follower index does not delete its ShardFollowTasks, potentially
leaving many persistent tasks in the cluster that cannot be allocated on
nodes and unnecessary fill the logs. This commit adds a cluster state listener
(ShardFollowTaskCleaner) that completes (with a failure) any persistent task
that refers to a non existent follower index.

I think that this bug has been introduced by #34404: before this change the
task would have been completed as failed and removed from the cluster state.

Backport of #44702 and #44801 on 7.x
2019-07-25 09:33:57 +02:00
Andrei Stefan fd74b63602 SQL: fix URI path being lost in case of hosted ES scenario (#44776)
(cherry picked from commit 06dea859e8fddada868941aaae15e83b4f64babe)
2019-07-25 10:27:51 +03:00
Andrei Stefan ee53f7e161 SQL: [Tests] Re-enable testDriverConfigurationWithSSLInURL test with more logging (#44800)
(cherry picked from commit 5b9ccd72e9a3bb65c8b7b06979a75cb795c17111)
2019-07-25 10:10:45 +03:00
Andrei Stefan 2633d11eb7
Switch from using docvalue_fields to extracting values from _source (#44062) (#44804)
* Switch from using docvalue_fields to extracting values from _source
where applicable. Doing this means parsing the _source and handling the
numbers parsing just like Elasticsearch is doing it when it's indexing
a document.
* This also introduces a minor limitation: aliases type of fields that
are NOT part of a tree of sub-fields will not be able to be retrieved
anymore. field_caps API doesn't shed any light into a field being an
alias or not and at _source parsing time there is no way to know if a
root field is an alias or not. Fields of the type "a.b.c.alias" can be
extracted from docvalue_fields, only if the field they point to can be
extracted from docvalue_fields. Also, not all fields in a hierarchy of
fields can be evaluated to being an alias.

(cherry picked from commit 8bf8a055e38f00df5f49c8d97f632f69d6e00c2c)
2019-07-25 10:02:41 +03:00
Igor Motov f9943a3e53 Geo: deprecate ShapeBuilder in QueryBuilders (#44715)
Removes unnecessary now timeline decompositions from shape builders
and deprecates ShapeBuilders in QueryBuilder in favor of libs/geo
shapes.

Relates to #40908
2019-07-24 14:27:58 -04:00
Ryan Ernst b3d8b39f29 Convert logging related gradle classes to java (#44771)
This commit converts the logging related classes (only used for vagrant)
to java from groovy.

relates #34459
2019-07-24 11:20:51 -07:00
Lisa Cawley 8bac13d5b2 [DOCS] Adds descriptions for put and start data frame transforms (#44724) 2019-07-24 11:12:56 -07:00
Enrico Zimuel a12be619f6 Fix URL documentation in API specs (#44487) 2019-07-24 10:33:15 -07:00
Lee Hinman fde6fcdfac Order ILM actions in policy definition documentation (#44773)
We already have a note that the order of actions is up to ILM for each
phase, this commit puts the actions in the same order as they will be
executed.

Resolves #41729
2019-07-24 11:28:38 -06:00
Gordon Brown 2ac54da60a
Fix swapped variables in error message (#44300)
The alias name and index were in the incorrect order in this error
message. This commit corrects the order.
2019-07-24 11:38:15 -04:00
David Turner 4cfd2fc6b2 Fix testFirstListElementsToCommaDelimitedStringReportsFirstElementsIfLong (#44785)
This test can fail (super-rarely) if it generates a list of length 11
containing a duplicate, because the `.distinct()` reduces the list length to 10
and then it is not abbreviated any more. This change generalises the test to
cover lists of any random length.
2019-07-24 16:10:41 +01:00
Armin Braun d8be9244f9
Fix Repository Cleanup Test Correctness (#44738) (#44751)
* The tests were creating the corruption and asserting its existence not on the repository base path but on a clean path.
As a result the consistency assertion on the repository wouldn't see the corruption ever an pass even if the cleanup was broken for repositories that have a non-root base path
2019-07-24 16:03:37 +02:00
Tanguy Leroux a8905ef142
[7.x] Add CloseIndexResponse to HLRC (#44349) (#44788)
The CloseIndexResponse was improved in #39687; this commit
exposes it in the HLRC.

Backport of #44349 to 7.x.
2019-07-24 15:51:01 +02:00
Andrei Stefan 04cb3aebd5 Use hasValue() methods from aggregations' InspectionHelpers (#44745)
Use InspectionHelper classes to decide if the aggregations should return null (in case there is no value) or the value itself.

(cherry picked from commit dafd7b039b0da072750e8f57e7572d24f7aad44a)
2019-07-24 16:14:45 +03:00