Commit Graph

46409 Commits

Author SHA1 Message Date
Alan Woodward 9de1c69c28 IndexAnalyzers doesn't need to extend AbstractIndexComponent (#43149)
AIC doesn't add anything here, and it removes the need to pass index settings
to the constructor.
2019-06-12 17:48:31 +01:00
Jay Modi b8b24ccd8e Fix mustache search template test (#43120)
This commit fixes a race in the test for the new response format with
search templates. The test indexes a document and then executes a
search with the expectation of 0 results. In some instances, the index
will refresh prior to the search execution and 1 hit will be found
causing the test fail.

Closes #42664
2019-06-12 09:38:01 -06:00
Ryan Ernst 5be0fb32f8 Move painless context api spec to test local (#43122)
The painless context api is internal and currently meant only for use in
generating docs. This commit moves the spec file for the api so that it
is only used by the test for this api, and not externally by any clients
building from the public rest spec.
2019-06-12 08:19:45 -07:00
Jim Ferenczi 79614aeb2d SearchRequest#allowPartialSearchResults does not handle successful retries (#43095)
When set to false, allowPartialSearchResults option does not check if the
shard failures have been reseted to null. The atomic array, that is used to record
shard failures, is filled with a null value if a successful request on a shard happens
after a failure on a shard of another replica. In this case the atomic array is not empty
but contains only null values so this shouldn't be considered as a failure since all
shards are successful (some replicas have failed but the retries on another replica succeeded).
This change fixes this bug by checking the content of the atomic array and fails the request only
if allowPartialSearchResults is set to false and at least one shard failure is not null.

Closes #40743
2019-06-12 16:27:10 +02:00
Christoph Büscher 7f690e8606 Fix suggestions for empty indices (#42927)
Currently suggesters return null values on empty shards. Usually this gets replaced
by results from other non-epmty shards, but if the index is completely epmty (e.g. after
creation) the search responses "suggest" is also "null" and we don't render a corresponding
output in the REST response. This is an irritating edge case that requires special handling on
the user side (see #42473) and should be fixed.

This change makes sure every suggester type (completion, terms, phrase) returns at least an
empty skeleton suggestion output, even for empty shards. This way, even if we don't find
any suggestions anywhere, we still return and output the empty suggestion.

Closes #42473
2019-06-12 15:42:23 +02:00
Shaunak Kashyap 5ae2460782
[7.x] Metricbeat monitoring Elasticsearch: Reorder/remove steps (#42917) (#43130) 2019-06-12 06:25:30 -07:00
Yannick Welsch 110f0c5b7e Mute testDataFrameTransformCrud
Relates to #43139
2019-06-12 14:12:01 +02:00
Alexander Reelsen 6f95038001 Upgrade HPPC to version 0.8.1 (#43025) 2019-06-12 13:14:16 +02:00
Dimitris Athanasiou b28e006f7c
[ML] Lock down extraction method when possible (#43104) (#43140) 2019-06-12 14:07:17 +03:00
Yannick Welsch e5a4a2272b Wipe repositories more often (#42511)
Fixes an issue where repositories are unintentionally shared among tests (given that the repo contents is captured in a static variable on the test class, to allow "sharing" among nodes) and two tests randomly chose the same snapshot name, leading to a conflict.

Closes #42519
2019-06-12 11:58:38 +02:00
Luca Cavanna 4da0fadedc [DOCS] Clarify phrase suggester docs smoothing parameter (#42947)
Closes #28512
2019-06-12 11:25:03 +02:00
Luca Cavanna afeda1a7b9 Split search in two when made against throttled and non throttled searches (#42510)
When a search on some indices takes a long time, it may cause problems to other indices that are being searched as part of the same search request and being written to as well, because their search context needs to stay open for a long time. This is especially a problem when searching against throttled and non-throttled indices as part of the same request. The problem can be generalized though: this may happen whenever read-only indices are searched together with indices that are being written to. Search contexts staying open for a long time is only an issue for indices that are being written to, in practice.

This commit splits the search in two sub-searches: one for read-only indices, and one for ordinary indices. This way the two don't interfere with each other. The split is done only when size is greater than 0, no scroll is provided and query_then_fetch is used as search type. Otherwise, the search executes like before. Note that the returned num_reduce_phases reflect the number of reduction phases that were run. If the search is split in two, there are three reductions: one non-final for each search, and a final one that merges the results of the previous two.

Closes #40900
2019-06-12 11:25:03 +02:00
Luca Cavanna e538592652 Update max_concurrent_shard_request parameter docs (#42227)
Some of the docs were outdated as they did not mention that the limit is
not per node. Also, The default value changed.

Relates to #31206
2019-06-12 11:25:03 +02:00
Luca Cavanna 31e8bff2ac Rename SearchRequest#crossClusterSearch (#42363)
The SearchRequest#crossClusterSearch method is currently used only as
part of cross cluster search request, when minimizing roundtrips.
It will soon be used also when splitting a search into two: one for
throttled and one for non throttled indices. It will probably be used
for other usecases as well in the future, hence it makes sense to generalize its name to subSearchRequest.
2019-06-12 11:25:03 +02:00
markharwood a75964d8fd Docs change for exists query. (#43092)
Now emphasises the test is for indexed values.
Previous documentation only mentioned the state of the input JSON doc (null values) but this is only one of several reasons why an indexed value may not exist.

Closes #24256
2019-06-12 09:28:18 +01:00
Jack Conradson f7d4ca7a14 Exclude internal SQL methods from Painless API Docs (#43116) 2019-06-11 15:42:54 -07:00
Nhat Nguyen 5692be2161 Fix timing issue in CcrRetentionLeaseIT (#43054)
In these tests, we sleep for a small multiple of the renew interval,
then check that the retention leases are not changed. If a renewal
request takes longer than that interval because of GC or slow CI, then
the retention leases are not the same as before sleep. With this change,
we relax to assert that we eventually stop the renewable process.

Closes #39509
2019-06-11 18:03:16 -04:00
Benjamin Trent 7ff3d86cf0
[ML][Data Frame] adding dest.index and id validations (#43053) (#43109)
* [ML][Data Frame] adding dest.index and id validations

* adjusting message format

* Adjusting id validity pattern

* Update DataFrameStrings.java
2019-06-11 15:55:18 -05:00
Benjamin Trent e384bf0276
[ML-DataFrame] stop task at completion of data frame function (#42955) (#43114)
* stop data frame task after it finishes

* test auto stop

* adapt tests

* persist the state correctly and move stop into listener

* Calling `onStop` even if persistence fails, changing `stop` to rely on doSaveState
2019-06-11 15:55:02 -05:00
Henning Andersen 30d8085d96 scheduleAtFixedRate would hang (#42993)
Though not in use in elasticsearch currently, it seems surprising that
ThreadPool.scheduler().scheduleAtFixedRate would hang. A recurring
scheduled task is never completed (except on failure) and we test for
exceptions using RunnableFuture.get(), which hangs for periodic tasks.
Fixed by checking that task is done before calling .get().
2019-06-11 19:46:37 +02:00
Ryan Ernst 172cd4dbfa Remove description from xpack feature sets (#43065)
The description field of xpack featuresets is optionally part of the
xpack info api, when using the verbose flag. However, this information
is unnecessary, as it is better left for documentation (and the existing
descriptions describe anything meaningful). This commit removes the
description field from feature sets.
2019-06-11 09:22:58 -07:00
Henning Andersen 6108899a2d Fix unresponsive network simulation (#42579)
Unresponsive network simulation would throw away requests. However, then
we no longer have any guarantees that a transport action either succeeds
or fails, which could lead to hangs (example: unclosed IndexShard
permits).

Closes #42244
2019-06-11 17:54:48 +02:00
David Roberts d3136f99e6 [ML] Fix race condition when closing time checker (#43098)
The tests for the ML TimeoutChecker rely on threads
not being interrupted after the TimeoutChecker is
closed.  This change ensures this by making the
close() and setTimeoutExceeded() methods synchronized
so that the code inside them cannot execute
simultaneously.

Fixes #43097
2019-06-11 16:39:17 +01:00
markharwood b17fbe2933 Docs enhancement for quote_field_suffix. (#43093)
* Docs enhancement for quote_field_suffix.
Mentions the use of a fall-back field when specified field is missing.
Closes #40778
2019-06-11 16:33:12 +01:00
Jason Tedor 0eeb080a72
Do not use runtime Java when starting BWC nodes (#43071)
When starting BWC nodes, it could be that runtime Java home is set. Yet,
runtime Java home can advance beyond what a BWC node might be compatible
with. For example, if runtime Java home is set to JDK 13 and we are
starting a 7.1.2 node, we do not have any guarantees that 7.1.2 is
compatible with JDK 13 (since we never did any work to make it so). This
will continue to be the case as JDK releases advance, but we still need
to test against BWC nodes. This commit stops applying runtime Java home
when starting a BWC node. Instead, we would use the bundled JDK.
2019-06-11 10:26:57 -04:00
Martijn Laarman 2c9a6cbf69
Documents the new deprecations options on the rest-api-spec (#41444) (#43090)
* Documents the new deprecations options on the rest-api-spec

Relates #41439 #38613 #35262

* remove reference to path now that #41452 is merged, also fixed missing a comma rendering the example json invalid

* removed one more instance of path

* make sure json examples are self contained and not excerpts

(cherry picked from commit 4430f99750a3bf98373d69d2be59d71475c7aaad)
2019-06-11 15:15:11 +02:00
David Turner 04cde1d6e2 Defer reroute when nodes join (#42855)
Today the master eagerly reroutes the cluster as part of processing node joins.
However, it is not necessary to do this reroute straight away, and it is
sometimes preferable to defer it until later. For instance, when the master
wins its election it processes joins and performs a reroute, but it would be
better to defer the reroute until after the master has become properly
established.

This change defers this reroute into a separate task, and batches multiple such
tasks together.
2019-06-11 14:00:18 +01:00
Nhat Nguyen 5d3849215b CCR should not replicate private/internal settings (#43067)
With this change, CCR will not replicate internal or private settings to
follower indices.

Closes #41268
2019-06-11 06:59:09 -04:00
Martijn Laarman cb7ce865b7
remove path from rest-api-spec (#41452) (#43084)
(cherry picked from commit f5fde1d0843d2f0f53d3b9a15b9cfc8b94471ab7)
2019-06-11 12:52:36 +02:00
Henning Andersen 1c7cd09375 Enable TRACE for testRecoverBrokenIndexMetadata (#43081)
Relates to #43034
2019-06-11 12:38:48 +02:00
Jim Ferenczi 900eb4f882 Handle empty terms index in TermsSliceQuery (#43078)
#40741 introduced a merge policy that can drop the postings for the `_id`
field on soft deleted documents. The TermsSliceQuery assumes that every document
has has an entry in the postings for that field so it doesn't check if the terms
index exists or not. This change fixes this bug by checking if the terms index for
the `_id` field is null and ignore the segment entirely if it's the case. This should
be harmless since segments without an `_id` terms index should only contain soft deleted
documents.

Closes #42996
2019-06-11 12:01:53 +02:00
Andrei Stefan 8de65daa45 Rename TESTRESPONSE[_cat] to TESTRESPONSE[non_json] (#43087)
(cherry picked from commit 897b24e0563f59c03e85096fdb64cbc1dd1a5d60)
2019-06-11 12:40:00 +03:00
Andrei Stefan 5b35ec1d9b Restructure the SQL Language section to have proper sub-sections (#43007)
Rest docs page update
- have the section be on separate pages
- add an Overview page
- add other formats examples

(cherry picked from commit 309bd691ff3f8625f67ca09fc1dd8e265f7e6c92)
2019-06-11 12:39:59 +03:00
Andrei Stefan 4a3287836d SQL: Clarify that the connections the jdbc driver creates are not pooled (#42992)
(cherry picked from commit 406d5281bdfe682fb7ec9fefcdb61cce1b9e7270)
2019-06-11 12:39:58 +03:00
Ioannis Kakavas 1776d6e055 Refresh remote JWKs on all errors (#42850)
It turns out that key rotation on the OP, can manifest as both
a BadJWSException and a BadJOSEException in nimbus-jose-jwt. As
such we cannot depend on matching only BadJWSExceptions to
determine if we should poll the remote JWKs for an update.

This has the side-effect that a remote JWKs source will be polled
exactly one additional time too for errors that have to do with
configuration, or for errors that might be caused by not synched
clocks, forged JWTs, etc. ( These will throw a BadJWTException
which extends BadJOSEException also )
2019-06-11 11:01:54 +03:00
Henning Andersen 6a77dde5ea Better test diag output on OOM (#42989)
If linearizability checking fails with OOM (or other exception), we did
not get the serialized history written into the log, making it difficult
to debug in cases where the problem is hard to reproduce. Fixed to
always attempt dumping the serialized history.

Related to #42244
2019-06-11 09:48:52 +02:00
Benjamin Trent 79052050bf
[ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds (#42969) (#43069)
* [ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds

* only supporting doc_values for geo_point fields

* moving validation into GeoPointField ctor
2019-06-10 21:52:53 -05:00
Benjamin Trent eadfe05587
[ML] Changes slice specification to auto. See #42996 (#43039) (#43070) 2019-06-10 21:52:22 -05:00
Nhat Nguyen 53eb630700 Fix NPE in CcrRetentionLeaseIT (#43059)
The retention leases stats is null if the processing shard copy is being
closed. In this the case, we should check against null then retry to
avoid failing a test.

Closes #41237
2019-06-10 17:58:37 -04:00
Nhat Nguyen f2e66e22eb Increase waiting time when check retention locks (#42994)
WriteActionsTests#testBulk and WriteActionsTests#testIndex sometimes
fail with a pending retention lock. We might leak retention locks when
switching to async recovery. However, it's more likely that ongoing
recoveries prevent the retention lock from releasing.

This change increases the waiting time when we check for no pending
retention lock and also ensures no ongoing recovery in
WriteActionsTests.

Closes #41054
2019-06-10 17:58:37 -04:00
Nhat Nguyen 4191df6e1d Unmute IndexFollowingIT#testFollowIndex
Fixed in #41987
2019-06-10 17:58:37 -04:00
Benjamin Trent 0a95b8c24d
Fixing handling of auto slices in bulk scroll requests (#43050) (#43063)
* Fixing handling of auto slices in bulk scroll requests

* adjusting assertions for tests
2019-06-10 16:47:40 -05:00
Benjamin Trent 1ddc4c8fc6
[ML][Data Frame] Removes slice specification from DBQ. See #42996 (#43036) (#43055) 2019-06-10 13:40:55 -05:00
Jay Modi f150443d9a
Default distro run creates elastic-admin user (#43004)
When using gradle run by itself, this uses the default distro with a
basic license and enables security. There is a setup command to create
a elastic-admin user but only when the license is a trial license. Now
that security is available with the basic license, we should always run
this command when using the default distribution.
2019-06-10 11:49:52 -06:00
Henning Andersen 437d2d6d9f Rename processor test fix (#43035)
If the source field name is a prefix of the target field name, the
source field still exists after rename processor has run. Adjusted test
case to handle that case.
2019-06-10 19:23:05 +02:00
Dimitris Athanasiou 76a92b49a8
[ML] Get resources action should be lenient when sort field is unmapped (#42991) (#43046)
Get resources action sorts on the resource id. When there are no resources at
all, then it is possible the index does not contain a mapping for the resource
id field. In that case, the search api fails by default.

This commit adjusts the search request to ignore unmapped fields.

Closes elastic/kibana#37870
2019-06-10 19:50:19 +03:00
Henning Andersen 014bad1f61 Mute AzureDiscoveryClusterFormationTests (#43049)
Relates #43048
2019-06-10 17:39:42 +02:00
David Roberts bf5d56053a
[TEST] Adding a BWC test for ML categorization config (#42988)
This test coverage was previously missing.

Backport of #42981
2019-06-10 15:39:28 +01:00
James Rodewig 5913723788 [DOCS] Change `// TESTRESPONSE[_cat]` to `// TESTRESPONSE[non_json]` (#43006) 2019-06-10 09:53:05 -04:00
Alan Woodward 8e23e4518a Move construction of custom analyzers into AnalysisRegistry (#42940)
Both TransportAnalyzeAction and CategorizationAnalyzer have logic to build
custom analyzers for index-independent analysis. A lot of this code is duplicated,
and it requires the AnalysisRegistry to expose a number of internal provider
classes, as well as making some assumptions about when analysis components are
constructed.

This commit moves the build logic directly into AnalysisRegistry, reducing the
registry's API surface considerably.
2019-06-10 14:33:25 +01:00