Commit Graph

46224 Commits

Author SHA1 Message Date
Henning Andersen 6a77dde5ea Better test diag output on OOM (#42989)
If linearizability checking fails with OOM (or other exception), we did
not get the serialized history written into the log, making it difficult
to debug in cases where the problem is hard to reproduce. Fixed to
always attempt dumping the serialized history.

Related to #42244
2019-06-11 09:48:52 +02:00
Benjamin Trent 79052050bf
[ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds (#42969) (#43069)
* [ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds

* only supporting doc_values for geo_point fields

* moving validation into GeoPointField ctor
2019-06-10 21:52:53 -05:00
Benjamin Trent eadfe05587
[ML] Changes slice specification to auto. See #42996 (#43039) (#43070) 2019-06-10 21:52:22 -05:00
Nhat Nguyen 53eb630700 Fix NPE in CcrRetentionLeaseIT (#43059)
The retention leases stats is null if the processing shard copy is being
closed. In this the case, we should check against null then retry to
avoid failing a test.

Closes #41237
2019-06-10 17:58:37 -04:00
Nhat Nguyen f2e66e22eb Increase waiting time when check retention locks (#42994)
WriteActionsTests#testBulk and WriteActionsTests#testIndex sometimes
fail with a pending retention lock. We might leak retention locks when
switching to async recovery. However, it's more likely that ongoing
recoveries prevent the retention lock from releasing.

This change increases the waiting time when we check for no pending
retention lock and also ensures no ongoing recovery in
WriteActionsTests.

Closes #41054
2019-06-10 17:58:37 -04:00
Nhat Nguyen 4191df6e1d Unmute IndexFollowingIT#testFollowIndex
Fixed in #41987
2019-06-10 17:58:37 -04:00
Benjamin Trent 0a95b8c24d
Fixing handling of auto slices in bulk scroll requests (#43050) (#43063)
* Fixing handling of auto slices in bulk scroll requests

* adjusting assertions for tests
2019-06-10 16:47:40 -05:00
Benjamin Trent 1ddc4c8fc6
[ML][Data Frame] Removes slice specification from DBQ. See #42996 (#43036) (#43055) 2019-06-10 13:40:55 -05:00
Jay Modi f150443d9a
Default distro run creates elastic-admin user (#43004)
When using gradle run by itself, this uses the default distro with a
basic license and enables security. There is a setup command to create
a elastic-admin user but only when the license is a trial license. Now
that security is available with the basic license, we should always run
this command when using the default distribution.
2019-06-10 11:49:52 -06:00
Henning Andersen 437d2d6d9f Rename processor test fix (#43035)
If the source field name is a prefix of the target field name, the
source field still exists after rename processor has run. Adjusted test
case to handle that case.
2019-06-10 19:23:05 +02:00
Dimitris Athanasiou 76a92b49a8
[ML] Get resources action should be lenient when sort field is unmapped (#42991) (#43046)
Get resources action sorts on the resource id. When there are no resources at
all, then it is possible the index does not contain a mapping for the resource
id field. In that case, the search api fails by default.

This commit adjusts the search request to ignore unmapped fields.

Closes elastic/kibana#37870
2019-06-10 19:50:19 +03:00
Henning Andersen 014bad1f61 Mute AzureDiscoveryClusterFormationTests (#43049)
Relates #43048
2019-06-10 17:39:42 +02:00
David Roberts bf5d56053a
[TEST] Adding a BWC test for ML categorization config (#42988)
This test coverage was previously missing.

Backport of #42981
2019-06-10 15:39:28 +01:00
James Rodewig 5913723788 [DOCS] Change `// TESTRESPONSE[_cat]` to `// TESTRESPONSE[non_json]` (#43006) 2019-06-10 09:53:05 -04:00
Alan Woodward 8e23e4518a Move construction of custom analyzers into AnalysisRegistry (#42940)
Both TransportAnalyzeAction and CategorizationAnalyzer have logic to build
custom analyzers for index-independent analysis. A lot of this code is duplicated,
and it requires the AnalysisRegistry to expose a number of internal provider
classes, as well as making some assumptions about when analysis components are
constructed.

This commit moves the build logic directly into AnalysisRegistry, reducing the
registry's API surface considerably.
2019-06-10 14:33:25 +01:00
Mayya Sharipova 81a3b6e2fe Improve documentation for smart_cn analyzer (#42822) 2019-06-10 08:59:30 -04:00
Alpar Torok 9def454ea9 Clean up configuration when docker isn't available (#42745)
We initially added `requireDocker` for a way for tasks to say that they
absolutely must have it, like the  build docker image tasks.
Projects using the test fixtures plugin are not in this both, as the
intent with these is that they will be skipped if docker and docker-compose
is not available.

Before this change we were lenient, the docker image build would succeed
but produce nothing. The implementation was also confusing as it was not
immediately obvious this was the case due to all the indirection in the
code.

The reason we have this leniency is that when we added the docker image
build, docker was a fairly new requirement for us, and we didn't have
it deployed in CI widely enough nor had CI configured to prefer workers
with docker when possible. We are in a much better position now.
The other reason was other stack teams running `./gradlew assemble`
in their respective CI and the possibility of breaking them if docker is
not installed. We have been advocating for building specific distros for
some time now and I will also send out an additional notice

The PR also removes the use of `requireDocker` from tests that actually
use test fixtures and are ok without it, and fixes a bug in test
fixtures that would cause incorrect configuration and allow some tasks
to run when docker was not available and they shouldn't have.

Closes  #42680 and #42829  see also #42719
2019-06-10 13:44:15 +03:00
Sachin Frayne 44aedcf97a Correct the description of generate_word_parts (#43026) 2019-06-10 11:36:31 +01:00
Sam Mingo 12962ee0a7 Update search-settings.asciidoc (#43016)
Grammar and spelling fixes
2019-06-10 10:14:03 +01:00
Shubham Vipul Majmudar b2e7045b50 Update regexp-syntax.asciidoc (#43021)
Corrects a typo.
2019-06-10 10:13:54 +01:00
Alpar Torok a5d4591af7 Don't run build-tools integ tests on FIPS (#42986)
These run Gradle and FIPS isn't supported

Closes #41721
2019-06-10 11:52:59 +03:00
David Turner 68339f90e9 Mute AutodetectMemoryLimitIT#testTooManyPartitions
Relates #43013
2019-06-10 09:20:36 +01:00
Jim Ferenczi 39cb1abc9d Fix auto fuzziness in query_string query (#42897)
Setting `auto` after the fuzzy operator (e.g. `"query": "foo~auto"`) in the `query_string`
does not take the length of the term into account when computing the distance and always use
a max distance of 1. This change fixes this disrepancy by ensuring that the term is passed when
the fuzziness is computed.
2019-06-10 10:13:16 +02:00
Vigya Sharma 25218733e6 Allow routing commands with ?retry_failed=true (#42658)
We respect allocation deciders, including the `MaxRetryAllocationDecider`, when
executing reroute commands. If you specify `?retry_failed=true` then the retry
counter is reset, but today this does not happen until after trying to execute
the reroute commands. This means that if an allocation has repeatedly failed,
but you want to take control and assign a shard to a particular node to work
around the repeated failures, you cannot execute the routing command in the
same call to `POST /_cluster/reroute` as the one that resets the failure
counter.

This commit fixes this by resetting the failure counter first, meaning that you
can now explicitly allocate a repeatedly-failed shard like this:

```
POST /_cluster/reroute?retry_failed=true
{
  "commands": [
    {
      "allocate_replica": {
        "index": "blahblah",
        "shard": 2,
        "node": "node-4"
      }
    }
  ]
}
```

Fixes #39546
2019-06-10 08:31:05 +01:00
Andrei Stefan 036f9c4a55 SQL: cover the Integer type when extracting values from _source (#42859)
* Take into consideration a wider range of Numbers when extracting the
values from source, more specifically - BigInteger and BigDecimal.

(cherry picked from commit 561b8d73dd7b03c50242e4e3f0128b2142959176)
2019-06-10 09:25:56 +03:00
Andrei Stefan 90485c6028 Since SQL is GA, remove the sql language plugin from this list (#41533)
(cherry picked from commit f715d722e8df54b3d3fe84d3ff57dfd6a198a2ac)
2019-06-10 09:25:55 +03:00
Jason Tedor b96ed1f9f7
Add note to CCR docs about mapping/alias updates
This commit adds a note to the docs clarifying that it is not possible
to manually update the mapping nor the aliases of a follower index.
2019-06-09 22:57:23 -04:00
Jason Tedor 25ca315d78
Add note to CCR docs regarding alias replication
This commit adds a note to the docs regarding the automatic replication
of aliases by a follower index from its leader index.
2019-06-09 22:55:20 -04:00
Jason Tedor 63bad28005
Do not allow modify aliases on followers (#43017)
Now that aliases are replicated by a follower from its leader, this
commit prevents directly modifying aliases on follower indices.
2019-06-09 22:53:54 -04:00
Nhat Nguyen 0ebcb21d2c Unmuted testRecoverBrokenIndexMetadata
These tests should be okay as we flush at the end of peer recovery.

Closes #40867
2019-06-09 10:26:57 -04:00
Nhat Nguyen afe65b5988 Fix assertion in ReadOnlyEngine (#43010)
We should execute the assertion before throwing an exception;
otherwise, it's a noop.
2019-06-09 10:26:56 -04:00
Jason Tedor 915d2f2daa
Refactor put mapping request validation for reuse (#43005)
This commit refactors put mapping request validation for reuse. The
concrete case that we are after here is the ability to apply effectively
the same framework to indices aliases requests. This commit refactors
the put mapping request validation framework to allow for that.
2019-06-09 10:19:04 -04:00
Nhat Nguyen 0a982fc57f Mute testLookupSeqNoByIdInLucene
Tracked at #42979
2019-06-08 00:30:12 -04:00
Jason Tedor 5f7d5920d7
Fix IOUtils#fsync on Windows fsyncing directories (#43008)
Fsyncing directories on Windows is not possible. We always suppressed
this by allowing that an AccessDeniedException is thrown when attemping
to open the directory for reading. Yet, this suppression also allowed
other IOExceptions to be suppressed, and that was a bug (e.g., the
directory not existing, or a filesystem error and reasons that we might
get an access denied there, like genuine permissions issues). This
leniency was previously removed yet it exposed that we were suppressing
this case on Windows. Rather than relying on exceptions for flow control
and continuing to suppress there, we simply return early if attempting
to fsync a directory on Windows (we will not put this burden on the
caller).
2019-06-07 23:00:26 -04:00
Jason Tedor b580677412
Fix put mapping request validators random test
This commit fixes a test bug in the request validators random test. In
particular, an assertion was not properly nested in a guard that would
ensure that was at least one failure.

Relates #43000
2019-06-07 17:47:51 -04:00
Benjamin Trent 553c73b22d
[ML][Data Frame] allow null values for aggs with sparse data (#42966) (#42998)
* [ML][Data Frame] allow null values for aggs with sparse data

* Making classes static, memory allocation optimization
2019-06-07 15:43:06 -05:00
Jason Tedor d6fe4b648d
Fix possible NPE in put mapping validators (#43000)
When applying put mapping validators, we apply all the validators in the
collection. If a failure occurs, we collect that as a top-level
exception, and suppress any additional failures into the top-level
exception. However, if a request passes the validator after a top-level
exception has been collected, we would try to suppress a null exception
into the top-level exception. This is a violation of the
Throwable#addSuppressed API. This commit addresses this, and adds test
to cover the logic of collecting the failures when validating a put
mapping request.
2019-06-07 16:24:12 -04:00
David Turner 5bc0dfce94
Improve translog corruption detection (#42980)
Today we test for translog corruption by incrementing a byte by 1 somewhere in
a file, and verify that this leads to a `TranslogCorruptionException`.
However, we rely on _all_ corruptions leading to this exception in the
`RemoveCorruptedShardDataCommand`: this command fails if a translog file
corruption leads to a different kind of exception, and `EOFException` and
`NegativeArraySizeException` are both possible. This commit strengthens the
translog corruption detection tests by simulating the following:

- a random value is written
- the file is truncated

It also makes sure that we return a `TranslogCorruptionException` in all such
cases.

Fixes #42661
Backport of #42744
2019-06-07 20:28:02 +01:00
Jason Tedor 479a1eeff6
Drop dead code for socket permissions for transport (#42990)
This code has not been needed since the removal of tribe nodes, it was
left behind when those were dropped (note that regular transport
permissions are handled through transport profiles, even if they are not
explicitly in use).
2019-06-07 15:22:10 -04:00
James Rodewig 5342616a23 [DOCS] Add explicit `articles_case` parameter to Elision Token Filter example (#42987) 2019-06-07 11:24:43 -04:00
Benjamin Trent 755ba72896
[ML][Data frame] make sure that fields exist when creating progress (#42943) (#42984) 2019-06-07 10:13:18 -05:00
markharwood 0719779a48
Search - enable low_level_cancellation by default. (#42291) (#42857)
Benchmarking on worst-case queries (max agg on match_all or popular-term query with large index) was not noticeably slower.

Closes #26258
2019-06-07 14:53:17 +01:00
jalvar08 b77be89c9a Remove Comma in Example (#41873)
The comma is there in error as there are no other parameter after 'value'
2019-06-07 08:39:27 -04:00
Jason Tedor 45bbd7f7f1
Only ignore IOException when fsyncing on dirs (#42972)
Today in the method IOUtils#fsync we ignore IOExceptions when fsyncing a
directory. However, the catch block here is too broad, for example it
would be ignoring IOExceptions when we try to open a non-existant
file. This commit addresses that by scoping the ignored exceptions only
to the invocation of FileChannel#force.
2019-06-07 08:35:09 -04:00
Henning Andersen dea935ac31
Reindex max_docs parameter name (#42942)
Previously, a reindex request had two different size specifications in the body:
* Outer level, determining the maximum documents to process
* Inside the source element, determining the scroll/batch size.

The outer level size has now been renamed to max_docs to
avoid confusion and clarify its semantics, with backwards compatibility and
deprecation warnings for using size.
Similarly, the size parameter has been renamed to max_docs for
update/delete-by-query to keep the 3 interfaces consistent.

Finally, all 3 endpoints now support max_docs in both body and URL.

Relates #24344
2019-06-07 12:16:36 +02:00
David Turner 5929803413 Relax timeout in NodeConnectionsServiceTests (#42934)
Today we assert that the connection thread is blocked by the time the test gets
to the barrier, but in fact this is not a valid assertion. The following
`Thread.sleep()` will cause the test to fail reasonably often.

```diff
diff --git a/server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java b/server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
index 193cde3180d..0e57211cec4 100644
--- a/server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
+++ b/server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
@@ -364,6 +364,7 @@ public class NodeConnectionsServiceTests extends ESTestCase {
             final CheckedRunnable<Exception> connectionBlock = nodeConnectionBlocks.get(node);
             if (connectionBlock != null) {
                 try {
+                    Thread.sleep(50);
                     connectionBlock.run();
                 } catch (Exception e) {
                     throw new AssertionError(e);
```

This change relaxes the test to allow some time for the connection thread to
hit the barrier.

Fixes #40170
2019-06-07 10:38:56 +01:00
Tim Vernum 090d42d3e6
Permit API Keys on Basic License (#42973)
Kibana alerting is going to be built using API Keys, and should be
permitted on a basic license.

This commit moves API Keys (but not Tokens) to the Basic license

Relates: elastic/kibana#36836
Backport of: #42787
2019-06-07 14:18:05 +10:00
Tim Brooks 667c613d9e
Remove `nonApplicationWrite` from `SSLDriver` (#42954)
Currently, when the SSLEngine needs to produce handshake or close data,
we must manually call the nonApplicationWrite method. However, this data
is only required when something triggers the need (starting handshake,
reading from the wire, initiating close, etc). As we have a dedicated
outbound buffer, this data can be produced automatically. Additionally,
with this refactoring, we combine handshake and application mode into a
single mode. This is necessary as there are non-application messages that
are sent post handshake in TLS 1.3. Finally, this commit modifies the
SSLDriver tests to test against TLS 1.3.
2019-06-06 17:44:40 -04:00
henryptung 61b62125b8 Wire query cache into sorting nested-filter computation (#42906)
Don't use Lucene's default query cache when filtering in sort.

Closes #42813
2019-06-06 21:16:58 +02:00
Tim Brooks d18f511327
Propogate version in reindex from remote search (#42958)
This is related to #31908. In order to use the external version in a
reindex from remote request, the search request must be configured to
request the version (as it is not returned by default). This commit
modifies the search request to request the version. Additionally, it
modifies our current reindex from remote tests to randomly use the
external version_type.
2019-06-06 14:50:06 -04:00