Commit Graph

47703 Commits

Author SHA1 Message Date
Nhat Nguyen db949847e5 Fix translog stats in testPrepareIndexForPeerRecovery (#46137)
When recovering a shard locally, we use a translog snapshot from
newSnapshotFromGen which consists of all readers from a certain
generation. In the test, we use newSnapshotFromMinSeqNo for the
expectation. The snapshot of this method includes only readers
containing operations in the requesting range.

Closes #46022
2019-08-30 08:53:27 -04:00
Alexander Reelsen 98c32c7846 Fix wrong URL encoding in watcher HTTP client (#45894)
The test assumption was calling the wrong method resulting in a URL
encoding before returning the data.

Closes #44970
2019-08-30 14:02:49 +02:00
István Zoltán Szabó 53f70ee996 [DOCS] Reformats URI search request (#45844)
* [DOCS] Reformats URI search request.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

Co-Authored-By: debadair <debadair@elastic.co>
2019-08-30 13:45:29 +02:00
Andrey Ershov 152ce62c58 Enhanced logging when transport is misconfigured to talk to HTTP port (#45964)
If a node is misconfigured to talk to remote node HTTP port (instead of
transport port) eventually it will receive an HTTP response from the
remote node on transport port (this happens when a node sends
accidentally line terminating byte in a transport request).
If this happens today it results in a non-friendly log message and a
long stack trace.
This commit adds a check if a malformed response is HTTP response. In
this case, a concise log message would appear.

(cherry picked from commit 911d02b7a9c3ce7fe316360c127a935ca4b11f37)
2019-08-30 13:02:08 +02:00
Tanguy Leroux 0c1b263e8d Inject random errors in S3BlobStoreRepositoryTests (#46125)
This commit modifies the HTTP server used in S3BlobStoreRepositoryTests 
so that it randomly returns server errors for any type of request executed by
 the SDK client. It is now possible to verify that the repository tests are s
uccessfully completed even if one or more errors were returned by the S3 
service in response of a blob upload, a blob deletion or a object listing request 
etc.

Because injecting errors forces the SDK client to retry requests, the test limits
 the maximum errors to send in response for each request at 3 retries.
2019-08-30 11:58:09 +02:00
Colin Goodheart-Smithe 57ac4f391b
Adds release notes for 7.4.0 2019-08-30 09:08:42 +01:00
Dimitris Athanasiou 5921ae53d8
[7.x][ML] Regression dependent variable must be numeric (#46072) (#46136)
* [ML] Regression dependent variable must be numeric

This adds a validation that the dependent variable of a regression
analysis must be numeric.

* Address review comments and fix some problems

In addition to addressing the review comments, this
commit fixes a few issues I found during testing.

In particular:

- if there were mappings for required fields but they were
not included we were not reporting the error
- if explicitly included fields had unsupported types we were
not reporting the error

Unfortunately, I couldn't get those fixed without refactoring
the code in `ExtractedFieldsDetector`.
2019-08-30 09:57:43 +03:00
Ryan Ernst e25f1ac8a2 Add package docs for bundled jdk location (#46153)
This commit expands the documented directory layout of the rpm and deb
packages to include the bundled jdk.

closes #45150
2019-08-29 14:14:02 -07:00
James Rodewig 2aafe8aecb
[DOCS] Bump docs version to 7.5.0 (#46143) 2019-08-29 15:10:10 -04:00
Paul Sanwald 8bdbc7d9bf
Bump version from 7.4 to 7.5 (#46142) 2019-08-29 15:03:26 -04:00
James Rodewig e7d33cc1c6 [DOCS] Add "delete index template" API docs (#46101) 2019-08-29 14:45:00 -04:00
Julie Tibshirani b5d8b364bb
Ensure top docs optimization is fully disabled for queries with unbounded max scores. (#46105) (#46139)
When a query contains a mandatory clause that doesn't track the max score per
block, we disable the max score optimization. Previously, we were doing this by
wrapping the collector with a FilterCollector that always returned
ScoreMode.COMPLETE.

However we weren't adjusting totalHitsThreshold, so the collector could still
call Scorer#setMinCompetitiveScore. It is against the method contract to call
setMinCompetitiveScore when the score mode is COMPLETE, and some scorers like
ReqOptSumScorer throw an error in this case.

This commit tries to disable the optimization by always setting
totalHitsThreshold to max int, as opposed to wrapping the collector.
2019-08-29 10:56:53 -07:00
James Rodewig c4aa8e0d90 [DOCS] Add "index template exists" API docs (#46095) 2019-08-29 13:20:20 -04:00
debadair bd50845855
[DOCS] Backporting Reformats delete by query API (#46051) (#46138)
* [DOCS] Reformats delete by query API (#46051)

* Reformats delete by query API

* Update docs/reference/docs/delete-by-query.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Updated common parms includes.

* [DOCS] Fixed issue in Common Parms.
2019-08-29 09:56:10 -07:00
Zachary Tong cf8a4171e1
Rename `data-science` plugin to `analytics` (#46133)
Rename `data-science` plugin to `analytics`. 
Also removes enabled flag. Backport of #46092
2019-08-29 12:45:39 -04:00
Simon Willnauer 9b2ea07b17
Flush engine after big merge (#46066) (#46111)
Today we might carry on a big merge uncommitted and therefore
occupy a significant amount of diskspace for quite a long time
if for instance indexing load goes down and we are not quickly
reaching the translog size threshold. This change will cause a
flush if we hit a significant merge (512MB by default) which
frees diskspace sooner.
2019-08-29 17:54:15 +02:00
James Rodewig f7a91239e7 [DOCS] Reformat update index aliases API docs (#46093) 2019-08-29 11:18:43 -04:00
James Rodewig 79f26f8308 [DOCS] Separate add index alias API docs (#46086) 2019-08-29 10:44:29 -04:00
James Rodewig 3e62cf9d74 [DOCS] Correct custom analyzer callouts (#46030) 2019-08-29 10:08:18 -04:00
James Rodewig 322d95f2f6 [DOCS] Add get index alias API docs (#46046) 2019-08-29 09:45:22 -04:00
Nhat Nguyen bb49124690 Only verify global checkpoint if translog sync occurred (#45980)
We only sync translog if the given offset hasn't synced yet. We can't
verify the global checkpoint from the latest translog checkpoint unless
a sync has occurred.

Closes #46065
Relates #45634
2019-08-29 09:44:40 -04:00
Nhat Nguyen 028e792e1d Remove already exist assertion while renew ccr lease (#46009)
If a CCR lease is disappeared while we are renewing it, then we will
issue asyncAddRetentionLease to add that lease. And if
asyncAddRetentionLease takes longer than retentionLeaseRenewInterval,
then we can issue another asyncAddRetentionLease request. One of
asyncAddRetentionLease requests will fail with
RetentionLeaseAlreadyExistsException, hence trip the assertion.

Closes #45192
2019-08-29 09:44:40 -04:00
James Rodewig 8145845fca [DOCS] Reformats analyze API (#45986) 2019-08-29 09:13:09 -04:00
István Zoltán Szabó a75348d1fb [DOCS] [PUT DFA] Documents inline the child params of source and dest (#45649)
* [DOCS] [PUT DFA] Documents inline the child params of source and dest.

* [DOCS] Fixes indentation issues and amends dfa definitions.
2019-08-29 15:09:02 +02:00
Jason Tedor d9be906afb
Start testing against AdoptOpenJDK (#45666)
This commit adds AdoptOpenJDK to the testing matrix.
2019-08-29 08:56:21 -04:00
Przemysław Witek b8a0379057
Refactor auditor-related classes (#45893) (#46120) 2019-08-29 14:21:03 +02:00
David Turner d340530a47 Avoid overshooting watermarks during relocation (#46079)
Today the `DiskThresholdDecider` attempts to account for already-relocating
shards when deciding how to allocate or relocate a shard. Its goal is to stop
relocating shards onto a node before that node exceeds the low watermark, and
to stop relocating shards away from a node as soon as the node drops below the
high watermark.

The decider handles multiple data paths by only accounting for relocating
shards that affect the appropriate data path. However, this mechanism does not
correctly account for _new_ relocating shards, which are unwittingly ignored.
This means that we may evict far too many shards from a node above the high
watermark, and may relocate far too many shards onto a node causing it to blow
right past the low watermark and potentially other watermarks too.

There are in fact two distinct issues that this PR fixes. New incoming shards
have an unknown data path until the `ClusterInfoService` refreshes its
statistics. New outgoing shards have a known data path, but we fail to account
for the change of the corresponding `ShardRouting` from `STARTED` to
`RELOCATING`, meaning that we fail to find the correct data path and treat the
path as unknown here too.

This PR also reworks the `MockDiskUsagesIT` test to avoid using fake data paths
for all shards. With the changes here, the data paths are handled in tests as
they are in production, except that their sizes are fake.

Fixes #45177
2019-08-29 12:40:55 +01:00
Tanguy Leroux b526309fbd Replace MockAmazonS3 usage in S3BlobStoreRepositoryTests by a HTTP server (#46081)
This commit removes the usage of MockAmazonS3 in S3BlobStoreRepositoryTests 
and replaces it by a HttpServer that emulates the S3 service. This allows the 
repository tests to use the real Amazon's S3 client under the hood in tests and will 
allow to test the behavior of the snapshot/restore feature for S3 repositories by 
simulating random server-side internal errors.

The HTTP server used to emulate the S3 service is intentionally simple and minimal 
to keep things understandable and maintainable. Testing full client options on the 
server side (like authentication, chunked encoding etc) remains the responsibility 
of the AmazonS3Fixture.
2019-08-29 13:16:59 +02:00
István Zoltán Szabó 93ede78b66 Revert "[DOCS] Adds search-related query parameters to the common parameters. (#46057)"
This reverts commit 95a50ae809.
2019-08-29 11:59:21 +02:00
István Zoltán Szabó 4b086fbef2 Revert "[DOCS] Reformats URI search request (#45844)"
This reverts commit 7f11c32400.
2019-08-29 11:58:28 +02:00
István Zoltán Szabó 95a50ae809 [DOCS] Adds search-related query parameters to the common parameters. (#46057)
@szabosteve Merging so I can make some additions. Will incorporate the comments from @jrodewig.
2019-08-29 11:42:08 +02:00
Przemysław Witek fbe9e8a530
Do not throw an exception if the process finished quickly but without any error. (#46073) (#46113) 2019-08-29 10:47:17 +02:00
Rory Hunter 3666bcfbd8
Handle multiple loopback addresses (#46061)
AbstractSimpleTransportTestCase.testTransportProfilesWithPortAndHost
expects a host to only have a single IPv4 loopback address, which isn't
necessarily the case. Allow for >= 1 address.

Backport of #45901.
2019-08-29 09:45:51 +01:00
Costin Leau 867cfe0223 DOC: Update SQL docs for DbVis and Workbench/J (#45981)
Refresh the setup for the new versions of DbVisualizer and SQL
Workbench/J which have Elasticsearch JDBC support out of the box.

(cherry picked from commit 6d257194c1055d060505e0faaaa37b41e21699f5)
2019-08-29 11:14:03 +03:00
István Zoltán Szabó 7f11c32400 [DOCS] Reformats URI search request (#45844)
* [DOCS] Reformats URI search request.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

Co-Authored-By: debadair <debadair@elastic.co>
2019-08-29 10:06:16 +02:00
Henning Andersen 0425f6a327 Docs _cat/health verification fix (#46064)
The _cat/health call in getting-started assumes that the master task max
wait time is always 0 (-), however, the test could sometimes run into a
short wait time (like some ms). Fixed to allow this.
2019-08-29 07:52:27 +02:00
Jason Tedor 9bc4a24118
Handle delete document level failures (#46100)
Today we assume that document failures can not occur for deletes. This
assumption is bogus, as they can fail for a variety of reasons such as
the Lucene index having reached the document limit. Because of this
assumption, we were asserting that such a document-level failure would
never happen. When this bogus assertion is violated, we fail the node, a
catastrophe. Instead, we need to treat this as a fatal engine exception.
2019-08-28 22:17:16 -04:00
Tim Brooks 70507e1041
Move netty numDirectArenas to jvm.options (#46104)
We currently configure io.netty.allocator.numDirectArenas to be 0 in the
jvm erconomics class. This is a config that we always want to set, so it
makes sense to move it to jvm.options.
2019-08-28 19:30:55 -06:00
Igor Motov 28006fe19f Fix GeoIpProcessorFactoryTests on windows (#45668)
Switches windows build to use geoip database loaded on heap instead
of memory mapping it.

Closes #44552
2019-08-28 18:02:25 -04:00
Gordon Brown 47bbd9d9a9
[7.x] Fix rollover alias in SLM history index template (#46001)
This commit adds the `rollover_alias` setting required for ILM to work
correctly to the SLM history index template and adds assertions to the
SLM integration tests to ensure that it works correctly.
2019-08-28 14:50:22 -07:00
Tal Levy a356bcff41
Add Circle Processor (#43851) (#46097)
add circle-processor that translates circles to polygons
2019-08-28 14:44:08 -07:00
Julie Tibshirani d94c4dcffb Use float instead of double for query vectors. (#46004)
Currently, when using script_score functions like cosineSimilarity, the query
vector is treated as an array of doubles. Since the stored document vectors use
floats, it seems like the least surprising behavior for the query vectors to
also be float arrays.

In addition to improving consistency, this change may help with some
optimizations we have been considering around vector dot product.
2019-08-28 11:03:14 -07:00
Jason Tedor 1249e6ba5d
Handle no-op document level failures (#46083)
Today we assume that document failures can not occur for no-ops. This
assumption is bogus, as they can fail for a variety of reasons such as
the Lucene index having reached the document limit. Because of this
assumption, we were asserting that such a document-level failure would
never happen. When this bogus assertion is violated, we fail the node, a
catastrophe. Instead, we need to treat this as a fatal engine exception.
2019-08-28 13:57:24 -04:00
Ryan Ernst 564b80303d Fix rest-api-spec dep for external plugins (#45949)
This commit fixes the maven coordinates for the rest-api-spec jar. It
was accidentally by #45107.

closes #45891
2019-08-28 10:54:03 -07:00
Ryan Ernst f20969959f Remove plugins dir reference from docs (#46047)
While the plugin installation directory used to be settable, it has not
been so for several major versions. This commit removes a lingering
reference to the plugins directory in upgrade docs.

closes #45889
2019-08-28 10:50:35 -07:00
Tanguy Leroux 9e14ffa8be Few clean ups in ESBlobStoreRepositoryIntegTestCase (#46068) 2019-08-28 16:29:46 +02:00
Martijn van Groningen f50c7cf88b
Add XContentType as parameter to HLRC ART#createServerTestInstance (#46036)
Add XContentType as parameter to the
AbstractResponseTestCase#createServerTestInstance method.

In the case a server side response class serializes xcontent as
bytes then the test needs to know what xcontent type was randomily selected.

This change is needed in #45970
2019-08-28 16:16:47 +02:00
Mark Tozzi 9ac85a4a2b Fix compilation in CumulativeCardinalityAggregatorTests 2019-08-28 09:31:48 -04:00
James Rodewig 54a882ada9 [DOCS] Add index alias exists API docs (#46042) 2019-08-28 09:13:51 -04:00
Mark Tozzi aec125faff
Support Range Fields in Histogram and Date Histogram (#46012)
Backport of 1a0dddf4ad24b3f2c751a1fe0e024fdbf8754f94 (AKA #445395)

     * Add support for a Range field ValuesSource, including decode logic for range doc values and exposing RangeType as a first class enum
     * Provide hooks in ValuesSourceConfig for aggregations to control ValuesSource class selection on missing & script values
     * Branch aggregator creation in Histogram and DateHistogram based on ValuesSource class, to enable specialization based on type.  This is similar to how Terms aggregator works.
     * Prioritize field type when available for selecting the ValuesSource class type to use for an aggregation
2019-08-28 09:06:09 -04:00