Commit Graph

39017 Commits

Author SHA1 Message Date
Simon Willnauer b50cf3c6b0
Side-step pending deletes check (#30571)
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.

Closes #30416
2018-05-15 11:51:54 +02:00
javanna 098b3b7fb4 [DOCS] Remove references to removed changelog 2018-05-15 11:47:56 +02:00
Yannick Welsch af2b9dd779 Revert "Mute ML upgrade test (#30458)"
This reverts commit 4b36ea7433.
2018-05-15 11:20:57 +02:00
David Roberts 1de5a3180a [ML] Adjust BWC version following backport of #30125 2018-05-15 09:51:56 +01:00
srini-raman 0592b685b9 [Docs] Improve section detailing translog usage (#30573) 2018-05-15 10:43:57 +02:00
Christoph Büscher bf2fb210cc
[Tests] Relax allowed delta in extended_stats aggregation (#30569)
The order in which double values are added in java can give different results
for the sum, so we need to allow a certain delta in the test assertions. The
current value was still a bit too low, resulting in rare test failures. This
change increases the allowed margin of error by a factor of ten.
2018-05-15 10:35:16 +02:00
David Roberts 50c34b2a9b
[ML] Reverse engineer Grok patterns from categorization results (#30125)
This change adds a grok_pattern field to the GET categories API
output in ML. It's calculated using the regex and examples in the
categorization result, and applying a list of candidate Grok
patterns to the bits in between the tokens that are considered to
define the category.

This can currently be considered a prototype, as the Grok patterns
it produces are not optimal. However, enough people have said it
would be useful for it to be worthwhile exposing it as experimental
functionality for interested parties to try out.
2018-05-15 09:02:38 +01:00
Tim Vernum 7dd816e77c Update build file due to doc file rename
A file with uncoverted snippets was change as part of 7f47ff9, but
build.gradle was not updated to reflect the rename.
2018-05-15 14:28:29 +10:00
Jason Tedor 0f85c6429c
Remove the changelog (#30593)
We are starting over on the changelog with a different approach. This
commit removes the existing incarnation of the changelog to remove
confusion that we need to continue adding entries to it.
2018-05-14 21:57:08 -04:00
Tim Brooks 848f240926
Fix issue with finishing handshake in ssl driver (#30580)
This is fixing an issue that has come up in some builds. In some
scenarios I see an assertion failure that we are trying to move to
application mode when we are not in handshake mode. What I think is
happening is that we are in handshake mode and have received the
completed handshake message AND an application message. While reading in
handshake mode we switch to application mode. However, there is still
data to be consumed so we attempt to continue to read in handshake mode.
This leads to us attempting to move to application mode again throwing
an assertion.

This commit fixes this by immediatly exiting the handshake mode read
method if we are not longer in handshake mode. Additionally if we swap
modes during a read we attempt to read with the new mode to see if there
is data that needs to be handled.
2018-05-14 19:19:53 -06:00
Tim Vernum 6517ac98eb
Fail if reading from closed KeyStoreWrapper (#30394)
In #28255 the implementation of the elasticsearch.keystore was changed
to no longer be built on top of a PKCS#12 keystore. A side effect of
that change was that calling getString or getFile on a closed
KeyStoreWrapper ceased to throw an exception, and would instead return
a value consisting of all 0 bytes.

This change restores the previous behaviour as closely as possible.
It is possible to retrieve the _keys_ from a closed keystore, but any
attempt to get or set the entries will throw an IllegalStateException.
2018-05-15 09:57:34 +10:00
Ioannis Kakavas 15790e1b56 Silence IndexUpgradeIT test failures. (#30430) 2018-05-15 02:14:35 +03:00
Lisa Cawley 21d67d1bd7
[DOCS] Adds release highlight pages (#30590) 2018-05-14 15:49:00 -07:00
Nik Everett 9881bfaea5
Docs: Document how to rebuild analyzers (#30498)
Adds documentation for how to rebuild all the built in analyzers and
tests for that documentation using the mechanism added in #29535.

Closes #29499
2018-05-14 18:40:54 -04:00
lcawl 7f47ff9fcd [DOCS] Fixes title capitalization in security content 2018-05-14 15:36:03 -07:00
Nik Everett 69481b4059
LLRest: Add equals and hashcode tests for Request (#30584)
Adds tests for the `Request` object's equals and hashcode to remove a
`TODO` and because we use the `equals` method in other testing.
2018-05-14 18:27:36 -04:00
Igor Motov 56d32bc8b2
SQL: Extract SQL request and response classes (#30457)
Extracts SQL request and response classes. This is the first step
towards creation of a small minimal dependencies jdbc driver.

Relates #29856
2018-05-14 16:43:29 -04:00
Jason Tedor 4e33443690
Adjust versions for resize copy settings (#30578)
Now that the change to deprecate copy settings and disallow it being
explicitly set to false is backported, this commit adjusts the BWC
versions in master.
2018-05-14 16:41:25 -04:00
Jack Conradson 1b0e6ee89f
Deprecate Empty Templates (#30194)
Deprecate the use of empty templates.  Bug fix allows empty
templates/scripts to be loaded on start up for upgrades/restarts, 
but empty templates can no longer be created.
2018-05-14 13:32:09 -07:00
Lisa Cawley 7928270610
[DOCS] Fix realm setting names (#30499) 2018-05-14 13:13:26 -07:00
Lisa Cawley fa45c6c9a6
[DOCS] Fix path info for various security files (#30502) 2018-05-14 13:07:27 -07:00
Igor Motov b30f2913cf
Docs: document precision limitations of geo_bounding_box (#30540)
The geo_bounding_box query might produce false positives alongside
the right and upper edges and false negatives alongside left and
bottom edges. This commit documents the behavior and defines the
maximum error.

Closes #29196
2018-05-14 15:54:42 -04:00
javanna df852fbdd9 Fix non existing javadocs link in RestClientTests 2018-05-14 20:24:00 +02:00
Yannick Welsch d5f028e085
Auto-expand replicas only after failing nodes (#30553)
#30423 combined auto-expansion in the same cluster state update where nodes are removed. As
the auto-expansion step would run before deassociating the dead nodes from the routing table, the
auto-expansion would possibly remove replicas from live nodes instead of dead ones. This commit
reverses the order to ensure that when nodes leave the cluster that the auto-expand-replica
functionality only triggers after failing the shards on the removed nodes. This ensures that active
shards on other live nodes are not failed if the primary resided on a now dead node.
Instead, one of the replicas on the live nodes first gets promoted to primary, and the auto-
expansion (removing replicas) only triggers in a follow-up step (but still same cluster state update).

Relates to #30456 and follow-up of #30423
2018-05-14 20:12:52 +02:00
Luca Cavanna 2f4212b80a
Fold RestGetAllSettingsAction in RestGetSettingsAction (#30561)
We currently have a separate endpoint for retrieving settings from all indices. We introduced such endpoint when removing comma-separated feature parsing for GetIndicesAction. The RestGetAllSettingsAction duplicates the code to print out the response that we already have in GetSettingsResponse (since it became a ToXContentObject), and uses the get index API internally instead of the get settings API, but the response is the same, hence we can fold get all settings and get settings in a single API, which is what this commit does.
2018-05-14 19:56:50 +02:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Ioannis Kakavas af10fd65e7 Unmute IndexUpgradeIT tests
The errors were caused because release tests would use a copy of
the public key that was formatted differently. The change to the
public key format was introduced in [1].

Release tests Jenkins job has now been updated to use the correct
key format depending on the branch they run on [2]

Closes #30430

[1] https://github.com/elastic/elasticsearch/pull/30251
[2] https://github.com/elastic/infra/pull/4944
2018-05-14 18:53:27 +03:00
Christoph Büscher cc93131318
Forbid expensive query parts in ranking evaluation (#30151)
Currently the ranking evaluation API accepts the full query syntax for
the queries specified in the evaluation set and executes them via multi
search. This potentially runs costly aggregations and suggestions too.
This change adds checks that forbid using aggregations, suggesters, 
highlighters and the explain and profile options in the queries that are 
run as part of the ranking evaluation since they are irrelevent in the 
context of this API.
2018-05-14 17:36:26 +02:00
Nik Everett 41148e4bb1
Docs: Update HighLevelRestClient migration docs (#30544)
The High Level REST Client's documentation suggested that users should
use the Low Level REST Client for index management activities. This
change removes that suggestion because the high level REST client
supports those APIs now.

This also changes the examples in the migration docs to that still use
the Low Level REST Client to use the non-deprecated varieats of
`performRequest`.
2018-05-14 11:11:27 -04:00
Nik Everett b8bf480742
Clients: Switch to new performRequest (#30543)
Switch several calls in the client projects from the deprecated
`performRequest` calls to the new version.
2018-05-14 10:37:53 -04:00
Zachary Tong 1a7110524f [TEST] Fix typo in MovAvgIT test
The second set of assertions was accidentally using the count's
moving average for the error delta in the value's moving average
assertion.  This fixes the typo, and unmutes the test.

Closes #29456
2018-05-14 13:38:36 +00:00
Alpar Torok 9a5555963b
Add missing dependencies on testClasses (#30527) 2018-05-14 16:06:56 +03:00
David Roberts 6a8aa99e3f [TEST] Mute ML test that needs updating to following ml-cpp changes
Relates #30399
2018-05-14 12:49:37 +01:00
Yannick Welsch c96f2d7bf7
Document woes between auto-expand-replicas and allocation filtering (#30531)
Relates to #2869
2018-05-14 12:14:37 +02:00
Martijn van Groningen 7b95470897
Moved tokenizers to analysis common module (#30538)
The following tokenizers were moved: classic, edge_ngram,
letter, lowercase, ngram, path_hierarchy, pattern, thai, uax_url_email and
whitespace.

Left keyword tokenizer factory in server module, because
normalizers directly depend on it.This should be addressed on a
follow up change.

Relates to #23658
2018-05-14 07:55:01 +02:00
Jason Tedor 901436148b Adjust copy settings versions
This commit adjusts the versions on the copy settings behavior now
that the default behavior is configured in 7.0.0.
2018-05-13 22:23:13 -04:00
Nhat Nguyen 73ec90f1b9 Mute ShrinkIndexIT suite
Relates #30416
2018-05-13 15:29:31 -04:00
Costin Leau 8dbe9198a1
SQL: SYS TABLES ordered according to *DBC specs (#30530)
To obey the *DBC specs, SYS TABLES returns information sorted by type
first and name second
2018-05-13 17:55:54 +03:00
Jason Tedor 593fdd40ed
Deprecate not copy settings and explicitly disallow (#30404)
We want copying settings to be the default behavior. This commit
deprecates not copying settings, and disallows explicitly not copying
settings. This gives users a transition path to the future default
behavior.
2018-05-13 10:30:05 -04:00
David Kyle 9dd629648d [ML] Improve state persistence log message 2018-05-12 09:20:08 +01:00
Ryan Ernst be8c094e8c
Build: Add mavenPlugin cluster configuration method (#30541)
This commit adds the ability to specify a plugin from maven for a
test cluster to use. Currently, only local projects may be used as
plugins, except when testing bwc, where the coordinates of the project
are used. However, that assumes all projects always keep the same
coordinates, or are even still plugins, which is no longer the case for
x-pack. The full cluster and rolling restart tests are changed to use
this new method when pulling x-pack versions before 6.3.0.
2018-05-11 21:58:18 -07:00
Nhat Nguyen 4c130a1054 Re-enable FlushIT tests
These tests failed due to in flight operations on the primary shard.
Sadly, we don't have any clue on those ops. This commit unmutes
these tests and logs the acquirers when checking for ongoing ops.

1> [2018-05-02T23:10:32,145][INFO ][o.e.i.f.FlushIT          ] Third
seal: Total shards: [2], failed: [true], reason: [[1] ongoing operations
on primary], detail: []

Relates #29392
2018-05-11 22:23:52 -04:00
Jason Tedor 07b962f31a
Bump Gradle heap to 2 GB (#30535)
We are still seeing rare failures with the Gradle heap set to 1792m,
especially on machines with high core count. Given it appears we are
close to the needed threshold, this commit bumps the heap one more time
to 2 GB.
2018-05-11 14:30:36 -04:00
Nik Everett 069fec83a8
SQL: Use request flavored methods in tests (#30345)
Modifies the SQL tests to use the new `Request` object flavored methods
introduced onto the `RestClient` in #29623. We'd like to remove the old
methods eventually so we should stop using them.
2018-05-11 13:10:01 -04:00
David Turner 15df911e41
Suppress hdfsFixture if there are spaces in the path (#30302)
HDFS sets its thread-name format based partly on a URL-encoded version of the
path, but the URL-encoding of spaces as `%20` is interpreted as a field in the
formatted string of type `2`, which is nonsensical. This change simply skips 
these tests in this case.
2018-05-11 13:36:31 +01:00
Yannick Welsch 323bcd84a0
Delete temporary blobs before creating index file (#30528)
Fixes an (un-released) bug introduced in #30332.

Closes #30507
2018-05-11 14:34:11 +02:00
Alexander Reelsen ed2d427c48
Watcher: Remove TriggerEngine.getJobCount() (#30395)
Since adding back the per-watch statistics, we do not need to access
every trigger engine implementation to get the current total job count.
This commit removes the unused methods to do so.
2018-05-11 10:04:58 +02:00
David Kyle 0ff8cf48e8
[ML] Fix wire BWC for JobUpdate (#30512)
Fix wire BWC for the JobUpdate class

Hide JobUpdate internal fields from the REST request parser
2018-05-11 09:03:45 +01:00
Yannick Welsch cdcd4a1129
Use simpler write-once semantics for FS repository (#30435)
The writeBlob method for FsBlobContainer already opens the file with StandardOpenOption.CREATE_NEW, so there's no need for an extra blobExists(blobName) check.
2018-05-11 10:02:07 +02:00
Daniel Mitterdorfer 09cf530f4b
Derive max composite buffers from max content len
With this commit we determine the maximum number of buffers that Netty keeps
while accumulating one HTTP request based on the maximum content length (default
1500 bytes, overridable with the system property `es.net.mtu`). Previously, we
kept the default value of 1024 which is too small for bulk requests which leads
to unnecessary copies of byte buffers internally.

Relates #29448
2018-05-11 10:01:09 +02:00