Commit Graph

44324 Commits

Author SHA1 Message Date
Armin Braun 63fe3c6ed6
Fix PrimaryAllocationIT Race Condition (#37355)
* Fix PrimaryAllocationIT Race Condition

* Forcing a stale primary allocation on a green index was tripping the assertion that was removed
   * Added a test that this case still errors out correctly
* Made the ability to wipe stopped datanode's data public on the internal test cluster and used it to ensure correct behaviour on the fixed test
   * Previously it simply passed because the test finished before the index went green and would NPE when the index was green at the time of the shard store status request, that would then come up empty
* Closes #37345
2019-01-11 23:26:04 +01:00
Marios Trivyzas 359222c55c
SQL: Make `FULL` non-reserved keyword in the grammar (#37377)
Since `full` can be common as a field name or part of a field name
(e.g.: `full.name` or `name.full`), it's nice if it's not a reserved
keyword of the grammar so a user can use it without resorting to quotes.

Fixes: #37376
2019-01-11 23:08:00 +02:00
Marios Trivyzas 85531f0285
SQL: [Tests] Fix and enable internalClusterTests (#37300)
SqlPlugin cannot have more than one public constructor, so for the testing
purposes the `getLicenseState()` should be overriden.

Fixes: #37191

Co-authored-by: Michael Basnight <mbasnight@gmail.com>
2019-01-11 22:43:17 +02:00
Benjamin Trent 5101e51891
ML: Fix testMigrateConfigs (#37373)
* ML: :s/execute/get

* Fixing other broken tests

* unmuting test
2019-01-11 13:29:30 -06:00
Zachary Tong de52ba1f78 Fix RollupDocumentation test to wait for job to stop
Also adds some extra state debug information to various log messages
2019-01-11 14:14:58 -05:00
Gordon Brown 827ece73c8 Mute MlConfigMigratorIT.testMigrateConfigs (#37374) 2019-01-11 11:11:58 -07:00
Nhat Nguyen 70cee18e56
Introduce StepListener (#37327)
This commit introduces StepListener which provides a simple way to write
a flow consisting of multiple asynchronous steps without having nested
callbacks.

Relates #37291
2019-01-11 13:06:17 -05:00
Gordon Brown 955d3aea19 Mute testRoundRobinWithFailures (#32190) 2019-01-11 09:38:40 -07:00
David Roberts 953fb9352f
[ML] Update error message for process update (#37363)
When this message was first added the model debug config was
the only thing that could be updated, but now more aspects of
the config can be updated so the message needs to be more
general.
2019-01-11 16:31:55 +00:00
Martijn van Groningen e4391afd98
Test fix, wait for auto follower to have stopped in the background
Relates to #36761
2019-01-11 17:26:17 +01:00
Christoph Büscher bb6d8784e7
Switch indices.get rest after backport of `include_type_name` (#37351)
With the `include_type_name` available now for indices.get on 6.x after the
backport, the corresponsing yaml test can include anything from 6.7 on.
Also changing the RestGetIndicesActionTests base test class.
2019-01-11 17:24:12 +01:00
Boaz Leskes 093d4a371c followup to #37284 with additional feedback 2019-01-11 08:07:35 -08:00
Boaz Leskes cae71cddfe
Document Seq No powered optimistic concurrency control (#37284)
Add documentation to describe the new sequence number powered optimistic concurrency control

Relates #36148
Relates #10708
2019-01-11 07:59:15 -08:00
Armin Braun 1eba1d1df9
Fix SnapshotDisruptionIT Race Condition (#37358)
* Due to a race between retrying the snapshot creation and the failed snapshot create trying to delete the snapshot there is no guarantee that the snapshot is eventually created by retries
   * Adjusted the assertion accordingly
* Closes #36779
2019-01-11 16:09:26 +01:00
Yannick Welsch f4abf9628a
Mock connections more accurately in DisruptableMockTransport (#37296)
This commit moves DisruptableMockTransport to use a more accurate representation of connection
management, which allows to use the full connection manager and does not require mocking out
any behavior. With this, we can implement restarting nodes in CoordinatorTests.
2019-01-11 16:06:48 +01:00
Jake Landis fad6329808
add link to freeze from cold phase (#37332) 2019-01-11 08:58:41 -06:00
Boaz Leskes d21df2a17a
Use Sequence number powered OCC for processing updates (#37308)
Updates perform realtime get, perform the requested update and then index the document again
using optimistic concurrency control. This PR changes the logic to use sequence numbers instead
of versioning. 

Note that the current versioning logic isn't suffering from the same problem as external OCC
requests because the get and indexing is always done on the same primary.

Relates #36148 
Relates #10708
2019-01-11 06:23:55 -08:00
Benjamin Trent 19a7e0f4eb
ML: update .ml-state actions to support > 1 index (#37307)
* ML: Updating .ml-state calls to be able to support > 1 index

* Matching bulk delete behavior with dbq

* Adjusting state name

* refreshing indices before search

* fixing line length

* adjusting index expansion options
2019-01-11 08:03:41 -06:00
David Roberts 1da59db3fb
[ML] Wait for autodetect to be ready in the datafeed (#37349)
This is a reinforcement of #37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes #36810
2019-01-11 13:22:35 +00:00
Yannick Welsch 4d3928d444 Increase timeouts in UnicastZenPingTests
Relates to #37268
2019-01-11 11:33:55 +01:00
Andrei Stefan ed2b5e80eb
Update discovery-ec2.asciidoc (#28925)
* Be more clear about the need to defining tags in EC2 to be able to filter
ES instances.
2019-01-11 11:24:31 +02:00
Alpar Torok 100ef2cc0e
Fix artifactId in plugin poms (#37315) 2019-01-11 11:23:54 +02:00
Yannick Welsch 3a929c7aea Increase assertBusy timeouts for RefreshListenersTests 2019-01-11 10:15:39 +01:00
Alpar Torok 82ca2d62de Mute CloseWhileRelocatingShardsIT.testCloseWhileRelocatingShards
Tracked by #37274
2019-01-11 10:54:53 +02:00
Jim Ferenczi b24dc2c541
[TEST] Awaits tasks termination in the RestHighLevelClient tests (#37302)
This change ensures that TasksIT#testGetValidTask and ReindexIT#testReindexTask
don't leave a non-completed task on the cluster when they finish.

Closes #35644
2019-01-11 09:47:46 +01:00
Alpar Torok 3e73911cbe Mute PrimaryAllocationIT.testForceStaleReplicaToBePromotedToPrimaryOnWrongNode
Tracking issue: #37345
2019-01-11 10:49:25 +02:00
Ignacio Vera 0a50821bb2
Geo: Do not normalize the longitude with value -180 for Lucene shapes (#37299)
Lucene based shapes should not normalize the longitude value -180 to 180.
2019-01-11 09:37:18 +01:00
Nhat Nguyen 360c430ad7
Add runAfter and notifyOnce wrapper to ActionListener (#37331)
Relates #37291
2019-01-11 03:33:06 -05:00
Alexander Reelsen 9f3da013d8
Date/Time parsing: Use java time API instead of exception handling (#37222)
* Add benchmark

* Use java time API instead of exception handling

when several formatters are used, the existing way of parsing those is
to throw an exception catch it, and try the next one. This is is
considerably slower than the approach taken in joda time, so that
indexing is reduced when a date format like `x||y` is used and y is the
date format being used.

This commit now uses the java API to parse the date by appending the
date time formatters to each other and does not rely on exception
handling.

* fix benchmark

* fix tests by changing formatter, also expose printer

* restore optional printing logic to fix tests

* fix tests

* incorporate review comments
2019-01-11 09:25:05 +01:00
Alexander Reelsen bbd093059f
Add whitelist to watcher HttpClient (#36817)
This adds a configurable whitelist to the HTTP client in watcher. By
default every URL is allowed to retain BWC. A dynamically configurable
setting named "xpack.http.whitelist" was added that allows to
configure an array of URLs, which can also contain simple regexes.

Closes #29937
2019-01-11 09:22:47 +01:00
Martijn van Groningen 37493c204d
Unmuted test now that #37239 has been merged and backported.
Relates to #37231
2019-01-11 09:02:46 +01:00
Ioannis Kakavas 80084138dd [DOCS] Fix link to role mapping doc 2019-01-11 09:22:40 +02:00
Armin Braun 860a8a7b23
Improve Precision for scaled_float (#37169)
* Use `toString` and `Bigdecimal` parsing to get intuitive behaviour for `scaled_float` as discussed in #32570
* Closes #32570
2019-01-11 08:07:55 +01:00
Jason Tedor 822626dadf
Make consistent empty retention lease supplier
This commit makes the use of empty retention lease suppliers to always
be an empty list as opposed to in some cases an empty set. This commit
is solely for consistency reasons, there is no functional change here.
2019-01-10 18:34:55 -08:00
Jason Tedor edc95c8a8e
Add validation for retention lease construction (#37312)
This commit adds some simple validation that the values input to the
retention lease constructor our valid values. We will later rely on
these values being within the validated range.
2019-01-10 18:13:05 -08:00
Jake Landis 9fd4d6ba4b
remove `+` references for xpack.monitoring.collection.indices (#37282)
`+` for index name inclusions is no longer supported for 6.x+. This
commit removes references of the `+` from the documenation. System
indices additional example is also included.

fixes #37237
2019-01-10 17:06:44 -06:00
Jack Conradson b5b93a2746
Rename ParameterMap to DeprecationMap (#37317)
Mechanical change to rename ParameterMap to DeprecationMap as this 
seems more appropriate for an extended Map to issue deprecation warnings.
2019-01-10 13:58:00 -08:00
markharwood 434430506b
Type removal - added deprecation warnings to _bulk apis (#36549)
Added warnings checks to existing tests
Added “defaultTypeIfNull” to DocWriteRequest interface so that Bulk requests can override a null choice of document type with any global custom choice.
Related to #35190
2019-01-10 21:35:19 +00:00
Lee Hinman 04dcb13ac4
Remove "beta" modifier from ILM documentation (#37326) 2019-01-10 13:38:14 -07:00
Jay Modi e6d3d85db4
Ensure latch is counted down in ssl reload test (#37313)
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See #36053
2019-01-10 13:27:25 -07:00
Costin Leau 83f7423cd6
SQL: Fix bug regarding alias fields with dots (#37279)
Field of types aliases that have dots in name are returned without a
hierarchy by field_caps, as oppose to the mapping api or field with
concrete types, which in turn breaks IndexResolver.
This commit fixes this by creating the backing hierarchy similar to the
mapping api.

Close #37224
2019-01-10 22:18:53 +02:00
Lee Hinman a2d63ecdc0
Link ILM management and policy information in ILM API documentation (#37324)
Previously these were only linked in a circuitous way rather than being
available from the top level API documentation and "Put Lifecycle" API docs.
This makes them slightly easier to find for a user.
2019-01-10 12:34:44 -07:00
lcawl 2d5a8ec59d [DOCS] Remove unused screenshots 2019-01-10 11:10:25 -08:00
Michael Basnight a57571045e
Fix rest reindex test for IPv4 addresses (#37310)
Some of our CI boxes end up giving out an IPv4 address for this
test. This commit allows both v4 and v6 addresses to be used.
2019-01-10 12:00:51 -06:00
David Roberts b65006e8cd
[ML] Fix ML memory tracker for old jobs (#37311)
Jobs created in version 6.1 or earlier can have a
null model_memory_limit.  If these are parsed from
cluster state following a full cluster restart then
we replace the null with 4096mb to make the meaning
explicit.  But if such jobs are streamed from an
old node in a mixed version cluster this does not
happen.  Therefore we need to account for the
possibility of a null model_memory_limit in the ML
memory tracker.
2019-01-10 17:28:00 +00:00
Julie Tibshirani a433c4012c
Support include_type_name in the field mapping and index template APIs. (#37210)
* Add include_type_name to the get field mappings API.
* Make sure the API specification lists include_type_name as a boolean.
* Add include_type_name to the get index templates API.
* Add include_type_name to the put index templates API.
2019-01-10 09:24:08 -08:00
Tim Brooks 9de62f1262
Increase IO direct byte buffers to 256KB (#37283)
Currently we read and write 64KB at a time in the nio libraries. As a
single byte buffer per event loop thread does not consume much memory,
there is little reason to not increase it further. This commit increases
the buffer to 256KB but still limits a single write to 64KB. The write
limit could be increased, but too high of a write limit will lead to
copying more data (if all the data is not flushed and needs to be copied
on the next call). This is something to explore in the future.
2019-01-10 09:17:20 -07:00
Jay Modi 71633775fd
Security: reorder realms based on last success (#36878)
This commit reorders the realm list for iteration based on the last
successful authentication for the given principal. This is an
optimization to prevent unnecessary iteration over realms if we can
make a smart guess on which realm to try first.
2019-01-10 09:06:16 -07:00
Ryan Ernst fcf7df3eda
Core: Handle security manager permission for deprecation log rolling (#37281)
When the deprecation log is written to within scripting support code
like ScriptDocValues, it runs under the reduces privileges of scripts.
Sometimes this can trigger log rolling, which then causes uncaught
security errors, as was handled in #28485. While doing individual
deprecation handling within each deprecation scripting location is
possible, there are a growing number of deprecations in scripts.

This commit wraps the logging call within the deprecation logger use a
doPrivileged block, just was we would within individual logging call
sites for scripting utilities.
2019-01-10 07:44:40 -08:00
Armin Braun 46237faa97
Fail Stale Primary Alloc. Req. without Data (#37226)
* Get indices shard store status before enqueuing the reallocation state update task to prevent
tasks that would fail because a node does not hold a stale copy of the shard on a best effort basis
* Closes #37098
2019-01-10 16:28:38 +01:00