Commit Graph

44810 Commits

Author SHA1 Message Date
Nhat Nguyen 820ba8169e Add retention leases replication tests (#38857)
This commit introduces the retention leases to ESIndexLevelReplicationTestCase,
then adds some tests verifying that the retention leases replication works
correctly in spite of the presence of the primary failover or out of order
delivery of retention leases sync requests.

Relates #37165
2019-02-20 19:21:00 -05:00
Tal Levy 0b676f07f6 mute failing test in 20_mix_typeless_typeful
awaits fix in #39198
2019-02-20 16:12:07 -08:00
Mirko Jotic a6ae146ccc Converting Derivative Pipeline Agg integration test into AggregatorTestsCase. (#38679)
Replicates the majority of existing Derivative pipeline integration tests into
an AggregatorTestCase, with the goal of removing the integration
tests in the near future.
2019-02-20 16:35:32 -05:00
Lisa Cawley bfc9a00d9f [DOCS] Unset machine learning upgrade mode (#39149) 2019-02-20 12:06:27 -08:00
Mark Vieira 24ac9da276
Mute CCR retention test that is consistently failing locally and in CI 2019-02-20 11:57:46 -08:00
Tal Levy cb7e3708bc
Rollup jobs should be cleaned up before indices are deleted (#38930) (#39144)
Rollup jobs should be stopped + deleted before the indices are removed.
It's possible for an active rollup job to issue a bulk request, the test
ends and the cleanup code deletes all indices.  The in-flight bulk
request will then stall + error because the index no-longer exists...
but this process might take longer than the StopRollup timeout.

Which means the test fails, and often fails several other tests since
the job is still active (e.g. other tests cannot create the same-named
job, or fail to stop the job in their cleanup because it's still stalled).

This tends to knock over several tests before the bulk finally times
out and the job shuts down.

Instead, we need to simply stop jobs first.  Inflight bulks will resolve
quickly, and we can carry on with deleting indices after the jobs are
confirmed inactive.

stop-job.asciidoc tended to trigger this issue because it executed
an async stop API and then exited, which setup the above situation. In
can and did happen with other tests though.  As an extra precaution,
the doc test was modified to substitute in wait_for_completion
to help head off these issues too.
2019-02-20 11:12:01 -08:00
Jay Modi af451459a5
Fix failures in SessionFactoryLoadBalancingTests (#39154)
This change aims to fix failures in the session factory load balancing
tests that mock failure scenarios. For these tests, we randomly shut
down ldap servers and bind a client socket to the port they were
listening on. Unfortunately, we would occasionally encounter failures
in these tests where a socket was already in use and/or the port
we expected to connect to was wrong and in fact was to one of the ldap
instances that should have been shut down.

The failures are caused by the behavior of certain operating systems
when it comes to binding ports and wildcard addresses. It is possible
for a separate application to be bound to a wildcard address and still
allow our code to bind to that port on a specific address. So when we
close the server socket and open the client socket, we are still able
to establish a connection since the other application is already
listening on that port on a wildcard address. Another variant is that
the os will allow a wildcard bind of a server socket when there is
already an application listening on that port for a specific address.

In order to do our best to prevent failures in these scenarios, this
change does the following:

1. Binds a client socket to all addresses in an awaitBusy
2. Adds assumption that we could bind all valid addresses
3. In the case that we still establish a connection to an address that
   we should not be able to, try to bind and expect a failure of not
   being connected

Closes #32190
2019-02-20 11:38:26 -07:00
Igor Motov 3d93011e32 Fix median calculation in MedianAbsoluteDeviationAggregatorTests (#38979)
Fixes an error in median calculation in
MedianAbsoluteDeviationAggregatorTests for odd number of sample points,
which causes some rare test failures.

Fixes #38937
2019-02-20 13:24:30 -05:00
Jason Tedor 90b1b36f50
Add cleanup logic to CCR retention lease test
This commit adds some logic to remove the mock transport rules at the
end of a CCR retention lease test.
2019-02-20 13:20:07 -05:00
Jason Tedor 224600f370
Bump jackson-databind version for AWS SDK (#39183)
This commit bumps the jackson-databind version for discovery-ec2 and
repository-s3 to 2.8.11.3.
2019-02-20 13:04:50 -05:00
Jason Tedor cfd7c77b64
Fix broken CCR retention lease unfollow test
This commit fixes a broken CCR retention lease unfollow test. The
problem with the test is that the random subset of shards that we picked
to disrupt would not necessarily overlap with the actual shards in
use. We could take a non-empty subset of [0, 3] (e.g., { 2 }) when the
only shard IDs in use were [0, 1]. This commit fixes this by taking into
account the number of shards in use in the test.

With this change, we also take measure to ensure that a successful
branch is tested more frequently than would otherwise be the case. On
that branch, we want to sometimes pretend that the retention lease is
already removed. The randomness here was also sometimes selecting a
subset of shards that did not overlap with the shards actually in use
during the test. While this does not break the test, it is confusing and
reduces the amount of coverage of that branch.

Relates #39185
2019-02-20 12:09:28 -05:00
Lisa Cawley d74e25a778 [DOCS] Edits the remote clusters documentation (#38996) 2019-02-20 09:01:16 -08:00
Tal Levy f1c8aa816f fix dangling tag in TasksClientDocumentationIT (#39157)
fix dangling tag in TasksClientDocumentationIT
and fix tag mentions from list-tasks to cancel-tasks
2019-02-20 08:48:58 -08:00
Jason Tedor 751c05eff9
Bump jackson-databind version for ingest-geoip (#39182)
This commit bumps the jackson-databind version for ingest-geoip to
2.8.11.3.
2019-02-20 11:40:31 -05:00
James Baiera fe9b89ef97
Add an exception throw if waiting on transport port file fails (#37574)
In the ClusterConfiguration class of the build source, there is an Ant waitfor block 
that runs to ensure that the seed node's transport ports file is created before 
trying to read it. If the wait times out, the file read fails with not much helpful info. 
This just adds a timeout property to the waitfor block and throws a descriptive 
exception instead.

Backports #37574
2019-02-20 11:09:43 -05:00
Albert Zaharovits af8ef1bb98 Do not create the missing index when invoking getRole (#39039)
In most of the places we avoid creating the `.security` index (or updating the mapping)
for read/search operations. This is more of a nit for the case of the getRole call,
that fixes a possible mapping update during a get role, and removes a dead if branch
about creating the `.security` index.
2019-02-20 17:33:10 +02:00
Martijn van Groningen 0594a467f2
Add support for ccr follow info api to HLRC. (#39115)
This API was introduces after #33824 was closed.
2019-02-20 16:10:54 +01:00
Jason Tedor 48984f647d
Mute failing CCR retention lease unfollow test
This commit mutes a CCR retention lease unfollow test that is failing
randomly, but frequently.
2019-02-20 09:47:17 -05:00
Ioannis Kakavas c783069804
Fix NPE on Stale Index in IndicesService(#39173)
This is a backport of  #38891 which closes #38845
2019-02-20 15:35:35 +02:00
Jason Tedor 09ea3ccd16
Remove retention leases when unfollowing (#39088)
This commit attempts to remove the retention leases on the leader shards
when unfollowing an index. This is best effort, since the leader might
not be available.
2019-02-20 07:06:49 -05:00
Darren Meiss eae2c9dd5c Edits to text in Phrase Suggester doc (#38966) 2019-02-20 12:35:21 +01:00
Darren Meiss 27f7ff157b Fix a typo in the cat shards API docs (#38536) 2019-02-20 11:36:46 +01:00
Adrien Grand c28b6fb9b6
Reenable test in `indices.put_mapping/20_mix_typeless_typeful.yml`. (#39056) (#39057)
This test had been disabled because of test failures, but it only affected the
6.x branch. The fix for 6.x is at #39054. On master/7.x/7.0 we can reenable the
test as-is.
2019-02-20 11:33:50 +01:00
Darren Meiss 988ce844e1 Edits to text in Bulk API doc (#39083) 2019-02-20 11:17:54 +01:00
Darren Meiss a070cd6822 Edits to text in Multi Get API doc (#39082) 2019-02-20 11:17:54 +01:00
Darren Meiss ad792669be Edits to text in Update By Query API doc (#39078) 2019-02-20 11:17:54 +01:00
Darren Meiss 07ebb8c49a Edits to text in Update API doc (#39069) 2019-02-20 11:17:54 +01:00
Darren Meiss 2f520c663c Edits to text in Delete By Query API doc (#39017) 2019-02-20 11:17:54 +01:00
Darren Meiss 1037aa0665 Edits to text in Index API doc (#39010) 2019-02-20 11:17:54 +01:00
Darren Meiss dc1ed80c73 Edits to text in API Conventions docs (#39001) 2019-02-20 11:17:54 +01:00
Andrei Stefan c1018db404 SQL: enforce JDBC driver - ES server version parity (#38972)
(cherry picked from commit 822a21f29491f295b22dacd04b747781a69ffa61)
2019-02-20 11:29:02 +02:00
Andrei Stefan 92206c8567 Added "validate.properties" property to JDBC's list of allowed properties. (#39050)
This defaults to "true" (current behavior) and will throw an exception
if there is a property that cannot be recognized. If "false", it will
ignore anything unrecognizable.

(cherry picked from commit 38fbf9792bcf4fe66bb3f17589e5fe6d29748d07)
2019-02-20 11:29:01 +02:00
David Turner efffb3d5b7 Simplify calculation in AwarenessAllocationDecider (#38091)
Today's calculation of the maximum number of shards per attribute is rather
convoluted. This commit clarifies that it returns
ceil(shardCount/numberOfAttributes).
2019-02-20 08:54:57 +00:00
Henning Andersen 00a26b9dd2 Blob store compression fix (#39073)
Blob store compression was not enabled for some of the files in
snapshots due to constructor accessing sub-class fields. Fixed to
instead accept compress field as constructor param. Also fixed chunk
size validation to work.

Deprecated repositories.fs.compress setting as well to be able to unify
in a future commit.
2019-02-20 09:24:41 +01:00
Tim Vernum 4aa50ed348
Resolve concurrency with watcher trigger service (#39164)
The watcher trigger service could attempt to modify the perWatchStats
map simultaneously from multiple threads. This would cause the
internal state to become inconsistent, in particular the count()
method may return an incorrect value for the number of watches.

This changes replaces the implementation of the map with a
ConcurrentHashMap so that its internal state remains consistent even
when accessed from mutiple threads.

Backport of: #39092
2019-02-20 19:18:00 +11:00
Ioannis Kakavas ec2b64af63 Disable date parsing test in non english locale (#39052)
This ensures we do not attempt to parse non english locale dates
in FIPS mode. The error, originally assumed to affect only Joda,
affects Java time in the same manner and manifests only with the
version of BouncyCastle FIPS certified provider we use in tests.
The upstream issue https://github.com/bcgit/bc-java/issues/405
indicates that the behavior is resolved in later versions of the
BouncyCastle library and should be tested again when the new
versions become FIPS 140 certified
2019-02-20 09:02:37 +02:00
Julie Tibshirani f5b28ca69d Enable test logging for TransformIntegrationTests#testSearchTransform.
There is already fairly detailed debug logging in the watcher framework, which
should hopefully help debug the failure.

Relates to #37882.
2019-02-19 18:15:34 -08:00
Lee Hinman 41ac6f9c55
Revert "Mute failing test 20_mix_typless_typefull (#38781)" (#38912) (#39141)
Backport of #38912

This reverts commit b91e0589fe1efdaa5061a75a3674a5cc8706b703.

This should be fixed by #38873

Resolves #38711
2019-02-19 16:14:31 -07:00
Tal Levy b5dbd1a027 AwaitsFix XPackUsageIT#testXPackCcrUsage.
relates to #39126.
2019-02-19 13:28:46 -08:00
Hendrik Muhs 50b3858f7c add version 6.6.2 2019-02-19 20:28:06 +01:00
Tal Levy f30f1fe9b6
fix RethrottleTests retry (#38978) (#39131)
the RethrottleTests assumed that tasks that were
unprepared to rethrottle would bubble up into the
Rethrottle response as an ElasticsearchException
wrapping an IllegalArgumentException. This seems to
have changed to potentially involve further levels of
wrapping.

This change makes the retry logic more resilient to
arbitrary nesting of the underlying IllegalArgumentException
2019-02-19 11:10:39 -08:00
Benjamin Trent 109b6451fd
ML refactor DatafeedsConfig(Update) so defaults are not populated in queries or aggs (#38822) (#39119)
* ML refactor DatafeedsConfig(Update) so defaults are not populated in queries or aggs

* Addressing pr feedback
2019-02-19 12:45:56 -06:00
Lee Hinman 80540a2fcc Document 'max_size' parameter as shard size for rollover (#38750)
It was not clear that this is *primary* shard size, not the entire shard size.

Resolves #37981
2019-02-19 11:17:54 -07:00
David Turner 0a9574c9d4 Add some missing toString() implementations (#39124)
Sometimes we turn objects into strings for logging or debugging using
`toString()`, but the default implementation is often unhelpful. This change
improves on this in two places I ran into recently.
2019-02-19 17:52:41 +00:00
Ioannis Kakavas 210f34f8e9 Remove BCryptTests (#39098)
This test was added to verify that we fixed a specific behavior in
Bcrypt and hasn't been running for almost 4 years now.
2019-02-19 18:12:18 +02:00
Jason Tedor fef9bdb23f
Allow retention lease operations under blocks (#39089)
This commit allows manipulating retention leases under blocks.
2019-02-19 10:26:49 -05:00
David Roberts 35e30b34f9 [ML] Stop the ML memory tracker before closing node (#39111)
The ML memory tracker does searches against ML results
and config indices.  These searches can be asynchronous,
and if they are running while the node is closing then
they can cause problems for other components.

This change adds a stop() method to the MlMemoryTracker
that waits for in-flight searches to complete.  Once
stop() has returned the MlMemoryTracker will not kick
off any new searches.

The MlLifeCycleService now calls MlMemoryTracker.stop()
before stopping stopping the node.

Fixes #37117
2019-02-19 15:12:40 +00:00
Jason Tedor 12f6963456
Fix retention leases sync on recovery test
This test had a bug. We attempt to allow only the primary to be
allocated, to force all replicas to recovery from the primary after we
had set the state of the retention leases on the primary. However, in
building the index settings, we were overwriting the settings that
exclude the replicas from being allocated. This means that some of the
replicas would end up assigned and rather than receive retention leases
during recovery, they would be part of the replication group receiving
retention leases as they are manipulated. Since retention lease renewals
are only synced periodically, this means that the replica could be
lagging a little behind in some cases leading to an assertion tripping
in the test. This commit addresses this by ensuring that the replicas
are indeed not allocated until after the retention leases are done being
manipulated on the replica. We did this by not overwriting the exclude
settings.

Closes #39105
2019-02-19 09:07:33 -05:00
David Roberts bbcdea43c5 [ML] Allow stop unassigned datafeed and relax unset upgrade mode wait (#39034)
These two changes are interlinked.

Before this change unsetting ML upgrade mode would wait for all
datafeeds to be assigned and not waiting for their corresponding
jobs to initialise.  However, this could be inappropriate, if
there was a reason other that upgrade mode why one job was unable
to be assigned or slow to start up.  Unsetting of upgrade mode
would hang in this case.

This change relaxes the condition for considering upgrade mode to
be unset to simply that an assignment attempt has been made for
each ML persistent task that did not fail because upgrade mode
was enabled.  Thus after unsetting upgrade mode there is no
guarantee that every ML persistent task is assigned, just that
each is not unassigned due to upgrade mode.

In order to make setting upgrade mode work immediately after
unsetting upgrade mode it was then also necessary to make it
possible to stop a datafeed that was not assigned.  There was
no particularly good reason why this was not allowed in the past.
It is trivial to stop an unassigned datafeed because it just
involves removing the persistent task.
2019-02-19 14:07:10 +00:00
Adrien Grand d8852b83d0
Don't swallow IOExceptions in InternalTestCluster. (#39068)
Relates #39030
2019-02-19 15:03:47 +01:00