Commit Graph

48574 Commits

Author SHA1 Message Date
Gordon Brown 50d7424e7d
Unmute and increase logging on flaky SLM tests (#48612)
The failures in these tests have been remarkably difficult to track
down, in part because they will not reproduce locally. This commit
unmutes the flaky tests and increases logging, as well as introducing
some additional logging, to attempt to pin down the failures.
2019-10-29 13:39:19 -07:00
Christoph Büscher 1de49d8a70 Remove Ranking Evaluation API experimental status (#48603)
The API has been released long enough to remove the experimental status.
2019-10-29 20:57:39 +01:00
Gordon Brown 4bd514715d
Increase timeout in ILM doc test slightly (#48606)
This assertBusy can occasionally time out on systems under heavy load,
such as CI, so this commit increases the timeout.
2019-10-29 11:12:41 -07:00
Andrei Dan 8b22e297ed
ILM open/close steps are noop if idx is open/close (#48614) (#48640)
The open and close follower steps didn't check if the index is open,
closed respectively, before executing the open/close request.
This changes the steps to check the index state and only perform the
open/close operation if the index is not already open/closed.
2019-10-29 17:43:56 +00:00
Lisa Cawley be9df101bf [DOCS] Adds missing references to oidc realms (#48224) 2019-10-29 09:41:34 -07:00
Lisa Cawley c6f4662038
[DOCS] Updates ML PRs in 7.4.1 release notes (#48600) 2019-10-29 09:35:11 -07:00
Gordon Brown cf235796c0
Use more reliable "never run" cron pattern in tests (#48608)
The cron schedule "1 2 3 4 5 ?" will run every May 4 at 03:02:01, which
may result in unnecessary test failures once a year. This commit
switches out uses of that schedule in tests for one which will never
execute (because it specifies a day which doesn't exist, Feb. 31). Also
factors the schedule out to a constant to make the intent clearer.
2019-10-29 09:33:14 -07:00
Daniel Andion d0cbbf9d58
SQL: [Docs] Typo in HAVING section (#48609)
`HAVING` section code states `GROUP BY`  instead of the appropriate keyword.

(cherry picked from commit 9d505dc3db51e250fdf1b44e4d952dcd97bf1bc1)
2019-10-29 16:37:39 +01:00
Przemysław Witek 7c944d26c5
[7.x] Assert that the results of classification analysis can be evaluated using _evaluate API. (#48626) (#48634) 2019-10-29 16:20:56 +01:00
Nhat Nguyen 2a863ac8ff Fix testCleanUpCommitsWhenGlobalCheckpointAdvanced
Relates #48559
2019-10-29 10:39:16 -04:00
Nhat Nguyen b08cd058bc Greedily advance safe commit on new global checkpoint (#48559)
Today we won't advance the safe commit on a new global checkpoint unless 
the last commit can become safe. This is not great if we have more than
two commits as we can have a new safe commit earlier.

Closes #4853
2019-10-29 10:39:16 -04:00
Christoph Büscher 09d68e7548
Support `search_type` in Rank Evaluation API (#48542) (#48631)
Adding support for the `search_type` request parameter to the Ranking Evaluation
API since this parameter can impact the ranking and the metric score and should
be choosen in the same way when evaluating the search as later in the real
search.

Closes #48503
2019-10-29 14:54:33 +01:00
Alpar Torok d004a560df Upgrade to Gradle 5.6.3 (#48235)
This PR upgrades to [5.6.3](https://github.com/gradle/gradle/releases/tag/v5.6.3).
Nothing particualrily of interest for us, just keeping up to date.
2019-10-29 14:48:03 +02:00
Jim Ferenczi aa70ff5ea4 Fix failures in ShuffleForcedMergePolicyTests#testDiagnostics (#48627)
This commit fixes intermittent failures in ShuffleForcedMergePolicyTests#testDiagnostics
by setting a more restricted merge policy that ensures that extra merging will not happen
before the forced merge.
2019-10-29 13:46:55 +01:00
Ioannis Kakavas a0362153e2
Update oauth2-oidc-sdk and nimbus-jose-jwt (#48537) (#48628)
Update two dependencies for our OpenID Connect realm implementation
to their latest versions
2019-10-29 14:18:59 +02:00
Alpar Torok 08bf89b92b Work around incorrect lintian version on debian 10 (#48620)
* Work around incorrect lintian version on debian 10

Relates to #48573 and intended as a work around to stop the CI failures
until we fix the image.

* Need to pass --allow-downgrades
2019-10-29 14:07:03 +02:00
Jim Ferenczi c6abe58f63 Fix expectations in SearchAfter integration tests (#48372)
This commit fixes the expectations of SearchAfterIT#shouldFail regarding the inner exceptions that should be thrown
when testing failures. The exception is sometimes wrapped in a QueryShardException so this change only checks that
the toString representation contains the expected message.

Closes #43143
2019-10-29 12:37:22 +01:00
Yannick Welsch 6af3ce58f8 Filter on node id in AllocationIdIT (#48623)
Makes the assertions more targeted.

Relates #48529
2019-10-29 12:10:48 +01:00
Rory Hunter da4654527b
Improve resiliency to auto-formatting in client (#48617)
Backport of #48447. Make a number of changes so that code in the
`client` directory is more resilient to automatic formatting. This
covers:

* Literal JSON handling:
   * Reformatting multiline JSON to embed whitespace in the strings
   * Remove string concatenation where JSON fits on a single line
   * Use `String.format` for large documents with variable content
* Remove some erroneous doc refs in `QueryDSLDocumentationTests`
* Move some comments around to they aren't auto-formatted to a strange
  place
2019-10-29 10:40:54 +00:00
Rory Hunter 61a4e24ecf
Refine the auto-formatting settings (#48618)
Backport of #48446. Update the Eclipse formatter settings as follows:

* Split arguments for methods, constructors etc into separate lines when
  they wrap
* Enable "joined wrapped lines", which basically means the formatter
  will ignore existing line breaks and join and re-wrap statements. I
  had originally hesitated about enabling this, but without it the
  codebase's style won't be as consistent as it should be, defeating the
  point of all this formatting business.
* Disable formatting between `// tag::` and `// end::` comments, which
  are the code snippet markers for the docs. They need a different line
  length, which isn't possible to configure separately.
* Detect and preserve aligned line comments at the end of consecutive
  lines.
* And a bunch of other tweaks.

Also update the contribution guidelines.
2019-10-29 10:40:03 +00:00
Rory Hunter 3c77c50f5f
Improve resiliency to auto-formatting in libs, modules (#48619)
Backport of #48448. Make a number of changes so that code in the libs and
modules directories are more resilient to automatic formatting. This covers:

* Remove string concatenation where JSON fits on a single line
* Move some comments around to they aren't auto-formatted to a strange
  place
2019-10-29 10:39:34 +00:00
Jim Ferenczi 028084ce23 Add a new merge policy that interleaves old and new segments on force merge (#48533)
This change adds a new merge policy that interleaves eldest and newest segments picked by MergePolicy#findForcedMerges
and MergePolicy#findForcedDeletesMerges. This allows time-based indices, that usually have the eldest documents
first, to be efficient at finding the most recent documents too. Although we wrap this merge policy for all indices
even though it is mostly useful for time-based but there should be no overhead for other type of indices so it's simpler
than adding a setting to enable it. This change is needed in order to ensure that the optimizations that we are working
on in # remain efficient even after running a force merge.

Relates #37043
2019-10-29 10:44:56 +01:00
lgypro abddf51672 [Docs] Fix syntax error leading to wrong doc ID (#48554)
In order to index a document with id 2, the "&" should be replaced by "?"
2019-10-29 10:27:23 +01:00
Ian Danforth 82e25c4ac7 [Docs] Fix typo in suggesters search API doc (#48477) 2019-10-29 09:58:05 +01:00
Tanguy Leroux 24f6985235 Reduce allocations when draining HTTP requests bodies in repository tests (#48541)
In repository integration tests, we drain the HTTP request body before 
returning a response. Before this change this operation was done using
Streams.readFully() which uses a 8kb buffer to read the input stream, it
 now uses a 1kb for the same operation. This should reduce the allocations 
made during the tests and speed them up a bit on CI.

Co-authored-by: Armin Braun <me@obrown.io>
2019-10-29 09:15:06 +01:00
Yannick Welsch 790cfc8ad2 Fix upgraded_scroll test (#48525)
I think the problem is that the master is trying to relocate the "upgraded_scroll" shard back to
the node on which it was previously allocated, but to which it can't be allocated now due to the
shard lock being held because of an in-progress scroll. As the master keeps on retrying and
retrying (and indefinitely tries so because max_retries does not apply to relocations, it blocks
any other lower-prioritized task from completing, which leads to the rolling upgrade tests failing
(see #48395). 

Closes #48395
2019-10-29 08:10:40 +01:00
Cris da Rocha 947f89a3a1 Update troubleshooting.asciidoc (#48516) 2019-10-28 18:44:24 -07:00
Ian Danforth 4a076f5e92 [Doc] Fix typo in indices module docs (#48598) 2019-10-28 21:40:09 +01:00
Mark Vieira e5c6440a4f
Simplify usage of Gradle Shadow plugin (#48478) (#48597)
This commit simplifies and standardizes our usage of the Gradle Shadow
plugin to conform more to plugin conventions. The custom "bundle" plugin
has been removed as it's not necessary and performs the same function
as the Shadow plugin's default behavior with existing configurations.

Additionally, this removes unnecessary creation of a "nodeps" artifact,
which is unnecessary because by default project dependencies will in
fact use the non-shadowed JAR unless explicitly depending on the
"shadow" configuration.

Finally, we've cleaned up the logic used for unit testing, so we are
now correctly testing against the shadow JAR when the plugin is applied.
This better represents a real-world scenario for consumers and provides
better test coverage for incorrectly declared dependencies.

(cherry picked from commit 3698131109c7e78bdd3a3340707e1c7b4740d310)
2019-10-28 12:11:55 -07:00
Julie Tibshirani 605500df7e
Add sparse vector deprecation to 7.6 migration docs. (#48435)
This note was accidentally omitted from the deprecation PR.
2019-10-28 11:57:20 -07:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Jason Tedor 97f48168d9
Bump bundled JDK to JDK 13.0.1+9 (#48587)
This commit bumps the bundled JDK to 13.0.1+9. Since AdoptOpenJDK did
not release 13.0.1+9 for Windows, this commit also enables that the
bundled JDK version can vary by platform.
2019-10-28 12:31:33 -04:00
Gordon Brown 5021410165
Retry on RepositoryException in SLM tests (#48548)
Due to a bug, GETing a snapshot can cause a RespositoryException to be
thrown. This error is transient and should be retried, rather than
causing the test to fail. This commit converts those
RepositoryExceptions into AssertionErrors so that they will be retried
in code wrapped in assertBusy.
2019-10-28 09:24:38 -07:00
Lisa Cawley 13ce179706
[DOCS] Re-enable code snippet testing in close anomaly detection job API (#48259) (#48585) 2019-10-28 08:42:09 -07:00
Michael Basnight 1ba57dbe08 [Docs] add missing snapshot restore reference (#45256) 2019-10-28 09:55:10 -05:00
Gordon Brown c353ad71fe
Wrap ResponseException in AssertionError in ILM/CCR tests (#48489)
When checking for the existence of a document in the ILM/CCR integration
tests, `assertDocumentExists` makes an HTTP request and checks the
response code. However, if the repsonse code is not successful, the call
will throw a `ResponseException`. `assertDocumentExists` is often called
inside an `assertBusy`, and wrapping the `ResponseException` in an
`AssertionError` will allow the `assertBusy` to retry.

In particular, this fixes an issue with `testCCRUnfollowDuringSnapshot`
where the index in question may still be closed when the document is
requested.
2019-10-28 07:37:52 -07:00
Marios Trivyzas 124f6d098b
SQL: [Tests] Renable CliSecurityIT (#48581)
Seems that the issue has been fixed with: #48098

Closes: #48117
(cherry picked from commit 470362361ffce794a6a12ce7a81a8029ec7d54de)
2019-10-28 15:08:38 +01:00
David Turner e821a22580 Mute SecurityDocumentationIT#testGetUsers - see #48440 2019-10-28 14:03:01 +01:00
Przemysław Witek 7e30277a37
Mute RegressionIT.testStopAndRestart (#48575) (#48576) 2019-10-28 13:08:11 +01:00
Jason Tedor 398c8028b3
Remove locale provider picked up on backport
This commit removes a locale provider that was inadvertently introduced
after a merge conflict during a backport.
2019-10-28 07:57:44 -04:00
Jason Tedor c4fbda3310
Introduce system JVM options (#48252)
This commit moves JVM options that we are setting on behalf of the user
that we do not expect them to fiddle with out of the jvm.options
configuration file and into the JVM options parser. In this way, we
discourage fiddling with these settings, but more importantly, we ensure
that as we evolve or add to these settings that a user would pick these
pick instead of being left behind if they have a modified jvm.options
file and do not pick any new that come with the distribution.
2019-10-28 07:55:15 -04:00
Armin Braun 53a22b8a8a
Fix Validity of RepositoryDataTests Randomness (#48564) (#48566)
Trivial point, but we were only testing shard generations
for a single shard here, accidentally, and not testing the
`null` generation case at all.
2019-10-28 11:04:57 +01:00
Alpar Torok 51c1b48b24 Increase timeouts for packaging tests (#48528)
This is in preparation to move to nested virtualization which is much slower
than the bare metal setup we use right now, but parallelizes better
resulting in a net win.t
2019-10-28 06:48:11 +02:00
Alpar Torok d235576452 Add a packagingTask for every os project (#48400)
We no longer run the sample tests in CI, so it's safe to create a task
for every project.
This will make it easier to set them up in a matrix like fashion.
2019-10-28 06:48:06 +02:00
Rory Hunter 30389c6660
Improve SAML tests resiliency to auto-formatting (#48517)
Backport of #48452.

The SAML tests have large XML documents within which various parameters
are replaced. At present, if these test are auto-formatted, the XML
documents get strung out over many, many lines, and are basically
illegible.

Fix this by using named placeholders for variables, and indent the
multiline XML documents.

The tests in `SamlSpMetadataBuilderTests` deserve a special mention,
because they include a number of certificates in Base64. I extracted
these into variables, for additional legibility.
2019-10-27 16:06:23 +00:00
Nhat Nguyen 1ef87c9a68 Refresh should not acquire readLock (#48414)
Today, we hold the engine readLock while refreshing. Although this 
choice simplifies the correctness reasoning, it can block IndexShard 
from closing if warming an external reader takes time. The current
implementation of refresh does not need to hold readLock as
ReferenceManager can handle errors correctly if the engine is closed in
midway.

This PR is a prerequisite that we need to solve #47186.
2019-10-25 17:32:35 -04:00
Dan Hermann 2e3db518c9
Do not reference values for filtered settings (#48066) (#48518) 2019-10-25 16:22:11 -05:00
Michael Basnight 5228956ecc Add slices to delete and update by query in HLRC (#48420)
The slices param was missing from both delete by query and update by
query in the HLRC request converters. This commit fixes the omission.
2019-10-25 15:23:17 -05:00
William Brafford 1dd7ab830c
Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) (#48365)
Our JVM ergonomics extract max heap size from JDK PrintFlagsFinal output. 
On JDK 8, there is a system-dependent bug where memory sizes are cast to 
32-bit integers. On affected systems (namely, Windows), when 1/4 of physical
memory is more than the maximum integer value, the output of PrintFlagsFinal
will be inaccurate. In the pathological case, where the max heap size would
be a multiple of 4g, the test will fail.

The practical effect of this bug, beyond test failures, is that we may set
MaxDirectMemorySize to an incorrect value on Windows. This commit adds a
warning about this situation during startup.
2019-10-25 13:27:43 -04:00
Shaunak Kashyap d27a307379 [DOCS] Remove extraneous comma in Enrich Stats API's JSON response (#48539) 2019-10-25 12:35:50 -04:00