Commit Graph

1103 Commits

Author SHA1 Message Date
Nhat Nguyen 4e5c305225 Mute testRelocationWithConcurrentIndexing
Tracked at #34950
2018-10-28 18:51:48 -04:00
Nik Everett 9f87fdc7ab
Drop deprecationLogger from AbstractComponent (#34859)
Drops the `deprecationLogger` from `AbstractComponent`, moving it to
places where we need it. This saves us from building a bunch of
`DeprecationLogger`s that we don't need.

Relates to #34488
2018-10-26 15:40:16 -04:00
Tanguy Leroux c42f350a81
[Test] Fix FullClusterRestartIT.testShrink() with copy_settings param (#34853)
The pull request #34338 added strict deprecation mode to the REST tests
and adds the copy_settings param when testing the shrink of an index.

This parameter has been added in 6.4.0 and will be removed in 8.0, so
the test now needs to take care of the old cluster version when adding
the copy_settings param.
2018-10-26 12:57:31 +02:00
Tal Levy e1fdd00420
Lowercase static final DeprecationLogger instance names (#34887)
After discussing on the team's FixItFriday, we concluded that
static final instance variables that are mutable should be lowercased.

Historically, DeprecationLogger was uppercased more frequently than lowercased.
2018-10-25 21:12:19 -07:00
Tanguy Leroux a69c540f12 [Test] Mute FullClusterRestartIT.testShrink() until test is fixed
See https://github.com/elastic/elasticsearch/pull/34853
2018-10-25 13:37:23 +02:00
lipsill d5ad3de42e [test] Introduce strict deprecation mode for REST tests (#34338)
#33708 introduced a strict deprecation mode that makes a REST request
fail if there is a warning header in the response returned by
Elasticsearch (usually a deprecation message signaling that a feature
or a field has been deprecated).

This change adds the strict deprecation mode into the REST integration
tests, and makes the tests fail if a deprecated feature is used. Also
any test using a deprecated feature has been modified to pass the build.

The YAML integration tests already analyzed HTTP warnings so they do
not use this mode, keeping their "expected vs actual" behavior.
2018-10-24 08:21:24 -04:00
Nhat Nguyen 100a18bd8d
TEST: Fix indentation in FullClusterRestartIT (#34420) 2018-10-12 15:11:55 -04:00
Ioannis Kakavas 080ddd5d9c
[WIP] Ingest Attachement: Upgrade tika to v1.19.1 (#33896)
Upgrades Tika to 1.19.1 and relevant transitive dependencies

Resolves: #31456, #31305
2018-10-12 17:09:14 +01:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Nhat Nguyen 7dbc403226
TEST: Index diff num docs in rolling upgrade tests (#34191)
Today we index the same number of documents (50 documents) in each round
of the rolling upgrade tests. If the actual count does not match, we can
not guess the problematic round.

Relates #27650
2018-10-02 09:04:03 -04:00
Vladimir Dolzhenko 2e2ae19b97
drop elasticsearch-translog for 7.0 (#33373)
#32281 adds elasticsearch-shard to provide bwc version of elasticsearch-translog for 6.x; have to remove elasticsearch-translog for 7.0

Relates to #31389
2018-10-01 16:21:14 +02:00
Nik Everett ddce9704d4
Logging: Drop two deprecated methods (#34055)
This drops two deprecated methods from `ESLoggerFactory`, switching all
calls to those methods to calls to methods of the same name on
`LogManager`.
2018-09-26 11:20:52 -04:00
Nhat Nguyen 002f763c48
Restore local history from translog on promotion (#33616)
If a shard was serving as a replica when another shard was promoted to
primary, then its Lucene index was reset to the global checkpoint.
However, if the new primary fails before the primary/replica resync
completes and we are now being promoted, we have to restore the reverted
operations by replaying the translog to avoid losing acknowledged writes.

Relates #33473
Relates #32867
2018-09-20 13:21:11 -04:00
Nik Everett 26c4f1fb6c
Core: Default node.name to the hostname (#33677)
Changes the default of the `node.name` setting to the hostname of the
machine on which Elasticsearch is running. Previously it was the first 8
characters of the node id. This had the advantage of producing a unique
name even when the node name isn't configured but the disadvantage of
being unrecognizable and not being available until fairly late in the
startup process. Of particular interest is that it isn't available until
after logging is configured. This forces us to use a volatile read
whenever we add the node name to the log.

Using the hostname is available immediately on startup and is generally
recognizable but has the disadvantage of not being unique when run on
machines that don't set their hostname or when multiple elasticsearch
processes are run on the same host. I believe that, taken together, it
is better to default to the hostname.

1. Running multiple copies of Elasticsearch on the same node is a fairly
advanced feature. We do it all the as part of the elasticsearch build
for testing but we make sure to set the node name then.
2. That the node.name defaults to some flavor of "localhost" on an
unconfigured box feels like it isn't going to come up too much in
production. I expect most production deployments to at least set the
hostname.

As a bonus, production deployments need no longer set the node name in
most cases. At least in my experience most folks set it to the hostname
anyway.
2018-09-19 15:21:29 -04:00
Vladimir Dolzhenko a3e8b831ee
add elasticsearch-shard tool (#32281)
Relates #31389
2018-09-19 10:28:22 +02:00
David Turner 421f58e172
Remove discovery-file plugin (#33257)
In #33241 we moved the file-based discovery functionality to core
Elasticsearch, but preserved the `discovery-file` plugin, and support for the
existing location of the `unicast_hosts.txt` file, for BWC reasons. This commit
completes the removal of this plugin.
2018-09-18 12:01:16 +01:00
markharwood 2fa09f062e
New plugin - Annotated_text field type (#30364)
New plugin for annotated_text field type.
Largely a copy of `text` field type but adds ability to include markdown-like syntax in the text.
The “AnnotatedText” class parses text+markup and converts into plain text and AnnotationTokens.
The annotation token values are injected unchanged alongside the regular text tokens to provide a
form of additional indexed overlay useful in positional searches and highlighting.
Annotated_text fields do not support fielddata as we want to phase this out.
Also includes a new "annotated" highlighter type that retains annotations and merges in search
hits as additional annotation markup.

Closes #29467
2018-09-18 10:25:27 +01:00
Or Bin a5bad4d92c Docs: Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...' (#33744)
Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...'

Closes #33728
2018-09-17 15:35:54 -04:00
Jason Tedor f7c131f118
Revert "Mute FullClusterRestartSettingsUpgradeIT"
This reverts commit e9826164bd.
2018-09-14 05:55:46 -04:00
Jason Tedor d65ff17b7e
Adjust BWC version on settings upgrade test (#33650)
The skip_unavailable setting did not exist until 6.1.0. This means that
we need to skip this test on versions prior to 6.1.0. We need to use
this setting because otherwise we will fail startup without it (since we
are not setting up a real remote cluster connection). This commit adds a
skip for all versions prior to 6.1.0.
2018-09-14 05:55:01 -04:00
Igor Motov e9826164bd Mute FullClusterRestartSettingsUpgradeIT
Tracked by #33697
2018-09-14 11:21:44 +04:00
Nhat Nguyen dcbbaad296 Mute testRecoveryWithConcurrentIndexing
Relates #33473
Relates #33616
2018-09-13 11:53:43 -04:00
Jason Tedor dcdacd2f3f
Lower version on full cluster restart settings test
The change to upgrade cross-cluster search settings was backported to
6.5.0. Therefore, this assumption needs to be reduced to the latest
version where settings were not automatically upgraded.
2018-09-12 01:19:05 -04:00
Jason Tedor c74c46edc3
Upgrade remote cluster settings (#33537)
This commit adds settings upgraders for the search.remote.* settings
that can be in the cluster state to automatically upgrade these settings
to cluster.remote.*. Because of the infrastructure that we have here,
these settings can be upgraded when recovering the cluster state, but
also when a user tries to make a dynamic update for these settings.
2018-09-12 01:14:43 -04:00
Jason Tedor 9f8dff9281
Remove debug logging in full cluster restart tests (#33612)
These logs are incredibly verbose, and it makes chasing normal failures
burdensome. This commit removes the debug logging, which can be
reenabled again if needed.
2018-09-11 19:11:30 -04:00
Jason Tedor ea3fdc90c6
Add full cluster restart base class (#33577)
This commit adds a base class for full cluster restart tests.
2018-09-10 20:06:42 -04:00
Nik Everett ab9e2cddf1 Logging: Clean up skipping test
Clean up on top of the last fix: if we skip the entire test case then
the test run would fail because we skipped all the tests. This adds a
dummy test case to prevent that. It is a fairly nasty work around I plan
to work on something that makes this not required any more anyway.
2018-09-07 21:54:38 -04:00
Nik Everett 97736ac46a Logging: Skip test if it'd fail
If we're running on a platform where we can't install syscall filters
Elasticsearch logs a message before it reads the data directory to get
the node name. Because that log message doesn't have a node name this
test will fail. Since we mostly run the test on OSes where we *can*
install the syscall filters we can fairly safely skip the test on OSes
where we can't install the syscall filters.

Closes #33540
2018-09-07 21:42:58 -04:00
Jason Tedor 9a404f3def
Include fallback settings when checking dependencies (#33522)
Today when checking settings dependencies, we do not check if fallback
settings are present. This means, for example, that if
cluster.remote.*.seeds falls back to search.remote.*.seeds, and
cluster.remote.*.skip_unavailable and search.remote.*.skip_unavailable
depend on cluster.remote.*.seeds, and we have set search.remote.*.seeds
and search.remote.*.skip_unavailable, then validation will fail because
it is expected that cluster.ermote.*.seeds is set here. This commit
addresses this by also checking fallback settings when validating
dependencies. To do this, we adjust the settings exist method to also
check for fallback settings, a case that it was not handling previously.
2018-09-07 20:09:53 -04:00
Nik Everett 190ea9a6de
Logging: Configure the node name when we have it (#32983)
Change the logging infrastructure to handle when the node name isn't
available in `elasticsearch.yml`. In that case the node name is not
available until long after logging is configured. The biggest change is
that the node name logging no longer fixed at pattern build time.
Instead it is read from a `SetOnce` on every print. If it is unset it is
printed as `unknown` so we have something that fits in the pattern.
On normal startup we don't log anything until the node name is available
so we never see the `unknown`s.
2018-09-07 14:31:23 -04:00
Nik Everett 0d45752e50
Fix IndexMetaData loads after rollover (#33394)
When we rollover and index we write the conditions of the rollover that
the old index met into the old index. Loading this index metadata
requires a working `NamedXContentRegistry` that has been populated with
parsers from the rollover infrastructure. We had a few loads that didn't
use a working `NamedXContentRegistry` and so would fail if they ever
encountered an index that had been rolled over. Here are the locations
of the loads and how I fixed them:

* IndexFolderUpgrader - removed entirely. It existed to support opening
indices made in Elasticsearch 2.x. Since we only need this change as far
back as 6.4.1 which will supports reading from indices created as far
back as 5.0.0 we should be good here.
* TransportNodesListGatewayStartedShards - wired the
`NamedXContentRegistry` into place.
* TransportNodesListShardStoreMetaData - wired the
`NamedXContentRegistry` into place.
* OldIndexUtils - removed entirely. It existed to support the zip based
index backwards compatibility tests which we've since replaced with code
that actually runs old versions of Elasticsearch.

In addition to fixing the actual problem I added full cluster restart
integration tests for rollover which would have caught this problem and
I added an extra assertion to IndexMetaData's deserialization code which
will trip if we try to deserialize and index's metadata without a fully
formed `NamedXContentRegistry`. It won't catch if use the *wrong*
`NamedXContentRegistry` but it is better than nothing.

Closes #33316
2018-09-06 17:55:24 -04:00
Jason Tedor d71ced1b00
Generalize search.remote settings to cluster.remote (#33413)
With features like CCR building on the CCS infrastructure, the settings
prefix search.remote makes less sense as the namespace for these remote
cluster settings than does a more general namespace like
cluster.remote. This commit replaces these settings with cluster.remote
with a fallback to the deprecated settings search.remote.
2018-09-05 20:43:44 -04:00
Jim Ferenczi f4e9729d64
Remove unsupported Version.V_5_* (#32937)
This change removes the es 5x version constants and their usages.
2018-08-24 09:51:21 +02:00
Jason Tedor 67bfb765ee
Refactor Netty4Utils#maybeDie (#33021)
In our Netty layer we have had to take extra precautions against Netty
catching throwables which prevents them from reaching the uncaught
exception handler. This code has taken on additional uses in NIO layer
and now in the scheduler engine because there are other components in
stack traces that could catch throwables and suppress them from reaching
the uncaught exception handler. This commit is a simple cleanup of the
iterative evolution of this code to refactor all uses into a single
method in ExceptionsHelper.
2018-08-22 10:18:07 -04:00
Alpar Torok 82d10b484a
Run forbidden api checks with runtimeJavaVersion (#32947)
Run forbidden APIs checks with runtime hava version
2018-08-22 09:05:22 +03:00
Nik Everett 2c81d7f77e
Build: Rework shadow plugin configuration (#32409)
This reworks how we configure the `shadow` plugin in the build. The major
change is that we no longer bundle dependencies in the `compile` configuration,
instead we bundle dependencies in the new `bundle` configuration. This feels
more right because it is a little more "opt in" rather than "opt out" and the
name of the `bundle` configuration is a little more obvious.

As an neat side effect of this, the `runtimeElements` configuration used when
one project depends on another now contains exactly the dependencies needed
to run the project so you no longer need to reference projects that use the
shadow plugin like this:

```
testCompile project(path: ':client:rest-high-level', configuration: 'shadow')
```

You can instead use the much more normal:

```
testCompile "org.elasticsearch.client:elasticsearch-rest-high-level-client:${version}"
```
2018-08-21 20:03:28 -04:00
Armin Braun 986c55b830
INGEST: Add Configuration Except. Data to Metdata (#32322)
* closes #27728
2018-08-15 19:02:19 +02:00
Ioannis Kakavas 106a4a6a48
Unmute WildFly tests in FIPS JVM (#32814)
WildflyIT test fails in a FIPS JVM due to the amount of output in stderr. The excessive stderr output is due to https://bugs.openjdk.java.net/browse/JDK-8202893 and is not an indication of a failure that should be tracked.
This commit adjusts the limit to something more lenient that would allow the test to succeed.
Reverts #32543
2018-08-13 21:08:10 +03:00
Nik Everett 294ab7ee96
Core: Remove some logging constructors (#32513)
Remove a few of the logger constructors that aren't widely used or
aren't used at all and deprecate a few more logger constructors in favor
of log4j2's `LogManager`.
2018-08-09 16:11:48 -04:00
Nik Everett abda9fdac1 Logging: Fix test on windows
Windows' `\` instead of `/` strikes again!

Closes #32546
2018-08-09 14:35:30 -04:00
David Roberts 2608012422
Add temporary directory cleanup workarounds (#32615)
On some Linux distributions tmpfiles.d cleans files and
directories under /tmp if they haven't been accessed for
10 days.

This can cause problems for ML as ML is currently the only
component that uses the temp directory more than a few
seconds after startup. If you didn't open an ML job for
10 days and then tried to open one then the temp directory
would have been deleted.

This commit prevents the problem occurring in the case of
Elasticsearch being managed by systemd, as systemd private
temp directories are not subject to periodic cleanup (by
default).

Additionally there are now some docs to warn people about
the risk and suggest a manual mitigation for .tar.gz users.
2018-08-07 16:59:56 +01:00
David Turner e3cc33756e
Suppress Wildfly test in FIPS JVMs (#32543)
WildflyIT fails on FIPS-enabled JVMs. This change mutes this test suite on such
JVMs. Relates #32534.
2018-08-03 17:57:30 +01:00
David Turner 7a3eb4b3e8 Add AwaitsFix to failing test - see #32546 2018-08-01 14:28:00 +01:00
Nik Everett 22459576d7
Logging: Make node name consistent in logger (#31588)
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.

Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```

The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```

Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```

The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.

This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.

Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
Andy Bristol ac04ba42df
[test] package pre-install java check (#32259)
This recreates a test that was added to the bats packaging tests
in #31343 but didn't make it over to the java project during when the
linux package tests were ported in #31943

When packages are installed but can not locate the java executable, they
should fail with a descriptive message
2018-07-23 11:45:04 -07:00
Nhat Nguyen 261002283b AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
Relates #31291
2018-07-23 08:56:43 -04:00
Nik Everett 042424b43b
Switch full-cluster-restart to new style Requests (#32140)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `qa/full-cluster-restart` project to use the new
versions.

It also fixes a small bug in the test for explain on the `_all` field
that was causing it to not properly invoke `_explain`.
2018-07-20 17:33:15 -04:00
Ioannis Kakavas aaa8f842d6
Remove BouncyCastle dependency from runtime (#32193)
* Remove BouncyCastle dependency from runtime

This commit introduces a new gradle  project that contains
 the classes that have a dependency on BouncyCastle. For 
the default distribution, It builds  a jar from those and
 in puts it in a subdirectory of lib
 (/tools/security-cli) along with the BouncyCastle jars. 
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools 
that use these classes.

BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
2018-07-21 00:03:58 +03:00
Nik Everett c6c9075ca4
Switch rolling restart to new style Requests (#32147)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `qa/rolling-upgrade` project to use the new
versions.
2018-07-20 12:01:50 -04:00
Andy Bristol aae0133847
[test] port linux package packaging tests (#31943)
Add packaging tests for the linux package distributions to the java test
project and remove them from bats. Most of the tests that lived in
30_deb_package.bats and 40_rpm_package.bats are applicable to both
package types and are combined into a single type of test case. Others
are separated out into separate cases to make their intent more clear

For #26741
2018-07-18 17:18:00 -07:00
Andy Bristol e20f59aa71
[test] use randomized runner in packaging tests (#32109)
Use the randomized runner from the test framework and add some basic
logging to make the packaging tests behave more similarly to how we use
junit in the rest of the project
2018-07-18 10:26:26 -07:00
Armin Braun 6de1f96cad
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
* UUID field was added for #31791 and only went into 6.4 and 7.0
* Fixes #32119
2018-07-18 15:32:36 +02:00
David Kyle b655c11dbe Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ 2018-07-17 10:34:56 +01:00
Luca Cavanna 049966a829
Check that client methods match API defined in the REST spec (#31825)
We have been encountering name mismatches between API defined in our
REST spec and method names that have been added to the high-level REST
client. We should check this automatically to prevent furher mismatches,
and correct all the current ones.

This commit adds a test for this and corrects the issues found by it.
2018-07-17 11:26:28 +02:00
Nik Everett d596447f3d
Switch non-x-pack to new style requests (#32106)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes most of the calls not in X-Pack to their new versions.
2018-07-16 17:44:19 -04:00
Nik Everett 4d83a0dd5a
HLREST: Bundle the x-pack protocol project (#31904)
The `:x-pack:protocol` project is an implementation detail shared by the
xpack projects and the high level rest client and really doesn't deserve
its own maven coordinants and published javadoc. This change bundles
`:x-pack:protocol` into the high level rest client.

Relates to #29827
2018-07-10 17:59:15 -04:00
Andy Bristol 006c2c9ab0
[test] port archive distribution packaging tests (#31314)
Recreates the rest of the bats packaging tests for the tar distribution
in the java packaging test project, with support for both tar and zip
packaging, both oss and default flavors, and on Linux and Windows. Most
tests are followed fairly closely, some have either been dropped if
unnecessary or folded into others if convenient.
2018-07-10 08:21:20 -07:00
Ben Abrams 909a18add7 Only set vm.max_map_count if greater than default (#31512)
So the issue here is that we want to avoid setting vm.max_map_count if
it is already equal to the desired value (the bootstrap check requires
262144). The reason we want to avoid this is because in some use-cases
using sysctl to set this will fail. In this case, we want to enable
users to set this value externally and then allow that to cause using
sysctl to set the value to be skipped so that cases where using sysctl
will fail to no longer fail.
2018-06-27 23:38:30 -04:00
Nik Everett d0c276c456
QA: Merge query-builder-bwc to restart test (#30979)
Merges the `query-builder-bwc` qa project into the
`full-cluster-restart` qa project, saving a cluster starts on every
build and *many* cluster starts on `./gradlew bwcTests`.
2018-06-27 16:37:04 -04:00
Nik Everett 232c71b6bf
QA: Create xpack yaml features (#31403)
This creates a YAML test "features" that indices if the cluster being
tested has xpack installed (`xpack`) or if it does *not* have xpack
installed (`no_xpack`). It uses those features to centralize skipping
a few tests that fail if xpack is installed.

The plan is to use this in a followup to skip docs tests that require
xpack when xpack is not installed. We *plan* to use the declaration
of required license level on the docs page to generate the required
`skip`.

Closes #30933.
2018-06-26 09:26:48 -04:00
Michael Basnight adfcea2af6
Add package pre-install check for java binary (#31343)
The package installation relies on java being in the path. If java is
not in the path, the tests fail at post-install time. This commit adds a
pre-install check to validate that java exists, and if it fails, the
package is never installed, and thus keeps a system clean, rather than
aborting at post-install and leaving behind a mess.

Closes #29665
2018-06-25 10:54:39 -05:00
Nhat Nguyen 51151027cd TEST: Add bwc recovery tests with synced-flush index
Although the master branch does not affect by #31482, it's helpful to
have BWC tests that verify the peer recovery with a synced-flush index.
This commit adds the bwc tests from #31506 to the master branch.

Relates #31482
Relates #31506
2018-06-22 20:14:06 -04:00
Ryan Ernst 59e7c6411a
Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
TransportRequestHandler currently contains 2 messageReceived methods,
one which takes a Task, and one that does not. The first just delegates
to the second. This commit changes all existing implementors of
TransportRequestHandler to implement the version which takes Task, thus
allowing the class to be a functional interface, and eliminating the
need to throw exceptions when a task needs to be ensured.
2018-06-22 07:36:03 -07:00
Christoph Büscher 02346c20a2
Rankeval: Fold template test project into main module (#31203)
This change moves tests in `smoke-test-rank-eval-with-mustache` into the main
ranking evaluation module by declaring that the integration testing cluster
requires the `lang-mustache` plugin. This avoids having to maintain the qa
project for only one basic test suite.
2018-06-15 15:55:39 +02:00
Ben Abrams 87a676e4d5 Do not set vm.max_map_count when unnecessary (#31285)
This commit modifies the Sys V init startup scripts to only modify
vm.max_map_count if needed. In this case, needed means that the current
value is less than our default value of 262144 maps.
2018-06-14 21:41:02 -04:00
Nik Everett d6d0727aac
QA: Fix resolution of default distribution (#31351)
If you run `./gradlew -p qa bwcTest -Dtests.distribution=zip` then we
need to resolve older versions of the default distribution. Since those
aren't available in maven central, we need add the elastic maven repo to
the project.
2018-06-14 18:25:47 -04:00
Andy Bristol 01d64b128b [test] opensuse packaging turn up debug logging 2018-06-13 11:15:04 -07:00
Jason Tedor 0bfd18cc8b
Revert upgrade to Netty 4.1.25.Final (#31282)
This reverts upgrading to Netty 4.1.25.Final until we have a cleaner
solution to dealing with the object cleaner thread.
2018-06-12 19:26:18 -04:00
Jason Tedor 563141c6c9
Upgrade to Netty 4.1.25.Final (#31232)
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
2018-06-11 16:55:07 -04:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Jason Tedor 94be9b471f
Rename elasticsearch-core to core (#31185)
This commit renames :libs:elasticsearch-core to :libs:core.
2018-06-07 16:50:21 -04:00
Nik Everett dfcc939ef8 QA: Better seed nodes for rolling restart
Use all running nodes as unicast seeds in the rolling restart tests to
avoid a race between pinging and the tests. Without this if the tests
are too fast then when a new node comes up and pings its single
configured seed node that node *might* not have a ping from the other
running node.
2018-06-07 13:30:37 -04:00
Tim Brooks 237f9b8930
Add nio-transport as option for http smoke tests (#31162)
This is related to #27260 and #28898. This commit adds the transport-nio
plugin as a random option when running the http smoke tests. As part of
this PR, I identified an issue where cors support was not properly
enabled causing these tests to fail when using transport-nio. This
commit also fixes that issue.
2018-06-07 09:46:36 -06:00
Nik Everett 56207ea43d QA: Set better node names on rolling restart tests
These should help with debugging failures.
2018-06-07 11:25:41 -04:00
Luca Cavanna be4a101ea1
Add high-level client methods that accept RequestOptions (#31069)
With #30490 we have introduced a new way to provide request options
whenever sending a request using the high-level REST client. Before you
could provide headers as the last argument varargs of each API method,
now you can provide `RequestOptions` that in the future will allow to
provide more options which can be specified per request.

This commit deprecates all of the client methods that accept a `Header`
varargs argument in favour of new methods that accept `RequestOptions`
instead. For some API we don't even go through deprecation given that
they were not released since they were added, hence in that case we can
just move them to the new method.
2018-06-06 23:17:45 +02:00
Nik Everett 7c59e7690e
QA: Switch xpack rolling upgrades to three nodes (#31112)
This is much more realistic and can find more issues. This causes the
"mixed cluster" tests to be run twice so I had to fix the tests to work
in that case. In most cases I did as little as possible to get them
working but in a few cases I went a little beyond that to make them
easier for me to debug while getting them to work. My test changes:

1. Remove the "basic indexing" tests and replace them with a copy of the
tests used in the OSS. We have no way of sharing code between these two
projects so for now I copy.
2. Skip the a few tests in the "one third" upgraded scenario:
  * creating a scroll to be reused when the cluster is fully upgraded
  * creating some ml data to be used when the cluster is fully ugpraded
3. Drop many "assert yellow and that the cluster has two nodes"
assertions. These assertions duplicate those made by the wait condition
and they fail now that we have three nodes.
4. Switch many "assert green and that the cluster has two nodes" to 3
nodes. These assertions are unique from the wait condition and, while
I imagine they aren't required in all cases, now is not the time to
find that out. Thus, I made them work.
5. Rework the index audit trail test so it is more obvious that it is
the same test expecting different numbers based on the shape of the
cluster. The conditions for which number are expected are fairly
complex because the index audit trail is shut down until the template
for it is upgraded and the template is upgraded when a master node is
elected that has the new version of the software.
6. Add some more information to debug the index audit trail test because
it helped me figure out what was going on.

I also dropped the `waitCondition` from the `rolling-upgrade-basic`
tests because it wasn't needed.

Closes #25336
2018-06-06 11:59:16 -04:00
Luca Cavanna 70749e01c4
Cross Cluster Search: preserve remote status code (#30976)
In case an error is returned when calling search_shards on a remote
cluster, which will lead to throwing an exception in the coordinating
 node, we should make sure that the status code returned by the
 coordinating node is the same as the one returned by the remote
 cluster. Up until now a 500 - Internal Server Error was always
 returned. This commit changes this behaviour so that for instance if an
 index is not found, which causes an 404, a 404 is also returned by the
 coordinating node to the client.

 Closes #27461
2018-06-01 08:53:53 +02:00
Nik Everett b225f5e5c6
HLRest: Allow caller to set per request options (#30490)
This modifies the high level rest client to allow calling code to
customize per request options for the bulk API. You do the actual
customization by passing a `RequestOptions` object to the API call
which is set on the `Request` that is generated by the high level
client. It also makes the `RequestOptions` a thing in the low level
rest client. For now that just means you use it to customize the
headers and the `httpAsyncResponseConsumerFactory` and we'll add
node selectors and per request timeouts in a follow up.

I only implemented this on the bulk API because it is the first one
in the list alphabetically and I wanted to keep the change small
enough to review. I'll convert the remaining APIs in a followup.
2018-05-31 13:59:52 -04:00
Christoph Büscher 1ea9f11b03
Change ScriptException status to 400 (bad request) (#30861)
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes #12315
2018-05-30 14:00:07 +02:00
Andy Bristol 4001097a68
[test] packaging: use shell when running commands (#30852)
When subprocesses are started with ProcessBuilder, they're forked by the
java process directly rather than from a shell, which can be surprising
for our use case here in the packaging tests which is similar to
scripting.

This commit changes the tests to run their subprocess commands in a
shell, using the bash -c <script> syntax for commands on linux and using
the powershell.exe -Command <script> syntax for commands on windows.
This syntax on windows is essentially what the tests were already doing.
2018-05-29 15:17:11 -07:00
Andy Bristol ba8bb1d4a1 [test] packaging test logging for suse distros 2018-05-29 11:06:57 -07:00
Alpar Torok f5de25c618 Remove left-over comment 2018-05-28 11:52:26 +03:00
Andy Bristol 4bd2607597
[docs] explainer for java packaging tests (#30825) 2018-05-24 17:05:21 -07:00
Christoph Büscher 9cb6b90a99
[Tests] Move templated _rank_eval tests (#30679)
This change moves the ranking evaluation tests that use templates to the
existing yml rest tests instead of extending ESIntegTestCase.

Closes #30628
2018-05-24 18:31:13 +02:00
Andy Bristol a1b538122c
[test] java tests for archive packaging (#30734)
Ports the first couple tests for archive distributions from the old bats
project to the new java project that includes windows platforms,
consolidating them into one test method that tests that the
distributions can be extracted and their contents verified. Includes the
zip distributions which were not tested in the bats project.
2018-05-23 10:37:57 -07:00
Nik Everett a5d90e919f
QA: Add xpack tests to rolling upgrade (#30795)
A rolling upgrade from oss Elasticsearch to the default distribution of
Elasticsearch is significantly different than a full cluster restart to
install a plugin and is again different from starting a new cluster with
xpack installed. So this adds some basic tests to make sure that the
rolling upgrade that enables xpack works at all.

This also removes some unused imports from the tests that I modified in
PR #30728. I didn't mean to leave them.
2018-05-22 17:17:34 -04:00
Luca Cavanna a17d6cab98
Replace Request#setHeaders with addHeader (#30588)
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
2018-05-22 20:32:30 +02:00
Nik Everett be8c0f27be
QA: Switch rolling upgrade to 3 nodes (#30728)
Switches the rolling upgrade tests from upgrading two nodes to upgrading
three nodes which is much more realistic and much better able to find
unexpected bugs. It upgrades the nodes one at a time and runs tests
between each upgrade. As such this now has four test runs:

1. Old
2. One third upgraded
3. Two thirds upgraded
4. Upgraded

It sets system properties so the tests can figure out which stage they
are in. It reuses the same yml tests for the "one third" and "two
thirds" cases because they are *almost* the same case.

This rewrites the yml-based indexing tests to be Java based because the
yml-based tests can't handle different expected values for the counts.
And the indexing tests need that when they are run twice.
2018-05-21 13:15:13 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Nik Everett 6695d11a97
Switch many QA projects to use new style requests (#30574)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the the old requests. This
changes many calls in the `qa` projects to use the new version.
2018-05-15 17:48:47 -04:00
Nik Everett 869b639d14
QA: System property to override distribution (#30591)
This configures all `qa` projects to use the distribution contained in
the `tests.distribution` system property if it is set. The goal is to
create a simple way to run tests against the default distribution which
has x-pack basic features enabled while not forcing these tests on all
contributors. You run these tests by doing something like:

```
./gradlew -p qa -Dtests.distribution=zip check
```

or

```
./gradlew -p qa -Dtests.distribution=zip bwcTest
```

x-pack basic *shouldn't* get in the way of any of these tests but
nothing is ever perfect so this we have to disable a few when running
with the zip distribution.
2018-05-15 17:16:16 -04:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Christoph Büscher cc93131318
Forbid expensive query parts in ranking evaluation (#30151)
Currently the ranking evaluation API accepts the full query syntax for
the queries specified in the evaluation set and executes them via multi
search. This potentially runs costly aggregations and suggestions too.
This change adds checks that forbid using aggregations, suggesters, 
highlighters and the explain and profile options in the queries that are 
run as part of the ranking evaluation since they are irrelevent in the 
context of this API.
2018-05-14 17:36:26 +02:00
Nik Everett 51fa8739ea
Reindex: Fold "with all deps" project into reindex (#30154)
This folds the `:qa:smoke-test-reindex-with-all-modules` project into
`:modules:reindex` by declaring the reindex's integration testing
cluster requires the `parent-join` and `lang-painless` plugins and then
moving all of the integration tests that depended on parent-join and
painless into reindex.

It saves us one cluster start up during the build at the cost of a
little of the reindex module's "purity". Since the reindex module *does*
have unit tests that test scripting without painless I'm fairly ok with
that.
2018-05-10 08:02:23 -04:00
Jason Tedor beee5fe004
Respect accept header on no handler (#30383)
Today when processing a request for a URL path for which we can not find
a handler we send back a plain-text response. Yet, we have the accept
header in our hand and can respect the accepted media type of the
request. This commit addresses this.
2018-05-04 18:13:50 -04:00
Jim Ferenczi 6a65b3b874 [Test] Add analysis-nori plugin to the vagrant tests
relates #30397
2018-05-04 22:46:04 +02:00
Ryan Ernst 3aec8c7725
Packaging: Set elasticsearch user to have non-existent homedir (#29007)
This commit adds setting the homedir for the elasticsearch user to the
adduser command in the packaging preinstall script. While the
elasticsearch user is a system user, it is sometimes conventient to have
an existing homedir (even if it is not writeable). For example, running
cron as the elasticsearch user will try to change dir to the homedir.

closes #14453
2018-05-02 14:33:34 -07:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Ryan Ernst fba2f00a73
Packaging: Unmark systemd service file as a config file (#29004)
Systemd overrides should happen through /etc/systemd/system, not
directly editing the service file. This commit removes marking the
service file as configuration for rpm and deb packages.
2018-05-02 09:48:49 -07:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00