Commit Graph

25142 Commits

Author SHA1 Message Date
Adrien Grand df4482fdc8 Do not cache the QueryShardContext in PercolatorFieldMapper: it is cheap to create. 2016-11-15 15:45:18 +01:00
Tanguy Leroux aa871f839a [Tests] Do not pass all env vars to sudo (#21562)
In #21348 the command executed to run the packaging tests has been changed to "sudo -E bats ...", forcing all environment variables from the vagrant user to be passed to the `sudo` command. This breaks a test on opensuse-13 (the one where it checks that elasticsearch cannot be started when `java` is not found) because all the PATH from the user is passed to the sudo command.

This commit restores the previous behavior while allowing only necessary testing environment variables to be passed using a /etc/sudoers.d file.
2016-11-15 15:19:34 +01:00
Boaz Leskes d6c2b4f7c5 Adapt InternalTestCluster to auto adjust `minimum_master_nodes` (#21458)
#20960 removed `LocalDiscovery` and we now use `ZenDiscovery` in all our tests. To keep cluster forming fast, we are using a `MockZenPing` implementation which uses static maps to return instant results making master election fast. Currently, we don't set `minimum_master_nodes` causing the occasional split brain when starting multiple nodes concurrently and their pinging is so fast that it misses the fact that one of the node has elected it self master. To solve this, `InternalTestCluster` is modified to behave like a true cluster and manage and set `minimum_master_nodes` correctly with every change to the number of nodes.

Tests that want to manage the settings themselves can opt out using a new `autoMinMasterNodes` parameter to the `ClusterScope` annotation. 

Having `min_master_nodes` set means the started node may need to wait for other nodes to be started as well. To combat this, we set `discovery.initial_state_timeout` to `0` and wait for the cluster to form once all node have been started. Also, because a node may wait and ping while other nodes are started, `MockZenPing` is adapted to wait rather than busy-ping.
2016-11-15 13:42:26 +00:00
Simon Willnauer 66fbb0dbc2 Don't fail in `afterExecute` if context is already closed (#21563)
We run an assert on an potentially closed thread context. this should
not bubble up the `IllegalStateException`.
2016-11-15 13:55:50 +01:00
Adrien Grand 54809065a6 Make PercolatorFieldMapper get a QueryShardContext lazily. 2016-11-15 12:02:40 +01:00
Simon Willnauer 4c53e55d34 [TEST] Add master_timeout to shrink call to fail faster 2016-11-15 10:59:59 +01:00
Sumit Gupta e53405f4f3 Update geohashgrid-aggregation.asciidoc (#21530) 2016-11-15 10:49:02 +01:00
Simon Willnauer 200a2850a9 [TEST] Don't stop MockAppender some nodes might concurrently use it 2016-11-15 10:48:39 +01:00
Boaz Leskes 6d9af2fff4 Uncommitted mapping updates should not efect existing indices (#21306)
When processing a mapping updates, the master current creates an `IndexService` and uses its mapper service to do the hard work. However, if the master is also a data node and it already has an instance of `IndexService`, we currently reuse the the `MapperService` of that instance. Sadly, since mapping updates are change the in memory objects, this means that a mapping change that can rejected later on during cluster state publishing will leave a side effect on the index in question, bypassing the cluster state safety mechanism.

This commit removes this optimization and replaces the `IndexService` creation with a direct creation of a `MapperService`. 

Also, this fixes an issue multiple from multiple shards for the same field caused unneeded cluster state publishing as the current code always created a new cluster state.

This were discovered while researching #21189
2016-11-15 10:47:34 +01:00
Adrien Grand ad94bea0bb Remove XPointValues. (#21541)
This class had been added to address a bug in PointValues, which has been fixed
since then.
2016-11-15 10:11:41 +01:00
Yannick Welsch 64a7a960d9 Use pre-JDK9 style FilePermissions on JDK9 (#21540)
JDK9 removed pathname canonicalization when constructing FilePermission objects, which breaks some of the FilePermissions added by Elasticsearch. This commit adds the system property jdk.io.permissionsUseCanonicalPath which makes JDK9 behave like JDK8 w.r.t. FilePermission objects (see #21534).
2016-11-15 09:31:32 +01:00
Martijn van Groningen 8a3a885058 inner_hits: Skip adding a parent field to nested documents.
Otherwise an empty string get added as _parent field.

Closes #21503
2016-11-15 07:32:28 +01:00
Ryan Ernst c7bd4f3454 Tests: Add TestZenDiscovery and replace uses of MockZenPing with it (#21488)
This changes adds a test discovery (which internally uses the existing
mock zenping by default). Having the mock the test framework selects be a discovery
greatly simplifies discovery setup (no more weird callback to a Node
method).
2016-11-14 21:46:10 -08:00
Ryan Ernst 2cdf3eeae0 Remove generics from ActionRequest (#21556)
closes #21368
2016-11-14 16:31:17 -08:00
Ryan Ernst d14c470b89 Remove generics from ActionRequest
closes #21368
2016-11-14 15:32:01 -08:00
Jason Tedor 48579cccab Add socket permissions for tribe nodes
Today when a node starts, we create dynamic socket permissions based on
the configured HTTP ports and transport ports. If no ports are
configured, we use the default port ranges. When a tribe node starts, a
tribe node creates an internal node client for connecting to each remote
cluster. If neither an explicit HTTP port nor transport ports were
specified, the default port ranges are large enough for the tribe node
and its internal node clients. If an explicit HTTP port or transport
port was specified for the tribe node, then socket permissions for those
ports will be created, but not for the internal node clients. Whether
the internal node clients have explicit ports specified, or attempt to
bind within the default range, socket permissions for these will not
have been created and the internal node clients will hit a permissions
issue when attempting to bind. This commit addresses this issue by also
accounting for tribe nodes when creating the dynamic socket
permissions. Additionally, we add our first real integration test for
tribe nodes.

Relates #21546
2016-11-14 15:09:45 -05:00
Jay Modi 87d76c3ff8 assert blocking calls are not made on the cluster state update thread
This commit adds an assertion to ensure that we do not introduce blocking calls in code
that is called in a ClusterStateListener or another part of the cluster state update process.
2016-11-14 14:30:01 -05:00
Jason Tedor 795c121852 Remove explicit task paths in tribe node test
This commit removes some explicit paths in the tribe node smoke test so
that these are easier to maintain (in the case of reorganization).
2016-11-14 13:52:57 -05:00
Yannick Welsch 6f69f54216 Use pre-JDK9 style FilePermissions on JDK9 when running from start scripts 2016-11-14 19:52:35 +01:00
Jason Tedor 9fb54f4ef8 Remove unnecessary hash map copy in o.e.b.Security
This commit removes an unnecessary copying of the tribe node group
settings in o.e.b.Security.
2016-11-14 13:49:16 -05:00
Jason Tedor a12f09317d Fallback to settings if transport profile is empty
If the transport profile does not contain a TCP port range, we fallback
to the top-level settings.
2016-11-14 13:48:12 -05:00
Jason Tedor 491a945ac8 Add socket permissions for tribe nodes
Today when a node starts, we create dynamic socket permissions based on
the configured HTTP ports and transport ports. If no ports are
configured, we use the default port ranges. When a tribe node starts, a
tribe node creates an internal node client for connecting to each remote
cluster. If neither an explicit HTTP port nor transport ports were
specified, the default port ranges are large enough for the tribe node
and its internal node clients. If an explicit HTTP port or transport
port was specified for the tribe node, then socket permissions for those
ports will be created, but not for the internal node clients. Whether
the internal node clients have explicit ports specified, or attempt to
bind within the default range, socket permissions for these will not
have been created and the internal node clients will hit a permissions
issue when attempting to bind. This commit addresses this issue by also
accounting for tribe nodes when creating the dynamic socket
permissions. Additionally, we add our first real integration test for
tribe nodes.
2016-11-14 11:58:44 -05:00
Simon Willnauer 1d8c8529ed Remove `IndexTemplateAlreadyExistsException` and `IndexShardAlreadyExistsException` (#21539)
Both exception can be replaced with java built-in exception, IAE and ISE respectively.
This should be back ported partially to 5.x which the transport layer code should be preserved.

Relates to #21494
2016-11-14 17:09:57 +01:00
Simon Willnauer 26375256ff Enable 5.x to 6.x BWC tests (#21537)
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 17:03:57 +01:00
Yannick Welsch d3e97ce6cd Fix line length in TCPTransportTests
Makes checkstyle happy
2016-11-14 16:55:14 +01:00
Yannick Welsch d42f7eec61 Check valid cluster service state transitions (#21538)
This commit adds assertions to check whether the cluster service state transitions in a way that we expect it to.

Relates to #21379.
2016-11-14 16:49:25 +01:00
Simon Willnauer 26a8a94e56 [TEST] Add test to ensure `transport.tcp.compress` works
This adds a basic unittest to ensure `transport.tcp.compress` has effect
on all basic TcpTransport implementations.

Relates to #21526
2016-11-14 16:13:44 +01:00
Simon Willnauer 7d4bde8e00 remove forbidden API 2016-11-14 15:30:07 +01:00
Yannick Welsch 8655cd7182 Add assertion that checks that the same shard with same id is not added to same node (#21498)
Adds an assertion that checks that the same shard with same id is not added to same node. Previously we would just silently ignore the second shard being added.
2016-11-14 15:14:14 +01:00
Simon Willnauer bdc942fa72 Enable 5.x to 6.x BWC tests
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 14:26:49 +01:00
Yannick Welsch ea65a01789 Use pre-JDK9 style FilePermissions on JDK9
JDK9 removed pathname canonicalization when constructing FilePermission objects, which breaks some of the FilePermissions added by
Elasticsearch. This commit adds the system property jdk.io.permissionsUseCanonicalPath which makes JDK9 behave like JDK8 w.r.t. FilePermissions (see
https://github.com/elastic/elasticsearch/issues/21534).
2016-11-14 14:13:23 +01:00
Tanguy Leroux b8f50b0219 Add Vagrant Gradle plugin (#21348)
This commit changes the current :elactisearch:qa:vagrant build file and transforms it into a Gradle plugin in order to reuse it in other projects.

Most of the code from the build.gradle file has been moved into the VagrantTestPlugin class. To avoid duplicated VMs when running vagrant tests, the Gradle plugin sets the following environment variables before running vagrant commands:
VAGRANT_CWD: absolute path to the folder that contains the Vagrantfile
VAGRANT_PROJECT_DIR: absolute path to the Gradle project that use the VagrantTestPlugin

The VAGRANT_PROJECT_DIR is used to share project folders and files with the vagrant VM. These folders and files are exported when running the task `gradle vagrantSetUp` which:
- collects all project archives dependencies and copies them into `${project.buildDir}/bats/archives`
- copy all project bats testing files from 'src/test/resources/packaging/tests' into `${project.buildDir}/bats/tests`
- copy all project bats utils files from 'src/test/resources/packaging/utils' into `${project.buildDir}/bats/utils`

It is also possible to inherit and grab the archives/tests/utils files from project dependencies using the plugin configuration:

apply plugin: 'elasticsearch.vagrant'
esvagrant {
    inheritTestUtils true|false
    inheritTestArchives true|false
    inheritTests true|false
}
dependencies {
    // Inherit Bats test utils from :qa:vagrant project
    bats project(path: ':qa:vagrant', configuration: 'bats')
}

The folders `${project.buildDir}/bats/archives`, `${project.buildDir}/bats/tests` and `${project.buildDir}/bats/utils` are then exported to the vagrant VMs and mapped to the BATS_ARCHIVES, BATS_TESTS and BATS_UTILS environnement variables.

The following Gradle tasks have also be renamed:

* gradle vagrantSetUp
This task copies all the necessary files to the project build directory (was `prepareTestRoot`)

* gradle vagrantSmokeTest
This task starts the VMs and echoes a "Hello world" within each VM (was: `smokeTest`)
2016-11-14 13:57:52 +01:00
Tanguy Leroux 7cfa76ec01 Set LoggedExec back 2016-11-14 12:11:53 +01:00
Tanguy Leroux c2c29339d0 Rebasing master 2016-11-14 10:44:06 +01:00
Tanguy Leroux 9cade1cc18 Add Vagrant Gradle plugin
This commit changes the current :elactisearch:qa:vagrant build file and transforms it into a Gradle plugin in order to reuse it in other projects.

Most of the code from the build.gradle file has been moved into the VagrantTestPlugin class. To avoid duplicated VMs when running vagrant tests, the Gradle plugin sets the following environment variables before running vagrant commands:
VAGRANT_CWD: absolute path to the folder that contains the Vagrantfile
VAGRANT_PROJECT_DIR: absolute path to the Gradle project that use the VagrantTestPlugin

The VAGRANT_PROJECT_DIR is used to share project folders and files with the vagrant VM. These folders and files are exported when running the task `gradle vagrantSetUp` which:
- collects all project archives dependencies and copies them into `${project.buildDir}/bats/archives`
- copy all project bats testing files from 'src/test/resources/packaging/tests' into `${project.buildDir}/bats/tests`
- copy all project bats utils files from 'src/test/resources/packaging/utils' into `${project.buildDir}/bats/utils`

It is also possible to inherit and grab the archives/tests/utils files from project dependencies using the plugin configuration:

apply plugin: 'elasticsearch.vagrant'
esvagrant {
    inheritTestUtils true|false
    inheritTestArchives true|false
    inheritTests true|false
}
dependencies {
    // Inherit Bats test utils from :qa:vagrant project
    bats project(path: ':qa:vagrant', configuration: 'bats')
}

The folders `${project.buildDir}/bats/archives`, `${project.buildDir}/bats/tests` and `${project.buildDir}/bats/utils` are then exported to the vagrant VMs and mapped to the BATS_ARCHIVES, BATS_TESTS and BATS_UTILS environnement variables.

The following Gradle tasks have also be renamed:

* gradle vagrantSetUp
This task copies all the necessary files to the project build directory (was `prepareTestRoot`)

* gradle vagrantSmokeTest
This task starts the VMs and echoes a "Hello world" within each VM (was: `smokeTest`)
2016-11-14 10:33:05 +01:00
Daniel Mitterdorfer 9d3d6c5409 [docs] clients need to add jackson-databind (#21527)
With ES 5.0 we do not include Jackson
Databind anymore with ES core. This commit
updates our docs to state that users need
to add this artifact now in their projects.
2016-11-14 10:07:07 +01:00
Clinton Gormley 30d342c87c Update significantterms-aggregation.asciidoc
Fix scripted significant terms example to use `params.` prefix for painless
2016-11-14 09:40:04 +01:00
Adrien Grand 1fd5c47e7f Upgrade to lucene-6.3.0. (#21464) 2016-11-14 09:36:45 +01:00
Jason Tedor 19decd7552 Hack around cluster service and logging race
When a cluster update task executes, there can be log messages after the
update task has finished processing and the new cluster state becomes
visible. The visibility of the cluster state allows the test thread in
UpdateSettingsIT#testUpdateAutoThrottleSettings and
UpdateSettingsiT#testUpdateMergeMaxThreadCount to proceed. The test
thread will remove and stop a mock appender setup at the beginning of
the test. The log messages in the cluster state update task that occur
after processing has finished can race with the removal of the
appender. Log4j will grab a reference to the appenders when processing
these log messages, and this races with the removal and stopping of the
appenders. If Log4j grabs a reference to the appenders before the mock
appender has been removed, and the test thread subsequently removes and
stops the appender before Log4j has appended the log message, Log4j will
get angry that we are appending to a stopped appender, causing the test
to fail. This commit addresses this race by waiting for the cluster
state update task to have finished processing before freeing the test
thread to make its assertions and finally remove and stop the
appender. Yes, this is a hack.

Relates #21518
2016-11-13 18:06:12 -05:00
Jason Tedor d273419d00 Do not prematurely shutdown Log4j
When a node closes, we shutdown logging as the last statement. This
statement must be last lest any subsequent attempts to log will blow up
by running into security permissions. Yet, in the case of a tribe node
this isn't enough. The first internal tribe node to close will shutdown
logging, and subsequent node closes will blow up with the aforementioned
problem. This commit migrate the Log4j shutdown to occur as part of the
shutdown hook that closes the node, after all nodes have
closed. Consequently, we can remove a hack in the test infrastructure to
prevent Log4j shutdowns when internal test nodes close and instead just
register a single shutdown hook that runs when the test JVM exits.

Relates #21519
2016-11-13 17:27:30 -05:00
Nik Everett 2a328034ef Support decimal constants with trailing [dD] in painless (#21412)
This adds support to painless for decimal constants with trailing `d` or
`D` to make it compatible with Java. It already supported integer
constants with a trailing `d` or `D` but this adds tests for it.

Closes #21116
2016-11-12 11:08:39 -05:00
Nik Everett a26b5a113c In painless suggest a long constant if int won't do (#21415)
In painless we prefer explicit types over implicit ones whereas
groovy is the other way around. Take this groovy code:

```
> 86400000.class
java.lang.Integer
> 864000000000.class
java.lang.Long
```

Painless accepts `86400000` just fine because that is a valid `int`
in the jvm. It rejects `864000000000` as an invlid `int` constant
because, in painless as in java, `long` constants always end in `L`
or `l`.

To ease the transition from groovy to painless, this changes the
compilation error returned from these invalid constants from:

```
Invalid int constant [864000000000].
```

to

```
Invalid int constant [864000000000]. If you want a long constant then change it to [864000000000L].
```

Inspired by #21313
2016-11-12 11:08:18 -05:00
Jason Tedor 404b9afeca Account for different paths for sysctl utilities
On some systems these utilities are in /usr/lib/systemd/systemd-sysctl
and /usr/sbin/sysctl, and on others the /usr is dropped. This commit
accounts for that fact.
2016-11-12 07:54:17 -05:00
Ali Beyad 38023fb58d [TEST] testRebalancePossible() may not have an assigned node id 2016-11-11 23:10:34 -05:00
Igor Motov ca639e8c86 Tests: Disable merge in SearchCancellationTests
We have to have at least 2 segments for the test to work and sometimes random merge policy merges them into one.
2016-11-11 18:22:28 -05:00
Igor Motov 058b6e019c Tests: clean search scroll at the end of SearchCancellationIT
Under some rare conditions search cancellation response might not fully clean scroll context. For now this commit adds the cleaning operation to the test, and we will address the root cause in https://github.com/elastic/elasticsearch/issues/21511
2016-11-11 18:22:15 -05:00
Jason Tedor 1fde26347a Set vm.max_map_count on systemd package install
Our docs claim that we set vm.max_map_count automatically. This is not
quite the case. The story is that on SysV init we set vm.max_map_count
each time the service starts, which is good. On systemd, we create a
sysctl.d conf file that sets vm.map_max_count, but this is only
meaningful if the system is rebooted after package install. This commit
modifies the post-install script so that we run systemd-sysctl so that
the vm.max_map_count change occurs after package install without a
reboot.

Relates #21507
2016-11-11 16:41:54 -05:00
Ali Beyad 5f1d108704 [TEST] reduce the number of snapshotted shards to 1 in testSnapshotSucceedsAfterSnapshotFailure()
so that we are more likely to trigger I/O exceptions on writing the control files during the
finalize phase of snapshotting (with the aim of triggering an I/O failure when writing pending-index-*).
2016-11-11 16:22:11 -05:00
David Pilato 5336e72fbb Add documentation for Logger with Transport Client
Backport of #21477 in master branch
2016-11-11 21:23:23 +01:00
Jason Tedor 9352d16602 Enable appender exceptions in UpdateSettingsIT
This commit sets the mock appender in UpdateSettingsIT to not ignore
exceptions. This means that when an exception is hit, we will see an
actual stack trace that could be useful in debugging a non-reproducible
test failure.

Relates #21461
2016-11-11 12:41:20 -05:00