Commit Graph

775 Commits

Author SHA1 Message Date
Jason Tedor 2a90e50d0f Add concurrent deprecation logger test
Since deprecation logging involves concurrency, this commit adds a test
that the concurrency here is handled safely.

Relates #25481
2017-06-29 18:44:06 -04:00
Jason Tedor d219a85b33 Use LRU set to reduce repeat deprecation messages
This commit adds an LRU set to used to determine if a keyed deprecation
message should be written to the deprecation logs, or only added to the
response headers on the thread context.

Relates #25474
2017-06-29 16:36:43 -04:00
Christoph Büscher 927111c91d Remove QueryParseContext from parsing QueryBuilders (#25448)
Currently QueryParseContext is only a thin wrapper around an XContentParser that
adds little functionality of its own. I provides helpers for long deprecated
field names which can be removed and two helper methods that can be made static
and moved to other classes. This is a first step in helping to remove
QueryParseContext entirely.
2017-06-29 17:10:20 +02:00
Martijn van Groningen 7f2bcf1f97
test: added not null assertion
Relates to #25311
2017-06-29 13:54:52 +02:00
Jason Tedor da59c178e2 Emit settings deprecation logging at most once
When a setting is deprecated, if that setting is used repeatedly we
currently emit a deprecation warning every time the setting is used. In
cases like hitting settings endpoints over and over against a node with
a lot of deprecated settings, this can lead to excessive deprecation
warnings which can crush a node. This commit ensures that a given
setting only sees deprecation logging at most once.

Relates #25457
2017-06-28 22:18:46 -04:00
Jay Modi b2901f536e Do not search locally if remote index pattern resolves to no indices (#25436)
This commit changes how we determine if there were any remote indices that a search should have
been executed against. Previously, we used the list of remote shard iterators but if the remote
index pattern resolved to no indices there would be no remote shard iterators even though the
request specified remote indices. The map of remote cluster names to the original indices is used
instead so that we can determine if there were remote indices even when there are no remote shard
iterators.

Closes #25426
2017-06-28 12:41:37 -06:00
Martijn van Groningen 11fcfaae68
test: get upgrade status for all indices
Relates to #25311
2017-06-27 16:56:48 +02:00
Martijn van Groningen 9f5aef7b6d
test: added extra log line
Relates to #25311
2017-06-27 10:54:37 +02:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Ryan Ernst 5de406debb Fix docs lucene version check error message 2017-06-26 15:45:13 -07:00
Ryan Ernst 5726d1394f Build: Add check on lucene version in docs (#25407)
This commit adds a version verifica†ion to ensure the lucene version in
the docs is up to date.
2017-06-26 14:22:54 -07:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Simon Willnauer d338a09812 Remove `mapping.single_type` from parent join test (#25391)
This removes the remaining usage of `mapping.single_type` from the parent join
module and moves it's bwc test to the mixed cluster tests

Relates to #24961
Relates to #20257
2017-06-26 17:33:07 +02:00
Nik Everett da0b991331 Remove `index.mapping.single_type=false` from reindex tests (#25365)
* Remove the setting from the yml tests and replace with tests using
`join` field. We can't use the setting in yml tests without lots of
backflips but we have `ReindexParentChildTests` for the coverage.
There weren't tests for `join` field with reindex before this. Adding
these tests discovered #25363.
* Remove the setting from `ReindexParentChildTests` and replace with
`index.version.created=V_5_6_0`. This test can be entirely removed
when legacy parent/child support is dropped from core.
* Port the yml tests that set _parent into integ tests so they
can set the index created version. These tests can be removed
when we drop support for _parent in core.
* Port a delete-by-query test for filtering based on type to an
`ESIntegTestCase` so it can use `index.version.created=5.6.0` to
setup documents of multiple types. This whole feature can be dropped
when we no longer support multiple types per index.

Relates to #24961
2017-06-23 17:14:59 -04:00
Martijn van Groningen 9c511bc447
test: Replace OldIndexBackwardsCompatibilityIT#testOldClusterStates with a full cluster restart qa test
OldIndexBackwardsCompatibilityIT#testOldClusterStates tested whether global and index metadata could be read from data directory,
 this can also be tested in full cluster qa test that checks cluster state via api.

 Relates to #24939
2017-06-23 09:54:05 +02:00
Nik Everett 4bbb7e828b Port most snapshot/restore static bwc tests to qa:full-cluster-restart (#25296)
Ports all of RepositoryUpgradabilityIT to qa:full-cluster-restart and ports as much of RestoreBackwardsCompatIT as possible into qa:full-cluster-restart.
2017-06-21 13:26:03 -04:00
Martijn van Groningen 926527adc3
test: verify `size_to_upgrade_in_bytes` in assertBusy(...)
Relates to #25311
2017-06-21 13:45:46 +02:00
Martijn van Groningen bcaa413b0b
test: Port the remaining old indices search tests to full cluster restart qa module
Also tweaked the qa module's gradle file to actually run bwc tests against all index compat versions.

Relates to #24939
2017-06-19 12:27:24 +02:00
Guillaume Le Floch a9014dfcc5 Deprecate tribe service
This commit deprecates the tribe service so that deprecation log
messages are delivered if a tribe node is configured.

Relates #24598
2017-06-15 12:41:05 -04:00
Martijn van Groningen fe02829aac test: Ported more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests. (#25173)
Relates to #24939
2017-06-15 14:48:06 +02:00
Ryan Ernst 106e373412 Build: Add master flag for disabling bwc tests (#25230)
This commit adds a gradle project, set inside the root build.gradle,
which controls all our bwc tests. This allows for seamless (ie no errant
CI failures) backporting of behavior.
2017-06-14 22:01:49 -07:00
Nik Everett ce11b894b4 Extract the snapshot/restore full cluster restart tests from the translog full cluster restart tests (#25204)
Extract the snapshot/restore full cluster restart tests from the translog full cluster restart tests. That way they are easier to read.
2017-06-14 13:03:59 -04:00
Jay Modi ed76b9a518 Test: allow setting socket timeout for rest client (#25221)
In #25201, a setting was added to allow setting the retry timeout for the rest client under the
impression that this would allow requests to go longer than 30s. However, there is also a socket
timeout that needs to be set to greater than 30s, which this change adds a setting for.
2017-06-14 08:21:56 -06:00
Jay Modi 190242fb1b Test: add setting to change request timeout for rest client (#25201)
This commit adds a setting to change the request timeout for the rest client. This is useful as the
default timeout is 30s, which is also the same default for calls like cluster health. If both are
the same then the response from the cluster health api will not be received as the client usually
times out first making test failures harder to debug.

Relates #25185
2017-06-13 12:19:17 -06:00
Simon Willnauer bc7ec68e76 Add Cross Cluster Search support for scroll searches (#25094)
To complete the cross cluster search capabilities for all search types and
function this change adds cross cluster search support for scroll searches.
2017-06-13 17:22:49 +02:00
Jason Tedor 99262e26a0 Use synchronized Wildfly shutdown
We need to use the variant of shutdown that blocks until the connection
to Wildfly is closed or we can get spurious build failures.
2017-06-12 21:38:58 -04:00
Jason Tedor bb66f3b76b Explicitly reject duplicate data paths
Duplicate data paths already fail to work because we would attempt to
take out a node lock on the directory a second time which will fail
after the first lock attempt succeeds. However, how this failure
manifests is not apparent at all and is quite difficult to
debug. Instead, we should explicitly reject duplicate data paths to make
the failure cause more obvious.

Relates #25178
2017-06-12 12:55:19 -04:00
James Baiera 2e29b69f6a Revert "Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)""
This reverts commit b9e2a1f989.
2017-06-12 09:41:35 -04:00
Jason Tedor 7ed3d6e75b Fix comment formatting in EvilLoggerTests
The comments here were formatted inconsistently so this commit fixes
them.
2017-06-10 13:25:44 -04:00
Jason Tedor 5108fa7529 Remove unneeded weak reference from prefix logger
We have a custom logger implementation known as a prefix logger that is
used to write every message by the logger with a given prefix. This is
useful for node-level, index-level, and shard-level messages where we
want to log the node name, index name, and shard ID, respectively, if
possible. The mechanism that we employ is that of a marker. Log4j has a
built-in facility for managing these markers, but its effectively a
memory leak because these markers are held in a map and can never be
released. This is problematic for us since indices and shards do not
necessarily have infinite life spans and so on a node where there are
many indices being creted and destroyed, this infinite lifespan can be a
problem indeed. To solve this, we use our own cache of markers. This is
necessary to prevent too many instances of the marker for the same
prefix from being created (just think of all the shard-level components
that exist in the system), and to workaround the effective leak in
Log4j. These markers are stored as weak references in a weak hash
map. It is these weak references that are unneeded. When a key is
removed from a weak hash map, the corresponding entry is placed on a
reference queue that is eventually cleared. This commit simplifies
prefix logger by removing this unnecessary weak reference wrapper.

Relates #22460
2017-06-10 13:20:45 -04:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Tal Levy 340909582f remove Ingest's Internal Template Service (#25085)
Ingest was using it's own wrapper around TemplateScripts and the ScriptService.
This commit removes that abstraction
2017-06-08 15:24:03 -07:00
Nik Everett 66007078d4 Build: Allow preserving shared dir (#24962)
This adds an option to `ClusterConfiguration` to preserve the
`shared` directory when starting up a new cluster and switches
the `qa:full-cluster-restart` tests to use it rather than
disable the clean shared task.

Relates to #24846
2017-06-04 20:40:25 -04:00
Lee Hinman b9e2a1f989 Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)"
This reverts commit 4ed0abe72d.
2017-06-02 14:42:52 -06:00
James Baiera 4ed0abe72d Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)
We're using Vagrant in more places now than before. This commit includes a plugin that verifies
the Vagrant and Virtualbox installations for projects that depend on them. This shared code
should fix up the errors we've seen from CI builds relating to the new Kerberos fixture.
2017-06-02 16:26:11 -04:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Martijn van Groningen c9aeb34d50
test: Always issue a refresh after we have indexed, otherwise subsequent searches may return different results. 2017-06-01 10:20:44 +02:00
Nik Everett 4a90bd2317 Test: be more careful while flushing
We don't actually want to flush all the indices in the full cluster
restart tests. Never. *Sometimes* we want to flush certain indices
though.
2017-05-31 13:25:33 -04:00
Martijn van Groningen 9531ef25ec
Move OldIndexBackwardsCompatibilityIT#assertBasicSearchWorks over to full cluster restart qa module.
Relates to #24939
2017-05-31 09:27:41 +02:00
Simon Willnauer 4d423bf2ba Add a dummy_index to upgrade tests to ensure we recover fine with replicas (#24937)
We default to 0 replicas in the rolling restart scenario already to ensure
we test against worst case. Yet, this adds a dummy index to ensure we also
recover and index with replicas just fine.
2017-05-29 17:36:44 +02:00
Jason Tedor d1318e4d23 Fix Lucene version expectation
The Lucene version expectation in the verify Lucene version test is
backwards, mixing up the expected and actual values. This commit
reorders them to fix this issue.
2017-05-27 18:31:44 -04:00
Jason Tedor 09dd03e19f Verify Lucene version constants
The Lucene version constants for 5.4.1 and 5.5.0 are wrong, they are
listed as 6.5.0 instead of 6.5.1. This commit fixes these issues, and
adds a test to ensure that this does not happen again.

Relates #24923
2017-05-27 15:46:16 -04:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Nik Everett e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Ryan Ernst 8aaea51a0a Scripting: Move context definitions to instance type classes (#24883)
This is a simple refactoring to move the context definitions into the
type that they use. While we have multiple context names for the same
class at the moment, this will eventually become one ScriptContext per
instance type, so the pattern of a static member on the interface called
CONTEXT can be used. This commit also moves the consolidated list of
contexts provided by core ES into ScriptModule.
2017-05-25 12:18:45 -07:00
Ryan Ernst 52d504bb5f Scripting: Simplify ScriptContext (#24818)
As we work towards contexts implying the return type of compilation, we
first need ScriptContext to not be an enum. This commit removes the
Standard enum and Plugin subclass of ScriptContext.
2017-05-22 13:11:15 -07:00
Ryan Ernst 2de748859f Scripting: Remove "inline script enabled" on script engines (#24815)
ScriptEngine implementations have an overridable method to indicate they
are safe to use as inline scripts. Since groovy was removed fro 6.0,
there are no longer any implementations which used the default false
value. Furthermore, the value was not actually read anywhere. This
commit removes the method. The ScriptEngineRegistry was also no longer
necessary as it only was used to build a map from language to engine.
2017-05-20 12:01:25 -07:00
Ryan Ernst 1964e5c1d0 Test: Make mixed cluster bwc test per wire compat version (#24780)
This commit renames the backwards-5.0 qa test to mixed-cluster and
creates a test within the project per wire compat version. Like with
rolling upgrade tests, the integTest task will run against the most
recent version, while all versions will be tested with the bwcTest task.
2017-05-18 14:20:23 -07:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings (#24756)
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Nik Everett 82d2c7a142 Remove vagrant testing versions (#24754)
Now that we generate the versions list from Versions.java we can
drop the list of versions maintained for vagrant testing. One nice
thing that the vagrant testing did was to check if the list of
versions was out of date. This moves that test to the core
project.
2017-05-18 09:33:13 -04:00
Ryan Ernst 0353bd1fb6 Test: Convert rolling upgrade test to have task per wire compat version (#24758)
This commit changes the rolling upgrade test to create a set of rest
test tasks per wire compat version. The most recent wire compat version
is always tested with the `integTest` task, and all versions can be
tested with `bwcTest`.
2017-05-18 01:14:24 -07:00
Ryan Ernst ec025f825b Fix leftover reference to scripts path in packaging test 2017-05-18 00:22:54 -07:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Ryan Ernst ff34434bba Build: Extract all ES versions into gradle properties (#24748)
This commit expands the logic for version extraction from Version.java
to include a list of all versions for backcompat purposes. The tests
using bwcVersion are converted to use this list, but those tests
(rolling upgrade and backwards-5.0) are still not randomized; that will
happen in another followup.
2017-05-17 12:58:37 -07:00
Jason Tedor 1a7a926a03 Fix jvm-example assertions in packaging tests
These assertions were on the yaml extension but we have migrated to yml
everywhere so these assertions need to be updated too.
2017-05-17 13:20:25 -04:00
Simon Willnauer cf846af0e5 Fix `_field_caps` serialization in order to support cross cluster search (#24722)
Today the `_field_caps` API doesn't implement its request serialization
correctly since indices and indices options are not serialized at all.
This will likely break with all transport clients etc. and if this request
must be send across the network. This commit fixes this and adds correct
handling if we have only remote indices to prevent the inclusion of
all local indices.
2017-05-17 14:02:45 +02:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Ryan Ernst 2e6dc04025 Re-enable centos-6 2017-05-15 13:00:24 -07:00
Ryan Ernst f1fd5350c9 Test: Fix how packaging test generates expected plugins 2017-05-15 12:58:11 -07:00
Yannick Welsch bd364c56a8 Mute packagingTest on CentOS 6
Relates #24645
2017-05-15 11:01:35 +02:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Nik Everett 4d86679f5a Disable reindex-from-old tests in windows
In windows we can't reliable git the pid so we skip the
reindex-from-remote tests from old versions of elasticsearch. This
is OK because we aren't really testing windows here anyway. It isn't
great, but should be safe.
2017-05-11 17:36:03 -04:00
Nik Everett 2ffdd4468d Skip reindex tests from old es if we can't run it
Two of the versions of Elasticsearch we need to run for these tests
can't run in Java 9 so we skip the entire test if we are running in
java 9. For now. I'd like to reenable it to run against java 8 if
there is one available, but that can wait for another time.

Relates to #24561
2017-05-11 15:32:31 -04:00
Nik Everett 9953a96143 Build: move dependency so it works in 5.x
Relates to #24561
2017-05-11 10:41:56 -04:00
Nik Everett 8188569fd1 Add qa module that tests reindex-from-remote against pre-5.0 versions of Elasticsearch (#24561)
Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes #23828
Closes #24520
2017-05-11 10:06:20 -04:00
Ryan Ernst 9ca7d28552 Scripting: Remove "service" from ScriptEngine interface name (#24574)
This commit renames ScriptEngineService to ScriptEngine.  It is often
confusing because we have the ScriptService, and then
ScriptEngineService implementations, but the latter are not services as
we see in other places in elasticsearch.
2017-05-10 00:47:33 -07:00
Ryan Ernst ebd3e5f73f Scripting: Deprecate file script settings (#24555)
File scripts have 2 related settings: the path of file scripts, and
whether they can be dynamically reloaded. This commit deprecates those
settings.

relates #21798
2017-05-09 16:14:57 -07:00
Tal Levy 945b3cd4ca fix ingest pipeline timestamp timezone format check to be even more generic 2017-05-09 15:31:32 -07:00
Tal Levy d2416a3a40 modify ingest timestamp format check to remove timezone 2017-05-09 14:33:43 -07:00
Jason Tedor 8f873620ee Inline global checkpoints
Today we rely on background syncs to relay the global checkpoint under
the mandate of the primary to its replicas. This means that the global
checkpoint on a replica can lag far behind the primary. The commit moves
to inlining global checkpoints with replication requests. When a
replication operation is performed, the primary will send the latest
global checkpoint inline with the replica requests. This keeps the
replicas closer in-sync with the primary.

However, consider a replication request that is not followed by another
replication request for an indefinite period of time. When the replicas
respond to the primary with their local checkpoint, the primary will
advance its global checkpoint. During this indefinite period of time,
the replicas will not be notified of the advanced global
checkpoint. This necessitates a need for another sync. To achieve this,
we perform a global checkpoint sync when a shard falls idle.

Relates #24513
2017-05-09 15:08:11 -04:00
Jason Tedor f222748506 Increase compilation limit in ingest tests
These tests are running into the script compilation limit. This commit
increases the limit for these tests.
2017-05-09 09:42:02 -04:00
Tal Levy 423b0f5e3d add option for _ingest.timestamp to use new ZonedDateTime (#24030)
Previously, Mustache would call `toString` on the `_ingest.timestamp`
field and return a date format that did not match Elasticsearch's
defaults for date-mapping parsing. The new ZonedDateTime class in Java 8
happens to do format itself in the same way ES is expecting.

This commit adds support for a feature flag that enables the usage of this new date format 
that has more native behavior.

Fixes #23168.

This new fix can be found in the form of a cluster setting called
`ingest.new_date_format`. By default, in 5.x, the existing behavior
will remain the same. One will set this property to `true` in order to
take advantage of this update for ingest-pipeline convenience.
2017-05-08 15:06:39 -07:00
Jason Tedor 3e485c2ca5 Fix plugin installation permissions
When installing plugin permissions, we try to set the permissions on all
installed files ourselves because a umask from the user could violate
everything needed to get the permissions right. Sadly, directories were
not handled correctly at all and so we were still left with broken
installations with umasks like 0077. This commit fixes this issue, adds
a thorough unit test for the situation, and most importantly, adds a
test that sets the umask before installing the plugin.

Relates #24527
2017-05-06 13:25:10 -04:00
Jim Ferenczi 953add8e70 Add 5.4.0 to bwc versions 2017-05-04 15:57:18 +02:00
Simon Willnauer 8356df0846 [TEST] Add a test that alias requests are dense for all indices 2017-05-04 14:29:59 +02:00
Simon Willnauer 07f106d39c [TEST] Rollback temporarily disabled field_caps test (#24483) 2017-05-04 14:14:22 +02:00
Simon Willnauer 14e57bf9f8 Add cross cluster support to `_field_caps` (#24463)
To support kibana this commit adds an internal optimization
to support the cross cluster syntax for indices on the `_field_caps`
API.

Closes #24334
2017-05-04 11:44:54 +02:00
Jason Tedor 5de2bcc624 Fix license header in WildflyIT.java
The license header in this file was after and on the same line as the
package statement. This commit moves the package statement to be after
the license header.
2017-05-02 11:56:49 -04:00
Jason Tedor 40ff169c54 Set available processors for Netty
Netty uses the number of processors for sizing various resources (e.g.,
thread pools, buffer pools, etc.). However, it uses the runtime number
of available processors which might not match the configured number of
processors as set in Elasticsearch to limit the number of threads (for
example, in Docker containers). A new feature was added to Netty that
enables configuring the number of processors Netty should see for sizing
this various resources. This commit takes advantage of this feature to
set this number of available processors to be equal to the configured
number of processors set in Elasticsearch.

Relates #24420
2017-05-01 19:27:28 -04:00
Yannick Welsch b4cff19842 Run gradle vagrantUpdateVersions to add version 5.3.2 2017-04-27 17:29:33 +02:00
Ryan Ernst 4a5c3c5a4a Test: Write node ports file before starting tribe service (#24351)
The tribe service can take a while to initialize, depending on how many cluster it needs to connect to. This change moves writing the ports file used by tests to before the tribe service is started.
2017-04-27 09:59:54 +02:00
Adrien Grand 1be2800120 Only allow one type on 7.0 indices (#24317)
This adds the `index.mapping.single_type` setting, which enforces that indices
have at most one type when it is true. The default value is true for 6.0+ indices
and false for old indices.

Relates #15613
2017-04-27 08:43:20 +02:00
Nik Everett 0c12d0ce37 Make bats tests refuse to start on non-VMs (#24315)
The bats tests are descructive and must be run as root. This is a
horrible combination on any sane system but perfectly fine to do
in a VM. This change modifies the tests so they revuse to start
unless they are in an environment with an `/etc/is_vagrant_vm`
file. The Vagrantfile creates it on startup.

Closes #24137
2017-04-26 10:40:35 -04:00
Jason Tedor c9a6d66bd5 Only test hard linking to directory on macOS
This skips trying to create a hard link to a directory in the evil node
tests on non-macOS operating systems.
2017-04-24 11:04:58 -04:00
Jason Tedor 1500beafc7 Check for default.path.data included in path.data
If the user explicitly configured path.data to include
default.path.data, then we should not fail the node if we find indices
in default.path.data. This commit addresses this.

Relates #24285
2017-04-24 09:31:54 -04:00
Jason Tedor d93d955563 Revert "Temporarily change Wildfly artifact location"
This reverts commit 8c0329d915.
2017-04-23 11:40:45 -04:00
Nik Everett 4423e1b78f Test search templates during rolling upgrade test (#24258)
In #24251 we fix an issue with stored search templates that
this test would have discovered: stored search templates cause
the node to refuse to start. Technically a "restart" test would
have caught this as well and would have caught it more quickly.
But we already *have* an upgrade test and we don't have restart tests.
And testing this on upgrade is a good thing too.
2017-04-22 13:37:13 -04:00
Jason Tedor c5396839c4 Skip Wildfly tests on JDK 9
It seems that Wildfly 10 can not be made to start in a fully-functional
form on JDK 9, so this commit skips running the Wildfly integration
tests on JDK 9.
2017-04-22 10:42:44 -04:00
Jason Tedor 8c0329d915 Temporarily change Wildfly artifact location
This commit temporarily changes the Wildfly artifact location to prevent
builds from failing due to a Red Hat datacenter outage.
2017-04-21 16:46:35 -04:00
Jason Tedor 2dd924bc15 Add Wildfly integration test
An important use case for our users is deploying our clients inside of
applications containers like Wildly. Sometimes, we make changes that
unintentionally break this use case. We need to know before we ship a
release that we have broken such use cases. As Wildfly is one of the
bigger application containers, this commit starts by adding an
integration test that deploys an application using the transport client
to Wildfly and ensures that all is well. Future work can add similar
integration tests for the low-level and high-level REST clients.

Relates #24147
2017-04-21 12:51:14 -04:00
Ryan Ernst ba48674695 Build: Move plugin cli and tests to distribution tool (#24220)
The plugin cli currently resides inside the elasticsearch jar. This
commit moves it into a plugin-cli jar. This is change alone is a no-op;
it does not change anything about what is loaded at runtime. But it will
allow easier testing (with fixtures in the future to test ES or maven
installation), as well as eventually not loading these classes when
starting elasticsearch.
2017-04-21 09:25:58 -07:00
Nik Everett a427d1dc5e Add 5.3.1 to bwc versions 2017-04-20 11:09:18 -04:00
Ryan Ernst c7e9231a86 Plugins: Remove leniency for missing plugins dir (#24173)
This leniency was left in after plugin installer refactoring for 2.0
because some tests still relied on it. However, the need for this
leniency no longer exists.
2017-04-19 09:09:34 -07:00
Areek Zillur 4f773e2dbb Replicate write failures (#23314)
* Replicate write failures

Currently, when a primary write operation fails after generating
a sequence number, the failure is not communicated to the replicas.
Ideally, every operation which generates a sequence number on primary
should be recorded in all replicas.

In this change, a sequence number is associated with write operation
failure. When a failure with an assinged seqence number arrives at a
replica, the failure cause and sequence number is recorded in the translog
and the sequence number is marked as completed via executing `Engine.noOp`
on the replica engine.

* use zlong to serialize seq_no

* Incorporate feedback

* track write failures in translog as a noop in primary

* Add tests for replicating write failures.

Test that document failure (w/ seq no generated) are recorded
as no-op in the translog for primary and replica shards

* Update to master

* update shouldExecuteOnReplica comment

* rename indexshard noop to markSeqNoAsNoOp

* remove redundant conditional

* Consolidate possible replica action for bulk item request
depanding on it's primary execution

* remove bulk shard result abstraction

* fix failure handling logic for bwc

* add more tests

* minor fix

* cleanup

* incorporate feedback

* incorporate feedback

* add assert to remove handling noop primary response when 5.0 nodes are not supported
2017-04-19 01:23:54 -04:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling (#21392)
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes #20240
2017-04-18 15:07:08 -07:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Nik Everett 9653551072 Add package test to catch leaking default.data.path setting (#24029)
Adds a packaging test that would have detected #23981. The test configures path.data and validates that nothing ends up in default.path.data.
2017-04-17 16:36:32 -04:00
Jason Tedor d4947c26ae Use jvm.options where needed in packaging tests
We have some packaging tests where we use a custom jvm.options file. The
flags we use here are barebones, just enough to exercise that using a
custom jvm.options files actually works. Due to this, we are missing a
Log4j flag that prevents Log4j from trying to use JMX. If Log4j tries to
use JMX, it hits a security manager exception and tries to log
this. This attempt to log happens before we've configured
logging. Previously, Elasticsearch was lenient here so this was treated
as harmless and the test could march on. Now, we fail startup if we
detect an attempt to log before logging is configured so this prevents
Elasticsearch from starting if we do not have jvm.options files in place
that prevent these log messages from being written before logging is
being configured. This commit adds jvm.options files in the places need
to prevent this.
2017-04-13 23:19:07 -04:00
Jason Tedor 52c9159fff Remove awaits fix from evil JNA native tests
These tests were marked as awaits fix due to JNA requiring a version of
glibc greater than or equal to version 2.14. Since we still support
systems that would not have this version, we have released our own JNA
dependency that is built to support earlier versions of glibc. This
commit removes some await fixes that were added to tests that failed as
a result of this situation.
2017-04-13 06:53:02 -04:00
Jason Tedor edd16fa27e Register error listener in evil logger tests
This test needs an error listener registered since we configure logging
here.
2017-04-12 21:23:05 -04:00