Commit Graph

1042 Commits

Author SHA1 Message Date
Nhat Nguyen 51151027cd TEST: Add bwc recovery tests with synced-flush index
Although the master branch does not affect by #31482, it's helpful to
have BWC tests that verify the peer recovery with a synced-flush index.
This commit adds the bwc tests from #31506 to the master branch.

Relates #31482
Relates #31506
2018-06-22 20:14:06 -04:00
Ryan Ernst 59e7c6411a
Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
TransportRequestHandler currently contains 2 messageReceived methods,
one which takes a Task, and one that does not. The first just delegates
to the second. This commit changes all existing implementors of
TransportRequestHandler to implement the version which takes Task, thus
allowing the class to be a functional interface, and eliminating the
need to throw exceptions when a task needs to be ensured.
2018-06-22 07:36:03 -07:00
Christoph Büscher 02346c20a2
Rankeval: Fold template test project into main module (#31203)
This change moves tests in `smoke-test-rank-eval-with-mustache` into the main
ranking evaluation module by declaring that the integration testing cluster
requires the `lang-mustache` plugin. This avoids having to maintain the qa
project for only one basic test suite.
2018-06-15 15:55:39 +02:00
Ben Abrams 87a676e4d5 Do not set vm.max_map_count when unnecessary (#31285)
This commit modifies the Sys V init startup scripts to only modify
vm.max_map_count if needed. In this case, needed means that the current
value is less than our default value of 262144 maps.
2018-06-14 21:41:02 -04:00
Nik Everett d6d0727aac
QA: Fix resolution of default distribution (#31351)
If you run `./gradlew -p qa bwcTest -Dtests.distribution=zip` then we
need to resolve older versions of the default distribution. Since those
aren't available in maven central, we need add the elastic maven repo to
the project.
2018-06-14 18:25:47 -04:00
Andy Bristol 01d64b128b [test] opensuse packaging turn up debug logging 2018-06-13 11:15:04 -07:00
Jason Tedor 0bfd18cc8b
Revert upgrade to Netty 4.1.25.Final (#31282)
This reverts upgrading to Netty 4.1.25.Final until we have a cleaner
solution to dealing with the object cleaner thread.
2018-06-12 19:26:18 -04:00
Jason Tedor 563141c6c9
Upgrade to Netty 4.1.25.Final (#31232)
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
2018-06-11 16:55:07 -04:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Jason Tedor 94be9b471f
Rename elasticsearch-core to core (#31185)
This commit renames :libs:elasticsearch-core to :libs:core.
2018-06-07 16:50:21 -04:00
Nik Everett dfcc939ef8 QA: Better seed nodes for rolling restart
Use all running nodes as unicast seeds in the rolling restart tests to
avoid a race between pinging and the tests. Without this if the tests
are too fast then when a new node comes up and pings its single
configured seed node that node *might* not have a ping from the other
running node.
2018-06-07 13:30:37 -04:00
Tim Brooks 237f9b8930
Add nio-transport as option for http smoke tests (#31162)
This is related to #27260 and #28898. This commit adds the transport-nio
plugin as a random option when running the http smoke tests. As part of
this PR, I identified an issue where cors support was not properly
enabled causing these tests to fail when using transport-nio. This
commit also fixes that issue.
2018-06-07 09:46:36 -06:00
Nik Everett 56207ea43d QA: Set better node names on rolling restart tests
These should help with debugging failures.
2018-06-07 11:25:41 -04:00
Luca Cavanna be4a101ea1
Add high-level client methods that accept RequestOptions (#31069)
With #30490 we have introduced a new way to provide request options
whenever sending a request using the high-level REST client. Before you
could provide headers as the last argument varargs of each API method,
now you can provide `RequestOptions` that in the future will allow to
provide more options which can be specified per request.

This commit deprecates all of the client methods that accept a `Header`
varargs argument in favour of new methods that accept `RequestOptions`
instead. For some API we don't even go through deprecation given that
they were not released since they were added, hence in that case we can
just move them to the new method.
2018-06-06 23:17:45 +02:00
Nik Everett 7c59e7690e
QA: Switch xpack rolling upgrades to three nodes (#31112)
This is much more realistic and can find more issues. This causes the
"mixed cluster" tests to be run twice so I had to fix the tests to work
in that case. In most cases I did as little as possible to get them
working but in a few cases I went a little beyond that to make them
easier for me to debug while getting them to work. My test changes:

1. Remove the "basic indexing" tests and replace them with a copy of the
tests used in the OSS. We have no way of sharing code between these two
projects so for now I copy.
2. Skip the a few tests in the "one third" upgraded scenario:
  * creating a scroll to be reused when the cluster is fully upgraded
  * creating some ml data to be used when the cluster is fully ugpraded
3. Drop many "assert yellow and that the cluster has two nodes"
assertions. These assertions duplicate those made by the wait condition
and they fail now that we have three nodes.
4. Switch many "assert green and that the cluster has two nodes" to 3
nodes. These assertions are unique from the wait condition and, while
I imagine they aren't required in all cases, now is not the time to
find that out. Thus, I made them work.
5. Rework the index audit trail test so it is more obvious that it is
the same test expecting different numbers based on the shape of the
cluster. The conditions for which number are expected are fairly
complex because the index audit trail is shut down until the template
for it is upgraded and the template is upgraded when a master node is
elected that has the new version of the software.
6. Add some more information to debug the index audit trail test because
it helped me figure out what was going on.

I also dropped the `waitCondition` from the `rolling-upgrade-basic`
tests because it wasn't needed.

Closes #25336
2018-06-06 11:59:16 -04:00
Luca Cavanna 70749e01c4
Cross Cluster Search: preserve remote status code (#30976)
In case an error is returned when calling search_shards on a remote
cluster, which will lead to throwing an exception in the coordinating
 node, we should make sure that the status code returned by the
 coordinating node is the same as the one returned by the remote
 cluster. Up until now a 500 - Internal Server Error was always
 returned. This commit changes this behaviour so that for instance if an
 index is not found, which causes an 404, a 404 is also returned by the
 coordinating node to the client.

 Closes #27461
2018-06-01 08:53:53 +02:00
Nik Everett b225f5e5c6
HLRest: Allow caller to set per request options (#30490)
This modifies the high level rest client to allow calling code to
customize per request options for the bulk API. You do the actual
customization by passing a `RequestOptions` object to the API call
which is set on the `Request` that is generated by the high level
client. It also makes the `RequestOptions` a thing in the low level
rest client. For now that just means you use it to customize the
headers and the `httpAsyncResponseConsumerFactory` and we'll add
node selectors and per request timeouts in a follow up.

I only implemented this on the bulk API because it is the first one
in the list alphabetically and I wanted to keep the change small
enough to review. I'll convert the remaining APIs in a followup.
2018-05-31 13:59:52 -04:00
Christoph Büscher 1ea9f11b03
Change ScriptException status to 400 (bad request) (#30861)
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes #12315
2018-05-30 14:00:07 +02:00
Andy Bristol 4001097a68
[test] packaging: use shell when running commands (#30852)
When subprocesses are started with ProcessBuilder, they're forked by the
java process directly rather than from a shell, which can be surprising
for our use case here in the packaging tests which is similar to
scripting.

This commit changes the tests to run their subprocess commands in a
shell, using the bash -c <script> syntax for commands on linux and using
the powershell.exe -Command <script> syntax for commands on windows.
This syntax on windows is essentially what the tests were already doing.
2018-05-29 15:17:11 -07:00
Andy Bristol ba8bb1d4a1 [test] packaging test logging for suse distros 2018-05-29 11:06:57 -07:00
Alpar Torok f5de25c618 Remove left-over comment 2018-05-28 11:52:26 +03:00
Andy Bristol 4bd2607597
[docs] explainer for java packaging tests (#30825) 2018-05-24 17:05:21 -07:00
Christoph Büscher 9cb6b90a99
[Tests] Move templated _rank_eval tests (#30679)
This change moves the ranking evaluation tests that use templates to the
existing yml rest tests instead of extending ESIntegTestCase.

Closes #30628
2018-05-24 18:31:13 +02:00
Andy Bristol a1b538122c
[test] java tests for archive packaging (#30734)
Ports the first couple tests for archive distributions from the old bats
project to the new java project that includes windows platforms,
consolidating them into one test method that tests that the
distributions can be extracted and their contents verified. Includes the
zip distributions which were not tested in the bats project.
2018-05-23 10:37:57 -07:00
Nik Everett a5d90e919f
QA: Add xpack tests to rolling upgrade (#30795)
A rolling upgrade from oss Elasticsearch to the default distribution of
Elasticsearch is significantly different than a full cluster restart to
install a plugin and is again different from starting a new cluster with
xpack installed. So this adds some basic tests to make sure that the
rolling upgrade that enables xpack works at all.

This also removes some unused imports from the tests that I modified in
PR #30728. I didn't mean to leave them.
2018-05-22 17:17:34 -04:00
Luca Cavanna a17d6cab98
Replace Request#setHeaders with addHeader (#30588)
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
2018-05-22 20:32:30 +02:00
Nik Everett be8c0f27be
QA: Switch rolling upgrade to 3 nodes (#30728)
Switches the rolling upgrade tests from upgrading two nodes to upgrading
three nodes which is much more realistic and much better able to find
unexpected bugs. It upgrades the nodes one at a time and runs tests
between each upgrade. As such this now has four test runs:

1. Old
2. One third upgraded
3. Two thirds upgraded
4. Upgraded

It sets system properties so the tests can figure out which stage they
are in. It reuses the same yml tests for the "one third" and "two
thirds" cases because they are *almost* the same case.

This rewrites the yml-based indexing tests to be Java based because the
yml-based tests can't handle different expected values for the counts.
And the indexing tests need that when they are run twice.
2018-05-21 13:15:13 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Nik Everett 6695d11a97
Switch many QA projects to use new style requests (#30574)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the the old requests. This
changes many calls in the `qa` projects to use the new version.
2018-05-15 17:48:47 -04:00
Nik Everett 869b639d14
QA: System property to override distribution (#30591)
This configures all `qa` projects to use the distribution contained in
the `tests.distribution` system property if it is set. The goal is to
create a simple way to run tests against the default distribution which
has x-pack basic features enabled while not forcing these tests on all
contributors. You run these tests by doing something like:

```
./gradlew -p qa -Dtests.distribution=zip check
```

or

```
./gradlew -p qa -Dtests.distribution=zip bwcTest
```

x-pack basic *shouldn't* get in the way of any of these tests but
nothing is ever perfect so this we have to disable a few when running
with the zip distribution.
2018-05-15 17:16:16 -04:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Christoph Büscher cc93131318
Forbid expensive query parts in ranking evaluation (#30151)
Currently the ranking evaluation API accepts the full query syntax for
the queries specified in the evaluation set and executes them via multi
search. This potentially runs costly aggregations and suggestions too.
This change adds checks that forbid using aggregations, suggesters, 
highlighters and the explain and profile options in the queries that are 
run as part of the ranking evaluation since they are irrelevent in the 
context of this API.
2018-05-14 17:36:26 +02:00
Nik Everett 51fa8739ea
Reindex: Fold "with all deps" project into reindex (#30154)
This folds the `:qa:smoke-test-reindex-with-all-modules` project into
`:modules:reindex` by declaring the reindex's integration testing
cluster requires the `parent-join` and `lang-painless` plugins and then
moving all of the integration tests that depended on parent-join and
painless into reindex.

It saves us one cluster start up during the build at the cost of a
little of the reindex module's "purity". Since the reindex module *does*
have unit tests that test scripting without painless I'm fairly ok with
that.
2018-05-10 08:02:23 -04:00
Jason Tedor beee5fe004
Respect accept header on no handler (#30383)
Today when processing a request for a URL path for which we can not find
a handler we send back a plain-text response. Yet, we have the accept
header in our hand and can respect the accepted media type of the
request. This commit addresses this.
2018-05-04 18:13:50 -04:00
Jim Ferenczi 6a65b3b874 [Test] Add analysis-nori plugin to the vagrant tests
relates #30397
2018-05-04 22:46:04 +02:00
Ryan Ernst 3aec8c7725
Packaging: Set elasticsearch user to have non-existent homedir (#29007)
This commit adds setting the homedir for the elasticsearch user to the
adduser command in the packaging preinstall script. While the
elasticsearch user is a system user, it is sometimes conventient to have
an existing homedir (even if it is not writeable). For example, running
cron as the elasticsearch user will try to change dir to the homedir.

closes #14453
2018-05-02 14:33:34 -07:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Ryan Ernst fba2f00a73
Packaging: Unmark systemd service file as a config file (#29004)
Systemd overrides should happen through /etc/systemd/system, not
directly editing the service file. This commit removes marking the
service file as configuration for rpm and deb packages.
2018-05-02 09:48:49 -07:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00
Andy Bristol 65e5868a55
[test] add java packaging test project (#30161)
[test] add java packaging test project

Adds a project for building and running packaging tests written in java
for portability. The vagrant tasks use jars on the packagingTest
configuration, which are built in the same project. No tests are added
yet.

Corresponding changes are not made to :x-pack:qa:vagrant because the
java packaging tests will all be consolidated into one project.

For #26741
2018-04-30 16:35:26 -07:00
Nik Everett f4ed902698
CCS: Drop http address from remote cluster info (#29568)
They are expensive to fetch and no longer needed by Kibana so they
*shouldn't* be needed by anyone else either.

Closes #29207
2018-04-27 14:19:00 -04:00
Nik Everett 912fbb2211
Reindex: Fold "from old" tests into reindex module (#30142)
This folds the `:qa:reindex-from-old` project into the `:modules:reindex`
project. This should speed up the build marginally by removing a single
clsuter start up at the cost of having to wait for old versions of
Elasticsearch to start up when checking reindex's integration tests.
Those don't take that long so this feels worth it.
2018-04-27 14:04:37 -04:00
Tanguy Leroux 7ae3b3b155
Move repository-s3 fixture tests to QA test project (#29372)
This commit moves the repository-s3 fixture test added in #29296 in a
new `repository-s3/qa/amazon-s3` project. This new project allows the
REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no env var
is provided, then the tests are executed using the fixture added
in #29296.

The REST tests located at the `repository-s3`plugin  project now only 
verify that the plugin is correctly loaded.

The REST tests have been adapted to allow a bucket name and a base 
path to be specified as env vars. This way it is possible to run the tests
with different base paths (could be anything, like a CI job name or a
branch name) without multiplicating buckets.

Related to #29349
2018-04-27 16:49:06 +02:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Tanguy Leroux df42358967
[Test] Fix docs check for DEB package in packaging tests (#30126)
The packaging tests for Debian based distro is loooking
for docs in /usr/share/elasticsearch, but it should be
/usr/share/elasticsearch-oss for the oss package.
2018-04-25 18:52:37 +02:00
Jason Tedor 3cadd5c40c Only enable modules to have native controllers
This commit removes the ability for a plugin to have a native controller
as leaves it as only modules can have a native controller.
2018-04-20 15:34:02 -07:00
Jason Tedor 0750f5d6f3 Add conflicts for default and oss packages
This commit adds conflicts between the default and oss packages so that
a user can not install both via a package manager (as they share paths).
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 7975280383
Remove remaining tribe node references (#29574)
While tribe node was removed in
https://github.com/elastic/elasticsearch/pull/28443, there remained a
couple lingering references to it in docs and code. This commit removes
those remaining references.
2018-04-19 18:02:01 -07:00
Ryan Ernst 4f282e9e32
Build: Move java home checks to pre-execution phase (#29548)
This commit moves the checks on JAVAX_HOME (where X is the java version
number) existing to the end of gradle's configuration phase, and based
on whether the tasks needing the java home are configured to execute.

relates #29519
2018-04-19 09:51:52 -07:00
Jason Tedor a548a7f2cb
Fix full cluster restart test recovery (#29545)
The test was using a parameter on GET /_cluster/health that older nodes
do not understand. Yet, we do no even need to make this call here, we
can use ensure green for the index.
2018-04-18 08:43:43 -04:00
Jason Tedor 21e73403d6 Mute full cluster restart test recovery test
This test is failing because of an addition of a call to GET
/_cluster/health with the parameter wait_for_no_initializing_shards set
to true. As older versions of Elasticsearch do not understand this
parameter, this request fails and the test fails. This commit marks this
test as awaiting a fix.
2018-04-16 17:26:35 -04:00
Boaz Leskes 8f91743768 FullClusterRestartIT.testRecovery should wait for all initializing shards
Shards that are not fully bake may not execute the flush call, causing
subsequent unexpected translog recoveries which fails the test.

Closes #27817
2018-04-16 17:51:08 +02:00
Jason Tedor b883e1217f
Lazy configure build tasks that require older JDKs (#29519)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.
2018-04-14 15:44:43 -04:00
Jason Tedor 27fafa24f5
Use proper Java version for BWC builds (#29493)
Today we have JAVA_HOME for the compiler Java home and RUNTIME_JAVA_HOME
for the test Java home. However, when we compile BWC nodes and run them,
neither of these Java homes might be the version that was suitable for
that BWC node (e.g., 5.6 requires JDK 8 to compile and to run). This
commit adds support for the environment variables JAVA\d+_HOME and uses
the appropriate Java home based on the version of the node being
started. We even do this for reindex-from-old which requires JDK 7 for
these very old nodes. Note that these environment variables are not
required if not running BWC tests, and they are strictly required if
running BWC tests.
2018-04-13 12:41:18 -04:00
Adrien Grand 4918924fae
Remove legacy mapping code. (#29224)
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
2018-04-11 09:41:37 +02:00
Adrien Grand 85f5382a3c
Fix more query extraction bugs. (#29388)
I found the following bugs:
 - The 6.0 logic for conjunctions didn't work when there were only `match_all`
   queries in MUST/FILTER clauses as they didn't propagate the `matchAllDocs`
   flag.
 - Some queries still had the same issue as `BooleanQuery` used to have with
   duplicate terms (see #28353), eg. `MultiPhraseQuery`.

Closes #29376
2018-04-06 10:44:34 +02:00
Adrien Grand e2d771e319 Disable failing query in QueryBuilderBWCIT.
Relates #29376
2018-04-04 22:09:26 +02:00
Jason Tedor 5cdd831a31
Remove silent batch mode from install plugin (#29359)
Today we have a silent batch mode in the install plugin command when
standard input is closed or there is no tty. It appears that
historically this was useful when running tests where we want to accept
plugin permissions without having to acknowledge them. Now that we have
an explicit batch mode flag, this use-case is removed. The motivation
for removing this now is that there is another place where silent batch
mode arises and that is when a user attempts to install a plugin inside
a Docker container without keeping standard input open and attaching a
tty. In this case, the install plugin command will treat the situation
as a silent batch mode and therefore the user will never have the chance
to acknowledge the additional permissions required by a plugin. This
commit removes this silent batch mode in favor of using the --batch flag
when running tests and requiring the user to take explicit action to
acknowledge the additional permissions (either by leaving standard input
open and attaching a tty, or by passing the --batch flags themselves).

Note that with this change the user will now see a null pointer
exception when they try to install a plugin in a Docker container
without keeping standard input open and attaching a tty. This will be
addressed in an immediate follow-up, but because the implications of
that change are larger, they should be handled separately from this one.
2018-04-03 21:23:01 -04:00
Andy Bristol 7bf9091942
[test] packaging: gradle tasks for groovy tests (#29046)
The vagrant test plugin adds tasks for the groovy packaging tests,
which run after the bats packaging test tasks.Rename the 'bats'
configuration to 'packaging' and remove the option to inherit
archives from this configuration.
2018-03-26 13:43:09 -07:00
Christoph Büscher e4b30071bb
RankEvalRequest should implement IndicesRequest (#29188)
Change RankEvalRequest to implement IndicesRequest, so it gets treated
in a similar fashion to regular search requests e.g. by security.
2018-03-22 11:58:55 +01:00
Nik Everett 5135484a27
Test: Expect extra Exception in die_with_dignity (#29138)
I did a little digging. It looks like IOException is thrown when the other
side closes its connection while we're waiting on our buffer to fill up. We
totally expect that in this test. It feels to me like we should throw a
`ConnectionClosedException` but upstream does not agree:
https://issues.apache.org/jira/browse/HTTPASYNC-134

While we *could* catch the exception and transform it ourselves that
seems like a bigger change than is merited at this point.

Closes #29136
2018-03-19 14:02:55 -04:00
Christoph Büscher 80532229a9
Move indices field from RankEvalSpec to RankEvalRequest (#28341)
Currently we store the indices specified in the request URL together with all
the other ranking evaluation specification in RankEvalSpec. This is not ideal
since e.g. the indices are not rendered to xContent and so cannot be parsed
back. Instead we should keep them in RankEvalRequest.
2018-03-19 16:26:02 +01:00
Jason Tedor 8342ba9108 Add logging output when starting Wildfly
This commit (which will be reverted soon) adds logging on the output of
starting Wildfly. This is needed to debug an issue with Wildfly not
starting in CI.
2018-03-16 11:10:23 -04:00
Lee Hinman 8e8fdc4f0e
Decouple XContentBuilder from BytesReference (#28972)
* Decouple XContentBuilder from BytesReference

This commit removes all mentions of `BytesReference` from `XContentBuilder`.
This is needed so that we can completely decouple the XContent code and move it
into its own dependency.

While this change appears large, it is due to two main changes, moving
`.bytes()` and `.string()` out of XContentBuilder itself into static methods
`BytesReference.bytes` and `Strings.toString` respectively. The rest of the
change is code reacting to these changes (the majority of it in tests).

Relates to #28504
2018-03-14 13:47:57 -06:00
Jack Conradson 42fe66162e
Fix Parsing Bug with Update By Query for Stored Scripts (#29039)
This changes the parsing logic for stored scripts in update by query to match the parsing logic for scripts in general Elasticsearch.

Closes #28002
2018-03-14 07:12:15 -07:00
Jason Tedor d3d7c04524 Fix description of die with dignity plugin
This commit adjusts the description of the die with dignity plugin which
was leftover from a previous iteration on this work.
2018-03-14 07:45:32 -04:00
Jason Tedor 8b6fbe2c11
Add test for dying with dignity (#28987)
I have long wanted an actual test that dying with dignity works. It is
tricky because if dying with dignity works, it means the test JVM dies
which is usually an abnormal condition. And anyway, how does one force a
fatal error to be thrown. I was motivated to investigate this again by
the fact that I missed a backport to one branch leading to an issue
where Elasticsearch would not successfully die with dignity. And now we
have a solution: we install a plugin that throws an out of memory error
when it receives a request. We hack the standalone test infrastructure
to prevent this from failing the test. To do this, we bypass the
security manager and remove the PID file for the node; this tricks the
test infrastructure into thinking that it does not need to stop the
node. We also bypass seccomp so that we can fork jps to make sure that
Elasticsearch really died. And to be extra paranoid, we parse the logs
of the dead Elasticsearch process to make sure it died with
dignity. Never forget.
2018-03-12 23:20:07 -04:00
Jason Tedor 6331bcaf76
Create keystore on package install (#28928)
This commit removes the ability to specify that a plugin requires the
keystore and instead creates the keystore on package installation or
when Elasticsearch is started for the first time. The reason that we opt
to create the keystore on package installation is to ensure that the
keystore has the correct permissions (the package installation scripts
run as root as opposed to Elasticsearch running as the elasticsearch
user) and to enable removing the keystore on package removal if the
keystore is not modified.
2018-03-12 12:48:00 -04:00
Ryan Ernst 4216fc9f64
Plugins: Allow modules to spawn controllers (#28968)
This commit makes the controller spawner also look under modules. It
also fixes a bug in module security policy loading where the module is a
meta plugin.
2018-03-11 09:01:27 -07:00
Ryan Ernst 3a64e6d121
Test: Remove specifying zip distribution in qa tests (#28805)
Applying the rest test gradle plugin already uses the zip distribution
by default, so specifying it explicitly is not necessary. These are
leftovers from before zip was the default for rest tests.
2018-02-23 13:45:38 -08:00
Yu 7d8fb69d50 version set in ingest pipeline (#27573)
Add support version and version_type in ingest pipelines

Add support for setting document version and version type in set
processor of an ingest pipeline.
2018-02-21 09:34:51 +01:00
Igor Motov c90d0fdf6b Tests: don't wait for completion while trying to get completed task
Nodes are reusing task ids after restart. So in some rare circumstances
the same task id might be assigned to the reindexing task stored by the
old cluster and the new task that is trying to retrieve the task
results. As a result, the get task request can timeout waiting on
itself. Since we already waited for the task to finish before restarting
the cluster, waiting for the task here doesn't make any sense to start
with.

Fixes #28732
2018-02-20 14:14:47 -05:00
Lee Hinman d4fddfa2a0
Remove log4j dependency from elasticsearch-core (#28705)
* Remove log4j dependency from elasticsearch-core

This removes the log4j dependency from our elasticsearch-core project. It was
originally necessary only for our jar classpath checking. It is now replaced by
a `Consumer<String>` so that the es-core dependency doesn't have external
dependencies.

The parts of #28191 which were moved in conjunction (like `ESLoggerFactory` and
`Loggers`) have been moved back where appropriate, since they are not required
in the core jar.

This is tangentially related to #28504

* Add javadocs for `output` parameter

* Change @code to @link
2018-02-20 09:15:54 -07:00
Jason Tedor 1fa701c18d
Fix using relative custom config path
Previously a user could set a custom config path to a relative directory
using ES_PATH_CONF. In a previous change related to enabling GC logging
by default, we forced the working directory for Elasticsearch to be
ES_HOME. This had the impact of causing all relative paths to be
relative to ES_HOME, against the intent of the user. This commit
addresses this by making ES_PATH_CONF absolute before we switch the
working directory to ES_HOME.

Relates #28700
2018-02-16 06:08:54 -05:00
Jason Tedor 3e846ab251
Handle throws on tasks submitted to thread pools
When we submit a task to a thread pool for asynchronous execution, we
are returned a future. Since we submitted to go asynchronous, these
futures are not inspected for failure (we would have to block a thread
to do that). While we have on failure handlers for exceptions that are
thrown during execution, we do not handle throwables that are not
exceptions and these end up silently lost. This commit adds a check
after the runnable returns that inspects the status of the future. If an
unhandled throwable occurred during execution, this throwable is
propogated out where it will land in the uncaught exception handler.

Relates #28667
2018-02-15 11:59:12 -05:00
Lee Hinman 7c1f5f5054
Move more XContent.createParser calls to non-deprecated version (#28670)
* Move more XContent.createParser calls to non-deprecated version

This moves more of the callers to pass in the DeprecationHandler.

Relates to #28504

* Use parser's deprecation handler where available
2018-02-14 09:01:40 -07:00
Andy Bristol 53d1cde7bc
[TEST] packaging: function to collect debug info (#28608)
[TEST] packaging: function to collect debug info

Sometimes when packaging tests fail in CI the test logs aren't enough to
tell what went wrong. This routine helps collect more info about the
state of the es installation at failure time
2018-02-13 10:51:13 -08:00
Michael Basnight e0bea70070
Generalize BWC logic (#28505)
Generalizing BWC building so that there is less code to modify for a release. This ensures we do not
need to think about what major or minor version is in the gradle code. It follows the general rules of the
elastic release structure. For more information on the rules, see the VersionCollection's javadoc.

This also removes the additional bwc snapshots that will never be released, such as 6.0.2, which were
being built and tested against every time we ran bwc tests.

Additionally, it creates 4 new projects that correspond to the different types of snapshots that may exist
for a given version. Its possible to now run those individual tasks to work out bwc logic whereas
previously it was impossible and the entire suite of bwc tests had to be run to work out any logic
changes in the build tools' bwc project. Please note that if the project does not make sense for the 
version that is current, that an error will be thrown from that individual project if an attempt is made to 
run it.

This should allow for automating the version bumps as well, since it removes all the hardcoded version
logic from the configs.
2018-02-09 14:55:10 -06:00
Nhat Nguyen 0edde25f09 TEST: use new nodes assumption in testUpdateSnapshotStatus
Using an assertion is not correct here as the list of new nodes can be
empty.
2018-02-08 13:11:21 -05:00
Lee Hinman 64adaffe11 [TEST] Expand failure message for wildfly integration tests 2018-02-06 15:34:41 -07:00
Luca Cavanna d860971572
REST high-level client: add support for split and shrink index API (#28425)
Relates to #27205
2018-02-01 16:37:01 +01:00
Simon Willnauer 3bf8554114
Remove tribe node support (#28443)
Tribe node has been superseeded by Cross-Cluster-Search. This change
removes the tribe node support entirely.
2018-01-30 20:40:19 +01:00
Ryan Ernst 3dd833ca0a
Plugins: Use one confirmation of all meta plugin permissions (#28366)
Currently meta plugins will ask for confirmation of security policy
exceptions for each bundled plugin. This commit collects the necessary
permissions of each bundled plugin, and asks for confirmation of all of
them at the same time.
2018-01-26 15:44:44 -08:00
Tanguy Leroux be74f11517
Replace jvm-example by two plugin examples (#28339)
This pull request replaces the jvm-example plugin (from the jvm/site plugins era) by two new plugins: a custom-settings that shows how to register and use custom settings (including secured settings) in a plugin, and rest-handler plugin that shows how to register a rest handler.

The two plugins now reside in the plugins/examples project. They can serve as sample plugins for users, a special attention has been put on documentation. The packaging tests have been adapted to use the custom-settings plugin.
2018-01-26 17:34:24 +01:00
Nik Everett 9d06cc09ae Reindex: Harden old ES version test
Our rest client throws exceptions in funny ways that might cause
`suppressed` exceptions to be eaten. This works around that in the
reindex-from-old tests so we don't stomp on a real failure. It is fairly
diryt so we should work on fixing the high level rest client.

Relates to #25453
2018-01-26 11:24:48 -05:00
Tanguy Leroux f4d0cf55be
Update packaging tests to work with meta plugins (#28336)
The current install_plugin() does not play well with meta plugins because 
it always checks for the plugin's descriptor file.

This commit changes the install_plugin() so that it only runs the install plugin 
command and lets the caller verify that the required files are correctly installed. 
It also adds a install_meta_plugin() function to install meta plugins.
2018-01-25 08:50:16 +01:00
Christoph Büscher a6bfe67f8b [Test] Lower bwc version for rank-eval rest tests
The API was backported to 6.2 so the version we test against on master can be
lowered to that.
2018-01-22 13:33:42 +01:00
Christoph Büscher 77dcaab34f
Simplify RankEvalResponse output (#28266)
Currenty the rest response of the ranking evaluation API wraps all inside an
enclosing `rank_eval` object. This is redundant since it is clear from the API
call and it doesn't provide any other useful information. This change removes
this.
2018-01-18 09:32:27 +01:00
Ryan Ernst 9435423844
Build: Automatically add projects under libs, qa, modules and plugins (#28279)
This commit lessens the burden on configuring settings.gradle when new
projects are added. In particular, this makes it trivial to move a
plugin to a module (or vice versa).
2018-01-17 19:54:43 -08:00
Jason Tedor 0a79555a12
Require JDK 9 for compilation (#28071)
This commit modifies the build to require JDK 9 for
compilation. Henceforth, we will compile with a JDK 9 compiler targeting
JDK 8 as the class file format. Optionally, RUNTIME_JAVA_HOME can be set
as the runtime JDK used for running tests. To enable this change, we
separate the meaning of the compiler Java home versus the runtime Java
home. If the runtime Java home is not set (via RUNTIME_JAVA_HOME) then
we fallback to using JAVA_HOME as the runtime Java home. This enables:
 - developers only have to set one Java home (JAVA_HOME)
 - developers can set an optional Java home (RUNTIME_JAVA_HOME) to test
   on the minimum supported runtime
 - we can test compiling with JDK 9 running on JDK 8 and compiling with
   JDK 9 running on JDK 9 in CI
2018-01-16 13:45:13 -05:00
Adrien Grand 05e851f0b0
Ignore the `-snapshot` suffix when comparing the Lucene version in the build and the docs. (#27927)
Currently if the Lucene version is `X.Y.Z-snapshot-{gitrev}`, then we will
expect the docs to have `X.Y.Z-snapshot` as a Lucene version. I would like
to change it to `X.Y.Z` so that this doesn't need changing when we move from a
snapshot to a final release.
2018-01-15 18:34:38 +01:00
Tim Brooks 3895add2ca
Introduce elasticsearch-core jar (#28191)
This is related to #27933. It introduces a jar named elasticsearch-core
in the lib directory. This commit moves the JarHell class from server to
elasticsearch-core. Additionally, PathUtils and some of Loggers are
moved as JarHell depends on them.
2018-01-15 09:59:01 -07:00
Igor Motov c75ac319a6
Add ability to associate an ID with tasks (#27764)
Adds support for capturing the X-Opaque-Id header from a REST request and storing it's value in the tasks that this request started. It works for all user-initiated tasks (not only search).

Closes #23250

Usage:
```
$ curl -H "X-Opaque-Id: imotov" -H "foo:bar" "localhost:9200/_tasks?pretty&group_by=parents"
{
  "tasks" : {
    "7qrTVbiDQKiZfubUP7DPkg:6998" : {
      "node" : "7qrTVbiDQKiZfubUP7DPkg",
      "id" : 6998,
      "type" : "transport",
      "action" : "cluster:monitor/tasks/lists",
      "start_time_in_millis" : 1513029940042,
      "running_time_in_nanos" : 266794,
      "cancellable" : false,
      "headers" : {
        "X-Opaque-Id" : "imotov"
      },
      "children" : [
        {
          "node" : "V-PuCjPhRp2ryuEsNw6V1g",
          "id" : 6088,
          "type" : "netty",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 67785,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        },
        {
          "node" : "7qrTVbiDQKiZfubUP7DPkg",
          "id" : 6999,
          "type" : "direct",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 98754,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        }
      ]
    }
  }
}
```
2018-01-12 15:34:17 -05:00
Jason Tedor ccaba016ac
Fix task ordering in rolling upgrade tests
The configuration of the upgraded cluster task was missing a dependency
on the stopping of the second old node in the cluster. In some cases
(e.g., --parallel) Gradle would then try to run the configuration of a
node in the upgraded cluster before it had even configured the old nodes
in the cluster.

Relates #28036
2018-01-09 17:20:55 -05:00
Jim Ferenczi 36729d1c46
Add the ability to bundle multiple plugins into a meta plugin (#28022)
This commit adds the ability to package multiple plugins in a single zip.
The zip file for a meta plugin must contains the following structure:

|____elasticsearch/
| |____   <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory)
| |____   <plugin2>  <-- The plugin files for plugin2
| |____   meta-plugin-descriptor.properties <-- example contents below
The meta plugin properties descriptor is mandatory and must contain the following properties:

description: simple summary of the meta plugin.
name: the meta plugin name
The installation process installs each plugin in a sub-folder inside the meta plugin directory.
The example above would create the following structure in the plugins directory:

|_____ plugins
| |____   <name_of_the_meta_plugin>
| | |____   meta-plugin-descriptor.properties
| | |____   <plugin1>
| | |____   <plugin2>
If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory.

|_____ config
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>

|_____ bin
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>
The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin
cannot have the same name than another plugin (or a sub-plugin inside another meta plugin).

It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed.

Closes #27316
2018-01-09 18:28:43 +01:00
Jason Tedor af3c2fcd65 Fix expected plugins test for transport-nio
We have a packaging test that tries to install all plugins, and then
asserts that all expected plugins are installed. The expected plugins
are dervied from the list of plugins in the plugins sub-project. The
plugin transport-nio was recently added here, but explicit commands to
install and remove this plugin were never added. This commit addresses
this.
2018-01-08 11:09:04 -05:00
Yannick Welsch ae9c3281fc [TEST] Wait for replicas to be allocated before shrinking
The full-cluster-restart tests are run with two nodes. This can lead to situations where the shrink tests fail because
the replicas are not allocated yet and the shrink operation needs the source shards to be available on the same node.
2018-01-08 10:02:56 +01:00
Tim Brooks 38701fb6ee
Create nio-transport plugin for NioTransport (#27949)
This is related to #27260. This commit moves the NioTransport from
:test:framework to a new nio-transport plugin. Additionally, supporting
tcp decoding classes are moved to this plugin. Generic byte reading and
writing contexts are moved to the nio library.

Additionally, this commit adds a basic MockNioTransport to
:test:framework that is a TcpTransport implementation for testing that
is driven by nio.
2018-01-05 09:41:29 -07:00
Yannick Welsch ca325194d9
Allow shrinking of indices from a previous major (#28076)
Lucene does not allow adding Lucene 6 files to a Lucene 7 index. This commit ensures that we carry over the Lucene version to the newly created Lucene index.

Closes #28061
2018-01-04 16:23:25 +01:00