Commit Graph

1344 Commits

Author SHA1 Message Date
Jason Tedor 0698dd017c
Enabling testing against an external cluster (#30885)
Today when executing REST tests we take full responsibility for cluster
configuration. Yet, there are use cases for brining your own cluster to
the REST tests. This commit is a small first step towards that effort by
skipping creating the cluster if the tests.rest.cluster and test.cluster
system properties are set. In this case, the user takes full
responsibility for configuring the cluster as expected by the REST
tests. This step is by no means meant to be perfect or complete, only a
baby step.
2018-05-26 08:04:53 -04:00
Ryan Ernst 2c7559c575
Packaging: Ensure upgrade_is_oss flag file is always deleted (#30732)
This commit ensures the delete of the upgrade_is_oss indicator for
the packaging tests is always deleted before each run. It works by
moving the check on version which skips the task into the doFirst block,
replacing the onlyIf.

closes #30682
2018-05-23 23:20:13 -07:00
Andy Bristol a1b538122c
[test] java tests for archive packaging (#30734)
Ports the first couple tests for archive distributions from the old bats
project to the new java project that includes windows platforms,
consolidating them into one test method that tests that the
distributions can be extracted and their contents verified. Includes the
zip distributions which were not tested in the bats project.
2018-05-23 10:37:57 -07:00
Nhat Nguyen 1918a30237
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.

Closes #30741
Closes #30577
2018-05-22 13:11:48 -04:00
Alpar Torok 8b7ff69479
Remove fedora 26, add 28 (#30683)
* Remove fedora 26, add 28

Closes #30579

* Update testing docs with new fedora 28 image
2018-05-22 07:22:00 +03:00
Alpar Torok 638383dc44
disable annotation processor for docs (#30610)
* disable annotation processor for docs

Could not find evidence that the log4j annotation processor is used.
The compiler flag enables the Gradle 5.0 behavior

Closes #30476

* Disable annotation processors for all tests

* remove redundant `-proc:none` already handled by required plugins

* Revert unintentional changes
2018-05-21 09:42:58 +03:00
Nhat Nguyen 67d8fc222d
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726)
This snapshot resolves issues related to ShrinkIndexIT.
2018-05-18 18:21:39 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Andy Bristol 890afadb37
[test] packaging: add windows boxes (#30402)
Adds windows server 2012r2 and 2016 vagrant boxes to packaging tests.
They can only be used if IDs for their images are specified, which are
passed to gradle and then to vagrant via env variables. Adds options
to the project property `vagrant.boxes` to choose between linux and
windows boxes.

Bats tests are run only on linux boxes, and portable packaging tests run
on all boxes. Platform tests are only run on linux boxes since they are
not being maintained.

For #26741
2018-05-16 16:42:08 -07:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Alpar Torok 9a5555963b
Add missing dependencies on testClasses (#30527) 2018-05-14 16:06:56 +03:00
Ryan Ernst be8c094e8c
Build: Add mavenPlugin cluster configuration method (#30541)
This commit adds the ability to specify a plugin from maven for a
test cluster to use. Currently, only local projects may be used as
plugins, except when testing bwc, where the coordinates of the project
are used. However, that assumes all projects always keep the same
coordinates, or are even still plugins, which is no longer the case for
x-pack. The full cluster and rolling restart tests are changed to use
this new method when pulling x-pack versions before 6.3.0.
2018-05-11 21:58:18 -07:00
Nhat Nguyen 519768b5d3
Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
This snapshot is to include LUCENE-8298 which allows DocValues updates
to reset a value. This is needed for the Lucene rollback work.
2018-05-10 12:31:45 -04:00
Nik Everett f9dc86836d
Docs: Test examples that recreate lang analyzers (#29535)
We have a pile of documentation describing how to rebuild the built in
language analyzers and, previously, our documentation testing framework
made sure that the examples successfully built *an* analyzer but they
didn't assert that the analyzer built by the documentation matches the
built in anlayzer. Unsuprisingly, some of the examples aren't quite
right.

This adds a mechanism that tests that the analyzers built by the docs.
The mechanism is fairly simple and brutal but it seems to be working:
build a hundred random unicode sequences and send them through the
`_analyze` API with the rebuilt analyzer and then again through the
built in analyzer. Then make sure both APIs return the same results.
Each of these calls to `_anlayze` takes about 20ms on my laptop which
seems fine.
2018-05-09 09:23:10 -04:00
Jason Tedor 92a0b0a00c
Stop forking groovyc (#30471)
This is a follow-up to our previous change to only fork javac if needed
to respect the Java compiler home (via JAVA_HOME). This commit makes the
same change for groovyc: we only fork groovyc if the JDK for Gradle is
not the JDK specified for the compiler (via JAVA_HOME).
2018-05-09 06:17:43 -04:00
Jason Tedor 3a912dfab2
Stop forking javac (#30462)
We started forking javac to avoid GC overhead when running builds. Yet,
we do not seem to have this problem anymore and not forking leads to a
substantial speed improvement. This commit stops forking javac.
2018-05-08 16:13:02 -04:00
Nik Everett 5b8bdcd111
Build: Switch to building javadoc with html5 (#30440)
We accidentally switched back to html4 in #30279 when we removed the
gradle hack that we were using to convert the projects one by one.
2018-05-08 07:38:28 -04:00
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Colin Goodheart-Smithe 09b918545d
Adds Eclipse config for xpack licence headers (#30299)
Uses a filter on the copy task for the eclipse settings files to
replace the token @@LICENSE_HEADER_TEXT@@ with the correct licence
header from the new buildSrc/src/main/resources/license-headers
directory
2018-05-03 08:48:49 +01:00
Ryan Ernst 916bf9d26d
Convert server javadoc to html5 (#30279)
This commit converts the remaining javadocs in :server using html4 to html5.
This was mostly converting `tt` to `{@code}`.
2018-05-02 08:08:54 -07:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
Andy Bristol 65e5868a55
[test] add java packaging test project (#30161)
[test] add java packaging test project

Adds a project for building and running packaging tests written in java
for portability. The vagrant tasks use jars on the packagingTest
configuration, which are built in the same project. No tests are added
yet.

Corresponding changes are not made to :x-pack:qa:vagrant because the
java packaging tests will all be consolidated into one project.

For #26741
2018-04-30 16:35:26 -07:00
Nik Everett 50945051b6
HTML5ify Javadoc for core and test framework (#30234)
`javadoc` will switch from detaulting to html4 to html5 in "a future
release". We should get ahead of it so we're not surprised. Also, HTML5
is the future! Er, the present. Anyway, this follows up from #30220 to
make the Javadoc for two of the four remaining projects HTML5
compatible.
2018-04-30 09:39:50 -04:00
Nik Everett 9c8e015552
Build: Mostly silence warning about html4 javadoc (#30220)
This *mostly* silences `javadoc`'s warning about defaulting to
generating html4 files by enabling generating html5 file for the
projects for which that works. It didn't work in a half dozen projects,
about half of which I've fixed in this PR, entirely by replacing
`<tt>thing</tt>` with `{@code thing}`.

There are a few remaining projects that contain javadoc with invalid
html5. I'll fix those projects in a followup.
2018-04-28 09:50:54 -04:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Nhat Nguyen 16490d7dfa
TEST: Update settings should go through cluster state (#29682)
Today we update index settings directly via IndexService instead of the
cluster state in IndexServiceTests. However, those changes will be lost
if there is a cluster state update. In general, we should update index
settings via client and limit the direct usage in only special tests.

This commit replaces direct usages by the updateSettings api of client.

Closes #24491
2018-04-26 09:28:14 -04:00
Jason Tedor b952a9954f
Set JAVA_HOME before forking setup commands (#29647)
Today when forking setup commands we do not set JAVA_HOME. This means
that we might not use a version of Java compatible with the version of
Java the command is expecting to run on (for example, 5.6 nodes would
expect JDK 8, and this is true even for their setup commands). This
commit sets JAVA_HOME when configuring setup command tasks.
2018-04-25 22:43:00 -04:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Jason Tedor 4fe1d64070 Add POM and license generation for X-Pack artifacts
With the switch to X-Pack as a module, we lost production of POMs for
the JARs that we publish, and did not have a license/notice file in the
zip archives nor the exploded module. This commit ensures that we
generate these POMs, and license/notice files.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 4f282e9e32
Build: Move java home checks to pre-execution phase (#29548)
This commit moves the checks on JAVAX_HOME (where X is the java version
number) existing to the end of gradle's configuration phase, and based
on whether the tasks needing the java home are configured to execute.

relates #29519
2018-04-19 09:51:52 -07:00
olcbean b3e3b80f1b REST high-level client: add support for Indices Update Settings API [take 2] (#29327)
Relates to #27205
2018-04-16 21:39:11 +02:00
Jason Tedor 9125684d86 Minor cleanup in NodeInfo.groovy
This commit is a minor cleanup of a code block in NodeInfo.groovy. We
remove an unused variable, make the formatting of the code consistent,
and cast a property that is typed as an Object to a String to avoid an
annoying IDE warning.
2018-04-14 20:48:48 -04:00
Jason Tedor b883e1217f
Lazy configure build tasks that require older JDKs (#29519)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.
2018-04-14 15:44:43 -04:00
Jason Tedor 85ac541ab3
Make NodeInfo#nodeVersion strongly-typed as Version (#29515)
Today we have a nodeVersion property on the NodeInfo class that we use
to carry around information about a standalone node that we will start
during tests. This property is a String which we usually end up parsing
to a Version anyway to do various checks on it. This can end up
happening a lot during configuration so it would be more efficient and
safer to have this already be strongly-typed as a Version and parsed
from a String only once for each instance of NodeInfo. Therefore, this
commit makes NodeInfo#nodeVersion strongly-typed as a Version.
2018-04-13 16:57:59 -04:00
Jason Tedor 82a753dcc7
Enable license header exclusions (#29379)
There are some scenarios where the license on a source file is one that
is compatible with our projects yet we do not want to add the license to
the list of approved license headers (to keep the number of files with
that compatible license contained). This commit adds the ability to
exclude a file from the license check.
2018-04-13 15:59:33 -04:00
Jason Tedor 27fafa24f5
Use proper Java version for BWC builds (#29493)
Today we have JAVA_HOME for the compiler Java home and RUNTIME_JAVA_HOME
for the test Java home. However, when we compile BWC nodes and run them,
neither of these Java homes might be the version that was suitable for
that BWC node (e.g., 5.6 requires JDK 8 to compile and to run). This
commit adds support for the environment variables JAVA\d+_HOME and uses
the appropriate Java home based on the version of the node being
started. We even do this for reindex-from-old which requires JDK 7 for
these very old nodes. Note that these environment variables are not
required if not running BWC tests, and they are strictly required if
running BWC tests.
2018-04-13 12:41:18 -04:00
Jay Modi e0ec8571ea
Build: introduce keystoreFile for cluster config (#29491)
This commit introduces built in support for adding files to the
keystore when configuring the integration test cluster for a project.

In order to use this support, simply add `keystoreFile` followed by the
secure setting name and the path to the source file inside the
integTestCluster closure for a project. The built in support will
handle the creation of the keystore and the addition of the file to the
keystore.
2018-04-12 07:28:34 -06:00
Christoph Büscher e2dbc77cd0 Remove redundant checkstyle LineLength suppressions 2018-04-06 19:49:33 +02:00
David Roberts d649449d37
[TEST] Put options that enable assertions earlier on command line (#29395)
This change moves the -ea and -esa options that enable assertions for
test nodes before the cluster-specific JVM arguments on the Java command
line.  This opens up the possibility for the cluster-specific JVM
arguments to disable assertions for one particular package or class,
which can be useful in BWC testing where incorrect assertions cannot be
removed from released versions of the product.
2018-04-06 11:34:08 +01:00
Alan Woodward dccd43af47
Upgrade to lucene 7.3.0 (#29387) 2018-04-05 10:34:44 +01:00
Jason Tedor 5cdd831a31
Remove silent batch mode from install plugin (#29359)
Today we have a silent batch mode in the install plugin command when
standard input is closed or there is no tty. It appears that
historically this was useful when running tests where we want to accept
plugin permissions without having to acknowledge them. Now that we have
an explicit batch mode flag, this use-case is removed. The motivation
for removing this now is that there is another place where silent batch
mode arises and that is when a user attempts to install a plugin inside
a Docker container without keeping standard input open and attaching a
tty. In this case, the install plugin command will treat the situation
as a silent batch mode and therefore the user will never have the chance
to acknowledge the additional permissions required by a plugin. This
commit removes this silent batch mode in favor of using the --batch flag
when running tests and requiring the user to take explicit action to
acknowledge the additional permissions (either by leaving standard input
open and attaching a tty, or by passing the --batch flags themselves).

Note that with this change the user will now see a null pointer
exception when they try to install a plugin in a Docker container
without keeping standard input open and attaching a tty. This will be
addressed in an immediate follow-up, but because the implications of
that change are larger, they should be handled separately from this one.
2018-04-03 21:23:01 -04:00
Uwe Schindler 7c6d5cbf1f Build: Fix Java9 MR build (#29312)
Correctly setup classpath/dependencies and fix checkstyle task that was partly broken because delayed setup of Java9 sourcesets. This also cleans packaging of META-INF. It also prepares forbiddenapis 2.6 upgrade

relates #29292
2018-04-03 10:22:12 -07:00
Adrien Grand 3bdfc8f3fb
Upgrade to lucene-7.3.0-snapshot-98a6b3d. (#29298)
Most notable changes include:
 - this release doesn't have the 7.2.1 version constant so I had to create one
 - spatial4j and jts were upgraded
2018-04-03 09:27:14 +02:00
Lee Hinman 6b2167f462
Begin moving XContent to a separate lib/artifact (#29300)
* Begin moving XContent to a separate lib/artifact

This commit moves a large portion of the XContent code from the `server` project
to the `libs/xcontent` project. For the pieces that have been moved, some
helpers have been duplicated to allow them to be decoupled from ES helper
classes. In addition, `Booleans` and `CheckedFunction` have been moved to the
`elasticsearch-core`  project.

This decoupling is a move so that we can eventually make things like the
high-level REST client not rely on the entire ES jar, only the parts it needs.

There are some pieces that are still not decoupled, in particular some of the
XContent tests still remain in the server project, this is because they test a
large portion of the pluggable xcontent pieces through
`XContentElasticsearchException`. They may be decoupled in future work.
Additionally, there may be more piecese that we want to move to the xcontent lib
in the future that are not part of this PR, this is a starting point.

Relates to #28504
2018-04-02 15:58:31 -06:00
Mayya Sharipova e70cd35bda
Revert "REST high-level client: add support for Indices Update Settings API (#28892)" (#29323)
This reverts commit b67b5b1bbd.
2018-03-30 16:26:46 -07:00
olcbean b67b5b1bbd REST high-level client: add support for Indices Update Settings API (#28892)
Relates to #27205
2018-03-30 10:53:29 +02:00
Uwe Schindler 6578d8a2a8 Update to forbiddenapis 2.5 (#29285) 2018-03-28 20:29:02 -07:00
Ryan Ernst 21c985122b
Build: Fix repos.mavenLocal casing (#29289)
The sysprop repos.mavenLocal may be used to add the local .m2 maven
repository for testing snapshots of locally build dependencies.
Unfortunately this has to be checked in two different places (they cannot
be shared, due to buildSrc being built essentially as a separate
project), and the casing of the string sysprop lookups did not align.
This commit fixes BuildPlugin's checking of repos.mavenLocal to use the
correct casing (camelCase, to match the gradle dsl element).
2018-03-28 20:21:30 -07:00
Jason Tedor 1f6a3c1d80
Fix building Javadoc JARs on JDK for client JARs (#29274)
When a module or plugin register that it has a client JAR, we copy
artifacts like the Javadoc and sources JARs as the JARs for the client
as well (with -client added to the name). I previously had to disable
the Javadoc task on JDK 10 due to a bug in bin/javadoc. After JDK 10
went GA without a fix for this bug, I added workaround to fix the
Javadoc task on JDK 10. However, I made a mistake reverting the
previously skipped Javadocs tasks and missed that one that copies the
Javadoc JAR for client JARs. This commit fixes that issue.
2018-03-27 22:58:44 -04:00
Jason Tedor 38fd9998e7
Require JDK 10 to build Elasticsearch (#29174)
This commit bumps the minimum compiler version required to build
Elasticsearch from JDK 9 to JDK 10.
2018-03-27 19:45:13 -04:00
Andy Bristol 7bf9091942
[test] packaging: gradle tasks for groovy tests (#29046)
The vagrant test plugin adds tasks for the groovy packaging tests,
which run after the bats packaging test tasks.Rename the 'bats'
configuration to 'packaging' and remove the option to inherit
archives from this configuration.
2018-03-26 13:43:09 -07:00
Christoph Büscher a9392f6d42 Add file permissions checks to precommit task
This adds a check for source files that have the execute bit set to the
precommit task.
2018-03-26 13:37:55 +02:00
Jason Tedor ad7e8bab6f
Revive build Javadocs on JDK 10 and workaround bug (#29173)
This commit reenables the Javadoc tasks on JDK 10. To reenable these
tasks, we have to workaround a bug in JDK 10 which trips on some deeply
nested anonymous classes that we have in the codebase (and are fine
as-is, this is not a problem with this code). The workaround is to
remove the compiled classes from the classpath. This has been reported
upstream and the workaround was suggested there (see the code comment).
2018-03-21 08:55:12 -04:00
Jason Tedor d766b68f7a
Set Java 9 checkstyle to depend on checkstyle conf (#28383)
We need to configure the Java 9 checkstyle task to depend on the
checkstyle configuration task or the task could run before the
checkstyle conf has been copied leading to runtime failures. We have to
do this after projects have been evaluated because the configuration of
these tasks can occur before the Java 9 source set has been added to a
project.
2018-03-20 09:08:12 -04:00
Ryan Ernst e5825100f0
Build: Fix meta plugin bundled plugin names (#29147)
This commit fixes the directory name bundled plugins are added under
within a meta plugin to be the configured name of the bundled plugin,
instead of the project name.
2018-03-19 19:54:01 -07:00
Ryan Ernst f7a1267e4f
Build: Simplify rest spec hack configuration (#29149)
This commit creates the copyRestSpec task for rest integ tests
immediately on creation of the RestIntegTestTask instead of lazily in
afterEvaluate. This allows other projects to add additional rest specs
to be copied, instead of needing to create another parallel copy task.
2018-03-19 19:46:43 -07:00
Ryan Ernst 1eb1d59de8
Build: Fix meta modules to not install as plugin in tests (#29150)
This commit fixes the meta plugin build to conditionally install the
plugin as a module or plugin, depending on whether it is a
module or plugin.
2018-03-19 16:10:16 -07:00
Nik Everett 762226bee9
Docs: Support triple quotes (#28915)
Adds support for triple quoted strings to the documentation test
generator. Kibana's CONSOLE tool has supported them for a year but we
were unable to use them in Elasticsearch's docs because the process that
converts example snippets into tests couldn't handle this. This change
adds code to convert them into standard JSON so we can pass them to
Elasticsearch.
2018-03-16 12:46:39 -04:00
Jason Tedor 55683d89c4
Clarify how to set compiler and runtime JDKs (#29101)
This commit enhances the error messages reported when JAVA_HOME and
RUNTIME_JAVA_HOME are not correctly set to point towards the minimum
compiler and minimum runtime JDKs that are expected by the builds. The
previous error message would say:

Java 1.9 or above is required to build Elasticsearch

which is confusing if the user does have a JDK 9 installation and is
even the version that they have on their path yet they have JAVA_HOME
pointing to another JDK installation. The error message reported after
this change is:

the environment variable JAVA_HOME must be set to a JDK installation directory for Java 1.9 but is [/usr/java/jdk-8] corresponding to [1.8]
2018-03-15 20:58:37 -04:00
Jason Tedor 5904d936fa
Copy Lucene IOUtils (#29012)
As we have factored Elasticsearch into smaller libraries, we have ended
up in a situation that some of the dependencies of Elasticsearch are not
available to code that depends on these smaller libraries but not server
Elasticsearch. This is a good thing, this was one of the goals of
separating Elasticsearch into smaller libraries, to shed some of the
dependencies from other components of the system. However, this now
means that simple utility methods from Lucene that we rely on are no
longer available everywhere. This commit copies IOUtils (with some small
formatting changes for our codebase) into the fold so that other
components of the system can rely on these methods where they no longer
depend on Lucene.
2018-03-13 12:49:33 -04:00
Jason Tedor 6331bcaf76
Create keystore on package install (#28928)
This commit removes the ability to specify that a plugin requires the
keystore and instead creates the keystore on package installation or
when Elasticsearch is started for the first time. The reason that we opt
to create the keystore on package installation is to ensure that the
keystore has the correct permissions (the package installation scripts
run as root as opposed to Elasticsearch running as the elasticsearch
user) and to enable removing the keystore on package removal if the
keystore is not modified.
2018-03-12 12:48:00 -04:00
Yannick Welsch be7f5dde24
Disallow logger methods with Object parameter (#28969)
Log4j2 provides a wide range of logging methods. Our code typically only uses a subset of them. In particular, uses of the methods trace|debug|info|warn|error|fatal(Object) or trace|debug|info|warn|error|fatal(Object, Throwable) have all been wrong, leading to not properly logging the provided message. To prevent these issues in the future, the corresponding Logger methods have been blacklisted.
2018-03-12 03:05:24 -07:00
Lee Hinman 46a79127ed
Remove FastStringReader in favor of vanilla StringReader (#28944)
This allows us to remove another dependency in the decoupling of the XContent
code. Rather than move this class over or decouple it, it can simply be removed.

Relates tangentially to #28504
2018-03-08 17:17:36 -07:00
Ryan Ernst d972db7eb7
Build: Fix ability to ignore when no tests are run (#28930)
Running any randomized testing task within Elasticsearch currently fails
if a project has zero tests. This was supposed to be overrideable, but
it was always set to 'fail', and the system property to override was
passed down to the test runner, but never read there. This commit
changes the value of the ifNoTests setting to randomized runner to be
read from system properties and continue to default to 'fail'.
2018-03-08 15:21:45 -08:00
Ryan Ernst ea5b6c4fc1
Build: Fix test logger NPE when no tests are run (#28929)
This commit fixes the test progress logging to not produce an NPE when
there are no tests run. The onQuit method is always called, but onStart
would not be called if no tests match the test patterns.
2018-03-07 15:51:49 -08:00
Ivan Brusic 6fb8319ffc Update outdated java version reference (#28870)
It is only a comment, but can confuse those reading the code

Used 6.0 as an arbitrary elasticsearch.version value since it is version that required Java 8
2018-03-01 14:17:24 -08:00
Jason Tedor 57225652b3 Revert "Destroy test VMs on clean (#28818)"
This reverts commit f66234bdd5.
2018-02-25 21:49:36 -05:00
Jason Tedor f66234bdd5
Destroy test VMs on clean (#28818)
This commit adds a dependency on the destroy task to the clean task so
that the VMs are destroyed when the clean task is executed.
2018-02-25 17:02:38 -05:00
Jason Tedor c9ed49da10 Specify working directory for destroy task
This commit specifies that the working directory of the destroy task for
destroying test VMs is the root of the build. This is necessary in case
the build was run from a sub-directory, the Vagrant command would then
not be able to locate the Vagrantfile for the VMs in question.
2018-02-25 11:56:54 -08:00
Jason Tedor 22736848b0
Always destroy Vagrant boxes before tests (#28813)
Today we do not destroy Vagrant boxes before tests. This is because
constantly reprovisioning these boxes is time-consuming. Yet, not
destroying these boxes can lead to state being left around that impacts
subsequent test runs. To address this, we now always destroy these boxes
before tests and provide a flag to set if this is not desired while
iterating locally.
2018-02-25 09:15:56 -05:00
Ryan Ernst 88e44dd133 Fix rest test to not specify zip distribution
This was lefter from
https://github.com/elastic/elasticsearch/pull/28805, and effectively did
not allow changing the distribution from zip (exception in esplugin
which sets it later).
2018-02-24 23:12:13 -08:00
Ryan Ernst 3a64e6d121
Test: Remove specifying zip distribution in qa tests (#28805)
Applying the rest test gradle plugin already uses the zip distribution
by default, so specifying it explicitly is not necessary. These are
leftovers from before zip was the default for rest tests.
2018-02-23 13:45:38 -08:00
Daniel Mitterdorfer d0831fdd3e
Reduce max javac memory to 512mb (#28783)
With this commit we reduce the maximum amount of memory that the javac
compiler can use from 1g to 512mb. While the build would succeed even
with 256mb, it influences compile time slightly negatively.

We have measured that the runtime overhead stays tolerable by running
the following command five times under repeatable conditions (i.e. we
execute `./gradlew clean`, then drop the caches and TRIM the disk):

```
./gradlew compileGroovy compileJava compileJava9Java\
 compileTestGroovy compileTestJava compileGroovy
```

The results in seconds (as reported by Gradle) are:

* 1gb: avg: 253s, min: 252s, max: 256s
* 512mb: avg: 257s, min: 256s, max: 259s
2018-02-22 16:36:01 +01:00
Ryan Ernst 014e90d903
Build: Consolidate archives and packages configuration (#28760)
This commit moves the distribution specific tasks into the respective
archives and packages builds. The collocation of common and distribution
specific tasks make it much easier to reason about what is expected in a
particular distribution.
2018-02-21 17:46:40 -08:00
Jason Tedor 86e5e38b41 Revert "Add startup logging for standalone tests"
This reverts commit 9e23b7ed72.
2018-02-20 20:00:09 -05:00
Ryan Ernst 53c38cc8fe
Build: Group archive and package distribution projects (#28673)
This commit adds intermediate gradle projects for archive based
distributions (zip, tar) and package based distributions (rpm, deb). The
grouping allows the common distribution build file to be considerably
shorter and clearly separated from the common zip/tar and rpm/deb
configuration.
2018-02-13 22:49:53 -08:00
Ryan Ernst 9a5199fae3
Docs: Remove references to elasticsearch directory in plugins (#28647)
This directory was removed from plugins in #28589, but docs still
referenced it. This commit cleans up the plugin author docs to no longer
refer to it.
2018-02-13 09:15:57 -08:00
Jason Tedor 9e23b7ed72
Add startup logging for standalone tests
We need to investigate why startup is taking so long in CI for
standalone tests. This commit adds logging for bootstrap and network
code that is executed before the node starts initializing in case this
is the source of the trouble.

Relates #28659
2018-02-13 10:14:42 -05:00
Jason Tedor fea1b20529 Revert "Increase the REST test timeout. (#28640)"
This reverts commit db87294a00.
2018-02-13 10:13:32 -05:00
Adrien Grand db87294a00
Increase the REST test timeout. (#28640)
By default we wait for 30 seconds for nodes to start. Looking at CI,
even for passing builds, the startup time is frequently over 20 seconds
(grep for `#wait (Thread[Task worker for ':',5,main]) completed` in any
console log), which is close to the threshold. This commit sets a
default timeout of 120 seconds instead, so that REST tests are less
likely to hit a timeout.
2018-02-13 11:05:26 +01:00
Ryan Ernst 65f1dd424e
Plugins: Remove intermediate "elasticsearch" directory within plugin zips (#28589)
This commit removes the extra layer of all plugin files existing under
"elasticsearch" within plugin zips. This simplifies building plugin zips
and removes the need for special logic of modules vs plugins.
2018-02-12 14:27:30 -08:00
Michael Basnight 4e0c1463d5
Fix build.snapshot bug in version collection (#28641)
The build.snapshot was mistakenly passed in to every snapshot version,
so when release tests were run, these versions were mistaken as released
entities and could not be found in maven, because they do not
exist. This fix removes that bug in logic, and always makes them proper
snapshots. This has a benefit of cleaning up the VersionUtilsTests
because they no longer rely on different sets of versions to check
against, which was also a bug.
2018-02-12 14:56:07 -06:00
David Turner c15e5d480c
Improve logging in ClusterFormationTasks#configureWaitTask (#28637)
We suspect a build failure might be due to a startup timeout, but there is 
insufficient information in the logs. This change enhances the information
available so we will be better-informed next time.
2018-02-12 18:46:25 +01:00
Michael Basnight e0bea70070
Generalize BWC logic (#28505)
Generalizing BWC building so that there is less code to modify for a release. This ensures we do not
need to think about what major or minor version is in the gradle code. It follows the general rules of the
elastic release structure. For more information on the rules, see the VersionCollection's javadoc.

This also removes the additional bwc snapshots that will never be released, such as 6.0.2, which were
being built and tested against every time we ran bwc tests.

Additionally, it creates 4 new projects that correspond to the different types of snapshots that may exist
for a given version. Its possible to now run those individual tasks to work out bwc logic whereas
previously it was impossible and the entire suite of bwc tests had to be run to work out any logic
changes in the build tools' bwc project. Please note that if the project does not make sense for the 
version that is current, that an error will be thrown from that individual project if an attempt is made to 
run it.

This should allow for automating the version bumps as well, since it removes all the hardcoded version
logic from the configs.
2018-02-09 14:55:10 -06:00
Ryan Ernst 20c37efea2
Build: Replace provided configuration with compileOnly (#28564)
When elasticsearch was originally moved to gradle, the "provided" equivalent in maven had to be done through a plugin. Since then, gradle added the "compileOnly" configuration. This commit removes the provided plugin and replaces all uses with compileOnly.
2018-02-09 11:30:24 -08:00
Ryan Ernst b637a5b028
Build: Improve test status logging (#28563)
Tests use the (internal) gradle progress logger. Since aroudn gradle
4.0, loggers can have children loggers which each get their own line of
updating output. This commit improves the test status output to give
each jvm its own status line, as well as lines for suite and test
counts. It also fixes an issue where the progress logger was not marked
as completed, which caused its last output to never be cleared.
2018-02-09 11:29:07 -08:00
Ryan Ernst a139c42964
Build: Fix test task to explicitly depend on testClasses task (#28490)
Gradle 4.5 now hides immutable task dependencies. We previously copied
the existing dependencies from the builtin test task to the
randomizedtesting task. This commit adds testClasses as an extra
dependency of the randomizedtesting task, to ensure the classes are
built.
2018-02-02 10:57:01 -08:00
David Turner ab8f5ea54c
Forbid trappy methods from java.time (#28476)
ava.time has the functionality needed to deal with timezones with varying 
offsets correctly, but it also has a bunch of methods that silently let you
forget about the hard cases, which raises the risk that we'll quietly do the
wrong thing at some point in the future.

This change adds the trappy methods to the list of forbidden methods to try and
help stop this from happening.

It also fixes the only use of these methods in the codebase so far:
IngestDocument#deepCopy() used ZonedDateTime.of() which may alter the offset of
the given time in cases where the offset is ambiguous.
2018-02-02 18:24:02 +00:00
Ryan Ernst 139deb535a
Test: Use --illegal-access=warn for tests (#28437)
This commit conditionally adds the --illegal-access=warn flag when tests
are run with java 9. Currently, testing on java 9 triggers a warning
about illegal access from mockito. While that should be fixed (by
updating to a newer mockito base for securemock), the stderr warning we
get is only the first one. Thankfully that is the only one, but this
change will enable finding all such illegal accesses in the future.
2018-01-30 17:40:18 -08:00
Simon Willnauer 3bf8554114
Remove tribe node support (#28443)
Tribe node has been superseeded by Cross-Cluster-Search. This change
removes the tribe node support entirely.
2018-01-30 20:40:19 +01:00
Jay Modi 37c9ac25de
Build: allow the amount of time we wait for a node to be configured (#28403)
This commit allows for configuration of the amount of time we wait for a node to startup. This is needed as some QA tests with plugins and tribe timeout when starting the tribe
node. Even regular node startup time with plugins is starting to approach the limit of 30 seconds.

Additionally, this commit provides better feedback when a wait has failed. The code checks to
ensure all expected files exist for each node; if they do not then we consider the wait task as
having failed. Prior to this change, when there was one or more missing file the build would
continue and attempt to execute the wait condition that typically makes an HTTP request to the
cluster. The output of this type of failure does include which files exist and which do not but
this change makes it clearer that the actual HTTP call did not time out, but the failure was before
the call was even made.
2018-01-29 12:45:15 -07:00
Nik Everett 711d79024b
Build: Print warn logs from cluster on failure (#28401)
Prints the warning level logs from the integration testing cluster if
there is a failure. We'd attempted to print those logs but I believe our
regex was missing a space.
2018-01-26 14:52:03 -05:00
Ryan Ernst 3a43bb1ba9 Build: Add pom generation to meta plugins (#28321)
This commit adds pom generation to meta plugins by using the same hacks
that PluginBuildPlugin already uses to get around "pom" type poms (ie
zip files).
2018-01-21 18:37:31 -05:00
Ryan Ernst 7a82bb94b3
Build: Fix meta plugin usage in integ test clusters (#28307)
This commit fixes places handling plugin projects in cluster formation
to work with both esplugin and es_meta_plugin.
2018-01-18 21:13:39 -08:00
Ryan Ernst 19a2b01e43
Build: Omit dependency licenses check for elasticsearch deps (#28304)
Sometimes modules/plugins depend on locally built elasticsearch jars.
This means not only that the jar is constantly changing (so no need for
a sha check), but also that the license falls under the Elasticsearch
license, and there is no need to keep another copy. This commit updates
the dependencies checked by dependencyLicenses to exclude those that are
built by elasticsearch.
2018-01-18 14:15:44 -08:00
Ryan Ernst e442a34acc
Build: Fix meta plugin integ test installation (#28286)
Integ test clusters should use the plugin method of ClusterConfiguration
to install plugins. Without it, meta plugins install based on the name
of the project directory, rather than the actual configured plugin name.
This commit fixes that, and also corrects the distribution used to be
the default integ-test-zip, to match that of PluginBuildPlugin. This
ensures plugins are tested in isolation by default.
2018-01-18 11:14:56 -08:00
Jason Tedor 4214a718ec
Fork Groovy compiler onto compile Java home
We use the --release flag which is only available starting in JDK
9. Since Gradle could be running on JDK 8 without forking the compiler,
compilation will occur with the Java home of Gradle. This commit adds a
fork flag to the compiler Java home so that we use the right compiler.

Relates #28300
2018-01-18 12:08:32 -05:00
Ryan Ernst 06f931fcc4
Build: Add run task to meta plugins (#28283)
This commit adds a run task to projects using the meta plugin plugin. It
also makes the project installable via ClusterConfiguration.plugin.
2018-01-18 01:02:27 -08:00
Ryan Ernst ac1c509844
Build: Fix subdirectories in meta plugins to be copied correctly (#28282)
This commit fixes the copying of files from bundled plugin zips.
Previously all files within each zip would be flattened under the
bundled plugin name, and the original directories would exist at the top
level of the plugin.
2018-01-17 21:27:59 -08:00
Ryan Ernst c15ae7eb20
Build: Add release flag to groovy compilation (#28277)
The build-tools project (ie buildSrc) has groovy code. The -source
option was set to java 8, but this is not correct on java 9, and
actually causes a warning about not setting boot classpath. This commit
adds the --release option to groovy compilation, just like is done for
java compilation.
2018-01-17 20:03:11 -08:00
Ryan Ernst cefea1a7c9
Build: Add gradle plugin for configuring meta plugin (#28276)
This commit adds a gradle plugin to ease development of meta plugins.
Applying the plugin will generated the meta plugin properties based on
the es_meta_plugin configuration object, which includes name and
description. The plugins to include within the meta plugin are
configured through the `plugins` list. An integ test task is also
automatically added.
2018-01-17 19:47:37 -08:00
Jason Tedor 045dd4ad48
Introduce multi-release JAR
This commit introduces the ability for the core Elasticsearch JAR to be
a multi-release JAR containing code that is compiled for JDK 8 and code
that is compiled for JDK 9. At runtime, a JDK 8 JVM will ignore the JDK
9 compiled classfiles, and a JDK 9 JVM will use the JDK 9 compiled
classfiles instead of the JDK 8 compiled classfiles. With this work, we
utilize the new JDK 9 API for obtaining the PID of the running JVM,
instead of relying on a hack.

For now, we want to keep IDEs on JDK 8 so when the build is in an IDE we
ignore the JDK 9 source set (as otherwise the IDE would give compilation
errors). However, with this change, running Gradle from the command-line
now requires JAVA_HOME and JAVA_9_HOME to be set. This will require
follow-up work in our CI infrastructure and our release builds to
accommodate this change.

Relates #28051
2018-01-16 15:10:29 -05:00
Jason Tedor 0a79555a12
Require JDK 9 for compilation (#28071)
This commit modifies the build to require JDK 9 for
compilation. Henceforth, we will compile with a JDK 9 compiler targeting
JDK 8 as the class file format. Optionally, RUNTIME_JAVA_HOME can be set
as the runtime JDK used for running tests. To enable this change, we
separate the meaning of the compiler Java home versus the runtime Java
home. If the runtime Java home is not set (via RUNTIME_JAVA_HOME) then
we fallback to using JAVA_HOME as the runtime Java home. This enables:
 - developers only have to set one Java home (JAVA_HOME)
 - developers can set an optional Java home (RUNTIME_JAVA_HOME) to test
   on the minimum supported runtime
 - we can test compiling with JDK 9 running on JDK 8 and compiling with
   JDK 9 running on JDK 9 in CI
2018-01-16 13:45:13 -05:00
Jim Ferenczi be012b1326
upgrade to lucene 7.2.1 (#28218) 2018-01-15 16:47:46 +01:00
Tim Brooks 99f88f15c5
Rename core module to server (#28180)
This is related to #27933. It renames the core module to server. This is
the first step towards introducing an elasticsearch-core jar.
2018-01-11 11:30:43 -07:00
Martijn van Groningen 7d0eb3292b
upgraded jna from 4.4.0-1 to 4.5.1 (#28183)
Closes #28172
2018-01-11 17:31:10 +01:00
Ryan Ernst 8ceae2a380
Build: Allow deps to be a direct jar (#28096)
Currently the code which disable transitive dependencies assumes all
deps are a "module dependency" in gradle. But a jar file dep is not.
This commit relaxes the closure signature to allow any dependency and
only enforce the transitive disabling for module dependencies.
2018-01-09 12:44:09 -08:00
Jim Ferenczi 36729d1c46
Add the ability to bundle multiple plugins into a meta plugin (#28022)
This commit adds the ability to package multiple plugins in a single zip.
The zip file for a meta plugin must contains the following structure:

|____elasticsearch/
| |____   <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory)
| |____   <plugin2>  <-- The plugin files for plugin2
| |____   meta-plugin-descriptor.properties <-- example contents below
The meta plugin properties descriptor is mandatory and must contain the following properties:

description: simple summary of the meta plugin.
name: the meta plugin name
The installation process installs each plugin in a sub-folder inside the meta plugin directory.
The example above would create the following structure in the plugins directory:

|_____ plugins
| |____   <name_of_the_meta_plugin>
| | |____   meta-plugin-descriptor.properties
| | |____   <plugin1>
| | |____   <plugin2>
If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory.

|_____ config
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>

|_____ bin
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>
The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin
cannot have the same name than another plugin (or a sub-plugin inside another meta plugin).

It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed.

Closes #27316
2018-01-09 18:28:43 +01:00
Maxime Gréau c46222ea15
Fix Licenses values for CDDL and Custom URL (#27999)
* Fix license SPDX identifiers (CDDL)
* Fix license type for Custom URL:
  * If the license is identified but not listed as an SPDX identifier, the character `;` is used after the identifier to set the license URL.
2018-01-08 19:36:33 +01:00
Jason Tedor eaa636d4bb Clarify reproduce info on Windows
This commit correct the test failure reproduction line on Windows.

Relates #28104
2018-01-06 22:49:14 -05:00
Jason Tedor d712f581ca
Fix reproduction info to point to Gradle wrapper
With the Gradle wrapper in place, we should point the reproduction info
to specify using the Gradle wrapper too.

Relates #28104
2018-01-06 08:47:23 -05:00
Jason Tedor f999d639be
Update platforms tests to use Gradle wrapper
Gradle will no longer be needed in the test VMs as the Gradle wrapper
can be used to run the platform tests. This commit updates the platform
tests to use the Gradle wrapper, and removes installing Gradle in the VM
which will speed up VM provisioning.

Relates #28105
2018-01-05 17:25:10 -05:00
Jason Tedor 4656c55639
Ignore GIT_COMMIT when calculating commit hash
When finding the commit hash for the build to place in the JAR manifest
(which is used to identity the build), the scm-info plugin assumes that
GIT_COMMIT is the commit for this build. That assumption is wrong, this
build could be a sub-build of another build that GIT_COMMIT belongs
to. If GIT_COMMIT is set, we ignore the commit hash calculated by
scm-info and calculate the hash ourselves.

Relates #28082
2018-01-04 14:58:34 -05:00
Ryan Ernst d36ec18029
Plugins: Add plugin extension capabilities (#27881)
This commit adds the infrastructure to plugin building and loading to
allow one plugin to extend another. That is, one plugin may extend
another by the "parent" plugin allowing itself to be extended through
java SPI. When all plugins extending a plugin are finished loading, the
"parent" plugin has a callback (through the ExtensiblePlugin interface)
allowing it to reload SPI.

This commit also adds an example plugin which uses as-yet implemented
extensibility (adding to the painless whitelist).
2018-01-03 11:12:43 -08:00
Daniel Mitterdorfer bcd72775ff Upgrade to JMH 1.19 2017-12-29 10:56:47 +01:00
Maxime Gréau 771defb97c
Build: Add 3rd party dependencies report generation (#27727)
* Adds task dependenciesInfo to BuildPlugin to generate a CSV file with dependencies information (name,version,url,license)
* Adds `ConcatFilesTask.groovy` to concatenates multiple files into one
* Adds task `:distribution:generateDependenciesReport` to concatenate `dependencies.csv` files into a single file (`es-dependencies.csv` by default)

 # Examples:
      $ gradle dependenciesInfo :distribution:generateDependenciesReport

 ## Use `csv` system property to customize the output file path
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dcsv=/tmp/elasticsearch-dependencies.csv

 ## When branch is not master, use `build.branch` system property to generate correct licenses URLs
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dbuild.branch=6.x -Dcsv=/tmp/elasticsearch-dependencies.csv
2017-12-26 10:51:47 +01:00
Jason Tedor 0423155fc6
Also skip Javadoc task for client JARs on JDK 10
We disabled the Javadoc task on JDK 10 due to an apparent bug in Javadoc
generation on JDK 10. However, the client JAR task sets up its own
Javadoc task for client JARs (because it merely copies the non-client
Javadoc JAR). This commit diables that task too, since the Javadocs for
the non-client JAR will not exist.

Relates #27962
2017-12-22 07:34:00 -05:00
Jason Tedor d4f914be2a
Disable Javadocs on JDK 10
There appears to be a bug in JDK 10 for generating Javadocs with some
nested anonymous classes. This commit disables these on JDK 10 until the
upstream issue is resolved.

Relates #27952
2017-12-21 17:26:20 -05:00
Jason Tedor 8bd7a19d61
Use full profile on JDK 10 builds
JDK 10 has gone fully-modular. This means that:
 - when targeting JDK 8 with a JDK 10 compiler, we have to use the full
   profile
 - when targeting JDK 10 with a JDK 10 compiler, we have to use
   -add-modules java.base

Today we only target JDK 8 (our minimum compatibility version) so we
need to change the compiler flags conditional on using a JDK 10
compiler. This commit does that.

Relates #27884
2017-12-20 16:22:21 -05:00
Jason Tedor c8371fa0b4
Require Gradle 4.3
This commit sets the minimum Gradle version to version 4.3. This the
minimum Gradle version that understands JDK 10 in code.

Relates #27885
2017-12-20 16:14:19 -05:00
Adrien Grand 77711508b0
Upgrade to Lucene 7.2.0. (#27910) 2017-12-20 14:17:40 +01:00
Jason Tedor ca70ca6698
Fix BWC release tests
When running the release tests, we set build.snapshot to false and this
causes all version numbers to not have "-SNAPSHOT". This is true even
for the tips of the branches (e.g., currently 5.6.6 on the 5.6
branch). Yet, if we do not set snapshot to false, then we would still be
trying to find artifacts with "-SNAPSHOT" appended which would not have
been build since build.snapshot is false. To fix this, we have to push
build.snapshot into the version logic.

Relates #27778
2017-12-12 10:55:24 -05:00
Jason Tedor cd474df972
Remove RPM and Debian integration tests
We have tests that manually unpackage the RPM and Debian package
distributions and start a cluster manually (not from the service) and
run a basic suite of integration tests against them. This is problematic
because it is not how the packages are intended to be used (instead,
they are intended to be installed using the package installation tools,
and started as services) and so violates assumptions that we make about
directory paths. This commit removes these integration tests, instead
relying on the packaging tests to ensure the packages are not
broken. Additionally, we add a sanity check that the package
distributions can be unpackaged. Finally, with this change we can remove
some leniency from elasticsearch-env about checking for the existence of
the environment file which the leniency was there solely for these
integration tests.

Relates #27725
2017-12-11 15:40:10 -05:00
Jason Tedor ec5e540174
Fix routing with leading or trailing whitespace
The problem here is that splitting was using a method that intentionally
trims whitespace (the method is really meant to be used for splitting
parameters where whitespace should be trimmed like list
settings). However, for routing values whitespace should not be trimmed
because we allow routing with leading and trailing spaces. This commit
switches the parsing of these routing values to a method that does not
trim whitespace.

Relates #27712
2017-12-08 11:23:24 -05:00
Adrien Grand 6323bb0d97
Upgrade to lucene-7.2.0-snapshot-8c94404. (#27619)
This new snapshot mostly brings a change to TopFieldCollector which can now
early terminate collection when trackTotalHits is `false`.

As a follow-up, we should replace our usage of
`EarlyTerminatingSortingCollector` with this new option.
2017-12-04 09:40:08 +01:00
Yannick Welsch 2738c783e5
Fix Gradle >=4.2 compatibility (#27591)
Gradle 4.2 introduced a feature for safer handling of stale output files. Unfortunately, due to the way some of our tasks are written, this broke execution of our REST tests tasks. The reason for this is that the extract task (which extracts the ES distribution) would clean its output directory, and thereby also remove the empty cwd subdirectory which was created by the clean task. The reason why Gradle would remove the directory is that the earlier running clean task would programmatically create the empty cwd directory, but not make Gradle aware of this fact, which would result in Gradle believing that it could just safely clean the directory.

This commit explicitly defines a task to create the cwd subdirectory, and marks the directory as output of the task, so that the subsequent extract task won't eagerly clean it. It thereby restores full compatibility of the build with Gradle 4.2 and above.

This commit also removes the @Input annotation on the waitCondition closure of AntFixture, which conflicted with the extended input/output validation of Gradle 4.3.
2017-11-30 10:03:16 +01:00
Adrien Grand 996990ad1f
Upgrade to lucene-7.2.0-snapshot-8c94404. (#27496)
The main highlight of this new snapshot is that it introduces the opportunity
for queries to opt out of caching. In case a query opts out of caching, not only
will it never be cached, but also no compound query that wraps it will be
cached.
2017-11-28 14:52:42 +01:00
David Turner e8c2e5d6dd
CURRENT should not be a -SNAPSHOT version if build.snapshot is false (#27512) 2017-11-24 11:19:41 +00:00
David Turner 8916935639
Remove unused method (#27508) 2017-11-24 08:09:31 +00:00
David Turner 89ba8996c6 Consolidate version numbering semantics (#27397)
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
2017-11-23 20:21:53 +00:00
Martijn van Groningen 9fbbc46ba4
Added 6.2 version 2017-11-22 17:37:19 +01:00
Dimitrios Liappis d3e3bc8656
Tests: Add Fedora-27 to packaging tests
Replace Fedora-25 with Fedora-27 and fix old references in
`TESTING.asciidoc`.

Relates #27434
2017-11-20 16:59:00 +02:00
Yannick Welsch 7b71cdd34e
Fix Gradle 4.3.1 compatibility for logging (#27382)
The build currently does not work with Gradle 4.3.1 as the Gradle team stopped publishing the gradle-logging dependency to jcenter, starting with 4.3.1 (not sure why). There are two options:

- Add the repository managed by Gradle team (https://repo.gradle.org/gradle/libs-releases-local) to our build
- Use an older version (4.3) of the dependency when running with Gradle 4.3.1.

Not to depend on another external repo, I've chosen solution 2. Note that this solution could break on future versions, but as this is a compileOnly dependency, and the interface we use has been super stable since forever, I don't envision this to be an issue (and easily detected by a breaking build).
2017-11-15 14:50:13 +01:00
Yannick Welsch a45f2bd225
Set minimum_master_nodes to all nodes for REST tests (#27344)
PR #26911 set minimum_master_nodes from number_of_nodes to (number_of_nodes / 2) + 1 in our REST tests. This has led to test failures (see #27233) as the REST tests only configure the first node in its unicast.hosts pinging list (see explanation here: #27233 (comment)). Until we have a proper fix for this, I'm reverting the change in #26911.
2017-11-10 15:07:49 +01:00
Jason Tedor 58a28dacbd
Remove colons from task and configuration names
Gradle 5.0 will remove support for colons in configuration and task
names. This commit fixes this for our build by removing all current uses
of colons in configuration and task names.

Relates #27305
2017-11-08 15:22:31 -05:00
Tanguy Leroux 43e7a4a349
Upgrade to Jackson 2.8.10 (#27230)
While it's not possible to upgrade the Jackson dependencies 
to their latest versions yet (see #27032 (comment) for more) 
it's still possible to upgrade to the latest 2.8.x version.
2017-11-06 10:20:05 +01:00
David Pilato 0635778c90
Add support for Gradle 4.3 (#27249)
Closes #26840
Related to #27087
2017-11-03 14:36:59 +01:00
Colin Goodheart-Smithe c1b8140c83
Upgrade to Lucene 7.1 (#27225) 2017-11-02 13:25:33 +00:00
Jason Tedor d5efc30968 Blacklist Gradle 4.2 and above
An upstream Gradle change has broken us starting on version 4.2. This
commit blacklists these versions until we can either find a workaround,
or the upstream issue is addressed.

Relates #27087
2017-10-24 09:11:30 -04:00
Tanguy Leroux 463e7e6fa3 Revert "Upgrade to Jackson 2.9.2 (#27032)"
This reverts commit 0b9acc5ace.
2017-10-20 08:25:41 +02:00
Tanguy Leroux 0b9acc5ace Upgrade to Jackson 2.9.2 (#27032)
Upgrade to Jackson 2.9.2 and also use a boolean `closed` flag to
indicate that a FastStringReader instance is closed, so that length
is still correctly reported after the reader is closed.
2017-10-19 15:15:02 +02:00
Yannick Welsch a4436195f8 Set minimum_master_nodes on rolling-upgrade test (#26911)
The rolling-upgrade test was only writing the "minimum_master_nodes" setting to the configuration file of the old nodes, but not the upgraded ones.

Also changes the value of "minimum_master_nodes" from "number_of_nodes" to "(number_of_nodes / 2) + 1".
2017-10-09 10:45:03 +02:00
Martijn van Groningen dca787ed8a
upgrade to Lucene 7.1.0 snapshot version 2017-10-05 09:06:56 +02:00
Uwe Schindler dd520cf0b6 Upgrade to forbiddenapis 2.4.1 (#26874)
Update provides:
 * Adds support for Gradle 4.0 (without deprecation warning)
 * Full Java 9 support through ASM 6.0

Version 2.4 was buggy (it broke Gradle dependencies on SourceSets), but this version fixes it.
2017-10-04 01:20:23 -06:00
Dimitrios Liappis 5079afd0bb Tests: Add Fedora-26 to packaging tests
Add Fedora-26 to packaging tests. Don't retire Fedora-25 yet, as it
will be EOL, one month after Fedora-27 is out.

Relates #26726
2017-10-03 12:01:20 +03:00
Jason Tedor 2e63a13c0a Upgrade to Log4j 2.9.1
This commit upgrades the Log4j dependency, picking up a fix for an issue
with handling stack traces on JDK 9.

Relates #26750
2017-09-22 11:57:06 -04:00
Jason Tedor e0db89bc35 Upgrade to Lucene 7.0.0
This commit upgrades to the GA release of Luence 7!

Relates #26744
2017-09-21 19:19:33 -04:00
David Pilato b6c6effa2a Move all repository-azure classes under one single package (#26624)
As we did for S3, we can collapse all packages within one single `org.elasticsearch.repositories.azure` package name.

Follow up for https://github.com/elastic/elasticsearch/pull/23518#issuecomment-328903585
2017-09-14 11:56:02 +02:00
Adrien Grand 93da7720ff Move non-core mappers to a module. (#26549)
Today we have all non-plugin mappers in core. I'd like to start moving those
that neither map to json datatypes nor are very frequently used like `date` or
`ip` to a module.

This commit creates a new module called `mappers-extra` and moves the
`scaled_float` and `token_count` mappers to it. I'd like to eventually move
`range` fields there but it's more complicated due to their intimate
relationship with range queries.

Relates #10368
2017-09-13 17:58:53 +02:00
Jason Tedor 670849894f Ensure module is bundled before installing in tests
This commit adds a dependency to the install module task on the task
that builds the module. This is needed for standalone integration
tests that require other modules to be installed. Without this, we do
not have a guarantee that the module is bundled.
2017-09-13 08:17:16 -04:00
Ryan Ernst 95d40758c2 Build: Remove norelease from forbidden patterns (#26592)
closes #26547
2017-09-11 15:55:02 -07:00
Ryan Ernst 8ba4ff3be0 Build: Move javadoc linking to root build.gradle (#26529)
Javadoc linking between projects currently relies on
projectSubstitutions. However, that is an extension variable that is not
part of BuildPlugin. This commit moves the javadoc linking into the root
build.gradle, alongside where projectSubstitutions are defined.
2017-09-11 15:43:34 -07:00
Jason Tedor e16cb0e4db Allow multiple digits in Vagrant 2.x minor versions
This commit allows the minor version in Vagrant 2.x versions to have two
digits when we perform the Vagrant version check.
2017-09-09 15:28:26 -04:00
Kevin Risden f476c10825 Support Vagrant 2.x
This commit adds support for Vagrant 2.x by allowing these versions to
pass the Vagrant version check.

Relates #26563
2017-09-09 15:27:21 -04:00
Jason Tedor 19a2156d18 Skip some logging tests on JDK 9
There is a bug in Log4j on JDK 9 for walking the stack to find where a
log line is coming from. This bug is impacting some of our testing, so
this commit marks these tests as skippable only on JDK 9 until the bug
is fixed upstream.

Relates #26467
2017-09-01 12:38:22 -04:00
Alexander Reelsen 563ff3cbce Tests: Use script.max_compilations_rate for BWC tests
As the setting is now backported to 6.0, the cluster
formation task can use only the new setting, no need
to use the old one anymore.
2017-09-01 14:43:00 +02:00
Alexander Reelsen 80d0a32f8e ScriptService: Replace max compilation per minute setting with max compilation rate (#26399)
The current script service has a script compilation limit for a one
minute window. This is set to a small default value of 15. Instead of
increasing that default value, this commit introduces a new setting 
that allows to configure a rate per time unit, so that the script service can deal with bursts better.

The new setting is named `script.max_compilations_rate`,
requires a nonnegative number and a positive time value.

The default is `75/5m`, which is equivalent to the existing 15 per minute.
2017-09-01 10:15:27 +02:00
Jason Tedor 697bc266ce Upgrade to Log4j 2.9.0
This commit upgrades the Log4j dependency from version 2.8.2 to version
2.9.0.

Relates #26450
2017-08-31 09:54:35 -04:00
Adrien Grand 78681bc9e5 Upgrade to lucene-7.0.0-snapshot-d94a5f0. (#26441) 2017-08-31 09:06:40 +02:00
Jason Tedor e8a7b1b8e5 Remove dead path conf BWC code in build
When changing how the config path is configured (from a command-line
flag to an environment variable) we had to add BWC code in the build so
that we could form clusters with 5.x nodes in them. Now that this branch
has moved to 7.x, we no longer need to be BWC with 5.x for starting
nodes. This commit removes this dead BWC code.

Relates #26446
2017-08-30 13:54:29 -04:00
Jason Tedor 53b7ad31e7 Use short name for invoking wrapper script
When starting a node in standalone tests, we sometimes use a wrapper
script as opposed to starting Elasticsearch directly (this is used for
backgrounding). On Windows, the path to this wrapper script can be
exceptionally long, exceeding the length of a batch script that cmd.exe
will invoke without whining. This commit replaces using the full path
name for this wrapper script by the short name for the wrapper script.

Additionally, the data, configuration, and jvm.options paths can also
end up being too long so we shorten those too. Care has to be taken with
the data directory because we usually rely on the node creating it on
startup but doing that will not be compatible with getting the short
name as that requires the directory already existing. Therefore, we
create that directory on-demand immediately before actually resolving
the short name.

Relates #26444
2017-08-30 13:13:03 -04:00
Tanguy Leroux f95dec797d [Docs] Convert more doc snippets (#26359)
This commit converts some remaining doc snippets so that they are now
testable.
2017-08-28 11:23:09 +02:00
Michael Basnight cfd14cd2b8 Revert shading for the low level rest client (#26367)
At current, we do not feel there is enough of a reason to shade the low
level rest client. It caused problems with commons logging and IDE's
during the brief time it was used. We did not know exactly how many
users will need this, and decided that leaving shading out until we
gather more information is best. Users can still shade the jar
themselves. For information and feeback, see issue #26366.

Closes #26328

This reverts commit 3a20922046.
This reverts commit 2c271f0f22.
This reverts commit 9d10dbea39.
This reverts commit e816ef89a2.
2017-08-25 14:13:12 -05:00
Ryan Ernst 661648b3aa Build: Allow build to configure which license/notice to embed in jars (#26373)
We currently add the apache license/notice for elasticsearch to any
plugin that uses our ES plugin gradle plugin. However, each plugin
should be able to use their own license. This commit adds a licenseFile
and noticeFile property to the root of project's using BuildPlugin,
which is added to jar files for that project.
2017-08-24 22:46:30 -07:00
Jason Tedor 911e1f6203 Handle long paths on Windows for standalone tests
In some cases our Windows builds fail due to long path names that arise
from a combination of long build job names plus long sub-project
names. While newer versions of Windows can handle long paths, invoking
batch scripts longer than 260 characters via cmd.exe is still
problematic. This leads to failing integration tests because we can not
run the commands to install plugins, create the keystore, and start the
node. This commit handles this by converting all paths on Windows used
to start an Elasticsearch node to short path names.

Relates #26365
2017-08-24 18:46:49 -04:00
Ryan Ernst 5202e7e93b Settings: Move keystore creation to plugin installation (#26329)
This commit removes the keystore creation on elasticsearch startup, and
instead adds a plugin property which indicates the plugin needs the
keystore to exist. It does still make sure the keystore.seed exists on
ES startup, but through an "upgrade" method that loading the keystore in
Bootstrap calls.

closes #26309
2017-08-24 12:12:47 -07:00
Yannick Welsch cfff71aa64 Use Javadoc tool from JAVA_HOME (#26330)
We currently run the javadoc executable from the JDK that runs Gradle, not the JDK from JAVA_HOME.

Relates to #24733
2017-08-23 10:18:27 +09:30
Yannick Welsch 3d8feff66e Use Java 9 FilePermission model (#26302)
This commit makes the security code aware of the Java 9 FilePermission changes (see #21534) and allows us to remove the `jdk.io.permissionsUseCanonicalPath` system property.
2017-08-22 11:22:00 +09:30
Yannick Welsch 41f81e2279 Use explicit test classes output dir for configuring test task
This allows the build to run with Gradle 4.1 as well, where oldTestTask.testClassesDir will return null
at configuration time.
2017-08-21 14:55:23 +09:30
Luca Cavanna 1309dfd44d Add links to external classes in clients javadoc (#25998)
The client sniffer depends on the low-level REST client, while the Java high-level REST client and the transport client depend on Elasticsearch itself. Javadoc are not that useful unless they have links to the Elasticsearch classes in the latter case, and to the low-level REST client in the sniffer javadoc. This commit adds those links.
2017-08-17 21:03:47 +02:00
Nik Everett 6d2c40e546 Enforce that responses in docs are valid json (#26249)
All of the snippets in our docs marked with `// TESTRESPONSE` are
checked against the response from Elasticsearch but, due to the
way they are implemented they are actually parsed as YAML instead
of JSON. Luckilly, all valid JSON is valid YAML! Unfurtunately
that means that invalid JSON has snuck into the exmples!

This adds a step during the build to parse them as JSON and fail
the build if they don't parse.

But no! It isn't quite that simple. The displayed text of some of
these responses looks like:
```
{
    ...
    "aggregations": {
        "range": {
            "buckets": [
                {
                    "to": 1.4436576E12,
                    "to_as_string": "10-2015",
                    "doc_count": 7,
                    "key": "*-10-2015"
                },
                {
                    "from": 1.4436576E12,
                    "from_as_string": "10-2015",
                    "doc_count": 0,
                    "key": "10-2015-*"
                }
            ]
        }
    }
}
```

Note the `...` which isn't valid json but we like it anyway and want
it in the output. We use substitution rules to convert the `...`
into the response we expect. That yields a response that looks like:
```
{
    "took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,
    "aggregations": {
        "range": {
            "buckets": [
                {
                    "to": 1.4436576E12,
                    "to_as_string": "10-2015",
                    "doc_count": 7,
                    "key": "*-10-2015"
                },
                {
                    "from": 1.4436576E12,
                    "from_as_string": "10-2015",
                    "doc_count": 0,
                    "key": "10-2015-*"
                }
            ]
        }
    }
}
```

That is what the tests consume but it isn't valid JSON! Oh no! We don't
want to go update all the substitution rules because that'd be huge and,
ultimately, wouldn't buy much. So we quote the `$body.took` bits before
parsing the JSON.

Note the responses that we use for the `_cat` APIs are all converted into
regexes and there is no expectation that they are valid JSON.

Closes #26233
2017-08-17 09:02:10 -04:00
Simon Willnauer d26c8b5c88 Evaluate settings values before passing it to task closure (#26243)
The secure settings tool reads from stdIn and we use a closure to
provide a value for this. Yet, we evaluate they value too late and end up
with the last provided value for all keys.
2017-08-16 22:31:49 +02:00
Jason Tedor e9687622bd Rename CONF_DIR to ES_PATH_CONF
The environment variable CONF_DIR was previously inconsistently used in
our packaging to customize the location of Elasticsearch configuration
files. The importance of this environment variable has increased
starting in 6.0.0 as it's now used consistently to ensure Elasticsearch
and all secondary scripts (e.g., elasticsearch-keystore) all use the
same configuration. The name CONF_DIR is there for legacy reasons yet
it's too generic. This commit renames CONF_DIR to ES_PATH_CONF.

Relates #26197
2017-08-15 06:19:06 +09:00
Nik Everett 99ac7beb8e Teach the build about betas and rcs (#26066)
The build was ignoring suffixes like "beta1" and "rc1" on the version numbers which was causing the backwards compatibility packaging tests to fail because they expected to be upgrading from 6.0.0 even though they were actually upgrading from 6.0.0-beta1. This adds the suffixes to the information that the build scrapes from Version.java. It then uses those suffixes when it resolves artifacts build from the bwc branch and for testing.

Closes #26017
2017-08-10 14:30:00 -04:00
Adrien Grand f0c1e30544 Upgrade to lucene-7.0.0-snapshot-a128fcb. (#26090) 2017-08-08 13:03:19 +02:00
Ryan Ernst 10cf5a8d3e Build: Fork javac to avoid GC overhead (#26047)
Compiling all of elasticsearch classes in one jvm, which is shared with
all of the loaded classes of gradle, can trip gc overhead limits. This
commit re-enables forking javac.
2017-08-03 15:47:29 -04:00
Ryan Ernst 072281d5aa Update version to 7.0.0-alpha1 (#25876)
This commit updates the version for master to 7.0.0-alpha1. It also adds
the 6.1 version constant, and fixes many tests, as well as marking some
as awaits fix.

Closes #25893
Closes #25870
2017-08-01 15:47:48 -04:00
Jason Tedor 6d68e8a10c Increase max compilation limit in standalone tests
Some REST tests can rapid-fire script compilations that exceed the
default script compilations per minute. Rather than subjecting ourselves
to spurious failures because of the limit being too low, we opt for a
larger limit here.
2017-08-01 14:49:47 +09:00
Yannick Welsch 1a01514081 Move tribe to a module (#25778)
This commit moves tribe to a module, stripping core from the tribe functionality.
2017-07-28 11:23:50 +02:00
Yannick Welsch efd79882a2 Allow build to directly run under JDK 9 (#25859)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

This commit also changes the `JavaCompile` tasks to only fork if necessary (i.e. when Gradle's JVM and JAVA_HOME's JVM differ).
2017-07-27 16:14:04 +02:00
Simon Willnauer b72c71083c Cleanup IndexFieldData visibility (#25900)
Today we expose `IndexFieldDataService` outside of IndexService to do maintenance
or lookup field data in different ways. Yet, we have a streamlined way to access IndexFieldData
via `QueryShardContext` that should encapsulate all access to it. This also ensures that we control all other functionality like cache clearing etc.

This change also removes the `recycler` option from `ClearIndicesCacheRequest` this option is a no-op and should have been removed long ago.
2017-07-26 20:03:42 +02:00
Martijn van Groningen a9ae52e78b
inner hits: Only access stored fields when needed
Stored fields were still being accessed for nested inner hits even if the _source was not requested.
This was done to figure out the id of the root document. However this is already known higher up the stack.
So instead this change adds the id to the nested search context, so that it is no longer required to be fetched via the stored fields.

In case the _source is large and no source is requested then hot threads like these ones would still appear:

```
100.3% (501.3ms out of 500ms) cpu usage by thread 'elasticsearch[AfXKKfq][search][T#6]'
     2/10 snapshots sharing following 22 elements
       org.apache.lucene.store.DataInput.skipBytes(DataInput.java:352)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.skipField(CompressingStoredFieldsReader.java:246)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:601)
       org.apache.lucene.index.CodecReader.document(CodecReader.java:88)
       org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
       org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:347)
       org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:219)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422)
```

and:

```
8/10 snapshots sharing following 27 elements
       org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:135)
       org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState$1.fillBuffer(CompressingStoredFieldsReader.java:531)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState$1.readBytes(CompressingStoredFieldsReader.java:550)
       org.apache.lucene.store.DataInput.readBytes(DataInput.java:87)
       org.apache.lucene.store.DataInput.skipBytes(DataInput.java:350)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.skipField(CompressingStoredFieldsReader.java:246)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:601)
       org.apache.lucene.index.CodecReader.document(CodecReader.java:88)
       org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
       org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:347)
       org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:219)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422)
```
2017-07-25 12:10:59 +02:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
Adrien Grand 91fe8d5366 Enforce that bash is used when running `gradle run`.
Using `sh` means we used whatever default the system has, which is `dash` on
Ubuntu, even though our startup script is written for bash (see the shebang).
2017-07-21 10:46:06 +02:00
Jason Tedor 0310a6a947 Introduce elasticsearch-env
This commit introduces the elasticsearch-env script. The purpose of this
script is threefold:
 - vastly simplify the various scripts used in Elasticsearch
 - provide a script that can be included in other scripts in the
   Elasticsearch ecosystem (e.g., plugins)
 - correctly establish the environment for all scripts (e.g., so that
   users can run `elasticsearch-keystore` from a package distribution
   without having to worry about setting `CONF_DIR` first, otherwise the
   keystore would be created in the wrong location)

Relates #25815
2017-07-21 09:38:49 +09:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Jason Tedor e165c405ac Add an underscore to flood stage setting
This is a minor nitty bikeshedding change that renames the suffix of the
disk flood stage setting to "flood_stage" from "floodstage".

Relates #25659
2017-07-11 22:02:00 -04:00
Adrien Grand 481d5d09b2 Upgrade to lucene-7.0.0-snapshot-00142c9. (#25641)
Lucene 7.0 is feature-frozen now, so there should not be many changes until GA.
2017-07-11 13:58:55 +02:00
Jason Tedor c084542731 Bump version to 6.0.0-beta1
This commit does two things:
 - bumps the version from 6.0.0-alpha3 to 6.0.0-beta1
 - renames the 6.0.0-alpha3 version constant to 6.0.0-beta1

Relates #25621
2017-07-09 18:12:50 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates #25600
2017-07-07 19:54:36 -04:00
Martijn van Groningen 6db708ef75
Move more token filters to analysis-common module
The following token filters were moved: common grams, limit token, pattern capture and pattern raplace.

Relates to #23658
2017-07-07 10:02:52 +02:00
Simon Willnauer 1c4ef0d214 Upgrade randomizedrunner to 2.5.2 (#25533)
An issue causing confusing error messages during test execution
has been fixed randomizedtesting/randomizedtesting#250
2017-07-04 16:48:11 +02:00
Dimitrios Liappis 39d42292dd Tests: Add Debian 9 (Stretch) to the packaging tests
Debian 9 aka Stretch is the current stable[1].

Add Debian-9 to the packaging tests.

[1] https://wiki.debian.org/DebianStretch

Relates #25494
2017-07-03 11:02:17 +03:00
Drew Raines 6deb18c0de Preliminary support for ARM
This commit adds preliminary support for 64-bit ARM architectures.

Relates #25318
2017-06-30 14:22:20 -04:00
Tim Brooks 5f8be0e090 Introduce NioTransport into framework for testing (#24262)
This commit introduces a nio based tcp transport into framework for
testing.

Currently Elasticsearch uses a simple blocking tcp transport for
testing purposes (MockTcpTransport). This diverges from production
where our current transport (netty) is non-blocking.

The point of this commit is to introduce a testing variant that more
closely matches the behavior of production instances.
2017-06-28 10:51:20 -05:00
Jason Tedor cca18a2c35 Make plugin loading stricter
Today we load plugins reflectively, looking for constructors that
conform to specific signatures. This commit tightens the reflective
operations here, not allowing plugins to have ambiguous constructors.

Relates #25405
2017-06-26 21:42:53 -04:00
Jason Tedor 22beb8d03c Remove hacky node version check in NodeInfo
This commit removes a hacky way of checking that a node is running
Elasticsearch 5.x when starting standalone nodes in tests.

Relates #25406
2017-06-26 20:32:13 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Nik Everett 4306315ff6 Throw useful error on bad docs snippets (#25389)
You can continue a test started in a previous snippet by marking the
next snippet with `// TEST[continued]`. The trouble is, if you mark the
first snippet in a file with `// TEST[continued]` you'd get difficult
to predict behavior because you'd continue the test started in another
file. This will usually create a test that fails the build but it isn't
easy to track down what you did wrong. This commit catches this
scenario up front and fails the build with a useful error message.

Similarly, if you put `// TEST[continued]` directly after a
`// TESTSETUP` section then the docs tests will fail to run but the
error message did not point you to the `// TEST[continued]` snippet.
This commit catches this scenario up front as well and fails the build
with a useful error message.
2017-06-26 11:52:16 -04:00
Martijn van Groningen a34f5fa812
Move more token filters to analysis-common module
The following token filters were moved: stemmer, stemmer_override, kstem, dictionary_decompounder, hyphenation_decompounder, reverse, elision and truncate.

Relates to #23658
2017-06-26 09:02:16 +02:00
Adrien Grand 44e9c0b947 Upgrade to lucene-7.0.0-snapshot-ad2cb77. (#25349)
Most notable changes:
 - better update concurrency: LUCENE-7868
 - TopDocs.totalHits is now a long: LUCENE-7872
 - QueryBuilder does not remove the boolean query around multi-term synonyms:
   LUCENE-7878
 - removal of Fields: LUCENE-7500

For the `TopDocs.totalHits` change, this PR relies on the fact that the encoding
of vInts and vLongs are compatible: you can write and read with any of them as
long as the value can be represented by a positive int.
2017-06-22 12:35:33 +02:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
Christoph Büscher 39d9c8aa67 Remove some redundant 140 character checkstyle suppressions 2017-06-16 13:58:22 +02:00
Martijn van Groningen 428e70758a
Moved more token filters to analysis-common module.
The following token filters were moved: `edge_ngram`, `ngram`, `uppercase`, `lowercase`, `length`, `flatten_graph` and `unique`.

Relates to #23658
2017-06-15 18:28:31 +02:00
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. (#25222)
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
James Baiera 2e29b69f6a Revert "Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)""
This reverts commit b9e2a1f989.
2017-06-12 09:41:35 -04:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
Jason Tedor 4034cd429d Fix formatting of random testing seed
The whitespace is not aligned correctly and it might be crazy-making to
some developers.
2017-06-07 15:16:46 -04:00
Nik Everett d6cb73b5ef Build: Pin the random seed at startup (#24990)
Pins the random testing seed at build start rather than letting
it vary with every randomized testing invocation. This is useful
for projects where random decisions in one randomized testing run
can effect the outcome of a second randomized testing run such as
the full cluster restart tests.

The goal isn't for tests to be able to assume that random decision
will be the same in both tests. It is more to make sure that the
seed printed when a test fails reproduces the appropriate random
decisions. And pinning the seed at startup should do just that.

This works by taking the key passed as a system property if one
is passed, otherwise picking a random long and getting it into
appropriate key format. The build just calls
`new Random().nextLong()` to get the seed while randomized testing
uses a Murmur3 hash of `System.nanoTime`.
2017-06-07 09:18:43 -04:00
Tim Brooks feca0a9f33 Bumping version to v6.0.0-alpha3 (#25077) 2017-06-06 15:47:23 -05:00
Nik Everett 66007078d4 Build: Allow preserving shared dir (#24962)
This adds an option to `ClusterConfiguration` to preserve the
`shared` directory when starting up a new cluster and switches
the `qa:full-cluster-restart` tests to use it rather than
disable the clean shared task.

Relates to #24846
2017-06-04 20:40:25 -04:00
Lee Hinman b9e2a1f989 Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)"
This reverts commit 4ed0abe72d.
2017-06-02 14:42:52 -06:00
James Baiera 4ed0abe72d Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)
We're using Vagrant in more places now than before. This commit includes a plugin that verifies
the Vagrant and Virtualbox installations for projects that depend on them. This shared code
should fix up the errors we've seen from CI builds relating to the new Kerberos fixture.
2017-06-02 16:26:11 -04:00
Nik Everett a301bbbe8b Build: fix version sorting
Prevents us from incorrectly thinking the current version is
the bwc version.
2017-06-01 14:53:08 -04:00
Ryan Ernst 160a049930 Build: Move verifyVersions to new branchConsistency task (#25009)
This commit adds a new `branchConsistency` task which will run in CI
once a day, instead of on every commit. This allows `verifyVersions` to
not break immediately once a new version is released in maven.
2017-06-01 10:29:51 -07:00
Dimitrios Liappis 8f8fc74b2e Tests: Switch to openSUSE 42 (Leap) for packaging tests
openSUSE-13 has reached [EOL](https://en.opensuse.org/Lifetime).

Replace openSUSE-13 with openSUSE-42 (Leap) for packaging tests and
update docs.

Relates #25001
2017-06-01 15:40:47 +03:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Nik Everett e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Ryan Ernst 8dbf8a230e Build: Add back explicit exclusions and remove gradle exclusions (#24879)
When transitive dependencies are disable for a dependency, gradle adds a
wildcard exclusion to the generated pom. However, some external tools
like ivy have bugs with wildcards. This commit adds back the explicit
generation of transitive excludes, and removes the gradle generated
exclusions element from the pom.

closes #24490
2017-05-25 12:22:08 -07:00
Nik Everett 13a86fec99 Add magic $_path stash key to docs tests (#24724)
Adds a "magic" key to the yaml testing stash mostly for use with
documentation tests. When unstashing an object, `$_path` is the
path into the current position in the object you are unstashing.
This means that in docs tests you can use
`// TESTRESPONSEs/somevalue/$body.${_path}/` to mean "replace
`somevalue` with whatever is the response in the same position."

Compare how you must carefully mock out all the numbers in the profile
response without this change:
```
// TESTRESPONSE[s/"id": "\[2aE02wS1R8q_QFnYu6vDVQ\]\[twitter\]\[1\]"/"id": $body.profile.shards.0.id/]
// TESTRESPONSE[s/"rewrite_time": 51443/"rewrite_time": $body.profile.shards.0.searches.0.rewrite_time/]
// TESTRESPONSE[s/"score": 51306/"score": $body.profile.shards.0.searches.0.query.0.breakdown.score/]
// TESTRESPONSE[s/"time_in_nanos": "1873811"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.time_in_nanos/]
// TESTRESPONSE[s/"build_scorer": 2935582/"build_scorer": $body.profile.shards.0.searches.0.query.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 919297/"create_weight": $body.profile.shards.0.searches.0.query.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 53876/"next_doc": $body.profile.shards.0.searches.0.query.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "391943"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.0.time_in_nanos/]
// TESTRESPONSE[s/"score": 28776/"score": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 784451/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 1669564/"create_weight": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 10111/"next_doc": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "210682"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.1.time_in_nanos/]
// TESTRESPONSE[s/"score": 4552/"score": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 42602/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 89323/"create_weight": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 2852/"next_doc": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "304311"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.time_in_nanos/]
// TESTRESPONSE[s/"time_in_nanos": "32273"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.children.0.time_in_nanos/]
```

To how you can cavalierly mock all the numbers at once with this change:
```
// TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
```
2017-05-23 15:33:48 -04:00
javanna 7a3e38eb8e Merge branch 'master' into feature/client_aggs_parsing 2017-05-22 12:25:14 +02:00
Jason Tedor 8e2ab43009 Fix handling of dirty PID file
When Elasticsearch dies during a standalone REST test we might leave a
dirty PID file laying around. We tried to log about this, but the log
messages contained references to undefined variables so we simply died
instead of providing a helpful message to run clean. This commit
addresses this issue.
2017-05-19 13:29:14 -04:00
javanna db0490343e Merge branch 'master' into feature/client_aggs_parsing 2017-05-19 18:17:06 +02:00
Nicholas Knize deb7caf4d3 Upgrade to lucene-7.0.0-snapshot-a0aef2f
This commit upgrades master to a current lucene snapshot with commit id a0aef2f.
2017-05-19 10:20:55 -05:00
Tanguy Leroux 83aa00b3f6 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-05-19 13:13:00 +02:00
Ryan Ernst 46530c1cba Build: Fix plugin installation for integ test to have unique configuration name (#24778) 2017-05-18 10:04:52 -07:00
Nik Everett 82d2c7a142 Remove vagrant testing versions (#24754)
Now that we generate the versions list from Versions.java we can
drop the list of versions maintained for vagrant testing. One nice
thing that the vagrant testing did was to check if the list of
versions was out of date. This moves that test to the core
project.
2017-05-18 09:33:13 -04:00
Ryan Ernst 0353bd1fb6 Test: Convert rolling upgrade test to have task per wire compat version (#24758)
This commit changes the rolling upgrade test to create a set of rest
test tasks per wire compat version. The most recent wire compat version
is always tested with the `integTest` task, and all versions can be
tested with `bwcTest`.
2017-05-18 01:14:24 -07:00
Tanguy Leroux eeef2e6c31 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-05-18 09:43:57 +02:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
javanna ce7326eb88 Merge branch 'master' into feature/client_aggs_parsing 2017-05-17 17:59:00 +02:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Nik Everett c38b3360b6 Allow unstashing values into keys (#24685)
This is almost exclusively for docs test which frequently match the
entire response. This allow something like:
```
  - set: {nodes.$master.http.publish_address: host}
  - match:
      $body:
        {
          "nodes": {
            $host: {
              ... stuff in here ...
            }
          }
        }
```

This should make it possible for the docs tests to work with
unpredictable keys.
2017-05-16 11:16:12 -04:00
Tanguy Leroux 5fb04fa603 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-05-15 17:08:26 +02:00
Jason Tedor bd5aee8cfa Split disruption test suite
The disruption tests sit in a single test suite which causes these tests
to be single-threaded. We can split this test suite into multiple suites
(logically, of course) enabling them to be run in parallel reducing the
total run time of all integration tests in core. This commit splits the
discovery with service disruptions test suite into three suites
 - master disruptions
 - discovery disruptions
 - cluster disruptions

The last one could probably be better named, it is meant to represent
performing actions in the cluster (indexing, failing a shard, etc.)
while a disruption is taking place.

Relates #24662
2017-05-15 07:47:42 -04:00
Christoph Büscher 42e8d4b761 Merge branch 'master' into feature/client_aggs_parsing
Conflicts:
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/filter/InternalFilterTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/global/InternalGlobalTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/missing/InternalMissingTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/nested/InternalNestedTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/nested/InternalReverseNestedTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/sampler/InternalSamplerTests.java
	modules/parent-join/src/test/java/org/elasticsearch/join/aggregations/InternalChildrenTests.java
	test/framework/src/main/java/org/elasticsearch/search/aggregations/InternalSingleBucketAggregationTestCase.java
2017-05-15 12:25:07 +02:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Christoph Büscher 744b1afcb2 Merge branch 'master' into feature/client_aggs_parsing
Conflicts:
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogramTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsTests.java
2017-05-12 10:55:05 +02:00
Simon Willnauer 1155615536 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.

This is re-adds this commit that was revered in 952feb58e4
2017-05-11 20:22:30 +02:00
Simon Willnauer 952feb58e4 Revert "Move DeleteByQuery and Reindex requests into core (#24578)"
This reverts commit 6ea2ae32b8.
2017-05-11 18:26:40 +02:00
Simon Willnauer 6ea2ae32b8 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.
2017-05-11 16:20:40 +02:00
Christoph Büscher 570390ac36 Merge branch 'master' into feature/client_aggs_parsing 2017-05-11 11:20:39 +02:00
Christoph Büscher 64ecf4f4bf Bumping version to v6.0.0-alpha2 2017-05-10 17:12:56 +02:00
Tanguy Leroux bf718a686f Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/InternalTermsTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalExtendedStatsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalMaxTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalStatsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvgTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/cardinality/InternalCardinalityTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/geobounds/InternalGeoBoundsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/geocentroid/InternalGeoCentroidTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/min/InternalMinTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/InternalPercentilesTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/hdr/InternalHDRPercentilesRanksTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/InternalTDigestPercentilesRanksTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/sum/InternalSumTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/valuecount/InternalValueCountTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/InternalSimpleValueTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/bucketmetrics/percentile/InternalPercentilesBucketTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/derivative/InternalDerivativeTests.java
#	test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
2017-05-10 13:40:39 +02:00
Ryan Ernst 3f1ef488cd fix checkstyle after script engine rename 2017-05-10 01:04:02 -07:00
Christoph Büscher 1959fd6a1a Merge branch 'master' into feature/client_aggs_parsing 2017-05-09 19:54:36 +02:00
Lee Hinman 1907c46689 Refactor UpdateHelper into unit-testable pieces
This starts breaking up the `UpdateHelper.prepare` method so that each piece can
be individually unit tested. No actual functionality has changed.

Note however, that I did add a TODO about `ctx.op` leniency, which I'd love to
remove as a separate PR if desired.
2017-05-08 09:00:47 -06:00
Jason Tedor 4e5537d9ed Add workaround so path.data can be set in run task
Previously you could set the system property tests.es.path.data and
start the run task with a custom data directory. A change in core to
detect duplicate settings broke this. That change should stay, but we
can work around this easily by only setting path.data to the data
directory if tests.es.path.data is not set.
2017-05-05 15:57:54 -04:00
Tanguy Leroux c7c524dd3f Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-05-05 21:54:37 +02:00
Jason Tedor 035494fa17 Remove obsolete JVM options from build
We start the test JVMs with various options. There are two that we
should remove, they do not make sense.
 - we require Java 8 yet there was a conditional build option for Java 7
 - we do not set MaxDirectMemorySize in our default JVM options, we
   should not in the test JVMs either

Relates #24501
2017-05-05 00:49:18 -04:00
Nik Everett 9bc7e210a0 Test: Move flag to painless tests (#24494)
The `-XX:-OmitStackTraceInFastThrow` flag is only required by Painless's
tests so we'll only set it there. This is much simpler.
2017-05-04 13:11:09 -04:00
James Baiera d928ae210d Add Vagrant based testing fixture (#24249) 2017-05-04 10:17:55 -04:00
Dimitrios Liappis ee6deb34d0 Tests: Switch to fedora-25 in test plugin
Use fedora-25 Vagrant box in VagrantTestPlugin, which was missing from
9a3ab3e800 causing packaging test
failures.

Additionally update TESTING.asciidoc
2017-05-04 14:29:35 +03:00
javanna 62c37339b8 Merge branch 'master' into feature/client_aggs_parsing 2017-05-03 22:04:06 +02:00
Jason Tedor 23801153c7 Fix JVM test argline (#24448)
* Fix JVM test argline

The argline was being overridden by '-XX:-OmitStackTraceInFastThrow'
which led to test failures that were expecting the JVM to be in a
certain state based on the value of tests.jvm.argline but they were not
since these arguments were never passed to the JVM. Additionally, we
need to respect the provided JVM argline if it is already provided with
a flag for OmitStackTraceInFastThrow. This commit fixes this by only
setting OmitStackTraceInFastThrow if it is not already set.

* Add comment

* Fix comment

* More elaborate comment

* Sigh
2017-05-02 19:59:07 -04:00
Simon Willnauer 2f9e9460d4 Move RemoteClusterService into TransportService (#24424)
TransportService and RemoteClusterService are closely coupled already today
and to simplify remote cluster integration down the road it can be a direct
dependency of TransportService. This change moves RemoteClusterService into
TransportService with the goal to make it a hidden implementation detail
of TransportService in followup changes.
2017-05-02 18:09:32 +02:00
Nik Everett 3b47355e56 Try not to lose stacktraces (#24426)
This adds `-XX:-OmitStackTraceInFastThrow` to the JVM arguments
which *should* prevent the JVM from omitting stack traces on
common exception sites. Even though these sites are common, we'd
still like the exceptions to debug them.

This also adds the flag when running tests and adapts some tests
that had workarounds for the absense of the flag.

Closes #24376
2017-05-02 11:34:12 -04:00
javanna 60866da4b7 Merge branch 'master' into feature/client_aggs_parsing 2017-05-02 12:48:30 +02:00
David Pilato 186b401fd8 Upgrade to JUnit 4.12 (#23877)
* Upgrade to JUnit 4.12
* Add permission to junit 4.12 and remove junit4-ant specific permission
2017-05-02 09:58:21 +02:00
Jason Tedor 9525c1c4fa Remove duplicates from checkstyle suppressions
This commit removes a few duplicates from the list of checkstyle
suppressions that accumulated while auto-generating the list of
suppressions based on the files that currently exceed the 140-column
limit.
2017-04-30 12:43:04 -04:00
Jason Tedor e339d894c5 Remove errant entry in checkstyle suppressions
The list of checkstyle suppressions was auto-generated based on the list
of files that currently violate the 140-column limit. An errant entry
was committed that did no harm, but this commit removes it.
2017-04-30 12:15:54 -04:00
Jason Tedor a0f4cde905 Adjust checkstyle suppressions to 140-column limit
We are back to having a 140-column limit. While at some point we will
apply auto-formatting tools to the code base, that is a down-the-road
thing and adjusting the suppressions now shaves minutes off the build
time.

Relates #24408
2017-04-30 12:11:57 -04:00
Christoph Büscher db07a34718 Merge branch 'master' into feature/client_aggs_parsing 2017-04-28 16:42:55 +02:00
Yannick Welsch 35f78d098a Separate publishing from applying cluster states (#24236)
Separates cluster state publishing from applying cluster states:

- ClusterService is split into two classes MasterService and ClusterApplierService. MasterService has the responsibility to calculate cluster state updates for actions that want to change the cluster state (create index, update shard routing table, etc.). ClusterApplierService has the responsibility to apply cluster states that have been successfully published and invokes the cluster state appliers and listeners.
- ClusterApplierService keeps track of the last applied state, but MasterService is stateless and uses the last cluster state that is provided by the discovery module to calculate the next prospective state. The ClusterService class is still kept around, which now just delegates actions to ClusterApplierService and MasterService.
- The discovery implementation is now responsible for managing the last cluster state that is used by the consensus layer and the master service. It also exposes the initial cluster state which is used by the ClusterApplierService. The discovery implementation is also responsible for adding the right cluster-level blocks to the initial state.
- NoneDiscovery has been renamed to TribeDiscovery as it is exclusively used by TribeService. It adds the tribe blocks to the initial state.
- ZenDiscovery is synchronized on state changes to the last cluster state that is used by the consensus layer and the master service, and does not submit cluster state update tasks anymore to make changes to the disco state (except when becoming master).

Control flow for cluster state updates is now as follows:

- State updates are sent to MasterService
- MasterService gets the latest committed cluster state from the discovery implementation and calculates the next cluster state to publish
- MasterService submits the new prospective cluster state to the discovery implementation for publishing
- Discovery implementation publishes cluster states to all nodes and, once the state is committed, asks the ClusterApplierService to apply the newly committed state.
- ClusterApplierService applies state to local node.
2017-04-28 09:34:31 +02:00
Christoph Büscher 1f221154a6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-26 23:09:45 +02:00
Nik Everett fc97e25b56 Add task to look for tests in src/main (#24298)
Creates a new task `namingConventionsMain`, that runs on the
`buildSrc` and `test:framework` projects and fails the build if
any of the classes in the main artifacts are named like tests or
are non-abstract subclasses of ESTestCase.

It also fixes the three tests that would cause it to fail.
2017-04-25 21:11:47 -04:00
Tanguy Leroux 35946a13d0 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/InternalPercentilesTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/hdr/InternalHDRPercentilesTests.java
2017-04-20 13:35:21 +02:00
Nik Everett caf376c8af Start building analysis-common module (#23614)
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Ryan Ernst e82d8007e3 Build: Add jstack output when starting integ test cluster if timeout occurs (#24193)
This commit adds a call to jstack to see where each node is stuck when
starting up, if a timeout occurs. This also decreases the timeout back
to 30 seconds.
2017-04-19 10:50:47 -07:00
Tanguy Leroux 5717ac3cc6 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/DocValueFormat.java
#	core/src/test/java/org/elasticsearch/search/aggregations/InternalAggregationTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalMaxTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvgTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/min/InternalMinTests.java
2017-04-19 10:12:11 +02:00
Tanguy Leroux e81bbc288a Remove Ubuntu 12.04 (#24161)
Ubuntu 12.04 will be EOL on April 28, 2017.
2017-04-19 09:39:42 +02:00
Ryan Ernst 8f666a74f8 Test: Fix plugin integ test to depend on bundling plugin (#24170)
This was broken in the recent refactoring to add dependsOn directly to
cluster configuration.
2017-04-18 17:26:40 -07:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Tanguy Leroux 67a9696e55 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-18 14:55:58 +02:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Yannick Welsch 7e581187a7 [TEST] Increase node startup timeouts
Tests are periodically failing on CI as the wait conditions are not met. This commit increases the timeouts.
2017-04-14 10:36:46 +02:00
Ryan Ernst 13489ff3a7 Test: Fix repro line for platformTest (#24095)
This was accidentally being added to packagingTest, which then had two
repro lines.
2017-04-13 14:17:11 -07:00
Tanguy Leroux 7f730c9489 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-13 09:24:03 +02:00
Ryan Ernst fb3a281755 Build: Switch jna dependency to an elastic version (#24081)
This new version of jna is rebuilt from the official release of jna, but
with native libs linked against older glibc in order to support all
platforms elasticsearch supports.

closes #23640
2017-04-13 00:17:50 -07:00
Christoph Büscher cf89fb86b5 Merge branch 'master' into feature/client_aggs_parsing 2017-04-12 12:00:40 +02:00
Ryan Ernst 7cebc0fc93 Cleanup outdated comments for fixing up pom dependencies (#24056) 2017-04-11 23:15:30 -07:00
Nik Everett 76603d4413 Move line split in eclipse back to 140 characters
Now that we've moved checkstyle back to 140 we should move the
split.
2017-04-11 13:23:03 -04:00
Simon Willnauer 8cfb9e446c Move back to 140 chars line length (#24013)
This reverts the line limit change in #23623 - this PR doesn't touch the suppression file since we are moving towards automatic code formatting which makes it mainly obsolete.
2017-04-10 17:07:18 +02:00
javanna f538d7b8d6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-10 14:46:57 +02:00
David Roberts 37aadb2adf Add the ability to include extra notices in a plugin's NOTICES file (#23898)
Adds the option for a plugin to specify extra directories containing notices
and licenses files to be incorporated into the overall notices file that is
generated for the plugin.

This can be useful, for example, where the plugin has a non-Java dependency
that itself incorporates many 3rd party components.
2017-04-10 12:37:42 +01:00
Luca Cavanna b73f87b0ea Make buffer limit configurable in HeapBufferedConsumerFactory (#23970)
The buffer limit should have been configurable already, but the factory constructor is package private so it is truly configurable only from the org.elasticsearch.client package. Also the HttpAsyncResponseConsumerFactory interface was package private, so it could only be implemented from the org.elasticsearch.client package.

Closes #23958
2017-04-10 12:27:42 +02:00
Jason Tedor 61c5976aee Upgrade to Log4j 2.8.2
This commit upgrades the Log4j dependencies from version 2.7 to version
2.8.2. This release includes a fix for a case where Log4j could lose
exceptions in the presence of a security manager.

Relates #23995
2017-04-09 07:19:16 -04:00
Ryan Ernst 776006bac5 Collapse repository gcs classes into a single java package (#23975)
This is a single reorge of the classes to simplify making them mostly
package protected.
2017-04-07 11:27:26 -07:00
javanna 67e087088d add class to suppressions list for line length check 2017-04-07 15:08:44 +02:00
Ryan Ernst 203f8433c2 Collapse packages in ec2 discovery plugin (#23909)
This commit collapses all the classes inside ec2 discovery to a single
package name.
2017-04-05 23:51:49 -07:00
Adrien Grand d5d0f140d6 The `filter` and `significant_terms` aggregations should parse the `filter` as a filter, not as a query. (#23797)
This is important for some queries like `bool`, which are parsed differently
depending on whether we want to get a query or a filter.
2017-04-05 16:46:21 +02:00
Ryan Ernst d31d2caf09 Collapse packages in repository-s3 (#23907)
This commit puts all the classes in the repository-s3 plugin into a
single package.  In addition to simplifying the plugin, it will make it
easier to test as things that should be package private will not be
difficult to use inside tests alone.
2017-04-04 15:15:25 -07:00
Jason Tedor 8c554215e0 Ban Boolean#getBoolean
The method Boolean#getBoolean is dangerous. It is too easy to mistakenly
invoke this method thinking that it is parsing a string as a
boolean. However, what it actually does is get a system property with
the specified string, and then attempts to use usual crappy boolean
parsing in the JDK to parse that system property as boolean with
complete leniency (it parses every input value into either true or
false); that is, this method amounts to invoking
Boolean#parseBoolean(String) on the result of
System#getProperty(String). Boo. This commit bans usage of this method.

Relates #23864
2017-04-01 17:02:19 -04:00
Ryan Ernst 1a5c36509f Build: Remove exclusions hack used for gradle 2.13 (#23809)
Now that we are on gradle 3.3, we can take advantage of a fix that was
made in 2.14 which properly handles disabling transitive dependencies in
pom generation.  As it was currently, we actually ended up generated two
exclusions sections in the generated pom. This is yet another example of
why we need validation on the pom files with our generation here, but I
leave that for another day because I still don't know a good way to do
it.
2017-03-29 21:55:48 -07:00
Lee Hinman c8081bde91 Further refactor and extend testing for `TransportShardBulkAction`
This moves `updateReplicaRequest` to `createPrimaryResponse` and separates the
translog updating to be a separate function so that the function purpose is more
easily understood (and testable).

It also separates the logic for `MappingUpdatePerformer` into two functions,
`updateMappingsIfNeeded` and `verifyMappings` so they don't do too much in a
single function. This allows finer-grained error testing for when a mapping
fails to parse or be applied.

Finally, it separates parsing and version validation for
`executeIndexRequestOnReplica` into a separate
method (`prepareIndexOperationOnReplica`) and adds a test for it.

Relates to #23359
2017-03-29 10:56:51 -06:00
Jason Tedor 6b461a1458 Remove line-length violations in SmokeTestClientIT
This commit removes the line-length violations in SmokeTestClientIT.java
and removes this file from the list of suppressions.
2017-03-27 19:59:48 -04:00
Jason Tedor defd0452e7 Modify permissions dialog for plugins
This commit modifies the handling of plugins that require special
permissions to cover a case that was not previously covered.

Relates #23742
2017-03-27 15:52:45 -04:00
Jim Ferenczi 0e95c90e9f Upgrade to Lucene 6.5.0 (#23750) 2017-03-27 15:57:54 +02:00
Ryan Ernst 8a0bc73435 Build: Use targetConfiguration to remove gradle deprecation warning (#23734)
In gradle 3.3 use of getConfiguration on a ModuleDependency was
deprecated. This commit changes it to use getTargetConfiguration.
2017-03-24 09:43:38 -07:00
Jason Tedor 2517cb3062 Fix line-length violations in gce/util/Access
This commit addresses all 100-column line-length violations in
gce/util/Access.java and removes this file from the suppressions list.
2017-03-22 21:34:15 -04:00
Ryan Ernst 25448e3046 Build: remove progress logger hack for gradle 2.13 (#23679)
We are now on minimum gradle 3.3, so we no longer need the groovy/gradle
hacks used to support both 2.13 and 2.14+.
2017-03-21 23:32:40 -07:00
Ryan Ernst ee802ad63c Test: Add dump of integ test cluster logs on failure (#23688)
This commit adds a build listener to the integ test runner which will
print out an excerpt of the logs from the integ test cluster if the test
run fails.  There are future improvements that can be made (for example,
to dump excerpts from dependent clusters like in tribe node tests), but
this is a start, and would help with simple rest tests failures that we
currently don't know what actually happened on the node.
2017-03-21 23:26:07 -07:00
Boaz Leskes 38a18db81e Revert "Revert "Build: Upgrade min gradle to 3.3 (#23544)""
This reverts commit f926a717ed.
2017-03-21 11:20:04 +01:00
Boaz Leskes f926a717ed Revert "Build: Upgrade min gradle to 3.3 (#23544)"
This reverts commit 5d5815416d.
2017-03-21 10:34:07 +01:00
Ryan Ernst 5d5815416d Build: Upgrade min gradle to 3.3 (#23544)
This will allow us to get rid of deprecation warnings that appear when
using 3.3, and also get rid of extra logic for 2.13 required because of
the progress logger.
2017-03-20 21:15:17 -07:00
Jason Tedor 7b17689458 Search took time should use a relative clock
Search took time uses an absolute clock to measure elapsed time, and
then tries to deal with the complexities of using an absolute clock for
this purpose. Instead, we should use a high-precision monotonic relative
clock that is designed exactly for measuring elapsed time. This commit
modifies the search infrastructure to use a relative clock for measuring
took time, but still provides an absolute clock for the components of
search that require a real clock (e.g., index name expression
resolution, etc.).

Relates #23662
2017-03-20 18:48:51 -04:00
Tim Brooks 15a5d1da7e Setup keystore during integration tests (#22966)
This commit creates a keystore and adds settings to it during the
cluster formation for integration tests. Users can define a
`keyStoreSetting` in build files for settings that need to be placed in
the keystore.
2017-03-20 10:56:17 -05:00
Jason Tedor 2eafe8310e Format RemovePluginCommand to 100-column limit
This commit formats RemovePluginCommand.java to the 100-column limit and
removes this file from the list of suppressions.
2017-03-19 22:50:13 -04:00
Jason Tedor 73097f55e7 Format EvilJNANativesTests to 100-column limit
This commit formats EvilJNANativesTests.java to the 100-column limit and
removes this file from the list of suppressions.
2017-03-18 11:13:59 -04:00
Jason Tedor 44d75db9e2 Upgrade from JNA 4.2.2 to JNA 4.4.0
This commit upgrades the JNA dependency from version 4.2.2 to version
4.4.0.

Relates #23636
2017-03-17 21:06:16 -04:00
Nik Everett f30f18285c Eclipse: move print margin to 100 columns
This comes "for free" when you run `gradle eclipse`.
2017-03-17 16:15:44 -04:00
Jason Tedor e8b2f9afd4 Migrate to max line length of 100
This commit moves the checkstyle rule of max line length from 140
characters to 100 characters. We whitelist all existing violations and
will address them in follow-ups.

Relates #23623
2017-03-17 11:19:58 -04:00
Yannick Welsch 5bd14424e6 Fix third-party audit task for Gradle 3.4 (#23612)
In Gradle 3.4, the buildSrc plugin seems to be packaged into a jar before it is accessed by the rest of the build and the signatures file for the third-party audit task cannot be accessed as
getClass().getResource('/forbidden/third-party-audit.txt') then points to a file entry in a JAR, which cannot be loaded directly as a File object. This commit changes the third-party audit task to pass the content of the signatures file as a String instead.
2017-03-17 02:05:46 -07:00
Ryan Ernst 9d4aff524c Build: Split output for jrunscript to stdout and stderr (#23553)
While trying to improve the failure output in #23547, the stderr was
also captured from jrunscript. This was under the assumption that stderr
is only written to in case of an error. However, with java 9, when
JAVA_TOOL_OPTIONS are set, they are output to stderr. And our CI sets
JAVA_TOOL_OPTIONS for some reason. This commit fixes the jrunscript call
to use a separate buffer for stderr.
2017-03-12 08:20:32 -07:00
Jason Tedor c51ef0b2ca Honor max concurrent searches in multi-search
A previous change to the multi-search request execution to avoid stack
overflows regressed on limiting the number of concurrent search requests
from a batched multi-search request. In particular, the replacement of
the tail-recursive call with a loop could asynchronously fire off all of
the remaining search requests in the batch while max concurrent search
requests are already executing. This commit attempts to address this
issue by taking a more careful approach to the initial problem of
recurisve calls. The cause of the initial problem was due to possibility
of individual requests completing on the same thread as invoked the
search action execution. This can happen, for example, in cases when an
individual request does not resolve to any shards. To address this
problem, when an individual request completes we check if it completed
on the same thread as fired off the request. In this case, we loop and
otherwise safely recurse. Sadly, there was a unit test to check that the
maximum number of concurrent search requests was not exceeded, but that
test was broken while modifying the test to reproduce a case that led to
the possibility of stack overflow. As such, we randomize whether or not
search actions execute on the same thread as the thread that invoked the
action.

Relates #23538
2017-03-12 00:45:40 -08:00
Ryan Ernst a2e838d286 Build: Give better output for java version introspection (#23547)
This commit improves the output when jrunscript fails to include the
full output of the command. It also makes the quoting that is needed for
windows only happen on windows (which worked on java 8, but for some
reason does not work with java 9)
2017-03-11 22:12:48 -08:00
Ryan Ernst 57c1d5f821 Build: Fix jrunscript command to quote correctly on windows (#23539)
As noted
in https://github.com/elastic/elasticsearch/issues/22898#issuecomment-277192425
2017-03-11 10:15:51 -08:00
Ryan Ernst 9488985775 Test: Upgrade randomized runner to 2.5.0 (#23513)
This commit upgrades to the newest version of randomized runner. There
is a new additional check that allows ensuring the working directory
for each child jvm is empty. By default, this check will fail the test
run. However, for elasticsearch, we default to wipe the directory. For
example, if you previously told the runner to not wipe the directory, in
order to investigate a failure, the wipe option will delete this data
upon re-running the test.
2017-03-09 11:56:43 -08:00
Ryan Ernst 188c2b3a26 Build: Fix notice generation to use configured notice file (#23472)
While the esplugin extension already had an input for the base notice
file of the plugin, the NoticeTask did not actually know how to use
that, and always used the base notice file from Elasticsearch.
2017-03-02 19:02:43 -08:00
Ryan Ernst 946b9794ad Fix notice generation to use configured notice file 2017-03-02 16:52:09 -08:00
Ryan Ernst 1768efd72d Build: Allow ignoring sha checks for dependencies (#23471)
This commit adds an `ignoreSha` configuration to the `dependencyLicense`
task, which allows to not check for a sha for a given dependency jar.
This is useful for locally built jars, which will constantly change.
2017-03-02 16:49:20 -08:00
Nik Everett a54daade33 Tests InternalSingleBucketAggregation subclasses (#23388)
Adds a common base class for testing subclasses of
`InternalSingleBucketAggregation`. They are so similar they
call into question the utility of having all of these classes.
We maybe could just use `InternalSingleBucketAggregation` in
all those cases.... But for now, let's test the classes!

Relates to #22278
2017-03-01 16:50:52 -05:00
Jason Tedor 640ab1ceab Set build hash to Unknown when unavailable
This commit changes the build hash to be the string "Unknown" when for
some reason this build hash is not available. This aligns the value with
the value we use when the hash is not available from the jar
manifest. This situation can occur when running tests from a worktree
which is not currently handled correctly by JGit, the upstream
dependency that is used to acquire the hash. This causes problems when
running tests locally because the warning header pattern expects a hash
or the string "Unknown". While the warning header pattern be changed to
allow "N/A" as well, it seemed more sensible to align the value here
with the value when the hash is not available from the jar manifest.

Relates #23421
2017-03-01 07:57:14 -05:00
Jim Ferenczi 5c84640126 Upgrade to lucene-6.5.0-snapshot-d00c5ca (#23385)
Lucene upgrade
2017-02-27 18:39:04 +01:00
Ryan Ernst 48280a9403 Tests: Add platformTest to vagrant (#23339)
This change adds a new test task, platformTest, which runs `gradle test
integTest` within a vagrant host. This will allow us to still test on
all the supported platforms, but be able to standardize on the tools
provided in the host system, for example, with a modern version of git
that can allow #22946.

In order to have sufficient memory and cpu to run the tests, the
vagrantfile has been updated to use 8GB and 4 cpus by default. This can
be customized with the `VAGRANT_MEMORY` and `VAGRANT_CPUS` environment
variables.  Also, to save time to show this can work, it currently uses
the same Vagrantfile the packaging tests do. There are a lot of cleanups
we can do to how the gradle-vagrant tasks work, including generating
Vagrantfile altogether, but I think this is fine for now as the same
machines that would run platformTest run packagingTest, and they are
relatively beefy machines, so the higher memory and cpu for them, with
either task, should not be an issue.
2017-02-27 09:21:28 -08:00
Ryan Ernst 37452dbe23 Tests: Fix reproduce line for packagingTest (#23365)
This commit fixes the reproduce line output when the vagrant packagingTest
fails. Before only the `gradle packagingTest` would be output, but the
seed and list of versions was swallowed by groovy with an ancillary
failure (due to the `+` being on the wrong line for a string
continuation).  With the new reproduce line, it is now output next to
the task right after failure, contains the actual task (specific to the
box that fails), and contains the seed. It also no longer contains the
upgrade versions list, as the seed is used to determine which of those
to use, and the same file would be read when testing a failure on a
particular git commit. Finally, this also ties bats test setup directly
to packagingTest, instead of to the vagrant up command.
2017-02-24 19:40:28 -08:00
Ryan Ernst de8049fd2a Tests: Ensure multi node integ tests wait on first node
When a rest integ test has multiple nodes, each node is supposed to not
start configuring itself until the first node has been started, so that
the unicast host information can be written. However, this was never
explicitly setup to occur, and we were just very lucky with the current
gradle version and stability of the code always produced a task graph
that had node0 starting first. With the recent refactorings to integ
tests, the order has changed. This commit fixes the ordering by adding
an explicit dependency between the first node and the other nodes.
2017-02-22 20:54:58 -08:00
Ryan Ernst 74ecd34fd7 Build: Change location in zip of license and notice inclusion for plugins (#23316)
This commit moves the LICENSE.txt and NOTICE.txt files for each plugin
to be alongside the other plugin files, inside the elasticsearch subdir.
This ensures those files are installed alongside the plugin.
2017-02-22 16:13:50 -08:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Nik Everett d1de9574ea Checkstyle: Fix link lengths in sampler aggregation 2017-02-17 15:03:57 -05:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ryan Ernst fbf824f6a6 Fix plugin properties generation to be exact about the file to copy to
the output zip
2017-02-16 13:43:07 -08:00
Ryan Ernst efb6259e3a Tests: Re-enable debian-8 and fedora-24 packaging tests (#23217)
These images have been rebuilt to be preloaded with java 8 installed.
This change re-enables the systems. It also removes some redundancy in
the rpm checks I found while testing the new images, and fixes a
potential issue with generated resources in plugins where a stale dir
can cause junk to get into the distribution.
2017-02-16 11:26:46 -08:00
Ryan Ernst 6cdf4f3f72 Plugins: Include license and notice files in zip (#23191)
This commit adds the elasticsearch LICENSE.txt to all plugins that
released with elasticsearch, as well as a generated NOTICE.txt specific
to the dependencies of each plugin.
2017-02-15 11:23:12 -08:00
Ryan Ernst 8453cf0622 Build: Add notice file generation (#23170)
This change improves the notice file present in our distributions to
include notice and license files from each included dependency.

closes #22546
2017-02-15 09:40:16 -08:00
Ryan Ernst c4625f230f Build: Support mavenLocal flag in buildSrc (#23157)
When configuring which repositories to pull from, we currently add
mavenLocal() when the `repos.mavenLocal` flag is set. However, this is
only done in normal projects, but not the special buildSrc project. This
change adds that support. Note that this was not possible before gradle
2.13, as there was a bug which prevented sys props from reaching the
buildSrc project (https://issues.gradle.org/browse/GRADLE-2475).
However, we already require 2.13+.
2017-02-14 21:31:56 -08:00
Uwe Schindler 82613198fe Update to forbiddenapis 2.3 (improves Gradle configuration time) (#23154) 2017-02-13 14:30:08 -08:00
Adrien Grand 709cc9ba65 Upgrade to lucene-6.5.0-snapshot-f919485. (#23087) 2017-02-10 15:08:47 +01:00
Tim Brooks fcc568fd8d Add methods requiring connect to forbidden apis (#22964)
This is related to #22116. This commit adds calls that require
SocketPermission connect to forbidden APIs.

The following calls are now forbidden:

- java.net.URL#openStream()
- java.net.URLConnection#connect()
- java.net.URLConnection#getInputStream()
- java.net.Socket#connect(java.net.SocketAddress)
- java.net.Socket#connect(java.net.SocketAddress, int)
- java.nio.channels.SocketChannel#open(java.net.SocketAddress)
- java.nio.channels.SocketChannel#connect(java.net.SocketAddress)
2017-02-07 14:41:50 -06:00
Martijn van Groningen c8d470f190 Change `org.elasticsearch.bootstrap.JNAKernel32Library$SizeT` constructor's modifier to public.
Otherwise `NativeMappedConverter` can't construct this class.

Closes #22991
2017-02-06 15:16:16 +01:00
Adrien Grand c8496fc4f4 Upgrade to Lucene 6.4.1. (#22978) 2017-02-06 09:28:43 +01:00
Ryan Ernst 796087a5a9 Build: Disable fedora for packaging tests (#22974)
Now that debian is disabled, we are seeing similar failures with fedora
not able to install java. This commit temporarily disables fedora until
it is once again stable.
2017-02-03 18:33:53 -08:00
Lee Hinman 39e7c30912 Change certain replica failures not to fail the replica shard
This changes the way that replica failures are handled such that not all
failures will cause the replica shard to be failed or marked as stale.

In some cases such as refresh operations, or global checkpoint syncs, it is
"okay" for the operation to fail without the shard being failed (because no data
is out of sync). In these cases, instead of failing the shard we should simply
fail the operation, and, in the event it is a user-facing operation, return a
5xx response code including the shard-specific failures.

This was accomplished by having two forms of the `Replicas` proxy, one that is
for non-write operations that does not fail the shard, and one that is for write
operations that will fail the shard when an operation fails.

Relates to #10708
2017-02-03 14:39:46 -07:00
Ryan Ernst bbf44394a8 Build: Temporarily remove debian 8 for packaging tests (#22967)
Debian 8 has been having issues with the openjdk package dependencies
being broken. This comment comments out debian-8 from the boxes which
packaging tests will run on CI.
2017-02-03 12:57:21 -08:00
Tim Brooks f70188ac58 Remove connect SocketPermissions from core (#22797)
This is related to #22116. Core no longer needs `SocketPermission`
`connect`.

This permission is relegated to these modules/plugins:
- transport-netty4 module
- reindex module
- repository-url module
- discovery-azure-classic plugin
- discovery-ec2 plugin
- discovery-gce plugin
- repository-azure plugin
- repository-gcs plugin
- repository-hdfs plugin
- repository-s3 plugin

And for tests:
- mocksocket jar
- rest client
- httpcore-nio jar
- httpasyncclient jar
2017-02-03 09:39:56 -06:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates #22960
2017-02-03 09:46:44 -05:00
Ryan Ernst be0236416b Build: Improve detection of older java versions (#22950)
This change switches to using jrunscript, instead of jjs, for detecting
version properties of java, which is available on java versions prior to 8.

closes #22898
2017-02-02 22:58:19 -08:00
Ryan Ernst 7f59bed87b Build: Add nicer error message for pre java 8 (#22911)
This commit adds clearer error message when trying to find the version of java, but
the jjs binary is not found.

closes #22898
2017-02-01 14:07:21 -08:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Jason Tedor 930282e161 Introduce sequence-number-based recovery
This commit introduces sequence-number-based recovery. When a replica
has fallen out of sync, rather than performing a file-based recovery we
first attempt to replay operations since the last local checkpoint on
the replica. To do this, at the start of recovery the replica tells the
primary what its local checkpoint is. The primary will then wait for all
operations between that local checkpoint and the current maximum
sequence number to complete; this is to ensure that there are no gaps in
the operations that will be replayed from the primary to the
replica. This is a best-effort attempt as we currently have no
guarantees on the primary that these operations will be available; if we
are not able to replay all operations in the desired range, we just
fallback to file-based recovery. Later work will strengthen the
guarantees.

Relates #22484
2017-01-27 08:16:38 -08:00
Christoph Büscher 33fd1a606b Remove unneeded checkstyle LineLength check suppression for more classes 2017-01-26 19:40:49 +01:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Nik Everett d704a880e7 Add tests for top_hits aggregation (#22754)
Add unit tests for `TopHitsAggregator` and convert some snippets in
docs for `top_hits` aggregation to `// CONSOLE`.

Relates to #22278
Relates to #18160
2017-01-25 16:15:50 -05:00
Yannick Welsch e37a7d73a2 Remove leftover build files and simplify ProgressLogger usage
These files should have been removed in an earlier commit. This commit also simplifies usage of ProgressLoggerWrapper by using the Groovy delegation instead of using explicit delegation.
2017-01-24 20:44:14 +01:00
Luca Cavanna 47c0e13a3b Stop returning "es." internal exception headers as http response headers (#22703)
move "es." internal headers to separate metadata set in ElasticsearchException and stop returning them as response headers

Closes #17593

* [TEST] remove ESExceptionTests, move its methods to ElasticsearchExceptionTests or ExceptionSerializationTests
2017-01-24 16:12:45 +01:00
Yannick Welsch 12b6ff5233 Use Gradle-version specific source sets
Instead of using Gradle-version specific compilation options, use distinct source sets. This also allows compilation of buildSrc/build-tools under IDEs that
don't understand the version-specific compilation options.

Relates to #22669
2017-01-24 11:09:57 +01:00
Yannick Welsch 36198e0275 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-24 11:09:57 +01:00
Jim Ferenczi 8028578305 Upgrade to Lucene 6.4.0 (#22724)
* Upgrade to Lucene 6.4.0

`ValueSource`s are now converted to `DoubleValueSource`s using the Lucene adapter made for the migration to the new API in 6.4.0.
2017-01-21 04:48:01 +01:00
Jason Tedor 8f6c074691 Revert "Make build Gradle 2.14 / 3.x compatible (#22669)"
This reverts commit 652cb7dbf7.

Relates #22727
2017-01-20 18:16:45 -05:00
Ryan Ernst c5b4bba30b S3 repository: Deprecate specifying credentials through env vars, sys props, and remove profile files (#22567)
* S3 repository: Deprecate specifying credentials through env vars and sys props

This is a follow up to #22479, where storing credentials secure way was
added.
2017-01-19 12:36:32 -08:00
Nik Everett 3ea37ec817 Fix Eclipse project generation
The new plugin causes Eclipse to go a bit crazy. So we skip it
for Eclipse.

Relates to #22646
2017-01-19 13:13:11 -05:00
Tim Brooks a10aa8aade Add TestWithDependenciesPlugin to build (#22646)
This commit adds a MessyRestTestPlugin to the gradle build. It extends 
StandaloneRestTestPlugin. The main piece of functionality that it adds 
is to copy plugin-metadata from dependencies into the 
generated-resources for the current test source. This is necessary to 
ensure that permissions for dependencies are applied when running the 
tests.

A current limitation is that the permissions are applied differently 
than in the distribution sources. When permissions are granted to all 
depedencies for a module or plugin, the permissions are granted to all 
dependencies on the classpath for tests besides a few hardcoded 
exclusions:
- es core
- es test framework
- lucene test framework
- randomized runner
- junit library
2017-01-19 09:43:53 -06:00
Yannick Welsch 652cb7dbf7 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-19 09:56:54 +01:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Nik Everett 1fe74a6b4b Better error when can't auto create index (#22488)
Changes the error message when `action.auto_create_index` or
`index.mapper.dynamic` forbids automatic creation of an index
from `no such index` to one of:
* `no such index and [action.auto_create_index] is [false]`
* `no such index and [index.mapper.dynamic] is [false]`
* `no such index and [action.auto_create_index] contains [-<pattern>] which forbids automatic creation of the index`
* `no such index and [action.auto_create_index] ([all patterns]) doesn't match`

This should make it more clear *why* there is `no such index`.

Closes #22435
2017-01-18 15:18:32 -05:00
Nik Everett e71b26f480 Improve unit test coverage of aggs (#22668)
Add tests for `GlobalAggregator`, `MaxAggregator`, and `InternalMax`.

Relates to #22278
2017-01-18 10:33:45 -05:00
Simon Willnauer f30b1f82ee Remove HttpServer and HttpServerAdapter in favor of a simple dispatch method (#22636)
Today we have quite some abstractions that are essentially providing a simple
dispatch method to the plugins defining a `HttpServerTransport`. This commit
removes `HttpServer` and `HttpServerAdaptor` and introduces a simple `Dispatcher` functional
interface that delegate to `RestController` by default.

Relates to #18482
2017-01-16 21:06:08 +01:00
Alexander Reelsen f6ee6e420b Indexing: Add shard id to indexing operation listener (#22606)
The IndexingOperationListener interface did not provide any
information about the shard id when a document was indexed.

This commit adds the shard id as the first parameter to all methods
in the IndexingOperationListener.
2017-01-16 09:08:16 +01:00
Tanguy Leroux 3a3ce61186 Update Jackson to 2.8.6 (#22596)
closes #22266
2017-01-13 09:05:48 +01:00
Nik Everett 4df2e182bf Fix some typos in error messages in the build 2017-01-10 16:11:05 -05:00
Luca Cavanna ddb93946aa use ElasticsearchException#renderException in BytesRestResponse#convert (#22531) 2017-01-10 20:58:31 +01:00
Nik Everett d50f96e122 Remove InternalAggregation.Type (#22511)
It is no longer needed. It used to contain a lot of strings
used by serialization but those have since been removed. Now
it is just another thing to pass around that we don't really
need.
2017-01-10 11:57:19 -05:00
Tanguy Leroux dc81afd4f8 [TESTS] Create vagrant up task for all boxes (#22496)
Currently, such tasks are only created for default boxes (centos-7, ubuntu-1404) and not all boxes and this can be misleading for developers who want to debug testing scripts on non-default boxes.
2017-01-10 09:45:33 +01:00
javanna 45d4938fcc Migrate some more responses to ToXContentObject 2017-01-06 23:31:48 +01:00
Ryan Ernst 5adb3f992c Merge pull request #22443 from rjernst/disable_assertions
Use tests.asserts flag to allow disabling assertions
2017-01-06 12:03:45 -08:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
javanna f0181b19f5 add REST high level client gradle submodule and first simple method
The RestHighLevelClient class takes as as an argument a low level client instance RestClient. The first method added is ping, which returns true if the call to HEAD / went ok and false if an IOException was thrown. Any other exception gets bubbled up.

There are two kinds of tests, a unit test (RestHighLevelClientTests) that verifies the interaction between high level and low level client, and an integration test (MainActionIT) which relies on an externally started es cluster to send requests to.
2017-01-05 10:55:47 +01:00
Ryan Ernst fff26ba39e Use tests.asserts flag to allow disabling assertions
Randomized runner uses a flag, tests.asserts, which we have previously
not used, but is used in lucene for disabling assertions. This change
modifies the gradle configuration to look for this flag and pass through
to the test runner to determine whether -ea and -esa are added to the
java commandline for tests.
2017-01-04 15:56:53 -08:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions (#22287)
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Adrien Grand f8998fece5 Upgrade to lucene-6.4.0-snapshot-084f7a0. (#22413) 2017-01-04 19:03:52 +01:00
javanna 0d67891a64 Remove ParseFieldMatcher usages from QueryParsers#parseRewriteMethod 2017-01-03 15:52:32 +01:00
javanna 41c7d3e092 Remove ParseFieldMatcher usage from Mappers 2017-01-03 15:52:32 +01:00
Jason Tedor 0d40608dc8 Remove leftover checkstyle suppression
This commit removes a leftover checkstyle suppression for a source file
that was temporarily forked into the codebase to hack around a bug in
Log4j. When that source file was removed, the suppression was left
behind.
2017-01-02 14:05:09 -05:00
Jay Modi 1243abfecc build: do not use cached snapshots for backwards compatibility tests
The backwards compatibility tests rely on gradle's built-in mechanisms for resolving dependencies
to get the zip of the older version we test against. By default, this will cache snapshots for
24 hours, which can lead to unexpected failures in CI. This change makes the special configurations
for backwards compatibility always update their snapshots by setting the amount of time to cache
to 0 seconds.
2016-12-23 08:17:16 -05:00
Jason Tedor 432ec54347 Apply logging levels in hierarchical order
This commit adds a test for applying logging levels in hierarchical
order, and addresses an issue with restoring the logging levels at the
end of a test or suite.
2016-12-23 07:51:19 -05:00
Jason Tedor faaa671fb6 Enable assertions in integration tests
When starting a standalone cluster, we do not able assertions. This is
problematic because it means that we miss opportunities to catch
bugs. This commit enables assertions for standalone integration tests,
and fixes a couple bugs that were uncovered by enabling these.

Relates #22334
2016-12-22 20:08:02 -05:00
javanna 55feae1e9a [TEST] remove deleted test classes from checkstyle suppressions
Relates to #21915
2016-12-21 13:28:43 +01:00
Luca Cavanna ae01a51b44 [TEST] make ESSingleNodeTestCase tests repeatable (#22283)
If we conditionally do random things, e.g. initialize a node only after the first test, we have to make sure that we unconditionally create a new seed calling random.nextLong(), then initialize the node under a private randomness context. This makes sure that any random usage through Randomness.get() will retrieve the proper random instance through RandomizedContext.current().getRandom(). When running under private randomness, the context will return the Random instance that was created with the provided seed (forked from the main random instance) rather than the main Random that's exposed to tests as well. Otherwise tests become non repeatable because that initialization part happens only before the first executed test.
2016-12-21 11:44:24 +01:00
Luca Cavanna 3421e54a42 Add fromXContent method to GetResponse (#22082)
Moved field values `toXContent` logic to `GetField` (from `GetResult`), which outputs its own fields, and can also parse them now. Also added `fromXContent` to `GetResult` and `GetResponse`.

 The start object and end object for `GetResponse` output have been moved to `GetResult#toXContent`, from the corresponding rest action. This makes it possible to have `toXContent` and `fromXContent` completely symmetric, as parsing requires looping till an end object is found which is weird when the corresponding `toXContent` doesn't print that out.

This also introduces the foundation for testing retrieval of _source and stored field values.
2016-12-19 17:21:26 +01:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
Daniel Mitterdorfer b2aaeb56f3 Update JMH to 1.17.3 2016-12-19 10:02:42 +01:00
Ryan Ernst 9e5cedae23 Fix line lengths in renamed seccomp file 2016-12-16 22:18:56 -08:00
Nik Everett e9bb8d8b38 Don't allow yaml tests with `warnings` that don't skip `warnings` (#21989)
If you write a yaml test with a `warnings` section in a `do` block
that doesn't also have a corresponding `skip` section for `warnings`
then client test runners that don't support `warnings` will fail.
This causes the elasticsearch build to fail so we catch these errors
earlier.

Related to #21811
2016-12-08 13:17:31 -05:00
Nik Everett 0f7c20ae81 Build: NORELEASE is the same as norelease (#22006)
Changes the build to recognize `NORELEASE` as well as `NOCOMMIT` to
mean the same thing as `norelease` and `nocommit` respectively. This
is useful because people have been using them that way but haven't
realized that only the lowercase versions worked.

This also explicitly forbids silly things like `NoReLeAsE` and
`noCOMMIT`, failing the build and telling you to spell them properly.
2016-12-08 11:50:03 -05:00
Simon Willnauer 20177f6eee [TEST] Add back ExternalTestCluster - downstream tests still use it 2016-12-02 10:54:27 +01:00
Jim Ferenczi d791ddf704 Upgrade to lucene-6.4.0-snapshot-ec38570 (#21853)
Set lucene version to 6.4.0-snapshot-ec38570 and update all the sha1s/license
Fix invalid combo after upgrade in query_string query. split_on_whitespace=false is disallowed if auto_generate_phrase_queries=true
Adapt the expectations of some tests to the new format of the Lucene explain output
2016-11-29 18:40:31 +01:00