322 lines
12 KiB
Plaintext
322 lines
12 KiB
Plaintext
[[Testing Framework Cheatsheet]]
|
|
= Testing
|
|
|
|
[partintro]
|
|
|
|
Elasticsearch uses jUnit for testing, it also uses randomness in the
|
|
tests, that can be set using a seed, the following is a cheatsheet of
|
|
options for running the tests for ES.
|
|
|
|
== Creating packages
|
|
|
|
To create a distribution without running the tests, simply run the
|
|
following:
|
|
|
|
-----------------------------
|
|
mvn clean package -DskipTests
|
|
-----------------------------
|
|
|
|
== Other test options
|
|
|
|
To disable and enable network transport, set the `Des.node.mode`.
|
|
|
|
Use network transport:
|
|
|
|
------------------------------------
|
|
-Des.node.mode=network
|
|
------------------------------------
|
|
|
|
Use local transport (default since 1.3):
|
|
|
|
-------------------------------------
|
|
-Des.node.mode=local
|
|
-------------------------------------
|
|
|
|
Alternatively, you can set the `ES_TEST_LOCAL` environment variable:
|
|
|
|
-------------------------------------
|
|
export ES_TEST_LOCAL=true && mvn test
|
|
-------------------------------------
|
|
|
|
=== Running Elasticsearch from a checkout
|
|
|
|
In order to run Elasticsearch from source without building a package, you can
|
|
run it using Maven:
|
|
|
|
-------------------------------------
|
|
./run.sh
|
|
-------------------------------------
|
|
|
|
=== Test case filtering.
|
|
|
|
- `tests.class` is a class-filtering shell-like glob pattern,
|
|
- `tests.method` is a method-filtering glob pattern.
|
|
|
|
Run a single test case (variants)
|
|
|
|
----------------------------------------------------------
|
|
mvn test -Dtests.class=org.elasticsearch.package.ClassName
|
|
mvn test "-Dtests.class=*.ClassName"
|
|
----------------------------------------------------------
|
|
|
|
Run all tests in a package and sub-packages
|
|
|
|
----------------------------------------------------
|
|
mvn test "-Dtests.class=org.elasticsearch.package.*"
|
|
----------------------------------------------------
|
|
|
|
Run any test methods that contain 'esi' (like: ...r*esi*ze...).
|
|
|
|
-------------------------------
|
|
mvn test "-Dtests.method=*esi*"
|
|
-------------------------------
|
|
|
|
You can also filter tests by certain annotations ie:
|
|
|
|
* `@Nightly` - tests that only run in nightly builds (disabled by default)
|
|
* `@Backwards` - backwards compatibility tests (disabled by default)
|
|
* `@AwaitsFix` - tests that are waiting for a bugfix (disabled by default)
|
|
* `@BadApple` - tests that are known to fail randomly (disabled by default)
|
|
|
|
Those annotation names can be combined into a filter expression like:
|
|
|
|
------------------------------------------------
|
|
mvn test -Dtests.filter="@nightly and not @slow"
|
|
------------------------------------------------
|
|
|
|
to run all nightly test but not the ones that are slow. `tests.filter` supports
|
|
the boolean operators `and, or, not` and grouping ie:
|
|
|
|
|
|
---------------------------------------------------------------
|
|
mvn test -Dtests.filter="@nightly and not(@slow or @backwards)"
|
|
---------------------------------------------------------------
|
|
|
|
=== Seed and repetitions.
|
|
|
|
Run with a given seed (seed is a hex-encoded long).
|
|
|
|
------------------------------
|
|
mvn test -Dtests.seed=DEADBEEF
|
|
------------------------------
|
|
|
|
=== Repeats _all_ tests of ClassName N times.
|
|
|
|
Every test repetition will have a different method seed
|
|
(derived from a single random master seed).
|
|
|
|
--------------------------------------------------
|
|
mvn test -Dtests.iters=N -Dtests.class=*.ClassName
|
|
--------------------------------------------------
|
|
|
|
=== Repeats _all_ tests of ClassName N times.
|
|
|
|
Every test repetition will have exactly the same master (0xdead) and
|
|
method-level (0xbeef) seed.
|
|
|
|
------------------------------------------------------------------------
|
|
mvn test -Dtests.iters=N -Dtests.class=*.ClassName -Dtests.seed=DEAD:BEEF
|
|
------------------------------------------------------------------------
|
|
|
|
=== Repeats a given test N times
|
|
|
|
(note the filters - individual test repetitions are given suffixes,
|
|
ie: testFoo[0], testFoo[1], etc... so using testmethod or tests.method
|
|
ending in a glob is necessary to ensure iterations are run).
|
|
|
|
-------------------------------------------------------------------------
|
|
mvn test -Dtests.iters=N -Dtests.class=*.ClassName -Dtests.method=mytest*
|
|
-------------------------------------------------------------------------
|
|
|
|
Repeats N times but skips any tests after the first failure or M initial failures.
|
|
|
|
-------------------------------------------------------------
|
|
mvn test -Dtests.iters=N -Dtests.failfast=true -Dtestcase=...
|
|
mvn test -Dtests.iters=N -Dtests.maxfailures=M -Dtestcase=...
|
|
-------------------------------------------------------------
|
|
|
|
=== Test groups.
|
|
|
|
Test groups can be enabled or disabled (true/false).
|
|
|
|
Default value provided below in [brackets].
|
|
|
|
------------------------------------------------------------------
|
|
mvn test -Dtests.nightly=[false] - nightly test group (@Nightly)
|
|
mvn test -Dtests.weekly=[false] - weekly tests (@Weekly)
|
|
mvn test -Dtests.awaitsfix=[false] - known issue (@AwaitsFix)
|
|
mvn test -Dtests.slow=[true] - slow tests (@Slow)
|
|
------------------------------------------------------------------
|
|
|
|
=== Load balancing and caches.
|
|
|
|
By default, the tests run sequentially on a single forked JVM.
|
|
|
|
To run with more forked JVMs than the default use:
|
|
|
|
----------------------------
|
|
mvn test -Dtests.jvms=8 test
|
|
----------------------------
|
|
|
|
Don't count hypercores for CPU-intense tests and leave some slack
|
|
for JVM-internal threads (like the garbage collector). Make sure there is
|
|
enough RAM to handle child JVMs.
|
|
|
|
=== Test compatibility.
|
|
|
|
It is possible to provide a version that allows to adapt the tests behaviour
|
|
to older features or bugs that have been changed or fixed in the meantime.
|
|
|
|
-----------------------------------------
|
|
mvn test -Dtests.compatibility=1.0.0
|
|
-----------------------------------------
|
|
|
|
|
|
=== Miscellaneous.
|
|
|
|
Run all tests without stopping on errors (inspect log files).
|
|
|
|
-----------------------------------------
|
|
mvn test -Dtests.haltonfailure=false test
|
|
-----------------------------------------
|
|
|
|
Run more verbose output (slave JVM parameters, etc.).
|
|
|
|
----------------------
|
|
mvn test -verbose test
|
|
----------------------
|
|
|
|
Change the default suite timeout to 5 seconds for all
|
|
tests (note the exclamation mark).
|
|
|
|
---------------------------------------
|
|
mvn test -Dtests.timeoutSuite=5000! ...
|
|
---------------------------------------
|
|
|
|
Change the logging level of ES (not mvn)
|
|
|
|
--------------------------------
|
|
mvn test -Des.logger.level=DEBUG
|
|
--------------------------------
|
|
|
|
Print all the logging output from the test runs to the commandline
|
|
even if tests are passing.
|
|
|
|
------------------------------
|
|
mvn test -Dtests.output=always
|
|
------------------------------
|
|
|
|
Configure the heap size.
|
|
|
|
------------------------------
|
|
mvn test -Dtests.heap.size=512m
|
|
------------------------------
|
|
|
|
Pass arbitrary jvm arguments.
|
|
|
|
------------------------------
|
|
mvn test -Dtests.jvm.argline="-XX:HeapDumpPath=/path/to/heapdumps"
|
|
------------------------------
|
|
|
|
== Backwards Compatibility Tests
|
|
|
|
Running backwards compatibility tests is disabled by default since it
|
|
requires a release version of elasticsearch to be present on the test system.
|
|
To run backwards compatibilty tests untar or unzip a release and run the tests
|
|
with the following command:
|
|
|
|
---------------------------------------------------------------------------
|
|
mvn test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.bwc.path=/path/to/elasticsearch -Dtests.security.manager=false
|
|
---------------------------------------------------------------------------
|
|
|
|
Note that backwards tests must be run with security manager disabled.
|
|
If the elasticsearch release is placed under `./backwards/elasticsearch-x.y.z` the path
|
|
can be omitted:
|
|
|
|
---------------------------------------------------------------------------
|
|
mvn test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.security.manager=false
|
|
---------------------------------------------------------------------------
|
|
|
|
To setup the bwc test environment execute the following steps (provided you are
|
|
already in your elasticsearch clone):
|
|
|
|
---------------------------------------------------------------------------
|
|
$ mkdir backwards && cd backwards
|
|
$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.tar.gz
|
|
$ tar -xzf elasticsearch-1.2.1.tar.gz
|
|
---------------------------------------------------------------------------
|
|
|
|
== Testing the REST layer
|
|
|
|
The available integration tests make use of the java API to communicate with
|
|
the elasticsearch nodes, using the internal binary transport (port 9300 by
|
|
default).
|
|
The REST layer is tested through specific tests that are shared between all
|
|
the elasticsearch official clients and consist of YAML files that describe the
|
|
operations to be executed and the obtained results that need to be tested.
|
|
|
|
The REST tests are run automatically when executing the maven test command. To run only the
|
|
REST tests use the following command:
|
|
|
|
---------------------------------------------------------------------------
|
|
mvn verify -Dtests.filter="@Rest"
|
|
---------------------------------------------------------------------------
|
|
|
|
`ElasticsearchRestTests` is the executable test class that runs all the
|
|
yaml suites available within the `rest-api-spec` folder.
|
|
|
|
The REST tests support all the options provided by the randomized runner, plus the following:
|
|
|
|
* `tests.rest[true|false]`: determines whether the REST tests need to be run (default) or not.
|
|
* `tests.rest.suite`: comma separated paths of the test suites to be run
|
|
(by default loaded from /rest-api-spec/test). It is possible to run only a subset
|
|
of the tests providing a sub-folder or even a single yaml file (the default
|
|
/rest-api-spec/test prefix is optional when files are loaded from classpath)
|
|
e.g. -Dtests.rest.suite=index,get,create/10_with_id
|
|
* `tests.rest.blacklist`: comma separated globs that identify tests that are
|
|
blacklisted and need to be skipped
|
|
e.g. -Dtests.rest.blacklist=index/*/Index document,get/10_basic/*
|
|
* `tests.rest.spec`: REST spec path (default /rest-api-spec/api)
|
|
|
|
Note that the REST tests, like all the integration tests, can be run against an external
|
|
cluster by specifying the `tests.cluster` property, which if present needs to contain a
|
|
comma separated list of nodes to connect to (e.g. localhost:9300). A transport client will
|
|
be created based on that and used for all the before|after test operations, and to extract
|
|
the http addresses of the nodes so that REST requests can be sent to them.
|
|
|
|
== Skip validate
|
|
|
|
To disable validation step (forbidden API or `// NOCOMMIT`) use
|
|
|
|
---------------------------------------------------------------------------
|
|
mvn test -Dvalidate.skip=true
|
|
---------------------------------------------------------------------------
|
|
|
|
You can also skip this by using the "dev" profile:
|
|
|
|
---------------------------------------------------------------------------
|
|
mvn test -Pdev
|
|
---------------------------------------------------------------------------
|
|
|
|
== Testing scripts
|
|
|
|
Shell scripts can be tested with the Bash Automate Testing System tool available
|
|
at https://github.com/sstephenson/bats. Once the tool is installed, you can
|
|
execute a .bats test file with the following command:
|
|
|
|
---------------------------------------------------------------------------
|
|
bats test_file.bats
|
|
---------------------------------------------------------------------------
|
|
|
|
When executing the test files located in the `/packaging/scripts` folder,
|
|
it's possible to add the flag `ES_CLEAN_BEFORE_TEST=true` to clean the test
|
|
environment before the tests are executed:
|
|
|
|
---------------------------------------------------------------------------
|
|
ES_CLEAN_BEFORE_TEST=true bats 30_deb_package.bats
|
|
---------------------------------------------------------------------------
|
|
|
|
The current mode of execution is to copy all the packages that should be tested
|
|
into one directory, then copy the bats files into the same directory and run
|
|
those.
|