This commit add a basic infrastructure as well as primitive tests
to ensure version backwards compatibility between the current
development trunk and an arbitrary previous version. The compatibility
tests are simple unit tests derived from a base class that starts
and manages nodes from a provided elasticsearch release package.
Use the following commandline executes all backwards compatiblity tests
in isolation:
```
mvn test -Dtests.bwc=true -Dtests.bwc.version=1.2.1 -Dtests.class=org.elasticsearch.bwcompat.*
```
These tests run basic checks like rolling upgrades and
routing/searching/get etc. against the specified version. The version
must be present in the `./backwards` folder as
`./backwards/elasticsearch-x.y.z`
Needed for REST backwards compatibility tests, since we need to run older tests with the latest runner, which randomizes shards and replicas, but the tests rely on defaults (5,1).
Done in a generic way based on compatibility versions e.g. `-Dtests.compatibility=1.0.0` allows to run tests in a special manner that is compatibile with 1.0.0 version.
Also moved back randomIndexTemplate to ElasticsearchIntegrationTest (from ImmutableCluster) where all the randomized aspects should be.
Closes#5897
The blacklist can be provided through -Dtests.rest.blacklist and supports a comma separated list of globs
e.g. -Dtests.rest.blacklist=get/10_basic/*,index/*/*
Also added some missing docs and made it clearer that the suite/test descriptions effectively contains their (relative) path (api/yaml_file/test section)
Closes#5881
ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner
We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.
Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)
Closes#5654
Multiple nodes are now started when running REST tests against the `TestCluster` (default randomized settings are now used instead of the hardcoded `1`)
Added also randomized round-robin based on all available nodes, and ability to provide multiple addresses when running tests against an external cluster to have the same behaviour
The REST layer can now be tested through tests that are shared between all the elasticsearch official clients.
The tests are based on REST specification that can be found on the elasticsearch-rest-api-spec project and consist of YAML files that describe the operations to be executed and the obtained results that need to be tested.
REST tests can be executed through the ElasticsearchRestTests class, which relies on the rest-spec git submodule that contains the rest spec and tests pulled from the elasticsearch-rest-spec-api project. The rest-spec submodule gets automatically initialized and updated through maven (generate-test-resources phase).
The REST runner and the needed classes are distributed within the test artifact.
The following are the options supported by the REST tests runner:
- tests.rest[true|false|host:port]: determines whether the REST tests need to be run and if so whether to rely on an external cluster (providing host and port) or fire a test cluster (default)
- tests.rest.suite: comma separated paths of the test suites to be run (by default loaded from /rest-spec/test classpath). it is possible to run only a subset of the tests providing a sub-folder or even a single yaml file (the default /rest-spec/test prefix is optional when files are loaded from classpath) e.g. -Dtests.rest.suite=index,get,create/10_with_id
- tests.rest.spec: REST spec path (default /rest-spec/api from classpath)
- tests.iters: runs multiple iterations
- tests.seed: seed to base the random behaviours on
- tests.appendseed[true|false]: enables adding the seed to each test section's description (default false)
- tests.cluster_seed: seed used to create the test cluster (if enabled)
Closes#4469
Simon Willnauer
javanna
Britta Weber
Dawid Weiss
Shay Banon
Boaz Leskes
Lee Hinman