OpenSearch/docs/reference/testing/testing-framework.asciidoc

236 lines
14 KiB
Plaintext
Raw Normal View History

[[testing-framework]]
== Java Testing Framework
[[testing-intro]]
Testing is a crucial part of your application, and as information retrieval itself is already a complex topic, there should not be any additional complexity in setting up a testing infrastructure, which uses elasticsearch. This is the main reason why we decided to release an additional file to the release, which allows you to use the same testing infrastructure we do in the elasticsearch core. The testing framework allows you to setup clusters with multiple nodes in order to check if your code covers everything needed to run in a cluster. The framework prevents you from writing complex code yourself to start, stop or manage several test nodes in a cluster. In addition there is another very important feature called randomized testing, which you are getting for free as it is part of the elasticsearch infrastructure.
[[why-randomized-testing]]
=== why randomized testing?
The key concept of randomized testing is not to use the same input values for every testcase, but still be able to reproduce it in case of a failure. This allows to test with vastly different input variables in order to make sure, that your implementation is actually independent from your provided test data.
If you are interested in the implementation being used, check out the http://labs.carrotsearch.com/randomizedtesting.html[RandomizedTesting webpage].
[[using-elasticsearch-test-classes]]
=== Using the elasticsearch test classes
First, you need to include the testing dependency in your project. If you use maven and its `pom.xml` file, it looks like this
[[source,xml]]
--------------------------------------------------
<dependencies>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-test-framework</artifactId>
<version>${lucene.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
--------------------------------------------------
Replace the elasticsearch version and the lucene versions with the current elasticsearch version and its accompanying lucene release.
There are already have a couple of classes, you can inherit from in your own test classes. The advantages of doing so is having already defined loggers, the whole randomized infrastructure is set up already.
[[unit-tests]]
=== unit tests
In case you only need to execute a unit test, because your implementation can be isolated that good and does not require an up and running elasticsearch cluster, you can use the `ElasticsearchTestCase`. If you are testing lucene features, use `ElasticsearchLuceneTestCase` and if you are testing concrete token streams, use the `ElasticsearchTokenStreamTestCase` class. Those specific classes execute additional checks, which ensure that no resources leaks are happening, after the test has run.
[[integration-tests]]
=== integration tests
[TEST] Randomized number of shards used for indices created during tests Introduced two levels of randomization for the number of shards (between 1 and 10) when running tests: 1) through the existing random index template, which now sets a random number of shards that is shared across all the indices created in the same test method unless overwritten 2) through `createIndex` and `prepareCreate` methods, similar to what happens using the `indexSettings` method, which changes for every `createIndex` or `prepareCreate` unless overwritten (overwrites index template for what concerns the number of shards) Added the following facilities to deal with the random number of shards: - `getNumShards` to retrieve the number of shards of a given existing index, useful when doing comparisons based on the number of shards and we can avoid specifying a static number. The method returns an object containing the number of primaries, number of replicas and the total number of shards for the existing index - added `assertFailures` that checks that a shard failure happened during a search request, either partial failure or total (all shards failed). Checks also the error code and the error message related to the failure. This is needed as without knowing the number of shards upfront, when simulating errors we can run into either partial (search returns partial results and failures) or total failures (search returns an error) - added common methods similar to `indexSettings`, to be used in combination with `createIndex` and `prepareCreate` method and explicitly control the second level of randomization: `numberOfShards`, `minimumNumberOfShards` and `maximumNumberOfShards`. Added also `numberOfReplicas` despite the number of replicas is not randomized (default not specified but can be overwritten by tests) Tests that specified the number of shards have been reviewed and the results follow: - removed number_of_shards in node settings, ignored anyway as it would be overwritten by both mechanisms above - remove specific number of shards when not needed - removed manual shards randomization where present, replaced with ordinary one that's now available - adapted tests that didn't need a specific number of shards to the new random behaviour - fixed a couple of test bugs (e.g. 3 levels parent child test could only work on a single shard as the routing key used for grand-children wasn't correct) - also done some cleanup, shared code through shard size facets and aggs tests and used common methods like `assertAcked`, `ensureGreen`, `refresh`, `flush` and `refreshAndFlush` where possible - made sure that `indexSettings()` is always used as a basis when using `prepareCreate` to inject specific settings - converted indexRandom(false, ...) + refresh to indexRandom(true, ...)
2014-02-22 05:58:05 -05:00
These kind of tests require firing up a whole cluster of nodes, before the tests can actually be run. Compared to unit tests they are obviously way more time consuming, but the test infrastructure tries to minimize the time cost by only restarting the whole cluster, if this is configured explicitly.
2014-11-28 09:38:53 -05:00
The class your tests have to inherit from is `ElasticsearchIntegrationTest`. As soon as you inherit, there is no need for you to start any elasticsearch nodes manually in your test anymore, though you might need to ensure that at least a certain number of nodes is up.
[TEST] Randomized number of shards used for indices created during tests Introduced two levels of randomization for the number of shards (between 1 and 10) when running tests: 1) through the existing random index template, which now sets a random number of shards that is shared across all the indices created in the same test method unless overwritten 2) through `createIndex` and `prepareCreate` methods, similar to what happens using the `indexSettings` method, which changes for every `createIndex` or `prepareCreate` unless overwritten (overwrites index template for what concerns the number of shards) Added the following facilities to deal with the random number of shards: - `getNumShards` to retrieve the number of shards of a given existing index, useful when doing comparisons based on the number of shards and we can avoid specifying a static number. The method returns an object containing the number of primaries, number of replicas and the total number of shards for the existing index - added `assertFailures` that checks that a shard failure happened during a search request, either partial failure or total (all shards failed). Checks also the error code and the error message related to the failure. This is needed as without knowing the number of shards upfront, when simulating errors we can run into either partial (search returns partial results and failures) or total failures (search returns an error) - added common methods similar to `indexSettings`, to be used in combination with `createIndex` and `prepareCreate` method and explicitly control the second level of randomization: `numberOfShards`, `minimumNumberOfShards` and `maximumNumberOfShards`. Added also `numberOfReplicas` despite the number of replicas is not randomized (default not specified but can be overwritten by tests) Tests that specified the number of shards have been reviewed and the results follow: - removed number_of_shards in node settings, ignored anyway as it would be overwritten by both mechanisms above - remove specific number of shards when not needed - removed manual shards randomization where present, replaced with ordinary one that's now available - adapted tests that didn't need a specific number of shards to the new random behaviour - fixed a couple of test bugs (e.g. 3 levels parent child test could only work on a single shard as the routing key used for grand-children wasn't correct) - also done some cleanup, shared code through shard size facets and aggs tests and used common methods like `assertAcked`, `ensureGreen`, `refresh`, `flush` and `refreshAndFlush` where possible - made sure that `indexSettings()` is always used as a basis when using `prepareCreate` to inject specific settings - converted indexRandom(false, ...) + refresh to indexRandom(true, ...)
2014-02-22 05:58:05 -05:00
[[number-of-shards]]
==== number of shards
The number of shards used for indices created during integration tests is randomized between `1` and `10` unless overwritten upon index creation via index settings.
Rule of thumb is not to specify the number of shards unless needed, so that each test will use a different one all the time.
[[helper-methods]]
==== generic helper methods
There are a couple of helper methods in `ElasticsearchIntegrationTest`, which will make your tests shorter and more concise.
[horizontal]
`refresh()`:: Refreshes all indices in a cluster
`ensureGreen()`:: Ensures a green health cluster state, waiting for relocations. Waits the default timeout of 30 seconds before failing.
`ensureYellow()`:: Ensures a yellow health cluster state, also waits for 30 seconds before failing.
`createIndex(name)`:: Creates an index with the specified name
`flush()`:: Flushes all indices in a cluster
`flushAndRefresh()`:: Combines `flush()` and `refresh()` calls
`optimize()`:: Waits for all relocations and optimized all indices in the cluster to one segment.
`indexExists(name)`:: Checks if given index exists
`admin()`:: Returns an `AdminClient` for administrative tasks
`clusterService()`:: Returns the cluster service java class
`cluster()`:: Returns the test cluster class, which is explained in the next paragraphs
[[test-cluster-methods]]
==== test cluster methods
The `TestCluster` class is the heart of the cluster functionality in a randomized test and allows you to configure a specific setting or replay certain types of outages to check, how your custom code reacts.
[horizontal]
`ensureAtLeastNumNodes(n)`:: Ensure at least the specified number of nodes is running in the cluster
`ensureAtMostNumNodes(n)`:: Ensure at most the specified number of nodes is running in the cluster
`getInstance()`:: Get a guice instantiated instance of a class from a random node
`getInstanceFromNode()`:: Get a guice instantiated instance of a class from a specified node
`stopRandomNode()`:: Stop a random node in your cluster to mimic an outage
`stopCurrentMasterNode()`:: Stop the current master node to force a new election
`stopRandomNonMaster()`:: Stop a random non master node to mimic an outage
`buildNode()`:: Create a new elasticsearch node
`startNode(settings)`:: Create and start a new elasticsearch node
[[accessing-clients]]
==== Accessing clients
In order to execute any actions, you have to use a client. You can use the `ElasticsearchIntegrationTest.client()` method to get back a random client. This client can be a `TransportClient` or a `NodeClient` - and usually you do not need to care as long as the action gets executed. There are several more methods for client selection inside of the `TestCluster` class, which can be accessed using the `ElasticsearchIntegrationTest.cluster()` method.
[horizontal]
`iterator()`:: An iterator over all available clients
`masterClient()`:: Returns a client which is connected to the master node
`nonMasterClient()`:: Returns a client which is not connected to the master node
`clientNodeClient()`:: Returns a client, which is running on a client node
`client(String nodeName)`:: Returns a client to a given node
`smartClient()`:: Returns a smart client
[[scoping]]
==== Scoping
By default the tests are run without restarting the cluster between tests or test classes in order to be as fast as possible. Of course all indices and templates are deleted between each test. However, sometimes you need to start a new cluster for each test or for a whole test suite - for example, if you load a certain plugin, but you do not want to load it for every test.
You can use the `@ClusterScope` annotation at class level to configure this behaviour
[source,java]
-----------------------------------------
@ClusterScope(scope=SUITE, numNodes=1)
public class CustomSuggesterSearchTests extends ElasticsearchIntegrationTest {
// ... tests go here
}
-----------------------------------------
The above sample configures an own cluster for this test suite, which is the class. Other values could be `GLOBAL` (the default) or `TEST` in order to spawn a new cluster for each test. The `numNodes` settings allows you to only start a certain number of nodes, which can speed up test execution, as starting a new node is a costly and time consuming operation and might not be needed for this test.
[[changing-node-configuration]]
==== Changing node configuration
As elasticsearch is using JUnit 4, using the `@Before` and `@After` annotations is not a problem. However you should keep in mind, that this does not have any effect in your cluster setup, as the cluster is already up and running when those methods are run. So in case you want to configure settings - like loading a plugin on node startup - before the node is actually running, you should overwrite the `nodeSettings()` method from the `ElasticsearchIntegrationTest` class and change the cluster scope to `SUITE`.
[source,java]
-----------------------------------------
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return ImmutableSettings.settingsBuilder()
.put("plugin.types", CustomSuggesterPlugin.class.getName())
.put(super.nodeSettings(nodeOrdinal)).build();
}
-----------------------------------------
[[randomized-testing]]
=== Randomized testing
The code snippets you saw so far did not show any trace of randomized testing features, as they are carefully hidden under the hood. However when you are writing your own tests, you should make use of these features as well. Before starting with that, you should know, how to repeat a failed test with the same setup, how it failed. Luckily this is quite easy, as the whole mvn call is logged together with failed tests, which means you can simply copy and paste that line and run the test.
[[generating-random-data]]
==== Generating random data
The next step is to convert your test using static test data into a test using randomized test data. The kind of data you could randomize varies a lot with the functionality you are testing against. Take a look at the following examples (note, that this list could go on for pages, as a distributed system has many, many moving parts):
* Searching for data using arbitrary UTF8 signs
* Changing your mapping configuration, index and field names with each run
* Changing your response sizes/configurable limits with each run
* Changing the number of shards/replicas when creating an index
So, how can you create random data. The most important thing to know is, that you never should instantiate your own `Random` instance, but use the one provided in the `RandomizedTest`, from which all elasticsearch dependent test classes inherit from.
[horizontal]
`getRandom()`:: Returns the random instance, which can recreated when calling the test with specific parameters
`randomBoolean()`:: Returns a random boolean
`randomByte()`:: Returns a random byte
`randomShort()`:: Returns a random short
`randomInt()`:: Returns a random integer
`randomLong()`:: Returns a random long
`randomFloat()`:: Returns a random float
`randomDouble()`:: Returns a random double
`randomInt(max)`:: Returns a random integer between 0 and max
`between()`:: Returns a random between the supplied range
`atLeast()`:: Returns a random integer of at least the specified integer
`atMost()`:: Returns a random integer of at most the specified integer
`randomLocale()`:: Returns a random locale
`randomTimeZone()`:: Returns a random timezone
In addition, there are a couple of helper methods, allowing you to create random ASCII and Unicode strings, see methods beginning with `randomAscii`, `randomUnicode`, and `randomRealisticUnicode` in the random test class. The latter one tries to create more realistic unicode string by not being arbitrary random.
If you want to debug a specific problem with a specific random seed, you can use the `@Seed` annotation to configure a specific seed for a test. If you want to run a test more than once, instead of starting the whole test suite over and over again, you can use the `@Repeat` annotation with an arbitrary value. Each iteration than gets run with a different seed.
[[assertions]]
=== Assertions
As many elasticsearch tests are checking for a similar output, like the amount of hits or the first hit or special highlighting, a couple of predefined assertions have been created. Those have been put into the `ElasticsearchAssertions` class.
[horizontal]
`assertHitCount()`:: Checks hit count of a search or count request
[TEST] Randomized number of shards used for indices created during tests Introduced two levels of randomization for the number of shards (between 1 and 10) when running tests: 1) through the existing random index template, which now sets a random number of shards that is shared across all the indices created in the same test method unless overwritten 2) through `createIndex` and `prepareCreate` methods, similar to what happens using the `indexSettings` method, which changes for every `createIndex` or `prepareCreate` unless overwritten (overwrites index template for what concerns the number of shards) Added the following facilities to deal with the random number of shards: - `getNumShards` to retrieve the number of shards of a given existing index, useful when doing comparisons based on the number of shards and we can avoid specifying a static number. The method returns an object containing the number of primaries, number of replicas and the total number of shards for the existing index - added `assertFailures` that checks that a shard failure happened during a search request, either partial failure or total (all shards failed). Checks also the error code and the error message related to the failure. This is needed as without knowing the number of shards upfront, when simulating errors we can run into either partial (search returns partial results and failures) or total failures (search returns an error) - added common methods similar to `indexSettings`, to be used in combination with `createIndex` and `prepareCreate` method and explicitly control the second level of randomization: `numberOfShards`, `minimumNumberOfShards` and `maximumNumberOfShards`. Added also `numberOfReplicas` despite the number of replicas is not randomized (default not specified but can be overwritten by tests) Tests that specified the number of shards have been reviewed and the results follow: - removed number_of_shards in node settings, ignored anyway as it would be overwritten by both mechanisms above - remove specific number of shards when not needed - removed manual shards randomization where present, replaced with ordinary one that's now available - adapted tests that didn't need a specific number of shards to the new random behaviour - fixed a couple of test bugs (e.g. 3 levels parent child test could only work on a single shard as the routing key used for grand-children wasn't correct) - also done some cleanup, shared code through shard size facets and aggs tests and used common methods like `assertAcked`, `ensureGreen`, `refresh`, `flush` and `refreshAndFlush` where possible - made sure that `indexSettings()` is always used as a basis when using `prepareCreate` to inject specific settings - converted indexRandom(false, ...) + refresh to indexRandom(true, ...)
2014-02-22 05:58:05 -05:00
`assertAcked()`:: Ensure the a request has been acknowledged by the master
`assertSearchHits()`:: Asserts a search response contains specific ids
`assertMatchCount()`:: Asserts a matching count from a percolation response
`assertFirstHit()`:: Asserts the first hit hits the specified matcher
`assertSecondHit()`:: Asserts the second hit hits the specified matcher
`assertThirdHit()`:: Asserts the third hits hits the specified matcher
`assertSearchHit()`:: Assert a certain element in a search response hits the specified matcher
`assertNoFailures()`:: Asserts that no shard failures have occurred in the response
[TEST] Randomized number of shards used for indices created during tests Introduced two levels of randomization for the number of shards (between 1 and 10) when running tests: 1) through the existing random index template, which now sets a random number of shards that is shared across all the indices created in the same test method unless overwritten 2) through `createIndex` and `prepareCreate` methods, similar to what happens using the `indexSettings` method, which changes for every `createIndex` or `prepareCreate` unless overwritten (overwrites index template for what concerns the number of shards) Added the following facilities to deal with the random number of shards: - `getNumShards` to retrieve the number of shards of a given existing index, useful when doing comparisons based on the number of shards and we can avoid specifying a static number. The method returns an object containing the number of primaries, number of replicas and the total number of shards for the existing index - added `assertFailures` that checks that a shard failure happened during a search request, either partial failure or total (all shards failed). Checks also the error code and the error message related to the failure. This is needed as without knowing the number of shards upfront, when simulating errors we can run into either partial (search returns partial results and failures) or total failures (search returns an error) - added common methods similar to `indexSettings`, to be used in combination with `createIndex` and `prepareCreate` method and explicitly control the second level of randomization: `numberOfShards`, `minimumNumberOfShards` and `maximumNumberOfShards`. Added also `numberOfReplicas` despite the number of replicas is not randomized (default not specified but can be overwritten by tests) Tests that specified the number of shards have been reviewed and the results follow: - removed number_of_shards in node settings, ignored anyway as it would be overwritten by both mechanisms above - remove specific number of shards when not needed - removed manual shards randomization where present, replaced with ordinary one that's now available - adapted tests that didn't need a specific number of shards to the new random behaviour - fixed a couple of test bugs (e.g. 3 levels parent child test could only work on a single shard as the routing key used for grand-children wasn't correct) - also done some cleanup, shared code through shard size facets and aggs tests and used common methods like `assertAcked`, `ensureGreen`, `refresh`, `flush` and `refreshAndFlush` where possible - made sure that `indexSettings()` is always used as a basis when using `prepareCreate` to inject specific settings - converted indexRandom(false, ...) + refresh to indexRandom(true, ...)
2014-02-22 05:58:05 -05:00
`assertFailures()`:: Asserts that shard failures have happened during a search request
`assertHighlight()`:: Assert specific highlights matched
`assertSuggestion()`:: Assert for specific suggestions
`assertSuggestionSize()`:: Assert for specific suggestion count
`assertThrows()`:: Assert a specific exception has been thrown
Common matchers
[horizontal]
`hasId()`:: Matcher to check for a search hit id
`hasType()`:: Matcher to check for a search hit type
`hasIndex()`:: Matcher to check for a search hit index
Usually, you would combine assertions and matchers in your test like this
[source,java]
----------------------------
SearchResponse seearchResponse = client().prepareSearch() ...;
assertHitCount(searchResponse, 4);
assertFirstHit(searchResponse, hasId("4"));
assertSearchHits(searchResponse, "1", "2", "3", "4");
----------------------------