mirror of https://github.com/apache/druid.git
310 lines
12 KiB
Markdown
310 lines
12 KiB
Markdown
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
# Test Creation Guide
|
|
|
|
You've played with the existing tests and you are ready to create a new test.
|
|
This section walks you through the process. If you are converting an existing
|
|
test, then see the [conversion guide](conversion.md) instead. The details
|
|
of each step are covered in other files, we'll link them from here.
|
|
|
|
## Category
|
|
|
|
The first quesetion is: should your new test go into an existing category,
|
|
or should you create a new one?
|
|
|
|
You should use an existing category if:
|
|
|
|
* Your test is a new case within an obviously-existing category.
|
|
* Your test needs the same setup as an existing category, and is quick
|
|
to run. Using the existing category avoids the need to fire up a
|
|
Docker cluster just for your test.
|
|
|
|
You should create a new category if:
|
|
|
|
* Your test uses a customized setup: set of services, service
|
|
configuration, set of external dependencies, instead.
|
|
* Your test will run for an extended time, and is best run in
|
|
parallel with other tests in a build envrionment. Your test
|
|
can share a cluster configuration with an existing test, but
|
|
the new category allows the test to run by itself.
|
|
|
|
When your test *can* reuse an existing cluser definition, then the question is
|
|
about time. It takes significan time (minutes) to start a Docker cluster. We clearly
|
|
don't want to pay that cost for a test that runs for seconds, if we could just add the
|
|
test to another category. On the other hand, if you've gone crazy and added a huge
|
|
suite of tests that take 20 minutes to run, then there is a huge win to be had by
|
|
running the tests in parallel, even if they reuse an existing cluster configuration.
|
|
Use your best judgment.
|
|
|
|
The existing categories are listed in the
|
|
`org.apache.druid.testsEx.categories` package. The classes there represent
|
|
[JUnit categories](
|
|
https://junit.org/junit4/javadoc/4.12/org/junit/experimental/categories/Categories.html).
|
|
See [Test Category](tests.md#Test+Category) for details.
|
|
|
|
If you create a new category, but want to reuse the configuration of
|
|
an existing category, add the `@Cluster` annotation as described in the above
|
|
link. Note: be sure to link to a "base" category, not to a category that, itself,
|
|
has a `@Cluster` annotation.
|
|
|
|
If you use the `@Cluster` annotation, you must also add a mapping in the
|
|
`cluster.sh` file. See the top of the file for an example.
|
|
|
|
## Cluster Configuration
|
|
|
|
If you create a new category, you must define a new cluster. There are two parts:
|
|
|
|
* Docker compose
|
|
* Test configuration
|
|
|
|
### Docker Compose
|
|
|
|
Create a new folder: `custer/<category>`, then create a `docker-compose.yaml` file
|
|
in that folder. Define your cluster by borrowing heavily from existing files.
|
|
See [compose](compose.md) for details.
|
|
|
|
The only trick is if you want to include a new external dependency. The preferred
|
|
approach is to use an "official" image. If you must, you can create a custom image
|
|
in the `it-image` module. (We've not yet done that, so if you need a custom image,
|
|
let us know and we'll figure it out.)
|
|
|
|
### Test Configuration
|
|
|
|
Tests need a variety of configuration information. This is, at present, more
|
|
complex than we might like. You will at least need:
|
|
|
|
* Describe the Docker Compose cluster
|
|
* Provide test-specific properties
|
|
|
|
You may also need:
|
|
|
|
* Test-specific Guice modules
|
|
* Environment variable bindings to various properties
|
|
* MySQL statements to pre-populate the Druid metastore DB
|
|
* And so on.
|
|
|
|
### Test Config File
|
|
|
|
The cluster and properties are defined in a config file. Create a folder
|
|
`src/test/resources/cluster/<category>`. Then add a file called `docker.yaml`.
|
|
Crib the contents from the same category from which you borrowed the Docker
|
|
Compose definitions. Strip out properties and metastore statements you don't
|
|
need. Add those you do need. See [Test Configuration](test-config.md) for the
|
|
gory details of this file.
|
|
|
|
### Test Config Code
|
|
|
|
You may also want to customize Guice, environment variable bindings, etc.
|
|
This is done in the [test setup](tests.md#Initialization) method in your test.
|
|
|
|
## Start Simple
|
|
|
|
There are *many* things that can go wrong. It is best to start simple.
|
|
|
|
### Verify the Cluster
|
|
|
|
Start by ensuring your cluster works.
|
|
|
|
* Define your cluster as described above. Or, pick one to reuse.
|
|
* Verify the cluster using `it.sh up <category>`.
|
|
* Look at the Docker desktop UI to ensure the cluster says up. if not,
|
|
track down what went wrong. Look at both the Docker (stdout) and
|
|
Druid (`target/<category>/logs/<service>.log`) files.
|
|
|
|
### Starter Test
|
|
|
|
Next, create your test file as described above and in [Tests](tests.md).
|
|
|
|
* Create the test class.
|
|
* Add the required annotations.
|
|
* Create a simple test function that just prints "hello, world".
|
|
* Create your `docker.yaml` file as decribed above.
|
|
* Start your cluster, as described above, if not already started.
|
|
* Run the test from your IDE.
|
|
* Verify that the test "passes" (that is, it prints the message.)
|
|
|
|
If so, then this means that your test connected to your custer and
|
|
verified the health of all the services declared in your `docker.yaml` file.
|
|
|
|
If something goes wrong, you'll know it is in the basics. Check your
|
|
cluster status. Double-check the `docker.yaml` structure. Check ports.
|
|
Etc.
|
|
|
|
### Client
|
|
|
|
Every test is a Druid client. Determine which service API you need. Find an
|
|
existing test client. The `DruidClusterAdminClient` is the "modern" way to
|
|
interact with the cluster, but thus far has a limited set of methods. There
|
|
are older clients as well, but they tend to be quirky. Feel free to extend
|
|
`DruidClusterAdminClient`, or use the older one: whatever works.
|
|
|
|
Inject the client into your test. See existing tests for how this is done.
|
|
|
|
Revise your "starter" test to do some trivial operation using the client.
|
|
Retest to ensure things work.
|
|
|
|
### Test Cases
|
|
|
|
From here, you can start writing tests. Explore the existing mechanisms
|
|
(including those in the original `druid-integration-tests` module which may
|
|
not yet have been ported to the new framework yet.) For example, there are
|
|
ways to store specs as files and parameterize them in tests. There is a
|
|
syntax for running queries and specifying expected results.
|
|
|
|
You may have to create a new tool to help with your test. If you do,
|
|
try to use the new mechanisms, such as `ResolvedClusterConfig` rather than
|
|
using the old, cumbersome ones. Post questions in Slack so we can help.
|
|
|
|
### Extensions
|
|
|
|
Your test may need a "non-default" extension. See [Special Environment Variables](
|
|
compose.md#Special+Environment+Variables) for how to specify test-specific
|
|
extensions. (Hint: don't copy/paste the full load list!)
|
|
|
|
Extensions have two aspects in ITs. They act like extensions in the Druid servers
|
|
running in Docker. So, the extension must be avaialble in the Docker image. All
|
|
standard Druid extensions which are available in the Druid distribution, are also
|
|
available in the image. The may not be enabled, however. Hence the need to define
|
|
the custom load list.
|
|
|
|
Your test may use code from the extension. To the *tests*, however, the extension
|
|
is just another jar: it must be listed in the `pom.xml` file. There is no such
|
|
thing as a "Druid extensions" to the tests themselves.
|
|
|
|
If you test an extension that is *not* part of the Druid distributeion, then it
|
|
has to get into the image. Reach out on the slack mailing list so we can discuss
|
|
solutions (such as mounting a directory that contains the extension).
|
|
|
|
### Retries
|
|
|
|
The old IT framework was very liberal in its use of retries. Retires were
|
|
used to handle:
|
|
|
|
* the time lag in starting a cluster,
|
|
* the latency inherent in events propagaing through a distributed system
|
|
(such as when segments get published),
|
|
* random network failures,
|
|
* flaky tests.
|
|
|
|
The new framework takes a stricter view. The framework itself will ensure
|
|
service are ready (using the Druid API for that purpose.) If a server reports
|
|
itself ready, but still fails on one of your API calls, then we've got a bug
|
|
to fix. Don't use retries to work around this issue because users won't know
|
|
to do this.
|
|
|
|
In the new framwork, tests should not be flaky. Flaky tests are a drag on
|
|
development; they waste time. If your test is flaky, please fix it. Don't count
|
|
on the amount of times things take: a busy build system will run much slower than
|
|
your dedicated laptop. And so on.
|
|
|
|
Ideally, Druid would provide a way to positively confirm that an action has
|
|
occurred. Perhaps this might be a test-only API. Otherwise, a retry is fine, but
|
|
should be coded into your test. (Or, better, implemented in a client.) Do this only
|
|
if we document that, for that API, users should poll. Otherwise, again, users of
|
|
the API under test won't know to retry, and so the test shouldn't do so either.
|
|
|
|
This leaves random failures. The right place to handle those is in the client,
|
|
since they are independent of the usage of the API.
|
|
|
|
The result of the above is that you should not need (or use) the `ITRetryUtil`
|
|
mechanism. No reason for your test to retry 240 times if something is really wrong
|
|
or your test is flaky.
|
|
|
|
This is an area under development. If you see a reason to retry, lets discuss it
|
|
and put it in the proper place.
|
|
|
|
### Travis
|
|
|
|
Run your tests in the IDE. Try them using `it.sh test <category>`. If that passes
|
|
add the test to Travis. The details on how to do so are still being worked out.
|
|
Likely, you will just copy/paste an existing test "stanza" to define your new
|
|
test. Your test will run in parallel with all other IT categories, which is why
|
|
we offered the advice above: the test has to have a good reason to fire up yet
|
|
another build task.
|
|
|
|
### Choosing the Middle Manager or Indexer
|
|
|
|
Tests should run on the Middle Manager by default. Tests can optionally run on the
|
|
Indexer. To run on Indexer:
|
|
|
|
* In the environment, `export USE_INDEXER=indexer`. (Use `middleManager`
|
|
otherwise. If the variable is not set, `middleManager` is the default.)
|
|
|
|
Then, there are two ways to handle indexer-specific configuration: the crude-but-effective
|
|
way and the subtle way.
|
|
|
|
#### Using Two Docker-Compose Files
|
|
|
|
The crude way, which involves much copy/paste and results in two files which must be maintained
|
|
in sync:
|
|
|
|
* The `cluster/<category>/docker-compose.yaml` file should be for the Middle manager. Create
|
|
a separate file called `cluster/<category>/docker-compose-indexer.yaml` to define the
|
|
Indexer-based cluster.
|
|
|
|
#### Generated Docker-Compose File
|
|
|
|
The fancy way is to use the `docker-compose.yaml` generation template described elsewhere.
|
|
In that case, the script will automatically generate either the Middle Manager, or the Indexer,
|
|
depending on the environment variable mentioned above.
|
|
|
|
#### Client Configuration
|
|
|
|
The client will choose Middle Manager or Indexer automatially if you set the
|
|
`USE_INDEXER` environment variable in your IDE. (When run via the build
|
|
process, the environment variable is already set.)
|
|
|
|
* The test `src/test/resources/cluster/<category>/docker.yaml` file should contain a conditional
|
|
entry to select define either the Middle Manager or Indexer. Example:
|
|
|
|
```yaml
|
|
middlemanager:
|
|
if: middleManager
|
|
instances:
|
|
- port: 8091
|
|
indexer:
|
|
if: indexer
|
|
instances:
|
|
- port: 8091
|
|
```
|
|
|
|
Now, the test will run on Indexer if the above environment variable is set, Middle Manager
|
|
otherwise.
|
|
|
|
#### Disable Individual Tests
|
|
|
|
You may have a test that can run only on Middle Manager or Indexer. The crude-but-effective
|
|
way to handle this is:
|
|
|
|
```
|
|
@Test
|
|
public void myMMOnlyTest()
|
|
{
|
|
if (ClusterConfig.isIndexer()) {
|
|
return; // Runs only on MM
|
|
}
|
|
// The MM-only test code here
|
|
}
|
|
```
|
|
|
|
It would be possible to define an annotation, managed by the `DruidTestRunner`, if this
|
|
becomes something we need to do often.
|
|
|