## Test Runtime Behavior This section explains how the various configuration pieces come together to run tests. See also: * [Docker setup](docker.md) * [Docker Compose setup](compose.md) * [Test setup](test-config.md) ## Shared Resources This module has a number of folders that are used by all tests: * `compose`: A collection of Docker Compose scripts that define the basics of the cluster. Each test "inherits" the bits that it needs. * `compose/environment-configs`: Files which define, as environment variables, the runtime properties for each service in the cluster. (See below for details.) * `assets`: The `log4j2.xml` file used by images for logging. ## Container Runtime Structure The container itself is a bit of a hybrid. The Druid distribution, along with some test-specific extensions, is reused. The container also contains libraries for Kafka, MySQL and MariaDB. Druid configuration is passed into the container as environment variables, and then converted to a `runtime.properties` file by the container launch script. Though a bit of a [Rube Goldberg](https://en.wikipedia.org/wiki/Rube_Goldberg) mechanism, it does have one important advantage over the usual Druid configs: we can support inheritance and overrides. The various `.env` files provide the standard configurations. Test-specific Docker Compose files can modify any setting. The container mounts a shared volume, defined in the `target/shared` directory of each test module. This volume can provide extra libraries and class path items. The one made available by default is `log4j2.xml`, but tests can add more as needed. Container "output" also goes into the shared folder: logs, "cold storage" and so on. Each container exposes the Java debugger on port 8000, mapped to a different host port for each container. Each container exposes the usual Druid ports so you can work with the container as you would a local cluster. Two handy tools are the Druid console and the scriptable [Python client](https://github.com/paul-rogers/druid-client). ## Test Execution Tests run using the Maven [failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) plugin which is designed for integration tests. The Maven phases are: * `pre-integration-test`: Starts the test cluster with `cluster.sh up` using Docker Compose. * `integration-test`: Runs tests that start or end with `IT`. * `post-integration-test`: Stops the test cluster using `cluster.sh down` * `verify`: Checks the result of the integration tests. See [this example](https://maven.apache.org/surefire/maven-failsafe-plugin/examples/junit.html) for JUnit setup with failsafe. The basic process for running a test group (sub-module) is: * Cluser startup builds a `target/shared` directory with items to be mounted into the containers, such as the `log4j2.xml` file, sample data, etc. The shared directory also holds log files, Druid persistent storage, the metastore (MySQL) DB, etc. See `test-image/README.md` for details. * The test is configured via a `druid-cluster/compose.yaml` file. This file defines the services to run and their configuration. * The `cluster.sh up` script builds the shared directory, loads the env vars defined when the image was created and starts the cluster. * Tests run on the local host within JUnit. * The `Initialization` class loads the cluster configuration (see below), optionally populates the Druid metadata storage, and is used to inject instances into the test. * The individual tests run. * The `cluster.sh down` script shuts down the cluster. `cluster.sh` uses the generated `test-image/target/env.sh` for versions and and other environment variables. This ensures that tests run with the same versions used to build the image. It also simplifies the Maven boilerplate to be copy/pasted into each test sub-project.