mirror of
https://github.com/apache/druid.git
synced 2025-02-07 02:28:19 +00:00
cfed036091
This commit is a first draft of the revised integration test framework which provides: - A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.) - Maven module druid-it-tools to hold code placed into the Docker image. - Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.) - Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes. The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
106 lines
4.6 KiB
Markdown
106 lines
4.6 KiB
Markdown
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
## Test Runtime Behavior
|
|
|
|
This section explains how the various configuration pieces come together
|
|
to run tests.
|
|
|
|
See also:
|
|
|
|
* [Docker setup](docker.md)
|
|
* [Docker Compose setup](compose.md)
|
|
* [Test setup](test-config.md)
|
|
|
|
## Shared Resources
|
|
|
|
This module has a number of folders that are used by all tests:
|
|
|
|
* `compose`: A collection of Docker Compose scripts that define the basics
|
|
of the cluster. Each test "inherits" the bits that it needs.
|
|
* `compose/environment-configs`: Files which define, as environment variables,
|
|
the runtime properties for each service in the cluster. (See below
|
|
for details.)
|
|
* `assets`: The `log4j2.xml` file used by images for logging.
|
|
|
|
## Container Runtime Structure
|
|
|
|
The container itself is a bit of a hybrid. The Druid distribution, along
|
|
with some test-specific extensions, is reused. The container also contains
|
|
libraries for Kafka, MySQL and MariaDB.
|
|
|
|
Druid configuration is passed into the container as environment variables,
|
|
and then converted to a `runtime.properties` file by the container launch
|
|
script. Though a bit of a [Rube Goldberg](https://en.wikipedia.org/wiki/Rube_Goldberg)
|
|
mechanism, it does have one important advantage over the usual Druid configs:
|
|
we can support inheritance and overrides. The various `<service>.env` files
|
|
provide the standard configurations. Test-specific Docker Compose files can
|
|
modify any setting.
|
|
|
|
The container mounts a shared volume, defined in the `target/shared` directory
|
|
of each test module. This volume can provide extra libraries and class path
|
|
items. The one made available by default is `log4j2.xml`, but tests can add
|
|
more as needed.
|
|
|
|
Container "output" also goes into the shared folder: logs, "cold storage"
|
|
and so on.
|
|
|
|
Each container exposes the Java debugger on port 8000, mapped to a different
|
|
host port for each container.
|
|
|
|
Each container exposes the usual Druid ports so you can work with the
|
|
container as you would a local cluster. Two handy tools are the Druid
|
|
console and the scriptable [Python client](https://github.com/paul-rogers/druid-client).
|
|
|
|
## Test Execution
|
|
|
|
Tests run using the Maven [failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/)
|
|
plugin which is designed for integration tests. The Maven phases are:
|
|
|
|
* `pre-integration-test`: Starts the test cluster with `cluster.sh up` using Docker Compose.
|
|
* `integration-test`: Runs tests that start or end with `IT`.
|
|
* `post-integration-test`: Stops the test cluster using `cluster.sh down`
|
|
* `verify`: Checks the result of the integration tests.
|
|
|
|
See [this example](https://maven.apache.org/surefire/maven-failsafe-plugin/examples/junit.html)
|
|
for JUnit setup with failsafe.
|
|
|
|
The basic process for running a test group (sub-module) is:
|
|
|
|
* Cluser startup builds a `target/shared` directory with items to be mounted
|
|
into the containers, such as the `log4j2.xml` file, sample data, etc.
|
|
The shared directory also holds log files, Druid persistent storage,
|
|
the metastore (MySQL) DB, etc. See `test-image/README.md` for details.
|
|
* The test is configured via a `druid-cluster/compose.yaml` file.
|
|
This file defines the services to run and their configuration.
|
|
* The `cluster.sh up` script builds the shared directory, loads the env vars
|
|
defined when the image was created and starts the cluster.
|
|
* Tests run on the local host within JUnit.
|
|
* The `Initialization` class loads the cluster configuration (see below),
|
|
optionally populates the Druid metadata storage, and is used to
|
|
inject instances into the test.
|
|
* The individual tests run.
|
|
* The `cluster.sh down` script shuts down the cluster.
|
|
|
|
`cluster.sh` uses the generated `test-image/target/env.sh` for versions and
|
|
and other environment variables. This ensures that tests run with the same
|
|
versions used to build the image. It also simplifies the Maven boilerplate to
|
|
be copy/pasted into each test sub-project.
|
|
|