mirror of
https://github.com/apache/druid.git
synced 2025-02-09 03:24:55 +00:00
cfed036091
This commit is a first draft of the revised integration test framework which provides: - A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.) - Maven module druid-it-tools to hold code placed into the Docker image. - Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.) - Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes. The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
96 lines
5.6 KiB
Markdown
96 lines
5.6 KiB
Markdown
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
|
|
## History
|
|
|
|
This revision of the integration test Docker scripts is based on a prior
|
|
integration test version, which is, in turn, based on
|
|
the build used for the public Docker image used in the Druid tutorial. If you are familiar
|
|
with the prior structure, here are some of the notable changes.
|
|
|
|
* Use of "official" images for third-party dependencies, rather than adding them
|
|
to the Druid image. (Results in *far* faster image builds.)
|
|
* This project splits the prior `druid-integration-tests` project into several parts. This
|
|
project holds the Druid Docker image, while sibling projects hold the cluster definition
|
|
and test for each test group.
|
|
This allows the projects to better utilize the standard Maven build phases, and allows
|
|
better partial build support.
|
|
* The prior approach built the Docker image in the `pre-integration-test` phase. Here, since
|
|
the project is separate, we can use the Maven `install` phase.
|
|
* The prior structure ran *before* the Druid `distribution` module, hence the Druid artifacts
|
|
were not available, and the scripts did its own build, which could end up polluting the
|
|
Maven build cache. This version runs after `distribution` so it can reuse the actual build
|
|
artifacts.
|
|
* The `pom.xml` file in this project does some of the work that that `build_run_cluster.sh`
|
|
previously did, such as passing Maven versions into Docker.
|
|
* The work from the prior Dockerfile and `base-setup.sh` are combined into the revised
|
|
`base-setup.sh` here so that the work is done in the target container.
|
|
* Since the prior approach was "all-in-one", it would pass test configuration options into
|
|
the container build process so that the container is specific to the test options. This
|
|
project attempts to create a generic container and instead handle test-specific options
|
|
at container run time.
|
|
* The detailed launch commands formerly in the Dockerfile now reside in
|
|
`$DRUID_HOME/launch.sh`.
|
|
* The prior version used a much-extended version of the public launch script. Those
|
|
extensions moved into `launch.sh` with the eventual goal of using the same launch
|
|
scripts in both cases.
|
|
* The various `generate_*_cert.sh` scripts wrote into the source directory. The revised
|
|
scripts write into `target/shared/tls`.
|
|
* The shared directory previously was in `~/shared`, but that places the directory outside
|
|
of the Maven build tree. The new location is `$DRUID_DEV/docker/base-docker/target/shared`.
|
|
As a result, the directory is removed and rebuild on each Maven build. The old location was
|
|
removed via scripts, but the new one is very clearly a Maven artifact, and thus to be
|
|
removed on a Maven `clean` operation.
|
|
* The prior approach had security enabled for all tests, which makes debugging hard.
|
|
This version makes security optional, it should be enabled for just a security test.
|
|
* The orginal design was based on TestNG. Revised tests are based on JUnit.
|
|
* The original tests had "test groups" within the single directory. This version splits
|
|
the former groups into projects, so each can have its own tailored cluster definition.
|
|
* Prior images would set up MySQL inline in the container by starting the MySQL engine.
|
|
This led to some redundancy (all images would do the same thing) and also some lost
|
|
work (since the DBs in each container are not those used when running.) Here, MySQL
|
|
is in its own image. Clients can update MySQL as needed using JDBC.
|
|
* Prior code used Supervisor to launch tasks. This version uses Docker directly and
|
|
runs one process per container (except for Middle Manager, which runs Peons.)
|
|
|
|
## History
|
|
|
|
The current work builds on the prior integration tests, with changes to
|
|
simplify and speed up the process.
|
|
|
|
* The prior tests required a separate Docker build for each test "group"
|
|
Here, the former groups are sub-projects. All use the same Docker image.
|
|
* The prior code used the long-obsolte TestNG. Tests here use JUnit.
|
|
* The prior test used a TestNG suite to create test intances and inject
|
|
various items using Guice. This version uses an `Initializer` class to
|
|
do roughly the same job.
|
|
* The prior tests required test configuration be passed in on the command
|
|
line, which is tedious when debugging. This version uses a cluster
|
|
configuation file instead.
|
|
* The prior version perfomed MySQL initialization in the Docker container.
|
|
But, since each test would launch multiple containers, that work was
|
|
done multiple times. Here the work is done by the test itself.
|
|
* The prior version had a single "shared" directory for all tests in
|
|
`~/shared`. This version creates a separate shared folder for each
|
|
test module, in `<module>/target/shared`. This ensures that Maven will
|
|
delete everything between test runs.
|
|
* This version removes many of the `druid-` prefixes on the container
|
|
names. We assume that the cluster runs as the only Docker app locally,
|
|
so the extra naming just clutters things. |