druid/integration-tests-ex/docs/history.md

History

This revision of the integration test Docker scripts is based on a prior integration test version, which is, in turn, based on the build used for the public Docker image used in the Druid tutorial. If you are familiar with the prior structure, here are some of the notable changes.

• Use of "official" images for third-party dependencies, rather than adding them to the Druid image. (Results in far faster image builds.)
• This project splits the prior druid-integration-tests project into several parts. This project holds the Druid Docker image, while sibling projects hold the cluster definition and test for each test group. This allows the projects to better utilize the standard Maven build phases, and allows better partial build support.
• The prior approach built the Docker image in the pre-integration-test phase. Here, since the project is separate, we can use the Maven install phase.
• The prior structure ran before the Druid distribution module, hence the Druid artifacts were not available, and the scripts did its own build, which could end up polluting the Maven build cache. This version runs after distribution so it can reuse the actual build artifacts.
• The pom.xml file in this project does some of the work that that build_run_cluster.sh previously did, such as passing Maven versions into Docker.
• The work from the prior Dockerfile and base-setup.sh are combined into the revised base-setup.sh here so that the work is done in the target container.
• Since the prior approach was "all-in-one", it would pass test configuration options into the container build process so that the container is specific to the test options. This project attempts to create a generic container and instead handle test-specific options at container run time.
• The detailed launch commands formerly in the Dockerfile now reside in $DRUID_HOME/launch.sh.
• The prior version used a much-extended version of the public launch script. Those extensions moved into launch.sh with the eventual goal of using the same launch scripts in both cases.
• The various generate_*_cert.sh scripts wrote into the source directory. The revised scripts write into target/shared/tls.
• The shared directory previously was in ~/shared, but that places the directory outside of the Maven build tree. The new location is $DRUID_DEV/docker/base-docker/target/shared. As a result, the directory is removed and rebuild on each Maven build. The old location was removed via scripts, but the new one is very clearly a Maven artifact, and thus to be removed on a Maven clean operation.
• The prior approach had security enabled for all tests, which makes debugging hard. This version makes security optional, it should be enabled for just a security test.
• The orginal design was based on TestNG. Revised tests are based on JUnit.
• The original tests had "test groups" within the single directory. This version splits the former groups into projects, so each can have its own tailored cluster definition.
• Prior images would set up MySQL inline in the container by starting the MySQL engine. This led to some redundancy (all images would do the same thing) and also some lost work (since the DBs in each container are not those used when running.) Here, MySQL is in its own image. Clients can update MySQL as needed using JDBC.
• Prior code used Supervisor to launch tasks. This version uses Docker directly and runs one process per container (except for Middle Manager, which runs Peons.)

History

The current work builds on the prior integration tests, with changes to simplify and speed up the process.

• The prior tests required a separate Docker build for each test "group" Here, the former groups are sub-projects. All use the same Docker image.
• The prior code used the long-obsolte TestNG. Tests here use JUnit.
• The prior test used a TestNG suite to create test intances and inject various items using Guice. This version uses an Initializer class to do roughly the same job.
• The prior tests required test configuration be passed in on the command line, which is tedious when debugging. This version uses a cluster configuation file instead.
• The prior version perfomed MySQL initialization in the Docker container. But, since each test would launch multiple containers, that work was done multiple times. Here the work is done by the test itself.
• The prior version had a single "shared" directory for all tests in ~/shared. This version creates a separate shared folder for each test module, in <module>/target/shared. This ensures that Maven will delete everything between test runs.
• This version removes many of the druid- prefixes on the container names. We assume that the cluster runs as the only Docker app locally, so the extra naming just clutters things.