Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-08 19:14:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
druid/integration-tests-ex/docs/next-steps.md
Paul Rogers cfed036091
Add the new integration test framework (#12368) 
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.

The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30

3.1 KiB
Raw Blame History

Future Work

The present version establishes the new IT framework. Work completed to date includes:

  • Restructure the Docker images to use the Druid produced from the Maven build. Use "official" images for dependencies.
  • Restructure the Docker compose files.
  • Create the cluster configuration mechanisms.
  • Convert one "test group" to a sub-module: "high-availability".
  • Create the pom.xml, scripts and other knick-knacks needed to tie everything together.
  • Create the initial test without using security settings to aid debugging.

However, much work remains:

  • Convert remaining tests.
  • Decide when we need full security. Convert the many certificate setup scripts.
  • Support cluster types other than Docker.

Open Tasks

The following detail items are open:

  • Disable a test if the disabled type is set in the test configuration file. Apply it to disable the HA tests for all but Docker.
  • Handle missing config files: generate a "dummy" that is disabled.
  • When launching a container build or test run from Maven, write environment variables to a target/env.sh file so that the user doesn't have to find them manually to run the helper scripts.
  • There is some redundancy in each test group project. Figure out solutions:
    • The cluster.sh script
    • Boilerplate in the pom.xml file.
  • Move test data from /resources to /shared/data. Build up the data directory from multiple sources during cluster launch.
  • Sort out which data and spec files are actually used. Remove those which are not used. Sort the files by test-specific and shared across tests by moving them into different directories.

Later Tasks

The "public" and "integration test" versions of the Docker images have diverged significantly, which makes it harder to "test what we ship." Differences include:

  • Different base image
  • Different ways to set up dependencies.
  • Different paths within the container.
  • Different launch scripts.
  • The test images place Druid in /usr/local, the public images in /opt.

The tests do want to do things beyond what the "public" image does. However, this should not require a fork of the builds. To address this issue:

  • Extend this project to create a base common to the "public" and integration test images.
  • Extend the integration test image to build on top of the public image.