mirror of
https://github.com/apache/druid.git
synced 2025-02-10 12:05:00 +00:00
cfed036091
This commit is a first draft of the revised integration test framework which provides: - A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.) - Maven module druid-it-tools to hold code placed into the Docker image. - Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.) - Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes. The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
139 lines
5.3 KiB
Markdown
139 lines
5.3 KiB
Markdown
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
# Druid Configuration
|
|
|
|
In a normal install, Druid obtains configuration from properties files:
|
|
|
|
* `<base>/_common/common.runtime.properties`
|
|
* `<base>/<service>/runtime.properties`
|
|
|
|
In the container, Druid uses the same mechanism, though the common properties
|
|
file is empty. The container could simply mount the `runtime.properties` file.
|
|
However, doing so runs into the normal issues with Druid configuration: Druid
|
|
provides no form of inheritance: we'd have to copy/paste the same properties
|
|
over and over, which would be a maintenance headache.
|
|
|
|
Instead, the images use the same technique as the
|
|
[production Docker image](https://druid.apache.org/docs/latest/tutorials/docker.html):
|
|
we pass in a large number of environment variables.
|
|
|
|
The test configuration extends the production set to include extra
|
|
variables. Thus there are two kinds:
|
|
|
|
* General configuration (capitalized)
|
|
* Druid configuration file settings (lower case)
|
|
|
|
## Configuration Flow
|
|
|
|
We use `docker-compose` to gather up the variables. From most specific
|
|
(highest priority) to most general, configuration comes from:
|
|
|
|
* An environment variable set by the script which launches Docker Compose.
|
|
(Use sparingly, only for different test "modes" such as choosing the
|
|
DB driver, when we will use a different mode across diffrerent test runs.)
|
|
* As in-line settings in the `environment` section in the Docker Compose
|
|
definition for each service.
|
|
* In the service-specific `compose/environment-configs/<service>.env` file.
|
|
* In the common `compose/environment-configs/common.env` file.
|
|
|
|
Make test-specific changes in the test-specific Docker compose file. Make
|
|
changes to the `*.env` files only if you are certain that the change should
|
|
apply to all tests. An example is when we change something in our product
|
|
configs.
|
|
|
|
The set of defined environment variables starts with the
|
|
`druid/conf/single-server/micro-quickstart` settings. It would be great to generate
|
|
these files directly from the latest quickstart files. For now, it is a manual
|
|
process to keep the definitions in sync.
|
|
|
|
These are defined in a hierarchy:
|
|
|
|
* `common.env` - roughly equivalent to the `_common` configuration area in Druid:
|
|
contains definitions common to all Druid services. Services can override any
|
|
of the definitions.
|
|
* `<service>.env` - base definitions for each service, assume it runs stand-alone.
|
|
Adjust if test cluster runs multiple instances. Rougly equivalent to the
|
|
service-specific `runtime.properties` file.
|
|
* `docker-compose.yaml` - test-specific settings.
|
|
|
|
The `launch.sh` script converts these variables to config files in
|
|
`/tmp/conf/druid`. Those files are then added to the class path.
|
|
|
|
## Druid Config Settings
|
|
|
|
To set a Druid config variable, replace dots in the name with underscores.
|
|
|
|
In the usual properties file:
|
|
|
|
```text
|
|
druid.auth.basic.common.maxSyncRetries=20
|
|
```
|
|
|
|
In an environment config file:
|
|
|
|
```text
|
|
druid_auth_basic_common_maxSyncRetries=20
|
|
```
|
|
|
|
```text
|
|
environment:
|
|
- druid_auth_basic_common_maxSyncRetries=20
|
|
```
|
|
|
|
For everyone's sanity, please include a comment to explain the reason for
|
|
the setting if it differs from the Quickstart defaults.
|
|
|
|
## Special Config Variables
|
|
|
|
The test configuration goes beyond the production Druid image configuration
|
|
to add several extensions specfically for tests. These are variables which
|
|
handle some specific part of the configuration to avoid what would otherwise
|
|
require redundant copy/paste. See the [Docker section](docker.md) for the
|
|
details.
|
|
|
|
## Shared Directory
|
|
|
|
Druid configuration includes not just the config files, but also items
|
|
on the Druid class path. These are provided via a `shared` directory mounted
|
|
into the container at `/shared`.
|
|
The shared directory is built in the `target/<category>` folder for each test
|
|
category.
|
|
|
|
The `launch.sh` script fills in a number of implicit configuration items:
|
|
|
|
| Item | Description |
|
|
| ---- | ----------- |
|
|
| Heap dump path | Set to `${SHARED}/logs/<instance>` |
|
|
| Log4J config | Optional at `${SHARED}/conf/log4j.xml` |
|
|
| Hadoop config | Optional at `${SHARED}/hadoop-xml` |
|
|
| Extra libraries | Optional at `${SHARED}/lib` |
|
|
| Extra resources | Optional at `${SHARED}/resources` |
|
|
|
|
`${SHARED}/resources` is the place to put things like a custom `log4j2.xml`
|
|
file.
|
|
|
|
## Security Setup
|
|
|
|
Tests can run with or without security enabled. (Security setup is a work in progress,
|
|
the prior integration tests enabled security for all tests.)
|
|
|
|
* `auth.env` - Additional definitions to create a secure cluster. Also requires that
|
|
the client certificates be created. Add this to tests which test security.
|