Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-17 07:25:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Instructions to run integration tests against quickstart (#9560)

Browse Source 
* Instructions to run integration tests against quickstart

* Address review comments

* actually exclude the test group

* Revert "actually exclude the test group"

This reverts commit 66f366409ee2e51af1ec0285276309bb6acc15c4.

* update comment
This commit is contained in:
Suneet Saldanha 2020-03-26 13:22:53 -07:00 committed by GitHub
parent 55c08e0746
commit e6e2836b0e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 66 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
integration-tests/README.md
View File

@ -74,31 +74,38 @@ Running Tests Using A Quickstart Cluster
When writing integration tests, it can be helpful to test against a quickstart
cluster so that you can set up remote debugging with in your developer
environment. This section walks you through setting up the integration tests
so that it can run against a quickstart cluster running on your development
so that it can run against a [quickstart cluster](../docs/tutorials/index.md#getting-started) running on your development
machine.
Note that not all features run by default on a quickstart cluster, so it may
not make sense to run the entire test suite against this configuration.
> NOTE: Not all features run by default on a quickstart cluster, so it may not make sense to run the entire test suite against this configuration.
> NOTE: Quickstart does not run with ssl, so to trick the integration tests we specify the `*_tls_url` in the config to be the same as the http url.
Make sure you have at least 6GB of memory available before you run the tests.
The tests rely on files in the test/resources folder to exist under the path /resources,
so create a symlink to make them available
so create a symlink to make them available:
```
ln -s ${DRUID_HOME}/integration-tests/src/test/resources /resources
```
Set the cluster config file environment variable to the quickstart config
Set the cluster config file environment variable to the quickstart config:
```
export CONFIG_FILE=${DRUID_HOME}/integration-tests/quickstart-it.json
```
Note that quickstart does not run with ssl, so to trick the integration tests
we specify the `*_tls_url` in the config to be the same as the http url
Then run the tests using a command similar to
The test group `quickstart-compatible` has tests that have been verified to work against the quickstart cluster.
There may be more tests that work, if you find that they do, please mark it as quickstart-compatible
(TestNGGroup#QUICKSTART_COMPATIBLE) and open a PR.
If you find some integration tests do not work, look at the docker files to see what setup they do. You may need to
do similar steps to get the test to work.
Then run the tests using a command similar to:
```
mvn verify -P int-tests-config-file -Dit.test=<test_name>
# Run all integration tests that have been verified to work against a quickstart cluster.
mvn verify -P int-tests-config-file -Dgroups=quickstart-compatible
```
Running Tests Using A Configuration File for Any Cluster

68
integration-tests/src/test/java/org/apache/druid/tests/TestNGGroup.java
View File

@ -33,29 +33,59 @@ public class TestNGGroup
// This group can only be run individually using -Dgroups=query since it requires specific test data setup.
public static final String QUERY = "query";
public static final String REALTIME_INDEX = "realtime-index";
// This group can only be run individually using -Dgroups=security since it requires specific test data setup.
/**
* This group can only be run individually using -Dgroups=security since it requires specific test data setup.
*/
public static final String SECURITY = "security";
// This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
// The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
// See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
/**
* This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
* The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
* See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
*/
public static final String S3_DEEP_STORAGE = "s3-deep-storage";
// This group is not part of CI. To run this group, gcs configs/credentials for your gcs must be provided in a file.
// The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
// See integration-tests/docker/environment-configs/override-examples/gcs for env vars to provide.
// The path to the folder that contains your GOOGLE_APPLICATION_CREDENTIALS file must also be pass
// to mvn with -Dresource.file.dir.path=<PATH_TO_FOLDER>
/**
* This group is not part of CI. To run this group, gcs configs/credentials for your gcs must be provided in a file.
* The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
* See integration-tests/docker/environment-configs/override-examples/gcs for env vars to provide.
* The path to the folder that contains your GOOGLE_APPLICATION_CREDENTIALS file must also be pass
* to mvn with -Dresource.file.dir.path=<PATH_TO_FOLDER>
*/
public static final String GCS_DEEP_STORAGE = "gcs-deep-storage";
// This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
// The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
// See integration-tests/docker/environment-configs/override-examples/azures for env vars to provide.
/**
* This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
* The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
* See integration-tests/docker/environment-configs/override-examples/azures for env vars to provide.
*/
public static final String AZURE_DEEP_STORAGE = "azure-deep-storage";
// This group is not part of CI. To run this group, hadoop configs must be provided in a file. The path of the file
// must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
// See integration-tests/docker/environment-configs/override-examples/hdfs for env vars to provide.
// Additionally, hadoop docker must be started by passing -Dstart.hadoop.docker=true to mvn.
/**
* This group is not part of CI. To run this group, hadoop configs must be provided in a file. The path of the file
* must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
* See integration-tests/docker/environment-configs/override-examples/hdfs for env vars to provide.
* Additionally, hadoop docker must be started by passing -Dstart.hadoop.docker=true to mvn.
*/
public static final String HDFS_DEEP_STORAGE = "hdfs-deep-storage";
// This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
// The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
// See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
/**
* This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
* The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
* See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
*/
public static final String S3_INGESTION = "s3-ingestion";
/**
* This group is not part of CI explicitly. It allows you to run all the tests that have been tested with
* against a quickstart deployment of Druid using the instructions in this project's README.
* No tests should belong exclusively to this group. It is only meant to be a helper group to run tests against
* a local quickstart deployment.
*
* As you run tests in your environment, mark the tests as quickstart compatible (see
* {@link org.apache.druid.tests.indexer.ITIndexerTest}) and add any additional instructions that were needed to
* get the tests to work to this project's README.
*/
public static final String QUICKSTART_COMPATIBLE = "quickstart-compatible";
}

2
integration-tests/src/test/java/org/apache/druid/tests/indexer/ITCompactionTaskTest.java
View File

@ -39,7 +39,7 @@ import java.lang.reflect.Method;
import java.nio.charset.StandardCharsets;
import java.util.List;
@Test(groups = TestNGGroup.OTHER_INDEX)
@Test(groups = {TestNGGroup.OTHER_INDEX, TestNGGroup.QUICKSTART_COMPATIBLE})
@Guice(moduleFactory = DruidTestModuleFactory.class)
public class ITCompactionTaskTest extends AbstractIndexerTest
{

2
integration-tests/src/test/java/org/apache/druid/tests/indexer/ITIndexerTest.java
View File

@ -26,7 +26,7 @@ import org.testng.annotations.Test;
import java.io.Closeable;
@Test(groups = TestNGGroup.BATCH_INDEX)
@Test(groups = {TestNGGroup.BATCH_INDEX, TestNGGroup.QUICKSTART_COMPATIBLE})
@Guice(moduleFactory = DruidTestModuleFactory.class)
public class ITIndexerTest extends AbstractITBatchIndexTest
{