diff --git a/TESTING.asciidoc b/TESTING.asciidoc index b99768a5e2d..72c4ba14530 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -3,9 +3,18 @@ [partintro] -Elasticsearch uses jUnit for testing, it also uses randomness in the +OpenSearch uses jUnit for testing, it also uses randomness in the tests, that can be set using a seed, the following is a cheatsheet of -options for running the tests for ES. +options for running the tests for OpenSearch. + +== Requirements + +You will need the following pieces of software to run these tests: +- Docker & Docker Compose +- Vagrant +- JDK 14 +- Gradle + == Creating packages @@ -16,18 +25,18 @@ following: ./gradlew assemble ----------------------------- -To create a platform-specific build including the x-pack modules, use the +To create a platform-specific build, use the following depending on your operating system: ----------------------------- -./gradlew :distribution:archives:linux-tar:assemble -./gradlew :distribution:archives:darwin-tar:assemble -./gradlew :distribution:archives:windows-zip:assemble +./gradlew :distribution:archives:oss-linux-tar:assemble +./gradlew :distribution:archives:oss-darwin-tar:assemble +./gradlew :distribution:archives:oss-windows-zip:assemble ----------------------------- -=== Running Elasticsearch from a checkout +=== Running OpenSearch from a checkout -In order to run Elasticsearch from source without building a package, you can +In order to run OpenSearch from source without building a package, you can run it using Gradle: ------------------------------------- @@ -36,7 +45,7 @@ run it using Gradle: ==== Launching and debugging from an IDE -If you want to run Elasticsearch from your IDE, the `./gradlew run` task +If you want to run OpenSearch from your IDE, the `./gradlew run` task supports a remote debugging option: --------------------------------------------------------------------------- @@ -52,8 +61,8 @@ is called "Auto restart" and needs to be checked. In case of Eclipse, "Connectio needs to be configured with a greater value (ie 10 or more). NOTE: If you have imported the project into IntelliJ according to the instructions in -link:/CONTRIBUTING.md#importing-the-project-into-intellij-idea[CONTRIBUTING.md] then a debug run configuration -named "Debug Elasticsearch" will be created for you and configured appropriately. +link:/DEVELOPER_GUIDE.md#importing-the-project-into-intellij-idea[DEVELOPER_GUIDE.md] then a debug run configuration +named "Debug OpenSearch" will be created for you and configured appropriately. ==== Distribution @@ -73,8 +82,8 @@ To for example start the open source distribution: - In order to use a custom data directory: `--data-dir=/tmp/foo` - In order to preserve data in between executions: `--preserve-data` - In order to remotely attach a debugger to the process: `--debug-jvm` -- In order to set a different keystore password: `--keystore-password` -- In order to set an Elasticsearch setting, provide a setting with the following prefix: `-Dtests.es.` +- In order to set a different keystore password: `--keystore-password yourpassword` +- In order to set an OpenSearch setting, provide a setting with the following prefix: `-Dtests.opensearch.` === Test case filtering. @@ -84,14 +93,14 @@ To for example start the open source distribution: Run a single test case (variants) ---------------------------------------------------------- -./gradlew test -Dtests.class=org.elasticsearch.package.ClassName +./gradlew test -Dtests.class=org.opensearch.package.ClassName ./gradlew test "-Dtests.class=*.ClassName" ---------------------------------------------------------- Run all tests in a package and its sub-packages ---------------------------------------------------- -./gradlew test "-Dtests.class=org.elasticsearch.package.*" +./gradlew test "-Dtests.class=org.opensearch.package.*" ---------------------------------------------------- Run any test methods that contain 'esi' (like: ...r*esi*ze...) @@ -198,7 +207,7 @@ Run all tests without stopping on errors (inspect log files). ./gradlew test -Dtests.haltonfailure=false ----------------------------------------- -Run more verbose output (slave JVM parameters, etc.). +Run more verbose output (JVM parameters, etc.). ---------------------- ./gradlew test -verbose @@ -211,10 +220,10 @@ tests (note the exclamation mark). ./gradlew test -Dtests.timeoutSuite=5000! ... --------------------------------------- -Change the logging level of ES (not Gradle) +Change the logging level of OpenSearch (not Gradle) -------------------------------- -./gradlew test -Dtests.es.logger.level=DEBUG +./gradlew test -Dtests.opensearch.logger.level=DEBUG -------------------------------- Print all the logging output from the test runs to the commandline @@ -271,11 +280,10 @@ The REST layer is tested through specific tests that are executed against a cluster that is configured and initialized via Gradle. The tests themselves can be written in either Java or with a YAML based DSL. -YAML based REST tests should be preferred since these are shared between all -the elasticsearch official clients. The YAML based tests describe the +YAML based REST tests should be preferred since these are shared between clients. The YAML based tests describe the operations to be executed and the obtained results that need to be tested. -The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification] +The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[OpenSearch REST API JSON specification] In order to run the YAML tests, the relevant API specification needs to be on the test classpath. Any gradle project that has support for REST tests will get the primary API on it's class path. However, to better support @@ -302,7 +310,7 @@ A specific test case can be run with the following command: --------------------------------------------------------------------------- ./gradlew ':rest-api-spec:yamlRestTest' \ - --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT" \ + --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" \ -Dtests.method="test {p0=cat.segments/10_basic/Help}" --------------------------------------------------------------------------- @@ -328,7 +336,7 @@ A specific test case can be run with the following syntax (fqn.test {params}): --------------------------------------------------------------------------- ./gradlew ':modules:mapper-extras:javaRestTest' \ - --tests "org.elasticsearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}" + --tests "org.opensearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}" --------------------------------------------------------------------------- yamlRestTest's and javaRestTest's are easy to identify, since they are found in a @@ -347,7 +355,7 @@ the http addresses of the nodes so that REST requests can be sent to them. == Testing packaging The packaging tests use Vagrant virtual machines or cloud instances to verify -that installing and running Elasticsearch distributions works correctly on +that installing and running OpenSearch distributions works correctly on supported operating systems. These tests should really only be run on ephemeral systems because they're destructive; that is, these tests install and remove packages and freely modify system settings, so you will probably regret it if @@ -361,7 +369,7 @@ runs the actual "destructive" test classes. . Install Virtual Box and Vagrant. + . (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze -a bit more performance out of the process: +a bit more performance out of the process (Note: as of 2021, vagrant-cachier is unmaintained): + -------------------------------------- vagrant plugin install vagrant-cachier @@ -385,7 +393,7 @@ If you want a quick test of the tarball and RPM packagings for Centos 7, you would run: + ------------------------------------------------------------------------------------------------- -./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive +./gradlew :qa:os:centos-7:distroTest.rpm :qa:os:centos-7:distroTest.linux-archive ------------------------------------------------------------------------------------------------- Note that if you interrupt Gradle in the middle of running these tasks, any boxes started @@ -468,11 +476,6 @@ vagrant destroy -f ubuntu-1604 && vagrant up ubuntu-1604 --provider virtualbox The whole process takes a minute and a half on a modern laptop, two and a half without vagrant-cachier. -Its possible that some downloads will fail and it'll be impossible to restart -them. This is a bug in vagrant. See the instructions here for how to work -around it: -https://github.com/mitchellh/vagrant/issues/4479 - Some vagrant commands will work on all VMs at once: ------------------ @@ -487,20 +490,20 @@ that'd consume a ton of ram. Because our packaging tests are capable of testing many combinations of OS (e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.), -Elasticsearch distribution type (e.g., default or OSS), and so forth, it's +OpenSearch distribution type (e.g. OSS), and so forth, it's faster to develop against smaller subsets of the tests. For example, to run tests for the default archive distribution on Fedora 28: ----------------------------------------------------------- -./gradlew :qa:os:fedora-28:distroTest.default-linux-archive +./gradlew :qa:os:fedora-28:distroTest.linux-archive ----------------------------------------------------------- These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like non-OS tests can. For example: ----------------------------------------------------------- -./gradlew :qa:os:fedora-28:distroTest.default-linux-archive \ - --tests "com.elasticsearch.packaging.test.ArchiveTests" +./gradlew :qa:os:fedora-28:distroTest.linux-archive \ + --tests "com.opensearch.packaging.test.ArchiveTests" ----------------------------------------------------------- == Testing backwards compatibility @@ -523,13 +526,13 @@ Use -Dtest.class and -Dtests.method to run a specific bwcTest test. For example to run a specific tests from the x-pack rolling upgrade from 7.7.0: ------------------------------------------------- ./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \ - -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \ + -Dtests.class=org.opensearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \ -Dtests.method="test {p0=*/40_ml_datafeed_crud/*}" ------------------------------------------------- Tests are ran for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. -See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection] +See link:./buildSrc/src/main/java/org/opensearch/gradle/VersionCollection.java[VersionCollection] and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle] for more information. @@ -573,15 +576,15 @@ fetching the latest from the remote. There are multiple base classes for tests: -* **`ESTestCase`**: The base class of all tests. It is typically extended +* **`OpenSearchTestCase`**: The base class of all tests. It is typically extended directly by unit tests. -* **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a +* **`OpenSearchSingleNodeTestCase`**: This test case sets up a cluster that has a single node. -* **`ESIntegTestCase`**: An integration test case that creates a cluster that +* **`OpenSearchIntegTestCase`**: An integration test case that creates a cluster that might have multiple nodes. -* **`ESRestTestCase`**: An integration tests that interacts with an external +* **`OpenSearchRestTestCase`**: An integration tests that interacts with an external cluster via the REST API. This is used for Java based REST tests. -* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run +* **`OpenSearchClientYamlSuiteTestCase` **: A subclass of `OpenSearchRestTestCase` used to run YAML based REST tests. === Good practices @@ -593,17 +596,17 @@ they are simpler to understand, more likely to reproduce, and unlikely to be affected by changes that are unrelated to the piece of functionality that is being tested. -The reason why `ESSingleNodeTestCase` exists is that all our components used to +The reason why `OpenSearchSingleNodeTestCase` exists is that all our components used to be very hard to set up in isolation, which had led us to having a number of -integration tests but close to no unit tests. `ESSingleNodeTestCase` is a +integration tests but close to no unit tests. `OpenSearchSingleNodeTestCase` is a workaround for this issue which provides an easy way to spin up a node and get access to components that are hard to instantiate like `IndicesService`. Whenever practical, you should prefer unit tests. -Many tests extend `ESIntegTestCase`, mostly because this is how most tests used +Many tests extend `OpenSearchIntegTestCase`, mostly because this is how most tests used to work in the early days of Elasticsearch. However the complexity of these tests tends to make them hard to debug. Whenever the functionality that is -being tested isn't intimately dependent on how Elasticsearch behaves as a +being tested isn't intimately dependent on how OpenSearch behaves as a cluster, it is recommended to write unit tests or REST tests instead. In short, most new functionality should come with unit tests, and optionally @@ -651,14 +654,14 @@ care. == Test coverage analysis -Generating test coverage reports for Elasticsearch is currently not possible through Gradle. +Generating test coverage reports for OpenSearch is currently not possible through Gradle. However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able to do the same using the EclEmma plugin. -Test coverage reporting used to be possible with JaCoCo when Elasticsearch was using Maven +Test coverage reporting used to be possible with JaCoCo when OpenSearch was using Maven as its build system. Since the switch to Gradle though, this is no longer possible, seeing as -the code currently used to build Elasticsearch does not allow JaCoCo to recognize its tests. +the code currently used to build OpenSearch does not allow JaCoCo to recognize its tests. For more information on this, see the discussion in https://github.com/elastic/elasticsearch/issues/28867[issue #28867]. --------------------------------------------------------------------------- @@ -666,10 +669,10 @@ For more information on this, see the discussion in https://github.com/elastic/e Read your IDE documentation for how to attach a debugger to a JVM process. == Building with extra plugins -Additional plugins may be built alongside elasticsearch, where their -dependency on elasticsearch will be substituted with the local elasticsearch -build. To add your plugin, create a directory called elasticsearch-extra as -a sibling of elasticsearch. Checkout your plugin underneath elasticsearch-extra +Additional plugins may be built alongside OpenSearch, where their +dependency on OpenSearch will be substituted with the local OpenSearch +build. To add your plugin, create a directory called opensearch-extra as +a sibling of OpenSearch. Checkout your plugin underneath opensearch-extra and the build will automatically pick it up. You can verify the plugin is included as part of the build by checking the projects of the build. @@ -686,23 +689,15 @@ takes a long time. To fix this, make sure you have your computer name (as returned by `hostname`) inside `/etc/hosts`, e.g.: .... -127.0.0.1 localhost ElasticMBP.local +127.0.0.1 localhost OpenSearchMBP.local 255.255.255.255 broadcasthost -::1 localhost ElasticMBP.local` +::1 localhost OpenSearchMBP.local` .... == Benchmarking -For changes that might affect the performance characteristics of Elasticsearch -you should also run macrobenchmarks. We maintain a macrobenchmarking tool +For changes that might affect the performance characteristics of OpenSearch +you should also run macrobenchmarks. There is also a macrobenchmarking tool called https://github.com/elastic/rally[Rally] -which you can use to measure the performance impact. It comes with a set of -default benchmarks that we also -https://elasticsearch-benchmarks.elastic.co/[run every night]. To get started, +which you can use to measure the performance impact. To get started, please see https://esrally.readthedocs.io/en/stable/[Rally's documentation]. - -== Test doc builds - -The Elasticsearch docs are in AsciiDoc format. You can test and build the docs -locally using the Elasticsearch documentation build process. See -https://github.com/elastic/docs.