honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

TESTING.asciidoc udpate (#390) 

* updated TESTING to reflect new project

Signed-off-by: Kyle Davis <kyledvs@amazon.com>

* fixing unresolved issue based on feedback

Signed-off-by: Kyle Davis <kyledvs@amazon.com>

* fixed remaining  ES and default- references

Signed-off-by: Kyle Davis <kyledvs@amazon.com>

* made references to clients less specific

Signed-off-by: Kyle Davis <kyledvs@amazon.com>
This commit is contained in:
Kyle J. Davis 2021-03-19 17:47:35 -06:00 committed by GitHub
parent f1f6e7109f
commit a7776b7546
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 61 additions and 66 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

127
TESTING.asciidoc
View File

@ -3,9 +3,18 @@
[partintro] [partintro]
Elasticsearch uses jUnit for testing, it also uses randomness in the OpenSearch uses jUnit for testing, it also uses randomness in the
tests, that can be set using a seed, the following is a cheatsheet of tests, that can be set using a seed, the following is a cheatsheet of
options for running the tests for ES. options for running the tests for OpenSearch.
== Requirements
You will need the following pieces of software to run these tests:
- Docker & Docker Compose
- Vagrant
- JDK 14
- Gradle
== Creating packages == Creating packages
@ -16,18 +25,18 @@ following:
./gradlew assemble ./gradlew assemble
----------------------------- -----------------------------
To create a platform-specific build including the x-pack modules, use the To create a platform-specific build, use the
following depending on your operating system: following depending on your operating system:
----------------------------- -----------------------------
./gradlew :distribution:archives:linux-tar:assemble ./gradlew :distribution:archives:oss-linux-tar:assemble
./gradlew :distribution:archives:darwin-tar:assemble ./gradlew :distribution:archives:oss-darwin-tar:assemble
./gradlew :distribution:archives:windows-zip:assemble ./gradlew :distribution:archives:oss-windows-zip:assemble
----------------------------- -----------------------------
=== Running Elasticsearch from a checkout === Running OpenSearch from a checkout
In order to run Elasticsearch from source without building a package, you can In order to run OpenSearch from source without building a package, you can
run it using Gradle: run it using Gradle:
------------------------------------- -------------------------------------
@ -36,7 +45,7 @@ run it using Gradle:
==== Launching and debugging from an IDE ==== Launching and debugging from an IDE
If you want to run Elasticsearch from your IDE, the `./gradlew run` task If you want to run OpenSearch from your IDE, the `./gradlew run` task
supports a remote debugging option: supports a remote debugging option:
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
@ -52,8 +61,8 @@ is called "Auto restart" and needs to be checked. In case of Eclipse, "Connectio
needs to be configured with a greater value (ie 10 or more). needs to be configured with a greater value (ie 10 or more).
NOTE: If you have imported the project into IntelliJ according to the instructions in NOTE: If you have imported the project into IntelliJ according to the instructions in
link:/CONTRIBUTING.md#importing-the-project-into-intellij-idea[CONTRIBUTING.md] then a debug run configuration link:/DEVELOPER_GUIDE.md#importing-the-project-into-intellij-idea[DEVELOPER_GUIDE.md] then a debug run configuration
named "Debug Elasticsearch" will be created for you and configured appropriately. named "Debug OpenSearch" will be created for you and configured appropriately.
==== Distribution ==== Distribution
@ -73,8 +82,8 @@ To for example start the open source distribution:
- In order to use a custom data directory: `--data-dir=/tmp/foo` - In order to use a custom data directory: `--data-dir=/tmp/foo`
- In order to preserve data in between executions: `--preserve-data` - In order to preserve data in between executions: `--preserve-data`
- In order to remotely attach a debugger to the process: `--debug-jvm` - In order to remotely attach a debugger to the process: `--debug-jvm`
- In order to set a different keystore password: `--keystore-password` - In order to set a different keystore password: `--keystore-password yourpassword`
- In order to set an Elasticsearch setting, provide a setting with the following prefix: `-Dtests.es.` - In order to set an OpenSearch setting, provide a setting with the following prefix: `-Dtests.opensearch.`
=== Test case filtering. === Test case filtering.
@ -84,14 +93,14 @@ To for example start the open source distribution:
Run a single test case (variants) Run a single test case (variants)
---------------------------------------------------------- ----------------------------------------------------------
./gradlew test -Dtests.class=org.elasticsearch.package.ClassName ./gradlew test -Dtests.class=org.opensearch.package.ClassName
./gradlew test "-Dtests.class=*.ClassName" ./gradlew test "-Dtests.class=*.ClassName"
---------------------------------------------------------- ----------------------------------------------------------
Run all tests in a package and its sub-packages Run all tests in a package and its sub-packages
---------------------------------------------------- ----------------------------------------------------
./gradlew test "-Dtests.class=org.elasticsearch.package.*" ./gradlew test "-Dtests.class=org.opensearch.package.*"
---------------------------------------------------- ----------------------------------------------------
Run any test methods that contain 'esi' (like: ...r*esi*ze...) Run any test methods that contain 'esi' (like: ...r*esi*ze...)
@ -198,7 +207,7 @@ Run all tests without stopping on errors (inspect log files).
./gradlew test -Dtests.haltonfailure=false ./gradlew test -Dtests.haltonfailure=false
----------------------------------------- -----------------------------------------
Run more verbose output (slave JVM parameters, etc.). Run more verbose output (JVM parameters, etc.).
---------------------- ----------------------
./gradlew test -verbose ./gradlew test -verbose
@ -211,10 +220,10 @@ tests (note the exclamation mark).
./gradlew test -Dtests.timeoutSuite=5000! ... ./gradlew test -Dtests.timeoutSuite=5000! ...
--------------------------------------- ---------------------------------------
Change the logging level of ES (not Gradle) Change the logging level of OpenSearch (not Gradle)
-------------------------------- --------------------------------
./gradlew test -Dtests.es.logger.level=DEBUG ./gradlew test -Dtests.opensearch.logger.level=DEBUG
-------------------------------- --------------------------------
Print all the logging output from the test runs to the commandline Print all the logging output from the test runs to the commandline
@ -271,11 +280,10 @@ The REST layer is tested through specific tests that are executed against
a cluster that is configured and initialized via Gradle. The tests a cluster that is configured and initialized via Gradle. The tests
themselves can be written in either Java or with a YAML based DSL. themselves can be written in either Java or with a YAML based DSL.
YAML based REST tests should be preferred since these are shared between all YAML based REST tests should be preferred since these are shared between clients. The YAML based tests describe the
the elasticsearch official clients. The YAML based tests describe the
operations to be executed and the obtained results that need to be tested. operations to be executed and the obtained results that need to be tested.
The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification] The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[OpenSearch REST API JSON specification]
In order to run the YAML tests, the relevant API specification needs In order to run the YAML tests, the relevant API specification needs
to be on the test classpath. Any gradle project that has support for REST to be on the test classpath. Any gradle project that has support for REST
tests will get the primary API on it's class path. However, to better support tests will get the primary API on it's class path. However, to better support
@ -302,7 +310,7 @@ A specific test case can be run with the following command:
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
./gradlew ':rest-api-spec:yamlRestTest' \ ./gradlew ':rest-api-spec:yamlRestTest' \
--tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT" \ --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" \
-Dtests.method="test {p0=cat.segments/10_basic/Help}" -Dtests.method="test {p0=cat.segments/10_basic/Help}"
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
@ -328,7 +336,7 @@ A specific test case can be run with the following syntax (fqn.test {params}):
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
./gradlew ':modules:mapper-extras:javaRestTest' \ ./gradlew ':modules:mapper-extras:javaRestTest' \
--tests "org.elasticsearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}" --tests "org.opensearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}"
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
yamlRestTest's and javaRestTest's are easy to identify, since they are found in a yamlRestTest's and javaRestTest's are easy to identify, since they are found in a
@ -347,7 +355,7 @@ the http addresses of the nodes so that REST requests can be sent to them.
== Testing packaging == Testing packaging
The packaging tests use Vagrant virtual machines or cloud instances to verify The packaging tests use Vagrant virtual machines or cloud instances to verify
that installing and running Elasticsearch distributions works correctly on that installing and running OpenSearch distributions works correctly on
supported operating systems. These tests should really only be run on ephemeral supported operating systems. These tests should really only be run on ephemeral
systems because they're destructive; that is, these tests install and remove systems because they're destructive; that is, these tests install and remove
packages and freely modify system settings, so you will probably regret it if packages and freely modify system settings, so you will probably regret it if
@ -361,7 +369,7 @@ runs the actual "destructive" test classes.
. Install Virtual Box and Vagrant. . Install Virtual Box and Vagrant.
+ +
. (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze . (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze
a bit more performance out of the process: a bit more performance out of the process (Note: as of 2021, vagrant-cachier is unmaintained):
+ +
-------------------------------------- --------------------------------------
vagrant plugin install vagrant-cachier vagrant plugin install vagrant-cachier
@ -385,7 +393,7 @@ If you want a quick test of the tarball and RPM packagings for Centos 7, you
would run: would run:
+ +
------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------
./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive ./gradlew :qa:os:centos-7:distroTest.rpm :qa:os:centos-7:distroTest.linux-archive
------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------
Note that if you interrupt Gradle in the middle of running these tasks, any boxes started Note that if you interrupt Gradle in the middle of running these tasks, any boxes started
@ -468,11 +476,6 @@ vagrant destroy -f ubuntu-1604 && vagrant up ubuntu-1604 --provider virtualbox
The whole process takes a minute and a half on a modern laptop, two and a half The whole process takes a minute and a half on a modern laptop, two and a half
without vagrant-cachier. without vagrant-cachier.
Its possible that some downloads will fail and it'll be impossible to restart
them. This is a bug in vagrant. See the instructions here for how to work
around it:
https://github.com/mitchellh/vagrant/issues/4479
Some vagrant commands will work on all VMs at once: Some vagrant commands will work on all VMs at once:
------------------ ------------------
@ -487,20 +490,20 @@ that'd consume a ton of ram.
Because our packaging tests are capable of testing many combinations of OS Because our packaging tests are capable of testing many combinations of OS
(e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.), (e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.),
Elasticsearch distribution type (e.g., default or OSS), and so forth, it's OpenSearch distribution type (e.g. OSS), and so forth, it's
faster to develop against smaller subsets of the tests. For example, to run faster to develop against smaller subsets of the tests. For example, to run
tests for the default archive distribution on Fedora 28: tests for the default archive distribution on Fedora 28:
----------------------------------------------------------- -----------------------------------------------------------
./gradlew :qa:os:fedora-28:distroTest.default-linux-archive ./gradlew :qa:os:fedora-28:distroTest.linux-archive
----------------------------------------------------------- -----------------------------------------------------------
These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like
non-OS tests can. For example: non-OS tests can. For example:
----------------------------------------------------------- -----------------------------------------------------------
./gradlew :qa:os:fedora-28:distroTest.default-linux-archive \ ./gradlew :qa:os:fedora-28:distroTest.linux-archive \
--tests "com.elasticsearch.packaging.test.ArchiveTests" --tests "com.opensearch.packaging.test.ArchiveTests"
----------------------------------------------------------- -----------------------------------------------------------
== Testing backwards compatibility == Testing backwards compatibility
@ -523,13 +526,13 @@ Use -Dtest.class and -Dtests.method to run a specific bwcTest test.
For example to run a specific tests from the x-pack rolling upgrade from 7.7.0: For example to run a specific tests from the x-pack rolling upgrade from 7.7.0:
------------------------------------------------- -------------------------------------------------
./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \ ./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \
-Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \ -Dtests.class=org.opensearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
-Dtests.method="test {p0=*/40_ml_datafeed_crud/*}" -Dtests.method="test {p0=*/40_ml_datafeed_crud/*}"
------------------------------------------------- -------------------------------------------------
Tests are ran for versions that are not yet released but with which the current version will be compatible with. Tests are ran for versions that are not yet released but with which the current version will be compatible with.
These are automatically checked out and built from source. These are automatically checked out and built from source.
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection] See link:./buildSrc/src/main/java/org/opensearch/gradle/VersionCollection.java[VersionCollection]
and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle] and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle]
for more information. for more information.
@ -573,15 +576,15 @@ fetching the latest from the remote.
There are multiple base classes for tests: There are multiple base classes for tests:
* **`ESTestCase`**: The base class of all tests. It is typically extended * **`OpenSearchTestCase`**: The base class of all tests. It is typically extended
directly by unit tests. directly by unit tests.
* **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a * **`OpenSearchSingleNodeTestCase`**: This test case sets up a cluster that has a
single node. single node.
* **`ESIntegTestCase`**: An integration test case that creates a cluster that * **`OpenSearchIntegTestCase`**: An integration test case that creates a cluster that
might have multiple nodes. might have multiple nodes.
* **`ESRestTestCase`**: An integration tests that interacts with an external * **`OpenSearchRestTestCase`**: An integration tests that interacts with an external
cluster via the REST API. This is used for Java based REST tests. cluster via the REST API. This is used for Java based REST tests.
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run * **`OpenSearchClientYamlSuiteTestCase` **: A subclass of `OpenSearchRestTestCase` used to run
YAML based REST tests. YAML based REST tests.
=== Good practices === Good practices
@ -593,17 +596,17 @@ they are simpler to understand, more likely to reproduce, and unlikely to be
affected by changes that are unrelated to the piece of functionality that is affected by changes that are unrelated to the piece of functionality that is
being tested. being tested.
The reason why `ESSingleNodeTestCase` exists is that all our components used to The reason why `OpenSearchSingleNodeTestCase` exists is that all our components used to
be very hard to set up in isolation, which had led us to having a number of be very hard to set up in isolation, which had led us to having a number of
integration tests but close to no unit tests. `ESSingleNodeTestCase` is a integration tests but close to no unit tests. `OpenSearchSingleNodeTestCase` is a
workaround for this issue which provides an easy way to spin up a node and get workaround for this issue which provides an easy way to spin up a node and get
access to components that are hard to instantiate like `IndicesService`. access to components that are hard to instantiate like `IndicesService`.
Whenever practical, you should prefer unit tests. Whenever practical, you should prefer unit tests.
Many tests extend `ESIntegTestCase`, mostly because this is how most tests used Many tests extend `OpenSearchIntegTestCase`, mostly because this is how most tests used
to work in the early days of Elasticsearch. However the complexity of these to work in the early days of Elasticsearch. However the complexity of these
tests tends to make them hard to debug. Whenever the functionality that is tests tends to make them hard to debug. Whenever the functionality that is
being tested isn't intimately dependent on how Elasticsearch behaves as a being tested isn't intimately dependent on how OpenSearch behaves as a
cluster, it is recommended to write unit tests or REST tests instead. cluster, it is recommended to write unit tests or REST tests instead.
In short, most new functionality should come with unit tests, and optionally In short, most new functionality should come with unit tests, and optionally
@ -651,14 +654,14 @@ care.
== Test coverage analysis == Test coverage analysis
Generating test coverage reports for Elasticsearch is currently not possible through Gradle. Generating test coverage reports for OpenSearch is currently not possible through Gradle.
However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage
analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able
to do the same using the EclEmma plugin. to do the same using the EclEmma plugin.
Test coverage reporting used to be possible with JaCoCo when Elasticsearch was using Maven Test coverage reporting used to be possible with JaCoCo when OpenSearch was using Maven
as its build system. Since the switch to Gradle though, this is no longer possible, seeing as as its build system. Since the switch to Gradle though, this is no longer possible, seeing as
the code currently used to build Elasticsearch does not allow JaCoCo to recognize its tests. the code currently used to build OpenSearch does not allow JaCoCo to recognize its tests.
For more information on this, see the discussion in https://github.com/elastic/elasticsearch/issues/28867[issue #28867]. For more information on this, see the discussion in https://github.com/elastic/elasticsearch/issues/28867[issue #28867].
--------------------------------------------------------------------------- ---------------------------------------------------------------------------
@ -666,10 +669,10 @@ For more information on this, see the discussion in https://github.com/elastic/e
Read your IDE documentation for how to attach a debugger to a JVM process. Read your IDE documentation for how to attach a debugger to a JVM process.
== Building with extra plugins == Building with extra plugins
Additional plugins may be built alongside elasticsearch, where their Additional plugins may be built alongside OpenSearch, where their
dependency on elasticsearch will be substituted with the local elasticsearch dependency on OpenSearch will be substituted with the local OpenSearch
build. To add your plugin, create a directory called elasticsearch-extra as build. To add your plugin, create a directory called opensearch-extra as
a sibling of elasticsearch. Checkout your plugin underneath elasticsearch-extra a sibling of OpenSearch. Checkout your plugin underneath opensearch-extra
and the build will automatically pick it up. You can verify the plugin is and the build will automatically pick it up. You can verify the plugin is
included as part of the build by checking the projects of the build. included as part of the build by checking the projects of the build.
@ -686,23 +689,15 @@ takes a long time.
To fix this, make sure you have your computer name (as returned by `hostname`) To fix this, make sure you have your computer name (as returned by `hostname`)
inside `/etc/hosts`, e.g.: inside `/etc/hosts`, e.g.:
.... ....
127.0.0.1 localhost ElasticMBP.local 127.0.0.1 localhost OpenSearchMBP.local
255.255.255.255 broadcasthost 255.255.255.255 broadcasthost
::1 localhost ElasticMBP.local` ::1 localhost OpenSearchMBP.local`
.... ....
== Benchmarking == Benchmarking
For changes that might affect the performance characteristics of Elasticsearch For changes that might affect the performance characteristics of OpenSearch
you should also run macrobenchmarks. We maintain a macrobenchmarking tool you should also run macrobenchmarks. There is also a macrobenchmarking tool
called https://github.com/elastic/rally[Rally] called https://github.com/elastic/rally[Rally]
which you can use to measure the performance impact. It comes with a set of which you can use to measure the performance impact. To get started,
default benchmarks that we also
https://elasticsearch-benchmarks.elastic.co/[run every night]. To get started,
please see https://esrally.readthedocs.io/en/stable/[Rally's documentation]. please see https://esrally.readthedocs.io/en/stable/[Rally's documentation].
== Test doc builds
The Elasticsearch docs are in AsciiDoc format. You can test and build the docs
locally using the Elasticsearch documentation build process. See
https://github.com/elastic/docs.