TESTING.asciidoc udpate (#390)
* updated TESTING to reflect new project Signed-off-by: Kyle Davis <kyledvs@amazon.com> * fixing unresolved issue based on feedback Signed-off-by: Kyle Davis <kyledvs@amazon.com> * fixed remaining ES and default- references Signed-off-by: Kyle Davis <kyledvs@amazon.com> * made references to clients less specific Signed-off-by: Kyle Davis <kyledvs@amazon.com>
This commit is contained in:
Kyle J. Davis
GitHub
parent
f1f6e7109f
commit
a7776b7546
127
TESTING.asciidoc
127
TESTING.asciidoc
|
|
@ -3,9 +3,18 @@
|
|||
|
||||
[partintro]
|
||||
|
||||
Elasticsearch uses jUnit for testing, it also uses randomness in the
|
||||
OpenSearch uses jUnit for testing, it also uses randomness in the
|
||||
tests, that can be set using a seed, the following is a cheatsheet of
|
||||
options for running the tests for ES.
|
||||
options for running the tests for OpenSearch.
|
||||
|
||||
== Requirements
|
||||
|
||||
You will need the following pieces of software to run these tests:
|
||||
- Docker & Docker Compose
|
||||
- Vagrant
|
||||
- JDK 14
|
||||
- Gradle
|
||||
|
||||
|
||||
== Creating packages
|
||||
|
||||
|
|
@ -16,18 +25,18 @@ following:
|
|||
./gradlew assemble
|
||||
-----------------------------
|
||||
|
||||
To create a platform-specific build including the x-pack modules, use the
|
||||
To create a platform-specific build, use the
|
||||
following depending on your operating system:
|
||||
|
||||
-----------------------------
|
||||
./gradlew :distribution:archives:linux-tar:assemble
|
||||
./gradlew :distribution:archives:darwin-tar:assemble
|
||||
./gradlew :distribution:archives:windows-zip:assemble
|
||||
./gradlew :distribution:archives:oss-linux-tar:assemble
|
||||
./gradlew :distribution:archives:oss-darwin-tar:assemble
|
||||
./gradlew :distribution:archives:oss-windows-zip:assemble
|
||||
-----------------------------
|
||||
|
||||
=== Running Elasticsearch from a checkout
|
||||
=== Running OpenSearch from a checkout
|
||||
|
||||
In order to run Elasticsearch from source without building a package, you can
|
||||
In order to run OpenSearch from source without building a package, you can
|
||||
run it using Gradle:
|
||||
|
||||
-------------------------------------
|
||||
|
|
@ -36,7 +45,7 @@ run it using Gradle:
|
|||
|
||||
==== Launching and debugging from an IDE
|
||||
|
||||
If you want to run Elasticsearch from your IDE, the `./gradlew run` task
|
||||
If you want to run OpenSearch from your IDE, the `./gradlew run` task
|
||||
supports a remote debugging option:
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
|
@ -52,8 +61,8 @@ is called "Auto restart" and needs to be checked. In case of Eclipse, "Connectio
|
|||
needs to be configured with a greater value (ie 10 or more).
|
||||
|
||||
NOTE: If you have imported the project into IntelliJ according to the instructions in
|
||||
link:/CONTRIBUTING.md#importing-the-project-into-intellij-idea[CONTRIBUTING.md] then a debug run configuration
|
||||
named "Debug Elasticsearch" will be created for you and configured appropriately.
|
||||
link:/DEVELOPER_GUIDE.md#importing-the-project-into-intellij-idea[DEVELOPER_GUIDE.md] then a debug run configuration
|
||||
named "Debug OpenSearch" will be created for you and configured appropriately.
|
||||
|
||||
==== Distribution
|
||||
|
||||
|
|
@ -73,8 +82,8 @@ To for example start the open source distribution:
|
|||
- In order to use a custom data directory: `--data-dir=/tmp/foo`
|
||||
- In order to preserve data in between executions: `--preserve-data`
|
||||
- In order to remotely attach a debugger to the process: `--debug-jvm`
|
||||
- In order to set a different keystore password: `--keystore-password`
|
||||
- In order to set an Elasticsearch setting, provide a setting with the following prefix: `-Dtests.es.`
|
||||
- In order to set a different keystore password: `--keystore-password yourpassword`
|
||||
- In order to set an OpenSearch setting, provide a setting with the following prefix: `-Dtests.opensearch.`
|
||||
|
||||
=== Test case filtering.
|
||||
|
||||
|
|
@ -84,14 +93,14 @@ To for example start the open source distribution:
|
|||
Run a single test case (variants)
|
||||
|
||||
----------------------------------------------------------
|
||||
./gradlew test -Dtests.class=org.elasticsearch.package.ClassName
|
||||
./gradlew test -Dtests.class=org.opensearch.package.ClassName
|
||||
./gradlew test "-Dtests.class=*.ClassName"
|
||||
----------------------------------------------------------
|
||||
|
||||
Run all tests in a package and its sub-packages
|
||||
|
||||
----------------------------------------------------
|
||||
./gradlew test "-Dtests.class=org.elasticsearch.package.*"
|
||||
./gradlew test "-Dtests.class=org.opensearch.package.*"
|
||||
----------------------------------------------------
|
||||
|
||||
Run any test methods that contain 'esi' (like: ...r*esi*ze...)
|
||||
|
|
@ -198,7 +207,7 @@ Run all tests without stopping on errors (inspect log files).
|
|||
./gradlew test -Dtests.haltonfailure=false
|
||||
-----------------------------------------
|
||||
|
||||
Run more verbose output (slave JVM parameters, etc.).
|
||||
Run more verbose output (JVM parameters, etc.).
|
||||
|
||||
----------------------
|
||||
./gradlew test -verbose
|
||||
|
|
@ -211,10 +220,10 @@ tests (note the exclamation mark).
|
|||
./gradlew test -Dtests.timeoutSuite=5000! ...
|
||||
---------------------------------------
|
||||
|
||||
Change the logging level of ES (not Gradle)
|
||||
Change the logging level of OpenSearch (not Gradle)
|
||||
|
||||
--------------------------------
|
||||
./gradlew test -Dtests.es.logger.level=DEBUG
|
||||
./gradlew test -Dtests.opensearch.logger.level=DEBUG
|
||||
--------------------------------
|
||||
|
||||
Print all the logging output from the test runs to the commandline
|
||||
|
|
@ -271,11 +280,10 @@ The REST layer is tested through specific tests that are executed against
|
|||
a cluster that is configured and initialized via Gradle. The tests
|
||||
themselves can be written in either Java or with a YAML based DSL.
|
||||
|
||||
YAML based REST tests should be preferred since these are shared between all
|
||||
the elasticsearch official clients. The YAML based tests describe the
|
||||
YAML based REST tests should be preferred since these are shared between clients. The YAML based tests describe the
|
||||
operations to be executed and the obtained results that need to be tested.
|
||||
|
||||
The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification]
|
||||
The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[OpenSearch REST API JSON specification]
|
||||
In order to run the YAML tests, the relevant API specification needs
|
||||
to be on the test classpath. Any gradle project that has support for REST
|
||||
tests will get the primary API on it's class path. However, to better support
|
||||
|
|
@ -302,7 +310,7 @@ A specific test case can be run with the following command:
|
|||
|
||||
---------------------------------------------------------------------------
|
||||
./gradlew ':rest-api-spec:yamlRestTest' \
|
||||
--tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT" \
|
||||
--tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" \
|
||||
-Dtests.method="test {p0=cat.segments/10_basic/Help}"
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
|
|
@ -328,7 +336,7 @@ A specific test case can be run with the following syntax (fqn.test {params}):
|
|||
|
||||
---------------------------------------------------------------------------
|
||||
./gradlew ':modules:mapper-extras:javaRestTest' \
|
||||
--tests "org.elasticsearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}"
|
||||
--tests "org.opensearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}"
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
yamlRestTest's and javaRestTest's are easy to identify, since they are found in a
|
||||
|
|
@ -347,7 +355,7 @@ the http addresses of the nodes so that REST requests can be sent to them.
|
|||
== Testing packaging
|
||||
|
||||
The packaging tests use Vagrant virtual machines or cloud instances to verify
|
||||
that installing and running Elasticsearch distributions works correctly on
|
||||
that installing and running OpenSearch distributions works correctly on
|
||||
supported operating systems. These tests should really only be run on ephemeral
|
||||
systems because they're destructive; that is, these tests install and remove
|
||||
packages and freely modify system settings, so you will probably regret it if
|
||||
|
|
@ -361,7 +369,7 @@ runs the actual "destructive" test classes.
|
|||
. Install Virtual Box and Vagrant.
|
||||
+
|
||||
. (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze
|
||||
a bit more performance out of the process:
|
||||
a bit more performance out of the process (Note: as of 2021, vagrant-cachier is unmaintained):
|
||||
+
|
||||
--------------------------------------
|
||||
vagrant plugin install vagrant-cachier
|
||||
|
|
@ -385,7 +393,7 @@ If you want a quick test of the tarball and RPM packagings for Centos 7, you
|
|||
would run:
|
||||
+
|
||||
-------------------------------------------------------------------------------------------------
|
||||
./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive
|
||||
./gradlew :qa:os:centos-7:distroTest.rpm :qa:os:centos-7:distroTest.linux-archive
|
||||
-------------------------------------------------------------------------------------------------
|
||||
|
||||
Note that if you interrupt Gradle in the middle of running these tasks, any boxes started
|
||||
|
|
@ -468,11 +476,6 @@ vagrant destroy -f ubuntu-1604 && vagrant up ubuntu-1604 --provider virtualbox
|
|||
The whole process takes a minute and a half on a modern laptop, two and a half
|
||||
without vagrant-cachier.
|
||||
|
||||
Its possible that some downloads will fail and it'll be impossible to restart
|
||||
them. This is a bug in vagrant. See the instructions here for how to work
|
||||
around it:
|
||||
https://github.com/mitchellh/vagrant/issues/4479
|
||||
|
||||
Some vagrant commands will work on all VMs at once:
|
||||
|
||||
------------------
|
||||
|
|
@ -487,20 +490,20 @@ that'd consume a ton of ram.
|
|||
|
||||
Because our packaging tests are capable of testing many combinations of OS
|
||||
(e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.),
|
||||
Elasticsearch distribution type (e.g., default or OSS), and so forth, it's
|
||||
OpenSearch distribution type (e.g. OSS), and so forth, it's
|
||||
faster to develop against smaller subsets of the tests. For example, to run
|
||||
tests for the default archive distribution on Fedora 28:
|
||||
|
||||
-----------------------------------------------------------
|
||||
./gradlew :qa:os:fedora-28:distroTest.default-linux-archive
|
||||
./gradlew :qa:os:fedora-28:distroTest.linux-archive
|
||||
-----------------------------------------------------------
|
||||
|
||||
These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like
|
||||
non-OS tests can. For example:
|
||||
|
||||
-----------------------------------------------------------
|
||||
./gradlew :qa:os:fedora-28:distroTest.default-linux-archive \
|
||||
--tests "com.elasticsearch.packaging.test.ArchiveTests"
|
||||
./gradlew :qa:os:fedora-28:distroTest.linux-archive \
|
||||
--tests "com.opensearch.packaging.test.ArchiveTests"
|
||||
-----------------------------------------------------------
|
||||
|
||||
== Testing backwards compatibility
|
||||
|
|
@ -523,13 +526,13 @@ Use -Dtest.class and -Dtests.method to run a specific bwcTest test.
|
|||
For example to run a specific tests from the x-pack rolling upgrade from 7.7.0:
|
||||
-------------------------------------------------
|
||||
./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \
|
||||
-Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
|
||||
-Dtests.class=org.opensearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
|
||||
-Dtests.method="test {p0=*/40_ml_datafeed_crud/*}"
|
||||
-------------------------------------------------
|
||||
|
||||
Tests are ran for versions that are not yet released but with which the current version will be compatible with.
|
||||
These are automatically checked out and built from source.
|
||||
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection]
|
||||
See link:./buildSrc/src/main/java/org/opensearch/gradle/VersionCollection.java[VersionCollection]
|
||||
and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle]
|
||||
for more information.
|
||||
|
||||
|
|
@ -573,15 +576,15 @@ fetching the latest from the remote.
|
|||
|
||||
There are multiple base classes for tests:
|
||||
|
||||
* **`ESTestCase`**: The base class of all tests. It is typically extended
|
||||
* **`OpenSearchTestCase`**: The base class of all tests. It is typically extended
|
||||
directly by unit tests.
|
||||
* **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a
|
||||
* **`OpenSearchSingleNodeTestCase`**: This test case sets up a cluster that has a
|
||||
single node.
|
||||
* **`ESIntegTestCase`**: An integration test case that creates a cluster that
|
||||
* **`OpenSearchIntegTestCase`**: An integration test case that creates a cluster that
|
||||
might have multiple nodes.
|
||||
* **`ESRestTestCase`**: An integration tests that interacts with an external
|
||||
* **`OpenSearchRestTestCase`**: An integration tests that interacts with an external
|
||||
cluster via the REST API. This is used for Java based REST tests.
|
||||
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
|
||||
* **`OpenSearchClientYamlSuiteTestCase` **: A subclass of `OpenSearchRestTestCase` used to run
|
||||
YAML based REST tests.
|
||||
|
||||
=== Good practices
|
||||
|
|
@ -593,17 +596,17 @@ they are simpler to understand, more likely to reproduce, and unlikely to be
|
|||
affected by changes that are unrelated to the piece of functionality that is
|
||||
being tested.
|
||||
|
||||
The reason why `ESSingleNodeTestCase` exists is that all our components used to
|
||||
The reason why `OpenSearchSingleNodeTestCase` exists is that all our components used to
|
||||
be very hard to set up in isolation, which had led us to having a number of
|
||||
integration tests but close to no unit tests. `ESSingleNodeTestCase` is a
|
||||
integration tests but close to no unit tests. `OpenSearchSingleNodeTestCase` is a
|
||||
workaround for this issue which provides an easy way to spin up a node and get
|
||||
access to components that are hard to instantiate like `IndicesService`.
|
||||
Whenever practical, you should prefer unit tests.
|
||||
|
||||
Many tests extend `ESIntegTestCase`, mostly because this is how most tests used
|
||||
Many tests extend `OpenSearchIntegTestCase`, mostly because this is how most tests used
|
||||
to work in the early days of Elasticsearch. However the complexity of these
|
||||
tests tends to make them hard to debug. Whenever the functionality that is
|
||||
being tested isn't intimately dependent on how Elasticsearch behaves as a
|
||||
being tested isn't intimately dependent on how OpenSearch behaves as a
|
||||
cluster, it is recommended to write unit tests or REST tests instead.
|
||||
|
||||
In short, most new functionality should come with unit tests, and optionally
|
||||
|
|
@ -651,14 +654,14 @@ care.
|
|||
|
||||
== Test coverage analysis
|
||||
|
||||
Generating test coverage reports for Elasticsearch is currently not possible through Gradle.
|
||||
Generating test coverage reports for OpenSearch is currently not possible through Gradle.
|
||||
However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage
|
||||
analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able
|
||||
to do the same using the EclEmma plugin.
|
||||
|
||||
Test coverage reporting used to be possible with JaCoCo when Elasticsearch was using Maven
|
||||
Test coverage reporting used to be possible with JaCoCo when OpenSearch was using Maven
|
||||
as its build system. Since the switch to Gradle though, this is no longer possible, seeing as
|
||||
the code currently used to build Elasticsearch does not allow JaCoCo to recognize its tests.
|
||||
the code currently used to build OpenSearch does not allow JaCoCo to recognize its tests.
|
||||
For more information on this, see the discussion in https://github.com/elastic/elasticsearch/issues/28867[issue #28867].
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
|
@ -666,10 +669,10 @@ For more information on this, see the discussion in https://github.com/elastic/e
|
|||
Read your IDE documentation for how to attach a debugger to a JVM process.
|
||||
|
||||
== Building with extra plugins
|
||||
Additional plugins may be built alongside elasticsearch, where their
|
||||
dependency on elasticsearch will be substituted with the local elasticsearch
|
||||
build. To add your plugin, create a directory called elasticsearch-extra as
|
||||
a sibling of elasticsearch. Checkout your plugin underneath elasticsearch-extra
|
||||
Additional plugins may be built alongside OpenSearch, where their
|
||||
dependency on OpenSearch will be substituted with the local OpenSearch
|
||||
build. To add your plugin, create a directory called opensearch-extra as
|
||||
a sibling of OpenSearch. Checkout your plugin underneath opensearch-extra
|
||||
and the build will automatically pick it up. You can verify the plugin is
|
||||
included as part of the build by checking the projects of the build.
|
||||
|
||||
|
|
@ -686,23 +689,15 @@ takes a long time.
|
|||
To fix this, make sure you have your computer name (as returned by `hostname`)
|
||||
inside `/etc/hosts`, e.g.:
|
||||
....
|
||||
127.0.0.1 localhost ElasticMBP.local
|
||||
127.0.0.1 localhost OpenSearchMBP.local
|
||||
255.255.255.255 broadcasthost
|
||||
::1 localhost ElasticMBP.local`
|
||||
::1 localhost OpenSearchMBP.local`
|
||||
....
|
||||
|
||||
== Benchmarking
|
||||
|
||||
For changes that might affect the performance characteristics of Elasticsearch
|
||||
you should also run macrobenchmarks. We maintain a macrobenchmarking tool
|
||||
For changes that might affect the performance characteristics of OpenSearch
|
||||
you should also run macrobenchmarks. There is also a macrobenchmarking tool
|
||||
called https://github.com/elastic/rally[Rally]
|
||||
which you can use to measure the performance impact. It comes with a set of
|
||||
default benchmarks that we also
|
||||
https://elasticsearch-benchmarks.elastic.co/[run every night]. To get started,
|
||||
which you can use to measure the performance impact. To get started,
|
||||
please see https://esrally.readthedocs.io/en/stable/[Rally's documentation].
|
||||
|
||||
== Test doc builds
|
||||
|
||||
The Elasticsearch docs are in AsciiDoc format. You can test and build the docs
|
||||
locally using the Elasticsearch documentation build process. See
|
||||
https://github.com/elastic/docs.
|
||||
|
|
|
|||
Loading…
Reference in New Issue