mirror of https://github.com/apache/lucene.git
218 lines
6.8 KiB
Plaintext
218 lines
6.8 KiB
Plaintext
Testing
|
|
=======
|
|
|
|
Examples below assume cwd at the gradlew script in the top directory of
|
|
the project's checkout.
|
|
|
|
|
|
Generic test/ checkup commands
|
|
------------------------------
|
|
|
|
Run all unit tests:
|
|
|
|
gradlew test
|
|
|
|
Run all(*) verification tasks, including tests:
|
|
|
|
gradlew check
|
|
|
|
(*) This step may omit certain validation tasks (errorprone, for example)
|
|
that are slow or require external resources. A pull request runs all
|
|
those locally omitted tasks on the CI (jenkins, gh actions).
|
|
For this reason, it's a good idea to run patches via the CI.
|
|
|
|
Run all verification tasks, excluding tests (-x is gradle's generic task
|
|
exclusion mechanism):
|
|
|
|
gradlew check -x test
|
|
|
|
Run verification for a selected module only:
|
|
|
|
gradlew :lucene:core:check # By full gradle project path
|
|
gradlew -p lucene/core check # By folder designation + task
|
|
|
|
|
|
Randomization
|
|
-------------
|
|
|
|
To run tests with the given starting seed pass 'tests.seed'
|
|
property:
|
|
|
|
gradlew :lucene:misc:test -Ptests.seed=DEADBEEF
|
|
|
|
There are a lot of other test randomization properties
|
|
available. To list them, their defaults and current values
|
|
run the testOpts task against a project that has tests.
|
|
For example:
|
|
|
|
gradlew -p lucene/core testOpts
|
|
|
|
|
|
Filtering
|
|
---------
|
|
|
|
Run tests of lucene-core module:
|
|
|
|
gradlew -p lucene/core test
|
|
|
|
Run a single test case (from a single module). Uses gradle's built-in filtering
|
|
(https://docs.gradle.org/current/userguide/java_testing.html#test_filtering):
|
|
|
|
gradlew -p lucene/core test --tests TestDemo
|
|
|
|
Run all tests in a package:
|
|
|
|
gradlew -p lucene/core test --tests "org.apache.lucene.document.*"
|
|
|
|
Run all test classes/ methods that match this pattern:
|
|
|
|
gradlew -p lucene/core test --tests "*testFeatureMissing*"
|
|
|
|
|
|
Test groups
|
|
-----------
|
|
|
|
Tests can be filtered by an annotation they're marked with.
|
|
Some test group annotations include: @AwaitsFix, @Nightly
|
|
|
|
This uses filtering infrastructure on the *runner* (randomizedtesting),
|
|
not gradle's built-in mechanisms (but it can be combined with "--tests").
|
|
For example, run all lucene-core tests annotated as @Nightly:
|
|
|
|
gradlew -p lucene/core test -Ptests.filter=@Nightly
|
|
|
|
Test group filters can be combined into Boolean expressions:
|
|
|
|
gradlew -p lucene/core test "default and not(@awaitsfix or @nightly)"
|
|
|
|
|
|
Reiteration ("beasting")
|
|
------------------------
|
|
|
|
Multiply each test case N times (this works by repeating the same test
|
|
within the same JVM; it also works in IDEs):
|
|
|
|
gradlew -p lucene/core test --tests TestDemo -Ptests.iters=5
|
|
|
|
Tests tasks will be (by default) re-executed on each invocation because
|
|
we pick a random global tests.seed and because we want them to (force
|
|
the test task to run). If you want to make the 'tests' task obey up-to-date
|
|
gradle rules, fix the seed and turn off up to date trigger:
|
|
|
|
gradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.neverUpToDate=false
|
|
|
|
These properties can be set in your local gradle.properties. To force re-execution
|
|
of tests, even for the same master seed, apply cleanTest task:
|
|
|
|
gradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef
|
|
|
|
The 'tests.iters' option should be sufficient for individual test cases
|
|
and is *much* faster than trying to duplicate re-runs of the entire
|
|
test suites. When it is absolutely needed to re-run an entire suite (because
|
|
of randomization in the static initialization, for example), you can do it
|
|
by running the 'beast' task with 'tests.dups' option:
|
|
|
|
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat
|
|
|
|
Note the filter (--tests) used to narrow down test reiterations to a particular
|
|
class. You can use any filter, including no filter at all, but it rarely makes
|
|
sense (will take ages). By default the test tasks generated by the 'beast' mode
|
|
use a random starting seed for randomization. If you pass an explicit seed, this
|
|
won't be the case (all tasks will use exactly the same starting seed):
|
|
|
|
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef
|
|
|
|
|
|
Verbose mode and debugging
|
|
--------------------------
|
|
|
|
The "tests.verbose" mode switch enables standard streams from tests
|
|
to be dumped directly to the console. Run your verbose tests explicitly
|
|
specifying the project and test task or a fully qualified task path. Example:
|
|
|
|
gradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo"
|
|
|
|
|
|
Larger heap size
|
|
--------------------------
|
|
|
|
By default tests run with a 512 MB max heap. But some tests (monster/nightly)
|
|
need more heap. Use "-Dtests.heapsize" for this:
|
|
|
|
gradlew -p lucene/core test --tests "Test2BFST" -Dtests.heapsize=32g
|
|
|
|
|
|
Run GUI tests headlessly with Xvfb (Linux only)
|
|
-----------------------------------------------
|
|
|
|
GUI test for Luke application launches a window, this might mess up your
|
|
monitor depending on the display manager you are using. In that case,
|
|
you can install Xvfb (X Virtual Frame Buffer) so that the test runs on the
|
|
virtual display and does not open a real window.
|
|
|
|
# redhat-type OS
|
|
$ sudo yum install Xvfb
|
|
|
|
# ubuntu/debian-type OS
|
|
$ sudo apt install xvfb
|
|
|
|
# arch
|
|
$ sudo pacman -S xorg-server-xvfb
|
|
|
|
|
|
Profiling slow tests
|
|
--------------------
|
|
|
|
The "tests.profile" mode switch turns on a sampling profiler during test execution,
|
|
and prints a simple summary at the end.
|
|
|
|
For example, top-10 histogram of methods (cpu samples):
|
|
|
|
gradlew -p lucene/core test -Ptests.profile=true
|
|
|
|
Alternatively, you can profile heap allocations instead:
|
|
|
|
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap
|
|
|
|
By default, results are computed (deduplicated) on just the method name, folding
|
|
together all events from the same method. To drill down further, you can increase the
|
|
stack size from the default of 1, to get a histogram of stacktraces instead:
|
|
|
|
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8
|
|
|
|
For big methods, it can also be helpful to include line numbers for more granularity:
|
|
|
|
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true
|
|
|
|
Using these additional options will make the results more sparse, so it may be useful
|
|
to increase the top-N count:
|
|
|
|
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100
|
|
|
|
|
|
Generating Coverage Reports
|
|
------------------------------
|
|
|
|
Running the "coverage" task (or setting the property "tests.coverage" to true)
|
|
will run the tests with instrumentation to record code coverage.
|
|
|
|
Example:
|
|
|
|
gradlew -p lucene/core coverage
|
|
open lucene/core/build/reports/jacoco/test/html/index.html
|
|
|
|
If you want to use test filtering to just check a particular test, specify
|
|
the "test" task explicitly before "coverage":
|
|
|
|
gradlew -p lucene/core test --tests TestDemo coverage
|
|
open lucene/core/build/reports/jacoco/test/html/index.html
|
|
|
|
|
|
External data sets
|
|
------------------
|
|
|
|
Some tests may require external (and large) data sets. To see relevant tasks
|
|
that download and extract these data files automatically, run the following:
|
|
|
|
gradlew tasks --group "Data set download"
|