OpenSearch/rest-api-spec/test
Clinton Gormley 34893c0570 Test: Temporarily change delete/put_mapping to wait for green
Delete and update mapping execution order has changed with #6762
2014-07-07 19:02:36 +02:00
..
abort_benchmark [TEST] Aborting a non-existent benchmark throws a missing request 2014-05-10 13:09:30 +02:00
benchmark [TEST] randomly added node.bench=true to client node in test cluster and re-enabled REST benchmark tests based on number of bench nodes available 2014-05-09 23:36:00 +02:00
bulk Corrected test names for "Bulk" test suite 2013-09-09 23:36:43 +02:00
cat.aliases [TEST] Replace folded blocks with literal blocks 2014-05-02 18:30:48 -04:00
cat.allocation [TEST] cat.allocation/One index: allow leading spaces in unassigned shards lines 2014-05-22 10:10:50 +02:00
cat.count [TEST] Replace folded blocks with literal blocks 2014-05-02 18:30:48 -04:00
cat.fielddata [TEST] Add test specifying fields field /_cat/fielddata REST test 2014-05-09 23:57:23 +02:00
cat.recovery Test: Python doesn't support atomic groups in regexes. Removing from cat.recovery 2014-05-29 16:57:09 +02:00
cat.shards [TEST] added wait for relocationg shards to cat.shards REST test 2014-06-03 10:48:11 +02:00
cat.thread_pool [TEST] fixed _cat/thread_pool REST tests with local transport, in case the transport port is not available and gets returned as '-' 2014-05-04 22:10:03 +02:00
cluster.pending_tasks [TEST] Added missing newline at end of YAML test file 2014-01-20 13:16:04 +01:00
cluster.put_settings [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
cluster.reroute [TEST] added support for replacing stashed values within objects and lists in our REST tests 2014-03-03 13:34:22 +01:00
cluster.state Cluster State API: Remove index template filtering 2014-05-05 14:54:14 +02:00
create [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
delete [TEST] specified number_of_shards 5 to make sure the two docs end up in different shards that can be refreshed separately 2014-04-07 19:01:19 +02:00
delete_by_query [TEST] removed REST test that checks for delete by query shard failures 2014-02-26 14:11:40 +01:00
exists [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
explain Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
get Version types `EXTERNAL` & `EXTERNAL_GTE` test for version equality in read operation & disallow them in the Update API 2014-04-25 23:06:12 +02:00
get_source [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
index Allow 0 as a valid external version 2014-05-16 22:10:16 +02:00
indices.analyze [TEST] remove old tests from yaml test suite 2014-01-07 16:19:06 +01:00
indices.clear_cache Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.create Added support for aliases to create index api 2014-02-17 14:54:21 +01:00
indices.delete_alias Consistent REST API changes for GETting data 2014-01-14 22:33:52 +01:00
indices.delete_mapping Test: Temporarily change delete/put_mapping to wait for green 2014-07-07 19:02:36 +02:00
indices.delete_warmer [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
indices.exists [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_alias [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_template Added setup task that clears the test template before running exists_* tests 2014-01-20 09:52:25 -07:00
indices.exists_type [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.get_alias Added support for aliases to create index api 2014-02-17 14:54:21 +01:00
indices.get_aliases Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
indices.get_field_mapping Added TransportGetFieldMappingsIndexAction that uses TransportSingleCustomOperationAction as base class, with the goal to reuse common logic (like: retry on failures, shard picking, connecting to nodes) 2014-02-24 15:54:24 +01:00
indices.get_mapping [TEST] Added check: test section names must be unique in the same REST test suite 2014-01-16 21:09:55 +01:00
indices.get_settings [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
indices.get_template Templates: GET templates doesn't honor the `flat_settings` parameter. 2014-07-02 08:42:31 +02:00
indices.get_warmer [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
indices.open [TEST] indices.open REST tests need 0 replicas 2014-05-22 11:04:28 +02:00
indices.optimize Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.put_alias [TEST] fixed typo s/mising/missing 2014-02-03 15:30:58 +01:00
indices.put_mapping Test: Temporarily change delete/put_mapping to wait for green 2014-07-07 19:02:36 +02:00
indices.put_settings Make PUT and DELETE consistent for _mapping, _alias and _warmer 2014-01-14 20:02:43 +01:00
indices.put_template Templates: GET templates doesn't honor the `flat_settings` parameter. 2014-07-02 08:42:31 +02:00
indices.put_warmer [TEST] added missing wait for yellow in put warmer REST test 2014-02-14 11:35:40 +01:00
indices.recovery [TEST] Don't use replicas if only primaries are expected or the first shard in the list shoudl be a primary 2014-03-22 18:36:34 +01:00
indices.segments Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.stats [TEST] Remove indentation on indices.stats test setups 2014-06-18 11:59:40 -04:00
indices.update_aliases PUT /_aliases should accept a numeric routing value 2014-03-20 10:38:32 +01:00
indices.validate_query Fixed validate query parsing issues 2014-05-12 12:49:25 +02:00
info Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
list_benchmarks [TEST] randomly added node.bench=true to client node in test cluster and re-enabled REST benchmark tests based on number of bench nodes available 2014-05-09 23:36:00 +02:00
mget [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
mlt [TEST] MLT Rest test needs a mapping since we randomized number of nodes/shards 2014-04-09 11:45:55 +02:00
mpercolate [TESTS] Added percolator rest tests. 2014-01-20 18:18:27 +01:00
msearch msearch tests 2013-07-22 00:40:12 +02:00
mtermvectors [SPEC] Renamed termvectors.* to termvector and mtermvectors 2014-01-21 16:31:50 +01:00
nodes.info [SPEC] Created snapshot.* and nodes.* namespaces 2014-01-17 11:58:44 +01:00
nodes.stats [SPEC] Created snapshot.* and nodes.* namespaces 2014-01-17 11:58:44 +01:00
percolate [TEST] Removed whitespace between } and , 2014-02-20 17:45:49 +01:00
ping Renamed "ok" and "not_ok" to "is_true" and "is_false" 2013-07-01 15:58:23 +02:00
scroll Return missing (404) is a scroll_id is cleared that no longer exists. 2014-05-12 09:43:56 +02:00
search significant terms: fix json response 2014-06-18 18:51:34 +02:00
search_shards Add REST API spec for /_search_shards endpoint 2014-04-23 13:29:29 -06:00
snapshot.get_repository Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
suggest Geo context suggester: Require precision in mapping 2014-04-02 23:51:14 +02:00
termvector [SPEC] Renamed termvectors.* to termvector and mtermvectors 2014-01-21 16:31:50 +01:00
update Switch to Groovy as the default scripting language 2014-06-25 12:15:12 +02:00
README.asciidoc REST Testing framework enhancement 2014-03-10 15:08:43 -07:00

README.asciidoc

Test Suite:
===========

A YAML test file consists of:
* an optional `setup` section, followed by
* one or more test sections

For instance:

    setup:
      - do: ....
      - do: ....

    ---
    "First test":
      - do: ...
      - match: ...

    ---
    "Second test":
      - do: ...
      - match: ...


A `setup` section contains a list of commands to run before each test
section in order to setup the same environment for each test section.

A test section represents an independent test, containing multiple `do`
statements and assertions. The contents of a test section must be run in
order, but individual test sections may be run in any order, as follows:

1. run `setup` (if any)
2. reset the `response` var and the `stash` (see below)
2. run test contents
3. run teardown

The `teardown` should delete all indices and all templates.

Dot notation:
-------------
Dot notation is used for (1) method calls and (2) hierarchical data structures.  For
instance, a method call like `cluster.health` would do the equivalent of:

    client.cluster.health(...params...)

A test against `_tokens.1.token` would examine the `token` key, in the second element
of the `tokens` array, inside the `response` var (see below):

    $val = $response->{tokens}[1]{token}  # Perl syntax roolz!

If one of the levels (eg `tokens`) does not exist, it should return an undefined value.
If no field name is given (ie the empty string) then return the current
$val -- used for testing the whole response body.

Use \. to specify paths that actually contain '.' in the key name, for example
in the `indices.get_settings` API.

Skipping tests:
---------------
If a test section should only be run on certain versions of Elasticsearch,
then the first entry in the section (after the title) should be called
`skip`, and should contain the range of versions to be
skipped, and the reason why the tests are skipped.  For instance:

....
    "Parent":
     - skip:
          version:     "0 - 0.90.2"
          reason:      Delete ignores the parent param

     - do:
       ... test definitions ...
....

All tests in the file following the skip statement should be skipped if:
`min <= current <= max`.

The `version` range should always have an upper bound. Versions should
either have each version part compared numerically, or should be converted
to a string with sufficient digits to allow string comparison, eg

    0.90.2 -> 000-090-002

Snapshot versions and versions of the form `1.0.0.Beta1` can be treated
as the rounded down version, eg `1.0.0`.

The skip section can also be used to list new features that need to be
supported in order to run a test. This way the up-to-date runners will
run the test, while the ones that don't support the feature yet can
temporarily skip it, and avoid having lots of test failures in the meantime.
Once all runners have implemented the feature, it can be declared supported
by default, thus the related skip sections can be removed from the tests.

....
    "Parent":
     - skip:
          features:    regex

     - do:
       ... test definitions ...
....

The `features` field can either be a string or an array of strings.
The skip section requires to specify either a `version` or a `features` list.

Required operators:
-------------------

=== `do`

The `do` operator calls a method on the client. For instance:

....
    - do:
        cluster.health:
            level: shards
....

The response from the `do` operator should be stored in the `response` var, which
is reset (1) at the beginning of a file or (2) on the next `do`.

If the arguments to `do` include `catch`, then we are expecting an error, which should
be caught and tested.  For instance:

....
    - do:
        catch:        missing
        get:
            index:    test
            type:    test
            id:        1
....

The argument to `catch` can be any of:

[horizontal]
`missing`::     a 404 response from ES
`conflict`::    a 409 response from ES
`request`::     a generic error response from ES
`param`::       a client-side error indicating an unknown parameter has been passed
                to the method
`/foo bar/`::   the text of the error message matches this regular expression

If `catch` is specified, then the `response` var must be cleared, and the test
should fail if no error is thrown.

=== `set`

For some tests, it is necessary to extract a value from the previous `response`, in
order to reuse it in a subsequent `do` and other tests.  For instance, when
testing indexing a document without a specified ID:

....
    - do:
        index:
            index: test
            type:  test
    - set:  { _id: id }   # stash the value of `response._id` as `id`
    - do:
        get:
            index: test
            type:  test
            id:    $id    # replace `$id` with the stashed value
    - match: { _id: $id } # the returned `response._id` matches the stashed `id`
....

The last response obtained gets always stashed automatically as a string, called `body`.
This is useful when needing to test apis that return text rather than json (e.g. cat api),
as it allows to treat the whole body as an ordinary string field.

Note that not only expected values can be retrieved from the stashed values (as in the
example above), but the same goes for actual values:

....
    - match: { $body: /^.+$/ } # the returned `body` matches the provided regex
....

The stash should be reset at the beginning of each test file.

=== `is_true`

The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null`
or the empty string), eg:

....
    - is_true:  fields._parent  # the _parent key exists in the fields hash and is "true"
....

=== `is_false`

The specified key doesn't exist or has a false value (ie `0`, `false`, `undefined`,
`null` or the empty string), eg:

....
    - is_false:  fields._source  # the _source key doesn't exist in the fields hash or is "false"
....

=== `match`

Used to compare two variables (could be scalars, arrays or hashes).  The two variables
should be identical, eg:

....
    - match: { _source: { foo: bar }}
....

Supports also regular expressions with flag X for more readability (accepts whitespaces and comments):

....
  - match:
      $body: >
               /^  epoch  \s+  timestamp          \s+  count  \s+  \n
                   \d+    \s+  \d{2}:\d{2}:\d{2}  \s+  \d+    \s+  \n  $/
....

=== `lt` and `gt`

Compares two numeric values, eg:

....
    - lt: { fields._ttl: 10000 }  # the `_ttl` value is less than 10,000
....

=== `lte` and `gte`

Compares two numeric values, eg:

....
    - lte: { fields._ttl: 10000 }  # the `_ttl` value is less than or equal to 10,000
....

=== `length`

This depends on the datatype of the value being examined, eg:

....
    - length: { _id: 22    }   # the `_id` string is 22 chars long
    - length: { _tokens: 3 }   # the `_tokens` array has 3 elements
    - length: { _source: 5 }   # the `_source` hash has 5 keys
....