OpenSearch/rest-api-spec/test
Clinton Gormley f4c0c3dae6 [TESTS] Fixed YAML in put_templates test 2014-03-06 12:16:01 +01:00
..
bulk Corrected test names for "Bulk" test suite 2013-09-09 23:36:43 +02:00
cat.aliases [TEST] Added cat.allocation and cat.aliases tests 2014-02-07 17:36:01 +01:00
cat.allocation [TEST] Removed nested quantifier in cat.allocation regex and made node name regex more liberal 2014-02-09 14:19:59 +01:00
cat.count [TEST] Made cat.count REST test more readable 2014-02-10 16:22:38 +01:00
cat.shards [TEST] Add basic test for /cat/shards/{index} 2014-02-10 12:23:01 +01:00
cat.thread_pool [TEST] Added REST tests for _cat/thread_pool endpoint 2014-02-10 16:20:03 +01:00
cluster.pending_tasks [TEST] Added missing newline at end of YAML test file 2014-01-20 13:16:04 +01:00
cluster.put_settings Add support for flat_settings flag to all REST APIs that output settings 2014-01-08 10:36:36 -05:00
cluster.reroute [TEST] added support for replacing stashed values within objects and lists in our REST tests 2014-03-03 13:34:22 +01:00
cluster.state Add `explain` flag support to the reroute API 2014-02-27 09:48:51 -07:00
create [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
delete [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
delete_by_query [TEST] removed REST test that checks for delete by query shard failures 2014-02-26 14:11:40 +01:00
exists [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
explain Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
get [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
get_source [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
index [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
indices.analyze [TEST] remove old tests from yaml test suite 2014-01-07 16:19:06 +01:00
indices.clear_cache Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.create Added support for aliases to create index api 2014-02-17 14:54:21 +01:00
indices.delete_alias Consistent REST API changes for GETting data 2014-01-14 22:33:52 +01:00
indices.delete_mapping [TEST] removed leftover delete alias test in delete mapping 2014-02-07 13:52:49 +01:00
indices.delete_warmer [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
indices.exists [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_alias [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_template Added setup task that clears the test template before running exists_* tests 2014-01-20 09:52:25 -07:00
indices.exists_type [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.get_alias Added support for aliases to create index api 2014-02-17 14:54:21 +01:00
indices.get_aliases Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
indices.get_field_mapping Added TransportGetFieldMappingsIndexAction that uses TransportSingleCustomOperationAction as base class, with the goal to reuse common logic (like: retry on failures, shard picking, connecting to nodes) 2014-02-24 15:54:24 +01:00
indices.get_mapping [TEST] Added check: test section names must be unique in the same REST test suite 2014-01-16 21:09:55 +01:00
indices.get_settings Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
indices.get_template Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
indices.get_warmer [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
indices.open Fix YAML in test/indices.open/20_multiple_indices.yaml 2014-01-13 13:17:30 +01:00
indices.optimize Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.put_alias [TEST] fixed typo s/mising/missing 2014-02-03 15:30:58 +01:00
indices.put_mapping [TEST] fixed typo s/mising/missing 2014-02-03 15:30:58 +01:00
indices.put_settings Make PUT and DELETE consistent for _mapping, _alias and _warmer 2014-01-14 20:02:43 +01:00
indices.put_template [TESTS] Fixed YAML in put_templates test 2014-03-06 12:16:01 +01:00
indices.put_warmer [TEST] added missing wait for yellow in put warmer REST test 2014-02-14 11:35:40 +01:00
indices.segments Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.snapshot_index Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.stats Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.status Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.update_aliases Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.validate_query [TEST] disabled replicas in validate query test 2014-02-11 17:16:43 +01:00
info Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
mget [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
mlt Wait for green in mlt tests 2014-02-07 19:20:36 +01:00
mpercolate [TESTS] Added percolator rest tests. 2014-01-20 18:18:27 +01:00
msearch msearch tests 2013-07-22 00:40:12 +02:00
mtermvectors [SPEC] Renamed termvectors.* to termvector and mtermvectors 2014-01-21 16:31:50 +01:00
nodes.info [SPEC] Created snapshot.* and nodes.* namespaces 2014-01-17 11:58:44 +01:00
nodes.stats [SPEC] Created snapshot.* and nodes.* namespaces 2014-01-17 11:58:44 +01:00
percolate [TEST] Removed whitespace between } and , 2014-02-20 17:45:49 +01:00
ping Renamed "ok" and "not_ok" to "is_true" and "is_false" 2013-07-01 15:58:23 +02:00
scroll Scroll test was missing an index - fails when other indices present 2013-09-16 20:56:32 +02:00
search Add mustache templating to query execution. 2014-02-20 12:21:59 +01:00
snapshot.get_repository Added base Request class for read operations that usually happen on the master but can be executed locally. 2014-01-20 12:35:48 +01:00
suggest [TEST] remove old tests from yaml test suite 2014-01-07 16:19:06 +01:00
termvector [SPEC] Renamed termvectors.* to termvector and mtermvectors 2014-01-21 16:31:50 +01:00
update [TEST] moved routing REST tests to 0 replicas 2014-02-10 12:32:34 +01:00
README.asciidoc [TEST] Added ability to test apis that don't return json 2014-02-03 11:26:38 +01:00

README.asciidoc

Test Suite:
===========

A YAML test file consists of:
* an optional `setup` section, followed by
* one or more test sections

For instance:

    setup:
      - do: ....
      - do: ....

    ---
    "First test":
      - do: ...
      - match: ...

    ---
    "Second test":
      - do: ...
      - match: ...


A `setup` section contains a list of commands to run before each test
section in order to setup the same environment for each test section.

A test section represents an independent test, containing multiple `do`
statements and assertions. The contents of a test section must be run in
order, but individual test sections may be run in any order, as follows:

1. run `setup` (if any)
2. reset the `response` var and the `stash` (see below)
2. run test contents
3. run teardown

The `teardown` should delete all indices and all templates.

Dot notation:
-------------
Dot notation is used for (1) method calls and (2) hierarchical data structures.  For
instance, a method call like `cluster.health` would do the equivalent of:

    client.cluster.health(...params...)

A test against `_tokens.1.token` would examine the `token` key, in the second element
of the `tokens` array, inside the `response` var (see below):

    $val = $response->{tokens}[1]{token}  # Perl syntax roolz!

If one of the levels (eg `tokens`) does not exist, it should return an undefined value.
If no field name is given (ie the empty string) then return the current
$val -- used for testing the whole response body.

Use \. to specify paths that actually contain '.' in the key name, for example
in the `indices.get_settings` API.

Skipping tests:
---------------
If a test section should only be run on certain versions of Elasticsearch,
then the first entry in the section (after the title) should be called
`skip`, and should contain the range of versions to be
skipped, and the reason why the tests are skipped.  For instance:

....
    "Parent":
     - skip:
          version:     "0 - 0.90.2"
          reason:      Delete ignores the parent param

     - do:
       ... test definitions ...
....

All tests in the file following the skip statement should be skipped if:
`min <= current <= max`.

The `version` range should always have an upper bound. Versions should
either have each version part compared numerically, or should be converted
to a string with sufficient digits to allow string comparison, eg

    0.90.2 -> 000-090-002

Snapshot versions and versions of the form `1.0.0.Beta1` can be treated
as the rounded down version, eg `1.0.0`.

The skip section can also be used to list new features that need to be
supported in order to run a test. This way the up-to-date runners will
run the test, while the ones that don't support the feature yet can
temporarily skip it, and avoid having lots of test failures in the meantime.
Once all runners have implemented the feature, it can be declared supported
by default, thus the related skip sections can be removed from the tests.

....
    "Parent":
     - skip:
          features:    regex

     - do:
       ... test definitions ...
....

The `features` field can either be a string or an array of strings.
The skip section requires to specify either a `version` or a `features` list.

Required operators:
-------------------

=== `do`

The `do` operator calls a method on the client. For instance:

....
    - do:
        cluster.health:
            level: shards
....

The response from the `do` operator should be stored in the `response` var, which
is reset (1) at the beginning of a file or (2) on the next `do`.

If the arguments to `do` include `catch`, then we are expecting an error, which should
be caught and tested.  For instance:

....
    - do:
        catch:        missing
        get:
            index:    test
            type:    test
            id:        1
....

The argument to `catch` can be any of:

[horizontal]
`missing`::     a 404 response from ES
`conflict`::    a 409 response from ES
`request`::     a generic error response from ES
`param`::       a client-side error indicating an unknown parameter has been passed
                to the method
`/foo bar/`::   the text of the error message matches this regular expression

If `catch` is specified, then the `response` var must be cleared, and the test
should fail if no error is thrown.

=== `set`

For some tests, it is necessary to extract a value from the previous `response`, in
order to reuse it in a subsequent `do` and other tests.  For instance, when
testing indexing a document without a specified ID:

....
    - do:
        index:
            index: test
            type:  test
    - set:  { _id: id }   # stash the value of `response._id` as `id`
    - do:
        get:
            index: test
            type:  test
            id:    $id    # replace `$id` with the stashed value
    - match: { _id: $id } # the returned `response._id` matches the stashed `id`
....

The last response obtained gets always stashed automatically as a string, called `body`.
This is useful when needing to test apis that return text rather than json (e.g. cat api),
as it allows to treat the whole body as an ordinary string field.

Note that not only expected values can be retrieved from the stashed values (as in the
example above), but the same goes for actual values:

....
    - match: { $body: /^.+$/ } # the returned `body` matches the provided regex
....

The stash should be reset at the beginning of each test file.

=== `is_true`

The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null`
or the empty string), eg:

....
    - is_true:  fields._parent  # the _parent key exists in the fields hash and is "true"
....

=== `is_false`

The specified key doesn't exist or has a false value (ie `0`, `false`, `undefined`,
`null` or the empty string), eg:

....
    - is_false:  fields._source  # the _source key doesn't exist in the fields hash or is "false"
....

=== `match`

Used to compare two variables (could be scalars, arrays or hashes).  The two variables
should be identical, eg:

....
    - match: { _source: { foo: bar }}
....

Supports also regular expressions with flag X for more readability (accepts whitespaces and comments):

....
  - match:
      $body: >
               /^  epoch  \s+  timestamp          \s+  count  \s+  \n
                   \d+    \s+  \d{2}:\d{2}:\d{2}  \s+  \d+    \s+  \n  $/
....

=== `lt` and `gt`

Compares two numeric values, eg:

....
    - lt: { fields._ttl: 10000 }  # the `_ttl` value is less than 10,000
....

=== `length`

This depends on the datatype of the value being examined, eg:

....
    - length: { _id: 22    }   # the `_id` string is 22 chars long
    - length: { _tokens: 3 }   # the `_tokens` array has 3 elements
    - length: { _source: 5 }   # the `_source` hash has 5 keys
....