OpenSearch/rest-api-spec/test
Ryan Ernst 68f75ea7b6 simplified rest skip range version parsing more, ranges can now be open
ended
2015-04-19 06:50:08 -07:00
..
bulk Corrected test names for "Bulk" test suite 2013-09-09 23:36:43 +02:00
cat.aliases Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.allocation Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.count Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.fielddata Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.health Test: cat health api didn't disable headers 2015-02-25 16:17:47 +01:00
cat.indices Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.nodes Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cat.plugins Cat api: added node id column to _cat/plugins response 2015-03-25 12:00:38 +01:00
cat.recovery Recovery: add total operations to the `_recovery` API 2015-03-17 07:31:29 -07:00
cat.segments Cat api: added node id column to _cat/segments response 2015-03-25 12:00:15 +01:00
cat.shards Cat api: added node id column to _cat/shards response 2015-03-25 11:59:35 +01:00
cat.thread_pool Default _cat APIs to verbose 2014-12-15 12:51:59 +00:00
cluster.health Test: cluster.health rest test should use 0 replicas 2015-02-25 20:35:39 +01:00
cluster.pending_tasks [TEST] Added missing newline at end of YAML test file 2014-01-20 13:16:04 +01:00
cluster.put_settings simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00
cluster.reroute typo fixes - https://github.com/vlajos/misspell_fixer 2014-11-08 18:55:57 +01:00
cluster.state Cluster state REST api: print routing_nodes out only when requested through specific flag 2015-04-08 16:10:36 +02:00
create Remove async replication from the docs and REST spec 2015-03-19 15:34:12 +01:00
delete Fix REST test delete/42_missing_parent to catch a RoutingMissingException 2015-03-25 06:48:05 +02:00
delete_by_query [TEST] removed REST test that checks for delete by query shard failures 2014-02-26 14:11:40 +01:00
exists [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
explain Java API: add index, type and id to ExplainResponse 2014-08-08 12:52:03 +02:00
get Version types `EXTERNAL` & `EXTERNAL_GTE` test for version equality in read operation & disallow them in the Update API 2014-04-25 23:06:12 +02:00
get_source [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
index [REPLICATION] Remove `async` replication 2015-03-19 14:44:21 -07:00
indices.analyze [TEST] remove old tests from yaml test suite 2014-01-07 16:19:06 +01:00
indices.clear_cache Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.create Core: Added the `index.query.parse.allow_unmapped_fields` setting to fail queries if they refer to unmapped fields. 2014-09-09 15:00:47 +02:00
indices.delete_alias Aliases: Throw exception if index is null or missing when creating an alias 2014-10-27 14:39:01 -04:00
indices.delete_warmer [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
indices.exists [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_alias [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.exists_template Rest: expose `master_timeout` flag on `GET _template` & `HEAD _template` 2015-02-13 11:25:26 +01:00
indices.exists_type [TEST] Added missing \n at end of YAML files, quoted * and fixed indentation 2014-01-20 13:25:16 +01:00
indices.get simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00
indices.get_alias Indices API: Fix to make GET Index API consistent with docs 2015-01-08 08:48:44 +00:00
indices.get_aliases simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00
indices.get_field_mapping Mappings: Remove support for new indexes using path setting in 2015-02-05 12:44:43 -08:00
indices.get_mapping Indices API: Fix to make GET Index API consistent with docs 2015-01-08 08:48:44 +00:00
indices.get_settings [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
indices.get_template Rest: expose `master_timeout` flag on `GET _template` & `HEAD _template` 2015-02-13 11:25:26 +01:00
indices.get_warmer Indices API: Fix to make GET Index API consistent with docs 2015-01-08 08:48:44 +00:00
indices.open [TEST] indices.open REST tests need 0 replicas 2014-05-22 11:04:28 +02:00
indices.optimize Remove hard-coded "ok": true from REST responses 2014-01-07 09:27:07 -07:00
indices.put_alias Tests: Refixed the put_alias tests 2014-10-28 13:39:05 +01:00
indices.put_mapping Mapping: Fixes Merging of default analyzer 2014-11-14 09:58:26 +00:00
indices.put_settings simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00
indices.put_template Templates: GET templates doesn't honor the `flat_settings` parameter. 2014-07-02 08:42:31 +02:00
indices.put_warmer [TEST] added missing wait for yellow in put warmer REST test 2014-02-14 11:35:40 +01:00
indices.recovery [CORE] Rename START phase into VERIFY_INDEX 2015-04-14 09:53:02 +02:00
indices.refresh Remove expansion of empty index arguments in RoutingTable 2015-04-02 12:29:47 +02:00
indices.segments Fixed cat.segments and indices.segments tests - bad YAML indenting 2015-02-12 17:35:19 +01:00
indices.stats Remove expansion of empty index arguments in RoutingTable 2015-04-02 12:29:47 +02:00
indices.update_aliases PUT /_aliases should accept a numeric routing value 2014-03-20 10:38:32 +01:00
indices.validate_query Fixed validate query parsing issues 2014-05-12 12:49:25 +02:00
info Rest: remove status code from main action 2015-01-12 12:37:46 +01:00
mget [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly 2014-04-07 17:08:05 +02:00
mlt MLT Query: Support for ignore docs 2014-11-28 14:48:43 +01:00
mpercolate [TESTS] Added percolator rest tests. 2014-01-20 18:18:27 +01:00
msearch Search: Remove the `count` search type. 2015-03-31 11:31:49 +02:00
mtermvectors [SPEC] Renamed termvectors.* to termvector and mtermvectors 2014-01-21 16:31:50 +01:00
nodes.info REST: Add skip "stash_in_path" feature to nodes.info/20_transport test 2015-02-02 17:11:04 +01:00
nodes.stats [SPEC] Created snapshot.* and nodes.* namespaces 2014-01-17 11:58:44 +01:00
percolate Percolate api: support encoded body as query string param consistently 2015-02-11 08:53:04 +11:00
ping Renamed "ok" and "not_ok" to "is_true" and "is_false" 2013-07-01 15:58:23 +02:00
script Indexed scripts/templates: return response body when script is not found 2015-04-08 10:44:11 +02:00
scroll [TEST] improved regular scroll REST test 2014-09-25 13:02:16 +02:00
search Search: Remove the `count` search type. 2015-03-31 11:31:49 +02:00
search.aggregation Test: Set number_of_replicas to zero so that wait_for_green succeeds in search.aggregations 2014-07-21 22:41:15 +02:00
search_shards Add REST API spec for /_search_shards endpoint 2014-04-23 13:29:29 -06:00
snapshot.get_repository [SNAPSHOT] Add repository validation 2014-10-07 10:50:16 -04:00
suggest Geo context suggester: Require precision in mapping 2014-04-02 23:51:14 +02:00
template Indexed scripts/templates: fix yaml test indentation 2015-04-08 15:36:23 -07:00
termvectors Term Vectors: support for version and version_type 2014-12-17 15:43:15 +01:00
update simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00
README.asciidoc simplified rest skip range version parsing more, ranges can now be open 2015-04-19 06:50:08 -07:00

README.asciidoc

Test Suite:
===========

A YAML test file consists of:
* an optional `setup` section, followed by
* one or more test sections

For instance:

    setup:
      - do: ....
      - do: ....

    ---
    "First test":
      - do: ...
      - match: ...

    ---
    "Second test":
      - do: ...
      - match: ...


A `setup` section contains a list of commands to run before each test
section in order to setup the same environment for each test section.

A test section represents an independent test, containing multiple `do`
statements and assertions. The contents of a test section must be run in
order, but individual test sections may be run in any order, as follows:

1. run `setup` (if any)
2. reset the `response` var and the `stash` (see below)
2. run test contents
3. run teardown

The `teardown` should delete all indices and all templates.

Dot notation:
-------------
Dot notation is used for (1) method calls and (2) hierarchical data structures.  For
instance, a method call like `cluster.health` would do the equivalent of:

    client.cluster.health(...params...)

A test against `_tokens.1.token` would examine the `token` key, in the second element
of the `tokens` array, inside the `response` var (see below):

    $val = $response->{tokens}[1]{token}  # Perl syntax roolz!

If one of the levels (eg `tokens`) does not exist, it should return an undefined value.
If no field name is given (ie the empty string) then return the current
$val -- used for testing the whole response body.

Use \. to specify paths that actually contain '.' in the key name, for example
in the `indices.get_settings` API.

Skipping tests:
---------------
If a test section should only be run on certain versions of Elasticsearch,
then the first entry in the section (after the title) should be called
`skip`, and should contain the range of versions to be
skipped, and the reason why the tests are skipped.  For instance:

....
    "Parent":
     - skip:
          version:     "0.20.1 - 0.90.2"
          reason:      Delete ignores the parent param

     - do:
       ... test definitions ...
....

All tests in the file following the skip statement should be skipped if:
`min <= current <= max`.

The `version` range can leave either bound empty, which means "open ended".
For instance:
....
    "Parent":
     - skip:
          version:     "1.0.0.Beta1 - "
          reason:      Delete ignores the parent param

     - do:
       ... test definitions ...
....

The skip section can also be used to list new features that need to be
supported in order to run a test. This way the up-to-date runners will
run the test, while the ones that don't support the feature yet can
temporarily skip it, and avoid having lots of test failures in the meantime.
Once all runners have implemented the feature, it can be declared supported
by default, thus the related skip sections can be removed from the tests.

....
    "Parent":
     - skip:
          features:    regex

     - do:
       ... test definitions ...
....

The `features` field can either be a string or an array of strings.
The skip section requires to specify either a `version` or a `features` list.

Required operators:
-------------------

=== `do`

The `do` operator calls a method on the client. For instance:

....
    - do:
        cluster.health:
            level: shards
....

The response from the `do` operator should be stored in the `response` var, which
is reset (1) at the beginning of a file or (2) on the next `do`.

If the arguments to `do` include `catch`, then we are expecting an error, which should
be caught and tested.  For instance:

....
    - do:
        catch:        missing
        get:
            index:    test
            type:    test
            id:        1
....

The argument to `catch` can be any of:

[horizontal]
`missing`::     a 404 response from ES
`conflict`::    a 409 response from ES
`request`::     a generic error response from ES
`param`::       a client-side error indicating an unknown parameter has been passed
                to the method
`/foo bar/`::   the text of the error message matches this regular expression

If `catch` is specified, then the `response` var must be cleared, and the test
should fail if no error is thrown.

=== `set`

For some tests, it is necessary to extract a value from the previous `response`, in
order to reuse it in a subsequent `do` and other tests.  For instance, when
testing indexing a document without a specified ID:

....
    - do:
        index:
            index: test
            type:  test
    - set:  { _id: id }   # stash the value of `response._id` as `id`
    - do:
        get:
            index: test
            type:  test
            id:    $id    # replace `$id` with the stashed value
    - match: { _id: $id } # the returned `response._id` matches the stashed `id`
....

The last response obtained gets always stashed automatically as a string, called `body`.
This is useful when needing to test apis that return text rather than json (e.g. cat api),
as it allows to treat the whole body as an ordinary string field.

Stashed values can be used in property names, eg:

....
  - do:
      cluster.state: {}

  - set: {master_node: master}

  - do:
      nodes.info:
        metric: [ transport ]

  - is_true: nodes.$master.transport.profiles
....


Note that not only expected values can be retrieved from the stashed values (as in the
example above), but the same goes for actual values:

....
    - match: { $body: /^.+$/ } # the returned `body` matches the provided regex if the body is text
    - match: { $body: {} } # the returned `body` matches the JSON object if the body is JSON
....

The stash should be reset at the beginning of each test file.

=== `is_true`

The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null`
or the empty string), eg:

....
    - is_true:  fields._parent  # the _parent key exists in the fields hash and is "true"
....

=== `is_false`

The specified key doesn't exist or has a false value (ie `0`, `false`, `undefined`,
`null` or the empty string), eg:

....
    - is_false:  fields._source  # the _source key doesn't exist in the fields hash or is "false"
....

=== `match`

Used to compare two variables (could be scalars, arrays or hashes).  The two variables
should be identical, eg:

....
    - match: { _source: { foo: bar }}
....

Supports also regular expressions with flag X for more readability (accepts whitespaces and comments):

....
  - match:
      $body: >
               /^  epoch  \s+  timestamp          \s+  count  \s+  \n
                   \d+    \s+  \d{2}:\d{2}:\d{2}  \s+  \d+    \s+  \n  $/
....

=== `lt` and `gt`

Compares two numeric values, eg:

....
    - lt: { fields._ttl: 10000 }  # the `_ttl` value is less than 10,000
....

=== `lte` and `gte`

Compares two numeric values, eg:

....
    - lte: { fields._ttl: 10000 }  # the `_ttl` value is less than or equal to 10,000
....

=== `length`

This depends on the datatype of the value being examined, eg:

....
    - length: { _id: 22    }   # the `_id` string is 22 chars long
    - length: { _tokens: 3 }   # the `_tokens` array has 3 elements
    - length: { _source: 5 }   # the `_source` hash has 5 keys
....