diff --git a/rest-api-spec/test/README.asciidoc b/rest-api-spec/test/README.asciidoc index c1d43bac6c5..68599fe2dcc 100644 --- a/rest-api-spec/test/README.asciidoc +++ b/rest-api-spec/test/README.asciidoc @@ -1,24 +1,49 @@ Test Suite: =========== -The tests in each YAML file should be run in order as a single set of dependent tests. -Before each file, the test cluster should be "reset" (eg deleting indices etc), +The tests in each YAML file should be run in order as a single set of dependent tests. +Before each file, the test cluster should be "reset" (eg deleting indices etc), the `response` var and the `stash` should be cleared (see below). Dot notation: ------------- -Dot notation is used for (1) method calls and (2) hierarchical data structures. For +Dot notation is used for (1) method calls and (2) hierarchical data structures. For instance, a method call like `cluster.health` would do the equivalent of: - client.cluster.health(...params...) + client.cluster.health(...params...) A test against `_tokens.1.token` would examine the `token` key, in the second element of the `tokens` array, inside the `response` var (see below): - $val = $response->{tokens}[1]{token} # Perl syntax roolz! + $val = $response->{tokens}[1]{token} # Perl syntax roolz! If one of the levels (eg `tokens`) does not exist, it should return an undefined value. +Skipping tests: +--------------- +If tests should be skipped on particular versions of Elasticsearch, the +first entry in the list of tests should be called `skip`, and contain +the range of versions to be skipped, and the reason why the tests are +skipped. For instance: + + "Parent": + - skip: + version: "0 - 0.90.2" + reason: Delete ignores the parent param + + - do: + ... test definitions ... + +All tests in the file should be skipped if: `min <= current <= max`. + +The `version` range should always have an upper bound. Versions should +either have each version part compared numerically, or should be converted +to a string with sufficient digits to allow string comparison, eg + + 0.90.2 -> 000-090-002 + +Snapshot versions and versions of the form `1.0.0.Beta1` can be treated +as the rounded down version, eg `1.0.0`. Required operators: ------------------- @@ -27,28 +52,28 @@ Required operators: The `do` operator calls a method on the client. For instance: - - do: - cluster.health: - level: shards + - do: + cluster.health: + level: shards -The response from the `do` operator should be stored in the `response` var, which +The response from the `do` operator should be stored in the `response` var, which is reset (1) at the beginning of a file or (2) on the next `do`. -If the arguments to `do` include `catch`, then we are expecting an error, which should +If the arguments to `do` include `catch`, then we are expecting an error, which should be caught and tested. For instance: - - do: - catch: missing - get: - index: test - type: test - id: 1 + - do: + catch: missing + get: + index: test + type: test + id: 1 The argument to `catch` can be any of: -`missing`:: a 404 response from ES -`conflict`:: a 409 response from ES -`request`:: a generic error response from ES +`missing`:: a 404 response from ES +`conflict`:: a 409 response from ES +`request`:: a generic error response from ES `param`:: a client-side error indicating an unknown parameter has been passed to the method `/foo bar/`:: the text of the error message matches this regular expression @@ -59,57 +84,57 @@ should fail if no error is thrown. === `set` For some tests, it is necessary to extract a value from the previous `response`, in -order to reuse it in a subsequent `do` and other tests. For instance, when +order to reuse it in a subsequent `do` and other tests. For instance, when testing indexing a document without a specified ID: - - do: - index: - index: test - type: test - - set: { _id: id } # stash the value of `response._id` as `id` - - do: - get: - index: test - type: test - id: $id # replace `$id` with the stashed value + - do: + index: + index: test + type: test + - set: { _id: id } # stash the value of `response._id` as `id` + - do: + get: + index: test + type: test + id: $id # replace `$id` with the stashed value - match: { _id: $id } # the returned `response._id` matches the stashed `id` The stash should be reset at the beginning of each test file. === `ok` -The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null` +The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null` or the empty string), eg: - - ok: fields._parent # the _parent key exists in the fields hash and is "true" + - ok: fields._parent # the _parent key exists in the fields hash and is "true" === `not_ok` -The specified key doesn't exist or has a false value (ie `0`, `false`, `undefined`, +The specified key doesn't exist or has a false value (ie `0`, `false`, `undefined`, `null` or the empty string), eg: - - not_ok: fields._source # the _source key doesn't exist in the fields hash + - not_ok: fields._source # the _source key doesn't exist in the fields hash === `match` -Used to compare two variables (could be scalars, arrays or hashes). The two variables +Used to compare two variables (could be scalars, arrays or hashes). The two variables should be identical, eg: - - + - === `lt` and `gt` Compares two numeric values, eg: - - lt: { fields._ttl: 10000 } # the `_ttl` value is less than 10,000 + - lt: { fields._ttl: 10000 } # the `_ttl` value is less than 10,000 === `length` This depends on the datatype of the value being examined, eg: - - length: { _id: 22 } # the `_id` string is 22 chars long - - length: { _tokens: 3 } # the `_tokens` array has 3 elements - - length: { _source: 5 } # the `_source` hash has 5 keys + - length: { _id: 22 } # the `_id` string is 22 chars long + - length: { _tokens: 3 } # the `_tokens` array has 3 elements + - length: { _source: 5 } # the `_source` hash has 5 keys