Build: Enable testing without magic comments (backports #46180) (#46325)

Previously we only turned on tests if we saw either `// CONSOLE` or
`// TEST`. These magic comments are difficult for the docs build to deal
with so it has moved away from using them where possible. We should
catch up. This adds another trigger to enable testing: marking a snippet
with the `console` language. It looks like this:

```
[source,console]
----
GET /
----
```

This saves a line which is nice, I guess. But it is more important to me
that this is consistent with the way the docs build works now.

Similarly this enables response testing when you mark a snippet with the
language `console-result`. That looks like:
```
[source,console-result]
----
{
  "result": "0.1"
}
----
```

`// TESTRESPONSE` is still available for situations like `// TEST`: when
the response isn't *in* the console-result language (like `_cat`) or
when you want to perform substitutions on the generated test.

Should unblock #46159.
This commit is contained in:
Nik Everett 2019-09-04 15:19:20 -04:00 committed by GitHub
parent 5ee336ff78
commit d5ad86dad9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 34 additions and 27 deletions

View File

@ -202,11 +202,12 @@ public class RestTestsFromSnippetsTask extends SnippetsTask {
previousTest = snippet previousTest = snippet
return return
} }
if (snippet.testResponse) { if (snippet.testResponse || snippet.language == 'console-result') {
response(snippet) response(snippet)
return return
} }
if (snippet.test || snippet.console) { if (snippet.test || snippet.console ||
snippet.language == 'console') {
test(snippet) test(snippet)
previousTest = snippet previousTest = snippet
return return

View File

@ -3,10 +3,14 @@ Elasticsearch documentation build process.
See: https://github.com/elastic/docs See: https://github.com/elastic/docs
Snippets marked with `// CONSOLE` are automatically annotated with "VIEW IN Snippets marked with `[source,console]` are automatically annotated with
CONSOLE" and "COPY AS CURL" in the documentation and are automatically tested "VIEW IN CONSOLE" and "COPY AS CURL" in the documentation and are automatically
by the command `gradle :docs:check`. To test just the docs from a single page, tested by the command `./gradlew -pdocs check`. To test just the docs from a
use e.g. `./gradlew :docs:integTestRunner --tests "*rollover*"`. single page, use e.g. `./gradlew -ddocs integTestRunner --tests "*rollover*"`.
NOTE: Previously we use `// CONSOLE` instead of `[source,console]`. This worked
well for a long time so you'll see it all over early branches but we're phasing
it out because it requires some unpleasant hackery on the docs build side.
NOTE: If you have an elasticsearch-extra folder alongside your elasticsearch NOTE: If you have an elasticsearch-extra folder alongside your elasticsearch
folder, you must temporarily rename it when you are testing 6.3 or later branches. folder, you must temporarily rename it when you are testing 6.3 or later branches.
@ -45,10 +49,21 @@ for its modifiers:
header. If the response doesn't include a `Warning` header with the exact header. If the response doesn't include a `Warning` header with the exact
text then the test fails. If the response includes `Warning` headers that text then the test fails. If the response includes `Warning` headers that
aren't expected then the test fails. aren't expected then the test fails.
* `// TESTRESPONSE`: Matches this snippet against the body of the response of * `[source,console-result]`: Matches this snippet against the body of the
the last test. If the response is JSON then order is ignored. If you add response of the last test. If the response is JSON then order is ignored. If
`// TEST[continued]` to the snippet after `// TESTRESPONSE` it will continue you add `// TEST[continued]` to the snippet after `[source,console-result]`
in the same test, allowing you to interleave requests with responses to check. it will continue in the same test, allowing you to interleave requests with
responses to check.
* `// TESTRESPONSE`: Explicitly marks a snippet as a test response even without
`[source,console-result]`. Similarly to `// TEST` this is mostly used for
its modifiers.
* You can't use `[source,console-result]` immediately after `// TESTSETUP`.
Instead, consider using `// TEST[continued]` or rearrange your snippets.
NOTE: Previously we only used `// TESTRESPONSE` instead of
`[source,console-result]` so you'll see that a lot in older branches but we
prefer `[source,console-result]` now.
* `// TESTRESPONSE[s/foo/bar/]`: Substitutions. See `// TEST[s/foo/bar]` for * `// TESTRESPONSE[s/foo/bar/]`: Substitutions. See `// TEST[s/foo/bar]` for
how it works. These are much more common than `// TEST[s/foo/bar]` because how it works. These are much more common than `// TEST[s/foo/bar]` because
they are useful for eliding portions of the response that are not pertinent they are useful for eliding portions of the response that are not pertinent
@ -62,8 +77,6 @@ for its modifiers:
"figures out" the path. This is especially useful for making sweeping "figures out" the path. This is especially useful for making sweeping
assertions like "I made up all the numbers in this example, don't compare assertions like "I made up all the numbers in this example, don't compare
them" which looks like `// TESTRESPONSE[s/\d+/$body.$_path/]`. them" which looks like `// TESTRESPONSE[s/\d+/$body.$_path/]`.
* You can't use `// TESTRESPONSE` immediately after `// TESTSETUP`. Instead,
consider using `// TEST[continued]` or rearrange your snippets.
* `// TESTRESPONSE[non_json]`: Add substitutions for testing responses in a * `// TESTRESPONSE[non_json]`: Add substitutions for testing responses in a
format other than JSON. Use this after all other substitutions so it doesn't format other than JSON. Use this after all other substitutions so it doesn't
make other substitutions difficult. make other substitutions difficult.
@ -98,7 +111,7 @@ endyaml
``` ```
This allows slightly more expressive testing of the snippets. Since that syntax This allows slightly more expressive testing of the snippets. Since that syntax
is not supported by CONSOLE the usual way to incorporate it is with a is not supported by `[source,console]` the usual way to incorporate it is with a
`// TEST[s//]` marker like this: `// TEST[s//]` marker like this:
``` ```

View File

@ -30,7 +30,7 @@ If no context is specified then this context is used by default.
Request: Request:
[source,js] [source,console]
---------------------------------------------------------------- ----------------------------------------------------------------
POST /_scripts/painless/_execute POST /_scripts/painless/_execute
{ {
@ -43,17 +43,15 @@ POST /_scripts/painless/_execute
} }
} }
---------------------------------------------------------------- ----------------------------------------------------------------
// CONSOLE
Response: Response:
[source,js] [source,console-result]
-------------------------------------------------- --------------------------------------------------
{ {
"result": "0.1" "result": "0.1"
} }
-------------------------------------------------- --------------------------------------------------
// TESTRESPONSE
===== Filter context ===== Filter context
@ -69,7 +67,7 @@ index:: The name of an index containing a mapping that is compatible with the do
*Example* *Example*
[source,js] [source,console]
---------------------------------------------------------------- ----------------------------------------------------------------
PUT /my-index PUT /my-index
{ {
@ -99,17 +97,15 @@ POST /_scripts/painless/_execute
} }
} }
---------------------------------------------------------------- ----------------------------------------------------------------
// CONSOLE
Response: Response:
[source,js] [source,console-result]
-------------------------------------------------- --------------------------------------------------
{ {
"result": true "result": true
} }
-------------------------------------------------- --------------------------------------------------
// TESTRESPONSE
===== Score context ===== Score context
@ -125,7 +121,7 @@ query:: If `_score` is used in the script then a query can specified that will b
*Example* *Example*
[source,js] [source,console]
---------------------------------------------------------------- ----------------------------------------------------------------
PUT /my-index PUT /my-index
{ {
@ -159,14 +155,12 @@ POST /_scripts/painless/_execute
} }
} }
---------------------------------------------------------------- ----------------------------------------------------------------
// CONSOLE
Response: Response:
[source,js] [source,console-result]
-------------------------------------------------- --------------------------------------------------
{ {
"result": 0.8 "result": 0.8
} }
-------------------------------------------------- --------------------------------------------------
// TESTRESPONSE

View File

@ -141,7 +141,7 @@ comma-separated list of patterns you want to allow, or prefix each pattern with
`+` or `-` to indicate whether it should be allowed or blocked. When a list is `+` or `-` to indicate whether it should be allowed or blocked. When a list is
specified, the default behaviour is to disallow. specified, the default behaviour is to disallow.
[source,js] [source,console]
-------------------------------------------------- --------------------------------------------------
PUT _cluster/settings PUT _cluster/settings
{ {
@ -164,7 +164,6 @@ PUT _cluster/settings
} }
} }
-------------------------------------------------- --------------------------------------------------
// CONSOLE
<1> Allow auto-creation of indices called `twitter` or `index10`, block the <1> Allow auto-creation of indices called `twitter` or `index10`, block the
creation of indices that match the pattern `index1*`, and allow creation of creation of indices that match the pattern `index1*`, and allow creation of