honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/build.gradle

387 lines
15 KiB
Groovy
Raw Normal View History

Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
apply plugin: 'elasticsearch.docs-test'
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
Fail build if new doc snippets aren't `// CONSOLE` This tracks the snippets that probably should be converted to `// CONSOLE` or `// TESTRESPONSE` and fails the build if the list of files with such snippets doesn't match the list in `docs/build.gradle`. Setting the file looks like ``` /* List of files that have snippets that probably should be converted to * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and * only remove entries from this list. When it is empty we'll remove it * entirely and have a party! There will be cake and everything.... */ buildRestTests.expectedUnconvertedCandidates = [ 'plugins/discovery-azure-classic.asciidoc', ... 'reference/search/suggesters/completion-suggest.asciidoc', ] ``` This list is in `build.gradle` because we expect it to be fairly temporary. In a few months we'll have converted all of the docs and won't ned it any more. From now on if you add now docs that contain a snippet that shows an interaction with elasticsearch you have three choices: 1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the responses. The build (specifically (`gradle docs:check`) will test that these interactions "work". If there isn't a `// TESTRESPONSE` snippet then "work" just means "Elasticsearch responds with a 200-level response code and no `WARNING` headers. This is way better than nothing. 2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with Elasticsearch. This should only be required for stuff like javascript source code or `curl` against an external service like AWS or GCE. The snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be tested. 3. Add `// TEST[skip:reason]` under the snippet. This will just skip the snippet in the test phase. This should really be reserved for snippets where we can't test them because they require an external service that we don't have at testing time. Please, please, please, please don't add more things to the list. After all, it sais there'll be cake when we remove it entirely! Relates to #18160
2016-09-09 11:10:13 -04:00
/* List of files that have snippets that probably should be converted to
* `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
* only remove entries from this list. When it is empty we'll remove it
* entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
'reference/aggregations/bucket/datehistogram-aggregation.asciidoc',
'reference/aggregations/bucket/daterange-aggregation.asciidoc',
'reference/aggregations/bucket/diversified-sampler-aggregation.asciidoc',
'reference/aggregations/bucket/filter-aggregation.asciidoc',
'reference/aggregations/bucket/geodistance-aggregation.asciidoc',
'reference/aggregations/bucket/geohashgrid-aggregation.asciidoc',
'reference/aggregations/bucket/global-aggregation.asciidoc',
'reference/aggregations/bucket/histogram-aggregation.asciidoc',
'reference/aggregations/bucket/iprange-aggregation.asciidoc',
'reference/aggregations/bucket/missing-aggregation.asciidoc',
'reference/aggregations/bucket/nested-aggregation.asciidoc',
'reference/aggregations/bucket/range-aggregation.asciidoc',
'reference/aggregations/bucket/reverse-nested-aggregation.asciidoc',
'reference/aggregations/bucket/sampler-aggregation.asciidoc',
'reference/aggregations/bucket/significantterms-aggregation.asciidoc',
'reference/aggregations/bucket/terms-aggregation.asciidoc',
'reference/aggregations/matrix/stats-aggregation.asciidoc',
'reference/aggregations/metrics/avg-aggregation.asciidoc',
'reference/aggregations/metrics/cardinality-aggregation.asciidoc',
'reference/aggregations/metrics/extendedstats-aggregation.asciidoc',
'reference/aggregations/metrics/geobounds-aggregation.asciidoc',
'reference/aggregations/metrics/geocentroid-aggregation.asciidoc',
'reference/aggregations/metrics/max-aggregation.asciidoc',
'reference/aggregations/metrics/min-aggregation.asciidoc',
'reference/aggregations/metrics/percentile-aggregation.asciidoc',
'reference/aggregations/metrics/percentile-rank-aggregation.asciidoc',
'reference/aggregations/metrics/scripted-metric-aggregation.asciidoc',
'reference/aggregations/metrics/stats-aggregation.asciidoc',
'reference/aggregations/metrics/sum-aggregation.asciidoc',
'reference/aggregations/metrics/tophits-aggregation.asciidoc',
'reference/aggregations/metrics/valuecount-aggregation.asciidoc',
'reference/aggregations/pipeline.asciidoc',
'reference/aggregations/pipeline/avg-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/bucket-script-aggregation.asciidoc',
'reference/aggregations/pipeline/bucket-selector-aggregation.asciidoc',
'reference/aggregations/pipeline/cumulative-sum-aggregation.asciidoc',
'reference/aggregations/pipeline/derivative-aggregation.asciidoc',
'reference/aggregations/pipeline/extended-stats-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/max-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/min-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/movavg-aggregation.asciidoc',
'reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/serial-diff-aggregation.asciidoc',
'reference/aggregations/pipeline/stats-bucket-aggregation.asciidoc',
'reference/aggregations/pipeline/sum-bucket-aggregation.asciidoc',
'reference/analysis/analyzers/lang-analyzer.asciidoc',
'reference/analysis/analyzers/pattern-analyzer.asciidoc',
'reference/analysis/charfilters/htmlstrip-charfilter.asciidoc',
'reference/analysis/charfilters/pattern-replace-charfilter.asciidoc',
'reference/analysis/tokenfilters/asciifolding-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/cjk-bigram-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/common-grams-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/compound-word-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/elision-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/hunspell-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/keep-types-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/keep-words-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/keyword-marker-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/keyword-repeat-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/limit-token-count-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/lowercase-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/pattern-capture-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/snowball-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/stemmer-override-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/stemmer-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/stop-tokenfilter.asciidoc',
'reference/analysis/tokenfilters/synonym-tokenfilter.asciidoc',
Synonym Graph Support (LUCENE-6664) (#21517) Integrate the patch from LUCENE-6664 into elasticsearch and add support for handling a graph token stream in match/multi-match queries. This fixes longstanding bugs with multi-token synonyms returning incorrect results with proximity queries.
2016-11-28 12:25:49 -05:00
'reference/analysis/tokenfilters/synonym-graph-tokenfilter.asciidoc',
Fail build if new doc snippets aren't `// CONSOLE` This tracks the snippets that probably should be converted to `// CONSOLE` or `// TESTRESPONSE` and fails the build if the list of files with such snippets doesn't match the list in `docs/build.gradle`. Setting the file looks like ``` /* List of files that have snippets that probably should be converted to * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and * only remove entries from this list. When it is empty we'll remove it * entirely and have a party! There will be cake and everything.... */ buildRestTests.expectedUnconvertedCandidates = [ 'plugins/discovery-azure-classic.asciidoc', ... 'reference/search/suggesters/completion-suggest.asciidoc', ] ``` This list is in `build.gradle` because we expect it to be fairly temporary. In a few months we'll have converted all of the docs and won't ned it any more. From now on if you add now docs that contain a snippet that shows an interaction with elasticsearch you have three choices: 1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the responses. The build (specifically (`gradle docs:check`) will test that these interactions "work". If there isn't a `// TESTRESPONSE` snippet then "work" just means "Elasticsearch responds with a 200-level response code and no `WARNING` headers. This is way better than nothing. 2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with Elasticsearch. This should only be required for stuff like javascript source code or `curl` against an external service like AWS or GCE. The snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be tested. 3. Add `// TEST[skip:reason]` under the snippet. This will just skip the snippet in the test phase. This should really be reserved for snippets where we can't test them because they require an external service that we don't have at testing time. Please, please, please, please don't add more things to the list. After all, it sais there'll be cake when we remove it entirely! Relates to #18160
2016-09-09 11:10:13 -04:00
'reference/analysis/tokenfilters/word-delimiter-tokenfilter.asciidoc',
'reference/cat/recovery.asciidoc',
'reference/cat/shards.asciidoc',
'reference/cat/snapshots.asciidoc',
[DOC] Add /cat/templates to list of excluded docs
2016-09-20 05:02:55 -04:00
'reference/cat/templates.asciidoc',
Fail build if new doc snippets aren't `// CONSOLE` This tracks the snippets that probably should be converted to `// CONSOLE` or `// TESTRESPONSE` and fails the build if the list of files with such snippets doesn't match the list in `docs/build.gradle`. Setting the file looks like ``` /* List of files that have snippets that probably should be converted to * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and * only remove entries from this list. When it is empty we'll remove it * entirely and have a party! There will be cake and everything.... */ buildRestTests.expectedUnconvertedCandidates = [ 'plugins/discovery-azure-classic.asciidoc', ... 'reference/search/suggesters/completion-suggest.asciidoc', ] ``` This list is in `build.gradle` because we expect it to be fairly temporary. In a few months we'll have converted all of the docs and won't ned it any more. From now on if you add now docs that contain a snippet that shows an interaction with elasticsearch you have three choices: 1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the responses. The build (specifically (`gradle docs:check`) will test that these interactions "work". If there isn't a `// TESTRESPONSE` snippet then "work" just means "Elasticsearch responds with a 200-level response code and no `WARNING` headers. This is way better than nothing. 2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with Elasticsearch. This should only be required for stuff like javascript source code or `curl` against an external service like AWS or GCE. The snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be tested. 3. Add `// TEST[skip:reason]` under the snippet. This will just skip the snippet in the test phase. This should really be reserved for snippets where we can't test them because they require an external service that we don't have at testing time. Please, please, please, please don't add more things to the list. After all, it sais there'll be cake when we remove it entirely! Relates to #18160
2016-09-09 11:10:13 -04:00
'reference/cat/thread_pool.asciidoc',
'reference/cluster/allocation-explain.asciidoc',
'reference/cluster/nodes-info.asciidoc',
'reference/cluster/nodes-stats.asciidoc',
'reference/cluster/pending.asciidoc',
'reference/cluster/reroute.asciidoc',
'reference/cluster/state.asciidoc',
'reference/cluster/stats.asciidoc',
'reference/cluster/tasks.asciidoc',
'reference/cluster/update-settings.asciidoc',
'reference/docs/bulk.asciidoc',
'reference/docs/delete-by-query.asciidoc',
'reference/docs/delete.asciidoc',
'reference/docs/index_.asciidoc',
'reference/docs/multi-get.asciidoc',
'reference/docs/multi-termvectors.asciidoc',
'reference/docs/reindex.asciidoc',
'reference/docs/termvectors.asciidoc',
'reference/docs/update-by-query.asciidoc',
'reference/docs/update.asciidoc',
'reference/index-modules/similarity.asciidoc',
'reference/index-modules/store.asciidoc',
'reference/index-modules/translog.asciidoc',
'reference/indices/analyze.asciidoc',
'reference/indices/flush.asciidoc',
'reference/indices/get-field-mapping.asciidoc',
'reference/indices/get-settings.asciidoc',
'reference/indices/put-mapping.asciidoc',
'reference/indices/recovery.asciidoc',
'reference/indices/segments.asciidoc',
'reference/indices/shadow-replicas.asciidoc',
'reference/indices/shard-stores.asciidoc',
'reference/indices/update-settings.asciidoc',
'reference/ingest/ingest-node.asciidoc',
'reference/mapping/dynamic/templates.asciidoc',
'reference/mapping/fields/all-field.asciidoc',
'reference/mapping/params/analyzer.asciidoc',
'reference/mapping/types/binary.asciidoc',
'reference/mapping/types/geo-point.asciidoc',
'reference/mapping/types/geo-shape.asciidoc',
'reference/mapping/types/ip.asciidoc',
'reference/mapping/types/nested.asciidoc',
'reference/mapping/types/object.asciidoc',
'reference/mapping/types/percolator.asciidoc',
'reference/modules/scripting/native.asciidoc',
'reference/modules/scripting/security.asciidoc',
'reference/modules/scripting/using.asciidoc',
'reference/modules/transport.asciidoc',
add basic docs
2017-01-05 10:10:34 -05:00
'reference/modules/cross-cluster-search.asciidoc', // this is hart to test since we need 2 clusters -- maybe we can trick it into referencing itself...
Fail build if new doc snippets aren't `// CONSOLE` This tracks the snippets that probably should be converted to `// CONSOLE` or `// TESTRESPONSE` and fails the build if the list of files with such snippets doesn't match the list in `docs/build.gradle`. Setting the file looks like ``` /* List of files that have snippets that probably should be converted to * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and * only remove entries from this list. When it is empty we'll remove it * entirely and have a party! There will be cake and everything.... */ buildRestTests.expectedUnconvertedCandidates = [ 'plugins/discovery-azure-classic.asciidoc', ... 'reference/search/suggesters/completion-suggest.asciidoc', ] ``` This list is in `build.gradle` because we expect it to be fairly temporary. In a few months we'll have converted all of the docs and won't ned it any more. From now on if you add now docs that contain a snippet that shows an interaction with elasticsearch you have three choices: 1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the responses. The build (specifically (`gradle docs:check`) will test that these interactions "work". If there isn't a `// TESTRESPONSE` snippet then "work" just means "Elasticsearch responds with a 200-level response code and no `WARNING` headers. This is way better than nothing. 2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with Elasticsearch. This should only be required for stuff like javascript source code or `curl` against an external service like AWS or GCE. The snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be tested. 3. Add `// TEST[skip:reason]` under the snippet. This will just skip the snippet in the test phase. This should really be reserved for snippets where we can't test them because they require an external service that we don't have at testing time. Please, please, please, please don't add more things to the list. After all, it sais there'll be cake when we remove it entirely! Relates to #18160
2016-09-09 11:10:13 -04:00
'reference/query-dsl/exists-query.asciidoc',
'reference/query-dsl/function-score-query.asciidoc',
'reference/query-dsl/geo-shape-query.asciidoc',
'reference/query-dsl/terms-query.asciidoc',
'reference/search/field-stats.asciidoc',
'reference/search/multi-search.asciidoc',
'reference/search/profile.asciidoc',
'reference/search/request/highlighting.asciidoc',
'reference/search/request/inner-hits.asciidoc',
'reference/search/request/rescore.asciidoc',
'reference/search/search-template.asciidoc',
]
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
integTest {
cluster {
setting 'script.inline', 'true'
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
setting 'script.stored', 'true'
Circuit break the number of inline scripts compiled per minute When compiling many dynamically changing scripts, parameterized scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>) should be preferred. This enforces a limit to the number of scripts that can be compiled within a minute. A new dynamic setting is added - `script.max_compilations_per_minute`, which defaults to 15. If more dynamic scripts are sent, a user will get the following exception: ```json { "error" : { "root_cause" : [ { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } ], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed_shards" : [ { "shard" : 0, "index" : "i", "node" : "a5V1eXcZRYiIk8lecjZ4Jw", "reason" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } } ], "caused_by" : { "type" : "general_script_exception", "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]", "caused_by" : { "type" : "circuit_breaking_exception", "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead", "bytes_wanted" : 0, "bytes_limit" : 0 } } }, "status" : 500 } ``` This also fixes a bug in `ScriptService` where requests being executed concurrently on a single node could cause a script to be compiled multiple times (many in the case of a powerful node with many shards) due to no synchronization between checking the cache and compiling the script. There is now synchronization so that a script being compiled will only be compiled once regardless of the number of concurrent searches on a node. Relates to #19396
2016-07-26 15:36:29 -04:00
setting 'script.max_compilations_per_minute', '1000'
Disable regexes by default in painless Adds a new node level, non-dynamic setting, `script.painless.regex.enabled` can be used to enable regexes. Closes #20397
2016-09-12 12:06:29 -04:00
/* Enable regexes in painless so our tests don't complain about example
* snippets that use them. */
setting 'script.painless.regex.enabled', 'true'
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
Closure configFile = {
extraConfigFile it, "src/test/cluster/config/$it"
}
configFile 'scripts/my_script.js'
configFile 'scripts/my_script.py'
Update docs for scripted metric agg Now that the default language is painless the examples didn't work at all. This fixes them. Closes #21536
2016-11-15 11:45:54 -05:00
configFile 'scripts/my_init_script.painless'
configFile 'scripts/my_map_script.painless'
configFile 'scripts/my_combine_script.painless'
configFile 'scripts/my_reduce_script.painless'
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
configFile 'userdict_ja.txt'
configFile 'KeywordTokenizer.rbbi'
Reindex from remote This adds a remote option to reindex that looks like ``` curl -POST 'localhost:9200/_reindex?pretty' -d'{ "source": { "remote": { "host": "http://otherhost:9200" }, "index": "target", "query": { "match": { "foo": "bar" } } }, "dest": { "index": "target" } }' ``` This reindex has all of the features of local reindex: * Using queries to filter what is copied * Retry on rejection * Throttle/rethottle The big advantage of this version is that it goes over the HTTP API which can be made backwards compatible. Some things are different: The query field is sent directly to the other node rather than parsed on the coordinating node. This should allow it to support constructs that are invalid on the coordinating node but are valid on the target node. Mostly, that means old syntax.
2016-05-19 12:40:22 -04:00
// Whitelist reindexing from the local node so we can test it.
Add "simple match" support for reindex-from-remote whitelist This allows you to whitelist `localhost:*` or `127.0.10.*:9200`. It explicitly checks for patterns like `*` in the whitelist and refuses to start if the whitelist would match everything. Beyond that the user is on their own designing a secure whitelist.
2016-10-18 15:00:49 -04:00
setting 'reindex.remote.whitelist', '127.0.0.1:*'
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
}
}
// Build the cluser with all plugins
Build: Simplify plugin configuration for rest tests This change removes the multiple ways that plugins can be added to the integ test cluster. It also removes the use of the default configuration, and instead adds a zip configuration to all plugins. This will enable using project substitutions with plugins, which must be done with the default configuration.
2016-07-15 17:34:21 -04:00
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
project.rootProject.subprojects.findAll { it.parent.path == ':plugins' }.each { subproj ->
/* Skip repositories. We just aren't going to be able to test them so it
* doesn't make sense to waste time installing them. */
if (subproj.path.startsWith(':plugins:repository-')) {
return
}
Build: Simplify plugin configuration for rest tests This change removes the multiple ways that plugins can be added to the integ test cluster. It also removes the use of the default configuration, and instead adds a zip configuration to all plugins. This will enable using project substitutions with plugins, which must be done with the default configuration.
2016-07-15 17:34:21 -04:00
subproj.afterEvaluate { // need to wait until the project has been configured
integTest {
cluster {
plugin subproj.path
}
Test docs for plugins We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
2016-05-13 16:15:51 -04:00
}
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
}
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
}
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
buildRestTests.docs = fileTree(projectDir) {
// No snippets in here!
exclude 'build.gradle'
// That is where the snippets go, not where they come from!
exclude 'build'
// This file simply doesn't pass yet. We should figure out how to fix it.
exclude 'reference/modules/snapshots.asciidoc'
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
}
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
Closure setupTwitter = { String name, int count ->
buildRestTests.setups[name] = '''
Consolify term and phrase suggester docs This includes a working example of reverse filters to support correcting prefix errors.
2016-07-22 18:51:36 -04:00
- do:
Add CONSOLE to more docs Relates to #18160
2016-09-01 17:08:18 -04:00
indices.create:
index: twitter
body:
settings:
number_of_shards: 1
number_of_replicas: 1
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
- do:
Add CONSOLE to more docs Relates to #18160
2016-09-01 17:08:18 -04:00
bulk:
index: twitter
type: tweet
refresh: true
body: |'''
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
for (int i = 0; i < count; i++) {
[docs] Add // CONSOLE to validate and uri-request Two of the snippets in validate weren't working properly so they are marked as skip and linked to this: https://github.com/elastic/elasticsearch/issues/18254 We didn't properly handle empty parameter values. We were sending them as the literal string "null". Now we do better and send them as the empty string.
2016-05-10 18:29:56 -04:00
String user, text
if (i == 0) {
user = 'kimchy'
text = 'trying out Elasticsearch'
} else {
user = 'test'
text = "some message with the number $i"
}
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
buildRestTests.setups[name] += """
Add CONSOLE to more docs Relates to #18160
2016-09-01 17:08:18 -04:00
{"index":{"_id": "$i"}}
{"user": "$user", "message": "$text", "date": "2009-11-15T14:12:12", "likes": $i}"""
Generate and run tests from the docs Adds infrastructure so `gradle :docs:check` will extract tests from snippets in the documentation and execute the tests. This is included in `gradle check` so it should happen on CI and during a normal build. By default each `// AUTOSENSE` snippet creates a unique REST test. These tests are executed in a random order and the cluster is wiped between each one. If multiple snippets chain together into a test you can annotate all snippets after the first with `// TEST[continued]` to have the generated tests for both snippets joined. Snippets marked as `// TESTRESPONSE` are checked against the response of the last action. See docs/README.asciidoc for lots more. Closes #12583. That issue is about catching bugs in the docs during build. This catches *some* bugs in the docs during build which is a good start.
2016-04-29 10:42:03 -04:00
}
}
Build a plugin for testing docs This makes it much easier to apply to other projects. Fixes to doc tests infrastructure: * Fix comparing lists. Was totally broken. * Fix order of actual vs expected parameters. * Allow multiple `// TESTRESPONSE` lines with substitutions to join into one big list of subtitutions. This makes lets the docs look tidier. * Exclude build from snippet scanning * Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-05 16:46:40 -04:00
setupTwitter('twitter', 5)
setupTwitter('big_twitter', 120)
Add CONSOLE to more docs Relates to #18160
2016-09-01 17:08:18 -04:00
setupTwitter('huge_twitter', 1200)
Reindex from remote This adds a remote option to reindex that looks like ``` curl -POST 'localhost:9200/_reindex?pretty' -d'{ "source": { "remote": { "host": "http://otherhost:9200" }, "index": "target", "query": { "match": { "foo": "bar" } } }, "dest": { "index": "target" } }' ``` This reindex has all of the features of local reindex: * Using queries to filter what is copied * Retry on rejection * Throttle/rethottle The big advantage of this version is that it goes over the HTTP API which can be made backwards compatible. Some things are different: The query field is sent directly to the other node rather than parsed on the coordinating node. This should allow it to support constructs that are invalid on the coordinating node but are valid on the target node. Mostly, that means old syntax.
2016-05-19 12:40:22 -04:00
buildRestTests.setups['host'] = '''
# Fetch the http host. We use the host of the master because we know there will always be a master.
- do:
cluster.state: {}
- set: { master_node: master }
- do:
nodes.info:
metric: [ http ]
- is_true: nodes.$master.http.publish_address
- set: {nodes.$master.http.publish_address: host}
'''
Add `// CONSOLE` to much of pipeline agg docs Most of the examples in the pipeline aggregation docs use a small "sales" test data set and I converted all of the examples that use it to `// CONSOLE`. There are still a bunch of snippets in the pipeline aggregation docs that aren't `// CONSOLE` so they aren't tested. Most of them are "this is the most basic form of this aggregation" so they are more immune to errors and bit rot then the examples that I converted. I'd like to do something with them as well but I'm not sure what. Also, the moving average docs and serial diff docs didn't get a lot of love from this pass because they don't use the test data set or follow the same general layout. Relates to #18160
2016-08-12 18:42:19 -04:00
Update docs for scripted metric agg Now that the default language is painless the examples didn't work at all. This fixes them. Closes #21536
2016-11-15 11:45:54 -05:00
// Used by scripted metric docs
buildRestTests.setups['ledger'] = '''
- do:
indices.create:
index: ledger
body:
settings:
number_of_shards: 2
number_of_replicas: 1
mappings:
sale:
properties:
type:
type: keyword
amount:
type: double
- do:
bulk:
index: ledger
type: item
refresh: true
body: |
{"index":{}}
{"date": "2015/01/01 00:00:00", "amount": 200, "type": "sale", "description": "something"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "amount": 10, "type": "expense", "decription": "another thing"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "amount": 150, "type": "sale", "description": "blah"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "amount": 50, "type": "expense", "description": "cost of blah"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "amount": 50, "type": "expense", "description": "advertisement"}'''
Add `// CONSOLE` to much of pipeline agg docs Most of the examples in the pipeline aggregation docs use a small "sales" test data set and I converted all of the examples that use it to `// CONSOLE`. There are still a bunch of snippets in the pipeline aggregation docs that aren't `// CONSOLE` so they aren't tested. Most of them are "this is the most basic form of this aggregation" so they are more immune to errors and bit rot then the examples that I converted. I'd like to do something with them as well but I'm not sure what. Also, the moving average docs and serial diff docs didn't get a lot of love from this pass because they don't use the test data set or follow the same general layout. Relates to #18160
2016-08-12 18:42:19 -04:00
// Used by pipeline aggregation docs
buildRestTests.setups['sales'] = '''
- do:
indices.create:
index: sales
body:
settings:
number_of_shards: 2
number_of_replicas: 1
mappings:
sale:
properties:
type:
type: keyword
- do:
bulk:
index: sales
type: sale
refresh: true
body: |
{"index":{}}
{"date": "2015/01/01 00:00:00", "price": 200, "type": "hat"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "price": 200, "type": "t-shirt"}
{"index":{}}
{"date": "2015/01/01 00:00:00", "price": 150, "type": "bag"}
{"index":{}}
{"date": "2015/02/01 00:00:00", "price": 50, "type": "hat"}
{"index":{}}
{"date": "2015/02/01 00:00:00", "price": 10, "type": "t-shirt"}
{"index":{}}
{"date": "2015/03/01 00:00:00", "price": 200, "type": "hat"}
{"index":{}}
{"date": "2015/03/01 00:00:00", "price": 175, "type": "t-shirt"}'''
CONSOLEify remaining getting-started docs Moves some test data used by those docs into the Elasticsearch repository so we can use it when we test the docs during the build. Relates to #18160
2016-10-02 23:16:21 -04:00
// Dummy bank account data used by getting-started.asciidoc
buildRestTests.setups['bank'] = '''
- do:
bulk:
index: bank
type: account
refresh: true
body: |
#bank_data#
'''
/* Load the actual accounts only if we're going to use them. This complicates
* dependency checking but that is a small price to pay for not building a
* 400kb string every time we start the build. */
File accountsFile = new File("$projectDir/src/test/resources/accounts.json")
buildRestTests.inputs.file(accountsFile)
buildRestTests.doFirst {
String accounts = accountsFile.getText('UTF-8')
// Indent like a yaml test needs
accounts = accounts.replaceAll('(?m)^', ' ')
buildRestTests.setups['bank'] =
buildRestTests.setups['bank'].replace('#bank_data#', accounts)
}
Add RangeFieldMapper for numeric and date range types Lucene 6.2 added index and query support for numeric ranges. This commit adds a new RangeFieldMapper for indexing numeric (int, long, float, double) and date ranges and creating appropriate range and term queries. The design is similar to NumericFieldMapper in that it uses a RangeType enumerator for implementing the logic specific to each type. The following range types are supported by this field mapper: int_range, float_range, long_range, double_range, date_range. Lucene does not provide a DocValue field specific to RangeField types so the RangeFieldMapper implements a CustomRangeDocValuesField for handling doc value support. When executing a Range query over a Range field, the RangeQueryBuilder has been enhanced to accept a new relation parameter for defining the type of query as one of: WITHIN, CONTAINS, INTERSECTS. This provides support for finding all ranges that are related to a specific range in a desired way. As with other spatial queries, DISJOINT can be achieved as a MUST_NOT of an INTERSECTS query.
2016-09-16 09:50:56 -04:00
buildRestTests.setups['range_index'] = '''
- do :
indices.create:
index: range_index
body:
settings:
number_of_shards: 2
number_of_replicas: 1
mappings:
my_type:
properties:
expected_attendees:
type: integer_range
time_frame:
type: date_range
format: yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis
- do:
bulk:
index: range_index
type: my_type
refresh: true
body: |
{"index":{"_id": 1}}
{"expected_attendees": {"gte": 10, "lte": 20}, "time_frame": {"gte": "2015-10-31 12:00:00", "lte": "2015-11-01"}}'''
Resolve index names in indices_boost This change allows specifying alias/wildcard expression in indices_boost. And added another format for specifying indices_boost. It accepts array of index name and boost pair. If an index is included in multiple aliases/wildcard expressions, the first match will be used. With new format, old format is marked as deprecated. Closes #4756
2016-11-07 03:20:06 -05:00
// Used by index boost doc
buildRestTests.setups['index_boost'] = '''
- do:
indices.create:
index: index1
- do:
indices.create:
index: index2
- do:
indices.put_alias:
index: index1
name: alias1
'''