A request level flag, defaults to be unset, to control the query cache. When not set, it defaults to the index level settings, when explicitly set, will override the index level setting
closes#7167
In the case of inserts the UpdateHelper class will now allow the script used to apply updates to run on the upsert doc provided by clients. This allows the logic for managing the internal state of the data item to be managed by the script and is not reliant on clients performing the initialisation of data structures managed by the script.
Closes#7143
Implements a new Exists API allowing users to do fast exists check on any matched documents for a given query.
This API should be faster then using the Count API as it will:
- early terminate the search execution once any document is found to exist
- return the response as soon as the first shard reports matched documents
closes#6995
This commit adds the ability to force blocking on the flush operaition
to make sure all files have been written and synced to disk. Without
this option a flush might be executing at the same time causing the
current flush to fail and return before all files being synced.
Closes#6996
This change allow elasticsearch users to store scripts and templates in an index for use at search time.
Scripts/Templates are stored in the .scripts index. The type of the events is set to the script language.
Templates use the mustache language so their type is be "mustache".
Adds the concept of a script type to calls to the ScriptService types are INDEXED,INLINE,FILE.
If a script type of INDEXED is supplied the script will be attempted to be loaded from the indexed, FILE will
look in the file cache and INLINE will treat the supplied script argument as the literal script.
REST endpoints are provided to do CRUD operations as is a java client library.
All query dsl points have been upgraded to allow passing in of explicit script ids and script file names.
Backwards compatible behavior has been preserved so this shouldn't break any existing querys that expect to
pass in a filename as the script/template name. The ScriptService will check the disk cache before parsing the
script.
Closes#5921#5637#5484
Concrete indices is now called multiple times when needed instead of changing what's inside the incoming request with the concrete indices. Ideally we want to keep the original aliases or indices or wildcard expressions in the request.
Also made sure that the check blocks is done against the concrete indices, which wasn't the case for delete index, delete mapping, open index, close index, types exists and indices exists.
Closes#6694Closes#6777
This reverts:
"Test: Temporarily change delete/put_mapping to wait for green": commit e408f8f638c2dd97a3ec86c8a9ac940f43ab37a0.
"[TEST] wait for green to update mapping": commit b3641a2ee6eb23318d49f5f04b39149e70c2b65b.
Commit fbd7c9aa5d introduced a regression that caused
the min_doc_count to be equal to the number of documents in the
background set. As a result no buckets were built when the
response for significant terms was created.
This only affected the final XContent response.
closes#6535
Bugs:
* "groups" and "types" were being ignored
* "completion_fields" as wildcards were not being resolved to fieldnames
Enhancements:
* Made "groups" and "types" support wildcards
* Added missing tests
Closes#6390
The put index template api supports the create parameter (defaults to false), which tells whether the template can replace an existing one with same name or not. Unified its behaviour between PUT and POST method, whereas the POST would previously force create to true.
Added create parameter to the rest spec (was missing before) and a REST test for create true scenario.
Adding the second index might cause relocation of existing shards depending on the number of nodes available, let's wait for relocation to be finished before going ahead and checking the cat shards outpu
Until now all version types have officially required the version to be a positive long number. Despite of this has being documented, ES versions <=1.0 did not enforce it when using the `external` version type. As a result people have succesfully indexed documents with 0 as a version. In 1.1. we introduced validation checks on incoming version values and causing indexing request to fail if the version was set to 0. While this is strictly speaking OK, we effectively have a situation where data already indexed does not match the version invariant.
To be lenient and adhere to spirit of our data backward compatibility policy, we have decided to allow 0 as a valid external version type. This is somewhat complicated as 0 is also the internal value of `MATCH_ANY`, which indicates requests should succeed regardles off the current doc version. To keep things simple, this commit changes the internal value of `MATCH_ANY` to `-3` for all version types.
Since we're doing this in a minor release (and because versions are stored in the transaction log), the default `internal` version type still accepts 0 as a `MATCH_ANY` value. This is not a problem for other version types as `MATCH_ANY` doesn't make sense in that context.
Closes#5662
Made sure that a match_all query is used when no query is specified and ensure no NPE is thrown either.
Also used the same code path as the search api to ensure that alias filters are taken into account, same for type filters.
Closes#6111Closes#6112Closes#6116
In our REST tests we already have support for features and skip sections that allow to skip tests if a feature is not supported.
We can then add a skip section based on the benchmark feature to the benchmark tests and execute them only when they are supported, knowing that they need at least a node with node.bench settings within the cluster. We can check that this requirement is met by calling the nodes info api.
This way we can dynamically decide whether to execute those tests or not and we don't need to have a node.bench around all the time. In fact, given that the REST tests use the GLOBAL cluster, we want to be able to randomize settings as much as possible and run tests against default settings as well. Also, this mechanism can be easily supported by the external cluster implementation that is used during the release process.
Introduced ability to disable benchmark nodes which is needed by BenchmarkNegativeTest.
This fixes a stack overflow in the test for the _cat/recovery API.
The regular expression that tests the response body was modified to
handle large responses properly.
The rest test for _cat/allocation was failing due to a regular
expression not accounting for space-padded right-justified text.
Also added Improvements to regular expressions to be smarter about optional values
and to use '+' instead of '*' where applicable.
A bad/non-existing scroll ID used to return a 200, however a 404 might be more useful.
Also, this PR returns the right Exception (SearchContextMissingException) in the Java API.
Additionally: Added StatusToXContent interface and RestStatusToXContentListener listener, so
the appropriate RestStatus can be returned
Closes#5729
The possibility of filtering for index templates in the cluster state API
had been introduced before there was a dedicated index templates API. This
commit removes this support from the cluster state API, as it was not really
clean, requiring you to specify the metadata and the index templates.
Closes#4954
The regex tests are formatted with blocks for readability. Previously,
they were formatted using folded style blocks (e.g. using `>`). Folded
blocks convert newlines into spaces. This is problematic for our regex,
since comments can only be terminated with a newline.
Effectively, anything after a comment will be commented out, making many
of the regex "silently pass".
This commit replaces them with scalar-style blocks (e.g. using `|`), which
treats newlines as significant, and thus correctly terminates comments
inside the regex.
Also fixes a regex test (`cat.thread_pool/10_basic.yaml`) that started
to fail after the block was fixed. The test was missing a `\s+` before
the closing newline.
The default number of clients nodes is randomized between 0 and 1, applied to all cluster scopes (global, suite and test). Can be changed through the newly added `@ClusterScope#numClientNodes`.
In our tests we currently refer to nodes in a generic way. All the tests that either stop or start nodes rely on the fact that those nodes hold data though. Made that clearer as that becomes more important when introducing other types of nodes within the test cluster. Reflected this by adapting and renaming the following methods in `TestCluster`:
- ensureAtLeastNumNodes to ensureAtLeastNumDataNodes
- ensureAtMostNumNodes to ensureAtMostNumDataNodes
- stopRandomNode to stopRandomDataNode
and the following ones in `ElasticsearchIntegrationTest`:
- allowNodes to allowDataNodes
- dataNodes to numDataNodes.
- @ClusterScope#numNodes to numDataNodes
- @ClusterScope#minNumNodes to minNumDataNodes
- @ClusterScope#maxNumNodes to maxNumDataNodes
Added facilities to be able to deal with data nodes specifically, like for instance retrieve a client to a data node, or retrieve an instance of a class through guice only from data nodes.
Adapted existing tests to successfully run although there's a node client around.
Fixed _cat/allocation REST tests to make disk.total, disk.avail and disk.percent optional as client nodes won't return that info.
Closes#5949
Separate version check logic for reads and writes for all version types, which allows different behavior in these cases.
Change `VersionType.EXTERNAL` & `VersionType.EXTERNAL_GTE` to behave the same as `VersionType.INTERNAL` for read operations.
The previous behavior was fit for writes but is useless in reads.
This commit also makes the usage of `EXTERNAL` & `EXTERNAL_GTE` in the update api raise a validation error as it make cause data to
be lost.
Closes#5663 , Closes#5661, Closes#5929
Currently the parser accepts queries like
```
"query" : {
"any_query": {
...
},
"any_field_name":...
}
```
The "any_field_name" is silently ignored. However, this also causes the parser
not to move to the next closing bracket which in turn can lead to additional query
paremters being ignored such as "fields", "highlight",...
This was the case in issue #4895
closes issue #4895
This is a fix for a bug whereby a cluster that has no nodes started with
-Des.node.bench=true will cause clients to hang if they attempt to
submit a benchmark.
Also adds REST tests to validate fix
Closes#5754
Add an API endpoint at /_bench for submitting, listing, and aborting
search benchmarks. This API can be used for timing search requests,
subject to various user-defined settings.
Benchmark results provide summary and detailed statistics on such
values as min, max, and mean time. Values are reported per-node so that
it is easy to spot outliers. Slow requests are also reported.
Long running benchmarks can be viewed with a GET request, or aborted
with a POST request.
Benchmark results are optionally stored in an index for subsequent
analysis.
Closes#5407
ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner
We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.
Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)
Closes#5654
The default precision was way too exact and could lead people to
think that geo context suggestions are not working. This patch now
requires you to set the precision in the mapping, as elasticsearch itself
can never tell exactly, what the required precision for the users
suggestions are.
Closes#5621
A bunch of minor fixes have been included here, especially due
to wrongly parsed mappings. Also using assertions resulted in an
NPE because they were disabled in the distribution.
Closes#5525
Some tests disable refresh and verify that documents cannot be found without an explicit refresh. With at least one replica and waiting for yellow though, it can happen that we start indexing docs when the replicas are not ready yet, and as soon as they become ready they get refreshed, causing the newly added docs to be unexpectedly found. Solution is to disable replicas for these specific tests.
The default mustache engine was using HTML escaping which breaks queries
if used with JSON etc. This commit adds escaping for:
```
\b Backspace (ascii code 08)
\f Form feed (ascii code 0C)
\n New line
\r Carriage return
\t Tab
\v Vertical tab
\" Double quote
\\ Backslash
```
Closes#5473
Adds a new API endpoint at /_recovery as well as to the Java API. The
recovery API allows one to see the recovery status of all shards in the
cluster. It will report on percent complete, recovery type, and which
files are copied.
Closes#4637
If we want to have a full picture of versions running in a cluster, we need to add a `_cat/plugins` endpoint.
Response could look like:
```sh
% curl es2:9200/_cat/plugins?v
node component version type url desc
es1 mapper-attachments 1.7.0 j Adds the attachment type allowing to parse difference attachment formats
es1 lang-javascript 1.4.0 j JavaScript plugin allowing to add javascript scripting support
es1 analysis-smartcn 1.9.0 j Smart Chinese analysis support
es1 marvel 1.1.0 j/s http://localhost:9200/_plugins/marvel Elasticsearch Management & Monitoring
es1 kopf 0.5.3 s http://localhost:9200/_plugins/kopf kopf - simple web administration tool for ElasticSearch
es2 mapper-attachments 2.0.0.RC1 j Adds the attachment type allowing to parse difference attachment formats
es2 lang-javascript 2.0.0.RC1 j JavaScript plugin allowing to add javascript scripting support
es2 analysis-smartcn 2.0.0.RC1 j Smart Chinese analysis support
```
Closes#4824.
The clients return an exception in case of failure and not the whole json response containing failures, thus this tests can only work with the Java REST tests runner
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time. Templates may be both quoted,
non-quoted and referencing templates stored in config/scripts/*.mustache by file
name.
See docs/reference/query-dsl/queries/template-query.asciidoc for templating
examples.
Implementation detail: mustache itself is being shaded as it depends directly on
guava - so having it marked optional but included in the final distribution
raises chances of version conflicts downstream.
Fixes#4879
It is now possible to specify aliases during index creation:
curl -XPUT 'http://localhost:9200/test' -d '
{
"aliases" : {
"alias1" : {},
"alias2" : {
"filter" : { "term" : {"field":"value"}}
}
}
}'
Closes#4920
In order to be consistent (and because in 1.0 we switched from
parameter driven information to specifzing the metrics as part of the URI)
this patch moves from 'plugin' to 'plugins' in the Nodes Info API.
REST tests get run against either 1 node or multiple nodes. Wait for yellow with replicas>0 is not enough when running against multiple nodes as replicas shard might get initialized during testing, which can cause timing issues.
Replaced also wait for yellow with wait for green when using no replicas.
get_source/60_realtime_refresh tests per shard refresh using refresh:true and realtime:true in get api. We might run into troubles though if we have a replica that gets initialized after a doc was indexed without a refresh, as that doc will be found when searching against that specific replica shard (as a refresh happens automatically before a replica gets exposed as started).
delete/50_refresh tests per shard refresh using refresh:true in delete api. We might run into troubles though if we have a replica that gets initialized after a doc was indexed and deleted, without a refresh, as that doc won't be found when searching against that specific replica shard (as a refresh happens automatically before a replica gets exposed as started).
The last response body gets now always stashed in the REST tests and can be retrieved via `$body`. This implies that not only expected values can be retrieved from the stashed values, but actual values as well.
Added support for regular expressions to `match` assertion, using `Pattern.COMMENTS` flag for better readability through new custom hamcrest matcher (adopted in do section as well). Functionality added through new feature called `regex` that needs to be mentioned in the skip sections whenever needed till all the runners support it.
Added also example tests for cat count api
As we have different runners for the REST tests we need a mechanism that allows us to add features to any of them without breaking all others builds.
The idea is to name a feature and temporarily use skip sections that mention the required new features, so that runners that don't support it will skip the test.
Added support for `features` field in skip section.
Added `Features` class that contains a static list of the features supported by the runner. If a feature mentioned in a skip section is not listed here, the test will be skipped.
If a get field mapping request is issued, and all but the field can be
found, the response should return an empty JSON object instead of a 404.
Closes#4738
In order to make sure, that only the requested data is returned to the client,
a couple of fixes have been applied in the ClusterState.toXContent() method.
Also some tests were added to the yaml test suite
Closes#4885
Added base TransportAction class for master read operations that execute locally or not depending on the request class (local flag).
Added support for local flag where missing, in a backwards compatible manner:
- IndicesExistsRequest
- GetAliasesRequest (get alias api, aliases exist api)
- TypesExistsRequest
- GetIndexTemplatesRequest (get template, template exists)
- GetSettingsRequest
- GetRepositoriesRequest
- PendingClusterTasks
Added parsing of the local flag where missing in Rest*Action.
Updated SPEC adding local flag param where missing and added REST tests that contain use of the local flag where it was just added.
Closes#3345
Including:
* change some string params to list
* make some params or bodies required
* removed incorrect URLs
* removed incorrect params
* Fixed name of pending_tasks
Although the empty paths are still registered in the java RestActions, that is only to return a meaningful error in case the index is not provided.
Index/indices are now mandatory.
The get field mapping API now includes a mappings element after the index in its JSON
Added more consistent endpoint /{index}/_mapping/{type}/field/{fields}
and added endpoint /_mapping/{type}/field/{fields}
which are also used in tests
Added rest spec tests for wildcards and _all
Relates #4071
NOTE: This is not yet complete for 1.0. We need to return an empty JSON document instead
of a 404 if the field of an existing index and type is not found. However this is not
possible with the current data structure being returned. Needs to be finished for 1.0.
* Made GET mappings consistent, supporting
* /{index}/_mappings/{type}
* /{index}/_mapping/{type}
* /_mapping/{type}
* Added "mappings" in the JSON response to align it with other responses
* Made GET warmers consistent, support /{index}/_warmers/{type} and /_warmer, /_warner/{name}
as well as wildcards and _all notation
* Made GET aliases consistent, support /{index}/_aliases/{name} and /_alias, /_aliases/{name}
as well as wildcards and _all notation
* Made GET settings consistent, added /{index}/_setting/{name}, /_settings/{name}
as well as supportings wildcards in settings name
* Returning empty JSON instead of a 404, if a specific warmer/
setting/alias/type is missing
* Added a ton of spec tests for all of the above
* Added a couple of more integration tests for several features
Relates #4071
See issue #4071
PUT options for _mapping:
Single type can now be added with
`[PUT|POST] {index|_all|*|regex|blank}/[_mapping|_mappings]/type`
and
`[PUT|POST] {index|_all|*|regex|blank}/type/[_mapping|_mappings]`
PUT options for _warmer:
PUT with a single warmer can now be done with
`[PUT|POST] {index|_all|*|prefix*|blank}/{type|_all|*|prefix*|blank}/[_warmer|_warmers]/warmer_name`
PUT options for _alias:
Single alias can now be PUT with
`[PUT|POST] {index|_all|*|prefix*|blank}/[_alias|_aliases]/alias`
DELETE options _mapping:
Several mappings can be deleted at once by defining several indices and types with
`[DELETE] /{index}/{type}`
`[DELETE] /{index}/{type}/_mapping`
`[DELETE] /{index}/_mapping/{type}`
where
`index= * | _all | glob pattern | name1, name2, …`
`type= * | _all | glob pattern | name1, name2, …`
Alternatively, the keyword `_mapings` can be used.
DELETE options for _warmer:
Several warmers can be deleted at once by defining several indices and names with
`[DELETE] /{index}/_warmer/{type}`
where
`index= * | _all | glob pattern | name1, name2, …`
`type= * | _all | glob pattern | name1, name2, …`
Alternatively, the keyword `_warmers` can be used.
DELETE options for _alias:
Several aliases can be deleted at once by defining several indices and names with
`[DELETE] /{index}/_alias/{type}`
where
`index= * | _all | glob pattern | name1, name2, …`
`type= * | _all | glob pattern | name1, name2, …`
Alternatively, the keyword `_aliases` can be used.
Java Builder apis drop old “len” methods in favour of new “length”
Rest APIs support both old “len: and new “length” forms using new ParseField class to a) provide compiler-checked consistency between Builder and Parser classes and
b) a common means of handling deprecated syntax in the DSL.
Documentation and rest specs only document the new “*length” forms
Closes#4083
When upgrading to ES 1.0 the existing mappings with a multi-field type automatically get replaced to a core field with the new `fields` option.
If a `multi_field` type-ed field doesn't have a main / default field, a default field will be chosen for the multi fields syntax. The new main field type
will be equal to the first `multi_field` fields' field or type string if no fields have been configured for the `multi_field` field and in both cases
the default index will not be indexed (`index=no` is set on the default field).
If a `multi_field` typed field has a default field, that field will replace the `multi_field` typed field.
Closes to #4521
If a type or path is missing in the REST test yaml file, it is
automatically replaced with _all. This makes it hard to test changes
in the api, for example adding the possibility to leave the index
blank in addition to _all and * in the uri.
closes#4657
Named wildcards were not always properly replaced with proper values by PathTrie.
Delete index (curl -XDELETE localhost:9200/*) worked anyway as the named wildcard is the last path element (and even if {index} didn't get replaced with '*', the empty string would have mapped to all indices anyway). When the named wildcard wasn't the last path element (e.g. curl -XPOST localhost:29200/*/_close), the variable didn't get replaced with the current '*' value, but with the empty string, which leads to an error as empty index is not allowed by open/close index.
Closes#4564
The new internal get index settings api is more efficient when it comes to sending the index settings from the master to the client via the
Also the get index settings support now all the indices options.
Closes#4620
Important: This breaks backwards compatibility with 0.90
* Removed endpoints: /_cluster/nodes, /_cluster/nodes/nodeId1,nodeId2
* Disallow usage of parameters, but make required metrics part of URI
* Changed NodesInfoRequest to return everything by default
* Fixed NPE in NodesInfoResponse
Closes#4055
Currently there are two get aliases apis that both have the same functionality, but have a different response structure. The reason for having 2 apis is historic.
The GET _alias api was added in 0.90.x and is more efficient since it only sends the needed alias data from the cluster state between the master node and the node that received the request. In the GET _aliases api the complete cluster state is send to the node that received the request and then the right information is filtered out and send back to the client.
The GET _aliases api should be removed in favour for the alias api
Closes to #4539
* `ignore_unavailable` - Controls whether to ignore if any specified indices are unavailable, this includes indices that don't exist or closed indices. Either `true` or `false` can be specified.
* `allow_no_indices` - Controls whether to fail if a wildcard indices expressions results into no concrete indices. Either `true` or `false` can be specified. For example if the wildcard expression `foo*` is specified and no indices are available that start with `foo` then depending on this setting the request will fail. This setting is also applicable when `_all`, `*` or no index has been specified.
* `expand_wildcards` - Controls to what kind of concrete indices wildcard indices expression expand to. If `open` is specified then the wildcard expression if expanded to only open indices and if `closed` is specified then the wildcard expression if expanded only to closed indices. Also both values (`open,closed`) can be specified to expand to all indices.
Closes to #4436
This change make single shard requests fail when no routing is specified and routing has been configured to be required in the mapping. Thi
Closes#4506
The following APIs now accept the query in a top level `query` field like:
* delete_by_query
* validate_query
* count
These APIs used to accept the query directly in the request body which was inconsistent with the search and explain APIs. For this reason t
Closes#4074
This should make easier any potential expansion such as:
client.update('index', 'type', 41, {...}, ignore: 409
Also restructured and renamed the test suite a bit as part of the commit.
---
This reverts commit 0b142cc8812d8d090e48ea0ad89c007767870b91, "Removed client-specific `ignore_missing` parameter".
* Fixed error in `skip` explanation for "get type" test
* Separated the tests for getting index/type mapping
I vote for removing the "get type" test entirely, since it just tests ancient versions.
When the runner contract is specified as "run all the tests in the file in sequence,
resetting cluster after each file", it makes is hard to translate the concept into
regular Test::Unit environment.
Changed the "Analyze API JSON format - index and field" test to perform the necessary
setup as part of the test code.
Possible to fail this test with a 500 NoShardsAvailable error, due
to the index not being fully allocated before the analyze call is
executed.
Changing for a wait_for_status: yellow fixes the occasional problem.
Clinton Gormley
Shay Banon
markharwood
Areek Zillur
Spencer Alger
Simon Willnauer
Lee Hinman
Colin Goodheart-Smithe
Brian Murphy
javanna
Boaz Leskes
Adrien Grand
Britta Weber
Zachary Tong
Honza Král
Karel Minarik
Martijn van Groningen
Andrew Selden
Alexander Reelsen
Chris Earle
Kevin Wang
Igor Motov
David Pilato
James Brook
Isabel Drost-Fromm
Alexander Reelsen
Karel Minarik
spenceralger
Karel Minarik