Commit Graph

1780 Commits

Author SHA1 Message Date
Simon Willnauer e81804cfa4 Add a shard filter search phase to pre-filter shards based on query rewriting (#25658)
Today if we search across a large amount of shards we hit every shard. Yet, it's quite
common to search across an index pattern for time based indices but filtering will exclude
all results outside a certain time range ie. `now-3d`. While the search can potentially hit
hundreds of shards the majority of the shards might yield 0 results since there is not document
that is within this date range. Kibana for instance does this regularly but used `_field_stats`
to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results.

This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards
and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters.
2017-07-12 22:19:20 +02:00
Adrien Grand f9fbce84b6 Optimize the order of bytes in uuids for better compression. (#24615)
Flake ids organize bytes in such a way that ids are ordered. However, we do not
need that property and could reorganize bytes in an order that would better suit
Lucene's terms dict instead.

Some synthetic tests suggest that this change decreases the disk footprint of
the `_id` field by about 50% in many cases (see `UUIDTests.testCompression`).
For instance, when simulating the indexing of 10M docs at a rate of 10k docs
per second, the current uid generator used 20.2 bytes per document on average,
while this new generator which only puts bytes in a different order uses 9.6
bytes per document on average.

We had already explored this idea in #18209 but the attempt to share long common
prefixes had had a bad impact on indexing speed. This time I have been more
careful about putting discriminant bytes early in the `_id` in a way that
preserves indexing speed on par with today, while still allowing for better
compression.
2017-07-11 17:28:23 +02:00
Simon Willnauer 98c91a3bd0 Limit the number of concurrent shard requests per search request (#25632)
This is a protection mechanism to prevent a single search request from
hitting a large number of shards in the cluster concurrently. If a search is
executed against all indices in the cluster this can easily overload the cluster
causing rejections etc. which is not necessarily desirable. Instead this PR adds
a per request limit of `max_concurrent_shard_requests` that throttles the number of
concurrent initial phase requests to `256` by default. This limit can be increased per request
and protects single search requests from overloading the cluster. Subsequent PRs can introduces
addiontional improvemetns ie. limiting this on a `_msearch` level, making defaults a factor of
the number of nodes or sort shards iters such that we gain the best concurrency across nodes.
2017-07-11 16:23:10 +02:00
Boaz Leskes a6db0ea908 Run Translog retention yaml tests with no replicas
Initializing replicas change the translog retention logic and confuses the test.

Switch to the solution suggested in https://github.com/elastic/elasticsearch/issues/25623, if implemented
2017-07-10 11:06:57 +02:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Jim Ferenczi 31614c3ddb Remove deprecated fielddata_fields from search request (#25566)
... and inner_hits
2017-07-06 13:02:28 +02:00
Colin Goodheart-Smithe 41abccf6c5 Adds rewrite phase to aggregations (#25495)
* Adds rewrite phase to aggregations

This change adds aggregations to the rewrite performed by the `SearchSourceBuilder`. This means that `AggregationBuilder`s are able to implement a `rewrite()` method where they can return a new `AggregationBuilder` which is functionally the same but in a more primitive form. This is exactly analogous to the rewrite done by the `QueryBuilder`s.

The first aggregation to implement the rewrite are the filter and filters aggregations so they can rewrite the filters they contain.

Closes #17676

* Removes rewrite from PipelineAggregationBuilder

Rewrite is based on shard level information. Since pipeline aggregation are run in the reduce phase it doesn’t make sense to rewrite them on the shards. In fact eventually we shouldn’t be transporting them to the shards at all and should be retaining them on the coordinating node for execution in the reduce phase

* Addresses review comments

* addresses more review comments

* Fixed imports
2017-07-04 16:47:48 +01:00
Jun Ohtani 6894ef6057 [Analysis] Support normalizer in request param (#24767)
* [Analysis] Support normalizer in request param

Support normalizer param
Support custom normalizer with char_filter/filter param

Closes #23347
2017-07-04 19:16:56 +09:00
Jason Tedor 6ae4497c13 Adjust BWC version on bad allocation request test
This commit adjusts the BWC version on the bad cluster allocation
explain request test as changing the API to respond with a bad request
status instead of an internal server error status was backported to 5.x
to be included in 5.6.0.

Relates #25503
2017-06-30 18:05:58 -04:00
Jason Tedor c70c440050 Adjust status on bad allocation explain requests
When a user requests a cluster allocation explain in a situation where
it does not make sense (for example, there are no unassigned shards), we
should consider this a bad request instead of a server error. Yet, today
by throwing an illegal state exception, these are treated as server
errors. This commit adjusts these so that they throw illegal argument
exceptions and are treated as bad requests.

Relates #25503
2017-06-30 17:50:20 -04:00
Glen Smith 1dd28808d5 Fix typo in name of test
This commit fixes a typo in the name of a REST test.

Relates #25451
2017-06-30 12:51:37 -04:00
Christoph Büscher 2708bcc6ed Merge branch 'master' into feature/rank-eval 2017-06-29 15:07:45 +02:00
Ali Beyad b18bfd6062 Output all empty snapshot info fields if in verbose mode (#25455)
In #24477, a less verbose option was added to retrieve snapshot info via
GET /_snapshot/{repo}/{snapshots}.  The point of adding this less
verbose option was so that if the repository is a cloud based one, and
there are many snapshots for which the snapshot info needed to be
retrieved, then each snapshot would require reading a separate snapshot
metadata file to pull out the necessary information.  This can be costly
(performance and cost) on cloud based repositories, so a less verbose
option was added that only retrieves very basic information about each
snapshot that is all available in the index-N blob - requiring only one
read!

In order to display this less verbose snapshot info appropriately, logic
was added to not display those fields which could not be populated.
However, this broke integrators (e.g. ECE) that required these fields to
be present, even if empty.  This commit is to return these fields in the
response, even if empty, if the verbose option is set.
2017-06-28 17:37:56 -05:00
Andreas Gebhardt a156ccd80e Expand `/_cat/nodes` to return information about hard drive (#21775)
Expand `/_cat/nodes` with already present information about available disk space `diskAvail` (alias: `d`, `disk`) by:

    * `diskTotal` (alias `dt`): total disk space
    * `diskUsed` (alias `du`): used disk space (`diskTotal - diskAvail`)
    * `diskUsedPercent` (alias `dup`): used disk space percentage

Note: The available disk space is the number of bytes available to the node's Java virtual machine. The size might be smaller than the real one. That means the used disk space (percentage) is larger.

Closes #21679
2017-06-28 18:20:20 +02:00
Simon Willnauer 4e4a104f4a Remove remaining `index.mapper.single_type` setting usage from tests (#25388)
This change removes the remaining explicitly specified `index.mapper.single_type`
settings from tests in order to allow the removal of the setting.
This is the already approved part of #25375 broken out to simplfiy reviews on
2017-06-25 12:25:41 +02:00
Boaz Leskes d963882053 Enable a long translog retention policy by default (#25294)
#25147  added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours.

In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations.

Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number.

Relates to #10708
2017-06-22 17:08:14 +02:00
Spencer c5b79cd460 [rest-api-spec/indices.refresh] Remove old params
Fixes #25234
2017-06-21 13:44:27 -07:00
Lee Hinman 50bac63210 [TEST] Add skip for 5.x BWC tests for custom filter in analyze API
Resolves #25316
2017-06-20 09:25:03 -06:00
Jun Ohtani 62d1969595 Parse synonyms with the same analysis chain (#8049)
* [Analysis] Parse synonyms with the same analysis chain

Synonym Token Filter / Synonym Graph Filter tokenize synonyms with whatever tokenizer and token filters appear before it in the chain.

Close #7199
2017-06-20 21:50:33 +09:00
Jim Ferenczi 68deda6d03 FastVectorHighlighter should not cache the field query globally (#25197)
This commit removes the global caching of the field query and replaces it with
a caching per field. Each field can use a different `highlight_query` and the rewriting of
some queries (prefix, automaton, ...) depends on the targeted field so the query used for highlighting
must be unique per field.
There might be a small performance penalty when highlighting multiple fields since the query needs to be rewritten
once per highlighted field with this change.

Fixes #25171
2017-06-15 00:33:01 +02:00
Christoph Büscher ac3db8c30f Merge branch 'master' into feature/rank-eval 2017-06-14 11:57:05 +02:00
Boaz Leskes 43f4ae5a7b Indices.rollover/10_basic should refresh to make the doc visible in lucene stats 2017-06-13 23:37:15 +02:00
Boaz Leskes d3c97615c1 Adapt skip version in rest-api-spec/test/indices.rollover/20_max_doc_condition.yml
The relevant change was backported.
2017-06-13 14:46:15 +02:00
Sergey Galkin 1c95cbc4e8 Rollover max docs should only count primaries (#24977)
max_doc condition for index rollover should use document count only from primary shards 

Fixes #24217
2017-06-13 14:30:46 +02:00
Spencer 88591fecac [docs] include two cluster doc pages missing from index (#25180)
* [docs] include two cluster doc pages missing from index

* [rest-api-spec] update link to remote-info docs
2017-06-12 12:33:56 -07:00
Jason Tedor 725f6b6983 Change BWC versions on get mapping 404s
This commit changes the BWC versions on the get mapping 404s now that
this API returning 404s when a type is missing is supported since 5.5.0.

Relates #23192
2017-06-11 16:59:12 -04:00
Jason Tedor dcf57f296e Fix get mappings HEAD requests
Get mappings HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for get
mappings HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates #23192
2017-06-11 14:58:56 -04:00
Jim Ferenczi 5cdbebec94 Test: remove faling test that relies on merge order 2017-06-10 11:55:41 +02:00
Jason Tedor 8a45c3105f Change BWC versions on create index response
This commit changes the BWC versions on the create index response now
that the index name in the response is supported since 5.6.0.

Relates #25139
2017-06-09 13:52:08 -04:00
Sergey Novikov 7c8657df0e Return the index name on a create index response
This commit modifies the create index response so that it includes the
index name.

Relates #25139
2017-06-09 13:47:47 -04:00
Tal Levy a771912a22 Add Ingest-Processor specific Rest Endpoints & Add Grok endpoint (#25059)
This PR enables Ingest plugins to leverage processor-scoped REST
endpoints. First of which being the Grok endpoint that retrieves
Grok Patterns for users to retrieve all the built-in patterns.
Example usage: Kibana Grok Autocomplete!
2017-06-08 15:24:35 -07:00
Lee Hinman 119f8ed9f0 Correctly enable _all for older 5.x indices
When we disabled `_all` by default for indices created in 6.0, we missed adding
a layer that would handle the situation where `_all` was not enabled in 5.x and
then the cluster was updated to 6.0, this means that when the cluster was
updated the `_all` field would be disabled for 5.x indices and field values
would not be added to the `_all` field.

This adds a compatibility layer for 5.x indices where we treat the default
enabled value for the `_all` field to be `true` if unset on 5.x indices.

Resolves #25068
2017-06-08 14:37:44 -06:00
Lee Hinman 050b7cd0f9 Include empty mappings in GET /{index}/_mappings requests (#25118)
Previously this would output:

```
GET /test-1/_mappings

{ }
```

And after this change:

```
GET /test-1/_mappings

{
  "test-1": {
    "mappings": {}
  }
}
```

To bring parity back to the REST output after #24723.

Relates to #25090
2017-06-08 10:57:04 -06:00
Lee Hinman 5b2ab96364 Return index name and empty map for /{index}/_alias with no aliases
Previously in #24723 we changed the `_alias` API to not go through the
`RestGetIndicesAction` endpoint, instead creating a `RestGetAliasesAction` that
did the same thing.

This changes the formatting so that it matches the old formatting of the
endpoint, before:

```
GET /test-1/_alias

{ }
```

And after this change:

```
GET /test-1/_alias

{
  "test-1": {
    "aliases": {}
  }
}
```

This is related to #25090
2017-06-08 10:03:03 -06:00
Jim Ferenczi 36a5cf8f35 Automatically early terminate search query based on index sorting (#24864)
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and 
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.

Relates #6720
2017-06-08 12:10:46 +02:00
Simon Willnauer d57641a747 Skip rest tests that use mutiple types in pure 6.x clusters (#24965)
This change skips rest tests that use mutlitple types if the cluster
is a pure 6.x cluster. This allows all indics to be created with a version
less than 6.0 and that means we can safely use the `mapping.single_type` setting.

Relates to #24961
2017-06-07 15:00:17 +02:00
Jason Tedor 5a0b159cb7 Modify skips for get missing alises tests
Previous work modified the status code on the get aliases API when an
alias is missing so that these requests 404 now. This change was also
backported to 5.5 so we can adjust the skips to skip everything before
5.5.0.
2017-06-06 15:35:30 -04:00
Jason Tedor e03c4938c5 GET aliases should 404 if aliases are missing
Previously the HEAD and GET aliases endpoints were misaigned in
behavior. The HEAD verb would 404 if any aliases are missing while the
GET verb would not if any aliases existed. When HEAD was aligned with
GET, this broke the previous usage of HEAD to serve as an existence
check for aliases. It is the behavior of GET that is problematic here
though, if any alias is missing the request should 404. This commit
addresses this by modifying the behavior of GET to behave in this
way. This fixes the behavior for HEAD to also 404 when aliases are
missing.

Relates #25043
2017-06-06 14:37:29 -04:00
Jim Ferenczi 4a8759ef4c Collapse inner hits rest test should not skip 5.x
Relates https://github.com/elastic/elasticsearch/pull/24517
2017-06-06 09:33:56 +02:00
jaymode e98d5676b3
Test: update missing body tests to run against versions >= 5.5.0
This updates the missing body tests to run against versions >= 5.5.0 after backporting the change
to the 5.x branch.

See #23497
2017-06-05 14:26:07 -06:00
Alex Benusovich 5463294ec4 Fixed NPEs caused by requests without content. (#23497)
REST handlers that require a body will throw an an ElasticsearchParseException "request body required".
REST handlers that require a body OR source param will throw an ElasticsearchParseException "request body or source param required".
Replaced asserts in BulkRequest parsing code with a more descriptive IllegalArgumentException if the line contains an empty object.
Updated bulk REST test to verify an empty action line is rejected properly.
Updated BulkRequestTests with randomized testing for an empty action line.
Used try-with-resouces for XContentParser in AbstractBulkByQueryRestHandler.
2017-06-05 09:08:14 -06:00
Lee Hinman 134b0d594e [TEST] Skip wildcard expansion test due to breaking change
Relates to #24723
2017-06-02 20:48:52 -06:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Ryan Ernst 8d88b94372 Scripting: Add optional context parameter to put stored script requests (#25014)
This commit adds an optional `context` url parameter to the put stored
script request. When a context is specified, the script is compiled
against that context before storing, as a validation the script will
work when used in that context.
2017-06-01 17:53:48 -07:00
Christoph Büscher 3d6fb4eb0b Merge branch 'master' into feature/rank-eval 2017-05-30 14:24:26 +02:00
Jim Ferenczi 47cf7825dd Move BWC version to 5.5 after backport
Relates to #24517
2017-05-26 14:57:07 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
markharwood a64937db7a Test fix - rest test missing version skip for new 6.0 significant_text agg 2017-05-24 17:05:02 +01:00
markharwood b7197f5e21 SignificantText aggregation - like significant_terms, but for text (#24432)
* SignificantText aggregation - like significant_terms but doesn’t require fielddata=true, recommended used with `sampler` agg to limit expense of tokenizing docs and takes optional `filter_duplicate_text`:true setting to avoid stats skew from repeated sections of text in search results.

Closes #23674
2017-05-24 13:46:43 +01:00
Christoph Büscher cd0941810f Merge branch 'master' into feature/rank-eval 2017-05-18 16:47:47 +02:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Jim Ferenczi 67c41d2e77 Fix ExpandSearchPhase when response contains no hits (#24688)
This change skips the expand search phase entirely when there is no search hits in the response.
2017-05-17 14:15:40 +02:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Jim Ferenczi 7b7e15023a Add rest test for sliced scroll (#24630) 2017-05-12 00:07:24 +02:00
qwerty4030 e7d352b489 Compound order for histogram aggregations. (#22343)
This commit adds support for histogram and date_histogram agg compound order by refactoring and reusing terms agg order code. The major change is that the Terms.Order and Histogram.Order classes have been replaced/refactored into a new class BucketOrder. This is a breaking change for the Java Transport API. For backward compatibility with previous ES versions the (date)histogram compound order will use the first order. Also the _term and _time aggregation order keys have been deprecated; replaced by _key.

Relates to #20003: now that all these aggregations use the same order code, it should be easier to move validation to parse time (as a follow up PR).

Relates to #14771: histogram and date_histogram aggregation order will now be validated at reduce time.

Closes #23613: if a single BucketOrder that is not a tie-breaker is added with the Java Transport API, it will be converted into a CompoundOrder with a tie-breaker.
2017-05-11 18:06:26 +01:00
Ali Beyad 743217a430 Enhances get snapshots API to allow retrieving repository index only (#24477)
Currently, the get snapshots API (e.g. /_snapshot/{repositoryName}/_all)
provides information about snapshots in the repository, including the
snapshot state, number of shards snapshotted, failures, etc.  In order
to provide information about each snapshot in the repository, the call
must read the snapshot metadata blob (`snap-{snapshot_uuid}.dat`) for
every snapshot.  In cloud-based repositories, this can be expensive,
both from a cost and performance perspective.  Sometimes, all the user
wants is to retrieve all the names/uuids of each snapshot, and the
indices that went into each snapshot, without any of the other status
information about the snapshot.  This minimal information can be
retrieved from the repository index blob (`index-N`) without needing to
read each snapshot metadata blob.

This commit enhances the get snapshots API with an optional `verbose`
parameter.  If `verbose` is set to false on the request, then the get
snapshots API will only retrieve the minimal information about each
snapshot (the name, uuid, and indices in the snapshot), and only read
this information from the repository index blob, thereby giving users
the option to retrieve the snapshots in a repository in a more
cost-effective and efficient manner.

Closes #24288
2017-05-10 15:48:40 -04:00
Isabel Drost-Fromm bd559d96d4
This adds max_concurrent_searches to multi-search-template endpoint.
Closes #20912
2017-05-10 11:23:24 +02:00
Adrien Grand a72eaa8e0f Identify documents by their `_id`. (#24460)
Now that indices have a single type by default, we can move to the next step
and identify documents using their `_id` rather than the `_uid`.

One notable change in this commit is that I made deletions implicitly create
types. This helps with the live version map in the case that documents are
deleted before the first type is introduced. Otherwise there would be no way
to differenciate `DELETE index/foo/1` followed by `PUT index/foo/1` from
`DELETE index/bar/1` followed by `PUT index/foo/1`, even though those are
different if versioning is involved.
2017-05-09 16:33:52 +02:00
Jim Ferenczi 4df636b5ef Fix single shard scroll within a cluster with nodes in version >= 5.3 and <= 5.3 (#24512)
If a node in version >= 5.3 acts as a coordinating node during a scroll request that targets a single shard, the scroll may return the same documents over and over iff the targeted shard is hosted by a node with a version <= 5.3.
The nodes in this version will advance the scroll only if the search_type has been set to `query_and_fetch` though this search type has been removed in 5.3.
This change handles this situation by adding the removed search_type in the request that targets a node in version <= 5.3.
2017-05-09 09:14:17 +02:00
Simon Willnauer 5bfb98ade4 [TEST] Reenable disabled tests for _field_caps and _search_shards (#24505) 2017-05-05 16:02:26 +02:00
Simon Willnauer 8055b14f2e Temporarily disable tests 2017-05-05 12:08:00 +02:00
Simon Willnauer 03267e03da Fix NPE if field caps request has a field that exists not in all indices (#24504)
If a field caps request contains a field name that doesn't exist in all indices
the response will be partial and we hide an NPE. The NPE is now fixed but we still
have the problem that we don't pass on errors on the shard level to the user. This will
be fixed in a followup.
2017-05-05 11:56:03 +02:00
Simon Willnauer 6b67e0bf2f Include all aliases including non-filtering in `_search_shards` response (#24489)
`_search_shards`API today only returns aliases names if there is an alias
filter associated with one of them. Now it can be useful to see which aliases
have been expanded for an index given the index expressions. This change also includes non-filtering aliases even without a filtering alias being present.
2017-05-05 09:34:12 +02:00
Simon Willnauer 07f106d39c [TEST] Rollback temporarily disabled field_caps test (#24483) 2017-05-04 14:14:22 +02:00
Simon Willnauer 14e57bf9f8 Add cross cluster support to `_field_caps` (#24463)
To support kibana this commit adds an internal optimization
to support the cross cluster syntax for indices on the `_field_caps`
API.

Closes #24334
2017-05-04 11:44:54 +02:00
Jim Ferenczi 6fcd24d264 Check index sorting with no replica since we cannot ensure that the replica index is ready when forceMerge is called. Closes #24416 2017-05-02 20:25:09 +02:00
Luca Cavanna 91fbb0ba28 Move IndicesAliasesRequest#concreteAliases to TransportIndicesAliasesAction (#24400)
This method has to do with how the transport action may or may not resolve wildcards expressions to aliases names. It is only needed in TransportIndicesAliasesAction and for this reason it should be a private method in it rather than part of a request class which is also part of the Java API and later in the high level REST client.
2017-05-01 19:59:06 +02:00
javanna 7863407b46 [TEST] fix _cat/allocation index size check
The check expected the size of the index to always be returned in bytes, but that can possibly be kb, mb, gb and tb depending on the actual size.
2017-05-01 14:38:14 +02:00
Guillaume Le Floch 382a617d34 Handle multiple aliases in _cat/aliases api (#23698)
The alias parameter was documented as a list in our rest-spec, yet only the first value out of a list was getting read and processed. This commit adds support for multiple aliases to _cat/aliases

Closes #23661
2017-04-28 15:21:44 +02:00
Adrien Grand 1be2800120 Only allow one type on 7.0 indices (#24317)
This adds the `index.mapping.single_type` setting, which enforces that indices
have at most one type when it is true. The default value is true for 6.0+ indices
and false for old indices.

Relates #15613
2017-04-27 08:43:20 +02:00
Guillaume Le Floch 739cb35d1b Allow passing single scrollID in clear scroll API body (#24242)
* Allow single scrollId in string format

Closes #24233
2017-04-25 13:43:21 +02:00
Christoph Büscher 5254731039 Merge branch 'master' into feature/rank-eval 2017-04-22 21:47:32 +02:00
Nik Everett caf376c8af Start building analysis-common module (#23614)
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Jim Ferenczi f05af0a382 Enable index-time sorting (#24055)
This change adds an index setting to define how the documents should be sorted inside each Segment.
It allows any numeric, date, boolean or keyword field inside a mapping to be used to sort the index on disk.
It is not allowed to use a `nested` fields inside an index that defines an index sorting since `nested` fields relies on the original sort of the index.
This change does not add early termination capabilities in the search layer. This will be added in a follow up.

Relates #6720
2017-04-19 14:36:11 +02:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates #24118
2017-04-15 10:36:03 -04:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Simon Willnauer 42e0b4f5e9 [TEST] Enable new REST test for 5.4 and BWC against 5.4.x 2017-04-11 13:30:45 +02:00
Simon Willnauer f22e0dc30b Add cross-cluster search remote cluster info API (#23969)
This commit adds an API to discover information like seed nodes,
http addresses and connection status of a configured remote cluster.

Closes #23925
2017-04-11 09:24:40 +02:00
Jim Ferenczi af49c46b76 Fix BWC tests for field_stats now that the deprecation has been back ported to 5.4 2017-04-10 12:40:37 +02:00
Jim Ferenczi 9b3c85dd88 Deprecate _field_stats endpoint (#23914)
_field_stats has evolved quite a lot to become a multi purpose API capable of retrieving the field capabilities and the min/max value for a field.
In the mean time a more focused API called `_field_caps` has been added, this enpoint is a good replacement for _field_stats since he can
retrieve the field capabilities by just looking at the field mapping (no lookup in the index structures).
Also the recent improvement made to range queries makes the _field_stats API obsolete since this queries are now rewritten per shard based on the min/max found for the field.
This means that a range query that does not match any document in a shard can return quickly and can be cached efficiently.
For these reasons this change deprecates _field_stats. The deprecation should happen in 5.4 but we won't remove this API in 6.x yet which is why
 this PR is made directly to 6.0.
 The rest tests have also been adapted to not throw an error while this change is backported to 5.4.
2017-04-10 10:10:16 +02:00
jaymode 53e3ddf2f0
Test: remove test that will never run on master
This test was added in #23950 for backporting and review, but it is always skipped on master so
this commit deletes it.
2017-04-06 15:50:08 -04:00
Jay Modi 495bf21b46 Preserve response headers when creating an index (#23950)
This commit preserves the response headers when creating an index and updating settings for an
index.

Closes #23947
2017-04-06 20:38:09 +01:00
Christoph Büscher 024ed1b6ca Merge branch 'master' into feature/rank-eval 2017-04-04 18:23:41 +02:00
Clinton Gormley 01b807f98e Adapted search_shards rest test to work with Perl
Because of the way Perl treats numbers, the boost is represented
as 1 instead of 1.0, which caused this test to fail.
2017-04-02 12:52:34 +02:00
Clinton Gormley 5b3c662145 To examine an exception in rest tests, the exception should be caught, not ignored 2017-04-02 12:52:30 +02:00
Clinton Gormley e5f99e0bde Fixed bad YAML in rest tests 2017-04-02 12:52:23 +02:00
Jim Ferenczi a8250b26e7 Add FieldCapabilities (_field_caps) API (#23007)
This change introduces a new API called `_field_caps` that allows to retrieve the capabilities of specific fields.

Example:

````
GET t,s,v,w/_field_caps?fields=field1,field2
````
... returns:
````
{
   "fields": {
      "field1": {
         "string": {
            "searchable": true,
            "aggregatable": true
         }
      },
      "field2": {
         "keyword": {
            "searchable": false,
            "aggregatable": true,
            "non_searchable_indices": ["t"]
            "indices": ["t", "s"]
         },
         "long": {
            "searchable": true,
            "aggregatable": false,
            "non_aggregatable_indices": ["v"]
            "indices": ["v", "w"]
         }
      }
   }
}
````

In this example `field1` have the same type `text` across the requested indices `t`, `s`, `v`, `w`.
Conversely `field2` is defined with two conflicting types `keyword` and `long`.
Note that `_field_caps` does not treat this case as an error but rather return the list of unique types seen for this field.
2017-03-31 15:34:46 +02:00
Nik Everett b3dca364ce Revert "Skip 5.4 bwc test for new name for now"
This reverts commit 490d29f4fc.
We should be able to run the 5.4 tests now that we have
8c53555b28.
2017-03-28 09:06:00 -04:00
Jason Tedor 742d929b56 Validate top-level keys when parsing mget requests
Today, when parsing mget requests, we silently ignore keys in the top
level that do not match "docs" or "ids". This commit addresses this
situation by throwing an exception if any other key occurs here, and
providing the names of valid keys.

Relates #23746
2017-03-28 08:27:31 -04:00
Christoph Büscher 96fc3aaf6f Merge branch 'master' into feature/rank-eval 2017-03-23 19:55:47 +01:00
Igor Motov f927a2708d Make it possible to validate a query on all shards instead of a single random shard (#23697)
This is especially useful when we rewrite the query because the result of the rewrite can be very different on different shards. See #18254 for example.
2017-03-22 17:39:21 -04:00
Nik Everett 490d29f4fc Skip 5.4 bwc test for new name for now
We have to wait for a clean snapshot build.
2017-03-22 15:19:02 -04:00
Jason Tedor 230c5b1ccf Count through the primary in list of strings test
A previous attempt to address a race condition in this test set wait for
active shards to all. However, there might not be any replicas if the
test is only running with one node so we end up waiting
forever. Instead, to address the intial race condition, we just count
through the primary.
2017-03-22 11:00:14 -04:00
Nik Everett 2e3ea918dc Skip testing new name if it isn't known
In #23638 we renamed `request_cache` to `request` in the
`_cache/clear` API. But it is only going to be committed back to
5.x so we can't test with the new name in a mixed version
cluster.
2017-03-22 10:56:03 -04:00
Jason Tedor 9646aca893 Wait for all shards in list of strings test
This test executes a bulk indexing operation with two documents. If this
test is running against multiple nodes, there are no guarantees that all
shards are green before we execute a search operation which might hit a
replica shard. This commit creates the index in advance, and waits for
all shards to be active before proceeding with the indexing request.
2017-03-22 09:35:12 -04:00
Stefan Gorgiovski 798c19dd7f Deprecate request_cache for clear-cache (#23638)
It is called `request` now.
2017-03-22 08:28:04 -04:00
Honza Král f8b7ec27a4 [API] change wait_for_completion defaults according to docs (#23672) 2017-03-22 12:53:47 +01:00
Jason Tedor f56900da9f Fix took assertion in response filter test
This test asserts that the took time exists by using the is_true
assertion. This assertion fails if the took time was zero as is_true
asserts that the field is not the empty string, not the string "false",
and not 0. If the search returns quickly, and took time is measured
using a high-precision monotonic clock, the took time can be zero. This
commit changes the assertion to use gte.
2017-03-20 21:11:20 -04:00
Christoph Büscher cf35545e2d Merge branch 'master' into feature/rank-eval 2017-03-13 17:36:13 -07:00
Karel Minarik 4ea6fc0059 [TEST] Fixed the "Msearch" typed keys YAML test
* Added the YAML document separator to the beginning of the file
* Fixed the incorrect JSON syntax in the query

Closes #23500
2017-03-08 08:38:12 -08:00
Ali Beyad 577d2a6a1d Adds cluster state size to /_cluster/state response (#23440)
This commit adds the size of the cluster state to the response for the
get cluster state API call (GET /_cluster/state).  The size that is
returned is the size of the full cluster state in bytes when compressed.
This is the same size of the full cluster state when serialized to
transmit over the network.  Specifying the ?human flag displays the
compressed size in a more human friendly manner.  Note that even if the
cluster state request filters items from the cluster state (so a subset
of the cluster state is returned), the size that is returned is the
compressed size of the entire cluster state.

Closes #3415
2017-03-02 14:20:29 -05:00
javanna 261f31f5b7 [TEST] move filters aggs wrapper query builder rewriting test to integ tests
This test makes little sense when sent from the REST layer, as WrapperQueryBuilder is supposed to be used from the Java api. Also, providing the inner query as base64 string will work only for string formats and break for binary formats like SMILE and CBOR, whcih doesn't play well with randomizing content type in our REST tests
2017-02-27 12:27:03 +01:00
Christoph Büscher 1f4c4d99b9 Merge branch 'master' into feature/rank-eval 2017-02-27 11:25:17 +01:00
Christoph Büscher 641c88dc29 Prevent negative `from` parameter in SearchSourceBuilder (#23358)
This prevents later errors like the one reported in #23324 and throws an
IllegalArgumentException early instead.
2017-02-27 09:45:10 +01:00
Ali Beyad 550f32354c [TEST] Removes timeout based wait_for_active_shards REST test (#23360)
This commit removes an necessary test that ensures if
wait_for_active_shards cannot be fulfilled on index creation, that the
response returns shardsAcknowledged=false.  However, this is already
tested in WaitForActiveShardsIT and it would improve the speed of the
test runs to get rid of any unnecessary tests, especially those that
depend on timeouts.
2017-02-24 14:55:42 -05:00
Ali Beyad 93a359e434 [TEST] increase timeout slightly in wait_for_active_shards test to allow
for index creation cluster state update to be processed before ensuring
the wait times out
2017-02-24 12:51:45 -05:00
Jim Ferenczi 57b5d1d29b disable BWC tests for the highlighters, need a new 5.x build to make it work 2017-02-24 08:50:39 +01:00
Jason Tedor e579629b16 Align REST specs for HEAD requests
Previous changes aligned HEAD requests to be consistent with GET
requests to the same endpoint. This commit aligns the REST spec for the
impacted endpoints.

Relates #23313
2017-02-23 08:55:13 -05:00
Luca Cavanna 495b24655b Update indices settings api to support CBOR and SMILE format (#23309)
Also expand testing on the different ways to provide index settings and remove dead code around ability to provide settings as query string parameters

Closes #23242
2017-02-22 17:51:10 +01:00
Simon Willnauer 5c1924ad19 Remove BWC layer for number of reduce phases (#23303)
Both PRs below have been backported to 5.4 such that we can enable
BWC tests of this feature as well as remove version dependend serialization
for search request / responses.

Relates to #23288
Relates to #23253
2017-02-22 15:03:09 +01:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Jason Tedor 2925a81cc9 Fix REST spec for exists
A previous change aligned the handling of the GET document and HEAD
document APIs. This commit aligns the specification for these two APIs
as well, and fixes a failing test.

Relates #23196
2017-02-16 08:56:38 -05:00
Christoph Büscher cfa52f8b9a Merge branch 'master' into feature/rank-eval 2017-02-16 10:39:07 +01:00
Jim Ferenczi 9316e8e8fe Restore BWC tests for field collapsing 2017-02-15 15:04:38 +01:00
Yannick Welsch 1aefbf57e1 Fix tests that check for deprecation message 2017-02-15 09:35:02 +01:00
Clinton Gormley 1556e81e9a Normalising REST spec query param types
integer -> number
filter_path: string -> list
2017-02-14 16:59:34 +01:00
Tanguy Leroux 072748cd67 [Tests] Fix msearch/typed_keys test
This test uses index_patterns which has been introduced in 6.0 and does not exist in 5.4.0, making the Bwc test fails. Instead of using index templates, it now uses explicitly create the required indices. Also, it fixes unmapped aggregations tests.
2017-02-13 13:39:11 +01:00
Jim Ferenczi 5808aea265 Temporary disable bwc tests on field collapsing that fail because snapshot build of 5.3 is broken 2017-02-10 15:42:58 +01:00
Tanguy Leroux e2e5937455 Use `typed_keys` parameter to prefix suggester names by type in search responses (#23080)
This pull request reuses the typed_keys parameter added in #22965, but this time it applies it to suggesters. When set to true, the suggester names in the search response will be prefixed with a prefix that reflects their type.
2017-02-10 10:53:38 +01:00
Jim Ferenczi 94087b3274 Removes ExpandCollapseSearchResponseListener, search response listeners and blocking calls
This changes removes the SearchResponseListener that was used by the ExpandCollapseSearchResponseListener to expand collapsed hits.
The removal of SearchResponseListener is not a breaking change because it was never released.
This change also replace the blocking call in ExpandCollapseSearchResponseListener by a single asynchronous multi search request. The parallelism of the expand request can be set via CollapseBuilder#max_concurrent_group_searches

Closes #23048
2017-02-09 18:06:10 +01:00
Tanguy Leroux 3553522328 Add parameter to prefix aggs name with type in search responses (#22965)
This pull request adds a new parameter to the REST Search API named `typed_keys`. When set to true, the aggregation names in the search response will be prefixed with a prefix that reflects the internal type of the aggregation.

Here is a simple example:
```
GET /_search?typed_keys
{
    "aggs": {
        "tweets_per_user": {
            "terms": {
                "field": "user"
            }
        }
    },
    "size": 0
}
```

And the response:

```
{
    "aggs": {
        "sterms:tweets_per_user": {
            ...
        }
    }
}
```

This parameter is intended to make life easier for REST clients that could parse back the prefix and could detect the type of the aggregation to parse. It could also be implemented for suggesters.
2017-02-09 11:19:04 +01:00
Christoph Büscher 4cb8d9d08c Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
        docs/reference/search.asciidoc
2017-02-03 17:27:20 +01:00
Ali Beyad 32707fa1ef [TEST] fix introduced typo 2017-02-02 12:18:20 -05:00
Ali Beyad 43aadef23a [TEST] upgrade backward compatibility version of rolling upgrade
tests to 5.3.0-SNAPSHOT
2017-02-02 09:56:38 -05:00
Ali Beyad bf439b9cc8 [TEST] temporarily disable the explain API rest test until
the 5.3.0-snapshot distribution is properly generated
2017-02-01 14:14:39 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Jim Ferenczi f6d38d480a Integrate UnifiedHighlighter (#21621)
* Integrate UnifiedHighlighter

This change integrates the Lucene highlighter called "unified" in the list of supported highlighters for ES.
This highlighter can extract offsets from either postings, term vectors, or via re-analyzing text.
The best strategy is picked automatically at query time and depends on the field and the query to highlight.
2017-01-31 19:06:03 +01:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Jim Ferenczi 868b12b548 Add BWC tests for field collapsing
Field collapsing is supported from version 5.3
2017-01-24 08:34:16 +01:00
Jim Ferenczi e48bc2eed7 Add field collapsing for search request (#22337)
* Add top hits collapsing to search request

The field collapsing is done with a custom top docs collector that "collapse" search hits with same field value.
The distributed aspect is resolve using the two passes that the regular search uses. The first pass "collapse" the top hits, then the coordinating node merge/collapse the top hits from each shard.

```
GET _search
{
   "collapse": {
      "field": "category",
   }
}
```

This change also adds an ExpandCollapseSearchResponseListener that intercepts the search response and expands collapsed hits using the CollapseBuilder#innerHit} options.
The retrieval of each inner_hits is done by sending a query to all shards filtered by the collapse key.

```
GET _search
{
   "collapse": {
      "field": "category",
      "inner_hits": {
	"size": 2
      }
   }
}
```
2017-01-23 16:33:51 +01:00
markharwood f01784205f New AdjacencyMatrix aggregation
Similar to the Filters aggregation but only supports "keyed" filter buckets and automatically "ANDs" pairs of filters to produce a form of adjacency matrix.
The intersection of buckets "A" and "B" is named "A&B" (the choice of separator is configurable). Empty intersection buckets are removed from the final results.

Closes #22169
2017-01-20 15:49:31 +00:00
Boaz Leskes 5d806bf93e Index creation and setting update may not return deprecation logging (#22702)
Those services validate their setting before submitting an AckedClusterStateUpdateTask to the cluster state service. An acked cluster state may be completed by a networking thread when the last acks as received. As such it needs special care to make sure that thread context headers are handled correctly.
2017-01-20 10:14:13 +01:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Nicholas Knize 84e4f91253 Add geo_point to FieldStats
This commit adds a new GeoPoint class to FieldStats for computing field stats over geo_point field types.
2017-01-18 14:37:03 -06:00
Boaz Leskes 1227044ddd Add a deprecation notice to shadow replicas (#22647)
Relates to #22024

On top of documentation, the PR adds deprecation loggers and deals with the resulting warning headers.

The yaml test is set exclude versions up to 6.0. This is need to make sure bwc tests pass until this is backported to 5.2.0 . Once that's done, I will change the yaml test version limits
2017-01-18 12:28:09 +01:00
Greg Marzouka e0f8d88d5c Include global query string parameters in the REST spec
Closes #11638
2017-01-17 07:35:14 -05:00
Lee Hinman 2db01b6127 Merge remote-tracking branch 'dakrone/disable-all-by-default' 2017-01-12 10:17:51 -07:00
Luca Cavanna 7674de9e1f Move human flag under always accepted query_string params (#22562)
There are some parameters that are accepted by each and every api we expose. Those (pretty, source, error_trace and filter_path)  are not explicitly listed in the spec of every api, rather whitelisted in clients test runners so that they are always accepted. The `human` flag has been treated up until now as a parameter that's accepted by only some stats and info api, but that doesn't reflect reality as es core treats it exactly like `pretty` (relevant especially now that we validate params and throw exception when we find one that is not supported). Furthermore, the human flag has effect on every api that outputs a date, time, percentage or byte size field. For instance the tasks api outputs a date field although they don't have the human flag explicitly listed in their spec. There are other similar cases. This commit removes the human flag from the rest spec and makes it an always accepted query_string param.
2017-01-12 10:04:45 +01:00
Lee Hinman 7a18bb50fc Disable _all by default
This change disables the _all meta field by default.

Now that we have the "all-fields" method of query execution, we can save both
indexing time and disk space by disabling it.

_all can no longer be configured for indices created after 6.0.

Relates to #20925 and #21341
Resolves #19784
2017-01-11 16:47:13 -07:00
Jim Ferenczi 433c822d4f Promote longs to doubles when a terms agg mixes decimal and non-decimal numbers (#22449)
* Promote longs to doubles when a terms agg mixes decimal and non-decimal number

This change makes the terms aggregation work when the buckets coming from different indices are a mix of decimal numbers and non-decimal numbers. In this case non-decimal number (longs) are promoted to decimal (double) which can result in a loss of precision for big numbers.

Fixes #22232
2017-01-10 11:50:56 +01:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
Karel Minarik 4f4b76cd41 [TEST] Fixed the incorrect indentation for the `skip` clauses in the REST tests
This patch fixes the incorrect indentation in the REST tests, which makes tests in language runners (eg. Ruby, Python) to fail, since the skip clause is parsed as an empty value. Tha Java YAML parser is smarter/lenient about whitespace, so it doesn't catch this.
2017-01-08 14:21:02 +01:00
Nik Everett 12923ef896 Close and flush refresh listeners on shard close
Right now closing a shard looks like it strands refresh listeners,
causing tests like
`delete/50_refresh/refresh=wait_for waits until changes are visible in search`
to fail. Here is a build that fails:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+multi_cluster_search+multijob-darwin-compatibility/4/console

This attempts to fix the problem by implements `Closeable` on
`RefreshListeners` and rejecting listeners when closed. More importantly
the act of closing the instance flushes all pending listeners
so we shouldn't have any stranded listeners on close.

Because it was needed for testing, this also adds the number of
pending listeners to the `CommonStats` object and all API to which
that flows: `_cat/nodes`, `_cat/indices`, `_cat/shards`, and
`_nodes/stats`.
2017-01-06 20:03:32 -05:00
Ali Beyad a487b90498 [TEST] fix explain API rest test that assumes there is only a single
node in the cluster (incorrect assumption)
2017-01-06 11:07:40 -05:00
Ali Beyad 2f510b38c3 [TEST] explain API rest test may have shard allocation throttled 2017-01-04 14:34:00 -05:00
Ali Beyad 85b754f0e0 [TEST] 5.x snapshot build is working again, so update the backwards
compatibility tests for the allocation explain API to include 5.2.0
2017-01-04 12:07:17 -05:00
Jim Ferenczi 1f35d2532b Fix BWC layer with field_stats and geo_point 2017-01-04 13:14:09 +01:00
Jim Ferenczi 360ce532eb Implement stats for geo_point and geo_shape field (#22391)
Currently `geo_point` and `geo_shape` field are treated as `text` field by the field stats API and we
try to extract the min/max values with MultiFields.getTerms.
This is ok in master because a `geo_point` field is always a Point field but it can cause problem in 5.x (and 2.x) because the legacy
 `geo_point` are indexed as terms.
 As a result the min and max are extracted and then printed in the FieldStats output using BytesRef.utf8ToString
 which can throw an IndexOutOfBoundException since it's not valid UTF8 strings.
 This change ensure that we never try to extract min/max information from a `geo_point` field.
 It does not add a new type for geo points in the fieldstats API so we'll continue to use `text` for this kind of field.
 This PR is targeted to master even though we could only commit this change to 5.x. I think it's cleaner to have it in master too before we make any decision on
  https://github.com/elastic/elasticsearch/pull/21947.

Fixes #22384
2017-01-04 10:42:22 +01:00
Ali Beyad 91917d6e91 [TEST] mute backwards compatability tests for explain API until 5.2
snapshot builds can be published again
2017-01-02 18:26:19 -05:00
Ali Beyad 20ab4be59f Cluster Explain API uses the allocation process to explain shard allocation decisions (#22182)
This PR completes the refactoring of the cluster allocation explain API and improves it in the following two high-level ways:

 1. The explain API now uses the same allocators that the AllocationService uses to make shard allocation decisions. Prior to this PR, the explain API would run the deciders against each node for the shard in question, but this was not executed on the same code path as the allocators, and many of the scenarios in shard allocation were not captured due to not executing through the same code paths as the allocators.

 2. The APIs have changed, both on the Java and JSON level, to accurately capture the decisions made by the system. The APIs also now report on shard moving and rebalancing decisions, whereas the previous API did not report decisions for moving shards which cannot remain on their current node or rebalancing shards to form a more balanced cluster.

Note: this change affects plugin developers who may have a custom implementation of the ShardsAllocator interface. The method weighShards has been removed and no longer has any utility. In order to support the new explain API, however, a custom implementation of ShardsAllocator must now implement ShardAllocationDecision decideShardAllocation(ShardRouting shard, RoutingAllocation allocation) which provides a decision and explanation for allocating a single shard. For implementations that do not support explaining a single shard allocation via the cluster allocation explain API, this method can simply return an UnsupportedOperationException.
2017-01-02 12:28:32 -06:00
Jim Ferenczi 02d4cbfeea Fix bwc integ test that tries to perform a term aggs on a scaled_float. This is broken when a node with version prior to 5.2.0 is used with another node > 5.2.0. This is because scaled_float fields are considered as longs in version < 5.2.0. This is fixed in 5.2.0 where scaled_float are recognized as doubles. 2016-12-27 21:52:27 +01:00
Jim Ferenczi e7444f7d77 Fix scaled_float numeric type in aggregations (#22351)
`scaled_float` should be used as DOUBLE in aggregations but currently they are used as LONG.
This change fixes this issue and adds a simple it test for it.

Fixes #22350
2016-12-27 09:23:22 +01:00
Ali Beyad 8261bd358a Synchronize snapshot deletions on the cluster state (#22313)
Before, snapshot/restore would synchronize all operations on the cluster
state except for deleting snapshots.  This meant that only one
snapshot/restore operation would be allowed in the cluster at any given
time, except for deletions - there could be two or more snapshot
deletions running at the same time, or a deletion could be running,
unbeknowest to the rest of the cluster, and thus a snapshot or restore
would be allowed at the same time as the snapshot deletion was still in
progress.  This could cause any number of synchronization issues,
including the situation where a snapshot that was deleted could reappear
in the index-N file, even though its data was no longer present in the
repository.

This commit introduces a new custom type to the cluster state to
represent deletions in progress.  Now, another deletion cannot start if
a deletion is currently in progress.  Similarily, a snapshot or restore
cannot be started if a deletion is currently in progress.  In each case,
if attempting to run another snapshot/restore operation while a deletion
is in progress, a ConcurrentSnapshotExecutionException will be thrown.
This is the same exception thrown if trying to snapshot while another
snapshot is in progress, or restore while a snapshot is in progress.

Closes #19957
2016-12-25 19:00:20 -05:00
Adrien Grand 70594a66c7 Only run the unmapped+missing tests on 5.2+. 2016-12-23 09:38:20 +01:00
Adrien Grand e39942fc02 `value_type` is useful regardless of scripting. (#22160)
Today we only expose `value_type` in scriptable aggregations, however it is
also useful with unmapped fields. I suspect we never noticed because
`value_type` was not documented (fixed) and most aggregations are scriptable.

Closes #20163
2016-12-22 14:35:12 +01:00
Boaz Leskes 6249f1092f x_refresh.yaml tests should use unique index names and doc ids to ease debugging
This is to make it easier to grep the node logs
2016-12-21 10:25:33 +01:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
Isabel Drost-Fromm b1e0d698ac Merge branch 'master' into feature/rank-eval 2016-12-19 10:16:16 +01:00
Areek Zillur d44de0cecc Remove deprecated _suggest endpoint (#22203)
In #20305, _suggest endpoint was deprecated
in favour of using _search endpoint. This
commit removes the dedicated _suggest endpoint
entirely from master.
2016-12-16 12:06:02 -05:00
Masaru Hasegawa a0185c83a7 Merge pull request #21393 from masaruh/alias_boost
Resolve index names in indices_boost
2016-12-16 15:07:51 +09:00
Jason Tedor 43f71015a8 Add skip for include segment file sizes REST tests
This commit adds a skip for the include segment file sizes REST tests on
nodes less than or equal to version 5.1.1 as the stats APIs did not
correctly account for this parameter prior to version 5.1.2.

Relates #21879
2016-12-15 21:08:51 -05:00
Aaron Spiegel 80d3d790ae Fix handling of segment file sizes in stats API
This commit addresses an issue in the stats APIs where
include_segment_file_sizes was not being consumed leading to requests
containing this parameter being rejected.

Relates #21879
2016-12-15 07:29:11 -05:00
Isabel Drost-Fromm 5618d6ca49 Merge branch 'master' into feature/rank-eval 2016-12-15 10:29:26 +01:00
Areek Zillur cdd5fbe3a1 Deprecate _suggest endpoint in favour of _search (#20305)
* Replace _suggest endpoint to _search in docs

In 5.0, the _suggest endpoint is just sugar for _search
with suggestions specified. Users should move away from
using the _suggest endpoint, as it is marked as deprecated in 5.x and
will be removed in 6.0

* update docs to use _search endpoint instead of _suggest

* Add deprecation logging to RestSuggestAction

* Use search endpoint instead of suggest endpoint in rest tests
2016-12-14 21:49:53 -05:00
Isabel Drost-Fromm ea9f11965d Merge branch 'master' into feature/rank-eval 2016-12-14 12:47:12 +01:00
Simon Willnauer b7bcb5bb3a [TEST] Skip term / int partitioning tests in bwc tests pre 5.2.0 2016-12-13 22:20:44 +01:00
markharwood 4c6d17a176 Added tests for toXContent and fromXContent for IncludeExclude class.
New REST test revealed an issue with inconsistent hashing in partitioned
term tests which is also fixed in this change.

Closes #22102
2016-12-13 15:23:09 +00:00
Isabel Drost-Fromm 58342d4c9a Add checks to RankEvalSpec to safe guard against missing parameters. (#22026)
Add checks to RankEvalSpec to safe guard against missing parameters.

Fail early in case no metric is supplied, no rated requests are supplied or the search source builder is missing but no template is supplied neither.

Add stricter checks around rank eval request parsing: Fail if in a rated request we see both, a verbatim request as well as request
template parameters.


Relates to #21260
2016-12-13 11:21:57 +01:00
Isabel Drost-Fromm b2e8455745 Merge branch 'master' into feature/rank-eval 2016-12-13 10:45:22 +01:00
Artur Nowosielski 726f5dccc0 Rewrite filter queries in FiltersAggregationBuilder (#22076)
Queries must be rewritten before the query phase executes otherwise non-executable queries like `wrapper` query or `terms`  will fail or queries that require resources like script service can't access these service unless rewritten.

Relates to #21303
2016-12-11 14:37:12 +01:00
Masaru Hasegawa 3df2a086d4 Resolve index names in indices_boost
This change allows specifying alias/wildcard expression in indices_boost.
And added another format for specifying indices_boost. It accepts array of index name and boost pair.
If an index is included in multiple aliases/wildcard expressions, the first match will be used.
With new format, old format is marked as deprecated.

Closes #4756
2016-12-11 21:41:49 +09:00
Nik Everett 7a74a41a0c Fix test for changed message
Message is only changed in 5.2.0 so we shouldn't assert on it
if we're running with any nodes less than that version.
2016-12-10 10:35:14 -05:00
Nik Everett ddade1b5ac Improve the error message if task and node isn't found (#22062)
Improves the error message returned when looking up a task that
belongs to a node that is no longer part of the cluster. The new
error message tells the user that the node isn't part of the cluster.
This is useful because if you start a task and the node goes down
there isn't a record of the task at all. This hints to the user that
the task might have died with the node.

Relates to #22027
2016-12-09 15:50:46 -05:00
Yannick Welsch fca4f92fee Fix BWC condition on REST test
Adds a missing skip section to a REST test that was forgotten in #21998
2016-12-09 19:05:00 +01:00
Yannick Welsch db0660a7ea Reject external versioning and explicit version numbers on create (#21998)
Fixes an issue where indexing requests with operation type "create" auto-convert external versioning to internal versioning and silently ignore the version number instead of failing with an error message.
2016-12-09 14:21:22 +01:00
Jason Tedor 4aae017891 Skip IP range query REST test prior to 5.1.2
This commit adds a skip for the IP range query REST test on version
prior to 5.1.2 due to a exclusive bug on the top end of the range.
2016-12-08 16:40:39 -05:00
Adrien Grand 8fe4bc1b74 Fix REST test for ip range aggregations.
Relates to #22018
2016-12-08 18:00:56 +01:00
Isabel Drost-Fromm 5601049014 Merge branch 'master' into feature/rank-eval 2016-12-05 11:22:39 +01:00
Jason Tedor 7861a9dc3e Add skip for missing types REST test
This commit adds a skip for the missing types REST test. There was a bug
for an unclosed XContent object on version prior to version 5.0.3 which
is going to lead to different responses on vresions prior to 5.0.3 and
versions on or after version 5.0.3.
2016-12-01 22:10:47 -05:00
Ryan Ernst a6ad89bee0 Mappings: Fix get mapping when no indexes exist to not fail in response generation (#21924)
When there are no indexes, get mapping has a series of special cases.
Two of those expect the response object already started, and the other
two respond with an exception. Those two cases (types passed in but no
indexes and vice versa) would fail in their error response generation
because it did not expect an object to already be started in the json
generator. This change moves the object start to where it is needed for
the empty responses.

closes #21916
2016-12-01 16:57:12 -08:00
Isabel Drost-Fromm 2b24091361 Merge branch 'master' into feature/rank-eval 2016-11-28 11:45:29 +01:00
Yannick Welsch 590a6372ad Disable disk watermarks on REST tests (#21803)
REST tests use the default OOTB low/high disk watermarks of 85%/90%, which can make some tests fail if run on a machine with a fuller disk. This commit changes the watermarks in the same way as in IntegTestCase so that they're essentially ignored.
2016-11-25 19:52:52 +01:00
Luca Cavanna 720b165350 Search shards to print out aliases array together with alias filter (#21784)
With #21738 we added an indices section to the search shards api, that will return the concrete indices hit by the request, and eventually the corresponding alias filter.

The java API returns the AliasFilter object, which holds the filter itself and an array of aliases that pointed to the index in the original request. The REST layer doesn't print out the aliases array though. This commit adds the aliases array as well and tests for this.
2016-11-25 10:58:06 +01:00
Simon Willnauer 9809760eb0 Fix settings diff generation for affix, list and group settings (#21788)
Group, List and Affix settings generate a bogus diff that turns the actual
diff into a string containing a json structure for instance:

```
"action" : {
  "search" : {
    "remote" : {
      "" : "{\"my_remote_cluster\":\"[::1]:60378\"}"
    }
  }
}
```

which make reading the setting impossible. This happens for instance
if a group or affix setting is rendered via `_cluster/settings?include_defaults=true`
This change fixes the issue as well as several minor issues with affix settings that
where not accepted as valid setting today.
2016-11-24 21:53:04 +01:00
Christoph Büscher e1fe0dc462 Merge branch 'master' into feature/rank-eval 2016-11-24 08:57:26 +01:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language (#21607)
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Luca Cavanna db5a72774b Add indices and filter information to search shards api output (#21738)
Add indices and filter information to search shards api output

The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.

This is relevant since we have moved parsing of alias filters to the coordinating node.

Relates to #20916
2016-11-22 23:00:25 +01:00
Luca Cavanna db8b2dceea Remove ignored type parameter in search_shards api (#21688)
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.

This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.

The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
2016-11-22 17:22:33 +01:00
Christoph Büscher 1d3e58ab9f Merge branch 'master' into feature/rank-eval 2016-11-22 11:31:46 +01:00
Luca Cavanna c174e364a4 Add indices options tests to search api REST tests (#21701)
This is a followup to #21689 where we removed a misplaced try catch for IndexMissingException and IndexClosedException which was related to #9047 (at least for the index closed case). The code block within the change was moved as part of #20890, which made the catch redundant. It was somehow used before (e.g. in 5.0) but it doesn't seem that this catch had any effect. Added tests to verify that. In fact a specific catch added to the search api only would defeat the purpose of having common indices options that work throughout all our APIs.

Relates to #21689
2016-11-21 14:52:43 +01:00
Christoph Büscher 1964d45453 Merge branch 'master' into feature/rank-eval 2016-11-17 17:37:39 +01:00
Dimitrios Athanasiou dbbba8ffbe Add comment to explain flexible assertion in indices rollover IT 2016-11-17 14:49:10 +00:00
Dimitris Athanasiou a75320f89b Replace IndexAlreadyExistsException with ResourceAlreadyExistsException (#21494) 2016-11-17 14:30:21 +00:00
Nik Everett 4bc8feea1d Fix shrink tests on disks w/<15% free
The tests rely on the allocation decider allowing allocation and
the disk threshold decider wasn't allowing it. This came up from
time to in CI but *all* the time on my local machine because it is
a small ssd.
2016-11-16 14:46:52 -05:00
Jason Tedor d06a8903fd Merge branch 'master' into feature/seq_no
* master: (22 commits)
  Add proper toString() method to UpdateTask (#21582)
  Fix `InternalEngine#isThrottled` to not always return `false`. (#21592)
  add `ignore_missing` option to SplitProcessor (#20982)
  fix trace_match behavior for when there is only one grok pattern (#21413)
  Remove dead code from GetResponse.java
  Fixes date range query using epoch with timezone (#21542)
  Do not cache term queries. (#21566)
  Updated dynamic mapper section
  Docs: Clarify date_histogram bucket sizes for DST time zones
  Handle release of 5.0.1
  Fix skip reason for stats API parameters test
  Reduce skip version for stats API parameter tests
  Strict level parsing for indices stats
  Remove cluster update task when task times out (#21578)
  [DOCS] Mention "all-fields" mode doesn't search across nested documents
  InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
  Fixed bad asciidoc in boolean mapping docs
  Fixed bad asciidoc ID in node stats
  Be strict when parsing values searching for booleans (#21555)
  Fix time zone rounding edge case for DST overlaps
  ...
2016-11-16 09:10:35 -05:00
Christoph Büscher 6c6ddf0245 Merge branch 'master' into feature/rank-eval 2016-11-16 11:57:22 +01:00
Nik Everett e66261eee9 Handle release of 5.0.1
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.

Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
2016-11-15 19:31:28 -05:00
Jason Tedor cca22088ac Fix skip reason for stats API parameters test
This commit corrects the reasons that the stats API parameters tests are
skipped against 5.0 nodes.
2016-11-15 17:52:45 -05:00
Jason Tedor 6a8a3e77ed Reduce skip version for stats API parameter tests
This commit reduces the skip version for the stats API unrecognized
parameter tests now that the logic for unrecognized parameters has been
backported to 5.x.
2016-11-15 17:43:19 -05:00
Jason Tedor f5ac0e5076 Remove lenient stats parsing
Today when parsing a stats request, Elasticsearch silently ignores
incorrect metrics. This commit removes lenient parsing of stats requests
for the nodes stats and indices stats APIs.

Relates #21417
2016-11-15 12:17:26 -05:00
Christoph Büscher d6738fa650 Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
2016-11-15 14:56:36 +01:00
Boaz Leskes c9f49039d3 Merge remote-tracking branch 'upstream/master' into feature/seq_no 2016-11-15 10:14:47 +00:00
Simon Willnauer 4c53e55d34 [TEST] Add master_timeout to shrink call to fail faster 2016-11-15 10:59:59 +01:00
Simon Willnauer bdc942fa72 Enable 5.x to 6.x BWC tests
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 14:26:49 +01:00
Jason Tedor 1e7c424479 Merge branch 'master' into feature/seq_no
* master:
  ShardActiveResponseHandler shouldn't hold to an entire cluster state
  Ensures cleanup of temporary index-* generational blobs during snapshotting (#21469)
  Remove (again) test uses of onModule (#21414)
  [TEST] Add assertBusy when checking for pending operation counter after tests
  Revert "Add trace logging when aquiring and releasing operation locks for replication requests"
  Allows multiple patterns to be specified for index templates (#21009)
  [TEST] fixes rebalance single shard check as it isn't guaranteed that a rebalance makes sense and the method only tests if rebalance is allowed
  Document _reindex with random_score
2016-11-11 11:25:27 -05:00
Jason Tedor d3417fb022 Merge branch 'master' into feature/seq_no
* master: (516 commits)
  Avoid angering Log4j in TransportNodesActionTests
  Add trace logging when aquiring and releasing operation locks for replication requests
  Fix handler name on message not fully read
  Remove accidental import.
  Improve log message in TransportNodesAction
  Clean up of Script.
  Update Joda Time to version 2.9.5 (#21468)
  Remove unused ClusterService dependency from SearchPhaseController (#21421)
  Remove max_local_storage_nodes from elasticsearch.yml (#21467)
  Wait for all reindex subtasks before rethrottling
  Correcting a typo-Maan to Man-in README.textile (#21466)
  Fix InternalSearchHit#hasSource to return the proper boolean value (#21441)
  Replace all index date-math examples with the URI encoded form
  Fix typos (#21456)
  Adapt ES_JVM_OPTIONS packaging test to ubuntu-1204
  Add null check in InternalSearchHit#sourceRef to prevent NPE (#21431)
  Add VirtualBox version check (#21370)
  Export ES_JVM_OPTIONS for SysV init
  Skip reindex rethrottle tests with workers
  Make forbidden APIs be quieter about classpath warnings (#21443)
  ...
2016-11-10 23:40:33 -05:00
Alexander Lin 0219a211d3 Allows multiple patterns to be specified for index templates (#21009)
* Allows for an array of index template patterns to be provided to an
index template, and rename the field from 'template' to 'index_pattern'.

Closes #20690
2016-11-10 18:00:30 -05:00
Nik Everett 31be683e57 Skip alias name validation rest test on 5.0 cluster
Alias name validation was introduced in 5.1.
2016-11-08 09:36:47 -05:00
Nik Everett 3787ea27bf Validate alias names the same as index names
Applied (almost) the same rules we use to validate index names
to new alias names. The only rule not applies it "must be lowercase".
We have tests that don't follow that rule and I except there are lots
of examples of camelCase alias names in the wild. We can add that
validation but I'm not sure it is worth it.

Closes #20748

Adds an alias that starts with `#` to the BWC index and validates
that you can read from it and remove it. Starting with `#` isn't
allowed after 5.1.0/6.0.0 so we don't create the alias or check it
after those versions.
2016-11-08 08:23:12 -05:00
Christoph Büscher dbef96c69e Merge branch 'master' of github.com:elastic/elasticsearch into feature/rank-eval 2016-11-08 11:10:10 +01:00
Clinton Gormley 8383af6734 The routing query string param is supported by mget but was missing from the rest spec (#21357) 2016-11-07 14:14:52 +01:00
Christoph Büscher 1702304ecd Merge branch 'master' into feature/rank-eval 2016-11-07 11:02:22 +01:00
Nik Everett a13a050271 Add automatic parallelization support to reindex and friends (#20767)
Adds support for `?slices=N` to reindex which automatically
parallelizes the process using parallel scrolls on `_uid`. Performance
testing sees a 3x performance improvement for simple docs
on decent hardware, maybe 30% performance improvement
for more complex docs. Still compelling, especially because
clusters should be able to get closer to the 3x than the 30%
number.

Closes #20624
2016-11-04 20:59:15 -04:00
Jason Tedor 555084a226 Write -1 on unbounded queue in cat thread pool
Today when writing an unbounded queue_size in the cat thread pool API,
we write null. This commit modifies this so that the output is -1 so
that the output is always present, and always a numeric value.

Relates #21342
2016-11-04 15:17:55 -04:00
Luca Cavanna c2160a88b5 Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api (#20712)
Exist requests are supposed to never throw an exception, but rather return true or false depending on whether some resource exists or not. Indices exists does that for indices and accepts wildcard expressions too. The way the api works internally is by resolving indices and catching IndexNotFoundException: if an exception is thrown the index does not exist hence it returns false, otherwise it returns true. That works ok only if ignore_unavailable and allow_no_indices indices options are both set to false, meaning that they are strict and any missing index or wildcard expressions that resolves to no indices will lead to an exception that can be thrown and cause false to be returned.

Unfortunately the indices options have  been configurable up until now for this request, meaning that one can set ignore_unavailable or allow_no_indices to true and have the indices exist request return true for indices that really don't exist, which makes very little sense in the context of this api.

This commit removes the indicesOptions setter from the IndicesExistsRequest and makes settable only expandWildcardsOpen and expandWildcardsClosed, hence a subset of the available indices options. This way we can guarantee more consistent behaviour of the indices exists api. We can then remove the ignore_unavailable and allow_no_indices option from indices exists api spec
2016-11-04 19:26:37 +01:00
Jayson Minard a978f13660 Fix cat thread pool API spec
This commit fixes an issue with the cat thread pool API spec. Namely,
the name of the variable for the thread pool patterns parameter was
misnamed in the spec.

Relates #21332
2016-11-04 12:25:11 -04:00
Simon Willnauer 436ba7b5fc Validate the `_rollover` target index name early to also fail if dry_run=true (#21330)
Today we validate the target index name late and therefore don't fail for instance
if the target index already exists and `dry_run=true` was specified. This change
validates the index name before we early terminate if dry_run is set.

Closes #21149
2016-11-04 13:53:28 +01:00
Christoph Büscher 619e4c1a44 Merge branch 'master' into feature/rank-eval 2016-11-04 12:48:42 +01:00
Simon Willnauer 68bed9cd3b [TEST] Skip new tests using ingnore_unavailable in BWC tests against 5.0.0
5.0.0 nodes don't understand these parameters and will throw an exception
if a 5.0.0 node will receive the request.

Relates to #21281
2016-11-03 21:47:32 +01:00
Luca Cavanna 00e7026778 Read indices options in indices upgrade API (#21281)
With #21099 we removed support for the ignored allow_no_indices parameter in indices upgrade API. Truth is that ignore_unavailable and expand_wildcards were also ignored, in indices upgrade as well as upgrade status API. Those parameters are though supported internally and settable through java API, hence they should be all supported on the REST layer too.
2016-11-03 18:05:18 +01:00
Simon Willnauer 9015062dcb Rewrite Queries/Filter in FilterAggregationBuilder and ensure client usage marks query as non-cachable (#21303)
`FilterAggregationBuilder` today misses to rewrite queries which causes failures
if a query that uses a client for instance to lookup terms since it must be rewritten first.
This change also ensures that if a client is used from the rewrite context we mark the query as
non-cacheable.

Closes #21301
2016-11-03 16:48:55 +01:00
Simon Willnauer d77d4fa63a Consume `full_id` request parameter early (#21270)
Since we now validate all consumed request parameter, users can't specify
`_cat/nodes?full_id=true|false` anymore since this parameter is consumed late.
This commit adds a test for this parameter and consumes it before request is processed.

Closes #21266
2016-11-03 10:31:35 +01:00
Simon Willnauer c4c4a3a504 Fix `_shrink` test to work under a mixed version cluster (#21262)
today the `_shrink` tests do relocate all shards to a single node in
the cluster. Yet, that is not always possible since the only node we can
safely identify in the cluster is the master and if the master is a BWC
node in such a cluster we won't be able to relocate shards that have a
primary on the newer version nodes since allocation deciders forbid this.

This change restricts allocation for that index when the index is created
to restrict allocation to the master that guarantees that all primaries
are on the same node which is sufficient for the `_shrink` API to run.
2016-11-02 17:09:30 +01:00
Adrien Grand 52de0645fb Remove `lowercase_expanded_terms` and `locale` from query-parser options. (#20208)
Lucene 6.2 introduces the new `Analyzer.normalize` API, which allows to apply
only character-level normalization such as lowercasing or accent folding, which
is exactly what is needed to process queries that operate on partial terms such
as `prefix`, `wildcard` or `fuzzy` queries. As a consequence, the
`lowercase_expanded_terms` option is not necessary anymore. Furthermore, the
`locale` option was only needed in order to know how to perform the lowercasing,
so this one can be removed as well.

Closes #9978
2016-11-02 14:25:08 +01:00
Isabel Drost-Fromm 0b8a2e40cb First step towards supporting templating in rank eval requests. (#20374)
This adds support for templating in rank eval requests.

Relates to #20231

Problem: In it's current state the rank-eval request API forces the user to repeat complete queries for each test request. In most use cases the structure of the query to test will be stable with only parameters changing across requests, so this looks like lots of boilerplate json for something that could be expressed in a more concise way.

Uses templating/ ScriptServices to enable users to submit only one test request template and let them only specify template parameters on a per test request basis.
2016-11-01 11:36:22 +01:00
Simon Willnauer 36c86c2b00 Fix `_cat/store` sort test to use a reliable sort value
using the store size is not reliable since it's cached and might
be stale depending on the timing when the cache entry was generated.
2016-10-28 16:04:33 +02:00
Simon Willnauer 97cc426a89 Fix bwc cluster formation in order to run BWC tests against a mixed version cluster (#21145)
This fixes our cluster formation task to run REST tests against a mixed version cluster.
Yet, due to some limitations in our test framework `indices.rollover` tests are currently
disabled for the BWC case since they select the current master as the merge node which
happens to be a BWC node and we can't relocate all shards to it since the primaries are on
a higher version node. This will be fixed in a followup.

Closes #21142

Note: This has been cherry-picked from 5.0 and fixes several rest tests
as well as a BWC break in `OsStats.java`
2016-10-27 17:03:53 +02:00
Christoph Büscher 51a2e3bf1e Merge branch 'master' into feature/rank-eval 2016-10-27 11:11:37 +02:00
Jun Ohtani a66c76eb44 Merge pull request #20704 from johtani/remove_request_params_in_analyze_api
Removing request parameters in _analyze API
2016-10-27 17:43:18 +09:00
Simon Willnauer e745015325 Return target index name even if _rollover conditions are not met (#21138)
Today we return the old index name as the target / new index name.
This change passes the correct rollover index name to the response.
2016-10-27 09:20:46 +02:00
Christoph Büscher 67e2de6702 Merge branch 'master' into feature/rank-eval 2016-10-25 11:05:31 +02:00
Greg Marzouka ab22307cf2 Remove allow_no_indices from indices.upgrade 2016-10-24 15:06:17 -04:00
Adrien Grand d88239ba63 `ip_range` aggregation should accept null bounds. (#21043)
* `ip_range` aggregation should accept null bounds.

Closes #21006

* test

* iter
2016-10-21 14:39:00 +02:00
javanna 7ef4076160 [TEST] fix multi_get REST test
After #20659 it's concreteSingleIndex that throws exception rather than resolvIndexRouting. The error message is slightly different.
2016-10-20 11:32:12 +02:00
Jason Tedor 7a55cca122 Whitelist node stats indices level parameter
When indices stats are requested via the node stats API, there is a
level parameter to request stats at the index, node, or shards
level. This parameter was not whitelisted when URL parsing was made
strict. This commit whitelists this parameter.

Additionally, there was some leniency in the parsing of this parameter
that has been removed.

Relates #21024
2016-10-19 22:01:17 -04:00
Honza Král 4bfd21b3dd [test] Required parameters are checked client-side 2016-10-19 15:43:52 +02:00
Christoph Büscher e8e65c3a1e Merge branch 'master' into feature/rank-eval 2016-10-19 11:37:29 +02:00
Clinton Gormley 718a86e256 Added missing POST method to create.json REST spec
Relates to #20924
2016-10-17 17:58:31 +02:00
Clinton Gormley 42458e836a Add a REST spec for the create API (#20924)
The create request now requires that an ID be present.
Currently the clients hard code a create method, but
we should just add a create REST spec so this method
can be autogenerated.
2016-10-17 17:57:25 +02:00
Christoph Büscher f927a235b3 Merge branch 'master' into feature/rank-eval 2016-10-17 14:50:10 +02:00
Clinton Gormley 8d602c6692 Updated REST test README to include required params when running tests 2016-10-16 13:25:25 +02:00
Christoph Büscher cd9d07b91b Merge branch 'master' into feature/rank-eval 2016-10-14 17:03:30 +02:00
Simon Willnauer cff5993318 Parse alias filters on the coordinating node (#20916)
Today we don't parse alias filters on the coordinating node, we only forward
the alias patters to executing node and resolve it late. This has several problems
like requests that go through filtered aliases are never cached if they use date math,
since the parsing happens very late in the process even without rewriting. It also used
to be processed on every shard while we can only do it once per index on the coordinating node.
Another nice side-effect is that we are never prone to cluster-state updates that change an alias,
all nodes will execute the exact same alias filter since they are process based on the same
cluster state.
2016-10-14 16:26:09 +02:00
Tanguy Leroux aabbbc2202 Remove duplicate timeout parameter in Delete/Update-By-Query REST Specs (#20934)
This commit removes the duplicated "timeout" parameter introduced in #20915
2016-10-14 13:21:39 +02:00
Tanguy Leroux f5e1c6d330 Update Delete/Update-By-Query REST Specs (#20915)
This commit removes unused parameters from the Update-By-Query and Delete-By-Query REST specification files.
2016-10-14 09:26:33 +02:00
Clinton Gormley 389d365697 Fix YAML formatting in several REST tests 2016-10-13 09:58:25 +02:00
Clinton Gormley 4c62e14c50 Made REST query param types consistent
text -> string
2016-10-12 14:49:01 +02:00
Clinton Gormley e1ad00f07a Made REST query param types consistent
duration -> time
integer,float -> number
2016-10-12 14:42:19 +02:00
Isabel Drost-Fromm 20c1e25609 Merge branch 'master' into feature/rank-eval 2016-10-12 14:41:17 +02:00
Alexander Lin 10ddcc4616 Adding built-in sorting capability to _cat apis. (#20658)
* Adding built-in sorting capability to _cat apis.

Closes #16975

* addressing pr comments

* changing value types back to original implementation and fixing cosmetic issues

* Changing compareTo, hashCode of value types to a better implementation

* Changed value compareTos to use Double.compare instead of if statements + fixed some failed unit tests
2016-10-11 12:29:22 -04:00
Tanguy Leroux 3b578db365 MultiGet should not fail entirely if alias resolves to many indices (#20858)
MultiGet should not fail entirely when one of the items of a multi get request refers to an alias that points to multiple indices.

closes #20845
2016-10-11 18:07:39 +02:00
Christoph Büscher 0c25cfbd16 Merge branch 'master' into feature/rank-eval 2016-10-10 12:10:34 +02:00
Jun Ohtani 945fa499d2 Deprecating request parameters in _analyze API
Remove params in indices.analyze.json
Fix REST changes

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani 370f0b885e Removing request parameters in _analyze API
Remove request params in _analyze API without index param
Change rest-api-test using JSON
Change docs using JSON

Closes #20246
2016-10-07 16:23:24 +09:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Jason Tedor 25fd9e26c4 Merge branch 'master' into feature/seq_no
* master: (1199 commits)
  [DOCS] Remove non-valid link to mapping migration document
  Revert "Default `include_in_all` for numeric-like types to false"
  test: add a test with ipv6 address
  docs: clearify that both ip4 and ip6 addresses are supported
  Include complex settings in settings requests
  Add production warning for pre-release builds
  Clean up confusing error message on unhandled endpoint
  [TEST] Increase logging level in testDelayShards()
  change health from string to enum (#20661)
  Provide error message when plugin id is missing
  Document that sliced scroll works for reindex
  Make reindex-from-remote ignore unknown fields
  Remove NoopGatewayAllocator in favor of a more realistic mock (#20637)
  Remove Marvel character reference from guide
  Fix documentation for setting Java I/O temp dir
  Update client benchmarks to log4j2
  Changes the API of GatewayAllocator#applyStartedShards and (#20642)
  Removes FailedRerouteAllocation and StartedRerouteAllocation
  IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource (#20638)
  Smoke tester: Adjust to latest changes (#20611)
  ...
2016-09-29 00:22:31 +02:00
Christoph Büscher 92baf14c0a Merge branch 'master' into feature/rank-eval 2016-09-27 11:59:35 +02:00
Martijn Laarman e5f1eb5dc9 change health from string to enum (#20661) 2016-09-26 22:44:30 +02:00
Christoph Büscher 1909877728 Merge branch 'master' into feature/rank-eval 2016-09-22 12:21:20 +02:00
Simon Willnauer 0151974500 `_flush` should block by default (#20597)
This commit changes the default behavior of `_flush` to block if other flushes are ongoing.
This also removes the use of `FlushNotAllowedException` and instead simply return immediately
by skipping the flush. Users should be aware if they set this option that the flush might or might
not flush everything to disk ie. no transactional behavior of some sort.

Closes #20569
2016-09-21 14:20:24 +02:00
Christoph Büscher 306ea1c613 Merge branch 'master' into feature/rank-eval 2016-09-21 10:58:36 +02:00
Tanguy Leroux 0fee59127e [Tests] Fix cat.templates to work when other templates exist
This commit fixes the cat.templates REST tests so that it works when other templates exist (like monitoring)
2016-09-20 12:05:21 +02:00
Christoph Büscher 8772725d46 Merge branch 'master' into feature/rank-eval 2016-09-20 11:23:24 +02:00
Alexander Lin d31a8e6558 Provides a cat api endpoint for templates. (#20545)
Adds a cat api endpoint: /_cat/templates and its more specific version, /_cat/templates/{name}.

It looks something like:

$ curl "localhost:9200/_cat/templates?v"
name                  template     order version
sushi_california_roll *avocado*    1     1
pizza_hawaiian        *pineapples* 1
pizza_pepperoni       *pepperoni*  1

The specified version (only allows * globs) looks like:

$ curl "localhost:9200/_cat/templates/pizza*"
name            template     order version
pizza_hawaiian  *pineapples* 1
pizza_pepperoni *pepperoni*  1

Partially specified columns:

$ curl "localhost:9200/_cat/templates/pizza*?v=true&h=name,template"
name            template
pizza_hawaiian  *pineapples*
pizza_pepperoni *pepperoni*

The help text:

$ curl "localhost:9200/_cat/templates/pizza*?help"
name     | n | template name
template | t | template pattern string
order    | o | template application order number
version  | v | version

Closes #20467
2016-09-20 10:40:23 +02:00
David Pilato ed4d0881b1 Add profile and explain parameters to template API
We can now run templates using `explain` and/or `profile` parameters.
Which is interesting when you have defined a complicated profile but want to debug it in an easier way than running the full query again.

You can use `explain` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
```

You can use `profile` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
```
2016-09-19 17:52:13 +02:00
Jason Tedor 9082c604c1 Fix command-line arguments in REST API spec
The command-line arguments for Elasticsearch must now be specified using
-E. This commit fixes the usage of command-line arguments in the REST
API spec README.
2016-09-19 07:47:02 -04:00
Jason Tedor b456823434 Fix leftover node attributes usage
Previously node attributes could be set via node.* but this now requires
using node.attr.*. This commit fixes some leftover usages of the old
way.
2016-09-19 07:45:31 -04:00
Christoph Büscher 8f88917c4e Merge branch 'master' into feature/rank-eval 2016-09-19 11:50:53 +02:00
Nik Everett d0be96df7b Clean up snapshots after each REST test
The only repository we can be sure is safe to clean is `fs` so we clean
any snapshots in those repositories after each test. Other repositories
like url and azure tend to throw exceptions rather than let us fetch
their contents during the REST test. So we clean what we can....

Closes #18159
2016-09-15 14:49:11 -04:00
Luca Cavanna 37489c3274 Add clusterUUID to RestMainAction output (#20503)
Add clusterUUID to RestMainAction output

GET / now returns the clusterUUID as well as part of its output for monitoring purposes
2016-09-15 16:25:17 +02:00
Christoph Büscher f91ee9a897 Merge branch 'master' into feature/rank-eval 2016-09-15 11:38:40 +02:00
Lee Hinman 863a199c42 Correct the description for refresh
The refresh description should indicate that the affected shards are
refreshed as opposed to the entire index.

This was raised as a discrepancy on
discuss (https://discuss.elastic.co/t/refresh-parameter-of-index-api/59008/2)
on the .NET client that originates from code generated from the rest api
spec. The description has been updated in master but should be updated
for the 2.4.0 release.
2016-09-14 10:58:03 -06:00
Ryan Ernst c42a5b330a Fix spacing for json formatting 2016-09-13 16:04:00 -07:00
Ryan Ernst 81e4cdfbc0 Build: Add pom building and associated files to rest api spec jar (#20460)
We put the rest api spec into a jar for upload to maven, so that we can
use within external rest tests. This change adds making a pom for maven
(as well as producing sources and javadoc jars, even though they will be
empty, because maven central requires them).
2016-09-13 14:05:52 -07:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs (#20166)
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155
2016-09-13 20:54:41 +02:00
Jason Tedor c7bfbe3e69 Add health status parameter to cat indices API
This commit adds a health status parameter to the cat indices API for
filtering on indices that match the specified status (green|yellow|red).

Relates #20393
2016-09-13 07:57:18 -04:00
Isabel Drost-Fromm 0e707f241e Add docs to template support for _msearch (#17382)
Add docs to template support for _msearch

Relates to #10885
Relates to #15674

* Reference those docs from the rest api spec for _msearch/template support.
2016-09-13 13:19:25 +02:00
Christoph Büscher 9e2ec53458 Merge branch 'master' into feature/rank-eval 2016-09-13 12:47:42 +02:00
Ali Beyad 494ad0d572 [TESTS] for the REST cluster stats test, if free or used (#20434)
memory are much less than the total memory, the percentage
returned could be 0%. The yaml tests check that the free/used
percentage are valid values by asserting `is_true`, but it
turns out that `is_true` returns false if the value is
assigned but it is 0 or even the string "0". This commit
changes the assertion in the yaml test to ensure the value
is greater than or equal to 0 instead.
2016-09-12 19:47:20 -04:00
Christoph Büscher 2fde4b2883 Merge branch 'master' into feature/rank-eval 2016-09-12 11:30:40 +02:00
Lee Hinman 49695af2ac Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-08 15:29:47 -06:00
Chris Earle 3713d7994d id to name 2016-09-08 17:27:01 -04:00
Chris Earle 07ccabbebc Add "version" field to Templates
This adds a version field to Templates, which is itself is unused by Elasticsearch, but exists for users to better manage their own templates. Like description, it's optional.
2016-09-08 16:01:42 -04:00
Lee Hinman b418889665 Revert "Remove FORCE version_type"
This reverts commit b4cc3cd35d.
2016-09-08 07:43:55 -06:00
Lee Hinman b4cc3cd35d Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-07 13:05:18 -06:00
Chris Earle 6a7309c09a Add "version" field to Pipelines
This adds a version field to Pipelines, which is itself is unused by Elasticsearch, but exists for users to better manage their own pipelines.
2016-09-07 10:27:40 -04:00
Christoph Büscher 11116db337 Merge branch 'master' into feature/rank-eval 2016-09-02 10:49:33 +02:00
Luca Cavanna faa03ad9fa Merge pull request #20255 from javanna/enhancement/cluster_stats_available_memory
Add mem section back to cluster stats
2016-09-02 10:19:51 +02:00
Adrien Grand 5bfab76c96 Source filtering should keep working when the source contains numbers greater than `Long.MAX_VALUE`. #20278
Currently it does not because our parsers do not support big integers/decimals
(on purpose) but we do not have to ask our parser for the number type, we can
just ask the jackson parser for a number representation of the value with the
right type.

Note that I did not add similar tests for big decimals because Jackson seems to
never return big decimals, even for decimal values that are out of the range of
values that can be represented by doubles.

Closes #11508
2016-09-02 08:56:04 +02:00
Christoph Büscher 1c02e48a8f Merge branch 'master' into feature/rank-eval
Conflicts:
	test/framework/src/main/java/org/elasticsearch/test/TestCluster.java
2016-09-01 13:06:21 +02:00
javanna 5f299ff46f add mem section back to cluster stats
The mem section was buggy in cluster stats and removed. It is now added back with the same structure as in node stats, containing total memory, available memory, used memory and percentages. All the values are the sum of all the nodes across the cluster (or at least the ones that we were able to get the values from).
2016-09-01 11:26:03 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Isabel Drost-Fromm 8f71c29c16 Merge branch 'master' into feature/rank-eval 2016-08-30 14:20:43 +02:00
Tanguy Leroux b4245c7ad9 Add exclusion filters support to filter_path
This commit adds the support for exclusion filter to the response filtering (filter_path) feature. It changes the XContentBuilder APIs so that it now accepts two types of filters: inclusive and exclusive. Filters are no more String arrays but sets of String instead.
2016-08-30 09:08:30 +02:00
Greg Marzouka 2363c7dcdd Merge pull request #20186 from gmarz/spec/wait_for_active_shards
[SPEC] Fix type for wait_for_active_shards (string => number)
2016-08-29 10:19:33 -04:00
Greg Marzouka 84f05cd7d5 [SPEC] Change type of wait_for_active_shards from number to string 2016-08-29 09:31:01 -04:00
Jun Ohtani 2a00c9dc46 Merge pull request #19860 from johtani/fix/validate_empty_field_name
Validate blank field name
2016-08-29 11:52:18 +09:00
Yannick Welsch 1b75cb63a2 Add recovery source to ShardRouting (#19516)
Adds an explicit recoverySource field to ShardRouting that characterizes the type of recovery to perform:

- fresh empty shard copy
- existing local shard copy
- recover from peer (primary)
- recover from snapshot
- recover from other local shards on same node (shrink index action)
2016-08-27 16:11:10 +02:00
Jun Ohtani 450f47d5b5 Validate blank field name
add validation and validate only 5.0+
Add tests before 5.0

Closes #19251
2016-08-26 20:10:33 +09:00
Adrien Grand 3ed0da5a58 GET operations should not extract fields from `_source`. #20158
This makes GET operations more consistent with `_search` operations which expect
`(stored_)fields` to work on stored fields and source filtering to work on the
`_source` field. This is now possible thanks to the fact that GET operations
do not read from the translog anymore (#20102) and also allows to get rid of
`FieldMapper#isGenerated`.

The `_termvectors` API (and thus more_like_this too) was relying on the fact
that GET operations would extract fields from either stored fields or the source
so the logic to do this that used to exist in `ShardGetService` has been moved
to `TermVectorsService`. It would be nice that term vectors do not rely on this,
but this does not seem to be a low hanging fruit.
2016-08-26 10:35:23 +02:00
Jason Tedor bc136a90d5 Add network types to cluster stats
The network types in use on a cluster can be useful information to have,
so this commit adds aggregate metrics for the network types in use in a
cluster to the cluster stats.

Relates #20144
2016-08-25 21:08:05 -04:00
Isabel Drost-Fromm 518e09d028 Merge branch 'master' into feature/rank-eval 2016-08-25 12:13:59 +02:00
Jim Ferenczi 4682fc34ae Add the ability to disable the retrieval of the stored fields entirely
This change adds a special field named _none_ that allows to disable the retrieval of the stored fields in a search request or in a TopHitsAggregation.

To completely disable stored fields retrieval (including disabling metadata fields retrieval such as _id or _type) use _none_ like this:

````
POST _search
{
   "stored_fields": "_none_"
}
````
2016-08-24 16:40:08 +02:00
Simon Willnauer c499427166 Use _refresh instead of reading from Translog in the RT GET case (#20102)
Today we do a lot of accounting inside the engine to maintain locations
of documents inside the transaction log. This is only needed to ensure
we can return the documents source from the engine if it hasn't been refreshed.
Aside of the added complexity to be able to read from the currently writing translog,
maintainance of pointers into the translog this also caused inconsistencies like different values
of the `_ttl` field if it was read from the tlog or not. TermVectors are totally different if
the document is fetched from the tranlog since copy fields are ignored etc.

This chance will simply call `refresh` if the documents latest version is not in the index. This
streamlines the semantics of the `_get` API and allows for more optimizations inside the engine
and on the transaction log. Note: `_refresh` is only called iff the requested document is not refreshed
yet but has recently been updated or added.

#Relates to #19787
2016-08-24 15:30:08 +02:00
Clinton Gormley 336ec0ac9a Renamed REST spec reindex.rethrottle to reindex_rethrottle
There is already a reindex method, so reindex can't
also be used as a namespace.
2016-08-24 13:30:48 +02:00
Isabel Drost-Fromm e5fd0cb3a2 Merge branch 'master' into feature/rank-eval 2016-08-24 12:44:37 +02:00
Ali Beyad 1c9b64e09a Adds ignoreUnavailable option to the snapshot status API (#20066)
Adds ignoreUnavailable to the snapshot status API to be consistent
with the get snapshots API which has a similar parameter. If
ignoreUnavailable is set to true, then the snapshot status request
will ignore any snapshots that were not found in the repository,
instead of throwing a SnapshotMissingException.

Closes #18522
2016-08-19 16:19:56 -04:00
Christoph Büscher 1082030fa0 Merge branch 'master' into feature/rank-eval 2016-08-19 10:15:22 +02:00
Adrien Grand a4ea7e7223 Switch indices.exists_type from `{index}/{type}` to `{index}/_mapping/{type}`. #20055
This will help remove types as we will need `{index}/{id}` to tell whether a
document exists.

Relates #15613
2016-08-19 09:18:24 +02:00
Christoph Büscher e58e25f338 Merge branch 'master' into feature/rank-eval 2016-08-18 10:43:48 +02:00
Clinton Gormley 7da9d826ff Update ingest.get_pipeline.json
The `id` parameter is not required

Closes #20010
2016-08-17 14:40:59 +02:00
Christoph Büscher ff17b9b38f Merge branch 'master' into feature/rank-eval 2016-08-17 11:06:32 +02:00
Adrien Grand d894db1590 Only use `PUT` for index creation, not POST. #20001
Currently both `PUT` and `POST` can be used to create indices. This commit
removes support for `POST index_name` so that we can use it to index documents
with auto-generated ids once types are removed.

Relates #15613
2016-08-17 10:15:42 +02:00
Christoph Büscher fe2bfc3a27 Merge branch 'master' into feature/rank-eval 2016-08-16 10:42:55 +02:00
Ali Beyad 5ba06b6487 Removes support for adding aliases to analyzers. Indices created pre 5.x (#19994)
that have analyzer aliases in their analysis settings will still work, but
any attempts to create an alias for analyzers in newly created indices
will result in an IllegalArgumentException.

As a result, the setting `index.analysis.analyzer.{analyzerName}.alias` is
no longer supported.

Closes #18244
2016-08-15 16:17:58 -04:00
Christoph Büscher ae721ef79b Merge branch 'master' into feature/rank-eval 2016-08-11 11:39:05 +02:00
Ryan Ernst d89540be9c Add rest put mapping test with dot in fieldname 2016-08-10 15:35:53 -07:00
Areek Zillur d107141bf6 Remove payload option from completion suggester
The payload option was introduced with the new completion
suggester implementation in v5, as a stop gap solution
to return additional metadata with suggestions.

Now we can return associated documents with suggestions
(#19536) through fetch phase using stored field (_source).
The additional fetch phase ensures that we only fetch
the _source for the global top-N suggestions instead of
fetching _source of top results for each shard.
2016-08-08 16:04:06 -04:00
Christoph Büscher 3895aa470a Merge branch 'master' into feature/rank-eval 2016-08-05 12:09:07 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Christoph Büscher 1b4b4a6298 Merge branch 'master' into feature/rank-eval 2016-08-04 12:52:23 +02:00
Jason Tedor 533412e36f Improve cat thread pool API
Today, when listing thread pools via the cat thread pool API, thread
pools are listed in a column-delimited format. This is unfriendly to
command-line tools, and inconsistent with other cat APIs. Instead,
thread pools should be listed in a row-delimited format.

Additionally, the cat thread pool API is limited to a fixed list of
thread pools that excludes certain built-in thread pools as well as all
custom thread pools. These thread pools should be available via the cat
thread pool API.

This commit improves the cat thread pool API by listing all thread pools
(built-in or custom), and by listing them in a row-delimited
format. Finally, for each node, the output thread pools are sorted by
thread pool name.

Relates #19721
2016-08-03 23:02:13 -04:00
Igor Motov 22e63b4783 Fixes cat tasks operation in detailed mode
Currently the cat tasks operation fails in the detailed mode.

Closes #19755
2016-08-02 15:21:31 -04:00
Christoph Büscher 0188363a40 Merge branch 'master' into feature/rank-eval 2016-08-02 12:05:44 +02:00
Ali Beyad d93f7d6085 Refactors ActiveShardCount 2016-08-01 13:35:29 -04:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Alexander Lin 9ac6389e43 Rename operation to result and reworking responses
* Rename operation to result and reworking responses
* Rename DocWriteResponse.Operation enum to DocWriteResponse.Result

These are just easier to interpret names.

Closes #19664
2016-08-01 10:42:58 -04:00
Christoph Büscher 8efccd1165 Merge branch 'master' into feature/rank-eval 2016-08-01 11:50:14 +02:00
Tanguy Leroux 64ca834722 Remove unnecessary indices refresh and wait for green status in REST tests 2016-08-01 11:09:56 +02:00
Tanguy Leroux 737db98bd7 /_cat/shards should support wilcards for indices
closes #19634
2016-08-01 11:09:48 +02:00
Tanguy Leroux e642187642 Remove unnecessary indices refresh from cluster.state/20_filtering.yaml file 2016-08-01 09:22:12 +02:00
Tanguy Leroux 7d4f557aa3 Allow routing table to be filtered by index pattern
Before this commit when an index pattern is used to filter the cluster state, only indices metadata are populated and routing table is just empty. This commit aligns the behavior of the filtering of cluster state's routing table with the filtering of cluster state's metadata so that coherent data are returned for both routing table & metadata when index pattern is requested.
2016-08-01 09:22:12 +02:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes #19585
2016-07-29 17:58:30 +02:00
Christoph Büscher f31fecc5a4 Merge branch 'master' into feature/rank-eval 2016-07-29 12:13:15 +02:00
Areek Zillur 4e3602a790 Add zero-padding to auto-generated rollover index name increment
closes #19484
2016-07-27 10:50:47 -04:00
Christoph Büscher 3ac35d1f4a Merge branch 'master' into feature/rank-eval 2016-07-27 11:14:30 +02:00
Chris Earle 0553ba9151 [Ingest] Add REST _ingest/pipeline to get all pipelines
This adds an extra REST handler for "_ingest/pipeline" so that users do not need to supply "_ingest/pipeline/*" to get all of them.

- Also adds a teardown section to related REST-tests for ingest.
2016-07-26 13:48:15 -04:00
Christoph Büscher e1415d6519 Merge pull request #19595 from cbuescher/fix-19422
Allow empty json object in request body in `_count` API.
2016-07-26 18:17:52 +02:00
Alexander Lin 8f2882a442 Add _operation field to index, update, delete responses
Performing the bulk request shown in #19267 now results in the following:
```
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"create","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":201}
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"noop","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":200}
```
2016-07-26 11:16:19 -04:00
Christoph Büscher 91f606988a Merge branch 'master' into feature/rank-eval 2016-07-26 15:49:07 +02:00
Christoph Büscher b861ec1cc0 Allow empty json object in request body in `_count` API
When the request body is missing, all documents in the target index are counted.
As mentioned in #19422, the same should happen when the request body is an empty
json object. This is also the behaviour for the `_search` endpoint and the two
APIs should behave in the same way.
2016-07-26 09:54:05 +02:00
Boaz Leskes 03fbc91816 allow for a `-` in a node name 2016-07-24 09:02:30 +02:00
Boaz Leskes cd596772ee Persistent Node Names (#19456)
With #19140 we started persisting the node ID across node restarts. Now that we have a "stable" anchor, we can use it to generate a stable default node name and make it easier to track nodes over a restarts. Sadly, this means we will not have those random fun Marvel characters but we feel this is the right tradeoff.

On the implementation side, this requires a bit of juggling because we now need to read the node id from disk before we can log as the node node is part of each log message. The PR move the initialization of NodeEnvironment as high up in the starting sequence as possible, with only one logging message before it to indicate we are initializing. Things look now like this:

```
[2016-07-15 19:38:39,742][INFO ][node                     ] [_unset_] initializing ...
[2016-07-15 19:38:39,826][INFO ][node                     ] [aAmiW40] node name set to [aAmiW40] by default. set the [node.name] settings to change it
[2016-07-15 19:38:39,829][INFO ][env                      ] [aAmiW40] using [1] data paths, mounts [[ /(/dev/disk1)]], net usable_space [5.5gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-07-15 19:38:39,830][INFO ][env                      ] [aAmiW40] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 19:38:39,837][INFO ][node                     ] [aAmiW40] version[5.0.0-alpha5-SNAPSHOT], pid[46048], build[473d3c0/2016-07-15T17:38:06.771Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2016-07-15 19:38:40,980][INFO ][plugins                  ] [aAmiW40] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy, transport-netty], plugins []
[2016-07-15 19:38:43,218][INFO ][node                     ] [aAmiW40] initialized
```

Needless to say, settings `node.name` explicitly still works as before.

The commit also contains some clean ups to the relationship between Environment, Settings and Plugins. The previous code suggested the path related settings could be changed after the initial Environment was changed. This did not have any effect as the security manager already locked things down.
2016-07-23 22:46:48 +02:00
Ali Beyad 83a137b25c Fixes REST test that is designed to timeout on index creation by
making the test wait until all urgent requests are completed before
finishing, so that tear down can properly delete the created index
and cleanup. Without this wait, it was possible that the test would
finish and cleanup the deleted indices would happen before the
index creation even processed, causing the test to leave a created
index behind.
2016-07-21 09:14:41 -04:00
Isabel Drost-Fromm 2d57fd10b1 Merge branch 'master' into feature/rank-eval 2016-07-21 14:54:49 +02:00
Karel Minarik 8c721b10af Test: Fixed incorrect YAML indentation in the `indices.put_template/10_basic.yaml` test
The Ruby YAML parser ignores the `do` actions when they are not indented,
making the test suite fail.

Related: #19506

Closes #19529
2016-07-21 14:17:17 +02:00
Simon Willnauer 302c7a521a Fix analyzer alias processing (#19506)
In the lack of tests the analyzer.alias feature was pretty much not working
at all on current master. Issues like #19163 showed some serious problems for users
using this feature upgrading to an alpha version.
This change fixes the processing order and allows aliases to be set for
existing analyzers like `default`. This change also ensures that if `default`
is aliased the correct analyzer is used for `default_search` etc.

Closes #19163
2016-07-21 09:32:47 +02:00
Jun Ohtani cebad703fe Analyze: Specify anonymous char_filters/tokenizer/token_filters in the analyze API
Add parser for anonymous char_filters/tokenizer/token_filters
Using Settings in AnalyzeRequest for anonymous definition
Add breaking changes document

Closed #8878
2016-07-21 11:06:36 +09:00
Karel Minarik 5bab65d886 Fixed incorrect YAML indentation in the "Rollover" tests again
As part of changes in d78f40fb1e, a fix to the YAML
indentation has been reverted, see location: d78f40fb1e (diff-eaf129528b571da2cafdfd5490c12453)

This patch fixes the YAML notation back.

/cc @abeyad

Closes #19482
2016-07-18 19:33:28 +02:00
Nik Everett d573541f66 Support requests_per_second=-1 to mean no throttling in reindex
This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes #19089
2016-07-18 13:05:06 -04:00
Ali Beyad 19d0dbcd17 Removes waiting for yellow cluster health upon index (#19460)
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).

Relates #19450
2016-07-15 17:18:34 -04:00
Ali Beyad 687e2e12b3 Merge pull request #19450 from elastic/feature/friendly-index-creation
Makes index creation more friendly
2016-07-15 11:48:21 -04:00
Ali Beyad d78f40fb1e Index creation waits for active shard copies before returning (#18985)
Before returning, index creation now waits for the configured number
of shard copies to be started. In the past, a client would create an
index and then potentially have to check the cluster health to wait
to execute write operations. With the cluster health semantics changing
so that index creation does not cause the cluster health to go RED,
this change enables waiting for the desired number of active shards
to be active before returning from index creation.

Relates #9126
2016-07-15 11:19:27 -04:00
Honza Král e21b1e8066 [TEST] add 'yaml' feature for the test runner (#19436)
Also renamed 30_yaml.yaml to 30_json.yaml since it tests json, not yaml
2016-07-14 17:30:32 +02:00
Simon Willnauer 4156a4bebb Add support for `wait_for_events` to the `_cluster/health` REST endpoint (#19432)
The Java API supports this while mostly used for tests it can also be useful in
production environments. For instance if something is automated like a settings change
and we execute some health right after it the settings update might have some consequences
like a reroute which hasn't been fully applied since the preconditions are not fulfilled yet.
For instance if not all shards started the settings update is applied but the reroute won't move
currently initializing shards like in the shrink API test. Sure this could be done by waiting for
green before but if the cluster moves shards due to some side-effects waiting for all events is
still useful. I also took the chance to add unittests to Priority.java

Closes #19419
2016-07-14 12:33:29 +02:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes #13729
2016-07-13 15:07:36 +02:00
Lee Hinman 95cf2407ee Merge remote-tracking branch 'dakrone/include-cluster-info-in-explain-api' 2016-07-12 16:26:46 -06:00
Lee Hinman 58db63b610 Expose the ClusterInfo object in the allocation explain output
This adds an optional parameter to the cluster allocation explain API
that will return the cluster info object, `include_disk_info`, the
output looks like:

GET /_cluster/allocation/explain?include_disk_info -d'
{"index": "i", "shard": 0, "primary": false}'

{
  ... other info ...

  "cluster_info" : {
    "nodes" : {
      "7Uws-vL7R6WVm3ZwQA1n5A" : {
        "node_name" : "Kraven the Hunter",
        "least_available" : {
          "path" : "/path/to/data1",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        },
        "most_available" : {
          "path" : "/path/to/data2",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        }
      }
    },
    "shard_sizes" : {
      "[i][2][p]_bytes" : 0,
      "[i][4][p]_bytes" : 130,
      "[i][1][p]_bytes" : 0,
      "[i][3][p]_bytes" : 0,
      "[i][0][p]_bytes" : 130
    },
    "shard_paths" : {
      "[i][3], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=LegZLDniTVaw0Y1urv7s3g]" : "/path/to/data1/nodes/0",
      "[i][1], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=lAU_4vf_SKmoRdtg0ACnjQ]" : "/path/to/data1/nodes/0",
      "[i][2], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Aurpeuj7SeGeyPDDpCtRgg]" : "/path/to/data1/nodes/0",
      "[i][0], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Vgg8GlQTQ82C2j6HYBq8DQ]" : "/path/to/data1/nodes/0",
      "[i][4], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=t8hQlVSxQe-58fSeaXcAqg]" : "/path/to/data1/nodes/0"
    }
  }
}

Resolves #14405
2016-07-12 15:52:20 -06:00
Nik Everett 3ea1360625 Limit batch size when scrolling
Limits the batch size from scrolling using the same setting as interactive
search: `index.max_result_window`.

Closes #19249
2016-07-11 15:29:45 -04:00
Jim Ferenczi 37725f640c Add missing field type in the FieldStats response.
This change adds the type of the field in the fieldstats response.
It can be one of the following:
  * "integer" for byte, short, integer and long
  * "float" for float, half-float and double
  * "date" for date
  * "ip" for ip
  * "text" for string, keyword and text.

Closes #17750
2016-07-06 15:24:09 +02:00
Jason Tedor d0765d0761 Merge branch 'master' into feature/seq_no
* master: (192 commits)
  [TEST] Fix rare OBOE in AbstractBytesReferenceTestCase
  Reindex from remote
  Rename writeThrowable to writeException
  Start transport client round-robin randomly
  Reword Refresh API reference (#19270)
  Update fielddata.asciidoc
  Fix stored_fields message
  Add missing footer notes in mapper size docs
  Remote BucketStreams
  Add doc values support to the _size field in the mapper-size plugin
  Bump version to 5.0.0-alpha5.
  Update refresh.asciidoc
  Update shrink-index.asciidoc
  Change Debian repository for Vagrant debian-8 box
  [TEST] fix test to account for internal empyt reference optimization
  Upgrade to netty 3.10.6.Final (#19235)
  [TEST] fix histogram test when extended bounds overlaps data
  Remove redundant modifier
  Simplify TcpTransport interface by reducing send code to a single send method (#19223)
  Fix style violation in InstallPluginCommand.java
  ...
2016-07-05 22:01:07 -04:00
Christoph Büscher 304e59cb04 Merge branch 'master' into feature/rank-eval 2016-07-05 11:23:21 +02:00
Jim Ferenczi 1924b46dbb Fix typo in rest test 2016-07-04 12:33:44 +02:00
Jim Ferenczi afe99fcdcd Restore reverted change now that alpha4 is out:
Rename `fields` to `stored_fields` and add `docvalue_fields`

`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-07-04 10:39:49 +02:00
Tanguy Leroux c8a9b28d1b Rename Search Template REST tests 2016-07-04 10:07:45 +02:00
Ali Beyad cb20776439 Includes the index UUID in the _cat/indices API and adds tests
for the _cat/indices functionality.

Closes #19204
Closes #19132
2016-07-01 14:45:55 -04:00
Christoph Büscher 3680069628 Merge branch 'master' into feature/rank-eval 2016-07-01 13:59:56 +02:00
javanna 43b82ce244 [TEST] remove feature yaml from REST tests
The only runner that supported it was the java runner, we can use json format instead given that the default one with cat apis is text
2016-07-01 11:13:10 +02:00
javanna 60bafa5d78 [TEST] parse yaml responses too through ObjectPath rather than only json responses
No need to match against yaml responses via regexes in REST tests, yaml responses can be properly parsed via ObjectPath instead. Few REST tests need to be updated accordingly.
2016-07-01 11:13:10 +02:00
jaymode 983a64c833 Add support for `teardown` section in REST tests
This commits adds support for a `teardown` section that can be defined in REST tests to
clean up any items that may have been created by the test and are not cleaned up by
deletion of indices and templates.
2016-06-30 11:33:29 -04:00
Isabel Drost-Fromm 6d4673fd58 Initial commit for Module to compute metrics on queries
This is an initial squashed commit of the work on a new feature for query metrics
proposed in #18798.
2016-06-30 16:15:08 +02:00
Britta Weber 39d38e513d [TEST] add wait for yellow 2016-06-29 13:30:54 +02:00
Honza Král f79851e23a [API] separate tasks.list and tasks.get APIs in the json definition (#19107) 2016-06-28 02:07:46 +02:00
Honza Král 0ec07833a8 [TEST] refactor search yaml tests (#19109) 2016-06-28 01:54:49 +02:00
Jim Ferenczi eb1e231a63 Revert "Rename `fields` to `stored_fields` and add `docvalue_fields`"
This reverts commit 2f46f53dc8.
2016-06-27 17:20:32 +02:00
Alex Benusovich 3ca909dfea Fix NPEs due to disabled source
This commit fixes several NPEs caused by implicitly performing a get request for a document that exists with its _source disabled and then trying to access the source. Instead of causing an NPE the following queries will throw an exception with a "source disabled" message (similar behavior as if the document does not exist).:
- GeoShape query for pre-indexed shape (throws IllegalArgumentException)
- Percolate query for an existing document (throws IllegalArgumentException)

A Terms query with a lookup will ignore the document if the source does not exist (same as if the document does not exist).

GET and HEAD requests for the document _source will return a 404 if the source is disabled (even if the document exists).
2016-06-24 22:03:03 -07:00
Areek Zillur c49ec957eb Merge pull request #19048 from karmi/fix_rollover_integration_test
Fixed incorrect indentation in the YAML integration test for the "Rollover" API
2016-06-23 12:57:16 -04:00
Areek Zillur ca322a12fc remove extra brace from rollover rest api spec 2016-06-23 12:37:04 -04:00
Karel Minarik ae8f885ef9 Fixed incorrect indentation in the YAML integration test for the "Rollover" API 2016-06-23 18:13:52 +02:00
Jason Tedor 112669daed Merge branch 'master' into feature/seq_no
* master: (416 commits)
  docs: removed obsolete information, percolator queries are not longer loaded into jvm heap memory.
  Upgrade JNA to 4.2.2 and remove optionality
  [TEST] Increase timeouts for Rest test client (#19042)
  Update migrate_5_0.asciidoc
  Add ThreadLeakLingering option to Rest client tests
  Add a MultiTermAwareComponent marker interface to analysis factories. #19028
  Attempt at fixing IndexStatsIT.testFilterCacheStats.
  Fix docs build.
  Move templates out of the Search API, into lang-mustache module
  revert - Inline reroute with process of node join/master election (#18938)
  Build valid slices in SearchSourceBuilderTests
  Docs: Convert aggs/misc to CONSOLE
  Docs: migration notes for _timestamp and _ttl
  Group client projects under :client
  [TEST] Add client-test module and make client tests use randomized runner directly
  Move upgrade test to upgrade from version 2.3.3
  Tasks: Add completed to the mapping
  Fail to start if plugin tries broken onModule
  Remove duplicated read byte array methods
  Rename `fields` to `stored_fields` and add `docvalue_fields`
  ...
2016-06-23 11:52:11 -04:00
Tanguy Leroux 04da1bda0d Move templates out of the Search API, into lang-mustache module
This commit moves template support out of the Search API to its own dedicated Search Template API in the lang-mustache module. It provides a new SearchTemplateAction that can be used to render templates before it gets delegated to the usual Search API. The current REST endpoint are identical, but the Render Search Template endpoint now uses the same Search Template API with a new "simulate" option. When this option is enabled, the Search Template API only renders template and returns immediatly, without executing the search.

Closes #17906
2016-06-23 09:30:53 +02:00
Jim Ferenczi 2f46f53dc8 Rename `fields` to `stored_fields` and add `docvalue_fields`
`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-06-22 17:38:30 +02:00
Adrien Grand db9af54ec0 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-22 08:35:54 +02:00
Martijn van Groningen b32d9a71e4 inner_hits: Also never serialize `_index` key for parent/child inner hits as the _index is always the same of the parent search hit 2016-06-21 18:23:40 +02:00
Martijn van Groningen 5ad2fdaa8e inner_hits: Don't include `_id`, `_type` and `_index` keys in search response for inner hits
Closes #18091
2016-06-21 14:13:38 +02:00
Areek Zillur 9356a6090f Merge branch 'master' into enhancement/rollover_api 2016-06-17 11:35:57 -04:00
Jim Ferenczi fb2a48d0f0 Revert "Remove support for sorting terms aggregation by ascending count"
This is delayed after alpha4 since Kibana relies on it.
2016-06-17 17:14:01 +02:00
Areek Zillur 545ffa7801 Merge branch 'master' into enhancement/rollover_api 2016-06-17 10:33:11 -04:00
Jim Ferenczi 755721953b Remove support for sorting terms aggregation by ascending count
closes #17614
2016-06-17 15:06:49 +02:00
Areek Zillur 6adffa6b7b Merge branch 'master' into enhancement/rollover_api 2016-06-16 17:27:32 -04:00
Alexander Kazakov 9eea1b6833 Fix flat_settings REST parameter
* Get XContent params from request in Nodes rest actions
* Adding test for nodes info rest api
2016-06-16 10:03:51 -04:00
Alexander Lin 7d42e7e716 Closes #18013. Added status field to _msearch response bodies. 2016-06-16 00:25:17 -07:00
Areek Zillur 1a59a8418a Merge branch 'master' into enhancement/shrink_request_parser 2016-06-15 15:49:40 -04:00
Nik Everett e392e0b1df Create get task API that falls back to the .tasks index
This adds a get task API that supports GET /_tasks/${taskId} and
removes that responsibility from the list tasks API. The get task
API supports wait_for_complation just as the list tasks API does
but doesn't support any of the list task API's filters. In exchange,
it supports falling back to the .results index when the task isn't
running any more. Like any good GET API it 404s when it doesn't
find the task.

Then we change reindex, update-by-query, and delete-by-query to
persist the task result when wait_for_completion=false. The leads
to the neat behavior that, once you start a reindex with
wait_for_completion=false, you can fetch the result of the task by
using the get task API and see the result when it has finished.

Also rename the .results index to .tasks.
2016-06-14 13:37:34 -04:00
Martijn van Groningen 3b96055b23 msearch: Cap the number of searches the msearch api will concurrently execute
By default the number of searches msearch executes is capped by the number of
nodes multiplied with the default size of the search threadpool. This default can be
overwritten by using the newly added `max_concurrent_searches` parameter.

Before the msearch api would concurrently execute all searches concurrently. If many large
msearch requests would be executed this could lead to some searches being rejected
while other searches in the msearch request would succeed.

The goal of this change is to avoid this exhausting of the search TP.

Closes #17926
2016-06-13 10:13:08 +02:00