Commit Graph

1549 Commits

Author SHA1 Message Date
Lee Hinman 119f8ed9f0 Correctly enable _all for older 5.x indices
When we disabled `_all` by default for indices created in 6.0, we missed adding
a layer that would handle the situation where `_all` was not enabled in 5.x and
then the cluster was updated to 6.0, this means that when the cluster was
updated the `_all` field would be disabled for 5.x indices and field values
would not be added to the `_all` field.

This adds a compatibility layer for 5.x indices where we treat the default
enabled value for the `_all` field to be `true` if unset on 5.x indices.

Resolves #25068
2017-06-08 14:37:44 -06:00
Lee Hinman 050b7cd0f9 Include empty mappings in GET /{index}/_mappings requests (#25118)
Previously this would output:

```
GET /test-1/_mappings

{ }
```

And after this change:

```
GET /test-1/_mappings

{
  "test-1": {
    "mappings": {}
  }
}
```

To bring parity back to the REST output after #24723.

Relates to #25090
2017-06-08 10:57:04 -06:00
Lee Hinman 5b2ab96364 Return index name and empty map for /{index}/_alias with no aliases
Previously in #24723 we changed the `_alias` API to not go through the
`RestGetIndicesAction` endpoint, instead creating a `RestGetAliasesAction` that
did the same thing.

This changes the formatting so that it matches the old formatting of the
endpoint, before:

```
GET /test-1/_alias

{ }
```

And after this change:

```
GET /test-1/_alias

{
  "test-1": {
    "aliases": {}
  }
}
```

This is related to #25090
2017-06-08 10:03:03 -06:00
Jim Ferenczi 36a5cf8f35 Automatically early terminate search query based on index sorting (#24864)
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and 
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.

Relates #6720
2017-06-08 12:10:46 +02:00
Simon Willnauer d57641a747 Skip rest tests that use mutiple types in pure 6.x clusters (#24965)
This change skips rest tests that use mutlitple types if the cluster
is a pure 6.x cluster. This allows all indics to be created with a version
less than 6.0 and that means we can safely use the `mapping.single_type` setting.

Relates to #24961
2017-06-07 15:00:17 +02:00
Jason Tedor 5a0b159cb7 Modify skips for get missing alises tests
Previous work modified the status code on the get aliases API when an
alias is missing so that these requests 404 now. This change was also
backported to 5.5 so we can adjust the skips to skip everything before
5.5.0.
2017-06-06 15:35:30 -04:00
Jason Tedor e03c4938c5 GET aliases should 404 if aliases are missing
Previously the HEAD and GET aliases endpoints were misaigned in
behavior. The HEAD verb would 404 if any aliases are missing while the
GET verb would not if any aliases existed. When HEAD was aligned with
GET, this broke the previous usage of HEAD to serve as an existence
check for aliases. It is the behavior of GET that is problematic here
though, if any alias is missing the request should 404. This commit
addresses this by modifying the behavior of GET to behave in this
way. This fixes the behavior for HEAD to also 404 when aliases are
missing.

Relates #25043
2017-06-06 14:37:29 -04:00
Jim Ferenczi 4a8759ef4c Collapse inner hits rest test should not skip 5.x
Relates https://github.com/elastic/elasticsearch/pull/24517
2017-06-06 09:33:56 +02:00
jaymode e98d5676b3
Test: update missing body tests to run against versions >= 5.5.0
This updates the missing body tests to run against versions >= 5.5.0 after backporting the change
to the 5.x branch.

See #23497
2017-06-05 14:26:07 -06:00
Alex Benusovich 5463294ec4 Fixed NPEs caused by requests without content. (#23497)
REST handlers that require a body will throw an an ElasticsearchParseException "request body required".
REST handlers that require a body OR source param will throw an ElasticsearchParseException "request body or source param required".
Replaced asserts in BulkRequest parsing code with a more descriptive IllegalArgumentException if the line contains an empty object.
Updated bulk REST test to verify an empty action line is rejected properly.
Updated BulkRequestTests with randomized testing for an empty action line.
Used try-with-resouces for XContentParser in AbstractBulkByQueryRestHandler.
2017-06-05 09:08:14 -06:00
Lee Hinman 134b0d594e [TEST] Skip wildcard expansion test due to breaking change
Relates to #24723
2017-06-02 20:48:52 -06:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Ryan Ernst 8d88b94372 Scripting: Add optional context parameter to put stored script requests (#25014)
This commit adds an optional `context` url parameter to the put stored
script request. When a context is specified, the script is compiled
against that context before storing, as a validation the script will
work when used in that context.
2017-06-01 17:53:48 -07:00
Christoph Büscher 3d6fb4eb0b Merge branch 'master' into feature/rank-eval 2017-05-30 14:24:26 +02:00
Jim Ferenczi 47cf7825dd Move BWC version to 5.5 after backport
Relates to #24517
2017-05-26 14:57:07 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
markharwood a64937db7a Test fix - rest test missing version skip for new 6.0 significant_text agg 2017-05-24 17:05:02 +01:00
markharwood b7197f5e21 SignificantText aggregation - like significant_terms, but for text (#24432)
* SignificantText aggregation - like significant_terms but doesn’t require fielddata=true, recommended used with `sampler` agg to limit expense of tokenizing docs and takes optional `filter_duplicate_text`:true setting to avoid stats skew from repeated sections of text in search results.

Closes #23674
2017-05-24 13:46:43 +01:00
Christoph Büscher cd0941810f Merge branch 'master' into feature/rank-eval 2017-05-18 16:47:47 +02:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Jim Ferenczi 67c41d2e77 Fix ExpandSearchPhase when response contains no hits (#24688)
This change skips the expand search phase entirely when there is no search hits in the response.
2017-05-17 14:15:40 +02:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Jim Ferenczi 7b7e15023a Add rest test for sliced scroll (#24630) 2017-05-12 00:07:24 +02:00
qwerty4030 e7d352b489 Compound order for histogram aggregations. (#22343)
This commit adds support for histogram and date_histogram agg compound order by refactoring and reusing terms agg order code. The major change is that the Terms.Order and Histogram.Order classes have been replaced/refactored into a new class BucketOrder. This is a breaking change for the Java Transport API. For backward compatibility with previous ES versions the (date)histogram compound order will use the first order. Also the _term and _time aggregation order keys have been deprecated; replaced by _key.

Relates to #20003: now that all these aggregations use the same order code, it should be easier to move validation to parse time (as a follow up PR).

Relates to #14771: histogram and date_histogram aggregation order will now be validated at reduce time.

Closes #23613: if a single BucketOrder that is not a tie-breaker is added with the Java Transport API, it will be converted into a CompoundOrder with a tie-breaker.
2017-05-11 18:06:26 +01:00
Ali Beyad 743217a430 Enhances get snapshots API to allow retrieving repository index only (#24477)
Currently, the get snapshots API (e.g. /_snapshot/{repositoryName}/_all)
provides information about snapshots in the repository, including the
snapshot state, number of shards snapshotted, failures, etc.  In order
to provide information about each snapshot in the repository, the call
must read the snapshot metadata blob (`snap-{snapshot_uuid}.dat`) for
every snapshot.  In cloud-based repositories, this can be expensive,
both from a cost and performance perspective.  Sometimes, all the user
wants is to retrieve all the names/uuids of each snapshot, and the
indices that went into each snapshot, without any of the other status
information about the snapshot.  This minimal information can be
retrieved from the repository index blob (`index-N`) without needing to
read each snapshot metadata blob.

This commit enhances the get snapshots API with an optional `verbose`
parameter.  If `verbose` is set to false on the request, then the get
snapshots API will only retrieve the minimal information about each
snapshot (the name, uuid, and indices in the snapshot), and only read
this information from the repository index blob, thereby giving users
the option to retrieve the snapshots in a repository in a more
cost-effective and efficient manner.

Closes #24288
2017-05-10 15:48:40 -04:00
Isabel Drost-Fromm bd559d96d4
This adds max_concurrent_searches to multi-search-template endpoint.
Closes #20912
2017-05-10 11:23:24 +02:00
Adrien Grand a72eaa8e0f Identify documents by their `_id`. (#24460)
Now that indices have a single type by default, we can move to the next step
and identify documents using their `_id` rather than the `_uid`.

One notable change in this commit is that I made deletions implicitly create
types. This helps with the live version map in the case that documents are
deleted before the first type is introduced. Otherwise there would be no way
to differenciate `DELETE index/foo/1` followed by `PUT index/foo/1` from
`DELETE index/bar/1` followed by `PUT index/foo/1`, even though those are
different if versioning is involved.
2017-05-09 16:33:52 +02:00
Jim Ferenczi 4df636b5ef Fix single shard scroll within a cluster with nodes in version >= 5.3 and <= 5.3 (#24512)
If a node in version >= 5.3 acts as a coordinating node during a scroll request that targets a single shard, the scroll may return the same documents over and over iff the targeted shard is hosted by a node with a version <= 5.3.
The nodes in this version will advance the scroll only if the search_type has been set to `query_and_fetch` though this search type has been removed in 5.3.
This change handles this situation by adding the removed search_type in the request that targets a node in version <= 5.3.
2017-05-09 09:14:17 +02:00
Simon Willnauer 5bfb98ade4 [TEST] Reenable disabled tests for _field_caps and _search_shards (#24505) 2017-05-05 16:02:26 +02:00
Simon Willnauer 8055b14f2e Temporarily disable tests 2017-05-05 12:08:00 +02:00
Simon Willnauer 03267e03da Fix NPE if field caps request has a field that exists not in all indices (#24504)
If a field caps request contains a field name that doesn't exist in all indices
the response will be partial and we hide an NPE. The NPE is now fixed but we still
have the problem that we don't pass on errors on the shard level to the user. This will
be fixed in a followup.
2017-05-05 11:56:03 +02:00
Simon Willnauer 6b67e0bf2f Include all aliases including non-filtering in `_search_shards` response (#24489)
`_search_shards`API today only returns aliases names if there is an alias
filter associated with one of them. Now it can be useful to see which aliases
have been expanded for an index given the index expressions. This change also includes non-filtering aliases even without a filtering alias being present.
2017-05-05 09:34:12 +02:00
Simon Willnauer 07f106d39c [TEST] Rollback temporarily disabled field_caps test (#24483) 2017-05-04 14:14:22 +02:00
Simon Willnauer 14e57bf9f8 Add cross cluster support to `_field_caps` (#24463)
To support kibana this commit adds an internal optimization
to support the cross cluster syntax for indices on the `_field_caps`
API.

Closes #24334
2017-05-04 11:44:54 +02:00
Jim Ferenczi 6fcd24d264 Check index sorting with no replica since we cannot ensure that the replica index is ready when forceMerge is called. Closes #24416 2017-05-02 20:25:09 +02:00
Luca Cavanna 91fbb0ba28 Move IndicesAliasesRequest#concreteAliases to TransportIndicesAliasesAction (#24400)
This method has to do with how the transport action may or may not resolve wildcards expressions to aliases names. It is only needed in TransportIndicesAliasesAction and for this reason it should be a private method in it rather than part of a request class which is also part of the Java API and later in the high level REST client.
2017-05-01 19:59:06 +02:00
javanna 7863407b46 [TEST] fix _cat/allocation index size check
The check expected the size of the index to always be returned in bytes, but that can possibly be kb, mb, gb and tb depending on the actual size.
2017-05-01 14:38:14 +02:00
Guillaume Le Floch 382a617d34 Handle multiple aliases in _cat/aliases api (#23698)
The alias parameter was documented as a list in our rest-spec, yet only the first value out of a list was getting read and processed. This commit adds support for multiple aliases to _cat/aliases

Closes #23661
2017-04-28 15:21:44 +02:00
Adrien Grand 1be2800120 Only allow one type on 7.0 indices (#24317)
This adds the `index.mapping.single_type` setting, which enforces that indices
have at most one type when it is true. The default value is true for 6.0+ indices
and false for old indices.

Relates #15613
2017-04-27 08:43:20 +02:00
Guillaume Le Floch 739cb35d1b Allow passing single scrollID in clear scroll API body (#24242)
* Allow single scrollId in string format

Closes #24233
2017-04-25 13:43:21 +02:00
Christoph Büscher 5254731039 Merge branch 'master' into feature/rank-eval 2017-04-22 21:47:32 +02:00
Nik Everett caf376c8af Start building analysis-common module (#23614)
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Jim Ferenczi f05af0a382 Enable index-time sorting (#24055)
This change adds an index setting to define how the documents should be sorted inside each Segment.
It allows any numeric, date, boolean or keyword field inside a mapping to be used to sort the index on disk.
It is not allowed to use a `nested` fields inside an index that defines an index sorting since `nested` fields relies on the original sort of the index.
This change does not add early termination capabilities in the search layer. This will be added in a follow up.

Relates #6720
2017-04-19 14:36:11 +02:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates #24118
2017-04-15 10:36:03 -04:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Simon Willnauer 42e0b4f5e9 [TEST] Enable new REST test for 5.4 and BWC against 5.4.x 2017-04-11 13:30:45 +02:00
Simon Willnauer f22e0dc30b Add cross-cluster search remote cluster info API (#23969)
This commit adds an API to discover information like seed nodes,
http addresses and connection status of a configured remote cluster.

Closes #23925
2017-04-11 09:24:40 +02:00
Jim Ferenczi af49c46b76 Fix BWC tests for field_stats now that the deprecation has been back ported to 5.4 2017-04-10 12:40:37 +02:00
Jim Ferenczi 9b3c85dd88 Deprecate _field_stats endpoint (#23914)
_field_stats has evolved quite a lot to become a multi purpose API capable of retrieving the field capabilities and the min/max value for a field.
In the mean time a more focused API called `_field_caps` has been added, this enpoint is a good replacement for _field_stats since he can
retrieve the field capabilities by just looking at the field mapping (no lookup in the index structures).
Also the recent improvement made to range queries makes the _field_stats API obsolete since this queries are now rewritten per shard based on the min/max found for the field.
This means that a range query that does not match any document in a shard can return quickly and can be cached efficiently.
For these reasons this change deprecates _field_stats. The deprecation should happen in 5.4 but we won't remove this API in 6.x yet which is why
 this PR is made directly to 6.0.
 The rest tests have also been adapted to not throw an error while this change is backported to 5.4.
2017-04-10 10:10:16 +02:00
jaymode 53e3ddf2f0
Test: remove test that will never run on master
This test was added in #23950 for backporting and review, but it is always skipped on master so
this commit deletes it.
2017-04-06 15:50:08 -04:00
Jay Modi 495bf21b46 Preserve response headers when creating an index (#23950)
This commit preserves the response headers when creating an index and updating settings for an
index.

Closes #23947
2017-04-06 20:38:09 +01:00
Christoph Büscher 024ed1b6ca Merge branch 'master' into feature/rank-eval 2017-04-04 18:23:41 +02:00
Clinton Gormley 01b807f98e Adapted search_shards rest test to work with Perl
Because of the way Perl treats numbers, the boost is represented
as 1 instead of 1.0, which caused this test to fail.
2017-04-02 12:52:34 +02:00
Clinton Gormley 5b3c662145 To examine an exception in rest tests, the exception should be caught, not ignored 2017-04-02 12:52:30 +02:00
Clinton Gormley e5f99e0bde Fixed bad YAML in rest tests 2017-04-02 12:52:23 +02:00
Jim Ferenczi a8250b26e7 Add FieldCapabilities (_field_caps) API (#23007)
This change introduces a new API called `_field_caps` that allows to retrieve the capabilities of specific fields.

Example:

````
GET t,s,v,w/_field_caps?fields=field1,field2
````
... returns:
````
{
   "fields": {
      "field1": {
         "string": {
            "searchable": true,
            "aggregatable": true
         }
      },
      "field2": {
         "keyword": {
            "searchable": false,
            "aggregatable": true,
            "non_searchable_indices": ["t"]
            "indices": ["t", "s"]
         },
         "long": {
            "searchable": true,
            "aggregatable": false,
            "non_aggregatable_indices": ["v"]
            "indices": ["v", "w"]
         }
      }
   }
}
````

In this example `field1` have the same type `text` across the requested indices `t`, `s`, `v`, `w`.
Conversely `field2` is defined with two conflicting types `keyword` and `long`.
Note that `_field_caps` does not treat this case as an error but rather return the list of unique types seen for this field.
2017-03-31 15:34:46 +02:00
Nik Everett b3dca364ce Revert "Skip 5.4 bwc test for new name for now"
This reverts commit 490d29f4fc.
We should be able to run the 5.4 tests now that we have
8c53555b28.
2017-03-28 09:06:00 -04:00
Jason Tedor 742d929b56 Validate top-level keys when parsing mget requests
Today, when parsing mget requests, we silently ignore keys in the top
level that do not match "docs" or "ids". This commit addresses this
situation by throwing an exception if any other key occurs here, and
providing the names of valid keys.

Relates #23746
2017-03-28 08:27:31 -04:00
Christoph Büscher 96fc3aaf6f Merge branch 'master' into feature/rank-eval 2017-03-23 19:55:47 +01:00
Igor Motov f927a2708d Make it possible to validate a query on all shards instead of a single random shard (#23697)
This is especially useful when we rewrite the query because the result of the rewrite can be very different on different shards. See #18254 for example.
2017-03-22 17:39:21 -04:00
Nik Everett 490d29f4fc Skip 5.4 bwc test for new name for now
We have to wait for a clean snapshot build.
2017-03-22 15:19:02 -04:00
Jason Tedor 230c5b1ccf Count through the primary in list of strings test
A previous attempt to address a race condition in this test set wait for
active shards to all. However, there might not be any replicas if the
test is only running with one node so we end up waiting
forever. Instead, to address the intial race condition, we just count
through the primary.
2017-03-22 11:00:14 -04:00
Nik Everett 2e3ea918dc Skip testing new name if it isn't known
In #23638 we renamed `request_cache` to `request` in the
`_cache/clear` API. But it is only going to be committed back to
5.x so we can't test with the new name in a mixed version
cluster.
2017-03-22 10:56:03 -04:00
Jason Tedor 9646aca893 Wait for all shards in list of strings test
This test executes a bulk indexing operation with two documents. If this
test is running against multiple nodes, there are no guarantees that all
shards are green before we execute a search operation which might hit a
replica shard. This commit creates the index in advance, and waits for
all shards to be active before proceeding with the indexing request.
2017-03-22 09:35:12 -04:00
Stefan Gorgiovski 798c19dd7f Deprecate request_cache for clear-cache (#23638)
It is called `request` now.
2017-03-22 08:28:04 -04:00
Honza Král f8b7ec27a4 [API] change wait_for_completion defaults according to docs (#23672) 2017-03-22 12:53:47 +01:00
Jason Tedor f56900da9f Fix took assertion in response filter test
This test asserts that the took time exists by using the is_true
assertion. This assertion fails if the took time was zero as is_true
asserts that the field is not the empty string, not the string "false",
and not 0. If the search returns quickly, and took time is measured
using a high-precision monotonic clock, the took time can be zero. This
commit changes the assertion to use gte.
2017-03-20 21:11:20 -04:00
Christoph Büscher cf35545e2d Merge branch 'master' into feature/rank-eval 2017-03-13 17:36:13 -07:00
Karel Minarik 4ea6fc0059 [TEST] Fixed the "Msearch" typed keys YAML test
* Added the YAML document separator to the beginning of the file
* Fixed the incorrect JSON syntax in the query

Closes #23500
2017-03-08 08:38:12 -08:00
Ali Beyad 577d2a6a1d Adds cluster state size to /_cluster/state response (#23440)
This commit adds the size of the cluster state to the response for the
get cluster state API call (GET /_cluster/state).  The size that is
returned is the size of the full cluster state in bytes when compressed.
This is the same size of the full cluster state when serialized to
transmit over the network.  Specifying the ?human flag displays the
compressed size in a more human friendly manner.  Note that even if the
cluster state request filters items from the cluster state (so a subset
of the cluster state is returned), the size that is returned is the
compressed size of the entire cluster state.

Closes #3415
2017-03-02 14:20:29 -05:00
javanna 261f31f5b7 [TEST] move filters aggs wrapper query builder rewriting test to integ tests
This test makes little sense when sent from the REST layer, as WrapperQueryBuilder is supposed to be used from the Java api. Also, providing the inner query as base64 string will work only for string formats and break for binary formats like SMILE and CBOR, whcih doesn't play well with randomizing content type in our REST tests
2017-02-27 12:27:03 +01:00
Christoph Büscher 1f4c4d99b9 Merge branch 'master' into feature/rank-eval 2017-02-27 11:25:17 +01:00
Christoph Büscher 641c88dc29 Prevent negative `from` parameter in SearchSourceBuilder (#23358)
This prevents later errors like the one reported in #23324 and throws an
IllegalArgumentException early instead.
2017-02-27 09:45:10 +01:00
Ali Beyad 550f32354c [TEST] Removes timeout based wait_for_active_shards REST test (#23360)
This commit removes an necessary test that ensures if
wait_for_active_shards cannot be fulfilled on index creation, that the
response returns shardsAcknowledged=false.  However, this is already
tested in WaitForActiveShardsIT and it would improve the speed of the
test runs to get rid of any unnecessary tests, especially those that
depend on timeouts.
2017-02-24 14:55:42 -05:00
Ali Beyad 93a359e434 [TEST] increase timeout slightly in wait_for_active_shards test to allow
for index creation cluster state update to be processed before ensuring
the wait times out
2017-02-24 12:51:45 -05:00
Jim Ferenczi 57b5d1d29b disable BWC tests for the highlighters, need a new 5.x build to make it work 2017-02-24 08:50:39 +01:00
Jason Tedor e579629b16 Align REST specs for HEAD requests
Previous changes aligned HEAD requests to be consistent with GET
requests to the same endpoint. This commit aligns the REST spec for the
impacted endpoints.

Relates #23313
2017-02-23 08:55:13 -05:00
Luca Cavanna 495b24655b Update indices settings api to support CBOR and SMILE format (#23309)
Also expand testing on the different ways to provide index settings and remove dead code around ability to provide settings as query string parameters

Closes #23242
2017-02-22 17:51:10 +01:00
Simon Willnauer 5c1924ad19 Remove BWC layer for number of reduce phases (#23303)
Both PRs below have been backported to 5.4 such that we can enable
BWC tests of this feature as well as remove version dependend serialization
for search request / responses.

Relates to #23288
Relates to #23253
2017-02-22 15:03:09 +01:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Jason Tedor 2925a81cc9 Fix REST spec for exists
A previous change aligned the handling of the GET document and HEAD
document APIs. This commit aligns the specification for these two APIs
as well, and fixes a failing test.

Relates #23196
2017-02-16 08:56:38 -05:00
Christoph Büscher cfa52f8b9a Merge branch 'master' into feature/rank-eval 2017-02-16 10:39:07 +01:00
Jim Ferenczi 9316e8e8fe Restore BWC tests for field collapsing 2017-02-15 15:04:38 +01:00
Yannick Welsch 1aefbf57e1 Fix tests that check for deprecation message 2017-02-15 09:35:02 +01:00
Clinton Gormley 1556e81e9a Normalising REST spec query param types
integer -> number
filter_path: string -> list
2017-02-14 16:59:34 +01:00
Tanguy Leroux 072748cd67 [Tests] Fix msearch/typed_keys test
This test uses index_patterns which has been introduced in 6.0 and does not exist in 5.4.0, making the Bwc test fails. Instead of using index templates, it now uses explicitly create the required indices. Also, it fixes unmapped aggregations tests.
2017-02-13 13:39:11 +01:00
Jim Ferenczi 5808aea265 Temporary disable bwc tests on field collapsing that fail because snapshot build of 5.3 is broken 2017-02-10 15:42:58 +01:00
Tanguy Leroux e2e5937455 Use `typed_keys` parameter to prefix suggester names by type in search responses (#23080)
This pull request reuses the typed_keys parameter added in #22965, but this time it applies it to suggesters. When set to true, the suggester names in the search response will be prefixed with a prefix that reflects their type.
2017-02-10 10:53:38 +01:00
Jim Ferenczi 94087b3274 Removes ExpandCollapseSearchResponseListener, search response listeners and blocking calls
This changes removes the SearchResponseListener that was used by the ExpandCollapseSearchResponseListener to expand collapsed hits.
The removal of SearchResponseListener is not a breaking change because it was never released.
This change also replace the blocking call in ExpandCollapseSearchResponseListener by a single asynchronous multi search request. The parallelism of the expand request can be set via CollapseBuilder#max_concurrent_group_searches

Closes #23048
2017-02-09 18:06:10 +01:00
Tanguy Leroux 3553522328 Add parameter to prefix aggs name with type in search responses (#22965)
This pull request adds a new parameter to the REST Search API named `typed_keys`. When set to true, the aggregation names in the search response will be prefixed with a prefix that reflects the internal type of the aggregation.

Here is a simple example:
```
GET /_search?typed_keys
{
    "aggs": {
        "tweets_per_user": {
            "terms": {
                "field": "user"
            }
        }
    },
    "size": 0
}
```

And the response:

```
{
    "aggs": {
        "sterms:tweets_per_user": {
            ...
        }
    }
}
```

This parameter is intended to make life easier for REST clients that could parse back the prefix and could detect the type of the aggregation to parse. It could also be implemented for suggesters.
2017-02-09 11:19:04 +01:00
Christoph Büscher 4cb8d9d08c Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
        docs/reference/search.asciidoc
2017-02-03 17:27:20 +01:00
Ali Beyad 32707fa1ef [TEST] fix introduced typo 2017-02-02 12:18:20 -05:00
Ali Beyad 43aadef23a [TEST] upgrade backward compatibility version of rolling upgrade
tests to 5.3.0-SNAPSHOT
2017-02-02 09:56:38 -05:00
Ali Beyad bf439b9cc8 [TEST] temporarily disable the explain API rest test until
the 5.3.0-snapshot distribution is properly generated
2017-02-01 14:14:39 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Jim Ferenczi f6d38d480a Integrate UnifiedHighlighter (#21621)
* Integrate UnifiedHighlighter

This change integrates the Lucene highlighter called "unified" in the list of supported highlighters for ES.
This highlighter can extract offsets from either postings, term vectors, or via re-analyzing text.
The best strategy is picked automatically at query time and depends on the field and the query to highlight.
2017-01-31 19:06:03 +01:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Jim Ferenczi 868b12b548 Add BWC tests for field collapsing
Field collapsing is supported from version 5.3
2017-01-24 08:34:16 +01:00
Jim Ferenczi e48bc2eed7 Add field collapsing for search request (#22337)
* Add top hits collapsing to search request

The field collapsing is done with a custom top docs collector that "collapse" search hits with same field value.
The distributed aspect is resolve using the two passes that the regular search uses. The first pass "collapse" the top hits, then the coordinating node merge/collapse the top hits from each shard.

```
GET _search
{
   "collapse": {
      "field": "category",
   }
}
```

This change also adds an ExpandCollapseSearchResponseListener that intercepts the search response and expands collapsed hits using the CollapseBuilder#innerHit} options.
The retrieval of each inner_hits is done by sending a query to all shards filtered by the collapse key.

```
GET _search
{
   "collapse": {
      "field": "category",
      "inner_hits": {
	"size": 2
      }
   }
}
```
2017-01-23 16:33:51 +01:00
markharwood f01784205f New AdjacencyMatrix aggregation
Similar to the Filters aggregation but only supports "keyed" filter buckets and automatically "ANDs" pairs of filters to produce a form of adjacency matrix.
The intersection of buckets "A" and "B" is named "A&B" (the choice of separator is configurable). Empty intersection buckets are removed from the final results.

Closes #22169
2017-01-20 15:49:31 +00:00
Boaz Leskes 5d806bf93e Index creation and setting update may not return deprecation logging (#22702)
Those services validate their setting before submitting an AckedClusterStateUpdateTask to the cluster state service. An acked cluster state may be completed by a networking thread when the last acks as received. As such it needs special care to make sure that thread context headers are handled correctly.
2017-01-20 10:14:13 +01:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Nicholas Knize 84e4f91253 Add geo_point to FieldStats
This commit adds a new GeoPoint class to FieldStats for computing field stats over geo_point field types.
2017-01-18 14:37:03 -06:00
Boaz Leskes 1227044ddd Add a deprecation notice to shadow replicas (#22647)
Relates to #22024

On top of documentation, the PR adds deprecation loggers and deals with the resulting warning headers.

The yaml test is set exclude versions up to 6.0. This is need to make sure bwc tests pass until this is backported to 5.2.0 . Once that's done, I will change the yaml test version limits
2017-01-18 12:28:09 +01:00
Greg Marzouka e0f8d88d5c Include global query string parameters in the REST spec
Closes #11638
2017-01-17 07:35:14 -05:00
Lee Hinman 2db01b6127 Merge remote-tracking branch 'dakrone/disable-all-by-default' 2017-01-12 10:17:51 -07:00
Luca Cavanna 7674de9e1f Move human flag under always accepted query_string params (#22562)
There are some parameters that are accepted by each and every api we expose. Those (pretty, source, error_trace and filter_path)  are not explicitly listed in the spec of every api, rather whitelisted in clients test runners so that they are always accepted. The `human` flag has been treated up until now as a parameter that's accepted by only some stats and info api, but that doesn't reflect reality as es core treats it exactly like `pretty` (relevant especially now that we validate params and throw exception when we find one that is not supported). Furthermore, the human flag has effect on every api that outputs a date, time, percentage or byte size field. For instance the tasks api outputs a date field although they don't have the human flag explicitly listed in their spec. There are other similar cases. This commit removes the human flag from the rest spec and makes it an always accepted query_string param.
2017-01-12 10:04:45 +01:00
Lee Hinman 7a18bb50fc Disable _all by default
This change disables the _all meta field by default.

Now that we have the "all-fields" method of query execution, we can save both
indexing time and disk space by disabling it.

_all can no longer be configured for indices created after 6.0.

Relates to #20925 and #21341
Resolves #19784
2017-01-11 16:47:13 -07:00
Jim Ferenczi 433c822d4f Promote longs to doubles when a terms agg mixes decimal and non-decimal numbers (#22449)
* Promote longs to doubles when a terms agg mixes decimal and non-decimal number

This change makes the terms aggregation work when the buckets coming from different indices are a mix of decimal numbers and non-decimal numbers. In this case non-decimal number (longs) are promoted to decimal (double) which can result in a loss of precision for big numbers.

Fixes #22232
2017-01-10 11:50:56 +01:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
Karel Minarik 4f4b76cd41 [TEST] Fixed the incorrect indentation for the `skip` clauses in the REST tests
This patch fixes the incorrect indentation in the REST tests, which makes tests in language runners (eg. Ruby, Python) to fail, since the skip clause is parsed as an empty value. Tha Java YAML parser is smarter/lenient about whitespace, so it doesn't catch this.
2017-01-08 14:21:02 +01:00
Nik Everett 12923ef896 Close and flush refresh listeners on shard close
Right now closing a shard looks like it strands refresh listeners,
causing tests like
`delete/50_refresh/refresh=wait_for waits until changes are visible in search`
to fail. Here is a build that fails:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+multi_cluster_search+multijob-darwin-compatibility/4/console

This attempts to fix the problem by implements `Closeable` on
`RefreshListeners` and rejecting listeners when closed. More importantly
the act of closing the instance flushes all pending listeners
so we shouldn't have any stranded listeners on close.

Because it was needed for testing, this also adds the number of
pending listeners to the `CommonStats` object and all API to which
that flows: `_cat/nodes`, `_cat/indices`, `_cat/shards`, and
`_nodes/stats`.
2017-01-06 20:03:32 -05:00
Ali Beyad a487b90498 [TEST] fix explain API rest test that assumes there is only a single
node in the cluster (incorrect assumption)
2017-01-06 11:07:40 -05:00
Ali Beyad 2f510b38c3 [TEST] explain API rest test may have shard allocation throttled 2017-01-04 14:34:00 -05:00
Ali Beyad 85b754f0e0 [TEST] 5.x snapshot build is working again, so update the backwards
compatibility tests for the allocation explain API to include 5.2.0
2017-01-04 12:07:17 -05:00
Jim Ferenczi 1f35d2532b Fix BWC layer with field_stats and geo_point 2017-01-04 13:14:09 +01:00
Jim Ferenczi 360ce532eb Implement stats for geo_point and geo_shape field (#22391)
Currently `geo_point` and `geo_shape` field are treated as `text` field by the field stats API and we
try to extract the min/max values with MultiFields.getTerms.
This is ok in master because a `geo_point` field is always a Point field but it can cause problem in 5.x (and 2.x) because the legacy
 `geo_point` are indexed as terms.
 As a result the min and max are extracted and then printed in the FieldStats output using BytesRef.utf8ToString
 which can throw an IndexOutOfBoundException since it's not valid UTF8 strings.
 This change ensure that we never try to extract min/max information from a `geo_point` field.
 It does not add a new type for geo points in the fieldstats API so we'll continue to use `text` for this kind of field.
 This PR is targeted to master even though we could only commit this change to 5.x. I think it's cleaner to have it in master too before we make any decision on
  https://github.com/elastic/elasticsearch/pull/21947.

Fixes #22384
2017-01-04 10:42:22 +01:00
Ali Beyad 91917d6e91 [TEST] mute backwards compatability tests for explain API until 5.2
snapshot builds can be published again
2017-01-02 18:26:19 -05:00
Ali Beyad 20ab4be59f Cluster Explain API uses the allocation process to explain shard allocation decisions (#22182)
This PR completes the refactoring of the cluster allocation explain API and improves it in the following two high-level ways:

 1. The explain API now uses the same allocators that the AllocationService uses to make shard allocation decisions. Prior to this PR, the explain API would run the deciders against each node for the shard in question, but this was not executed on the same code path as the allocators, and many of the scenarios in shard allocation were not captured due to not executing through the same code paths as the allocators.

 2. The APIs have changed, both on the Java and JSON level, to accurately capture the decisions made by the system. The APIs also now report on shard moving and rebalancing decisions, whereas the previous API did not report decisions for moving shards which cannot remain on their current node or rebalancing shards to form a more balanced cluster.

Note: this change affects plugin developers who may have a custom implementation of the ShardsAllocator interface. The method weighShards has been removed and no longer has any utility. In order to support the new explain API, however, a custom implementation of ShardsAllocator must now implement ShardAllocationDecision decideShardAllocation(ShardRouting shard, RoutingAllocation allocation) which provides a decision and explanation for allocating a single shard. For implementations that do not support explaining a single shard allocation via the cluster allocation explain API, this method can simply return an UnsupportedOperationException.
2017-01-02 12:28:32 -06:00
Jim Ferenczi 02d4cbfeea Fix bwc integ test that tries to perform a term aggs on a scaled_float. This is broken when a node with version prior to 5.2.0 is used with another node > 5.2.0. This is because scaled_float fields are considered as longs in version < 5.2.0. This is fixed in 5.2.0 where scaled_float are recognized as doubles. 2016-12-27 21:52:27 +01:00
Jim Ferenczi e7444f7d77 Fix scaled_float numeric type in aggregations (#22351)
`scaled_float` should be used as DOUBLE in aggregations but currently they are used as LONG.
This change fixes this issue and adds a simple it test for it.

Fixes #22350
2016-12-27 09:23:22 +01:00
Ali Beyad 8261bd358a Synchronize snapshot deletions on the cluster state (#22313)
Before, snapshot/restore would synchronize all operations on the cluster
state except for deleting snapshots.  This meant that only one
snapshot/restore operation would be allowed in the cluster at any given
time, except for deletions - there could be two or more snapshot
deletions running at the same time, or a deletion could be running,
unbeknowest to the rest of the cluster, and thus a snapshot or restore
would be allowed at the same time as the snapshot deletion was still in
progress.  This could cause any number of synchronization issues,
including the situation where a snapshot that was deleted could reappear
in the index-N file, even though its data was no longer present in the
repository.

This commit introduces a new custom type to the cluster state to
represent deletions in progress.  Now, another deletion cannot start if
a deletion is currently in progress.  Similarily, a snapshot or restore
cannot be started if a deletion is currently in progress.  In each case,
if attempting to run another snapshot/restore operation while a deletion
is in progress, a ConcurrentSnapshotExecutionException will be thrown.
This is the same exception thrown if trying to snapshot while another
snapshot is in progress, or restore while a snapshot is in progress.

Closes #19957
2016-12-25 19:00:20 -05:00
Adrien Grand 70594a66c7 Only run the unmapped+missing tests on 5.2+. 2016-12-23 09:38:20 +01:00
Adrien Grand e39942fc02 `value_type` is useful regardless of scripting. (#22160)
Today we only expose `value_type` in scriptable aggregations, however it is
also useful with unmapped fields. I suspect we never noticed because
`value_type` was not documented (fixed) and most aggregations are scriptable.

Closes #20163
2016-12-22 14:35:12 +01:00
Boaz Leskes 6249f1092f x_refresh.yaml tests should use unique index names and doc ids to ease debugging
This is to make it easier to grep the node logs
2016-12-21 10:25:33 +01:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
Isabel Drost-Fromm b1e0d698ac Merge branch 'master' into feature/rank-eval 2016-12-19 10:16:16 +01:00
Areek Zillur d44de0cecc Remove deprecated _suggest endpoint (#22203)
In #20305, _suggest endpoint was deprecated
in favour of using _search endpoint. This
commit removes the dedicated _suggest endpoint
entirely from master.
2016-12-16 12:06:02 -05:00
Masaru Hasegawa a0185c83a7 Merge pull request #21393 from masaruh/alias_boost
Resolve index names in indices_boost
2016-12-16 15:07:51 +09:00
Jason Tedor 43f71015a8 Add skip for include segment file sizes REST tests
This commit adds a skip for the include segment file sizes REST tests on
nodes less than or equal to version 5.1.1 as the stats APIs did not
correctly account for this parameter prior to version 5.1.2.

Relates #21879
2016-12-15 21:08:51 -05:00
Aaron Spiegel 80d3d790ae Fix handling of segment file sizes in stats API
This commit addresses an issue in the stats APIs where
include_segment_file_sizes was not being consumed leading to requests
containing this parameter being rejected.

Relates #21879
2016-12-15 07:29:11 -05:00
Isabel Drost-Fromm 5618d6ca49 Merge branch 'master' into feature/rank-eval 2016-12-15 10:29:26 +01:00
Areek Zillur cdd5fbe3a1 Deprecate _suggest endpoint in favour of _search (#20305)
* Replace _suggest endpoint to _search in docs

In 5.0, the _suggest endpoint is just sugar for _search
with suggestions specified. Users should move away from
using the _suggest endpoint, as it is marked as deprecated in 5.x and
will be removed in 6.0

* update docs to use _search endpoint instead of _suggest

* Add deprecation logging to RestSuggestAction

* Use search endpoint instead of suggest endpoint in rest tests
2016-12-14 21:49:53 -05:00
Isabel Drost-Fromm ea9f11965d Merge branch 'master' into feature/rank-eval 2016-12-14 12:47:12 +01:00
Simon Willnauer b7bcb5bb3a [TEST] Skip term / int partitioning tests in bwc tests pre 5.2.0 2016-12-13 22:20:44 +01:00
markharwood 4c6d17a176 Added tests for toXContent and fromXContent for IncludeExclude class.
New REST test revealed an issue with inconsistent hashing in partitioned
term tests which is also fixed in this change.

Closes #22102
2016-12-13 15:23:09 +00:00
Isabel Drost-Fromm 58342d4c9a Add checks to RankEvalSpec to safe guard against missing parameters. (#22026)
Add checks to RankEvalSpec to safe guard against missing parameters.

Fail early in case no metric is supplied, no rated requests are supplied or the search source builder is missing but no template is supplied neither.

Add stricter checks around rank eval request parsing: Fail if in a rated request we see both, a verbatim request as well as request
template parameters.


Relates to #21260
2016-12-13 11:21:57 +01:00
Isabel Drost-Fromm b2e8455745 Merge branch 'master' into feature/rank-eval 2016-12-13 10:45:22 +01:00
Artur Nowosielski 726f5dccc0 Rewrite filter queries in FiltersAggregationBuilder (#22076)
Queries must be rewritten before the query phase executes otherwise non-executable queries like `wrapper` query or `terms`  will fail or queries that require resources like script service can't access these service unless rewritten.

Relates to #21303
2016-12-11 14:37:12 +01:00
Masaru Hasegawa 3df2a086d4 Resolve index names in indices_boost
This change allows specifying alias/wildcard expression in indices_boost.
And added another format for specifying indices_boost. It accepts array of index name and boost pair.
If an index is included in multiple aliases/wildcard expressions, the first match will be used.
With new format, old format is marked as deprecated.

Closes #4756
2016-12-11 21:41:49 +09:00
Nik Everett 7a74a41a0c Fix test for changed message
Message is only changed in 5.2.0 so we shouldn't assert on it
if we're running with any nodes less than that version.
2016-12-10 10:35:14 -05:00
Nik Everett ddade1b5ac Improve the error message if task and node isn't found (#22062)
Improves the error message returned when looking up a task that
belongs to a node that is no longer part of the cluster. The new
error message tells the user that the node isn't part of the cluster.
This is useful because if you start a task and the node goes down
there isn't a record of the task at all. This hints to the user that
the task might have died with the node.

Relates to #22027
2016-12-09 15:50:46 -05:00
Yannick Welsch fca4f92fee Fix BWC condition on REST test
Adds a missing skip section to a REST test that was forgotten in #21998
2016-12-09 19:05:00 +01:00
Yannick Welsch db0660a7ea Reject external versioning and explicit version numbers on create (#21998)
Fixes an issue where indexing requests with operation type "create" auto-convert external versioning to internal versioning and silently ignore the version number instead of failing with an error message.
2016-12-09 14:21:22 +01:00
Jason Tedor 4aae017891 Skip IP range query REST test prior to 5.1.2
This commit adds a skip for the IP range query REST test on version
prior to 5.1.2 due to a exclusive bug on the top end of the range.
2016-12-08 16:40:39 -05:00
Adrien Grand 8fe4bc1b74 Fix REST test for ip range aggregations.
Relates to #22018
2016-12-08 18:00:56 +01:00
Isabel Drost-Fromm 5601049014 Merge branch 'master' into feature/rank-eval 2016-12-05 11:22:39 +01:00
Jason Tedor 7861a9dc3e Add skip for missing types REST test
This commit adds a skip for the missing types REST test. There was a bug
for an unclosed XContent object on version prior to version 5.0.3 which
is going to lead to different responses on vresions prior to 5.0.3 and
versions on or after version 5.0.3.
2016-12-01 22:10:47 -05:00
Ryan Ernst a6ad89bee0 Mappings: Fix get mapping when no indexes exist to not fail in response generation (#21924)
When there are no indexes, get mapping has a series of special cases.
Two of those expect the response object already started, and the other
two respond with an exception. Those two cases (types passed in but no
indexes and vice versa) would fail in their error response generation
because it did not expect an object to already be started in the json
generator. This change moves the object start to where it is needed for
the empty responses.

closes #21916
2016-12-01 16:57:12 -08:00
Isabel Drost-Fromm 2b24091361 Merge branch 'master' into feature/rank-eval 2016-11-28 11:45:29 +01:00
Yannick Welsch 590a6372ad Disable disk watermarks on REST tests (#21803)
REST tests use the default OOTB low/high disk watermarks of 85%/90%, which can make some tests fail if run on a machine with a fuller disk. This commit changes the watermarks in the same way as in IntegTestCase so that they're essentially ignored.
2016-11-25 19:52:52 +01:00
Luca Cavanna 720b165350 Search shards to print out aliases array together with alias filter (#21784)
With #21738 we added an indices section to the search shards api, that will return the concrete indices hit by the request, and eventually the corresponding alias filter.

The java API returns the AliasFilter object, which holds the filter itself and an array of aliases that pointed to the index in the original request. The REST layer doesn't print out the aliases array though. This commit adds the aliases array as well and tests for this.
2016-11-25 10:58:06 +01:00
Simon Willnauer 9809760eb0 Fix settings diff generation for affix, list and group settings (#21788)
Group, List and Affix settings generate a bogus diff that turns the actual
diff into a string containing a json structure for instance:

```
"action" : {
  "search" : {
    "remote" : {
      "" : "{\"my_remote_cluster\":\"[::1]:60378\"}"
    }
  }
}
```

which make reading the setting impossible. This happens for instance
if a group or affix setting is rendered via `_cluster/settings?include_defaults=true`
This change fixes the issue as well as several minor issues with affix settings that
where not accepted as valid setting today.
2016-11-24 21:53:04 +01:00
Christoph Büscher e1fe0dc462 Merge branch 'master' into feature/rank-eval 2016-11-24 08:57:26 +01:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language (#21607)
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Luca Cavanna db5a72774b Add indices and filter information to search shards api output (#21738)
Add indices and filter information to search shards api output

The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.

This is relevant since we have moved parsing of alias filters to the coordinating node.

Relates to #20916
2016-11-22 23:00:25 +01:00
Luca Cavanna db8b2dceea Remove ignored type parameter in search_shards api (#21688)
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.

This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.

The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
2016-11-22 17:22:33 +01:00
Christoph Büscher 1d3e58ab9f Merge branch 'master' into feature/rank-eval 2016-11-22 11:31:46 +01:00
Luca Cavanna c174e364a4 Add indices options tests to search api REST tests (#21701)
This is a followup to #21689 where we removed a misplaced try catch for IndexMissingException and IndexClosedException which was related to #9047 (at least for the index closed case). The code block within the change was moved as part of #20890, which made the catch redundant. It was somehow used before (e.g. in 5.0) but it doesn't seem that this catch had any effect. Added tests to verify that. In fact a specific catch added to the search api only would defeat the purpose of having common indices options that work throughout all our APIs.

Relates to #21689
2016-11-21 14:52:43 +01:00
Christoph Büscher 1964d45453 Merge branch 'master' into feature/rank-eval 2016-11-17 17:37:39 +01:00
Dimitrios Athanasiou dbbba8ffbe Add comment to explain flexible assertion in indices rollover IT 2016-11-17 14:49:10 +00:00
Dimitris Athanasiou a75320f89b Replace IndexAlreadyExistsException with ResourceAlreadyExistsException (#21494) 2016-11-17 14:30:21 +00:00
Nik Everett 4bc8feea1d Fix shrink tests on disks w/<15% free
The tests rely on the allocation decider allowing allocation and
the disk threshold decider wasn't allowing it. This came up from
time to in CI but *all* the time on my local machine because it is
a small ssd.
2016-11-16 14:46:52 -05:00
Jason Tedor d06a8903fd Merge branch 'master' into feature/seq_no
* master: (22 commits)
  Add proper toString() method to UpdateTask (#21582)
  Fix `InternalEngine#isThrottled` to not always return `false`. (#21592)
  add `ignore_missing` option to SplitProcessor (#20982)
  fix trace_match behavior for when there is only one grok pattern (#21413)
  Remove dead code from GetResponse.java
  Fixes date range query using epoch with timezone (#21542)
  Do not cache term queries. (#21566)
  Updated dynamic mapper section
  Docs: Clarify date_histogram bucket sizes for DST time zones
  Handle release of 5.0.1
  Fix skip reason for stats API parameters test
  Reduce skip version for stats API parameter tests
  Strict level parsing for indices stats
  Remove cluster update task when task times out (#21578)
  [DOCS] Mention "all-fields" mode doesn't search across nested documents
  InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
  Fixed bad asciidoc in boolean mapping docs
  Fixed bad asciidoc ID in node stats
  Be strict when parsing values searching for booleans (#21555)
  Fix time zone rounding edge case for DST overlaps
  ...
2016-11-16 09:10:35 -05:00
Christoph Büscher 6c6ddf0245 Merge branch 'master' into feature/rank-eval 2016-11-16 11:57:22 +01:00
Nik Everett e66261eee9 Handle release of 5.0.1
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.

Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
2016-11-15 19:31:28 -05:00
Jason Tedor cca22088ac Fix skip reason for stats API parameters test
This commit corrects the reasons that the stats API parameters tests are
skipped against 5.0 nodes.
2016-11-15 17:52:45 -05:00
Jason Tedor 6a8a3e77ed Reduce skip version for stats API parameter tests
This commit reduces the skip version for the stats API unrecognized
parameter tests now that the logic for unrecognized parameters has been
backported to 5.x.
2016-11-15 17:43:19 -05:00
Jason Tedor f5ac0e5076 Remove lenient stats parsing
Today when parsing a stats request, Elasticsearch silently ignores
incorrect metrics. This commit removes lenient parsing of stats requests
for the nodes stats and indices stats APIs.

Relates #21417
2016-11-15 12:17:26 -05:00
Christoph Büscher d6738fa650 Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
2016-11-15 14:56:36 +01:00
Boaz Leskes c9f49039d3 Merge remote-tracking branch 'upstream/master' into feature/seq_no 2016-11-15 10:14:47 +00:00
Simon Willnauer 4c53e55d34 [TEST] Add master_timeout to shrink call to fail faster 2016-11-15 10:59:59 +01:00
Simon Willnauer bdc942fa72 Enable 5.x to 6.x BWC tests
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 14:26:49 +01:00
Jason Tedor 1e7c424479 Merge branch 'master' into feature/seq_no
* master:
  ShardActiveResponseHandler shouldn't hold to an entire cluster state
  Ensures cleanup of temporary index-* generational blobs during snapshotting (#21469)
  Remove (again) test uses of onModule (#21414)
  [TEST] Add assertBusy when checking for pending operation counter after tests
  Revert "Add trace logging when aquiring and releasing operation locks for replication requests"
  Allows multiple patterns to be specified for index templates (#21009)
  [TEST] fixes rebalance single shard check as it isn't guaranteed that a rebalance makes sense and the method only tests if rebalance is allowed
  Document _reindex with random_score
2016-11-11 11:25:27 -05:00
Jason Tedor d3417fb022 Merge branch 'master' into feature/seq_no
* master: (516 commits)
  Avoid angering Log4j in TransportNodesActionTests
  Add trace logging when aquiring and releasing operation locks for replication requests
  Fix handler name on message not fully read
  Remove accidental import.
  Improve log message in TransportNodesAction
  Clean up of Script.
  Update Joda Time to version 2.9.5 (#21468)
  Remove unused ClusterService dependency from SearchPhaseController (#21421)
  Remove max_local_storage_nodes from elasticsearch.yml (#21467)
  Wait for all reindex subtasks before rethrottling
  Correcting a typo-Maan to Man-in README.textile (#21466)
  Fix InternalSearchHit#hasSource to return the proper boolean value (#21441)
  Replace all index date-math examples with the URI encoded form
  Fix typos (#21456)
  Adapt ES_JVM_OPTIONS packaging test to ubuntu-1204
  Add null check in InternalSearchHit#sourceRef to prevent NPE (#21431)
  Add VirtualBox version check (#21370)
  Export ES_JVM_OPTIONS for SysV init
  Skip reindex rethrottle tests with workers
  Make forbidden APIs be quieter about classpath warnings (#21443)
  ...
2016-11-10 23:40:33 -05:00
Alexander Lin 0219a211d3 Allows multiple patterns to be specified for index templates (#21009)
* Allows for an array of index template patterns to be provided to an
index template, and rename the field from 'template' to 'index_pattern'.

Closes #20690
2016-11-10 18:00:30 -05:00
Nik Everett 31be683e57 Skip alias name validation rest test on 5.0 cluster
Alias name validation was introduced in 5.1.
2016-11-08 09:36:47 -05:00
Nik Everett 3787ea27bf Validate alias names the same as index names
Applied (almost) the same rules we use to validate index names
to new alias names. The only rule not applies it "must be lowercase".
We have tests that don't follow that rule and I except there are lots
of examples of camelCase alias names in the wild. We can add that
validation but I'm not sure it is worth it.

Closes #20748

Adds an alias that starts with `#` to the BWC index and validates
that you can read from it and remove it. Starting with `#` isn't
allowed after 5.1.0/6.0.0 so we don't create the alias or check it
after those versions.
2016-11-08 08:23:12 -05:00
Christoph Büscher dbef96c69e Merge branch 'master' of github.com:elastic/elasticsearch into feature/rank-eval 2016-11-08 11:10:10 +01:00
Clinton Gormley 8383af6734 The routing query string param is supported by mget but was missing from the rest spec (#21357) 2016-11-07 14:14:52 +01:00
Christoph Büscher 1702304ecd Merge branch 'master' into feature/rank-eval 2016-11-07 11:02:22 +01:00
Nik Everett a13a050271 Add automatic parallelization support to reindex and friends (#20767)
Adds support for `?slices=N` to reindex which automatically
parallelizes the process using parallel scrolls on `_uid`. Performance
testing sees a 3x performance improvement for simple docs
on decent hardware, maybe 30% performance improvement
for more complex docs. Still compelling, especially because
clusters should be able to get closer to the 3x than the 30%
number.

Closes #20624
2016-11-04 20:59:15 -04:00
Jason Tedor 555084a226 Write -1 on unbounded queue in cat thread pool
Today when writing an unbounded queue_size in the cat thread pool API,
we write null. This commit modifies this so that the output is -1 so
that the output is always present, and always a numeric value.

Relates #21342
2016-11-04 15:17:55 -04:00
Luca Cavanna c2160a88b5 Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api (#20712)
Exist requests are supposed to never throw an exception, but rather return true or false depending on whether some resource exists or not. Indices exists does that for indices and accepts wildcard expressions too. The way the api works internally is by resolving indices and catching IndexNotFoundException: if an exception is thrown the index does not exist hence it returns false, otherwise it returns true. That works ok only if ignore_unavailable and allow_no_indices indices options are both set to false, meaning that they are strict and any missing index or wildcard expressions that resolves to no indices will lead to an exception that can be thrown and cause false to be returned.

Unfortunately the indices options have  been configurable up until now for this request, meaning that one can set ignore_unavailable or allow_no_indices to true and have the indices exist request return true for indices that really don't exist, which makes very little sense in the context of this api.

This commit removes the indicesOptions setter from the IndicesExistsRequest and makes settable only expandWildcardsOpen and expandWildcardsClosed, hence a subset of the available indices options. This way we can guarantee more consistent behaviour of the indices exists api. We can then remove the ignore_unavailable and allow_no_indices option from indices exists api spec
2016-11-04 19:26:37 +01:00
Jayson Minard a978f13660 Fix cat thread pool API spec
This commit fixes an issue with the cat thread pool API spec. Namely,
the name of the variable for the thread pool patterns parameter was
misnamed in the spec.

Relates #21332
2016-11-04 12:25:11 -04:00
Simon Willnauer 436ba7b5fc Validate the `_rollover` target index name early to also fail if dry_run=true (#21330)
Today we validate the target index name late and therefore don't fail for instance
if the target index already exists and `dry_run=true` was specified. This change
validates the index name before we early terminate if dry_run is set.

Closes #21149
2016-11-04 13:53:28 +01:00
Christoph Büscher 619e4c1a44 Merge branch 'master' into feature/rank-eval 2016-11-04 12:48:42 +01:00
Simon Willnauer 68bed9cd3b [TEST] Skip new tests using ingnore_unavailable in BWC tests against 5.0.0
5.0.0 nodes don't understand these parameters and will throw an exception
if a 5.0.0 node will receive the request.

Relates to #21281
2016-11-03 21:47:32 +01:00
Luca Cavanna 00e7026778 Read indices options in indices upgrade API (#21281)
With #21099 we removed support for the ignored allow_no_indices parameter in indices upgrade API. Truth is that ignore_unavailable and expand_wildcards were also ignored, in indices upgrade as well as upgrade status API. Those parameters are though supported internally and settable through java API, hence they should be all supported on the REST layer too.
2016-11-03 18:05:18 +01:00
Simon Willnauer 9015062dcb Rewrite Queries/Filter in FilterAggregationBuilder and ensure client usage marks query as non-cachable (#21303)
`FilterAggregationBuilder` today misses to rewrite queries which causes failures
if a query that uses a client for instance to lookup terms since it must be rewritten first.
This change also ensures that if a client is used from the rewrite context we mark the query as
non-cacheable.

Closes #21301
2016-11-03 16:48:55 +01:00
Simon Willnauer d77d4fa63a Consume `full_id` request parameter early (#21270)
Since we now validate all consumed request parameter, users can't specify
`_cat/nodes?full_id=true|false` anymore since this parameter is consumed late.
This commit adds a test for this parameter and consumes it before request is processed.

Closes #21266
2016-11-03 10:31:35 +01:00
Simon Willnauer c4c4a3a504 Fix `_shrink` test to work under a mixed version cluster (#21262)
today the `_shrink` tests do relocate all shards to a single node in
the cluster. Yet, that is not always possible since the only node we can
safely identify in the cluster is the master and if the master is a BWC
node in such a cluster we won't be able to relocate shards that have a
primary on the newer version nodes since allocation deciders forbid this.

This change restricts allocation for that index when the index is created
to restrict allocation to the master that guarantees that all primaries
are on the same node which is sufficient for the `_shrink` API to run.
2016-11-02 17:09:30 +01:00
Adrien Grand 52de0645fb Remove `lowercase_expanded_terms` and `locale` from query-parser options. (#20208)
Lucene 6.2 introduces the new `Analyzer.normalize` API, which allows to apply
only character-level normalization such as lowercasing or accent folding, which
is exactly what is needed to process queries that operate on partial terms such
as `prefix`, `wildcard` or `fuzzy` queries. As a consequence, the
`lowercase_expanded_terms` option is not necessary anymore. Furthermore, the
`locale` option was only needed in order to know how to perform the lowercasing,
so this one can be removed as well.

Closes #9978
2016-11-02 14:25:08 +01:00
Isabel Drost-Fromm 0b8a2e40cb First step towards supporting templating in rank eval requests. (#20374)
This adds support for templating in rank eval requests.

Relates to #20231

Problem: In it's current state the rank-eval request API forces the user to repeat complete queries for each test request. In most use cases the structure of the query to test will be stable with only parameters changing across requests, so this looks like lots of boilerplate json for something that could be expressed in a more concise way.

Uses templating/ ScriptServices to enable users to submit only one test request template and let them only specify template parameters on a per test request basis.
2016-11-01 11:36:22 +01:00
Simon Willnauer 36c86c2b00 Fix `_cat/store` sort test to use a reliable sort value
using the store size is not reliable since it's cached and might
be stale depending on the timing when the cache entry was generated.
2016-10-28 16:04:33 +02:00
Simon Willnauer 97cc426a89 Fix bwc cluster formation in order to run BWC tests against a mixed version cluster (#21145)
This fixes our cluster formation task to run REST tests against a mixed version cluster.
Yet, due to some limitations in our test framework `indices.rollover` tests are currently
disabled for the BWC case since they select the current master as the merge node which
happens to be a BWC node and we can't relocate all shards to it since the primaries are on
a higher version node. This will be fixed in a followup.

Closes #21142

Note: This has been cherry-picked from 5.0 and fixes several rest tests
as well as a BWC break in `OsStats.java`
2016-10-27 17:03:53 +02:00
Christoph Büscher 51a2e3bf1e Merge branch 'master' into feature/rank-eval 2016-10-27 11:11:37 +02:00