Commit Graph

1800 Commits

Author SHA1 Message Date
Jim Ferenczi 8a7f3f75f3
Add support for rest_total_hits_as_int (#36051)
The support for rest_total_hits_as_int has already been merged to 6x
in #35848 so this change adds this new option to master. The plan was
to add this new option as part of #35848 but we've decided to wait a few
days before merging this breaking change so this commit just handles
the new option as a noop exactly like 6x for now. This will allow
users to migrate to this parameter before #35848 is merged.

Relates #33028
2018-11-29 18:36:16 +01:00
Martijn van Groningen 7624734f14
Added wait_for_metadata_version parameter to cluster state api. (#35535)
The `wait_for_metadata_version` parameter will instruct the cluster state
api to only return a cluster state until the metadata's version is equal or
greater than the version specified in `wait_for_metadata_version`. If  
the specified `wait_for_timeout` has expired then a timed out response 
is returned. (a response with no cluster state and wait for timed out flag set to true)
In  the case metadata's version is equal or higher than  `wait_for_metadata_version`
then the api will immediately return.

This feature is useful to avoid external components from constantly
polling the cluster state to whether somethings have changed in the
cluster state's metadata.
2018-11-26 08:50:08 +01:00
Yannick Welsch 2970abfce9
Add read-only repository verification (#35731)
Adds a verification mode for read-only repositories. It also makes the extra bucket check on
repository creation obsolete, which fixes #35703.
2018-11-23 14:45:05 +01:00
Alpar Torok 8659af68e0
Auto skip license headers on no source (#35640)
* Unmute BuildExamplePluginsIT

* Skip licenseHeaders when there are no sources
2018-11-20 13:02:33 +02:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
Julie Tibshirani ec53288fc0
Remove include_type_name from the relevant APIs. (#35192)
We've decided that the bulk, delete, get, index, update, and search APIs should not
contain this request parameter, and we will instead accept both typed and typeless calls.
2018-11-06 14:33:48 -08:00
Simon Willnauer 833e0f8ecf
Prevent throttled indices to be searched through wildcards by default (#34354)
Today if a wildcard, date-math expression or alias expands/resolves
to an index that is search-throttled we still search it. This is likely
not the desired behavior since it can unexpectedly slow down searches
significantly.

This change adds a new indices option that allows `search`, `count`
and `msearch` to ignore throttled indices by default. Users can
force expansion to throttled indices by using `ignore_throttled=true`
on the rest request to expand also to throttled indices.

Relates to #34352
2018-11-06 09:45:30 +01:00
lipsill d181d1bab1 Remove deprecated url parameters `_source_include` and `_source_exclude` (#35097)
Removes `_source_include` and `_source_exclude` url parameters. 
These parameters have been deprecated in #33475.

Closes #22792
2018-10-31 17:11:59 -04:00
Andy Bristol 010845b233 [test] dont skip last version for mad 2018-10-31 11:36:25 -07:00
Nik Everett 70e939ee38 Search: Test deprecated field in mixed cluster
Update the test for the deprecated `_source_exclude` parameter to run
against 6.6.0+ now that we've backported the deprecation.

Relates to #33475
2018-10-30 11:59:09 -04:00
Andy Bristol b8280ea7cc
median absolute deviation agg (#34482)
This commit adds a new single value metric aggregation that calculates
the statistic called median absolute deviation, which is a measure of
variability that works on more types of data than standard deviation

Our calculation of MAD is approximated using t-digests. In the collect
phase, we collect each value visited into a t-digest. In the reduce
phase, we merge all value t-digests, then create a t-digest of
deviations using the first t-digest's median and centroids
2018-10-30 07:22:52 -07:00
Spencer 6169e9345e
[REST-API] fix url for reload_secure_settings 2018-10-29 12:09:53 -07:00
lipsill 6df1c9e818 Deprecate `_source_include` and `_source_exclude` url parameters (#33475)
Deprecates `_source_include` and `_source_exclude` url parameters
in favor of `_source_inclues` and `_source_excludes` because those
are consistent with the rest of Elasticsearch's APIs.

Relates to #22792
2018-10-29 12:06:38 -04:00
Emily S eaa05fe762
[TEST] Minor updates to rest api spec tests (#34551) 2018-10-23 19:09:09 +02:00
Julie Tibshirani fbb9ac34f9
Deprecate type exists requests. (#34663) 2018-10-22 08:46:11 -07:00
Christoph Büscher a1c441f78a
HLRC: Add throttling for update & delete-by-query (#33951)
This change adds throttling to the update-by-query and delete-by-query cases
similar to throttling for reindex. This mostly means additional methods on the
client class itself, since the request hits the same RestHandler, just with
slightly different endpoints, and also the return values are similar.
2018-10-02 21:44:15 +02:00
Przemyslaw Gomulka 3f8cc89c9f
Completion types with multi-fields support (#34081)
Mappings with completion type and multi-fields, were not able to index array or
object format on completion fields. Only string format was supported.
This is fixed by providing multiField parser with externalValueContext with already parsed object

closes #15115
2018-10-02 14:32:56 +02:00
Russ Cam 94df8ec358 Rest-Api-Spec: Correct spelling in filter_path description (#33154)
This commit corrects the spelling of response in the description of
filter_path in the REST API spec.
2018-09-28 11:20:51 +02:00
Mayya Sharipova 80c5d30f30
XContentBuilder to handle BigInteger and BigDecimal (#32888)
Although we allow to index BigInteger and BigDecimal into a keyword
field, source filtering on these fields would fail
as XContentBuilder was not able to deserialize BigInteger and BigDecimal
to json.

This modifies XContentBuilder to allow to handle BigInteger and
BigDecimal.

Closes #32395
2018-09-26 14:24:31 -04:00
Adrien Grand 3c2841d493
REST test for typeless APIs. (#33934)
This commit duplicates REST tests for the
 - `indices.create`
 - `indices.put_mapping`
 - `indices.get_mapping`
 - `index`
 - `get`
 - `delete`
 - `update`
 - `bulk`
APIs, so that we both test them when used without types (include_type_name=false)
and with types, mostly for mixed-version cluster tests.

Given a suite called `X_test_name.yml`, I first copied it to
`(X+1)_test_name_with_types.yml` and then changed `X_test_name.yml` to set
`include_type_name=false` on every API that supports it.

Relates #15613
2018-09-26 17:11:37 +02:00
Jim Ferenczi a255880497
Add nested and object fields to field capabilities response (#33803)
This commit adds nested and object fields to the field capabilities response.

Closes #33237
2018-09-26 08:59:41 +02:00
Tomas Della Vedova 7bed915499 [DOCS] Fixed list formatting (#32963) 2018-09-18 17:05:10 +02:00
Jim Ferenczi 5f495c18df Adapt skip version for doc_values format deprecation
This commit fixes bwc rest tests for the doc_values format deprecation
in search. The message of the deprecation changed in 6.4.1 so the bwc test
should not check against 6.4.0.
2018-09-14 10:22:11 +02:00
Colin Goodheart-Smithe 624b84f897
Improves doc values format deprecation message (#33576)
* Improves doc values format deprecation message

This changes the deprecation message when doc values fields do not
supply a format form logging a deprecation warning for each offending
field individually to logging a single message which lists all
offending fields

Closes #33572

* Updates YAML test with new deprecation message

Also adds a test to ensure multiple deprecation warnings are collated
into one message

* Condenses collection of fields without format check

Moves the collection of fields that don't have a format to a separate
loop and moves the logging of the deprecation warning to be next to it
at the expesnse of looping through the field list twice

* fixes typo

* Fixes test
2018-09-11 14:32:43 +01:00
Jim Ferenczi dbc7102c86
Fix inner hits retrieval when stored fields are disabled (_none_) (#33018)
Now that types are unique per mapping we can retrieve the document mapper
without referencing the type. This fixes an NPE when stored fields are disabled.
For 6x we'll need a different fix since mappings can still have multiple types.

Relates #32941
2018-09-04 16:25:52 +02:00
Albert Zaharovits fbe609d589
Reload Secure Settings REST specs & docs (#32990)
This is a minimal REST API spec and docs for the REST handler
for the `_nodes/reload_secure_settings endpoint`.

Relates #29135
2018-08-26 14:49:32 +03:00
Mayya Sharipova 6f1ee76443 Revert "Do NOT allow termvectors on nested fields (#32728)"
This reverts commit fdff8f3db0.
2018-08-24 10:12:16 -04:00
Mayya Sharipova fdff8f3db0
Do NOT allow termvectors on nested fields (#32728)
Requesting _termvectors on a nested field or any sub-fields of a nested field
returns empty results.

Closes #21625
2018-08-23 16:46:47 -04:00
Luca Cavanna 393eec1482
Set maxScore for empty TopDocs to Nan rather than 0 (#32938)
We used to set `maxScore` to `0` within `TopDocs` in situations where there is really no score as the size was set to `0` and scores were not even tracked. In such scenarios, `Float.Nan` is more appropriate, which gets converted to `max_score: null` on the REST layer. That's also more consistent with lucene which set `maxScore` to `Float.Nan` when merging empty `TopDocs` (see `TopDocs#merge`).
2018-08-22 17:23:54 +02:00
Simon Willnauer ffb1a5d5b7
Expose `max_concurrent_shard_requests` in `_msearch` (#33016)
Today `_msearch` doesn't allow modifying the `max_concurrent_shard_requests`
per sub search request. This change adds support for setting this parameter on
all sub-search requests in an `_msearch`.

Relates to #31877
2018-08-22 08:45:08 +02:00
Julie Tibshirani 99c3d8ab26 Fix a bug in the BWC tests where we added a field alias in unsupported versions. 2018-08-08 15:24:23 -07:00
Julie Tibshirani 0281846c6b Adjust the test version now that the field alias bug has been backported. 2018-08-07 16:48:41 -07:00
Julie Tibshirani d7183f8f3d
Make sure that field collapsing supports field aliases. (#32648) 2018-08-07 16:20:09 -07:00
Shaunak Kashyap 0a83968650
Add cluster UUID to Cluster Stats API response (#32206)
* Make cluster stats response contain cluster UUID

* Updating constructor usage in Monitoring tests

* Adding cluster_uuid field to Cluster Stats API reference doc

* Adding rest api spec test for expecting cluster_uuid in cluster stats response

* Adding missing newline

* Indenting do section properly

* Missed a spot!

* Fixing the test cluster ID
2018-08-02 17:14:19 -07:00
Zachary Tong 7c43da280b [TEST] add version skip to weighted_avg tests
Added in 6.4, so we don't want this running on pre-6.4 BWC tests
2018-07-23 19:50:26 -04:00
Zachary Tong 6ba144ae31
Add WeightedAvg metric aggregation (#31037)
Adds a new single-value metrics aggregation that computes the weighted 
average of numeric values that are extracted from the aggregated 
documents. These values can be extracted from specific numeric
fields in the documents.

When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value.  In contrast, weighted averages
scale each datapoint differently.  The amount that each datapoint contributes 
to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`

A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.

Closes #15731
2018-07-23 18:33:15 -04:00
Boaz Leskes f7da55aa3c Rest test - allow for snapshots to take 0 milliseconds
Sometimes we get lucky and things are fast :)
2018-07-19 11:32:36 +02:00
Armin Braun 6de1f96cad
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
* UUID field was added for #31791 and only went into 6.4 and 7.0
* Fixes #32119
2018-07-18 15:32:36 +02:00
Armin Braun 4b5071f2d0
Add Index UUID to `/_stats` Response (#31871)
* Add "uuid" field to each index's section in the `/_stats` response
* closes #31791
2018-07-17 06:50:21 +02:00
Jim Ferenczi 44f0c1df39 Unmute field collapsing rest tests
BWC tests can run now that master and 6x branch are aligned.
Closes #32055
2018-07-16 12:03:28 +02:00
Christoph Büscher 466235288e Mute failing tests
Relates to #32055
2018-07-13 23:09:07 +02:00
Mayya Sharipova 80492cacfc
Add second level of field collapsing (#31808)
* Put second level collapse under inner_hits

Closes #24855
2018-07-13 11:40:03 -04:00
Tal Levy 68a8d13828
add support for is_write_index in put-alias body parsing (#31674)
* add support for is_write_index in put-alias body parsing

The Rest Put-Alias Action does separate parsing of the alias body
to construct the IndicesAliasesRequest. This extra parsing
was missed in #30703.

* test flag was not just ignored by the parser

* disable backcompat tests
2018-07-09 16:03:17 -04:00
Jim Ferenczi 584fa261cc
Remove the ability to index or query context suggestions without context (#31007)
This is a follow up of #30712 that removes the ability to index or query
and context enabled completion field without context.

Relates #30712
2018-07-09 16:01:01 +02:00
Alan Woodward d3a8094d35
Remove obsolete parameters from analyze rest spec (#31795)
This commit also fixes a typo in the analyze high-level client
documentation.
2018-07-06 09:05:34 +01:00
Sohaib Iftikhar 09e8ac8167 REST high-level client: add get index API (#31703)
Also added master_timeout parameter for the indices.get spec

Relates to #27205
2018-07-05 13:52:25 -04:00
Julie Tibshirani 1a54bca712 Mute 'Test typed keys parameter for suggesters' as we await a fix. 2018-06-29 12:57:54 -07:00
Nik Everett 232c71b6bf
QA: Create xpack yaml features (#31403)
This creates a YAML test "features" that indices if the cluster being
tested has xpack installed (`xpack`) or if it does *not* have xpack
installed (`no_xpack`). It uses those features to centralize skipping
a few tests that fail if xpack is installed.

The plan is to use this in a followup to skip docs tests that require
xpack when xpack is not installed. We *plan* to use the declaration
of required license level on the docs page to generate the required
`skip`.

Closes #30933.
2018-06-26 09:26:48 -04:00
Luca Cavanna 0352d88621
Get Mapping API to honour allow_no_indices and ignore_unavailable (#31507)
Get Mapping currently throws index not found exception (and returns
404 status code) from the REST layer whenever an index was specified
and no indices have been returned. We should not have this logic in the
REST layer though as only our index resolver should decide whether we
need to throw exceptions or not based on provided indices and corresponding
indices options.

Closes #31485
2018-06-22 09:57:32 +02:00
Vladimir Dolzhenko 04e4e44409
Add get stored script and delete stored script to high level REST API (#31355)
Add get stored script and delete stored script to high level REST API

Relates to #27205
2018-06-19 14:21:11 +02:00
Tal Levy 3b70e943eb
add is-write-index flag to aliases (#30942)
This commit adds the is-write-index flag for aliases.
It allows requests to set the flag, and responses to display the flag.
It does not validate and/or affect any indexing/getting/updating behavior
of Elasticsearch -- this will be done in a follow-up PR.
2018-06-15 08:45:29 -07:00
Nik Everett 856936c286
REST Client: NodeSelector for node attributes (#31296)
Add a `NodeSelector` so that users can filter the nodes that receive
requests based on node attributes.

I believe we'll need this to backport #30523 and we want it anyway.

I also added a bash script to help with rebuilding the sniffer parsing
test documents.
2018-06-15 08:04:54 -04:00
Nik Everett 0d9b78834f
LLClient: Support host selection (#30523)
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the  `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```

This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.

I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.

Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:

```
---
"example test":
  - skip:
      features: host_selector
  - do:
      host_selector:
        version: " - 7.0.0" # same syntax as skip
      apiname:
        something: true
```

The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.

The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.

Closes #21888
2018-06-11 17:07:27 -04:00
Simon Willnauer f825a530b8
Limit the number of concurrent requests per node (#31206)
With `max_concurrent_shard_requests` we used to throttle / limit
the number of concurrent shard requests a high level search request
can execute per node. This had several problems since it limited the
number on a global level based on the number of nodes. This change
now throttles the number of concurrent requests per node while still
allowing concurrency across multiple nodes.

Closes #31192
2018-06-11 08:49:18 +02:00
Martijn van Groningen 07a57cc131
Move number of language analyzers to analysis-common module (#31143)
The following analyzers were moved from server module to analysis-common module:
`snowball`, `arabic`, `armenian`, `basque`, `bengali`, `brazilian`, `bulgarian`,
`catalan`, `chinese`, `cjk`, `czech`, `danish`, `dutch`, `english`, `finnish`,
`french`, `galician` and `german`.

Relates to #23658
2018-06-08 08:58:46 +02:00
Jim Ferenczi b30aa3137d
Reject long regex in query_string (#31136)
This change applies the existing `index.max_regex_length` to regex queries
produced by the `query_string` query.

Relates #28344
2018-06-07 09:29:26 +02:00
Lee Hinman b22a055bcf
Add get mappings support to high-level rest client (#30889)
This adds support for the get mappings API to the high level rest client.

Relates to #27205
2018-06-04 14:31:08 -06:00
Jim Ferenczi f94a75778c
Fix index prefixes to work with span_multi (#31066)
* Fix index prefixes to work with span_multi

Text fields that use `index_prefixes` can rewrite `prefix` queries into
`term` queries internally. This commit fix the handling of this rewriting
in the `span_multi` query.
This change also copies the index options of the text field into the
prefix field in order to be able to run positional queries. This is mandatory
for `span_multi` to work but this could also be useful to optimize `match_phrase_prefix`
queries in a follow up. Note that this change can only be done on indices created
after 6.3 since we set the index options to doc only in this version.

Fixes #31056
2018-06-04 21:48:56 +02:00
Alan Woodward 0427339ab0
Index phrases (#30450)
Specifying `index_phrases: true` on a text field mapping will add a subsidiary
[field]._index_phrase field, indexing two-term shingles from the parent field.
The parent analysis chain is re-used, wrapped with a FixedShingleFilter.

At query time, if a phrase match query is executed, the mapping will redirect it
to run against the subsidiary field.

This should trade faster phrase querying for a larger index and longer indexing
times.

Relates to #27049
2018-06-04 08:50:35 +01:00
Michael Basnight d826cb36c3
Remove version read/write logic in Verify Response (#30879)
Since master will always communicate with a >=6.4 node, the logic for
checking if the node is 6.4 and conditionally reading and writing based
on that can be removed from master. This logic will stay in 6.x as it is
the bridge to the cleaner response in master. This also unmutes the
failing test due to this bwc change.

Closes #30807
2018-05-31 12:10:01 -05:00
Jim Ferenczi 0f5e570184
Deprecates indexing and querying a context completion field without context (#30712)
This change deprecates completion queries and documents without context that target a
context enabled completion field. Querying without context degrades the search
performance considerably (even when the number of indexed contexts is low).
This commit targets master but the deprecation will take place in 6.x and the functionality
will be removed in 7 in a follow up.

Closes #29222
2018-05-31 16:09:48 +02:00
Jim Ferenczi f582418ada Fix missing option serialization after backport
Relates #29465
2018-05-30 12:55:31 +02:00
Jim Ferenczi e33d107f84
Add missing_bucket option in the composite agg (#29465)
This change adds a new option to the composite aggregation named `missing_bucket`.
This option can be set by source and dictates whether documents without a value for the
source should be ignored. When set to true, documents without a value for a field emits
an explicit `null` value which is then added in the composite bucket.
The `missing` option that allows to set an explicit value (instead of `null`) is deprecated in this change and will be removed in a follow up (only in 7.x).
This commit also changes how the big arrays are allocated, instead of reserving
the provided `size` for all sources they are created with a small intial size and they grow
depending on the number of buckets created by the aggregation:
Closes #29380
2018-05-30 09:48:40 +02:00
Alan Woodward 67905c85a5
Rename index_prefix to index_prefixes (#30932)
This commit also adds index_prefixes tests to TextFieldMapperTests to ensure that cloning and wire-serialization work correctly
2018-05-30 08:32:31 +01:00
Martijn van Groningen 544822c78b
Moved keyword tokenizer to analysis-common module (#30642)
Relates to #23658
2018-05-29 19:22:28 +02:00
Vladimir Dolzhenko b55b079a90
Include size of snapshot in snapshot metadata #18543, bwc clean up (#30890) 2018-05-26 21:20:44 +02:00
Vladimir Dolzhenko 81eb8ba0f0
Include size of snapshot in snapshot metadata (#29602)
Include size of snapshot in snapshot metadata

Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.

Closes #18543
2018-05-25 21:04:50 +02:00
Tom Callahan 36fbb4cb48
Harmonize include_defaults tests (#30700)
This PR breaks the include_defaults functionality of the get settings API into its own
test, which is skipped for mixed-mode clusters containing pre-6.4 nodes.
2018-05-25 09:41:16 -04:00
David Roberts 40534ccabc [TEST] Mute {p0=snapshot.get_repository/10_basic/Verify created repository} YAML test
Issue is #30807
2018-05-25 12:58:02 +01:00
Michael Basnight e1ffbeb824
Fix bad version check writing Repository nodes (#30846)
The writeTo method of VerifyRepositoryResponse incorrectly used its
local version to determine what it was receiving, rather than the
sender's version. This fixes a bug that ocassionally happened when a 6.4
master node sent data to a 7.0 client, causing the number of bytes to be
improperly read. This also unmutes the test.

Closes #30807
2018-05-24 19:21:57 -05:00
Julie Tibshirani f55b09bae4 Update the version checks around ip_range bucket keys, now that the change was backported. 2018-05-24 12:04:18 -07:00
Julie Tibshirani 638a719370
Ensure that ip_range aggregations always return bucket keys. (#30701) 2018-05-24 08:55:14 -07:00
Adrien Grand 405eb7a751 Change serialization version of doc-value fields.
Relates #29639
2018-05-23 18:34:05 +02:00
Adrien Grand a19df4ab3b
Add a `format` option to `docvalue_fields`. (#29639)
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.

Closes #27740
2018-05-23 14:39:04 +02:00
Colin Goodheart-Smithe 483b25330b
Mustes {p0=snapshot.get_repository/10_basic/*} YAML test
This is awaiting a fix for
https://github.com/elastic/elasticsearch/issues/30807
2018-05-23 11:32:58 +01:00
olcbean af8ad8d172 Add more yaml tests for get alias API (#29513) 2018-05-22 11:48:28 +02:00
Jason Tedor d68c44b76c
Default copy settings to true and deprecate on the REST layer (#30598)
This commit defaults the copy_settings REST parameter to the shrink and
split APIs to true, and deprecates the parameter.
2018-05-18 10:12:08 -04:00
Zachary Tong d120fb222c [TEST] Adjust version skips for movavg/movfn tests
Since the MovFn PR was backported to 6.x, we can adjust
the version skip numbers in master to correctly
match 6.3.99 instead of 6.4.0
2018-05-17 18:07:52 +00:00
Mayya Sharipova 3dfa93ef7c
Improve explanation in rescore (#30629)
Currently in a rescore request if window_size is smaller than
the top N documents returned (N=size), explanation of scores could be incorrect
for documents that were a part of topN and not part of rescoring.
This PR corrects this, but saving in RescoreContext docIDs of documents
for which rescoring was applied, and adding rescoring explanation
only for these docIDs.

Closes #28725
2018-05-17 07:09:18 -04:00
Tanguy Leroux 2ac1f9fe89
Fix _cluster/state to always return cluster_uuid (#30656)
Since #30143, the Cluster State API should always returns the current
cluster_uuid in the response body, regardless of the metrics filters.

This is not exactly true as it is returned only if metadata metrics and
no specific indices are requested.

This commit fixes the behavior to always return the cluster_uuid and
add new test.
2018-05-17 10:58:25 +02:00
Zachary Tong df853c49c0
Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (#29594)
This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points.  It is the scripted version of MovingAvg pipeline agg.

Through custom script contexts, we expose a number of convenience methods:

 - MovingFunctions.max()
 - MovingFunctions.min()
 - MovingFunctions.sum()
 - MovingFunctions.unweightedAvg()
 - MovingFunctions.linearWeightedAvg()
 - MovingFunctions.ewma()
 - MovingFunctions.holt()
 - MovingFunctions.holtWinters()
 - MovingFunctions.stdDev()

The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
2018-05-16 10:57:00 -04:00
Simon Willnauer b50cf3c6b0
Side-step pending deletes check (#30571)
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.

Closes #30416
2018-05-15 11:51:54 +02:00
Jason Tedor 4e33443690
Adjust versions for resize copy settings (#30578)
Now that the change to deprecate copy settings and disallow it being
explicitly set to false is backported, this commit adjusts the BWC
versions in master.
2018-05-14 16:41:25 -04:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Martijn van Groningen 7b95470897
Moved tokenizers to analysis common module (#30538)
The following tokenizers were moved: classic, edge_ngram,
letter, lowercase, ngram, path_hierarchy, pattern, thai, uax_url_email and
whitespace.

Left keyword tokenizer factory in server module, because
normalizers directly depend on it.This should be addressed on a
follow up change.

Relates to #23658
2018-05-14 07:55:01 +02:00
Jason Tedor 593fdd40ed
Deprecate not copy settings and explicitly disallow (#30404)
We want copying settings to be the default behavior. This commit
deprecates not copying settings, and disallows explicitly not copying
settings. This gives users a transition path to the future default
behavior.
2018-05-13 10:30:05 -04:00
Tal Levy 34f92df2d3 AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
there are two tests that have failed multiple times in one day on windows CI.

This commit AwaitsFixes them until their timeout issues are resolved.

tracking here: https://github.com/elastic/elasticsearch/issues/30503
2018-05-09 18:25:25 -07:00
Boaz Leskes 9f5fe49cec Disable REST default settings testing until #29229 is back-ported
That PR changed the execution path of index settings default to be on the master
until the PR is back-ported the old master will not return default settings.
2018-05-07 13:30:14 +02:00
tomcallahan 0a93956194
Add Get Settings API support to java high-level rest client (#29229)
This PR adds support for the Get Settings API to the java high-level rest client.
Furthermore, logic related to the retrieval of default settings has been moved from the rest layer into the transport layer and now default settings may be retrieved consistency via both the rest API and the transport API.
2018-05-04 11:14:28 -04:00
Adrien Grand bcdf3d5c61 Post backport of #29658. 2018-05-02 11:43:50 +02:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00
Adrien Grand 7358946bda
Add a new `_ignored` meta field. (#29658)
This adds a new `_ignored` meta field which indexes and stores fields that have
been ignored at index time because of the `ignore_malformed` option. It makes
malformed documents easier to identify by using `exists` or `term(s)` queries
on the `_ignored` field.

Closes #29494
2018-05-02 10:47:02 +02:00
Jason Tedor 5de6f4ff7b Adjust copy settings on resize BWC version
This commit adjusts the BWC version for copy settings on resize
operations after the behavior was backported to 6.x.
2018-05-01 08:49:16 -04:00
Jason Tedor 50535423ff
Allow copying source settings on resize operation (#30255)
Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.

This change is the first step in our adventure:
 - this flag is added here in 7.0.0 and immediately deprecated
 - this flag will be backported to 6.4.0 and remain deprecated
 - then, we will remove the ability to set this flag to false in 7.0.0
 - finally, in 8.0.0 we will remove this flag and the only behavior will
   be for settings to be copied
2018-05-01 08:48:19 -04:00
Chris Earle 421bd9bd7a
_cluster/state Skip Test for pre-6.4, not pre-7.0 (#30264)
This updates the skip section for the new `_cluster/state` responses to
include 6.4+ now that it has been backported.
2018-04-30 14:53:48 -04:00
Chris Earle 725a5af2c6
_cluster/state should always return cluster_uuid (#30143)
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
2018-04-30 10:16:11 -04:00
Julie Tibshirani f5978d6d33
In the field capabilities API, remove support for providing fields in the request body. (#30185) 2018-04-27 16:14:11 -07:00
Alexander Reelsen e1a16a6018
REST: Remove GET support for clear cache indices (#29525)
Clearing the cache indices can be done via GET and POST. As GET should
only support read only operations, this removes the support for using
GET for clearing the indices caches.
2018-04-27 08:41:36 +02:00
Jason Tedor c12c2a6cc9 Rename the bulk thread pool to write thread pool (#29593)
This commit renames the bulk thread pool to the write thread pool. This
is to better reflect the fact that the underlying thread pool is used to
execute any document write request (single-document index/delete/update
requests, and bulk requests).

With this change, we add support for fallback settings
thread_pool.bulk.* which will be supported until 7.0.0.

We also add a system property so that the display name of the thread
pool remains as "bulk" if needed to avoid breaking users.
2018-04-19 08:18:58 -04:00
Martijn van Groningen 8afa7c174f
Added painless execute api. (#29164)
Added an api that allows to execute an arbitrary script and a result to be returned.

```
POST /_scripts/painless/_execute
{
  "script": {
    "source": "params.var1 / params.var2",
    "params": {
      "var1": 1,
      "var2": 1
    }
  }
}
```

Relates to #27875
2018-04-19 09:33:34 +02:00
Jason Tedor 2b47d67d95
Remove the index thread pool (#29556)
Now that single-document indexing requests are executed on the bulk
thread pool the index thread pool is no longer needed. This commit
removes this thread pool from Elasticsearch.
2018-04-18 09:18:08 -04:00
Adrien Grand d223bcf7ab
Add the `include_type_name` option to the search and document APIs. (#29506)
This commit add the `include_type_name` option to the `index`, `update`,
`delete`, `get`, `bulk` and `search` APIs. When set to `false`, the response
will omit the `_type` in the response. This option doesn't work if the endpoint
contains a type. For instance, the following call would succeed:

```
GET index/_doc/1?include_type_name=false
```

But the following one would fail:

```
GET index/some_type/1?include_type_name=false
```

Relates #15613
2018-04-17 11:29:08 +02:00
olcbean b3e3b80f1b REST high-level client: add support for Indices Update Settings API [take 2] (#29327)
Relates to #27205
2018-04-16 21:39:11 +02:00
Ke Li 0bfb59dcf2 Using ObjectParser in UpdateRequest (#29293)
CRUD: Parsing changes for UpdateRequest (#29293)

Use `ObjectParser` to parse `UpdateRequest` so we reject unknown fields
and drop support for the `_fields` parameter because it was deprecated
in 5.x.
2018-04-16 08:39:35 -04:00
Adrien Grand 553c718d66
Make index APIs work without types. (#29479)
Unlike the `indices.create`, `indices.get_mapping` and `indices.put_mapping`
APIs, the index APIs do not need the `include_type_name` option, they can work
work with and without types withouth knowing whether types are being used.

Internally, `_doc` is used as a type if no type is provided, like for the
`indices.put_mapping` API.
2018-04-13 09:08:45 +02:00
Adrien Grand ebd6b5b7ba
Deprecate filtering on `_type`. (#29468)
As indices are only allowed to have one type now, and types are going away in
the future, we should deprecate filtering by `_type`.

Relates #15613
2018-04-13 09:07:51 +02:00
Nhat Nguyen 067fbb8ecd Backport periodic flush count to v6.3.0
Relates #29360
2018-04-11 17:14:28 -04:00
Nhat Nguyen 4e6a8900a3
Add periodic flush count to flush stats (#29360)
Currently, a flush stats contains only the total flush which is the sum
of manual flush (via API) and periodic flush (async triggered when the
uncommitted translog size is exceeded the flush threshold). Sometimes,
it's useful to know these two numbers independently. This commit tracks
and returns a periodic flush count in a flush stats.
2018-04-11 11:15:33 -04:00
Adrien Grand 6a6c0ea5e6
Add an `include_type_name` option. (#29453)
This adds an `include_type_name` option to the `indices.create`,
`indices.get_mapping` and `indices.put_mapping` APIs, which defaults to `true`.
When set to `false`, then mappings will be returned directly in the body of
the `indices.get_mapping` API, without keying them by the type name, the
`indices.create` will expect mappings directly under the `mappings` key, and
the `indices.put_mapping` will use `_doc` as a type name and fail if a `type`
is provided explicitly.

Relates #15613
2018-04-11 15:54:16 +02:00
Adrien Grand 4918924fae
Remove legacy mapping code. (#29224)
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
2018-04-11 09:41:37 +02:00
Adrien Grand 6949c888bf Remove useless skip.
Relates #29353
2018-04-11 09:40:28 +02:00
Adrien Grand a091d950a7
Deprecate slicing on `_uid`. (#29353)
Deprecate slicing on `_uid`.

`_id` should be used instead on 6.x.
2018-04-10 14:28:30 +02:00
Lee Hinman a93c942927
Move ObjectParser into the x-content lib (#29373)
* Move ObjectParser into the x-content lib

This moves `ObjectParser`, `AbstractObjectParser`, and
`ConstructingObjectParser` into the libs/x-content dependency. This decoupling
allows them to be used for parsing for projects that don't want to depend on the
entire Elasticsearch jar.

Relates to #28504
2018-04-06 09:41:14 -06:00
Christoph Büscher 570f1d9ac7
Add indices options support to _rank_eval (#29386)
Currently the ranking evaluation API doesn't support many of the
standard parameters of the search API. Some of these make sense, like
adding support for the common indices options parameters, which this
change adds.
2018-04-06 16:23:19 +02:00
Jason Tedor 8fdca6a89a
Align cat thread pool info to thread pool config (#29195)
Today we report thread pool info using a common object. This means that
we use a shared set of terminology that is not consistent with the
terminology used to the configure thread pools. This holds in particular
for the minimum and maximum number of threads in the thread pool where
we use the following terminology:
 thread pool info | fixed | scaling
  min                core    size
  max                max     size

A previous change addressed this for the nodes info API. This commit
changes the display of thread pool info in the cat thread pool API too
to be dependent on the type of the thread pool so that we can align the
terminology in the output of thread pool info with the terminology used
to configure a thread pool.
2018-04-03 17:27:26 -04:00
Johnny Marnell 4db6fc9a08 Reindex: Fix error in delete-by-query rest spec (#29318) 2018-04-03 11:22:04 -04:00
Mayya Sharipova e70cd35bda
Revert "REST high-level client: add support for Indices Update Settings API (#28892)" (#29323)
This reverts commit b67b5b1bbd.
2018-03-30 16:26:46 -07:00
olcbean b67b5b1bbd REST high-level client: add support for Indices Update Settings API (#28892)
Relates to #27205
2018-03-30 10:53:29 +02:00
Jim Ferenczi dd77d7fd0a #28745: remove extra option in the composite rest tests
`allow_partial_search_results` is not needed for these tests.
2018-03-26 14:32:59 +02:00
Christoph Büscher 318b0af953 Remove execute mode bit from source files
Some source files seem to have the execute bit (a+x) set, which doesn't
really seem to hurt but is a bit odd. This change removes those, making
the permissions similar to other source files in the repository.
2018-03-26 13:37:55 +02:00
Jim Ferenczi 5288235ca3
Optimize the composite aggregation for match_all and range queries (#28745)
This change refactors the composite aggregation to add an execution mode that visits documents in the order of the values
present in the leading source of the composite definition. This mode does not need to visit all documents since it can early terminate
the collection when the leading source value is greater than the lowest value in the queue.
Instead of collecting the documents in the order of their doc_id, this mode uses the inverted lists (or the bkd tree for numerics) to collect documents
in the order of the values present in the leading source.
For instance the following aggregation:

```
"composite" : {
  "sources" : [
    { "value1": { "terms" : { "field": "timestamp", "order": "asc" } } }
  ],
  "size": 10
}
```
... can use the field `timestamp` to collect the documents with the 10 lowest values for the field instead of visiting all documents.
For composite aggregation with more than one source the execution can early terminate as soon as one of the 10 lowest values produces enough
composite buckets. For instance if visiting the first two lowest timestamp created 10 composite buckets we can early terminate the collection since it
is guaranteed that the third lowest timestamp cannot create a composite key that compares lower than the one already visited.

This mode can execute iff:
 * The leading source in the composite definition uses an indexed field of type `date` (works also with `date_histogram` source), `integer`, `long` or `keyword`.
 * The query is a match_all query or a range query over the field that is used as the leading source in the composite definition.
 * The sort order of the leading source is the natural order (ascending since postings and numerics are sorted in ascending order only).

If these conditions are not met this aggregation visits each document like any other agg.
2018-03-26 09:51:37 +02:00
Nhat Nguyen 8cfe619f03 Revert "[TEST] Mute index synced flush rest tests"
The BWC issue was fixed.
This reverts commit f8830b7b43.
2018-03-21 16:51:52 -04:00
David Roberts f8830b7b43 [TEST] Mute index synced flush rest tests
Awaiting fix of #29162
2018-03-21 10:09:21 -04:00
Luca Cavanna ff09c82319
REST high-level client: add clear cache API (#28866)
* REST high-level client: add clear cache API

Relates to #27205

Also Closes #26947 (rest-spec were outdated)
2018-03-20 10:39:36 +01:00
olcbean 3d81497f25 REST: Clear Indices Cache API remove deprecated url params (#29068)
By the time the master branch is released the deprecated url
parameters in the `/_cache/clear` API will have been deprecated
for a couple of minor releases. Since master will be the next
major release we are fine with removing these parameters.
2018-03-14 16:37:50 -04:00
Nik Everett 3e0e6444cf Mark field_data as deprecated in /_cache/clear
It was deprecated in #28943.
2018-03-14 11:39:35 -04:00
Nik Everett 8f0da37c5e Revert "REST tests: Increase version to skip to include 6.3"
This reverts commit 9f2c4df94d which isn't
needed now that the backport is complete.
2018-03-13 20:22:32 -04:00
Alexander Reelsen 9f2c4df94d REST tests: Increase version to skip to include 6.3
In order to fix the tests, the correct version needs to be skipped until
the backport is done.

Relates #28943
2018-03-13 17:00:01 -07:00
Paul Sanwald 6dae955b6a
Document and test date_range "missing" support (#28983)
* Add a REST integration test that documents date_range support

Add a test case that exercises date_range aggregations using the missing
option.

Addresses #17597

* Test cleanup and correction

Adding a document with a null date to exercise `missing` option, update
test name to something reasonable.

* Update documentation to explain how the "missing" parameter works for
date_range aggregations.

* Wrap lines at 80 chars in docs.

* Change format of test to YAML for readability.
2018-03-13 12:58:30 -07:00
olcbean edc57f6f34 REST: deprecate `field_data` in Clear Cache API (#28943)
We call it `fielddata` everywhere else in the code and API so we may as
well be consistent.
2018-03-13 15:16:27 -04:00
olcbean 3cf599be4c REST api specs : remove unsupported `wait_for_merge` param (#28959) 2018-03-13 12:12:52 +01:00
Spencer 9709d89daf
[rest-api-spec] update doc link for /_rank_eval 2018-03-08 16:06:23 -07:00
Jim Ferenczi bc8b3fc71c Revert "Rescore collapsed documents (#28521)"
This reverts commit f057fc294a.
The rescorer does not resort the collapsed values inside the top docs
during rescoring. For this reason the Lucene rescorer is not compatible
with collapsing.
Relates #27243
2018-03-08 11:20:29 +01:00
Sergey Galkin f057fc294a Rescore collapsed documents (#28521)
This change adds the ability to rescore collapsed documents.
2018-03-04 13:39:50 -08:00
Andrew Odendaal f207aacac0 Missing `timeout` parameter from the REST API spec JSON files (#28328)
Closes #28200
2018-03-02 14:47:06 +01:00
olcbean 25834e2d26 REST api spec: remove a common param from nodes.usage.json (#28835)
"human" is a common parameter defined in `_common.json`
It should not be repeated again for a concrete api.

Closes #28226
2018-03-01 10:58:38 +01:00
olcbean ec0a1b36c9 REST api spec: remove unsupported parameter `parent_node` (#28841) 2018-03-01 10:57:25 +01:00
Ke Li a77273fc01 Reject regex search if regex string is too long (#28542)
* Reject regex search if regex string is too long (#28344)

* Add docs

* Introduce index level setting `index.max_regex_length`
 to control the maximum length of the regular expression

Closes #28344
2018-02-23 10:41:24 -08:00
Nhat Nguyen 30598627ce Fix BWC issue of the translog last modified age stats
We added a rest test for the translog last modified age without a
version check. This causes BWC failed because the stats are not
available in the old versions.

Relates #28613
2018-02-16 14:55:20 -05:00
Justin Wyer 5aeb479ffd Add translog file age to Translog Stats (#28613)
Expose the age of translog files in the translog stats. This is useful to reason about your translog retention policy.

Closes #28189
2018-02-16 16:23:33 +01:00
Boaz Leskes beb55d148a
Simplify the Translog constructor by always expecting an existing translog (#28676)
Currently the Translog constructor is capable both of opening an existing translog and creating a
new one (deleting existing files). This PR separates these two into separate code paths. The
constructors opens files and a dedicated static methods creates an empty translog.
2018-02-15 09:24:09 +01:00
Ke Li fc406c9a5a Upgrade t-digest to 3.2 (#28295) (#28305) 2018-02-15 08:23:20 +00:00
Jim Ferenczi 37e938f9de Fix indices.sort rest test
Set the number of replicas to 0 in order to avoid race condition during the test
Fixes #24416
2018-02-12 09:03:55 +01:00
Martijn van Groningen 2a35b4ee2b
Use right skip versions.
Closes #27570
2018-02-06 12:22:42 +01:00
markharwood 77d2dd203e
Search - add allow_partial_search_results flag with default setting false (#28440)
Adds allow_partial_search_results flag to search requests with default setting = true.
When false, will error if search either timeouts, has partial errors or has missing shards rather
than returning partial search results. A cluster-level setting provides a default for search requests with no flag.

Closes #27435
2018-01-31 15:51:29 +00:00
Alan Woodward 424ecb3c7d
Add ability to index prefixes on text fields (#28290)
This adds the ability to index term prefixes into a hidden subfield, enabling prefix queries to be run without multitermquery rewrites. The subfield reuses the analysis chain of its parent text field, appending an EdgeNGramTokenFilter. It can be configured with minimum and maximum ngram lengths. Query terms with lengths outside this min-max range fall back to using prefix queries against the parent text field.

The mapping looks like this:

"my_text_field" : {
"type" : "text",
"analyzer" : "english",
"index_prefix" : { "min_chars" : 1, "max_chars" : 10 }
}

Relates to #27049
2018-01-30 08:26:56 +00:00
Jim Ferenczi 95c45aeb5d Adapt bwc version after backport #28358 2018-01-25 09:26:10 +01:00
Jim Ferenczi c26d4ac6c1
Always return the after_key in composite aggregation response (#28358)
This change adds the `after_key` of a composite aggregation directly in the response.
It is redundant when all buckets are not filtered/removed by a pipeline aggregation since in this case the `after_key` is always the last bucket
in the response. Though when using a pipeline aggregation to filter composite buckets, the `after_key` can be lost if the last bucket is filtered.
This commit fixes this situation by always returning the `after_key` in a dedicated section.
2018-01-25 09:15:27 +01:00
Jim Ferenczi b10d166190 Adapt bwc version after backport #28310 2018-01-24 09:17:30 +01:00
Jim Ferenczi 19cfc25873
Adds the ability to specify a format on composite date_histogram source (#28310)
This commit adds the ability to specify a date format on the `date_histogram` composite source.
If the format is defined, the key for the source is returned as a formatted date.

Closes #27923
2018-01-23 15:14:49 +01:00