Commit Graph

701 Commits

Author SHA1 Message Date
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. (#25222)
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
Christoph Büscher ac3db8c30f Merge branch 'master' into feature/rank-eval 2017-06-14 11:57:05 +02:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Jim Ferenczi 5e8b569255 fix highlighting docs 2017-06-09 14:42:08 +02:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
Andrey Groshev e4fd8485ce Made the same length of opening and closing lines (#23583) 2017-06-09 00:50:43 -07:00
Jim Ferenczi 36a5cf8f35 Automatically early terminate search query based on index sorting (#24864)
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and 
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.

Relates #6720
2017-06-08 12:10:46 +02:00
Yibin Lin fbf2e3d574 Tiny correction in inner-hits.asciidoc (#25066) 2017-06-06 13:26:37 +02:00
Christoph Büscher 3d6fb4eb0b Merge branch 'master' into feature/rank-eval 2017-05-30 14:24:26 +02:00
Clinton Gormley 0656d0236b Update context-suggest.asciidoc
Removed incorrect parameter
2017-05-26 17:41:40 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
António Ribeiro 85a1b2b406 Fix link to perl docs (#24842)
* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Issue #24606 - Changed the link text to Search::Elasticsearch::Client::5_0::Bulk and
Search::Elasticsearch::Client::5_0::Scroll.
2017-05-24 11:43:54 +02:00
Nik Everett 13a86fec99 Add magic $_path stash key to docs tests (#24724)
Adds a "magic" key to the yaml testing stash mostly for use with
documentation tests. When unstashing an object, `$_path` is the
path into the current position in the object you are unstashing.
This means that in docs tests you can use
`// TESTRESPONSEs/somevalue/$body.${_path}/` to mean "replace
`somevalue` with whatever is the response in the same position."

Compare how you must carefully mock out all the numbers in the profile
response without this change:
```
// TESTRESPONSE[s/"id": "\[2aE02wS1R8q_QFnYu6vDVQ\]\[twitter\]\[1\]"/"id": $body.profile.shards.0.id/]
// TESTRESPONSE[s/"rewrite_time": 51443/"rewrite_time": $body.profile.shards.0.searches.0.rewrite_time/]
// TESTRESPONSE[s/"score": 51306/"score": $body.profile.shards.0.searches.0.query.0.breakdown.score/]
// TESTRESPONSE[s/"time_in_nanos": "1873811"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.time_in_nanos/]
// TESTRESPONSE[s/"build_scorer": 2935582/"build_scorer": $body.profile.shards.0.searches.0.query.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 919297/"create_weight": $body.profile.shards.0.searches.0.query.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 53876/"next_doc": $body.profile.shards.0.searches.0.query.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "391943"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.0.time_in_nanos/]
// TESTRESPONSE[s/"score": 28776/"score": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 784451/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 1669564/"create_weight": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 10111/"next_doc": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "210682"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.1.time_in_nanos/]
// TESTRESPONSE[s/"score": 4552/"score": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 42602/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 89323/"create_weight": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 2852/"next_doc": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "304311"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.time_in_nanos/]
// TESTRESPONSE[s/"time_in_nanos": "32273"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.children.0.time_in_nanos/]
```

To how you can cavalierly mock all the numbers at once with this change:
```
// TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
```
2017-05-23 15:33:48 -04:00
Jack Conradson 0aa380b770 Fix search template documentation reference to scripting security. 2017-05-18 14:27:58 -07:00
Christoph Büscher cd0941810f Merge branch 'master' into feature/rank-eval 2017-05-18 16:47:47 +02:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Martijn van Groningen 840da4aebf
Removed deprecated template query.
Relates to #19390
2017-05-11 14:56:45 +02:00
Adrien Grand a72eaa8e0f Identify documents by their `_id`. (#24460)
Now that indices have a single type by default, we can move to the next step
and identify documents using their `_id` rather than the `_uid`.

One notable change in this commit is that I made deletions implicitly create
types. This helps with the live version map in the case that documents are
deleted before the first type is introduced. Otherwise there would be no way
to differenciate `DELETE index/foo/1` followed by `PUT index/foo/1` from
`DELETE index/bar/1` followed by `PUT index/foo/1`, even though those are
different if versioning is involved.
2017-05-09 16:33:52 +02:00
Anupam 0b36fb052c Update completion-suggest.asciidoc (#24506) 2017-05-05 11:34:41 -04:00
Simon Willnauer 6b67e0bf2f Include all aliases including non-filtering in `_search_shards` response (#24489)
`_search_shards`API today only returns aliases names if there is an alias
filter associated with one of them. Now it can be useful to see which aliases
have been expanded for an index given the index expressions. This change also includes non-filtering aliases even without a filtering alias being present.
2017-05-05 09:34:12 +02:00
Nik Everett 9f431543fc CONSOLEify inner hits docs
Rewrites most of the snippets in the `innert_hits` docs to be
complete examples and enables `VIEW IN CONSOLE`, `COPY AS CURL`,
and automatic testing of the snippets.
2017-05-04 17:30:54 -04:00
Adrien Grand 977016ba25 Do not index `_type` when there is at most one type. (#24363)
This change makes `_type` behave pretty much like `_index` when
`index.mapping.single_type` is true.
2017-05-04 16:29:35 +02:00
Clinton Gormley 582b3c06b6 Added docs for batched_reduce_size
Relates to #23288
2017-05-02 14:25:03 +02:00
Jim Ferenczi 9d8254fadf Fix FieldCaps documentation
Fix the expected output for field_caps call.
Fixes #24413
2017-05-02 10:14:47 +02:00
Martijn van Groningen b77254871b
docs: document alternative for nested inner hits source
Closes #24110
2017-04-28 11:09:24 +02:00
Guillaume Le Floch 739cb35d1b Allow passing single scrollID in clear scroll API body (#24242)
* Allow single scrollId in string format

Closes #24233
2017-04-25 13:43:21 +02:00
Christoph Büscher 5254731039 Merge branch 'master' into feature/rank-eval 2017-04-22 21:47:32 +02:00
Suhas Karanth f97d8bc78d Update reference docs for Highlighter fragmenter (#23754)
Explain the fragmenter and add examples.
2017-04-17 14:00:24 -04:00
Simon Willnauer 040b86a76b Set shard count limit to unlimited (#24012)
Now that we have incremental reduce functions for topN and aggregations
we can set the default for `action.search.shard_count.limit` to unlimited.
This still allows users to restrict these settings while by default we executed
across all shards matching the search requests index pattern.
2017-04-10 17:09:21 +02:00
Jim Ferenczi 9b3c85dd88 Deprecate _field_stats endpoint (#23914)
_field_stats has evolved quite a lot to become a multi purpose API capable of retrieving the field capabilities and the min/max value for a field.
In the mean time a more focused API called `_field_caps` has been added, this enpoint is a good replacement for _field_stats since he can
retrieve the field capabilities by just looking at the field mapping (no lookup in the index structures).
Also the recent improvement made to range queries makes the _field_stats API obsolete since this queries are now rewritten per shard based on the min/max found for the field.
This means that a range query that does not match any document in a shard can return quickly and can be cached efficiently.
For these reasons this change deprecates _field_stats. The deprecation should happen in 5.4 but we won't remove this API in 6.x yet which is why
 this PR is made directly to 6.0.
 The rest tests have also been adapted to not throw an error while this change is backported to 5.4.
2017-04-10 10:10:16 +02:00
Nik Everett 7fad7c675d Rewrite the scripting security docs (#23930)
They needed to be updated now that Painless is the default and
the non-sandboxed scripting languages are going away or gone.

I dropped the entire section about customizing the classloader
whitelists. In master this barely does anything (exposes more
things to expressions).
2017-04-07 11:46:41 -04:00
Nik Everett 048191ceb6 CONSOLEify highlighting a function_score docs
Converts many of the partial examples into full search requests.

Relates #18160
2017-04-06 08:13:56 -04:00
Christoph Büscher 024ed1b6ca Merge branch 'master' into feature/rank-eval 2017-04-04 18:23:41 +02:00
Jim Ferenczi a8250b26e7 Add FieldCapabilities (_field_caps) API (#23007)
This change introduces a new API called `_field_caps` that allows to retrieve the capabilities of specific fields.

Example:

````
GET t,s,v,w/_field_caps?fields=field1,field2
````
... returns:
````
{
   "fields": {
      "field1": {
         "string": {
            "searchable": true,
            "aggregatable": true
         }
      },
      "field2": {
         "keyword": {
            "searchable": false,
            "aggregatable": true,
            "non_searchable_indices": ["t"]
            "indices": ["t", "s"]
         },
         "long": {
            "searchable": true,
            "aggregatable": false,
            "non_aggregatable_indices": ["v"]
            "indices": ["v", "w"]
         }
      }
   }
}
````

In this example `field1` have the same type `text` across the requested indices `t`, `s`, `v`, `w`.
Conversely `field2` is defined with two conflicting types `keyword` and `long`.
Note that `_field_caps` does not treat this case as an error but rather return the list of unique types seen for this field.
2017-03-31 15:34:46 +02:00
Glen Smith c62d4b7b0f Clarify preference docs
This commit clarifies the preference docs regarding the explanation of
how operations are routed by default. In particular, the previous use of
"shard replicas" was confusing as it could imply an operation would only
be routed to replicas by default.

Relates #23794
2017-03-29 12:55:47 -04:00
Christoph Büscher 96fc3aaf6f Merge branch 'master' into feature/rank-eval 2017-03-23 19:55:47 +01:00
Igor Motov f927a2708d Make it possible to validate a query on all shards instead of a single random shard (#23697)
This is especially useful when we rewrite the query because the result of the rewrite can be very different on different shards. See #18254 for example.
2017-03-22 17:39:21 -04:00
Jim Ferenczi b8c352fc3f Add support for fragment_length in the unified highlighter (#23431)
* Add support for fragment_length in the unified highlighter

This commit introduce a new break iterator (a BoundedBreakIterator) designed for the unified highlighter
 that is able to limit the size of fragments produced by generic break iterator like `sentence`.
The `unified` highlighter now supports `boundary_scanner` which can `words` or `sentence`.
The `sentence` mode will use the bounded break iterator in order to limit the size of the sentence to `fragment_length`.
When sentences bigger than `fragment_length` are produced, this mode will break the sentence at the next word boundary **after**
 `fragment_length` is reached.
2017-03-17 18:10:13 +01:00
Jack Conradson 8e04561c0d Change params._source to params['_source'] in example. 2017-03-15 17:29:31 -07:00
Jack Conradson 4c11ebc8b9 Fix example in documentation for Painless using _source. (#21322) 2017-03-15 17:18:34 -07:00
Christoph Büscher cf35545e2d Merge branch 'master' into feature/rank-eval 2017-03-13 17:36:13 -07:00
NFM f8fa5c96aa Fix indentation in sort docs
This commit fixes the indentation in an example query in the sort docs.

Relates #23561
2017-03-12 17:08:06 -07:00
Christoph Büscher 1f4c4d99b9 Merge branch 'master' into feature/rank-eval 2017-02-27 11:25:17 +01:00
Shai Erera eeac6d27f2 Add BreakIteratorBoundaryScanner support for FVH (#23248)
This commit adds a boundary_scanner property to the search highlight
request so the user can specify different boundary scanners:

* `chars` (default,  current behavior)
* `word` Use a WordBreakIterator
* `sentence` Use a SentenceBreakIterator

This commit also adds "boundary_scanner_locale" to define which locale
should be used when scanning the text.
2017-02-23 23:32:22 +01:00
Christoph Büscher cfa52f8b9a Merge branch 'master' into feature/rank-eval 2017-02-16 10:39:07 +01:00
Adrien Grand 8d6a41f671 Nested queries should avoid adding unnecessary filters when possible. (#23079)
When nested objects are present in the mappings, many queries get deoptimized
due to the need to exclude documents that are not in the right space. For
instance, a filter is applied to all queries that prevents them from matching
non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all
child queries of `nested` queries in order to make sure that the child query
only matches child documents (`_type:__nested_path`), which is required by
`ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested`
queries).

These additional filters slow down `nested` queries. In 1.7-, the cost was
somehow amortized by the fact that we cached filters very aggressively. However,
this has proven to be a significant source of slow downs since 2.0 for users
of `nested` mappings and queries, see #20797.

This change makes the filtering a bit smarter. For instance if the query is a
`match_all` query, then we need to exclude nested docs. However, if the query
is `foo: bar` then it may only match root documents since `foo` is a top-level
field, so no additional filtering is required.

Another improvement is to use a `FILTER` clause on all types rather than a
`MUST_NOT` clause on all nested paths when possible since `FILTER` clauses
are more efficient.

Here are some examples of queries and how they get rewritten:

```
"match_all": {}
```

This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and
`ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})`
with this change. The automaton is the complement of `_type:__*` so it matches
the same documents, but is faster since it is now a positive clause. Simplistic
performance testing on a 10M index where each root document has 5 nested
documents on average gave a latency of 420ms on master and 90ms with this change
applied.

```
"term": {
  "foo": {
    "value": "0"
  }
}
```

This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on
master and `foo:0` with this change: we do not need to filter nested docs out
since the query cannot match nested docs. While doing performance testing in
the same conditions as above, response times went from 250ms to 50ms.

```
"nested": {
  "path": "nested",
  "query": {
    "term": {
      "nested.foo": {
        "value": "0"
      }
    }
  }
}
```

This query is rewritten to
`+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0`
on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The
top-level filter (`-_type:__*`) could be removed since `nested` queries only
match documents of the parent space, as well as the child filter
(`#_type:__nested`) since the child query may only match nested docs since the
`nested` object has both `include_in_parent` and `include_in_root` set to
`false`. While doing performance testing in the same conditions as above,
response times went from 850ms to 270ms.
2017-02-14 16:05:19 +01:00
Tanguy Leroux e2e5937455 Use `typed_keys` parameter to prefix suggester names by type in search responses (#23080)
This pull request reuses the typed_keys parameter added in #22965, but this time it applies it to suggesters. When set to true, the suggester names in the search response will be prefixed with a prefix that reflects their type.
2017-02-10 10:53:38 +01:00
Tanguy Leroux 63ea6f7168 [Docs] Remove unnecessary // TEST[continued] in search-template doc
It has been explained in e39b96f257
2017-02-10 10:08:24 +01:00
Jim Ferenczi 94087b3274 Removes ExpandCollapseSearchResponseListener, search response listeners and blocking calls
This changes removes the SearchResponseListener that was used by the ExpandCollapseSearchResponseListener to expand collapsed hits.
The removal of SearchResponseListener is not a breaking change because it was never released.
This change also replace the blocking call in ExpandCollapseSearchResponseListener by a single asynchronous multi search request. The parallelism of the expand request can be set via CollapseBuilder#max_concurrent_group_searches

Closes #23048
2017-02-09 18:06:10 +01:00
Tanguy Leroux 832952cb29 [Docs] Fix consoleify search-template.asciidoc
It does not reproduce well, hopefully this will fix the failure on DELETE _search/template/<templatename>.
2017-02-08 21:23:38 +01:00
Jay Modi 7f3769c745 Remove ldjson support and document ndjson for bulk/msearch (#23049)
This commit removes support for the `application/x-ldjson` Content-Type header as this was only used in the first draft
of the spec and had very little uptake. Additionally, the docs for bulk and msearch have been updated to specifically
call out ndjson and mention that the newline character may be preceded by a carriage return.

Finally, the bulk request handling of the carriage return has been improved to remove this character from the source.

Closes #23025
2017-02-08 11:55:50 -05:00
Tanguy Leroux 477d1aa8bf [Docs] Consoleify multi-search and search-template docs (#23047)
Relates #23001
2017-02-08 17:05:22 +01:00
Nik Everett 0e98c9107a Docs: CONSOLEify some more docs
These need to be CONSOLEified *now* because we're starting to
require Content-Type headers and they didn't have any.

* cluster/reroute: Marked as CONSOLE but skipped because the docs
build runs with a single node.
* docs/bulk: Marked as NOTCONSOLE because the snippets describe
either examples or `curl` commands. Fixed the `curl` command to
include the `Content-Type` header.
* query-dsl/terms-query: Marked as CONSOLE.
* search/request/rescore: Marked as CONSOLE. Fixed deprecated
syntax.

Relates #23001
Relates #18160
2017-02-07 16:49:01 -05:00
Jim Ferenczi bbf62e3472 CONSOLify indices/analyze.asciidoc and search/field-stats.asciidoc
Relates #23001
2017-02-07 20:15:09 +01:00
Clinton Gormley e181a020a9 Replaced absolute URLs in docs with attributes 2017-02-04 12:05:03 +01:00
Christoph Büscher 4cb8d9d08c Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
        docs/reference/search.asciidoc
2017-02-03 17:27:20 +01:00
Clinton Gormley 8ace37e214 Fix asciidoc in stored fields 2017-02-03 10:18:01 +01:00
Nicholas Knize b41d5747f0 Reduce GeoDistance insanity
GeoDistance query, sort, and scripts make use of a crazy GeoDistance enum for handling 4 different ways of computing geo distance: SLOPPY_ARC, ARC, FACTOR, and PLANE. Only two of these are necessary: ARC, PLANE. This commit removes SLOPPY_ARC, and FACTOR and cleans up the way Geo distance is computed.
2017-02-02 12:39:42 -06:00
Jim Ferenczi f6d38d480a Integrate UnifiedHighlighter (#21621)
* Integrate UnifiedHighlighter

This change integrates the Lucene highlighter called "unified" in the list of supported highlighters for ES.
This highlighter can extract offsets from either postings, term vectors, or via re-analyzing text.
The best strategy is picked automatically at query time and depends on the field and the query to highlight.
2017-01-31 19:06:03 +01:00
Jim Ferenczi e7e871acdd Fix link to keyword and numeric type 2017-01-30 13:57:28 +01:00
Clinton Gormley 938f5194ef Include field-collapsing docs in request-body search 2017-01-30 11:47:12 +01:00
Jim Ferenczi e48bc2eed7 Add field collapsing for search request (#22337)
* Add top hits collapsing to search request

The field collapsing is done with a custom top docs collector that "collapse" search hits with same field value.
The distributed aspect is resolve using the two passes that the regular search uses. The first pass "collapse" the top hits, then the coordinating node merge/collapse the top hits from each shard.

```
GET _search
{
   "collapse": {
      "field": "category",
   }
}
```

This change also adds an ExpandCollapseSearchResponseListener that intercepts the search response and expands collapsed hits using the CollapseBuilder#innerHit} options.
The retrieval of each inner_hits is done by sending a query to all shards filtered by the collapse key.

```
GET _search
{
   "collapse": {
      "field": "category",
      "inner_hits": {
	"size": 2
      }
   }
}
```
2017-01-23 16:33:51 +01:00
Christoph Büscher 9ed867ea83 [DOCS] Fix inconsistent formatting for fieldnames in profile.asciidoc 2017-01-18 10:41:22 +01:00
Clinton Gormley 401438819e Docs: Fix the first highlighting example to work
Closes #22642
2017-01-17 12:20:03 +01:00
Christoph Büscher 2791c69960 Update profile.asciidoc
Making the "Human readable output" section a note instead of an own section.
2017-01-16 16:19:07 +01:00
Christoph Büscher 49a49da3f5 [Docs] Fix section title in profile.asciidoc 2017-01-16 14:53:06 +01:00
Christoph Büscher 59a48ffc41 ProfileResult and CollectorResult should print machine readable timing information (#22561)
Currently both ProfileResult and CollectorResult print the time field in a human readable string format
 (e.g. "time": "55.20315000ms"). When trying to parse this back to a long value, for example to use in 
the planned high level java rest client, we can lose precision because of conversion and rounding issues. 
This change adds a new additional field (`time_in_nanos`) to the profile response to be able to get the 
original time value in nanoseconds back. 

The old `time` field is only printed when the `?`human=true` flag in the url is set. This follow the behaviour for 
all other stats-related apis. Also the format of the `time` field is slightly changed. Instead of always formatting 
the output as a 10-digit ms value, by using the `XContentBuilder#timeValueField()` method we now print 
the largest time unit present is used (e.g. "s", "ms", "micros").
2017-01-16 14:27:55 +01:00
maciejkula b4c8c21553 State default sort order on missing values
Closes #19099
2017-01-13 17:05:13 +01:00
Lee Hinman 2db01b6127 Merge remote-tracking branch 'dakrone/disable-all-by-default' 2017-01-12 10:17:51 -07:00
Tanguy Leroux df703dce0a [DOC] Document {{url}} mustache function (#22549)
This function introduced in #20838 wasn't documented at all.

Related to #22459
2017-01-12 14:57:03 +01:00
Lee Hinman 7a18bb50fc Disable _all by default
This change disables the _all meta field by default.

Now that we have the "all-fields" method of query execution, we can save both
indexing time and disk space by disabling it.

_all can no longer be configured for indices created after 6.0.

Relates to #20925 and #21341
Resolves #19784
2017-01-11 16:47:13 -07:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
Isabel Drost-Fromm 46c30e6bc3 Make maximum number of parallel search requests configurable. (#22192)
Problem: So far all rank eval requests are being executed in parallel. If there
are more than the search thread pool can handle, or if there are other search
requests executed in parallel rank eval can fail.

Solution: Make number of max_concurrent_searches configurable.

Name of configuration parameter is analogous to msearch. Default
max_concurrent_searches set to 10: Rank_eval isn't particularly time critical so
trying to avoid being more clever than probably needed here. Can set this value
through the API to a higher value anytime.

Fixes #21403
2016-12-19 13:05:49 +01:00
Isabel Drost-Fromm bdc32be8b7 Support specifying multiple templates (#22139)
Problem: We introduced the ability to shorten the rank eval request by using a
template in #20231. When playing with the API it turned out that there might be
use cases where - e.g. due to various heuristics - folks might want to translate
the original user query into more than just one type of Elasticsearch query.

Solution: Give each template an id that can later be referenced in the
actual requests.

Closes #21257
2016-12-19 12:49:15 +01:00
Isabel Drost-Fromm b1e0d698ac Merge branch 'master' into feature/rank-eval 2016-12-19 10:16:16 +01:00
Masaru Hasegawa a0185c83a7 Merge pull request #21393 from masaruh/alias_boost
Resolve index names in indices_boost
2016-12-16 15:07:51 +09:00
Isabel Drost-Fromm 5618d6ca49 Merge branch 'master' into feature/rank-eval 2016-12-15 10:29:26 +01:00
Areek Zillur cdd5fbe3a1 Deprecate _suggest endpoint in favour of _search (#20305)
* Replace _suggest endpoint to _search in docs

In 5.0, the _suggest endpoint is just sugar for _search
with suggestions specified. Users should move away from
using the _suggest endpoint, as it is marked as deprecated in 5.x and
will be removed in 6.0

* update docs to use _search endpoint instead of _suggest

* Add deprecation logging to RestSuggestAction

* Use search endpoint instead of suggest endpoint in rest tests
2016-12-14 21:49:53 -05:00
Isabel Drost-Fromm eeff9fb100 Adjust docs to reflect removed template endpoint
The dedicated template endpoint for rank_eval queries was removed, reflect this in the docs as well.
2016-12-13 13:59:11 +01:00
Masaru Hasegawa 3df2a086d4 Resolve index names in indices_boost
This change allows specifying alias/wildcard expression in indices_boost.
And added another format for specifying indices_boost. It accepts array of index name and boost pair.
If an index is included in multiple aliases/wildcard expressions, the first match will be used.
With new format, old format is marked as deprecated.

Closes #4756
2016-12-11 21:41:49 +09:00
Isabel Drost-Fromm 5c6cdb90ad Merge branch 'master' into feature/rank-eval 2016-12-07 10:59:05 +01:00
Jim Ferenczi b42ca6bcc9 Include unindexed field in FieldStats response (#21821)
* Include unindexed field in FieldStats response

This change adds non-searchable fields to the FieldStats response. These fields do not have min/max informations but they can be aggregatable. Fields that are only stored in _source (store:no, index:no, doc_values:no) will still be missing since they do not have any useful information to show. Indices and clients must be at least on V_5_2_0 to see this change.
2016-12-06 13:32:57 +01:00
Isabel Drost-Fromm 2fecd8e394 Merge branch 'master' into feature/rank-eval 2016-11-30 11:17:56 +01:00
Jim Ferenczi d791ddf704 Upgrade to lucene-6.4.0-snapshot-ec38570 (#21853)
Set lucene version to 6.4.0-snapshot-ec38570 and update all the sha1s/license
Fix invalid combo after upgrade in query_string query. split_on_whitespace=false is disallowed if auto_generate_phrase_queries=true
Adapt the expectations of some tests to the new format of the Lucene explain output
2016-11-29 18:40:31 +01:00
Isabel Drost-Fromm 3e0aedd7da Merge branch 'master' into feature/rank-eval 2016-11-29 11:09:22 +01:00
Adrin Jalali eec05ec208 then -> than (#21829) 2016-11-28 17:04:56 +01:00
Isabel Drost-Fromm 2b24091361 Merge branch 'master' into feature/rank-eval 2016-11-28 11:45:29 +01:00
Adrin Jalali 953928b2c5 typo fix (it self -> itself) (#21781)
* typo fix.

* apply "stored field value"

* replaced "whereas" with "on the contrary"
2016-11-24 17:11:43 +01:00
Adrin Jalali 0871073f9b clarification on geo distance sorting (#21779)
* clarification on geo distance sorting

* applying the suggested change
2016-11-24 16:06:10 +01:00
Christoph Büscher e1fe0dc462 Merge branch 'master' into feature/rank-eval 2016-11-24 08:57:26 +01:00
Luca Cavanna db5a72774b Add indices and filter information to search shards api output (#21738)
Add indices and filter information to search shards api output

The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.

This is relevant since we have moved parsing of alias filters to the coordinating node.

Relates to #20916
2016-11-22 23:00:25 +01:00
Luca Cavanna db8b2dceea Remove ignored type parameter in search_shards api (#21688)
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.

This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.

The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
2016-11-22 17:22:33 +01:00
Christoph Büscher 1d3e58ab9f Merge branch 'master' into feature/rank-eval 2016-11-22 11:31:46 +01:00
Lee Hinman 11da09e9bc Allow overriding all-field leniency when `lenient` option is specified
As part of #20925 and #21341 we added an "all-fields" mode to the
`query_string` and `simple_query_string`. This would expand the query to
all fields and automatically set `lenient` to true.

However, we should still allow a user to override the `lenient` flag to
whichever value they desire, should they add it in the request. This
commit does that.
2016-11-21 21:32:25 -07:00
Luca Wintergerst 277f4b8d24 fix two errors in suggester docs
The first changed referred to an example of the 2.4 documentation. I removed the no longer relevant parts. We should consider adding a little more here. 

The second change was just then->than in the suggest_mode popular section
2016-11-18 12:05:49 +01:00
Christoph Büscher 1f5f2e312a Use the twitter index in documentation snippets 2016-11-17 17:03:46 +01:00
Isabel Drost-Fromm e5a08937e3 Fix failing doc test, add missing CONSOLE 2016-11-17 15:01:04 +01:00
Isabel Drost-Fromm 19bb0a928d Reference documentation for rank evaluation API (#21427)
* Reference documentation for rank evaluation API

This adds a first page of reference documentation to the current state of the
rank evaluation API.

Closes to #21402

* Add default values for precision metric.

Add information on default relevant_rating_threshold and ignore_unlabeled
settings.

Relates to #21304

* Move under search request docs, fix formatting

Also removes some detail where it seemed unneeded for reference docs
2016-11-17 10:27:57 +01:00
Nik Everett 593d47efe2 Make it clear _suggest doesn't support source filtering (#21268)
We plan to deprecate `_suggest` during 5.0 so it isn't worth fixing
it to support the `_source` parameter for `_source` filtering. But we
should fix the docs so they are accurate.

Since this removes the last non-`// CONSOLE` line in
`completion-suggest.asciidoc` this also removes it from the list of
files that have non-`// CONSOLE` docs.

Closes #20482
2016-11-06 20:15:45 -05:00
Adrien Grand 52de0645fb Remove `lowercase_expanded_terms` and `locale` from query-parser options. (#20208)
Lucene 6.2 introduces the new `Analyzer.normalize` API, which allows to apply
only character-level normalization such as lowercasing or accent folding, which
is exactly what is needed to process queries that operate on partial terms such
as `prefix`, `wildcard` or `fuzzy` queries. As a consequence, the
`lowercase_expanded_terms` option is not necessary anymore. Furthermore, the
`locale` option was only needed in order to know how to perform the lowercasing,
so this one can be removed as well.

Closes #9978
2016-11-02 14:25:08 +01:00