Commit Graph

773 Commits

Author SHA1 Message Date
Adrien Grand 5c237fe834 Add new option `min_doc_count` to terms and histogram aggregations.
`min_doc_count` is the minimum number of hits that a term or histogram key
should match in order to appear in the response.

`min_doc_count=0` replaces `compute_empty_buckets` for histograms and will
behave exactly like facets' `all_terms=true` for terms aggregations.

Close #4662
2014-01-13 10:09:38 +01:00
Martijn van Groningen 943b62634c Replaced the multi-field type in favour for the multi fields option that can be set on any core field.
When upgrading to ES 1.0 the existing mappings with a multi-field type automatically get replaced to a core field with the new `fields` option.

If a `multi_field` type-ed field doesn't have a main / default field, a default field will be chosen for the multi fields syntax. The new main field type
will be equal to the first `multi_field` fields' field or type string if no fields have been configured for the `multi_field` field and in both cases
the default index will not be indexed (`index=no` is set on the default field).

If a `multi_field` typed field has a default field, that field will replace the `multi_field` typed field.

Closes to #4521
2014-01-13 09:21:53 +01:00
Florian Schilling 464037e0c1 Geo clean Up
============
The default unit for measuring distances is *MILES* in most cases. This commit moves ES
over to the *International System of Units* and make it work on a default which relates
to *METERS* . Also the current structures of the `GeoBoundingBox Filter` changed in
order to define the *Bounding* by setting abitrary corners.

Distances
---------
Since the default unit for measuring distances has changed to a default unit
`DistanceUnit.DEFAULT` relating to *meters*, the **REST API** has changed at the
following places:

  * `ScriptDocValues.factorDistance()` returns *meters* instead of *miles*
  * `ScriptDocValues.factorDistanceWithDefault()` returns *meters* instead of *miles*
  * `ScriptDocValues.arcDistance()` returns *meters* instead of *miles*
        one might use `ScriptDocValues.arcDistanceInMiles()`
  * `ScriptDocValues.arcDistanceWithDefault()` returns *meters* instead of *miles*
  * `ScriptDocValues.distance()` returns *meters* instead of *miles*
        one might use `ScriptDocValues.distanceInMiles()`
  * `ScriptDocValues.distanceWithDefault()` returns *meters* instead of *miles*
        one might use `ScriptDocValues.distanceInMilesWithDefault()`
  * `GeoDistanceFilter` default unit changes from *kilometers* to *meters*
  * `GeoDistanceRangeFilter` default unit changes from *miles* to *meters*
  * `GeoDistanceFacet` default unit changes from *miles* to *meters*

Geo Bounding Box Filter
-----------------------
The naming of the GeoBoundingBoxFilter properties allows to set arbitrary corners
(see #4084) namely `top_right`, `top_left`, `bottom_right` and `bottom_left`. This
change also includes the fields `topRight` and `bottomLeft` Also it is be possible to
set the single values by using just `top`, `bottom`, `left` and `right` parameters.

Closes #4515, #4084
2014-01-11 21:30:29 +09:00
Simon Willnauer bc5a9ca342 Rename edit_distance/min_similarity to fuzziness
A lot of different API's currently use different names for the
same logical parameter. Since lucene moved away from the notion
of a `similarity` and now uses an `fuzziness` we should generalize
this and encapsulate the generation, parsing and creation of these
settings across all queries.

This commit adds a new `Fuzziness` class that handles the renaming
and generalization in a backwards compatible manner.

This commit also added a ParseField class to better support deprecated
Query DSL parameters

The ParseField class allows specifying parameger that have been deprecated.
Those parameters can be more easily tracked and removed in future version.
This also allows to run queries in `strict` mode per index to throw
exceptions if a query is executed with deprected keys.

Closes #4082
2014-01-09 15:14:51 +01:00
Martijn van Groningen 7e341cefd0 Change the `sort` boolean option in percolate api to the sort dsl available in search api.
Closes #4625
2014-01-09 09:58:34 +01:00
Clinton Gormley 2e4b70d40f [DOCS] Fixed duplicate ID in highlighting 2014-01-09 00:37:18 +01:00
Nik Everett bbf0ec52de Add warning phrase suggester's max_errors
large number can badly impact performance.
2014-01-08 23:06:41 +01:00
Nik Everett 8bd9e34e39 Stop FVH from throwing away some query boosts
The FVH was throwing away some boosts on queries stopping a number of
ways to boost phrase matches to the top of the list of fragments from
working.

The plain highlighter also doesn't work for this but that is because it
doesn't support the concept of the same term having a different score at
different positions.

Also update documentation claiming that FHV is nicer for weighing terms
found by query combinations.

Closes #4351
2014-01-08 11:51:48 +01:00
Nik Everett 522d620eb6 Use FHV's phraseLimit
This prevents poisoning the FVH with documents that contain TONS of matches
which take tons of memory and time to highlight.

Closes #4645
2014-01-08 11:27:58 +01:00
markharwood 602de04692 A GeoHashGrid aggregation that buckets GeoPoints into cells whose dimensions are determined by a choice of GeoHash resolution.
Added a long-based representation of GeoHashes to GeoHashUtils for fast evaluation in aggregations.
The new BucketUtils provides a common heuristic for determining the number of results to obtain from each shard in "top N" type requests.
2014-01-07 18:03:33 +00:00
Lee Hinman 2cb40fcb17 Rename "exists" to "found" in TermVector and Get responses
- Adds the "created" field to the index action response
- Reverses Delete class' notFound to Found to avoid double negative
2014-01-07 09:47:07 -07:00
Simon Willnauer fa16969360 Cleanup comments and class names s/ElasticSearch/Elasticsearch
* Clean up s/ElasticSearch/Elasticsearch on docs/*
 * Clean up s/ElasticSearch/Elasticsearch on src/* bin/* & pom.xml
 * Clean up s/ElasticSearch/Elasticsearch on NOTICE.txt and README.textile

Closes #4634
2014-01-07 11:21:51 +01:00
Martijn van Groningen 32c5471d33 Rename `score` to `track_scores` in percolate api.
Closes #4624
2014-01-06 14:57:39 +01:00
Martijn van Groningen f1bf585089 The `fields` option should always return an array for json document fields and single valued field for metadata fields.
Also the `fields` option can only be used to fetch leaf fields, trying to do fetch object fields will return in a client error.

Closes #4542
2014-01-03 17:29:12 +01:00
Martijn van Groningen f4bf0d5112 Replaced `ignore_indices` with `ignore_unavailable`, `expand_wildcards` and `allow_no_indices`.
* `ignore_unavailable` - Controls whether to ignore if any specified indices are unavailable, this includes indices that don't exist or closed indices. Either `true` or `false` can be specified.
* `allow_no_indices` - Controls whether to fail if a wildcard indices expressions results into no concrete indices. Either `true` or `false` can be specified. For example if the wildcard expression `foo*` is specified and no indices are available that start with `foo` then depending on this setting the request will fail. This setting is also applicable when `_all`, `*` or no index has been specified.
* `expand_wildcards` - Controls to what kind of concrete indices wildcard indices expression expand to. If `open` is specified then the wildcard expression if expanded to only open indices and if `closed` is specified then the wildcard expression if expanded only to closed indices. Also both values (`open,closed`) can be specified to expand to all indices.

Closes to #4436
2014-01-02 12:19:45 +01:00
Florian Schilling bc452dff84 * setup accurate GeoDistance Function
* adapt tests
* introduced default GeoDistance function
* Updated docs

closes #4498
2013-12-27 19:15:19 +09:00
Chris Simpson 4f8c916eed [Docs] Fix Typo
Fixes small typo in the geo_distance aggregation docs.
2013-12-18 11:21:21 +01:00
Clinton Gormley 34b9b16233 [DOCS] Fixed some bad link refs 2013-12-16 18:07:33 +01:00
Martijn van Groningen 23d2b1ea7b Renamed top level `filter` to `post_filter`.
Closes #4119
2013-12-16 17:10:14 +01:00
Adrien Grand 36bd9cc432 Aggregations: Ordinals-based string bucketing support.
When the ValuesSource has ordinals, terms ordinals are used as a cache key to
bucket ordinals. This can make terms aggregations on String terms significantly
faster.

Close #4350
2013-12-13 15:34:02 +01:00
Martijn van Groningen 10e2528cce Added the `force_source` option to highlighting that enforces to use of the _source even if there are stored fields.
The percolator uses this option to deal with the fact that the MemoryIndex doesn't support stored fields,
this is possible b/c the _source of the document being percolated is always present.

Closes #4348
2013-12-13 13:39:53 +01:00
Martijn van Groningen ebf6519965 Added aggs option to percolate api documentation. 2013-12-10 14:09:37 +01:00
Martijn van Groningen 8c1de501e7 Update percolator highlighting docs. 2013-12-07 16:40:49 -05:00
Adrien Grand 32eb5ffa92 [Docs] Document which encoding should be used in order to make sense of the offsets returned by the term vectors API.
Close #4363
2013-12-06 22:39:08 +01:00
Markus Fischer 2da0611dfb [DOCS] Completion suggest: Clarify de-duplication, optimize/merge
This contribution is based on the feedback given in issue #4254 and
issue #4255, and should clear things up, when suggestions are being
removed and not displayed anymore after deletion of data.
2013-12-05 11:10:56 +01:00
Nik Everett 8e34057bc0 Add support for combining fields to the FVH
The Fast Vector Highlighter can combine matches on multiple fields to
highlight a single field using `matched_fields`.  This is most
intuitive for multifields that analyze the same string in different
ways.  Example:
{
    "query": {
        "query_string": {
            "query": "content.plain:running scissors",
            "fields": ["content"]
        }
    },
    "highlight": {
        "order": "score",
        "fields": {
            "content": {
                "matched_fields": ["content", "content.plain"],
                "type" : "fvh"
            }
        }
    }
}

Closes #3750
2013-12-03 11:10:01 +01:00
Clinton Gormley d9a480c97a [DOCS] Typos in aggregations 2013-12-02 15:14:25 +01:00
Conrad Pankoff 87246af256 [DOCS] Fixed typos and corrected grammar 2013-12-02 10:08:26 +01:00
uboness cdc7dfbb2c Changed the "script_lang" parameter to "lang" in all value source based aggs - to be consistent with all other script based APIs. 2013-12-02 02:01:03 +01:00
Clinton Gormley bc393b6d79 Changed the minScore comparator from > to >=
Closes #4303
2013-11-29 20:29:20 +01:00
uboness 0d6a35b9a7 - Added support for term filtering based on include/exclude regex on the terms agg
- Added javadoc to the TermsBuilder

Closes #4267
2013-11-29 13:46:48 +01:00
uboness afb0d119e4 - Added docs for the value_count aggregation
- Fixed typos in the terms facets docs
- Fixed aggregation docs layout
- Added docs for shard_size in term aggregation
2013-11-29 12:35:42 +01:00
Clinton Gormley cdc1935b6e [DOCS] Documented rest.action.multi.allow_explicit_index 2013-11-27 17:33:09 +01:00
Boaz Leskes c63d8c4fb5 [Docs] Added _source filtering to documentation
Relates to #3301
2013-11-26 19:16:24 +01:00
Britta Weber dbef64009f [DOC] add doc for multi term vector api
closes #3998
2013-11-26 17:03:14 +01:00
Alexander Reelsen bf74f49fdd Updated Analyzing/Fuzzysuggester from lucene trunk
* Minor alignments (like setter to ctor)
* FuzzySuggester has a unicode aware flag, which is not exposed in the fuzzy completion request parameters
* Made XAnalyzingSuggester flags (PAYLOAD_SEP, END_BYTE, SEP_LABEL) to be written into the postings format, so we can retain backwards compatibility
* The above change also implies, that these flags can be set per instantiated XAnalyzingSuggester
* CompletionPostingsFormatTest now uses a randomProvider for writing data to check for bwc
2013-11-26 12:52:06 +01:00
uboness c7f6c5266d initial commit of the aggregations module
Closes #3300
2013-11-24 03:13:08 -08:00
Clinton Gormley 3465e69e83 [DOCS] Changed all store:yes/no to store:true/false
which is how this setting is stored internally
2013-11-07 16:57:18 +01:00
Simon Willnauer 77bc5d5ecf release [1.0.0.Beta1] 2013-11-06 15:32:43 +01:00
Shay Banon 7c32269f4f Dist. Percolation: Use .percolator instead of _percolator for type name
Use .percolator as the internal (hidden) type name for percolators within the index. Seems nicer name to represent "hidden" types within an index.
closes #4090
2013-11-05 20:02:59 +01:00
Martijn van Groningen 30ab6f841d [DOCS] Fixed percolate docs errors 2013-11-01 11:44:07 +01:00
Clinton Gormley 8b2efd4849 [DOCS] Added a version flag to percolation 2013-10-30 13:59:03 +01:00
Luca Cavanna 48ac9747a8 Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.

Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same

The lucene postings highlighter api is  quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.

Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.

Closes #3704
2013-10-24 23:38:00 +02:00
Luca Cavanna e981e411d7 [DOCS] rephrased docs for highlight no_match_size parameter
(removed 0.90.6 coming tag as it's needed only in 0.90 branch)
2013-10-24 14:38:32 +02:00
Nik Everett 14a709f563 Highlighting can return excerpt with no highlights
You can configure the highlighting api to return an excerpt of a field
even if there wasn't a match on the field.

The FVH makes excerpts from the beginning of the string to the first
boundary character after the requested length or the boundary_max_scan,
whichever comes first.  The Plain highlighter makes excerpts from the
beginning of the string to the end of the last token before the requested
length.

Closes #1171
2013-10-24 14:38:32 +02:00
Markus Fischer 782d315da3 Fix markup 2013-10-21 16:11:09 +02:00
Clinton Gormley b2d82d7e75 [DOCS] Reorganised the highlight_query docs and added a version flag 2013-10-18 18:03:31 +02:00
Nik Everett 60550e4cc2 phrase_len is not called phrase_length 2013-10-18 09:29:53 -04:00
Martijn van Groningen 1d0841e2b8 Added initial documentation for the redesigned percolator. 2013-10-16 14:12:19 +02:00
Clinton Gormley 9a062e465c [DOCS] Reorganised common API conventions 2013-10-13 16:46:56 +02:00
Clinton Gormley ba1b4886e3 [DOCS] Moved "named filters/queries" up one level 2013-10-10 11:23:08 +02:00
Clinton Gormley 264a00a40f [DOCS] Added pages explaining lucene query parser syntax and regular expression syntax 2013-10-07 14:42:49 +02:00
Clinton Gormley 7a53d41446 [DOCS] Changed capitalization of operator in rescore query 2013-10-05 17:18:15 +02:00
uboness f3c6108b71 introduced support for "shard_size" for terms & terms_stats facets. The "shard_size" is the number of term entries each shard will send back to the coordinating node. "shard_size" > "size" will increase the accuracy (both in terms of the counts associated with each term and the terms that will actually be returned the user) - of course, the higher "shard_size" is, the more expensive the processing becomes as bigger queues are maintained on a shard level and larger lists are streamed back from the shards.
closes #3821
2013-10-02 22:02:00 +02:00
Nik Everett 6b000d8c6d Support specifing score query on highlight.
This is useful if you want to highlight terms not in the search query or
you want sort highlighted snippets based on another query.

Closes #3630
2013-10-02 15:46:24 -04:00
Lee Hinman ba40aa374e Uniquify anchor links to fix asciidoc/docbook generation 2013-09-30 15:32:00 -06:00
Lee Hinman 0442b737be Add more anchor links to documentation
Related to #3679
2013-09-30 13:13:16 -06:00
Adrien Grand 90524d7ad2 Fix formatting of the documentation.
Remaining '@'s have been replaced with '`'s.
2013-09-18 12:35:44 +02:00
David Pilato 1e3ffa0df7 Add distance supported units 2013-09-17 14:21:45 +02:00
Clinton Gormley 85bba668f7 [DOCS] Tidied up various doc formatting errors 2013-09-16 16:13:01 +02:00
Martijn van Groningen f6f4b5014f Added docs for named queries.
Relates to #3581
2013-09-16 11:17:01 +02:00
David Pilato 169cd007b5 Fix typo
Thanks to @ybonnel for finding it ;-)
2013-09-12 11:00:59 +02:00
Martijn van Groningen 8ddb809f98 If all scroll ids should be removed then the `_all` value should be used instead of not specifying any scroll ids. 2013-09-12 10:41:38 +02:00
Martijn van Groningen 0efa78710b Added clear scroll api.
The clear scroll api allows clear all resources associated with a `scroll_id` by deleting the `scroll_id` and its associated SearchContext.

Closes #3657
2013-09-10 21:17:34 +02:00
Clinton Gormley 6d667e5d41 [DOCS] Missing sort values now works for all field types 2013-09-04 23:20:55 +02:00
Clinton Gormley e1c6f45ff0 [DOCS] Added clarification about global scope in facets 2013-09-04 23:20:55 +02:00
Clinton Gormley 08f8e77b8f [DOCS] Added fuzzy options to completion suggester 2013-09-04 23:20:55 +02:00
Clinton Gormley 9f5d0b6e89 [DOCS] Added a few clarifications to the docs from the issues list 2013-09-04 23:20:55 +02:00
Clinton Gormley 5b60506b2e [DOCS] Added highlighting to the phrase suggester 2013-09-04 23:20:54 +02:00
Clinton Gormley 53ad7330fc [DOCS] Added docs for term vectors 2013-09-04 23:20:54 +02:00
Clinton Gormley 393c28bee4 [DOCS] Removed outdated new/deprecated version notices 2013-09-03 21:28:31 +02:00
Simon Willnauer eb2fed85f1 Add 'min_input_len' to completion suggester
Restrict the size of the input length to a reasonable size otherwise very
long strings can cause StackOverflowExceptions deep down in lucene land.
Yet, this is simply a saftly limit set to `50` UTF-16 codepoints by default.
This limit is only present at index time and not at query time. If prefix
completions > 50 UTF-16 codepoints are expected / desired this limit should be raised.
Critical string sizes are beyone the 1k UTF-16 Codepoints limit.

Closes #3596
2013-09-03 10:26:37 +02:00
Clinton Gormley 822043347e Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00