2013-08-28 19:24:34 -04:00
|
|
|
[[search-request-highlighting]]
|
|
|
|
=== Highlighting
|
|
|
|
|
|
|
|
Allows to highlight search results on one or more fields. The
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
implementation uses either the lucene `highlighter`, `fast-vector-highlighter`
|
|
|
|
or `postings-highlighter`. The following is an example of the search request
|
|
|
|
body:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"fields" : {
|
|
|
|
"content" : {}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
In the above case, the `content` field will be highlighted for each
|
|
|
|
search hit (there will be another element in each search hit, called
|
|
|
|
`highlight`, which includes the highlighted fields and the highlighted
|
|
|
|
fragments).
|
|
|
|
|
|
|
|
In order to perform highlighting, the actual content of the field is
|
|
|
|
required. If the field in question is stored (has `store` set to `yes`
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
in the mapping) it will be used, otherwise, the actual `_source` will
|
2013-08-28 19:24:34 -04:00
|
|
|
be loaded and the relevant field will be extracted from it.
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
The field name supports wildcard notation. For example, using `comment_*`
|
|
|
|
will cause all fields that match the expression to be highlighted.
|
|
|
|
|
|
|
|
==== Postings highlighter
|
|
|
|
|
|
|
|
If `index_options` is set to `offsets` in the mapping the postings highlighter
|
|
|
|
will be used instead of the plain highlighter. The postings highlighter:
|
|
|
|
|
|
|
|
* Is faster since it doesn't require to reanalyze the text to be highlighted:
|
|
|
|
the larger the documents the better the performance gain should be
|
|
|
|
* Requires less disk space than term_vectors, needed for the fast vector
|
|
|
|
highlighter
|
|
|
|
* Breaks the text into sentences and highlights them. Plays really well with
|
|
|
|
natural languages, not as well with fields containing for instance html markup
|
|
|
|
* Treats the document as the whole corpus, and scores individual sentences as
|
|
|
|
if they were documents in this corpus, using the BM25 algorithm
|
|
|
|
|
|
|
|
Here is an example of setting the `content` field to allow for
|
|
|
|
highlighting using the postings highlighter on it:
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"type_name" : {
|
|
|
|
"content" : {"index_options" : "offsets"}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
Note that the postings highlighter is meant to perform simple query terms
|
|
|
|
highlighting, regardless of their positions. That means that when used for
|
|
|
|
instance in combination with a phrase query, it will highlight all the terms
|
|
|
|
that the query is composed of, regardless of whether they are actually part of
|
|
|
|
a query match, effectively ignoring their positions.
|
|
|
|
|
|
|
|
|
|
|
|
==== Fast vector highlighter
|
|
|
|
|
|
|
|
If `term_vector` information is provided by setting `term_vector` to
|
|
|
|
`with_positions_offsets` in the mapping then the fast vector highlighter
|
|
|
|
will be used instead of the plain highlighter. The fast vector highlighter:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
* Is faster especially for large fields (> `1MB`)
|
2013-10-18 12:03:31 -04:00
|
|
|
* Can be customized with `boundary_chars`, `boundary_max_scan`, and
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
`fragment_offset` (see <<boundary-characters,below>>)
|
2013-10-18 12:03:31 -04:00
|
|
|
* Requires setting `term_vector` to `with_positions_offsets` which
|
2013-08-28 19:24:34 -04:00
|
|
|
increases the size of the index
|
|
|
|
|
|
|
|
Here is an example of setting the `content` field to allow for
|
|
|
|
highlighting using the fast vector highlighter on it (this will cause
|
|
|
|
the index to be bigger):
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"type_name" : {
|
|
|
|
"content" : {"term_vector" : "with_positions_offsets"}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
==== Force highlighter type
|
|
|
|
|
|
|
|
The `type` field allows to force a specific highlighter type. This is useful
|
|
|
|
for instance when needing to use the plain highlighter on a field that has
|
|
|
|
`term_vectors` enabled. The allowed values are: `plain`, `postings` and `fvh`.
|
|
|
|
The following is an example that forces the use of the plain highlighter:
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"fields" : {
|
|
|
|
"content" : { "type" : "plain"}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2013-09-25 12:17:40 -04:00
|
|
|
[[tags]]
|
2013-08-28 19:24:34 -04:00
|
|
|
==== Highlighting Tags
|
|
|
|
|
|
|
|
By default, the highlighting will wrap highlighted text in `<em>` and
|
|
|
|
`</em>`. This can be controlled by setting `pre_tags` and `post_tags`,
|
|
|
|
for example:
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"pre_tags" : ["<tag1>"],
|
|
|
|
"post_tags" : ["</tag1>"],
|
|
|
|
"fields" : {
|
|
|
|
"_all" : {}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
Using the fast vector highlighter there can be more tags, and the "importance"
|
|
|
|
is ordered.
|
|
|
|
|
2013-08-28 19:24:34 -04:00
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"pre_tags" : ["<tag1>", "<tag2>"],
|
|
|
|
"post_tags" : ["</tag1>", "</tag2>"],
|
|
|
|
"fields" : {
|
|
|
|
"_all" : {}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
There are also built in "tag" schemas, with currently a single schema
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
called `styled` with the following `pre_tags`:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
<em class="hlt1">, <em class="hlt2">, <em class="hlt3">,
|
|
|
|
<em class="hlt4">, <em class="hlt5">, <em class="hlt6">,
|
|
|
|
<em class="hlt7">, <em class="hlt8">, <em class="hlt9">,
|
|
|
|
<em class="hlt10">
|
|
|
|
--------------------------------------------------
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
and `</em>` as `post_tags`. If you think of more nice to have built in tag
|
2013-08-28 19:24:34 -04:00
|
|
|
schemas, just send an email to the mailing list or open an issue. Here
|
|
|
|
is an example of switching tag schemas:
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"tags_schema" : "styled",
|
|
|
|
"fields" : {
|
|
|
|
"content" : {}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
|
|
|
|
==== Encoder
|
|
|
|
|
2013-08-28 19:24:34 -04:00
|
|
|
An `encoder` parameter can be used to define how highlighted text will
|
|
|
|
be encoded. It can be either `default` (no encoding) or `html` (will
|
|
|
|
escape html, if you use html highlighting tags).
|
|
|
|
|
|
|
|
==== Highlighted Fragments
|
|
|
|
|
|
|
|
Each field highlighted can control the size of the highlighted fragment
|
|
|
|
in characters (defaults to `100`), and the maximum number of fragments
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
to return (defaults to `5`).
|
|
|
|
For example:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"fields" : {
|
|
|
|
"content" : {"fragment_size" : 150, "number_of_fragments" : 3}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
The `fragment_size` is ignored when using the postings highlighter, as it
|
|
|
|
outputs sentences regardless of their length.
|
|
|
|
|
|
|
|
On top of this it is possible to specify that highlighted fragments need
|
|
|
|
to be sorted by score:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"order" : "score",
|
|
|
|
"fields" : {
|
|
|
|
"content" : {"fragment_size" : 150, "number_of_fragments" : 3}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
2013-10-24 08:30:14 -04:00
|
|
|
If the `number_of_fragments` value is set to `0` then no fragments are
|
2013-10-18 12:03:31 -04:00
|
|
|
produced, instead the whole content of the field is returned, and of
|
|
|
|
course it is highlighted. This can be very handy if short texts (like
|
|
|
|
document title or address) need to be highlighted but no fragmentation
|
|
|
|
is required. Note that `fragment_size` is ignored in this case.
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"fields" : {
|
|
|
|
"_all" : {},
|
|
|
|
"bio.title" : {"number_of_fragments" : 0}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
When using `fast-vector-highlighter` one can use `fragment_offset`
|
|
|
|
parameter to control the margin to start highlighting from.
|
|
|
|
|
2013-10-24 08:30:14 -04:00
|
|
|
In the case where there is no matching fragment to highlight, the default is
|
|
|
|
to not return anything. Instead, we can return a snippet of text from the
|
|
|
|
beginning of the field by setting `no_match_size` (default `0`) to the length
|
|
|
|
of the text that you want returned. The actual length may be shorter than
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
specified as it tries to break on a word boundary. When using the postings
|
|
|
|
highlighter it is not possible to control the actual size of the snippet,
|
|
|
|
therefore the first sentence gets returned whenever `no_match_size` is
|
|
|
|
greater than `0`.
|
2013-09-03 14:25:58 -04:00
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"fields" : {
|
|
|
|
"content" : {
|
|
|
|
"fragment_size" : 150,
|
|
|
|
"number_of_fragments" : 3,
|
|
|
|
"no_match_size": 150
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
2013-10-18 12:03:31 -04:00
|
|
|
==== Highlight query
|
|
|
|
|
2013-09-05 12:39:01 -04:00
|
|
|
It is also possible to highlight against a query other than the search
|
|
|
|
query by setting `highlight_query`. This is especially useful if you
|
|
|
|
use a rescore query because those are not taken into account by
|
|
|
|
highlighting by default. Elasticsearch does not validate that
|
|
|
|
`highlight_query` contains the search query in any way so it is possible
|
|
|
|
to define it so legitimate query results aren't highlighted at all.
|
|
|
|
Generally it is better to include the search query in the
|
|
|
|
`highlight_query`. Here is an example of including both the search
|
|
|
|
query and the rescore query in `highlight_query`.
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"fields": [ "_id" ],
|
|
|
|
"query" : {
|
|
|
|
"match": {
|
|
|
|
"content": {
|
|
|
|
"query": "foo bar"
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"rescore": {
|
|
|
|
"window_size": 50,
|
|
|
|
"query": {
|
|
|
|
"rescore_query" : {
|
|
|
|
"match_phrase": {
|
|
|
|
"content": {
|
|
|
|
"query": "foo bar",
|
|
|
|
"phrase_slop": 1
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"rescore_query_weight" : 10
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"highlight" : {
|
|
|
|
"order" : "score",
|
|
|
|
"fields" : {
|
|
|
|
"content" : {
|
|
|
|
"fragment_size" : 150,
|
|
|
|
"number_of_fragments" : 3,
|
|
|
|
"highlight_query": {
|
|
|
|
"bool": {
|
|
|
|
"must": {
|
|
|
|
"match": {
|
|
|
|
"content": {
|
|
|
|
"query": "foo bar"
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"should": {
|
|
|
|
"match_phrase": {
|
|
|
|
"content": {
|
|
|
|
"query": "foo bar",
|
|
|
|
"phrase_slop": 1,
|
|
|
|
"boost": 10.0
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"minimum_should_match": 0
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
Note that the score of text fragment in this case is calculated by the Lucene
|
|
|
|
highlighting framework. For implementation details you can check the
|
|
|
|
`ScoreOrderFragmentsBuilder.java` class. On the other hand when using the
|
|
|
|
postings highlighter the fragments are scored using, as mentioned above,
|
|
|
|
the BM25 algorithm.
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2013-09-30 17:32:00 -04:00
|
|
|
[[highlighting-settings]]
|
2013-08-28 19:24:34 -04:00
|
|
|
==== Global Settings
|
|
|
|
|
|
|
|
Highlighting settings can be set on a global level and then overridden
|
|
|
|
at the field level.
|
|
|
|
|
|
|
|
[source,js]
|
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"query" : {...},
|
|
|
|
"highlight" : {
|
|
|
|
"number_of_fragments" : 3,
|
|
|
|
"fragment_size" : 150,
|
|
|
|
"tag_schema" : "styled",
|
|
|
|
"fields" : {
|
|
|
|
"_all" : { "pre_tags" : ["<em>"], "post_tags" : ["</em>"] },
|
|
|
|
"bio.title" : { "number_of_fragments" : 0 },
|
|
|
|
"bio.author" : { "number_of_fragments" : 0 },
|
|
|
|
"bio.content" : { "number_of_fragments" : 5, "order" : "score" }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
2013-09-25 12:17:40 -04:00
|
|
|
[[field-match]]
|
2013-08-28 19:24:34 -04:00
|
|
|
==== Require Field Match
|
|
|
|
|
|
|
|
`require_field_match` can be set to `true` which will cause a field to
|
|
|
|
be highlighted only if a query matched that field. `false` means that
|
|
|
|
terms are highlighted on all requested fields regardless if the query
|
|
|
|
matches specifically on them.
|
|
|
|
|
2013-09-25 12:17:40 -04:00
|
|
|
[[boundary-characters]]
|
2013-08-28 19:24:34 -04:00
|
|
|
==== Boundary Characters
|
|
|
|
|
Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.
Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same
The lucene postings highlighter api is quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.
Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.
Closes #3704
2013-08-08 11:10:42 -04:00
|
|
|
When highlighting a field using the fast vector highlighter,
|
2013-08-28 19:24:34 -04:00
|
|
|
`boundary_chars` can be configured to define what constitutes a boundary
|
|
|
|
for highlighting. It's a single string with each boundary character
|
|
|
|
defined in it. It defaults to `.,!? \t\n`.
|
|
|
|
|
|
|
|
The `boundary_max_scan` allows to control how far to look for boundary
|
|
|
|
characters, and defaults to `20`.
|