181 lines
6.5 KiB
Plaintext
181 lines
6.5 KiB
Plaintext
[[fielddata]]
|
|
=== `fielddata`
|
|
|
|
Most fields are <<mapping-index,indexed>> by default, which makes them
|
|
searchable. Sorting, aggregations, and accessing field values in scripts,
|
|
however, requires a different access pattern from search.
|
|
|
|
Search needs to answer the question _"Which documents contain this term?"_,
|
|
while sorting and aggregations need to answer a different question: _"What is
|
|
the value of this field for **this** document?"_.
|
|
|
|
Most fields can use index-time, on-disk <<doc-values,`doc_values`>> for this
|
|
data access pattern, but <<text,`text`>> fields do not support `doc_values`.
|
|
|
|
Instead, `text` fields use a query-time *in-memory* data structure called
|
|
`fielddata`. This data structure is built on demand the first time that a
|
|
field is used for aggregations, sorting, or in a script. It is built by
|
|
reading the entire inverted index for each segment from disk, inverting the
|
|
term ↔︎ document relationship, and storing the result in memory, in the JVM
|
|
heap.
|
|
|
|
==== Fielddata is disabled on `text` fields by default
|
|
|
|
Fielddata can consume a *lot* of heap space, especially when loading high
|
|
cardinality `text` fields. Once fielddata has been loaded into the heap, it
|
|
remains there for the lifetime of the segment. Also, loading fielddata is an
|
|
expensive process which can cause users to experience latency hits. This is
|
|
why fielddata is disabled by default.
|
|
|
|
If you try to sort, aggregate, or access values from a script on a `text`
|
|
field, you will see this exception:
|
|
|
|
[quote]
|
|
--
|
|
Fielddata is disabled on text fields by default. Set `fielddata=true` on
|
|
[`your_field_name`] in order to load fielddata in memory by uninverting the
|
|
inverted index. Note that this can however use significant memory.
|
|
--
|
|
|
|
[[before-enabling-fielddata]]
|
|
==== Before enabling fielddata
|
|
|
|
Before you enable fielddata, consider why you are using a `text` field for
|
|
aggregations, sorting, or in a script. It usually doesn't make sense to do
|
|
so.
|
|
|
|
A text field is analyzed before indexing so that a value like
|
|
`New York` can be found by searching for `new` or for `york`. A `terms`
|
|
aggregation on this field will return a `new` bucket and a `york` bucket, when
|
|
you probably want a single bucket called `New York`.
|
|
|
|
Instead, you should have a `text` field for full text searches, and an
|
|
unanalyzed <<keyword,`keyword`>> field with <<doc-values,`doc_values`>>
|
|
enabled for aggregations, as follows:
|
|
|
|
[source,js]
|
|
---------------------------------
|
|
PUT my_index
|
|
{
|
|
"mappings": {
|
|
"my_type": {
|
|
"properties": {
|
|
"my_field": { <1>
|
|
"type": "text",
|
|
"fields": {
|
|
"keyword": { <2>
|
|
"type": "keyword"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
---------------------------------
|
|
// CONSOLE
|
|
<1> Use the `my_field` field for searches.
|
|
<2> Use the `my_field.keyword` field for aggregations, sorting, or in scripts.
|
|
|
|
==== Enabling fielddata on `text` fields
|
|
|
|
You can enable fielddata on an existing `text` field using the
|
|
<<indices-put-mapping,PUT mapping API>> as follows:
|
|
|
|
[source,js]
|
|
-----------------------------------
|
|
PUT my_index/_mapping/my_type
|
|
{
|
|
"properties": {
|
|
"my_field": { <1>
|
|
"type": "text",
|
|
"fielddata": true
|
|
}
|
|
}
|
|
}
|
|
-----------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
<1> The mapping that you specify for `my_field` should consist of the existing
|
|
mapping for that field, plus the `fielddata` parameter.
|
|
|
|
TIP: The `fielddata.*` parameter must have the same settings for fields of the
|
|
same name in the same index. Its value can be updated on existing fields
|
|
using the <<indices-put-mapping,PUT mapping API>>.
|
|
|
|
|
|
[[global-ordinals]]
|
|
.Global ordinals
|
|
*****************************************
|
|
|
|
Global ordinals is a data-structure on top of fielddata and doc values, that
|
|
maintains an incremental numbering for each unique term in a lexicographic
|
|
order. Each term has a unique number and the number of term 'A' is lower than
|
|
the number of term 'B'. Global ordinals are only supported on <<text,`text`>>
|
|
and <<keyword,`keyword`>> fields.
|
|
|
|
Fielddata and doc values also have ordinals, which is a unique numbering for
|
|
all terms in a particular segment and field. Global ordinals just build on top
|
|
of this, by providing a mapping between the segment ordinals and the global
|
|
ordinals, the latter being unique across the entire shard.
|
|
|
|
Global ordinals are used for features that use segment ordinals, such as
|
|
sorting and the terms aggregation, to improve the execution time. A terms
|
|
aggregation relies purely on global ordinals to perform the aggregation at the
|
|
shard level, then converts global ordinals to the real term only for the final
|
|
reduce phase, which combines results from different shards.
|
|
|
|
Global ordinals for a specified field are tied to _all the segments of a
|
|
shard_, while fielddata and doc values ordinals are tied to a single segment.
|
|
which is different than for field data for a specific field which is tied to a
|
|
single segment. For this reason global ordinals need to be entirely rebuilt
|
|
whenever a once new segment becomes visible.
|
|
|
|
The loading time of global ordinals depends on the number of terms in a field,
|
|
but in general it is low, since it source field data has already been loaded.
|
|
The memory overhead of global ordinals is a small because it is very
|
|
efficiently compressed. Eager loading of global ordinals can move the loading
|
|
time from the first search request, to the refresh itself.
|
|
|
|
*****************************************
|
|
|
|
[[field-data-filtering]]
|
|
==== `fielddata_frequency_filter`
|
|
|
|
Fielddata filtering can be used to reduce the number of terms loaded into
|
|
memory, and thus reduce memory usage. Terms can be filtered by _frequency_:
|
|
|
|
The frequency filter allows you to only load terms whose document frequency falls
|
|
between a `min` and `max` value, which can be expressed an absolute
|
|
number (when the number is bigger than 1.0) or as a percentage
|
|
(eg `0.01` is `1%` and `1.0` is `100%`). Frequency is calculated
|
|
*per segment*. Percentages are based on the number of docs which have a
|
|
value for the field, as opposed to all docs in the segment.
|
|
|
|
Small segments can be excluded completely by specifying the minimum
|
|
number of docs that the segment should contain with `min_segment_size`:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT my_index
|
|
{
|
|
"mappings": {
|
|
"my_type": {
|
|
"properties": {
|
|
"tag": {
|
|
"type": "text",
|
|
"fielddata": true,
|
|
"fielddata_frequency_filter": {
|
|
"min": 0.001,
|
|
"max": 0.1,
|
|
"min_segment_size": 500
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|