OpenSearch/docs/reference/analysis.asciidoc

[[analysis]]
= Text analysis

[partintro]
--

_Text analysis_ is the process of converting text, like the body of any email,
into _tokens_ or _terms_ which are added to the inverted index for searching.
Analysis is performed by an <<analysis-analyzers,_analyzer_>> which can be
either a built-in analyzer or a <<analysis-custom-analyzer,`custom`>> analyzer
defined per index.

[float]
== Index time analysis

For instance, at index time the built-in <<english-analyzer,`english`>> _analyzer_ 
will first convert the sentence:

[source,text]
------
"The QUICK brown foxes jumped over the lazy dog!"
------

into distinct tokens. It will then lowercase each token, remove frequent
stopwords ("the") and reduce the terms to their word stems (foxes -> fox,
jumped -> jump, lazy -> lazi). In the end, the following terms will be added
to the inverted index:

[source,text]
------
[ quick, brown, fox, jump, over, lazi, dog ]
------

[float]
[[specify-index-time-analyzer]]
=== Specifying an index time analyzer

{es} determines which index-time analyzer to use by
checking the following parameters in order:

. The <<analyzer,`analyzer`>> mapping parameter of the field
. The `default` analyzer parameter in the index settings

If none of these parameters are specified, the
<<analysis-standard-analyzer,`standard` analyzer>> is used.

[discrete]
[[specify-index-time-field-analyzer]]
==== Specify the index-time analyzer for a field

Each <<text,`text`>> field in a mapping can specify its own
<<analyzer,`analyzer`>>:

[source,console]
-------------------------
PUT my_index
{
  "mappings": {
    "properties": {
      "title": {
        "type":     "text",
        "analyzer": "standard"
      }
    }
  }
}
-------------------------

[discrete]
[[specify-index-time-default-analyzer]]
==== Specify a default index-time analyzer

When <<indices-create-index,creating an index>>, you can set a default
index-time analyzer using the `default` analyzer setting:

[source,console]
----
PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "type": "whitespace"
        }
      }
    }
  }
}
----

A default index-time analyzer is useful when mapping multiple `text` fields that
use the same analyzer. It's also used as a general fallback analyzer for both
index-time and search-time analysis.

[float]
== Search time analysis

This same analysis process is applied to the query string at search time in
<<full-text-queries,full text queries>> like the
<<query-dsl-match-query,`match` query>>
to convert the text in the query string into terms of the same form as those
that are stored in the inverted index.

For instance, a user might search for:

[source,text]
------
"a quick fox"
------

which would be analysed by the same `english` analyzer into the following terms:

[source,text]
------
[ quick, fox ]
------

Even though the exact words used in the query string don't appear in the
original text (`quick` vs `QUICK`, `fox` vs `foxes`), because we have applied
the same analyzer to both the text and the query string, the terms from the
query string exactly match the terms from the text in the inverted index,
which means that this query would match our example document.

[float]
=== Specifying a search time analyzer

Usually the same analyzer should be used both at
index time and at search time, and <<full-text-queries,full text queries>>
like the  <<query-dsl-match-query,`match` query>> will use the mapping to look
up the analyzer to use for each field.

The analyzer to use to search a particular field is determined by
looking for:

* An `analyzer` specified in the query itself.
* The <<search-analyzer,`search_analyzer`>> mapping parameter.
* The <<analyzer,`analyzer`>> mapping parameter.
* An analyzer in the index settings called `default_search`.
* An analyzer in the index settings called `default`.
* The `standard` analyzer.

--

include::analysis/overview.asciidoc[]

include::analysis/anatomy.asciidoc[]

include::analysis/testing.asciidoc[]

include::analysis/analyzers.asciidoc[]

include::analysis/normalizers.asciidoc[]

include::analysis/tokenizers.asciidoc[]

include::analysis/tokenfilters.asciidoc[]

include::analysis/charfilters.asciidoc[]