mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-20 03:45:02 +00:00
36a3b84fc9
* Default include_type_name to false for get and put mappings. * Default include_type_name to false for get field mappings. * Add a constant for the default include_type_name value. * Default include_type_name to false for get and put index templates. * Default include_type_name to false for create index. * Update create index calls in REST documentation to use include_type_name=true. * Some minor clean-ups around the get index API. * In REST tests, use include_type_name=true by default for index creation. * Make sure to use 'expression == false'. * Clarify the different IndexTemplateMetaData toXContent methods. * Fix FullClusterRestartIT#testSnapshotRestore. * Fix the ml_anomalies_default_mappings test. * Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests. We make sure to specify include_type_name=true during xContent parsing, so we continue to test the legacy typed responses. XContent generation for the typeless responses is currently only covered by REST tests, but we will be adding unit test coverage for these as we implement each typeless API in the Java HLRC. This commit also refactors GetMappingsResponse to follow the same appraoch as the other mappings-related responses, where we read include_type_name out of the xContent params, instead of creating a second toXContent method. This gives better consistency in the response parsing code. * Fix more REST tests. * Improve some wording in the create index documentation. * Add a note about types removal in the create index docs. * Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL. * Make sure to mention include_type_name in the REST docs for affected APIs. * Make sure to use 'expression == false' in FullClusterRestartIT. * Mention include_type_name in the REST templates docs.
126 lines
3.3 KiB
Plaintext
126 lines
3.3 KiB
Plaintext
[[analysis]]
|
|
= Analysis
|
|
|
|
[partintro]
|
|
--
|
|
|
|
_Analysis_ is the process of converting text, like the body of any email, into
|
|
_tokens_ or _terms_ which are added to the inverted index for searching.
|
|
Analysis is performed by an <<analysis-analyzers,_analyzer_>> which can be
|
|
either a built-in analyzer or a <<analysis-custom-analyzer,`custom`>> analyzer
|
|
defined per index.
|
|
|
|
[float]
|
|
== Index time analysis
|
|
|
|
For instance, at index time the built-in <<english-analyzer,`english`>> _analyzer_
|
|
will first convert the sentence:
|
|
|
|
[source,text]
|
|
------
|
|
"The QUICK brown foxes jumped over the lazy dog!"
|
|
------
|
|
|
|
into distinct tokens. It will then lowercase each token, remove frequent
|
|
stopwords ("the") and reduce the terms to their word stems (foxes -> fox,
|
|
jumped -> jump, lazy -> lazi). In the end, the following terms will be added
|
|
to the inverted index:
|
|
|
|
[source,text]
|
|
------
|
|
[ quick, brown, fox, jump, over, lazi, dog ]
|
|
------
|
|
|
|
[float]
|
|
=== Specifying an index time analyzer
|
|
|
|
Each <<text,`text`>> field in a mapping can specify its own
|
|
<<analyzer,`analyzer`>>:
|
|
|
|
[source,js]
|
|
-------------------------
|
|
PUT my_index?include_type_name=true
|
|
{
|
|
"mappings": {
|
|
"_doc": {
|
|
"properties": {
|
|
"title": {
|
|
"type": "text",
|
|
"analyzer": "standard"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
-------------------------
|
|
// CONSOLE
|
|
|
|
At index time, if no `analyzer` has been specified, it looks for an analyzer
|
|
in the index settings called `default`. Failing that, it defaults to using
|
|
the <<analysis-standard-analyzer,`standard` analyzer>>.
|
|
|
|
|
|
[float]
|
|
== Search time analysis
|
|
|
|
This same analysis process is applied to the query string at search time in
|
|
<<full-text-queries,full text queries>> like the
|
|
<<query-dsl-match-query,`match` query>>
|
|
to convert the text in the query string into terms of the same form as those
|
|
that are stored in the inverted index.
|
|
|
|
For instance, a user might search for:
|
|
|
|
[source,text]
|
|
------
|
|
"a quick fox"
|
|
------
|
|
|
|
which would be analysed by the same `english` analyzer into the following terms:
|
|
|
|
[source,text]
|
|
------
|
|
[ quick, fox ]
|
|
------
|
|
|
|
Even though the exact words used in the query string don't appear in the
|
|
original text (`quick` vs `QUICK`, `fox` vs `foxes`), because we have applied
|
|
the same analyzer to both the text and the query string, the terms from the
|
|
query string exactly match the terms from the text in the inverted index,
|
|
which means that this query would match our example document.
|
|
|
|
[float]
|
|
=== Specifying a search time analyzer
|
|
|
|
Usually the same analyzer should be used both at
|
|
index time and at search time, and <<full-text-queries,full text queries>>
|
|
like the <<query-dsl-match-query,`match` query>> will use the mapping to look
|
|
up the analyzer to use for each field.
|
|
|
|
The analyzer to use to search a particular field is determined by
|
|
looking for:
|
|
|
|
* An `analyzer` specified in the query itself.
|
|
* The <<search-analyzer,`search_analyzer`>> mapping parameter.
|
|
* The <<analyzer,`analyzer`>> mapping parameter.
|
|
* An analyzer in the index settings called `default_search`.
|
|
* An analyzer in the index settings called `default`.
|
|
* The `standard` analyzer.
|
|
|
|
--
|
|
|
|
include::analysis/anatomy.asciidoc[]
|
|
|
|
include::analysis/testing.asciidoc[]
|
|
|
|
include::analysis/analyzers.asciidoc[]
|
|
|
|
include::analysis/normalizers.asciidoc[]
|
|
|
|
include::analysis/tokenizers.asciidoc[]
|
|
|
|
include::analysis/tokenfilters.asciidoc[]
|
|
|
|
include::analysis/charfilters.asciidoc[]
|
|
|