mirror of https://github.com/apache/lucene.git
116 lines
7.5 KiB
Plaintext
116 lines
7.5 KiB
Plaintext
= Analyzers
|
|
// Licensed to the Apache Software Foundation (ASF) under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing,
|
|
// software distributed under the License is distributed on an
|
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
// KIND, either express or implied. See the License for the
|
|
// specific language governing permissions and limitations
|
|
// under the License.
|
|
|
|
An analyzer examines the text of fields and generates a token stream.
|
|
|
|
Analyzers are specified as a child of the `<fieldType>` element in the `schema.xml` configuration file (in the same `conf/` directory as `solrconfig.xml`).
|
|
|
|
In normal usage, only fields of type `solr.TextField` will specify an analyzer. The simplest way to configure an analyzer is with a single `<analyzer>` element whose class attribute is a fully qualified Java class name. The named class must derive from `org.apache.lucene.analysis.Analyzer`. For example:
|
|
|
|
[source,xml]
|
|
----
|
|
<fieldType name="nametext" class="solr.TextField">
|
|
<analyzer class="org.apache.lucene.analysis.core.WhitespaceAnalyzer"/>
|
|
</fieldType>
|
|
----
|
|
|
|
In this case a single class, `WhitespaceAnalyzer`, is responsible for analyzing the content of the named text field and emitting the corresponding tokens. For simple cases, such as plain English prose, a single analyzer class like this may be sufficient. But it's often necessary to do more complex analysis of the field content.
|
|
|
|
Even the most complex analysis requirements can usually be decomposed into a series of discrete, relatively simple processing steps. As you will soon discover, the Solr distribution comes with a large selection of tokenizers and filters that covers most scenarios you are likely to encounter. Setting up an analyzer chain is very straightforward; you specify a simple `<analyzer>` element (no class attribute) with child elements that name factory classes for the tokenizer and filters to use, in the order you want them to run.
|
|
|
|
For example:
|
|
|
|
[source,xml]
|
|
----
|
|
<fieldType name="nametext" class="solr.TextField">
|
|
<analyzer>
|
|
<tokenizer class="solr.StandardTokenizerFactory"/>
|
|
<filter class="solr.StandardFilterFactory"/>
|
|
<filter class="solr.LowerCaseFilterFactory"/>
|
|
<filter class="solr.StopFilterFactory"/>
|
|
<filter class="solr.EnglishPorterFilterFactory"/>
|
|
</analyzer>
|
|
</fieldType>
|
|
----
|
|
|
|
Note that classes in the `org.apache.solr.analysis` package may be referred to here with the shorthand `solr.` prefix.
|
|
|
|
In this case, no Analyzer class was specified on the `<analyzer>` element. Rather, a sequence of more specialized classes are wired together and collectively act as the Analyzer for the field. The text of the field is passed to the first item in the list (`solr.StandardTokenizerFactory`), and the tokens that emerge from the last one (`solr.EnglishPorterFilterFactory`) are the terms that are used for indexing or querying any fields that use the "nametext" `fieldType`.
|
|
|
|
.Field Values versus Indexed Terms
|
|
[IMPORTANT]
|
|
====
|
|
The output of an Analyzer affects the _terms_ indexed in a given field (and the terms used when parsing queries against those fields) but it has no impact on the _stored_ value for the fields. For example: an analyzer might split "Brown Cow" into two indexed terms "brown" and "cow", but the stored value will still be a single String: "Brown Cow"
|
|
====
|
|
|
|
== Analysis Phases
|
|
|
|
Analysis takes place in two contexts. At index time, when a field is being created, the token stream that results from analysis is added to an index and defines the set of terms (including positions, sizes, and so on) for the field. At query time, the values being searched for are analyzed and the terms that result are matched against those that are stored in the field's index.
|
|
|
|
In many cases, the same analysis should be applied to both phases. This is desirable when you want to query for exact string matches, possibly with case-insensitivity, for example. In other cases, you may want to apply slightly different analysis steps during indexing than those used at query time.
|
|
|
|
If you provide a simple `<analyzer>` definition for a field type, as in the examples above, then it will be used for both indexing and queries. If you want distinct analyzers for each phase, you may include two `<analyzer>` definitions distinguished with a type attribute. For example:
|
|
|
|
[source,xml]
|
|
----
|
|
<fieldType name="nametext" class="solr.TextField">
|
|
<analyzer type="index">
|
|
<tokenizer class="solr.StandardTokenizerFactory"/>
|
|
<filter class="solr.LowerCaseFilterFactory"/>
|
|
<filter class="solr.KeepWordFilterFactory" words="keepwords.txt"/>
|
|
<filter class="solr.SynonymFilterFactory" synonyms="syns.txt"/>
|
|
</analyzer>
|
|
<analyzer type="query">
|
|
<tokenizer class="solr.StandardTokenizerFactory"/>
|
|
<filter class="solr.LowerCaseFilterFactory"/>
|
|
</analyzer>
|
|
</fieldType>
|
|
----
|
|
|
|
In this theoretical example, at index time the text is tokenized, the tokens are set to lowercase, any that are not listed in `keepwords.txt` are discarded and those that remain are mapped to alternate values as defined by the synonym rules in the file `syns.txt`. This essentially builds an index from a restricted set of possible values and then normalizes them to values that may not even occur in the original text.
|
|
|
|
At query time, the only normalization that happens is to convert the query terms to lowercase. The filtering and mapping steps that occur at index time are not applied to the query terms. Queries must then, in this example, be very precise, using only the normalized terms that were stored at index time.
|
|
|
|
=== Analysis for Multi-Term Expansion
|
|
|
|
In some types of queries (i.e., Prefix, Wildcard, Regex, etc...) the input provided by the user is not natural language intended for Analysis. Things like Synonyms or Stop word filtering do not work in a logical way in these types of Queries.
|
|
|
|
The analysis factories that _can_ work in these types of queries (such as Lowercasing, or Normalizing Factories) are known as {lucene-javadocs}/analyzers-common/org/apache/lucene/analysis/util/MultiTermAwareComponent.html[`MultiTermAwareComponents`]. When Solr needs to perform analysis for a query that results in Multi-Term expansion, only the `MultiTermAwareComponents` used in the `query` analyzer are used, Factory that is not Multi-Term aware will be skipped.
|
|
|
|
For most use cases, this provides the best possible behavior, but if you wish for absolute control over the analysis performed on these types of queries, you may explicitly define a `multiterm` analyzer to use, such as in the following example:
|
|
|
|
[source,xml]
|
|
----
|
|
<fieldType name="nametext" class="solr.TextField">
|
|
<analyzer type="index">
|
|
<tokenizer class="solr.StandardTokenizerFactory"/>
|
|
<filter class="solr.LowerCaseFilterFactory"/>
|
|
<filter class="solr.KeepWordFilterFactory" words="keepwords.txt"/>
|
|
<filter class="solr.SynonymFilterFactory" synonyms="syns.txt"/>
|
|
</analyzer>
|
|
<analyzer type="query">
|
|
<tokenizer class="solr.StandardTokenizerFactory"/>
|
|
<filter class="solr.LowerCaseFilterFactory"/>
|
|
</analyzer>
|
|
<!-- No analysis at all when doing queries that involved Multi-Term expansion -->
|
|
<analyzer type="multiterm">
|
|
<tokenizer class="solr.KeywordTokenizerFactory" />
|
|
</analyzer>
|
|
</fieldType>
|
|
----
|