mirror of https://github.com/apache/lucene.git
Updated scoring.xml per suggestions by Doug and Chris on issue 664. Moved the Query information and Similarity info to the o.a.l.s package.html in the Javadocs and provided links to the javadocs from the scoring file.
Added scoring.html into the project.xml so that it will now be live on the Lucene java site. git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@442406 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Grant Ingersoll
parent
6c53bc3c49
commit
d0b971a021
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -89,6 +89,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -83,6 +83,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -83,6 +83,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -91,6 +91,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="../queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="../fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="../scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="../api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -83,6 +83,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -87,6 +87,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
@ -336,126 +338,8 @@ limitations under the License.
|
|||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
</h4>
|
||||
<p>Of the various implementations of
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a>, the
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
is the easiest to understand and the most often used in applications. A <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> matches all the documents that contain the specified
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>,
|
||||
which is a word that occurs in a certain
|
||||
<a href="api/org/apache/lucene/document/Field.html">Field</a>.
|
||||
Thus, a <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> identifies and scores all
|
||||
<a href="api/org/apache/lucene/document/Document.html">Document</a>s that have a <a href="api/org/apache/lucene/document/Field.html">Field</a> with the specified string in it.
|
||||
Constructing a <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
is as simple as:
|
||||
<pre>
|
||||
TermQuery tq = new TermQuery(new Term("fieldName", "term");
|
||||
</pre>In this example, the <a href="api/org/apache/lucene/search/Query.html">Query</a> identifies all <a href="api/org/apache/lucene/document/Document.html">Document</a>s that have the <a href="api/org/apache/lucene/document/Field.html">Field</a> named <tt>"fieldName"</tt> and
|
||||
contain the word <tt>"term"</tt>.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>
|
||||
</h4>
|
||||
<p>Things start to get interesting when one combines multiple
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> instances into a <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>.
|
||||
A <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a> contains multiple
|
||||
<a href="api/org/apache/lucene/search/BooleanClause.html">BooleanClause</a>s,
|
||||
where each clause contains a sub-query (<a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
instance) and an operator (from <a href="api/org/apache/lucene/search/BooleanClause.Occur.html">BooleanClause.Occur</a>)
|
||||
describing how that sub-query is combined with the other clauses:
|
||||
<ol>
|
||||
|
||||
<li><p>SHOULD -- Use this operator when a clause can occur in the result set, but is not required.
|
||||
If a query is made up of all SHOULD clauses, then every document in the result
|
||||
set matches at least one of these clauses.</p></li>
|
||||
|
||||
<li><p>MUST -- Use this operator when a clause is required to occur in the result set. Every
|
||||
document in the result set will match
|
||||
all such clauses.</p></li>
|
||||
|
||||
<li><p>MUST NOT -- Use this operator when a
|
||||
clause must not occur in the result set. No
|
||||
document in the result set will match
|
||||
any such clauses.</p></li>
|
||||
</ol>
|
||||
Boolean queries are constructed by adding two or more
|
||||
<a href="api/org/apache/lucene/search/BooleanClause.html">BooleanClause</a>
|
||||
instances. If too many clauses are added, a <a href="api/org/apache/lucene/search/BooleanQuery.TooManyClauses.html">TooManyClauses</a>
|
||||
exception will be thrown during searching. This most often occurs
|
||||
when a <a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
is rewritten into a <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a> with many
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> clauses,
|
||||
for example by <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>.
|
||||
The default setting for the maximum number
|
||||
of clauses 1024, but this can be changed via the
|
||||
static method <a href="api/org/apache/lucene/search/BooleanQuery.html#setMaxClauseCount(int)">setMaxClauseCount</a>
|
||||
in <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>.
|
||||
</p>
|
||||
<h4>Phrases</h4>
|
||||
<p>Another common search is to find documents containing certain phrases. This
|
||||
is handled in two different ways.
|
||||
<ol>
|
||||
<li>
|
||||
<p><a href="api/org/apache/lucene/search/PhraseQuery.html">PhraseQuery</a>
|
||||
-- Matches a sequence of
|
||||
<a href="api/org/apache/lucene/index/Term.html">Terms</a>.
|
||||
<a href="api/org/apache/lucene/search/PhraseQuery.html">PhraseQuery</a> uses a slop factor to determine
|
||||
how many positions may occur between any two terms in the phrase and still be considered a match.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><a href="api/org/apache/lucene/search/spans/SpanNearQuery.html">SpanNearQuery</a>
|
||||
-- Matches a sequence of other
|
||||
<a href="api/org/apache/lucene/search/spans/SpanQuery.html">SpanQuery</a>
|
||||
instances. <a href="api/org/apache/lucene/search/spans/SpanNearQuery.html">SpanNearQuery</a> allows for much more
|
||||
complicated phrase queries since it is constructed from other to <a href="api/org/apache/lucene/search/spans/SpanQuery.html">SpanQuery</a>
|
||||
instances, instead of only <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> instances.</p>
|
||||
</li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/RangeQuery.html">RangeQuery</a>
|
||||
</h4>
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/RangeQuery.html">RangeQuery</a>
|
||||
matches all documents that occur in the
|
||||
exclusive range of a lower
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>
|
||||
and an upper
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>.
|
||||
For example, one could find all documents
|
||||
that have terms beginning with the letters <tt>a</tt> through <tt>c</tt>. This type of <a href="api/org/apache/lucene/search/Query.html">Query</a> is frequently used to
|
||||
find
|
||||
documents that occur in a specific date range.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a>,
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>
|
||||
</h4>
|
||||
<p>While the
|
||||
<a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a>
|
||||
has a different implementation, it is essentially a special case of the
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>.
|
||||
The <a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a> allows an application
|
||||
to identify all documents with terms that begin with a certain string. The <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> generalizes this by allowing
|
||||
for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> (matches exactly one character) wildcards. Note that the <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> can be quite slow. Also note that
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> should
|
||||
not start with <tt>*</tt> and <tt>?</tt>, as these are extremely slow. For tricks on how to search using a wildcard at
|
||||
the beginning of a term, see
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/13373#13373">
|
||||
Starts With x and Ends With x Queries</a>
|
||||
from the Lucene users's mailing list.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/FuzzyQuery.html">FuzzyQuery</a>
|
||||
</h4>
|
||||
<p>A
|
||||
<a href="api/org/apache/lucene/search/FuzzyQuery.html">FuzzyQuery</a>
|
||||
matches documents that contain terms similar to the specified term. Similarity is
|
||||
determined using
|
||||
<a href="http://en.wikipedia.org/wiki/Levenshtein">Levenshtein (edit) distance</a>.
|
||||
This type of query can be useful when accounting for spelling variations in the collection.
|
||||
<p>For information on the Query Classes, refer to the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#query">search package javadocs</a>
|
||||
</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
|
|
@ -469,36 +353,9 @@ limitations under the License.
|
|||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>Chances are <a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a> is sufficient for all your searching needs.
|
||||
However, in some applications it may be necessary to customize your <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a> implementation. For instance, some applications do not need to
|
||||
distinguish between shorter and longer documents (see <a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">a "fair" similarity</a>).</p>
|
||||
<p>To change <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a>, one must do so for both indexing and searching, and the changes must happen before
|
||||
either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen.
|
||||
</p>
|
||||
<p>To make this change, implement your own <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a> (likely you'll want to simply subclass
|
||||
<a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a>) and then use the new
|
||||
class by calling
|
||||
<a href="api/org/apache/lucene/index/IndexWriter.html#setSimilarity(org.apache.lucene.search.Similarity)">IndexWriter.setSimilarity</a> before indexing and
|
||||
<a href="api/org/apache/lucene/search/Searcher.html#setSimilarity(org.apache.lucene.search.Similarity)">Searcher.setSimilarity</a> before searching.
|
||||
</p>
|
||||
<p>
|
||||
If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at <a href="http://www.nabble.com/Overriding-Similarity-tf2128934.html">Overriding Similarity</a>.
|
||||
In summary, here are a few use cases:
|
||||
<ol>
|
||||
<li><p><a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> -- <a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> gives small increases as the frequency increases a small amount
|
||||
and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.</p></li>
|
||||
<li><p>Overriding tf -- In some applications, it doesn't matter what the score of a document is as long as a matching term occurs. In these
|
||||
cases people have overridden Similarity to return 1 from the tf() method.</p></li>
|
||||
<li><p>Changing Length Normalization -- By overriding <a href="api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String,%20int)">lengthNorm</a>, it is possible to discount how the length of a field contributes
|
||||
to a score. In <a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a>, lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be
|
||||
1 / (numTerms in field), all fields will be treated
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">"fairly"</a>.</p></li>
|
||||
</ol>
|
||||
In general, Chris Hostetter sums it up best in saying (from <a href="http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125">the Lucene users's mailing list</a>):
|
||||
<blockquote>[One would override the Similarity in] ... any situation where you know more about your data then just that
|
||||
it's "text" is a situation where it *might* make sense to to override your
|
||||
Similarity method.</blockquote>
|
||||
</p>
|
||||
<p>One of the ways of changing the scoring characteristics of Lucene is to change the similarity factors. For information on
|
||||
how to do this, see the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#changingSimilarity">search package javadocs</a></p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
|
|
@ -516,169 +373,10 @@ limitations under the License.
|
|||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>Changing scoring is an expert level task, so tread carefully and be prepared to share your code if
|
||||
you want help.
|
||||
<p>At a much deeper level, one can affect scoring by implementing their own Query classes (and related scoring classes.) To learn more
|
||||
about how to do this, refer to the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#scoring">search package javadocs</a>
|
||||
</p>
|
||||
<p>With the warning out of the way, it is possible to change a lot more than just the Similarity
|
||||
when it comes to scoring in Lucene. Lucene's scoring is a complex mechanism that is grounded by
|
||||
<span class="highlight-for-editing">three main classes</span>:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a> -- The abstract object representation of the user's information need.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a> -- The internal interface representation of the user's Query, so that Query objects may be reused.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a> -- An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities.</li>
|
||||
</ol>
|
||||
Details on each of these classes, and their children can be found in the subsections below.
|
||||
</p>
|
||||
<table border="0" cellspacing="0" cellpadding="2" width="100%">
|
||||
<tr><td bgcolor="#828DA6">
|
||||
<font color="#ffffff" face="arial,helvetica,sanserif">
|
||||
<a name="The Query Class"><strong>The Query Class</strong></a>
|
||||
</font>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>In some sense, the
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
class is where it all begins. Without a Query, there would be
|
||||
nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
|
||||
is often responsible
|
||||
for creating them or coordinating the functionality between them. The
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a> class has several methods that are important for
|
||||
derived classes:
|
||||
<ol>
|
||||
<li>createWeight(Searcher searcher) -- A
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a> is the internal representation of the Query, so each Query implementation must
|
||||
provide an implementation of Weight. See the subsection on <a href="#The Weight Interface">The Weight Interface</a> below for details on implementing the Weight interface.</li>
|
||||
<li>rewrite(IndexReader reader) -- Rewrites queries into primitive queries. Primitive queries are:
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>,
|
||||
<a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>, <span class="highlight-for-editing">OTHERS????</span></li>
|
||||
</ol>
|
||||
</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
</table>
|
||||
<table border="0" cellspacing="0" cellpadding="2" width="100%">
|
||||
<tr><td bgcolor="#828DA6">
|
||||
<font color="#ffffff" face="arial,helvetica,sanserif">
|
||||
<a name="The Weight Interface"><strong>The Weight Interface</strong></a>
|
||||
</font>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a>
|
||||
interface provides an internal representation of the Query so that it can be reused. Any
|
||||
<a href="api/org/apache/lucene/search/Searcher.html">Searcher</a>
|
||||
dependent state should be stored in the Weight implementation,
|
||||
not in the Query class. The interface defines 6 methods that must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#getQuery()">Weight#getQuery()</a> -- Pointer to the Query that this Weight represents.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#getValue()">Weight#getValue()</a> -- The weight for this Query. For example, the TermQuery.TermWeight value is
|
||||
equal to the idf^2 * boost * queryNorm <!-- DOUBLE CHECK THIS --></li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#sumOfSquaredWeights()">
|
||||
Weight#sumOfSquaredWeights()</a> -- The sum of squared weights. Tor TermQuery, this is (idf *
|
||||
boost)^2</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#normalize(float)">
|
||||
Weight#normalize(float)</a> -- Determine the query normalization factor. The query normalization may
|
||||
allow for comparing scores between queries.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#scorer(IndexReader)">
|
||||
Weight#scorer(IndexReader)</a> -- Construct a new
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a>
|
||||
for this Weight. See
|
||||
<a href="#The Scorer Class">The Scorer Class</a>
|
||||
below for help defining a Scorer. As the name implies, the
|
||||
Scorer is responsible for doing the actual scoring of documents given the Query.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#explain(IndexReader, int)">
|
||||
Weight#explain(IndexReader, int)</a> -- Provide a means for explaining why a given document was scored
|
||||
the way it was.</li>
|
||||
</ol>
|
||||
</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
</table>
|
||||
<table border="0" cellspacing="0" cellpadding="2" width="100%">
|
||||
<tr><td bgcolor="#828DA6">
|
||||
<font color="#ffffff" face="arial,helvetica,sanserif">
|
||||
<a name="The Scorer Class"><strong>The Scorer Class</strong></a>
|
||||
</font>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a>
|
||||
abstract class provides common scoring functionality for all Scorer implementations and
|
||||
is the heart of the Lucene scoring process. The Scorer defines the following abstract methods which
|
||||
must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#next()">Scorer#next()</a> -- Advances to the next document that matches this Query, returning true if and only
|
||||
if there is another document that matches.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#doc()">Scorer#doc()</a> -- Returns the id of the
|
||||
<a href="api/org/apache/lucene/document/Document.html">Document</a>
|
||||
that contains the match. Is not valid until next() has been called at least once.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#score()">Scorer#score()</a> -- Return the score of the current document. This value can be determined in any
|
||||
appropriate way for an application. For instance, the
|
||||
<a href="http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/search/TermScorer.java?view=log">TermScorer</a>
|
||||
returns the tf * Weight.getValue() * fieldNorm.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#skipTo(int)">Scorer#skipTo(int)</a> -- Skip ahead in the document matches to the document whose id is greater than
|
||||
or equal to the passed in value. In many instances, skipTo can be
|
||||
implemented more efficiently than simply looping through all the matching documents until
|
||||
the target document is identified.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#explain(int)">Scorer#explain(int)</a> -- Provides details on why the score came about.</li>
|
||||
</ol>
|
||||
</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
</table>
|
||||
<table border="0" cellspacing="0" cellpadding="2" width="100%">
|
||||
<tr><td bgcolor="#828DA6">
|
||||
<font color="#ffffff" face="arial,helvetica,sanserif">
|
||||
<a name="Why would I want to add my own Query?"><strong>Why would I want to add my own Query?</strong></a>
|
||||
</font>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p>In a nutshell, you want to add your own custom Query implementation when you think that Lucene's
|
||||
aren't appropriate for the
|
||||
task that you want to do. You might be doing some cutting edge research or you need more information
|
||||
back
|
||||
out of Lucene (similar to Doug adding SpanQuery functionality).</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
</table>
|
||||
<table border="0" cellspacing="0" cellpadding="2" width="100%">
|
||||
<tr><td bgcolor="#828DA6">
|
||||
<font color="#ffffff" face="arial,helvetica,sanserif">
|
||||
<a name="Examples"><strong>Examples</strong></a>
|
||||
</font>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<blockquote>
|
||||
<p class="highlight-for-editing">FILL IN HERE</p>
|
||||
</blockquote>
|
||||
</td></tr>
|
||||
<tr><td><br/></td></tr>
|
||||
</table>
|
||||
</blockquote>
|
||||
</p>
|
||||
</td></tr>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -87,6 +87,8 @@ limitations under the License.
|
|||
<li> <a href="./queryparsersyntax.html">Query Syntax</a>
|
||||
</li>
|
||||
<li> <a href="./fileformats.html">File Formats</a>
|
||||
</li>
|
||||
<li> <a href="./scoring.html">Scoring</a>
|
||||
</li>
|
||||
<li> <a href="./api/index.html">Javadoc</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -3,13 +3,356 @@
|
|||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
|
||||
<meta name="Author" content="Doug Cutting">
|
||||
<meta content="Grant Ingersoll" name="Author">
|
||||
</head>
|
||||
<body>
|
||||
<h2>Table Of Contents</h2>
|
||||
<p>
|
||||
<ol>
|
||||
<li><a href="#search">Search Basics</a></li>
|
||||
<li><a href="#query">The Query Classes</a></li>
|
||||
<li><a href="#scoring">Changing the Scoring</a></li>
|
||||
</ol>
|
||||
</p>
|
||||
<a name="search"></a>
|
||||
<h2>Search</h2>
|
||||
<p>
|
||||
Search over indices.
|
||||
|
||||
Applications usually call {@link
|
||||
org.apache.lucene.search.Searcher#search(Query)} or {@link
|
||||
org.apache.lucene.search.Searcher#search(Query,Filter)}.
|
||||
|
||||
<!-- FILL IN MORE HERE -->
|
||||
</p>
|
||||
<a name="query"></a>
|
||||
<h2>Query Classes</h2>
|
||||
<h4>
|
||||
<a href="TermQuery.html">TermQuery</a>
|
||||
</h4>
|
||||
|
||||
<p>Of the various implementations of
|
||||
<a href="Query.html">Query</a>, the
|
||||
<a href="TermQuery.html">TermQuery</a>
|
||||
is the easiest to understand and the most often used in applications. A <a
|
||||
href="TermQuery.html">TermQuery</a> matches all the documents that contain the
|
||||
specified
|
||||
<a href="../index/Term.html">Term</a>,
|
||||
which is a word that occurs in a certain
|
||||
<a href="../document/Field.html">Field</a>.
|
||||
Thus, a <a href="TermQuery.html">TermQuery</a> identifies and scores all
|
||||
<a href="../document/Document.html">Document</a>s that have a <a
|
||||
href="../document/Field.html">Field</a> with the specified string in it.
|
||||
Constructing a <a
|
||||
href="TermQuery.html">TermQuery</a>
|
||||
is as simple as:
|
||||
<pre>
|
||||
TermQuery tq = new TermQuery(new Term("fieldName", "term");
|
||||
</pre>In this example, the <a href="Query.html">Query</a> identifies all <a
|
||||
href="../document/Document.html">Document</a>s that have the <a
|
||||
href="../document/Field.html">Field</a> named <tt>"fieldName"</tt> and
|
||||
contain the word <tt>"term"</tt>.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="BooleanQuery.html">BooleanQuery</a>
|
||||
</h4>
|
||||
|
||||
<p>Things start to get interesting when one combines multiple
|
||||
<a href="TermQuery.html">TermQuery</a> instances into a <a
|
||||
href="BooleanQuery.html">BooleanQuery</a>.
|
||||
A <a href="BooleanQuery.html">BooleanQuery</a> contains multiple
|
||||
<a href="BooleanClause.html">BooleanClause</a>s,
|
||||
where each clause contains a sub-query (<a href="Query.html">Query</a>
|
||||
instance) and an operator (from <a
|
||||
href="BooleanClause.Occur.html">BooleanClause.Occur</a>)
|
||||
describing how that sub-query is combined with the other clauses:
|
||||
<ol>
|
||||
|
||||
<li><p>SHOULD -- Use this operator when a clause can occur in the result set, but is not required.
|
||||
If a query is made up of all SHOULD clauses, then every document in the result
|
||||
set matches at least one of these clauses.</p></li>
|
||||
|
||||
<li><p>MUST -- Use this operator when a clause is required to occur in the result set. Every
|
||||
document in the result set will match
|
||||
all such clauses.</p></li>
|
||||
|
||||
<li><p>MUST NOT -- Use this operator when a
|
||||
clause must not occur in the result set. No
|
||||
document in the result set will match
|
||||
any such clauses.</p></li>
|
||||
</ol>
|
||||
Boolean queries are constructed by adding two or more
|
||||
<a href="BooleanClause.html">BooleanClause</a>
|
||||
instances. If too many clauses are added, a <a href="BooleanQuery.TooManyClauses.html">TooManyClauses</a>
|
||||
exception will be thrown during searching. This most often occurs
|
||||
when a <a href="Query.html">Query</a>
|
||||
is rewritten into a <a href="BooleanQuery.html">BooleanQuery</a> with many
|
||||
<a href="TermQuery.html">TermQuery</a> clauses,
|
||||
for example by <a href="WildcardQuery.html">WildcardQuery</a>.
|
||||
The default setting for the maximum number
|
||||
of clauses 1024, but this can be changed via the
|
||||
static method <a href="BooleanQuery.html#setMaxClauseCount(int)">setMaxClauseCount</a>
|
||||
in <a href="BooleanQuery.html">BooleanQuery</a>.
|
||||
</p>
|
||||
|
||||
<h4>Phrases</h4>
|
||||
|
||||
<p>Another common search is to find documents containing certain phrases. This
|
||||
is handled in two different ways.
|
||||
<ol>
|
||||
<li>
|
||||
<p><a href="PhraseQuery.html">PhraseQuery</a>
|
||||
-- Matches a sequence of
|
||||
<a href="../index/Term.html">Terms</a>.
|
||||
<a href="PhraseQuery.html">PhraseQuery</a> uses a slop factor to determine
|
||||
how many positions may occur between any two terms in the phrase and still be considered a match.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><a href="spans/SpanNearQuery.html">SpanNearQuery</a>
|
||||
-- Matches a sequence of other
|
||||
<a href="spans/SpanQuery.html">SpanQuery</a>
|
||||
instances. <a href="spans/SpanNearQuery.html">SpanNearQuery</a> allows for
|
||||
much more
|
||||
complicated phrase queries since it is constructed from other to <a
|
||||
href="spans/SpanQuery.html">SpanQuery</a>
|
||||
instances, instead of only <a href="TermQuery.html">TermQuery</a>
|
||||
instances.</p>
|
||||
</li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>
|
||||
<a href="RangeQuery.html">RangeQuery</a>
|
||||
</h4>
|
||||
|
||||
<p>The
|
||||
<a href="RangeQuery.html">RangeQuery</a>
|
||||
matches all documents that occur in the
|
||||
exclusive range of a lower
|
||||
<a href="../index/Term.html">Term</a>
|
||||
and an upper
|
||||
<a href="../index/Term.html">Term</a>.
|
||||
For example, one could find all documents
|
||||
that have terms beginning with the letters <tt>a</tt> through <tt>c</tt>. This type of <a
|
||||
href="Query.html">Query</a> is frequently used to
|
||||
find
|
||||
documents that occur in a specific date range.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="PrefixQuery.html">PrefixQuery</a>,
|
||||
<a href="WildcardQuery.html">WildcardQuery</a>
|
||||
</h4>
|
||||
|
||||
<p>While the
|
||||
<a href="PrefixQuery.html">PrefixQuery</a>
|
||||
has a different implementation, it is essentially a special case of the
|
||||
<a href="WildcardQuery.html">WildcardQuery</a>.
|
||||
The <a href="PrefixQuery.html">PrefixQuery</a> allows an application
|
||||
to identify all documents with terms that begin with a certain string. The <a
|
||||
href="WildcardQuery.html">WildcardQuery</a> generalizes this by allowing
|
||||
for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> (matches exactly one character) wildcards.
|
||||
Note that the <a href="WildcardQuery.html">WildcardQuery</a> can be quite slow. Also
|
||||
note that
|
||||
<a href="WildcardQuery.html">WildcardQuery</a> should
|
||||
not start with <tt>*</tt> and <tt>?</tt>, as these are extremely slow. For tricks on how to search using a wildcard
|
||||
at
|
||||
the beginning of a term, see
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/13373#13373">
|
||||
Starts With x and Ends With x Queries</a>
|
||||
from the Lucene users's mailing list.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="FuzzyQuery.html">FuzzyQuery</a>
|
||||
</h4>
|
||||
|
||||
<p>A
|
||||
<a href="FuzzyQuery.html">FuzzyQuery</a>
|
||||
matches documents that contain terms similar to the specified term. Similarity is
|
||||
determined using
|
||||
<a href="http://en.wikipedia.org/wiki/Levenshtein">Levenshtein (edit) distance</a>.
|
||||
This type of query can be useful when accounting for spelling variations in the collection.
|
||||
</p>
|
||||
<a name="changingSimilarity"></a>
|
||||
<h2>Changing Similarity</h2>
|
||||
|
||||
<p>Chances are <a href="DefaultSimilarity.html">DefaultSimilarity</a> is sufficient for all
|
||||
your searching needs.
|
||||
However, in some applications it may be necessary to customize your <a
|
||||
href="Similarity.html">Similarity</a> implementation. For instance, some
|
||||
applications do not need to
|
||||
distinguish between shorter and longer documents (see <a
|
||||
href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">a "fair" similarity</a>).</p>
|
||||
|
||||
<p>To change <a href="Similarity.html">Similarity</a>, one must do so for both indexing and
|
||||
searching, and the changes must happen before
|
||||
either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it
|
||||
just isn't well-defined what is going to happen.
|
||||
</p>
|
||||
|
||||
<p>To make this change, implement your own <a href="Similarity.html">Similarity</a> (likely
|
||||
you'll want to simply subclass
|
||||
<a href="DefaultSimilarity.html">DefaultSimilarity</a>) and then use the new
|
||||
class by calling
|
||||
<a href="../index/IndexWriter.html#setSimilarity(org.apache.lucene.search.Similarity)">IndexWriter.setSimilarity</a>
|
||||
before indexing and
|
||||
<a href="Searcher.html#setSimilarity(org.apache.lucene.search.Similarity)">Searcher.setSimilarity</a>
|
||||
before searching.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at <a
|
||||
href="http://www.nabble.com/Overriding-Similarity-tf2128934.html">Overriding Similarity</a>.
|
||||
In summary, here are a few use cases:
|
||||
<ol>
|
||||
<li><p><a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> -- <a
|
||||
href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> gives small increases
|
||||
as the frequency increases a small amount
|
||||
and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is
|
||||
more significant.</p></li>
|
||||
<li><p>Overriding tf -- In some applications, it doesn't matter what the score of a document is as long as a
|
||||
matching term occurs. In these
|
||||
cases people have overridden Similarity to return 1 from the tf() method.</p></li>
|
||||
<li><p>Changing Length Normalization -- By overriding <a
|
||||
href="Similarity.html#lengthNorm(java.lang.String,%20int)">lengthNorm</a>,
|
||||
it is possible to discount how the length of a field contributes
|
||||
to a score. In <a href="DefaultSimilarity.html">DefaultSimilarity</a>,
|
||||
lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be
|
||||
1 / (numTerms in field), all fields will be treated
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">"fairly"</a>.</p></li>
|
||||
</ol>
|
||||
In general, Chris Hostetter sums it up best in saying (from <a
|
||||
href="http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125">the Lucene users's mailing list</a>):
|
||||
<blockquote>[One would override the Similarity in] ... any situation where you know more about your data then just
|
||||
that
|
||||
it's "text" is a situation where it *might* make sense to to override your
|
||||
Similarity method.</blockquote>
|
||||
</p>
|
||||
<a name="scoring"></a>
|
||||
<h2>Changing Scoring -- Expert Level</h2>
|
||||
|
||||
<p>Changing scoring is an expert level task, so tread carefully and be prepared to share your code if
|
||||
you want help.
|
||||
</p>
|
||||
|
||||
<p>With the warning out of the way, it is possible to change a lot more than just the Similarity
|
||||
when it comes to scoring in Lucene. Lucene's scoring is a complex mechanism that is grounded by
|
||||
<span >three main classes</span>:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="Query.html">Query</a> -- The abstract object representation of the
|
||||
user's information need.</li>
|
||||
<li>
|
||||
<a href="Weight.html">Weight</a> -- The internal interface representation of
|
||||
the user's Query, so that Query objects may be reused.</li>
|
||||
<li>
|
||||
<a href="Scorer.html">Scorer</a> -- An abstract class containing common
|
||||
functionality for scoring. Provides both scoring and explanation capabilities.</li>
|
||||
</ol>
|
||||
Details on each of these classes, and their children can be found in the subsections below.
|
||||
</p>
|
||||
<h4>The Query Class</h4>
|
||||
<p>In some sense, the
|
||||
<a href="Query.html">Query</a>
|
||||
class is where it all begins. Without a Query, there would be
|
||||
nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
|
||||
is often responsible
|
||||
for creating them or coordinating the functionality between them. The
|
||||
<a href="Query.html">Query</a> class has several methods that are important for
|
||||
derived classes:
|
||||
<ol>
|
||||
<li>createWeight(Searcher searcher) -- A
|
||||
<a href="Weight.html">Weight</a> is the internal representation of the
|
||||
Query, so each Query implementation must
|
||||
provide an implementation of Weight. See the subsection on <a
|
||||
href="#The Weight Interface">The Weight Interface</a> below for details on implementing the Weight
|
||||
interface.</li>
|
||||
<li>rewrite(IndexReader reader) -- Rewrites queries into primitive queries. Primitive queries are:
|
||||
<a href="TermQuery.html">TermQuery</a>,
|
||||
<a href="BooleanQuery.html">BooleanQuery</a>, <span
|
||||
>OTHERS????</span></li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>The Weight Interface</h4>
|
||||
<p>The
|
||||
<a href="Weight.html">Weight</a>
|
||||
interface provides an internal representation of the Query so that it can be reused. Any
|
||||
<a href="Searcher.html">Searcher</a>
|
||||
dependent state should be stored in the Weight implementation,
|
||||
not in the Query class. The interface defines 6 methods that must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="Weight.html#getQuery()">Weight#getQuery()</a> -- Pointer to the
|
||||
Query that this Weight represents.</li>
|
||||
<li>
|
||||
<a href="Weight.html#getValue()">Weight#getValue()</a> -- The weight for
|
||||
this Query. For example, the TermQuery.TermWeight value is
|
||||
equal to the idf^2 * boost * queryNorm <!-- DOUBLE CHECK THIS --></li>
|
||||
<li>
|
||||
<a href="Weight.html#sumOfSquaredWeights()">
|
||||
Weight#sumOfSquaredWeights()</a> -- The sum of squared weights. Tor TermQuery, this is (idf *
|
||||
boost)^2</li>
|
||||
<li>
|
||||
<a href="Weight.html#normalize(float)">
|
||||
Weight#normalize(float)</a> -- Determine the query normalization factor. The query normalization may
|
||||
allow for comparing scores between queries.</li>
|
||||
<li>
|
||||
<a href="Weight.html#scorer(IndexReader)">
|
||||
Weight#scorer(IndexReader)</a> -- Construct a new
|
||||
<a href="Scorer.html">Scorer</a>
|
||||
for this Weight. See
|
||||
<a href="#The Scorer Class">The Scorer Class</a>
|
||||
below for help defining a Scorer. As the name implies, the
|
||||
Scorer is responsible for doing the actual scoring of documents given the Query.
|
||||
</li>
|
||||
<li>
|
||||
<a href="Weight.html#explain(IndexReader, int)">
|
||||
Weight#explain(IndexReader, int)</a> -- Provide a means for explaining why a given document was
|
||||
scored
|
||||
the way it was.</li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>The Scorer Class</h4>
|
||||
<p>The
|
||||
<a href="Scorer.html">Scorer</a>
|
||||
abstract class provides common scoring functionality for all Scorer implementations and
|
||||
is the heart of the Lucene scoring process. The Scorer defines the following abstract methods which
|
||||
must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="Scorer.html#next()">Scorer#next()</a> -- Advances to the next
|
||||
document that matches this Query, returning true if and only
|
||||
if there is another document that matches.</li>
|
||||
<li>
|
||||
<a href="Scorer.html#doc()">Scorer#doc()</a> -- Returns the id of the
|
||||
<a href="../document/Document.html">Document</a>
|
||||
that contains the match. Is not valid until next() has been called at least once.
|
||||
</li>
|
||||
<li>
|
||||
<a href="Scorer.html#score()">Scorer#score()</a> -- Return the score of the
|
||||
current document. This value can be determined in any
|
||||
appropriate way for an application. For instance, the
|
||||
<a href="http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/search/TermScorer.java?view=log">TermScorer</a>
|
||||
returns the tf * Weight.getValue() * fieldNorm.
|
||||
</li>
|
||||
<li>
|
||||
<a href="Scorer.html#skipTo(int)">Scorer#skipTo(int)</a> -- Skip ahead in
|
||||
the document matches to the document whose id is greater than
|
||||
or equal to the passed in value. In many instances, skipTo can be
|
||||
implemented more efficiently than simply looping through all the matching documents until
|
||||
the target document is identified.</li>
|
||||
<li>
|
||||
<a href="Scorer.html#explain(int)">Scorer#explain(int)</a> -- Provides
|
||||
details on why the score came about.</li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>Why would I want to add my own Query?</h4>
|
||||
|
||||
<p>In a nutshell, you want to add your own custom Query implementation when you think that Lucene's
|
||||
aren't appropriate for the
|
||||
task that you want to do. You might be doing some cutting edge research or you need more information
|
||||
back
|
||||
out of Lucene (similar to Doug adding SpanQuery functionality).</p>
|
||||
<h4>Examples</h4>
|
||||
<p >FILL IN HERE</p>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
|
|
@ -184,281 +184,22 @@
|
|||
</p>
|
||||
</subsection>
|
||||
<subsection name="Query Classes">
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
</h4>
|
||||
<p>Of the various implementations of
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a>, the
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
is the easiest to understand and the most often used in applications. A <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> matches all the documents that contain the specified
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>,
|
||||
which is a word that occurs in a certain
|
||||
<a href="api/org/apache/lucene/document/Field.html">Field</a>.
|
||||
Thus, a <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> identifies and scores all
|
||||
<a href="api/org/apache/lucene/document/Document.html">Document</a>s that have a <a href="api/org/apache/lucene/document/Field.html">Field</a> with the specified string in it.
|
||||
Constructing a <a
|
||||
href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>
|
||||
is as simple as:
|
||||
<pre>
|
||||
TermQuery tq = new TermQuery(new Term("fieldName", "term");
|
||||
</pre>In this example, the <a href="api/org/apache/lucene/search/Query.html">Query</a> identifies all <a href="api/org/apache/lucene/document/Document.html">Document</a>s that have the <a href="api/org/apache/lucene/document/Field.html">Field</a> named <tt>"fieldName"</tt> and
|
||||
contain the word <tt>"term"</tt>.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>
|
||||
</h4>
|
||||
<p>Things start to get interesting when one combines multiple
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> instances into a <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>.
|
||||
A <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a> contains multiple
|
||||
<a href="api/org/apache/lucene/search/BooleanClause.html">BooleanClause</a>s,
|
||||
where each clause contains a sub-query (<a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
instance) and an operator (from <a href="api/org/apache/lucene/search/BooleanClause.Occur.html">BooleanClause.Occur</a>)
|
||||
describing how that sub-query is combined with the other clauses:
|
||||
<ol>
|
||||
|
||||
<li><p>SHOULD -- Use this operator when a clause can occur in the result set, but is not required.
|
||||
If a query is made up of all SHOULD clauses, then every document in the result
|
||||
set matches at least one of these clauses.</p></li>
|
||||
|
||||
<li><p>MUST -- Use this operator when a clause is required to occur in the result set. Every
|
||||
document in the result set will match
|
||||
all such clauses.</p></li>
|
||||
|
||||
<li><p>MUST NOT -- Use this operator when a
|
||||
clause must not occur in the result set. No
|
||||
document in the result set will match
|
||||
any such clauses.</p></li>
|
||||
</ol>
|
||||
Boolean queries are constructed by adding two or more
|
||||
<a href="api/org/apache/lucene/search/BooleanClause.html">BooleanClause</a>
|
||||
instances. If too many clauses are added, a <a href="api/org/apache/lucene/search/BooleanQuery.TooManyClauses.html">TooManyClauses</a>
|
||||
exception will be thrown during searching. This most often occurs
|
||||
when a <a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
is rewritten into a <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a> with many
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> clauses,
|
||||
for example by <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>.
|
||||
The default setting for the maximum number
|
||||
of clauses 1024, but this can be changed via the
|
||||
static method <a href="api/org/apache/lucene/search/BooleanQuery.html#setMaxClauseCount(int)">setMaxClauseCount</a>
|
||||
in <a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>.
|
||||
</p>
|
||||
|
||||
<h4>Phrases</h4>
|
||||
<p>Another common search is to find documents containing certain phrases. This
|
||||
is handled in two different ways.
|
||||
<ol>
|
||||
<li>
|
||||
<p><a href="api/org/apache/lucene/search/PhraseQuery.html">PhraseQuery</a>
|
||||
-- Matches a sequence of
|
||||
<a href="api/org/apache/lucene/index/Term.html">Terms</a>.
|
||||
<a href="api/org/apache/lucene/search/PhraseQuery.html">PhraseQuery</a> uses a slop factor to determine
|
||||
how many positions may occur between any two terms in the phrase and still be considered a match.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><a href="api/org/apache/lucene/search/spans/SpanNearQuery.html">SpanNearQuery</a>
|
||||
-- Matches a sequence of other
|
||||
<a href="api/org/apache/lucene/search/spans/SpanQuery.html">SpanQuery</a>
|
||||
instances. <a href="api/org/apache/lucene/search/spans/SpanNearQuery.html">SpanNearQuery</a> allows for much more
|
||||
complicated phrase queries since it is constructed from other to <a href="api/org/apache/lucene/search/spans/SpanQuery.html">SpanQuery</a>
|
||||
instances, instead of only <a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a> instances.</p>
|
||||
</li>
|
||||
</ol>
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/RangeQuery.html">RangeQuery</a>
|
||||
</h4>
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/RangeQuery.html">RangeQuery</a>
|
||||
matches all documents that occur in the
|
||||
exclusive range of a lower
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>
|
||||
and an upper
|
||||
<a href="api/org/apache/lucene/index/Term.html">Term</a>.
|
||||
For example, one could find all documents
|
||||
that have terms beginning with the letters <tt>a</tt> through <tt>c</tt>. This type of <a href="api/org/apache/lucene/search/Query.html">Query</a> is frequently used to
|
||||
find
|
||||
documents that occur in a specific date range.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a>,
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>
|
||||
</h4>
|
||||
<p>While the
|
||||
<a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a>
|
||||
has a different implementation, it is essentially a special case of the
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a>.
|
||||
The <a href="api/org/apache/lucene/search/PrefixQuery.html">PrefixQuery</a> allows an application
|
||||
to identify all documents with terms that begin with a certain string. The <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> generalizes this by allowing
|
||||
for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> (matches exactly one character) wildcards. Note that the <a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> can be quite slow. Also note that
|
||||
<a href="api/org/apache/lucene/search/WildcardQuery.html">WildcardQuery</a> should
|
||||
not start with <tt>*</tt> and <tt>?</tt>, as these are extremely slow. For tricks on how to search using a wildcard at
|
||||
the beginning of a term, see
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/13373#13373">
|
||||
Starts With x and Ends With x Queries</a>
|
||||
from the Lucene users's mailing list.
|
||||
</p>
|
||||
<h4>
|
||||
<a href="api/org/apache/lucene/search/FuzzyQuery.html">FuzzyQuery</a>
|
||||
</h4>
|
||||
<p>A
|
||||
<a href="api/org/apache/lucene/search/FuzzyQuery.html">FuzzyQuery</a>
|
||||
matches documents that contain terms similar to the specified term. Similarity is
|
||||
determined using
|
||||
<a href="http://en.wikipedia.org/wiki/Levenshtein">Levenshtein (edit) distance</a>.
|
||||
This type of query can be useful when accounting for spelling variations in the collection.
|
||||
<p>For information on the Query Classes, refer to the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#query">search package javadocs</a>
|
||||
</p>
|
||||
</subsection>
|
||||
<subsection name="Changing Similarity">
|
||||
<p>Chances are <a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a> is sufficient for all your searching needs.
|
||||
However, in some applications it may be necessary to customize your <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a> implementation. For instance, some applications do not need to
|
||||
distinguish between shorter and longer documents (see <a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">a "fair" similarity</a>).</p>
|
||||
|
||||
<p>To change <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a>, one must do so for both indexing and searching, and the changes must happen before
|
||||
either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen.
|
||||
</p>
|
||||
|
||||
<p>To make this change, implement your own <a href="api/org/apache/lucene/search/Similarity.html">Similarity</a> (likely you'll want to simply subclass
|
||||
<a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a>) and then use the new
|
||||
class by calling
|
||||
<a href="api/org/apache/lucene/index/IndexWriter.html#setSimilarity(org.apache.lucene.search.Similarity)">IndexWriter.setSimilarity</a> before indexing and
|
||||
<a href="api/org/apache/lucene/search/Searcher.html#setSimilarity(org.apache.lucene.search.Similarity)">Searcher.setSimilarity</a> before searching.
|
||||
</p>
|
||||
<p>
|
||||
If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at <a href="http://www.nabble.com/Overriding-Similarity-tf2128934.html">Overriding Similarity</a>.
|
||||
In summary, here are a few use cases:
|
||||
<ol>
|
||||
<li><p><a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> -- <a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> gives small increases as the frequency increases a small amount
|
||||
and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.</p></li>
|
||||
<li><p>Overriding tf -- In some applications, it doesn't matter what the score of a document is as long as a matching term occurs. In these
|
||||
cases people have overridden Similarity to return 1 from the tf() method.</p></li>
|
||||
<li><p>Changing Length Normalization -- By overriding <a href="api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String,%20int)">lengthNorm</a>, it is possible to discount how the length of a field contributes
|
||||
to a score. In <a href="api/org/apache/lucene/search/DefaultSimilarity.html">DefaultSimilarity</a>, lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be
|
||||
1 / (numTerms in field), all fields will be treated
|
||||
<a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">"fairly"</a>.</p></li>
|
||||
</ol>
|
||||
In general, Chris Hostetter sums it up best in saying (from <a href="http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125">the Lucene users's mailing list</a>):
|
||||
<blockquote>[One would override the Similarity in] ... any situation where you know more about your data then just that
|
||||
it's "text" is a situation where it *might* make sense to to override your
|
||||
Similarity method.</blockquote>
|
||||
</p>
|
||||
<p>One of the ways of changing the scoring characteristics of Lucene is to change the similarity factors. For information on
|
||||
how to do this, see the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#changingSimilarity">search package javadocs</a></p>
|
||||
</subsection>
|
||||
|
||||
</section>
|
||||
<section name="Changing your Scoring -- Expert Level">
|
||||
<p>Changing scoring is an expert level task, so tread carefully and be prepared to share your code if
|
||||
you want help.
|
||||
<p>At a much deeper level, one can affect scoring by implementing their own Query classes (and related scoring classes.) To learn more
|
||||
about how to do this, refer to the
|
||||
<a href="api/org/apache/lucene/search/package-summary.html#scoring">search package javadocs</a>
|
||||
</p>
|
||||
<p>With the warning out of the way, it is possible to change a lot more than just the Similarity
|
||||
when it comes to scoring in Lucene. Lucene's scoring is a complex mechanism that is grounded by
|
||||
<span class="highlight-for-editing">three main classes</span>:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a> -- The abstract object representation of the user's information need.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a> -- The internal interface representation of the user's Query, so that Query objects may be reused.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a> -- An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities.</li>
|
||||
</ol>
|
||||
Details on each of these classes, and their children can be found in the subsections below.
|
||||
</p>
|
||||
<subsection name="The Query Class">
|
||||
<p>In some sense, the
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a>
|
||||
class is where it all begins. Without a Query, there would be
|
||||
nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
|
||||
is often responsible
|
||||
for creating them or coordinating the functionality between them. The
|
||||
<a href="api/org/apache/lucene/search/Query.html">Query</a> class has several methods that are important for
|
||||
derived classes:
|
||||
<ol>
|
||||
<li>createWeight(Searcher searcher) -- A
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a> is the internal representation of the Query, so each Query implementation must
|
||||
provide an implementation of Weight. See the subsection on <a
|
||||
href="#The Weight Interface">The Weight Interface</a> below for details on implementing the Weight interface.</li>
|
||||
<li>rewrite(IndexReader reader) -- Rewrites queries into primitive queries. Primitive queries are:
|
||||
<a href="api/org/apache/lucene/search/TermQuery.html">TermQuery</a>,
|
||||
<a href="api/org/apache/lucene/search/BooleanQuery.html">BooleanQuery</a>, <span class="highlight-for-editing">OTHERS????</span></li>
|
||||
</ol>
|
||||
</p>
|
||||
</subsection>
|
||||
<subsection name="The Weight Interface">
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/Weight.html">Weight</a>
|
||||
interface provides an internal representation of the Query so that it can be reused. Any
|
||||
<a href="api/org/apache/lucene/search/Searcher.html">Searcher</a>
|
||||
dependent state should be stored in the Weight implementation,
|
||||
not in the Query class. The interface defines 6 methods that must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#getQuery()">Weight#getQuery()</a> -- Pointer to the Query that this Weight represents.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#getValue()">Weight#getValue()</a> -- The weight for this Query. For example, the TermQuery.TermWeight value is
|
||||
equal to the idf^2 * boost * queryNorm <!-- DOUBLE CHECK THIS --></li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#sumOfSquaredWeights()">
|
||||
Weight#sumOfSquaredWeights()</a> -- The sum of squared weights. Tor TermQuery, this is (idf *
|
||||
boost)^2</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#normalize(float)">
|
||||
Weight#normalize(float)</a> -- Determine the query normalization factor. The query normalization may
|
||||
allow for comparing scores between queries.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#scorer(IndexReader)">
|
||||
Weight#scorer(IndexReader)</a> -- Construct a new
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a>
|
||||
for this Weight. See
|
||||
<a href="#The Scorer Class">The Scorer Class</a>
|
||||
below for help defining a Scorer. As the name implies, the
|
||||
Scorer is responsible for doing the actual scoring of documents given the Query.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Weight.html#explain(IndexReader, int)">
|
||||
Weight#explain(IndexReader, int)</a> -- Provide a means for explaining why a given document was scored
|
||||
the way it was.</li>
|
||||
</ol>
|
||||
</p>
|
||||
</subsection>
|
||||
<subsection name="The Scorer Class">
|
||||
<p>The
|
||||
<a href="api/org/apache/lucene/search/Scorer.html">Scorer</a>
|
||||
abstract class provides common scoring functionality for all Scorer implementations and
|
||||
is the heart of the Lucene scoring process. The Scorer defines the following abstract methods which
|
||||
must be implemented:
|
||||
<ol>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#next()">Scorer#next()</a> -- Advances to the next document that matches this Query, returning true if and only
|
||||
if there is another document that matches.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#doc()">Scorer#doc()</a> -- Returns the id of the
|
||||
<a href="api/org/apache/lucene/document/Document.html">Document</a>
|
||||
that contains the match. Is not valid until next() has been called at least once.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#score()">Scorer#score()</a> -- Return the score of the current document. This value can be determined in any
|
||||
appropriate way for an application. For instance, the
|
||||
<a href="http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/search/TermScorer.java?view=log">TermScorer</a>
|
||||
returns the tf * Weight.getValue() * fieldNorm.
|
||||
</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#skipTo(int)">Scorer#skipTo(int)</a> -- Skip ahead in the document matches to the document whose id is greater than
|
||||
or equal to the passed in value. In many instances, skipTo can be
|
||||
implemented more efficiently than simply looping through all the matching documents until
|
||||
the target document is identified.</li>
|
||||
<li>
|
||||
<a href="api/org/apache/lucene/search/Scorer.html#explain(int)">Scorer#explain(int)</a> -- Provides details on why the score came about.</li>
|
||||
</ol>
|
||||
</p>
|
||||
</subsection>
|
||||
<subsection name="Why would I want to add my own Query?">
|
||||
<p>In a nutshell, you want to add your own custom Query implementation when you think that Lucene's
|
||||
aren't appropriate for the
|
||||
task that you want to do. You might be doing some cutting edge research or you need more information
|
||||
back
|
||||
out of Lucene (similar to Doug adding SpanQuery functionality).</p>
|
||||
</subsection>
|
||||
<subsection name="Examples">
|
||||
<p class="highlight-for-editing">FILL IN HERE</p>
|
||||
</subsection>
|
||||
</section>
|
||||
|
||||
<section name="Appendix">
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@
|
|||
<item name="Getting Started" href="/gettingstarted.html"/>
|
||||
<item name="Query Syntax" href="/queryparsersyntax.html"/>
|
||||
<item name="File Formats" href="/fileformats.html"/>
|
||||
<item name="Scoring" href="/scoring.html"/>
|
||||
<item name="Javadoc" href="/api/index.html"/>
|
||||
<item name="Contributions" href="/contributions.html"/>
|
||||
<item name="Benchmarks" href="/benchmarks.html"/>
|
||||
|
|
|
|||
Loading…
Reference in New Issue