2010-05-05 12:27:58 -04:00
|
|
|
Analysis Module Change Log
|
|
|
|
|
|
2011-05-31 09:03:40 -04:00
|
|
|
For more information on past and future Lucene versions, please see:
|
|
|
|
|
http://s.apache.org/luceneversions
|
|
|
|
|
|
2010-05-05 12:27:58 -04:00
|
|
|
======================= Trunk (not yet released) =======================
|
2010-10-31 09:56:46 -04:00
|
|
|
|
2010-05-05 12:27:58 -04:00
|
|
|
API Changes
|
|
|
|
|
|
|
|
|
|
* LUCENE-2413: Deprecated PatternAnalyzer in common/miscellaneous, in favor
|
|
|
|
|
of the pattern package (CharFilter, Tokenizer, TokenFilter). (Robert Muir)
|
|
|
|
|
|
|
|
|
|
* LUCENE-2413: Removed the AnalyzerUtil in common/miscellaneous. (Robert Muir)
|
|
|
|
|
|
2011-01-06 08:51:10 -05:00
|
|
|
* LUCENE-2167,LUCENE-2699,LUCENE-2763,LUCENE-2847: StandardTokenizer/Analyzer
|
|
|
|
|
in common/standard/ now implement the Word Break rules from the Unicode 6.0.0
|
|
|
|
|
Text Segmentation algorithm (UAX#29), covering the full range of Unicode code
|
|
|
|
|
points, including values from U+FFFF to U+10FFFF
|
2010-10-15 01:41:54 -04:00
|
|
|
|
2011-01-06 08:51:10 -05:00
|
|
|
ClassicTokenizer/Analyzer retains the old (pre-Lucene 3.1) StandardTokenizer/
|
|
|
|
|
Analyzer implementation and behavior. Only the Unicode Basic Multilingual
|
|
|
|
|
Plane (code points from U+0000 to U+FFFF) is covered.
|
2010-12-07 09:53:13 -05:00
|
|
|
|
|
|
|
|
UAX29URLEmailTokenizer tokenizes URLs and E-mail addresses according to the
|
|
|
|
|
relevant RFCs, in addition to implementing the UAX#29 Word Break rules.
|
|
|
|
|
(Steven Rowe, Robert Muir, Uwe Schindler)
|
|
|
|
|
|
2010-10-09 12:55:23 -04:00
|
|
|
* LUCENE-1370: Added ShingleFilter option to output unigrams if no shingles
|
|
|
|
|
can be generated. (Chris Harris via Steven Rowe)
|
|
|
|
|
|
2011-02-28 00:15:50 -05:00
|
|
|
* LUCENE-2514, LUCENE-2551: JDK and ICU CollationKeyAnalyzers were changed to
|
|
|
|
|
use pure byte keys when Version >= 4.0. This cuts sort key size approximately
|
|
|
|
|
in half. (Robert Muir)
|
2011-08-25 06:32:21 -04:00
|
|
|
|
|
|
|
|
* LUCENE-3400: Removed DutchAnalyzer.setStemDictionary (Chris Male)
|
2011-09-07 00:43:10 -04:00
|
|
|
|
2011-09-13 23:33:50 -04:00
|
|
|
* LUCENE-3431: Removed QueryAutoStopWordAnalyzer.addStopWords* deprecated methods
|
|
|
|
|
since they prevented reuse. Stopwords are now generated at instantiation through
|
|
|
|
|
the Analyzer's constructors. (Chris Male)
|
|
|
|
|
|
2011-09-14 23:21:17 -04:00
|
|
|
* LUCENE-3434: Removed ShingleAnalyzerWrapper.set* and PerFieldAnalyzerWrapper.addAnalyzer
|
|
|
|
|
since they prevent reuse. Both Analyzers should be configured at instantiation.
|
|
|
|
|
(Chris Male)
|
|
|
|
|
|
2010-05-05 12:27:58 -04:00
|
|
|
New Features
|
2011-06-30 15:12:54 -04:00
|
|
|
|
|
|
|
|
* LUCENE-2341: A new analyzer/ filter: Morfologik - a dictionary-driven lemmatizer
|
|
|
|
|
(accurate stemmer) for Polish (includes morphosyntactic annotations).
|
|
|
|
|
(Michał Dybizbański, Dawid Weiss)
|
|
|
|
|
|
2010-05-05 12:27:58 -04:00
|
|
|
* LUCENE-2413: Consolidated Solr analysis components into common.
|
|
|
|
|
New features from Solr now available to Lucene users include:
|
|
|
|
|
- o.a.l.analysis.commongrams: Constructs n-grams for frequently occurring terms
|
|
|
|
|
and phrases.
|
|
|
|
|
- o.a.l.analysis.charfilter.HTMLStripCharFilter: CharFilter that strips HTML
|
|
|
|
|
constructs.
|
|
|
|
|
- o.a.l.analysis.miscellaneous.WordDelimiterFilter: TokenFilter that splits words
|
|
|
|
|
into subwords and performs optional transformations on subword groups.
|
|
|
|
|
- o.a.l.analysis.miscellaneous.RemoveDuplicatesTokenFilter: TokenFilter which
|
|
|
|
|
filters out Tokens at the same position and Term text as the previous token.
|
|
|
|
|
- o.a.l.analysis.miscellaneous.TrimFilter: Trims leading and trailing whitespace
|
|
|
|
|
from Tokens in the stream.
|
|
|
|
|
- o.a.l.analysis.miscellaneous.KeepWordFilter: A TokenFilter that only keeps tokens
|
|
|
|
|
with text contained in the required words (inverse of StopFilter).
|
|
|
|
|
- o.a.l.analysis.miscellaneous.HyphenatedWordsFilter: A TokenFilter that puts
|
|
|
|
|
hyphenated words broken into two lines back together.
|
2010-06-23 07:25:17 -04:00
|
|
|
- o.a.l.analysis.miscellaneous.CapitalizationFilter: A TokenFilter that applies
|
|
|
|
|
capitalization rules to tokens.
|
2010-05-05 12:27:58 -04:00
|
|
|
- o.a.l.analysis.pattern: Package for pattern-based analysis, containing a
|
|
|
|
|
CharFilter, Tokenizer, and Tokenfilter for transforming text with regexes.
|
2010-05-20 06:46:00 -04:00
|
|
|
- o.a.l.analysis.synonym.SynonymFilter: A synonym filter that supports multi-word
|
|
|
|
|
synonyms.
|
2010-06-23 07:25:17 -04:00
|
|
|
- o.a.l.analysis.phonetic: Package for phonetic search, containing various
|
|
|
|
|
phonetic encoders such as Double Metaphone.
|
2010-05-05 12:27:58 -04:00
|
|
|
|
|
|
|
|
* LUCENE-2413: Consolidated all Lucene analyzers into common.
|
2010-05-25 16:16:44 -04:00
|
|
|
- o.a.l.analysis.KeywordAnalyzer -> o.a.l.analysis.core.KeywordAnalyzer
|
|
|
|
|
- o.a.l.analysis.KeywordTokenizer -> o.a.l.analysis.core.KeywordTokenizer
|
|
|
|
|
- o.a.l.analysis.LetterTokenizer -> o.a.l.analysis.core.LetterTokenizer
|
|
|
|
|
- o.a.l.analysis.LowerCaseFilter -> o.a.l.analysis.core.LowerCaseFilter
|
|
|
|
|
- o.a.l.analysis.LowerCaseTokenizer -> o.a.l.analysis.core.LowerCaseTokenizer
|
|
|
|
|
- o.a.l.analysis.SimpleAnalyzer -> o.a.l.analysis.core.SimpleAnalyzer
|
|
|
|
|
- o.a.l.analysis.StopAnalyzer -> o.a.l.analysis.core.StopAnalyzer
|
|
|
|
|
- o.a.l.analysis.StopFilter -> o.a.l.analysis.core.StopFilter
|
|
|
|
|
- o.a.l.analysis.WhitespaceAnalyzer -> o.a.l.analysis.core.WhitespaceAnalyzer
|
|
|
|
|
- o.a.l.analysis.WhitespaceTokenizer -> o.a.l.analysis.core.WhitespaceTokenizer
|
2010-05-05 12:27:58 -04:00
|
|
|
- o.a.l.analysis.PorterStemFilter -> o.a.l.analysis.en.PorterStemFilter
|
|
|
|
|
- o.a.l.analysis.ASCIIFoldingFilter -> o.a.l.analysis.miscellaneous.ASCIIFoldingFilter
|
|
|
|
|
- o.a.l.analysis.ISOLatin1AccentFilter -> o.a.l.analysis.miscellaneous.ISOLatin1AccentFilter
|
2010-05-20 09:23:12 -04:00
|
|
|
- o.a.l.analysis.KeywordMarkerFilter -> o.a.l.analysis.miscellaneous.KeywordMarkerFilter
|
2010-05-05 12:27:58 -04:00
|
|
|
- o.a.l.analysis.LengthFilter -> o.a.l.analysis.miscellaneous.LengthFilter
|
|
|
|
|
- o.a.l.analysis.PerFieldAnalyzerWrapper -> o.a.l.analysis.miscellaneous.PerFieldAnalyzerWrapper
|
|
|
|
|
- o.a.l.analysis.TeeSinkTokenFilter -> o.a.l.analysis.sinks.TeeSinkTokenFilter
|
2010-05-25 18:28:32 -04:00
|
|
|
- o.a.l.analysis.CharFilter -> o.a.l.analysis.charfilter.CharFilter
|
2010-05-05 12:27:58 -04:00
|
|
|
- o.a.l.analysis.BaseCharFilter -> o.a.l.analysis.charfilter.BaseCharFilter
|
|
|
|
|
- o.a.l.analysis.MappingCharFilter -> o.a.l.analysis.charfilter.MappingCharFilter
|
|
|
|
|
- o.a.l.analysis.NormalizeCharMap -> o.a.l.analysis.charfilter.NormalizeCharMap
|
2010-05-25 18:28:32 -04:00
|
|
|
- o.a.l.analysis.CharArraySet -> o.a.l.analysis.util.CharArraySet
|
|
|
|
|
- o.a.l.analysis.CharArrayMap -> o.a.l.analysis.util.CharArrayMap
|
2010-05-25 16:16:44 -04:00
|
|
|
- o.a.l.analysis.ReusableAnalyzerBase -> o.a.l.analysis.util.ReusableAnalyzerBase
|
|
|
|
|
- o.a.l.analysis.StopwordAnalyzerBase -> o.a.l.analysis.util.StopwordAnalyzerBase
|
|
|
|
|
- o.a.l.analysis.WordListLoader -> o.a.l.analysis.util.WordListLoader
|
2011-05-02 20:29:47 -04:00
|
|
|
- o.a.l.analysis.CharTokenizer -> o.a.l.analysis.util.CharTokenizer
|
|
|
|
|
- o.a.l.util.CharacterUtils -> o.a.l.analysis.util.CharacterUtils
|
2010-05-05 12:27:58 -04:00
|
|
|
|
2011-02-04 05:19:52 -05:00
|
|
|
* SOLR-1057: Add PathHierarchyTokenizer that represents file path hierarchies as synonyms of
|
|
|
|
|
/something, /something/something, /something/something/else. (Ryan McKinley, Koji Sekiguchi)
|
|
|
|
|
|
2011-09-10 02:00:39 -04:00
|
|
|
* LUCENE-3414: Added HunspellStemFilter which uses a provided pure Java implementation of the
|
|
|
|
|
Hunspell algorithm. (Chris Male)
|
|
|
|
|
|
2010-05-05 12:27:58 -04:00
|
|
|
Build
|
|
|
|
|
|
2010-05-20 06:46:00 -04:00
|
|
|
* LUCENE-2413: All analyzers in contrib/analyzers and contrib/icu were moved to the
|
2010-05-20 09:23:12 -04:00
|
|
|
analysis module. The 'smartcn' and 'stempel' components now depend on 'common'.
|
|
|
|
|
(Robert Muir)
|
2011-09-12 01:50:26 -04:00
|
|
|
|
|
|
|
|
* LUCENE-3376: Moved ReusableAnalyzerBase into lucene core. (Chris Male)
|
