Michael McCandless
|
87de0c9688
|
LUCENE-1466: added chainable CharFilter stage before Tokenizer to allow mapping of characters before tokenization
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@787795 13f79535-47bb-0310-9956-ffa450edef68
|
2009-06-23 19:15:31 +00:00 |
|
Michael McCandless
|
ec8088654d
|
bulk fix svn:eol-style to native for text files
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@787424 13f79535-47bb-0310-9956-ffa450edef68
|
2009-06-22 22:18:56 +00:00 |
|
Michael McCandless
|
bb6b711718
|
LUCENE-1333: improvements to Token reuse API and full cutover to reuse API for all core and contrib analyzers
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@687357 13f79535-47bb-0310-9956-ffa450edef68
|
2008-08-20 14:38:07 +00:00 |
|
Michael McCandless
|
44daebab7c
|
LUCENE-1150: put back public tokenImage/TOKEN_TYPES in StandardTokenizer and WikipediaTokenizer
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@648183 13f79535-47bb-0310-9956-ffa450edef68
|
2008-04-15 08:48:41 +00:00 |
|
Michael McCandless
|
38f8d4e837
|
LUCENE-1150: make StandardAnalyzer tokenizer constants public again (public access was accidentally removed with LUCENE-966)
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@616248 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-29 10:51:44 +00:00 |
Grant Ingersoll
|
7f09c9f00e
|
Added javadocs about UNTOKENIZED_TOKEN_FLAG conundrum
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@614905 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-24 15:42:50 +00:00 |
|
Grant Ingersoll
|
305c47f500
|
LUCENE-1133: Adds ability to keep certain strings as single tokens
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@614895 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-24 15:05:53 +00:00 |
|
Grant Ingersoll
|
0a54ca1920
|
LUCENE-1135. Added Javadoc marking the WikipediaTokenizer as experimental
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@612244 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-15 22:01:21 +00:00 |
|
Grant Ingersoll
|
79e09db401
|
LUCENE-1103: Internal links should increment as all tokens do, since the first token is valid too
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608989 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-04 20:36:23 +00:00 |
|
Grant Ingersoll
|
b18f6ae959
|
LUCENE-1103: The link is now incremented 1, but then the next token in the link is not incremented. This way, the link is not associated with the previous term. Instead it associated with the next term in the link, which would be the display tokens. If there are no display tokens, then it will take it's proper place in the token chain.
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608978 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-04 20:15:22 +00:00 |
|
Grant Ingersoll
|
f715fc6031
|
LUCENE-1103
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608852 13f79535-47bb-0310-9956-ffa450edef68
|
2008-01-04 14:29:15 +00:00 |