78e45c92a7
LUCENE-2207: CJKTokenizer generates tokens with incorrect offsets
...
LUCENE-2219: Chinese, SmartChinese, Wikipedia tokenizers generate incorrect offsets, test end() in BaseTokenStreamTestCase
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@900196 13f79535-47bb-0310-9956-ffa450edef68
2010-01-17 19:25:57 +00:00
3f722b66a5
LUCENE-2211: Fix various missing clearAttributes() and improve BaseTokenStreamTestCase to check for this trap
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@899627 13f79535-47bb-0310-9956-ffa450edef68
2010-01-15 13:42:18 +00:00
b756112999
LUCENE-1257: Generics:
...
demo,contrib/swing,contrib/wikipedia,contrib/wordnet,contrib/xml-query-parser
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@834414 13f79535-47bb-0310-9956-ffa450edef68
2009-11-10 10:18:26 +00:00
786eb6ce0d
LUCENE-2012: add remaining @overrides (contrib,demo)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@833867 13f79535-47bb-0310-9956-ffa450edef68
2009-11-08 12:45:12 +00:00
066eac49a4
LUCENE-2022: remove deprecated api from contrib/analysis and wikipedia
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@831425 13f79535-47bb-0310-9956-ffa450edef68
2009-10-30 19:04:30 +00:00
4cded8042c
LUCENE-1946, LUCENE-1753: Remove deprecated TokenStream API. What a pity, my wonderful backwards layer is gone! :-( Enforce decorator pattern by making the rest of TokenStreams final.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@824116 13f79535-47bb-0310-9956-ffa450edef68
2009-10-11 17:35:09 +00:00
af0e97fd72
LUCENE-1257: Replace StringBuffer by StringBuilder where possible
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@821185 13f79535-47bb-0310-9956-ffa450edef68
2009-10-02 22:11:10 +00:00
ec90bc2202
LUCENE-1855: Change AttributeSource API to use generics
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@820553 13f79535-47bb-0310-9956-ffa450edef68
2009-10-01 07:53:43 +00:00
4666489857
LUCENE-1906: Fix backwards problems with CharStream and Tokenizers with custom reset(Reader) method.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@813671 13f79535-47bb-0310-9956-ffa450edef68
2009-09-11 06:12:13 +00:00
c56f4c224f
LUCENE-1884: massive javadoc and comment cleanup -- primarily dealing with typos
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@811070 13f79535-47bb-0310-9956-ffa450edef68
2009-09-03 18:31:41 +00:00
a02eb9ee42
LUCENE-1876: add missing package.html to some contribs
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@810923 13f79535-47bb-0310-9956-ffa450edef68
2009-09-03 13:02:16 +00:00
26c5af3a33
LUCENE-1865 Add a ton of missing license headers throughout test/demo/contrib
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@808567 13f79535-47bb-0310-9956-ffa450edef68
2009-08-27 18:48:16 +00:00
56985e3136
add overview.html stubs for contrib to avoid error messages when building javadoc
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@808233 13f79535-47bb-0310-9956-ffa450edef68
2009-08-27 00:23:09 +00:00
a1bd9277ca
fix last attempted AttributeFactory link fix and update some other simple javadoc
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@807841 13f79535-47bb-0310-9956-ffa450edef68
2009-08-25 22:27:31 +00:00
1e4c3cdf86
{@link AttributeFactory} should be {@link AttributeSource.AttributeFactory}
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@807809 13f79535-47bb-0310-9956-ffa450edef68
2009-08-25 21:26:35 +00:00
367b35f0cb
LUCENE-1843: Update contrib tests to conform to onlyUseNewAPI; refactored assertAnalyzesTo and others into the new BaseTokenStreamTestCase class; Rewrote TestMappingCharFilter to use the new assert functions, too; performance improvements of Token.copyTo(); new impl of SingleTokenTokenStream
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@807190 13f79535-47bb-0310-9956-ffa450edef68
2009-08-24 12:44:13 +00:00
64ed5f39a5
LUCENE-1826: Add constructors that take AttributeSource and AttributeFactory to all Tokenizer implementations.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@806942 13f79535-47bb-0310-9956-ffa450edef68
2009-08-23 08:34:22 +00:00
b16e0aa31b
LUCENE-1801: All Tokenizers/TokenStreams that are source of tokens call AttributeSource.clearAttributes() first. Made Token.clear() consistent to AttributeImpl (clear everything)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@804392 13f79535-47bb-0310-9956-ffa450edef68
2009-08-14 22:01:42 +00:00
1743081b07
LUCENE-1460: Changed TokenStreams/TokenFilters in contrib to use the new TokenStream API.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@799953 13f79535-47bb-0310-9956-ffa450edef68
2009-08-01 22:52:32 +00:00
63402f49c7
Javadoc updates
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@795553 13f79535-47bb-0310-9956-ffa450edef68
2009-07-19 15:06:57 +00:00
87de0c9688
LUCENE-1466: added chainable CharFilter stage before Tokenizer to allow mapping of characters before tokenization
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@787795 13f79535-47bb-0310-9956-ffa450edef68
2009-06-23 19:15:31 +00:00
ec8088654d
bulk fix svn:eol-style to native for text files
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@787424 13f79535-47bb-0310-9956-ffa450edef68
2009-06-22 22:18:56 +00:00
bb6b711718
LUCENE-1333: improvements to Token reuse API and full cutover to reuse API for all core and contrib analyzers
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@687357 13f79535-47bb-0310-9956-ffa450edef68
2008-08-20 14:38:07 +00:00
44daebab7c
LUCENE-1150: put back public tokenImage/TOKEN_TYPES in StandardTokenizer and WikipediaTokenizer
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@648183 13f79535-47bb-0310-9956-ffa450edef68
2008-04-15 08:48:41 +00:00
38f8d4e837
LUCENE-1150: make StandardAnalyzer tokenizer constants public again (public access was accidentally removed with LUCENE-966)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@616248 13f79535-47bb-0310-9956-ffa450edef68
2008-01-29 10:51:44 +00:00
7f09c9f00e
Added javadocs about UNTOKENIZED_TOKEN_FLAG conundrum
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@614905 13f79535-47bb-0310-9956-ffa450edef68
2008-01-24 15:42:50 +00:00
305c47f500
LUCENE-1133: Adds ability to keep certain strings as single tokens
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@614895 13f79535-47bb-0310-9956-ffa450edef68
2008-01-24 15:05:53 +00:00
0a54ca1920
LUCENE-1135. Added Javadoc marking the WikipediaTokenizer as experimental
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@612244 13f79535-47bb-0310-9956-ffa450edef68
2008-01-15 22:01:21 +00:00
79e09db401
LUCENE-1103: Internal links should increment as all tokens do, since the first token is valid too
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608989 13f79535-47bb-0310-9956-ffa450edef68
2008-01-04 20:36:23 +00:00
b18f6ae959
LUCENE-1103: The link is now incremented 1, but then the next token in the link is not incremented. This way, the link is not associated with the previous term. Instead it associated with the next term in the link, which would be the display tokens. If there are no display tokens, then it will take it's proper place in the token chain.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608978 13f79535-47bb-0310-9956-ffa450edef68
2008-01-04 20:15:22 +00:00
f715fc6031
LUCENE-1103
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@608852 13f79535-47bb-0310-9956-ffa450edef68
2008-01-04 14:29:15 +00:00
Robert Muir
Uwe Schindler
Chris M. Hostetter
Mark Robert Miller
Michael Busch
Grant Ingersoll
Michael McCandless