f597b9a1cc
LUCENE-3883: Irish Analyzer
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1304836 13f79535-47bb-0310-9956-ffa450edef68
2012-03-24 15:59:04 +00:00
63f1c48b7d
Added katakana stem filter (LUCENE-3901)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1304719 13f79535-47bb-0310-9956-ffa450edef68
2012-03-24 06:38:53 +00:00
7291d38535
LUCENE-3905: sometimes run real-ish content (from LineFileDocs) through the analyzers too; fix end() offset bugs in the ngram tokenizers/filters
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1304525 13f79535-47bb-0310-9956-ffa450edef68
2012-03-23 17:39:13 +00:00
86c2da0eac
happy new year
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303828 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 15:21:17 +00:00
c3305a50ff
add some more kuromoji javadocs
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303746 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 12:21:48 +00:00
d2eebf9330
Fix for LUCENE-3897 (KuromojiTokenizer fails with large docs)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303739 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 11:41:54 +00:00
a6fd306dfb
add missing license headers
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303738 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 11:33:45 +00:00
1a191f4edc
LUCENE-3898: reset() was missing some state
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303441 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 15:22:28 +00:00
fb395f66a3
use MockTokenizer instead of WhitespaceTokenizer for better testing
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303382 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 13:10:38 +00:00
595744089a
LUCENE-3896: CharacterUtils.fill must call Reader.read again if it only got a single high surrogate char on the first read
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303374 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 12:53:27 +00:00
f75d40dad5
LUCENE-3894: try toning down for this tokenizer (it builds lots of tokens from the input treated as a path)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303276 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 04:30:11 +00:00
1156de050f
LUCENE-3894: add large docs tests for more tokenizers
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303273 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 03:59:14 +00:00
dd7bfc78d9
LUCENE-3894: for tokenizers, add some tests for larger documents
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303258 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 02:54:07 +00:00
3d73a3014e
LUCENE-3896: beef up TestDuelingAnalyzers for larger documents
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303253 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 01:52:22 +00:00
c20242721f
LUCENE-3894: some tokenizers weren't reading all input chars
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303193 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 23:02:37 +00:00
b7a7e5a625
LUCENE-3889: remove unnecessary/unused base class
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303026 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 17:28:26 +00:00
5648222e86
SOLR-2764: Fix testcase for minimal stemmer
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302872 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 13:12:39 +00:00
54d48eb98b
SOLR-2764: Create a NorwegianLightStemmer and NorwegianMinimalStemmer
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302833 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 10:57:50 +00:00
790323780f
basic javadocs improvements, mostly simple descriptions where the class had nothing before
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302752 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 02:09:25 +00:00
4a2b1d974a
javadocs: add missing package.htmls
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302713 13f79535-47bb-0310-9956-ffa450edef68
2012-03-19 23:20:25 +00:00
c4f72f61ac
LUCENE-3880: UAX29URLEmailTokenizer now recognizes emails when the mailto: scheme is prepended.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302265 13f79535-47bb-0310-9956-ffa450edef68
2012-03-19 03:13:52 +00:00
3d2d144f92
LUCENE-3848: don't produce tokenstreams that start with posinc=0
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1301478 13f79535-47bb-0310-9956-ffa450edef68
2012-03-16 13:06:30 +00:00
3d8b22ffd0
LUCENE-3850: Fix rawtypes warnings for Java 7 compiler ( #2 )
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1297162 13f79535-47bb-0310-9956-ffa450edef68
2012-03-05 18:48:04 +00:00
989530e17e
LUCENE-3850: Fix rawtypes warnings for Java 7 compiler
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1297048 13f79535-47bb-0310-9956-ffa450edef68
2012-03-05 13:34:40 +00:00
430365f7cc
Kuromoji now produces both compound words and the segmentation of those words in search mode (LUCENE-3767)
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1296805 13f79535-47bb-0310-9956-ffa450edef68
2012-03-04 13:34:13 +00:00
8c2e3cef8f
LUCENE-3820: limiting the amount of input for pattern matching to go past exponential time patterns, even if they happen. A nice catch from Mike too -- un-ignore testNastyPattern and look at processing time go wild with each additional input character...
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1294797 13f79535-47bb-0310-9956-ffa450edef68
2012-02-28 19:26:05 +00:00
f3cc65733b
Sysout of the randomized pattern.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1294518 13f79535-47bb-0310-9956-ffa450edef68
2012-02-28 08:15:38 +00:00
4d401ca87d
Test thread's name reflects the current seed.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1294514 13f79535-47bb-0310-9956-ffa450edef68
2012-02-28 08:04:42 +00:00
493bd8b42f
LUCENE-3820: optimistic limit on running time for the randomized pattern test. This doesn't eliminate the possibility of hitting an exponential time pattern, but I re-run a few times and it seems to be pretty stbale.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1294322 13f79535-47bb-0310-9956-ffa450edef68
2012-02-27 20:50:24 +00:00
7be5533989
LUCENE-3820: Wrong trailing index calculation in PatternReplaceCharFilter.
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1294141 13f79535-47bb-0310-9956-ffa450edef68
2012-02-27 13:13:10 +00:00
482c0610fd
[LUCENE-3731] - refactored analyzeText method to initializeIterator and made it abstract inside BaseUIMATokenizer
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1293614 13f79535-47bb-0310-9956-ffa450edef68
2012-02-25 14:14:00 +00:00
930816cc5b
LUCENE-3731 - AEProviderFactory getAEProvider logic cleaned
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1292585 13f79535-47bb-0310-9956-ffa450edef68
2012-02-22 23:39:51 +00:00
e51795be39
LUCENE-3731: remove unnecessary code
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244714 13f79535-47bb-0310-9956-ffa450edef68
2012-02-15 20:53:53 +00:00
c97e3edbb9
LUCENE-3731: performance improvements and thread safety fixes to UIMA tokenizers
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244688 13f79535-47bb-0310-9956-ffa450edef68
2012-02-15 20:29:20 +00:00
c454ae6a66
[LUCENE-3731] - creating and using simple wst and pos tagger implementations for analyzers' random string testing
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244474 13f79535-47bb-0310-9956-ffa450edef68
2012-02-15 13:17:57 +00:00
cea3acb111
LUCENE-3731: fix javadoc warnings, add uima to eclipse project
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244350 13f79535-47bb-0310-9956-ffa450edef68
2012-02-15 04:41:32 +00:00
8d9bfe9245
LUCENE-3731: adding missing overview.html
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244340 13f79535-47bb-0310-9956-ffa450edef68
2012-02-15 04:01:57 +00:00
d66d97790b
[LUCENE-3731] - Creating the analysis-uima module for UIMA based tokenizers/analyzers
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1244236 13f79535-47bb-0310-9956-ffa450edef68
2012-02-14 22:13:34 +00:00
087f1e3126
LUCENE-3774: Optimized and streamlined license and notice file validation
...
by refactoring the build task into an ANT task and modifying build scripts
to perform top-level checks. (Dawid Weiss, Steve Rowe, Robert Muir)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1243527 13f79535-47bb-0310-9956-ffa450edef68
2012-02-13 14:12:59 +00:00
6a07201844
don't fail test due to jre bugs in String.toLowerCase
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1243415 13f79535-47bb-0310-9956-ffa450edef68
2012-02-13 04:50:12 +00:00
590741dcfe
LUCENE-3766: Remove Tokenizer's default ctor
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242890 13f79535-47bb-0310-9956-ffa450edef68
2012-02-10 19:12:35 +00:00
8a50cefc6b
LUCENE-3748: EnglishPossessiveFilter did not work with a proper right quotation mark
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242740 13f79535-47bb-0310-9956-ffa450edef68
2012-02-10 11:01:11 +00:00
9f783ead67
SOLR-3115: improve japanese stopwords.txt description
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242557 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 22:17:44 +00:00
509f4c557d
LUCENE-3751: align default japanese configurations for lucene/solr
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242543 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 21:45:41 +00:00
72ae3171be
LUCENE-3765: Trappy behavior with StopFilter/ignoreCase
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242497 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 19:59:50 +00:00
c0319d5928
SOLR-3056: document expectations in these files
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1241960 13f79535-47bb-0310-9956-ffa450edef68
2012-02-08 16:27:47 +00:00
dac1b58277
SOLR-3097, SOLR-3105: add fieldtypes for different languages to the example
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1241878 13f79535-47bb-0310-9956-ffa450edef68
2012-02-08 12:07:52 +00:00
bef6e3664d
LUCENE-3726: additional tests
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240760 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 16:16:02 +00:00
03497e7595
LUCENE-3745: add proper Japanese stopping
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240714 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 13:05:42 +00:00
009608d9f2
LUCENE-3726: default Kuromoji to search mode
...
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240710 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 12:41:13 +00:00
Robert Muir
Christian Moen
Michael McCandless
Jan Høydahl
Steven Rowe
Uwe Schindler
Dawid Weiss
Tommaso Teofili
Ryan McKinley