Commit Graph

37 Commits

Author SHA1 Message Date
Michael McCandless 78b4be5dc6 LUCENE-3940: fix Kuromoji to not produce invalid token graph due to UNK with punctuation being decompounded
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1311072 13f79535-47bb-0310-9956-ffa450edef68
2012-04-08 19:17:17 +00:00
Michael McCandless 755ebafa49 LUCENE-3873: add MockGraphTokenFilter, inserting random graph tokens
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1310910 13f79535-47bb-0310-9956-ffa450edef68
2012-04-07 23:06:12 +00:00
Robert Muir 6c7c89c3f9 LUCENE-1866: add exclusion for bocchan test file
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1309255 13f79535-47bb-0310-9956-ffa450edef68
2012-04-04 05:36:52 +00:00
Robert Muir e5448e2e20 LUCENE-3947: fix rat-sources task to work with tools/ directories
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1309207 13f79535-47bb-0310-9956-ffa450edef68
2012-04-04 01:51:56 +00:00
Robert Muir 6b16efdc22 LUCENE-3930: kuromoji steals icu's jar
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1308423 13f79535-47bb-0310-9956-ffa450edef68
2012-04-02 16:31:59 +00:00
Robert Muir 8f0d7cc135 LUCENE-3930: nuke jars from source tree and use ivy
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1307563 13f79535-47bb-0310-9956-ffa450edef68
2012-03-30 18:04:43 +00:00
Christian Moen ec18632428 Fixed various related to config and user dictionaries for Kuromoji (SOLR-3276)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1306476 13f79535-47bb-0310-9956-ffa450edef68
2012-03-28 17:20:48 +00:00
Robert Muir 620f9a5739 small opto when charfilter is used: don't call this method twice in end
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305742 13f79535-47bb-0310-9956-ffa450edef68
2012-03-27 06:06:51 +00:00
Christian Moen f5770479e3 Move and rename Kuromoji (LUCENE-3909)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305297 13f79535-47bb-0310-9956-ffa450edef68
2012-03-26 10:31:48 +00:00
Robert Muir 35705cc396 LUCENE-3919: fix czechstemmer aioobe on the empty term
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305177 13f79535-47bb-0310-9956-ffa450edef68
2012-03-25 23:40:44 +00:00
Michael McCandless cb1a9a0cdf LUCENE-3897: if best scoring path is ahead of current pos, move forward
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305149 13f79535-47bb-0310-9956-ffa450edef68
2012-03-25 21:37:55 +00:00
Michael McCandless a278ba7a0c LUCENE-3897: fix silly bug in forced backtrace
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305086 13f79535-47bb-0310-9956-ffa450edef68
2012-03-25 17:51:26 +00:00
Christian Moen c3ddb9dc67 Added KuromojiReadingFormFilter (LUCENE-3915)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1305046 13f79535-47bb-0310-9956-ffa450edef68
2012-03-25 14:17:23 +00:00
Christian Moen 63f1c48b7d Added katakana stem filter (LUCENE-3901)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1304719 13f79535-47bb-0310-9956-ffa450edef68
2012-03-24 06:38:53 +00:00
Robert Muir c3305a50ff add some more kuromoji javadocs
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303746 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 12:21:48 +00:00
Christian Moen d2eebf9330 Fix for LUCENE-3897 (KuromojiTokenizer fails with large docs)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303739 13f79535-47bb-0310-9956-ffa450edef68
2012-03-22 11:41:54 +00:00
Robert Muir dd7bfc78d9 LUCENE-3894: for tokenizers, add some tests for larger documents
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1303258 13f79535-47bb-0310-9956-ffa450edef68
2012-03-21 02:54:07 +00:00
Robert Muir 790323780f basic javadocs improvements, mostly simple descriptions where the class had nothing before
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302752 13f79535-47bb-0310-9956-ffa450edef68
2012-03-20 02:09:25 +00:00
Robert Muir 4a2b1d974a javadocs: add missing package.htmls
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1302713 13f79535-47bb-0310-9956-ffa450edef68
2012-03-19 23:20:25 +00:00
Christian Moen 430365f7cc Kuromoji now produces both compound words and the segmentation of those words in search mode (LUCENE-3767)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1296805 13f79535-47bb-0310-9956-ffa450edef68
2012-03-04 13:34:13 +00:00
Robert Muir 9f783ead67 SOLR-3115: improve japanese stopwords.txt description
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242557 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 22:17:44 +00:00
Robert Muir 509f4c557d LUCENE-3751: align default japanese configurations for lucene/solr
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242543 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 21:45:41 +00:00
Robert Muir 72ae3171be LUCENE-3765: Trappy behavior with StopFilter/ignoreCase
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1242497 13f79535-47bb-0310-9956-ffa450edef68
2012-02-09 19:59:50 +00:00
Robert Muir bef6e3664d LUCENE-3726: additional tests
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240760 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 16:16:02 +00:00
Robert Muir 03497e7595 LUCENE-3745: add proper Japanese stopping
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240714 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 13:05:42 +00:00
Robert Muir 009608d9f2 LUCENE-3726: default Kuromoji to search mode
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1240710 13f79535-47bb-0310-9956-ffa450edef68
2012-02-05 12:41:13 +00:00
Michael McCandless 60c36c24fb don't let prefix's output bleed into full string's output (potential/latent bug)
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1239658 13f79535-47bb-0310-9956-ffa450edef68
2012-02-02 15:01:13 +00:00
Robert Muir 995c5b9ef1 LUCENE-3730: improve Kuromoji search mode heuristics
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1239061 13f79535-47bb-0310-9956-ffa450edef68
2012-02-01 11:03:17 +00:00
Michael McCandless d1165b1972 LUCENE-3725: add optional packing to FSTs
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1237500 13f79535-47bb-0310-9956-ffa450edef68
2012-01-29 22:48:45 +00:00
Robert Muir e869b1fbf7 LUCENE-3700: give enough ram so that you can build naist-jdic with java 5
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232274 13f79535-47bb-0310-9956-ffa450edef68
2012-01-17 02:27:31 +00:00
Robert Muir f562a8a0dc LUCENE-3700: optionally support naist-jdic for kuromoji
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232268 13f79535-47bb-0310-9956-ffa450edef68
2012-01-17 02:20:24 +00:00
Robert Muir 48c01e5a2b LUCENE-3699: share baseform with surface and flag if the reading can be computed from surface
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232265 13f79535-47bb-0310-9956-ffa450edef68
2012-01-17 02:12:27 +00:00
Robert Muir c902f63125 unbreak clover/nightly builds until we do this right
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232254 13f79535-47bb-0310-9956-ffa450edef68
2012-01-17 01:37:28 +00:00
Robert Muir 12c9b8b4bf LUCENE-3699: simplify dictionary access and reduce tokeninfodictionary 1.5MB
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232120 13f79535-47bb-0310-9956-ffa450edef68
2012-01-16 19:19:48 +00:00
Robert Muir 354a3be78f LUCENE-3696: fix dictionary construction to work on java5
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1232012 13f79535-47bb-0310-9956-ffa450edef68
2012-01-16 14:50:09 +00:00
Robert Muir 05a65507af LUCENE-3305: optimization, don't retrieve the base form twice in this filter
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1230769 13f79535-47bb-0310-9956-ffa450edef68
2012-01-12 20:36:58 +00:00
Robert Muir cd372bdc83 LUCENE-3305: add Kuromoji Japanese morphological analyzer
git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1230748 13f79535-47bb-0310-9956-ffa450edef68
2012-01-12 20:10:48 +00:00