lucene/contrib/benchmark
Michael McCandless 198aa70b35 LUCENE-1500: fix compilation errors in contrib/benchmark by allowing doHighlight method to throw InvalidTokenOfsetsException
git-svn-id: https://svn.apache.org/repos/asf/lucene/java/trunk@758575 13f79535-47bb-0310-9956-ffa450edef68
2009-03-26 10:03:34 +00:00
..
conf LUCENE-1527: open reader non-readOnly for doing deletes 2009-01-23 20:33:51 +00:00
lib LUCENE-848: make sure we use the right version (2.9.0) of XercesJ 2007-07-02 01:06:07 +00:00
src LUCENE-1500: fix compilation errors in contrib/benchmark by allowing doHighlight method to throw InvalidTokenOfsetsException 2009-03-26 10:03:34 +00:00
.rsync-filter LUCENE-848. Add Wikipedia benchmarking support 2007-07-01 02:19:10 +00:00
CHANGES.txt LUCENE-1495: fix time-based test to reduce change of false failure 2008-12-21 11:07:28 +00:00
README.enwiki LUCENE-848. Add Wikipedia benchmarking support 2007-07-01 02:19:10 +00:00
build.xml LUCENE-1090: remove relative paths from benchmark's build.xml. 2008-06-10 11:58:00 +00:00
pom.xml.template LUCENE-1525: add missing maven dependencies for contrib/{benchmark,highlighter}; fix svn eolstyle to native 2009-01-21 16:59:39 +00:00

README.enwiki

Support exists for downloading, parsing, and loading the English
version of wikipedia (enwiki).

The build file can automatically try to download the most current
enwiki dataset (pages-articles.xml.bz2) from the "latest" directory,
http://download.wikimedia.org/enwiki/latest/. However, this file
doesn't always exist, depending on where wikipedia is in the dump
process and whether prior dumps have succeeded. If this file doesn't
exist, you can sometimes find an older or in progress version by
looking in the dated directories under
http://download.wikimedia.org/enwiki/. For example, as of this
writing, there is a page file in
http://download.wikimedia.org/enwiki/20070402/. You can download this
file manually and put it in temp. Note that the file you download will
probably have the date in the name, e.g.,
http://download.wikimedia.org/enwiki/20070402/enwiki-20070402-pages-articles.xml.bz2. When
you put it in temp, rename it to enwiki-latest-pages-articles.xml.bz2.

After that, ant enwiki should process the data set and run a load
test. Ant targets get-enwiki, expand-enwiki, and extract-enwiki can
also be used to download, decompress, and extract (to individual files
in work/enwiki) the dataset, respectively.