lucene/lucene/benchmark
Alan Woodward 2f2e00ffe2 LUCENE-7737: Remove spatial-extras dependency on queries module 2017-06-27 15:53:39 +01:00
..
conf LUCENE-7815: Removed the PostingsHighlighter 2017-05-23 14:39:51 -04:00
scripts LUCENE-4723: Add AnalyzerFactoryTask to benchmark, and enable analyzer creation via the resulting factories using NewAnalyzerTask. 2013-01-28 17:18:48 +00:00
src LUCENE-7737: Remove spatial-extras dependency on queries module 2017-06-27 15:53:39 +01:00
.gitignore LUCENE-7438: Renovate benchmark module's support for highlighting 2016-10-07 09:57:11 -04:00
README.enwiki LUCENE-7438: Renovate benchmark module's support for highlighting 2016-10-07 09:57:11 -04:00
build.xml LUCENE-7546: people.apache.org -> home.apache.org 2016-11-11 13:00:37 -05:00
ivy.xml LUCENE-7060: Spatial4j 0.6 upgrade. Package com.spatial4j.core -> org.locationtech.spatial4j 2016-03-02 21:05:32 -05:00

README.enwiki

Support exists for downloading, parsing, and loading the English
version of wikipedia (enwiki).

The build file can automatically try to download the most current
enwiki dataset (pages-articles.xml.bz2) from the "latest" directory,
http://download.wikimedia.org/enwiki/latest/. However, this file
doesn't always exist, depending on where wikipedia is in the dump
process and whether prior dumps have succeeded. If this file doesn't
exist, you can sometimes find an older or in progress version by
looking in the dated directories under
http://download.wikimedia.org/enwiki/. For example, as of this
writing, there is a page file in
http://download.wikimedia.org/enwiki/20070402/. You can download this
file manually and put it in temp. Note that the file you download will
probably have the date in the name, e.g.,
http://download.wikimedia.org/enwiki/20070402/enwiki-20070402-pages-articles.xml.bz2.

If you use the EnwikiContentSource then the data will be decompressed on the fly
during the benchmark.  If you want to benchmark indexing, you should probably decompress
it beforehand using the "enwiki" Ant target which will produce a work/enwiki.txt, after
which you can use LineDocSource in your benchmark.

After that, ant enwiki should process the data set and run a load
test. Ant target enwiki will download, decompress, and extract (to individual files
in work/enwiki) the dataset, respectively.