mirror of https://github.com/apache/lucene.git
47f8c1baa2
Add new stored fields and termvectors interfaces: IndexReader.storedFields() and IndexReader.termVectors(). Deprecate IndexReader.document() and IndexReader.getTermVector(). The new APIs do not rely upon ThreadLocal storage for each index segment, which can greatly reduce RAM requirements when there are many threads and/or segments. Co-authored-by: Adrien Grand <jpountz@gmail.com> |
||
|---|---|---|
| .. | ||
| conf | ||
| scripts | ||
| src | ||
| .gitignore | ||
| README.enwiki | ||
| build.gradle | ||
README.enwiki
Support exists for downloading, parsing, and loading the English version of wikipedia (enwiki). The build file can automatically try to download the most current enwiki dataset (pages-articles.xml.bz2) from the "latest" directory, http://download.wikimedia.org/enwiki/latest/. However, this file doesn't always exist, depending on where wikipedia is in the dump process and whether prior dumps have succeeded. If this file doesn't exist, you can sometimes find an older or in progress version by looking in the dated directories under http://download.wikimedia.org/enwiki/. For example, as of this writing, there is a page file in http://download.wikimedia.org/enwiki/20070402/. You can download this file manually and put it in temp. Note that the file you download will probably have the date in the name, e.g., http://download.wikimedia.org/enwiki/20070402/enwiki-20070402-pages-articles.xml.bz2. If you use the EnwikiContentSource then the data will be decompressed on the fly during the benchmark. If you want to benchmark indexing, you should probably decompress it beforehand using the "enwiki" Ant target which will produce a work/enwiki.txt, after which you can use LineDocSource in your benchmark. After that, ant enwiki should process the data set and run a load test. Ant target enwiki will download, decompress, and extract (to individual files in work/enwiki) the dataset, respectively.