Lucene Benchmark Contrib Change Log The Benchmark contrib package contains code for benchmarking Lucene in a variety of ways. $Id:$ 8/4/2009 LUCENE-1770: Add EnwikiQueryMaker (Mark Miller) 8/04/2009 LUCENE-1773: Add FastVectorHighlighter tasks. This change is a non-backwards compatible change in how subclasses of ReadTask define a highlighter. The methods doHighlight, isMergeContiguousFragments, maxNumFragments and getHighlighter are no longer used and have been mark deprecated and package protected private so there's a compile time error. Instead, the new getBenchmarkHighlighter method should return an appropriate highlighter for the task. The configuration of the highlighter tasks (maxFrags, mergeContiguous, etc.) is now accepted as params to the task. (Koji Sekiguchi via Mike McCandless) 8/03/2009 LUCENE-1778: Add support for log.step setting per task type. Perviously, if you included a log.step line in the .alg file, it had been applied to all tasks. Now, you can include a log.step.AddDoc, or log.step.DeleteDoc (for example) to control logging for just these tasks. If you want to ommit logging for any other task, include log.step=-1. The syntax is "log.step." together with the Task's 'short' name (i.e., without the 'Task' part). (Shai Erera via Mark Miller) 7/24/2009 LUCENE-1595: Deprecate LineDocMaker and EnwikiDocMaker in favor of using DocMaker directly, with content.source = LineDocSource or EnwikiContentSource. NOTE: with this change, the "id" field from the Wikipedia XML export is now indexed as the "docname" field (previously it was indexed as "docid"). Additionaly, the SearchWithSort task now accepts all types that SortField can accept and no longer falls back to SortField.AUTO, which has been deprecated. (Mike McCandless) 7/20/2009 LUCENE-1755: Fix WriteLineDocTask to output a document if it contains either a title or body (or both). (Shai Erera via Mark Miller) 7/14/2009 LUCENE-1725: Fix the example Sort algorithm - auto is now deprecated and no longer works with Benchmark. Benchmark will now throw an exception if you specify sort fields without a type. The example sort algorithm is now typed. (Mark Miller) 7/6/2009 LUCENE-1730: Fix TrecContentSource to use ISO-8859-1 when reading the TREC files, unless a different encoding is specified. Additionally, ContentSource now supports a content.source.encoding parameter in the configuration file. (Shai Erera via Mark Miller) 6/26/2009 LUCENE-1716: Added the following support: doc.tokenized.norms: specifies whether to store norms doc.body.tokenized.norms: special attribute for the body field doc.index.props: specifies whether DocMaker should index the properties set on DocData writer.info.stream: specifies the info stream to set on IndexWriter (supported values are: SystemOut, SystemErr and a file name). (Shai Erera via Mike McCandless) 6/23/09 LUCENE-1714: WriteLineDocTask incorrectly normalized text, by replacing only occurrences of "\t" with a space. It now replaces "\r\n" in addition to that, so that LineDocMaker won't fail. (Shai Erera via Michael McCandless) 6/17/09 LUCENE-1595: This issue breaks previous external algorithms. DocMaker has been replaced with a concrete class which accepts a ContentSource for iterating over a content source's documents. Most of the old DocMakers were changed to a ContentSource implementation, and DocMaker is now a default document creation impl that provides an easy way for reusing fields. When [doc.maker] is not defined in an algorithm, the new DocMaker is the default. If you have .alg files which specify a DocMaker (like ReutersDocMaker), you should change the [doc.maker] line to: [content.source=org.apache.lucene.benchmark.byTask.feeds.ReutersContentSource] i.e. doc.maker=org.apache.lucene.benchmark.byTask.feeds.ReutersDocMaker becomes content.source=org.apache.lucene.benchmark.byTask.feeds.ReutersContentSource doc.maker=org.apache.lucene.benchmark.byTask.feeds.SimpleDocMaker becomes content.source=org.apache.lucene.benchmark.byTask.feeds.SingleDocSource Also, PerfTask now logs a message in tearDown() rather than each Task doing its own logging. A new setting called [log.step] is consulted to determine how often to log. [doc.add.log.step] is no longer a valid setting. For easy migration of current .alg files, rename [doc.add.log.step] to [log.step] and [doc.delete.log.step] to [delete.log.step]. Additionally, [doc.maker.forever] should be changed to [content.source.forever]. (Shai Erera via Mark Miller) 6/12/09 LUCENE-1539: Added DeleteByPercentTask which enables deleting a percentage of documents and searching on them. Changed CommitIndex to optionally accept a label (recorded as userData=