Lucene Retrieval Machine
Lucene Framework
Mission Document

Revision: 5 (cmarschn, 2002-12-01)

Clemens Marschner - Otis Gospodnetic - Peter Carlson - Kelvin Tan

1	Mission
“
The Lucene Retrieval Machine forms a complete and highly scalable search solution for end-users of the Lucene search engine: Capable of intelligently indexing data from various sources, preprocessing of source documents configurable by the end user, up to a best-practice implementation of online search functionality. 

It will be based on the Lucene Framework that provides implementations for data aggregation and indexing functionality utilizing the Lucene indexing API, while being easily extensible and constructible by application developers or researchers.
”
2	
Background and goals
Lucene 1.2 has become an increasingly popular library for incorporating search functionality into Java applications. As it stands, the Lucene codebase is an API, as opposed to a search framework, or a full-fledged search application. The consequence is that developers (users of the API) often need to implement commonly-used but non-API functionality, in addition to any custom search requirements, to arrive at a working search application.

This proposal is a plan to leverage existing contributions and integrate them into a Lucene search framework. We believe it will significantly reduce the time taken to implement a searching and indexing solution using Lucene, provide deeper functionality for a basic application and lower the time it takes for a new user to be productive. It will also constitute a best-practice model of how the Lucene user community creates efficient, flexible search applications, by means of the framework.

This framework is a foundation of what we call the Lucene Retrieval Machine (LARM). In contrast to the framework, the retrieval machine forms an application that can be installed and used by an end user. We think that the framework can be so modular that LARM can in fact form many different applications, from indexing database tables to file systems, up to indexing a large portion of the web. Many of these configurations can come out-of-the box.

To take a familiar example, the Apache Web Server provides a framework for pluggable modules (the Apache modules) that allow for easily extending the web server using standard interfaces. On the other hand it is also easily configurable by an end user, and can be used without further programming. We want LARM to be even more flexible, to consist of merely a microkernel and consisting mainly of pluggable components. 

3	
Fulfilled User Needs 
The challenges faced by a typical Lucene user are:

1. Retrieving data from a myriad of data sources
2. Processing this data to be added to the search index
3. Searching the index and displaying the search results
4. Provide a consistent mechanism to access and display the detailed content

Correspondingly, Lucene Framework addresses these challenges by 
providing

- A data gather interface for retrieving data from various sources
- An indexing framework for updating indexes and indexing different kinds of data 
- A Search API which simplifies searching through abstraction, and provides display-related functionality

In addition, Lucene Framework also provides

- IndexReader/IndexSearcher pooling mechanism (?)
- Ability to run in single-VM or client/server mode (?)
-	Ability to access multiple indexes to search
-	Provide consistent link between search and index analyzer

Hooks in the framework allow users to customize how the framework functions. These exist in the form of pluggable implementations of classes and user-defined "steps" in a pipeline. These are:

- Data sources which allows the user to abstract the data to be retrieved
- Filters for pre- and post(?)-processing of what data needs to be indexed
- Content handlers which define strategies for indexing different file types (e.g XML, PDF, etc)
-	Storage handlers so there can be a consistent way to access data from multiple data sources

With these, Lucene Framework should address most, if not all, of a developer's requirements for a Lucene application.

4	
Scope/Requirements
We see LARM to be comprised of the following major components:
1) Data Gathering (web pages, database, file system)
2) Index creation (ability to handle data in different formats with 
configuration)
3) Search interface 
4) Data display (ability to see content details of the data)
4.1	Data Gathering
4.1.1	Data Sources
A major area of work will be the document sources section. It will comprise of components that may vary from very simple to complex

•	Database 
•	File System
•	Web: This (assumingly large) subproject seeks for efficiently crawling the web. It’s described elsewhere
•	Something else: JMS Queues, Mailing lists, mail servers, or whatever else contains text or can be converted to text
4.1.2	Operating Modes
LARM will mostly be concerned with updating an already existing index. We don’t want a solution that has to rebuild the whole index from scratch. It should do only as little work as necessary while indexing. That means, if possible, only the data that has changed is updated. We see two modes
-	Scheduled full (re)indexing. The index is completely rebuilt from time to time. We want to avoid having to take external programs to do that, so a configurable scheduler that triggers the process should be incorporated 
-	Incremental re-indexing: If possible, re-indexing should take place incrementally: the index should only be updated if documents were added, modified or deleted. This requires a notification from the data source, e.g. by database triggers or file system notification. As data sources will typically not be aware of LARM indexing, LARM will poll these data sources periodically. LARM will check for any changes such as additions, modifications or deletions.
4.2	Indexing
4.2.1	Data Processing
-	Preprocessing means making different kinds of data available for the indexer. This means HTML, PDF etc. must first be transformed to a format that can be understood by the indexer and other storage devices. This means fielded data in text, number or date format.
-	The data is then put into a pipeline of components that may work on the data in order to transform different input formats into a set of fields and tokens. This step can involve putting headings and body text into different fields, or doing linguistic analysis on the content. [I think a distinction needs to be made between the Lucene analyzer/tokenizer and the pre or post processor]
-	Also are we going to support field types?
4.2.2	Indexing Step
For indexing itself we want to use Lucene. One of the components in the data processing pipeline will use the Lucene API to add incoming data to the index. It puts tokenized fields into a searchable index. It can also store the raw data or index the raw data. The index option will be configurable. No other components in the pipeline should know about Lucene. For them this is just another step in the pipeline, another component.
4.3	Searching
The best practices way of searching and returning search results, including functionality such as index reader pooling, sorting search results and paging.
Proposed: SearchBean, in conjunction with the indexing framework.
Implementation:
Create a standard API to search the index. When items are updated, deleted or added, by the Indexing engine, have the index automatically reflect this updated information including creating new sorted indexes and not having to worry about which index searcher to use. Also, increase performance by using pooled index searchers, which get updated at the appropriate time.
4.4	Nonfunctional Requirements
4.4.1	Scalability
We want the whole system to scale up to hundreds of millions of documents. This is what we see as one of the major challenges why we work on this project. 
Scalability means 
-	We have to avoid algorithms that have a higher than constant complexity, 
-	We have to avoid internal I/O (hard drive or network accesses), 
-	We will have to use efficient data structures and/or compression, where applicable. 
-	In order to avoid more than necessary synchronization points between processes, we have to be able to exchange much of the data in batch mode, as well to cater the several orders of magnitude slower communication through network or hard drive I/O.
4.4.2	Configuration
LARM provides a standard way of configuration of all components in the system. Parameters that may be configured include:
-	Mapping of name extensions or MIME types to data handlers.
-	Index handler configuration
-	Sorted fields
-	Default conjunction operator
-	[Full list needs to be defined]
4.4.3	Inherently Distributable 
Constrained system resources (hard drive & network I/O, CPU and RAM) require that different components may be distributed over multiple machines. This calls for a message-oriented paradigm with coarse-grained data exchange packets.
4.4.4	Component-Based Approach
Our intention is to use Avalon as the basis for the framework. Whilst we recognize that this might significantly increase the learning curve to using the framework, we feel that its an acceptable tradeoff in exchange for a well-designed, production-quality server framework.
The component-based approach means different parts of the system (e.g. data sources and the pipeline) are only loosely coupled. This will enable us to assemble a system using configuration files, not source code, to incorporate external developments more easily through standard interfaces, and to be able to break up different steps of the pipeline into several processes, connected by data exchange components.
4.4.5	Fail-Safety/Check-pointing
If LARM is to be used in a production environment there have to be possibilities to shut the system down at least at specified checkpoints, and to recover after a system crash. Since indexing operations may take days or even weeks, it must be possible to save the global status of an indexing operation, at least every now and then. 
4.4.6	Other non-functional Requirements
-	Common logging mechanism
-	Common configuration mechanism
4.5	Out of Scope
We see the following aspects to be out of scope of our approach:
-	Complete transaction safety. This would involve an XA based architecture that would guarantee that a change, e.g. in a database table, would result in an index update in the LARM engine. Core-Lucene is not transaction safe at this time, so this wouldn’t make sense. In general, we favor a lean approach that allows slight differences between the index and the underlying data

[Peter: I think we should include the transactional like updating. We need to be able to update a live index.]
4.6	Development Standards
- Coding Style: Every LARM developer agrees to use the Avalon guidelines available at http://jakarta.apache.org/avalon/code-standards.html
- Logging: In case we use Avalon, Avalon LogKit is used. Otherwise we use Jakarta Log4J [configuration to be defined]