Lucene Retrieval Machine
Lucene Framework
System Overview Document

Revision: 6 (cmarschn, 2002-12-02)

Clemens Marschner - Otis Gospodnetic - Peter Carlson - Kelvin Tan

System Overview

1	System Layers
The high level parts are not dependent on each other. They communicate through the message system provided by existing solutions [Avalon Event] and processing pipelines. They may all have to use standard services, exposed to them by an external configuration mechanism [aka IOC pattern/Phoenix]

Services 
(e.g. Scheduler)

Search

Indexing

Source

LARM Message System (Messages, Sources, Pipelines etc.)

2	Subsystem Interactivity
2.1	Overview
The following shows one possible LARM configuration:

Some Container

Avalon Phoenix Container: Scheduler

Search&Sort component

Lucene Index

Scheduler

searches

get docs to reindex

Request

HTTP

RPC

Avalon Phoenix Container: Gatherer

Local App

Gatherer

ext. Data Sources

Message

Message

Avalon Phoenix Container: Indexer

Servlet Runner / AppServer

Avalon Phoenix Container: Processor

e.g. HTTP

Message

Message

Indexing Component

Processor2

Processor1

(read) access
message passing
process boundary

This diagram actually consists of two pipelines:
-	The first one between the Scheduler and the Gatherer, which conveys request messages. Since, as in case of the web crawler, new requests may be put back at any time, this pipeline runs independently of the other one
-	The second one between the Gatherer and the Indexing component, which runs through several steps and (optionally) through process boundaries
The diagram is only an example on how process boundaries could be defined. These boundaries only make sense if there is an advantage, e.g. to provide a means for load balancing. It could all take place in only one container. (the term “container” meaning the same as in J2EE)
It is important to note that the most important dependency among the different parts is the message itself. 

Notes [Otis]:
1.	Scheduler: Peter Donald recognized the need for this right away.  He also called it a Job Server.  The 'Cornerstone?' reference in the diagram refers to a Scheduler component that is a part of Cornerstone.

2.	Scheduler gets documents that need to be re-indexed from an existing index, wraps them in a Request Container, and puts them in the Request Queue of the Gatherer (or Fetcher, same thing, lets pick one name).  We could also provide an interface where a user can manually add Request Containers into the Request Queue.  We could also have an implementation of Scheduler that looks at some external locations for documents to index
(e.g. it may occasionally pass a Request Container to the Gatherer telling it to fetch something that is not already in the index).
 
In this example with 3 components they all run under Avalon, just like Scheduler, and Gatherer, but only the one that deals with Lucene-specific stuff (e.g. Fields, Documents, Analyzers, IndexWriter...) has the knowledge of Lucene and its classes. Some components may have multiple instances of itself in the Pipeline, some may not.

3.	Index is a standard Lucene Index.  If we can process Messages that end up in multiple indices, that's great, but not needed right away, I think.

4.	Finally, the search component doesn't participate in document processing, only index searching.  I am not sure what would be hosting it, whether it would be a Servlet container (e.g. SearchBean would be called from a servlet, like it probably is currently in most places that use it), or whether it would be Avalon (how do we serve HTTP requests by a component that runs under Avalon?  Apples and oranges?). I don’t think we worry about this we just provide an interface.

2.2	Another Example
The following example shows a possible configuration in 3 processes (A, B, C). In this example a crawler forms the “gatherer” part. Process (A) receives a list of URLs as input and outputs a list of (filtered) URLs that represent “crawling requests” for the crawler. In process (B) the documents are loaded and then transferred to a process (C) in which links are extracted from the documents and put back into the crawler, and which puts the documents into a Lucene index. It is important to note that the requests in (1) and (2) may not only consist of URLs but also of extra info. In incremental operation for example each request may contain a date for each doc. The crawler in (B) may load the HTTP header and stop reading the file if the date hasn’t changed.
Messages transferred in (1) to (3) will usually be a collection of data units, e.g. 100 URLs or 10 documents. A dispatcher may be used to put groups of messages together, based on time or volume thresholds. 

(HostInfo)

:JMSSource

:CLIURL
Source
(Command line interface)

URLPipeline:NonblockingPipeline

DNSResolver

Robots

URLSeen

ScopeFilter

Queue

SeenFilter

PatternFilter

Robots
Filter

DispatchPipeline:NonblockingPipeline

Queue

Dispatcher

Scheduler

:JMSDrain

DB

1

2

A

:JMSSource

URLPipeline:NonblockingPipeline

DocPipeline:
NonblockingPipeline

:JMSDrain

Queue

:Fetcher

loads doc.
nothing more
(less functionality than in
 LARM-TOS)

Dispatcher

DB

Scheduler

Queue

3

2

B

C

:LuceneDrain

DocProcessingPipeline:NonblockingPipeline

LinkExtractionPipeline:NonblockingPipeline

:JMSSource

:RuleBasedDocConverter

:MimeTypeHandler

:HTMLToDocument

Queue

:MimeTypeHandler

Queue

3

:HTMLLinkParser

:PDFToDocument

:PDFLinkParser

:TextToDocument

:NoOpParser

:JMSDrain

URLPipeline:NonblockingPipeline

Dispatcher

Queue

1

DB

Crawler Boundary

3	The Pipeline Framework
The Lucene Framework is based on a set of components that provide complete data aggregation and indexing functionality. The integration of these various components provide an ease of use for extracting data from various data source and indexing the information. 
The Lucene Framework creates the ability to take data from different data sources including web pages, databases, and file systems, and put them into a Lucene index. Optionally, take the data from the data source at a unit level (a web pages, a row or a file) and store that into a file in a standard format for later display.
For the Web pages, the framework includes a complete crawler with the ability to provide a link handling strategy that indicates how to handle embedded links within the web pages.

The pipeline framework is a set of classes that allows for a separation of different “assembly” parts of the whole system. That way different parts of the pipeline can be put in different classes and can be developed rather independently. 
It’s an in-process framework (compared to message queue systems which are mostly used for interprocess communications) and very low level: If it is known that only thread is involved, the components need not even be thread safe. The aim is to be able to process a very large number of small messages rapidly.
It’s also not a real workflow system, since it is a lot less complex and may become, for the sake of speed, more specialized.
3.1	Active and Passive Components
Active Components run in their own thread. They may respond to external events (socket calls, timer events or whatsoever). Passive Components just provide services to other passive or active components. MessageSources will mostly be active components. That is, they operate the subsequent pipeline. 

3.2	MessageProcessor
A MessageProcessor (MP) is called by a pipeline to handle a message. It may alter the message, filter it, save it somewhere, etc. It either returns null (forming a message sink) or it returns the message (most of the time the same message it got, but it may also return a different one). Examples of an MP would be a RobotExclusionFilter (filtering some of the URLs from the URL list), a PDF to XML converter (reducing PDF to a common metaformat that is understood by the indexing component), a FileSystemStorage that saves incoming documents on disk, a JMSStorage that saves them to a message queue, or a LuceneStorage that adds a document to a Lucene index. An MP could as well contain a BlockingPipeline (see below) forming a nested pipeline.
[Is the storage a required part of the pipeline? If so I think we should break it up into more distinct pieces to there can be some control programmatically. If not is there a required order?]
3.3	MessagePipelines
MessageProcessors are put together into message pipelines.

A BlockingPipeline processes a message by calling (at most) all of its MessageProcessors in a row. A MessageProcessor gets the Message, may alter it, and returns it again. The reference returned is passed to the next processor in the row. After the last MP the Message is discarded [why is the message discarded?] and the thread returns the message again.
BlockingPipeline may be designed not to be thread safe (i.e. because it is used from within an NonBlockingPipeline and thus only accessed by one thread), as do the MPs.
(A BlockingPipeline may as well be an MP, which allows for nesting pipelines).

(thread safe) BlockingPipeline

MP 1

MP 1

Threads

A NonBlockingPipeline has an extra thread that handles the messages. Therefore, at a putMessage() call, the message is written into a message queue. The processor thread handles all messages until the queue is empty. Internally the AsynchronousProcessor consists of a BlockingPipeline that is operated by the ProcessorThread.
In that scenario, an MP may create new messages and put them back into the same queue without causing the queue to block.

The Queue implementation will usually be a FIFO queue, but may be exchanged depending on the needs. A queue may refuse a message (i.e. if it is full).
(This concept is currently implemented as the MessageHandler. There’s currently no representation of a BlockingPipeline) 

MP 1

MP 1

BlockingPipeline

may put message into queue

Processor Thread

NonBlockingPipeline

Threads

(FIFO)
Queue

3.4	Message Sources
A MessageSource is an active component that (somehow) creates messages that are then processed, i. e. by a message queue. This view as a “source” only reflects that the pipeline framework doesn’t know where the messages come from. In reality they may come from the network (i.e. a JMS topic or a web service call) such that a broader view may reveal that the source only forms a part of a larger assembly line. This can be compared to a car manufacturer that gets car parts from other manufacturers who also have assembly lines. 

some trigger

SourceThread

MessageSource

Processor Thread

MP 1

(FIFO)
Queue

MP 1

BlockingPipeline

NonBlockingPipeline

(MessageSources are not yet implemented in LARM. The only two message sources now are FetcherMain which puts URLs into the queue after the start, and the FetcherTasks which put extracted URLs into the queue after crawling)
3.5	Messages
The contents of a message may be diverse, ranging from a single URL to a bunch of documents. A document should then be represented in a common form, regardless of where the data comes from (the web or the file system). That way, a document [let’s not call it a document] may consist of field/value pairs, just like a Lucene document
-	They must be very memory efficient, since there may be hundreds of thousands of them. A simple HashMap is definitely too high level, so there must be a distinction of different types of messages (just like in JMS, but not as complex)
-	There should be a object reuse as much as possible, i.e. by having an object pool of messages that Sources (acquire) and Pipelines (release) may access
-	There should also be a notion of a CompoundMessage that carries a lot of other (i.e. similar) messages that are passed to the MPs in one turn. The reason for that is that this way a lot of synchronization may be prevented. [I think this will be tricky since for some data sources we want a message to be broken up, like a web site and others we want them in a single thread like a database query. We might have a pre-processor on the messageQueue?]

(in LARM-old there is no message pooling and no compound messages)
4	Document Sources
4.1	Request
Contains all the information required to obtain the desired data from a data source and process it through pipeline. Is a Message.
4.2	Data Source
An abstract grouping of sources of data. Each data source must provide a way to obtain the data and then get information at a field level. Is a Source
4.3	File System Data Source
A data source that gets the data from a file system.
4.4	RDBMS Data Source
A data source that gets the data from a relational database.
4.5	FTP Data Source
A data source that gets the data from an FTP server.

(Are the following still consistent? No, but this document isn’t either. Let’s add a terminology section)
•	Request Result
An abstract grouping of results items. A request result contains the resulting single level unit from the data source, such as a file or database row. Each result must provide a way to get an iterator of fields and values for each result. There can also be an analyzer associated with each request result type to handle indexing of certain data.
•	HTML Request Result
A request result which assumes the result as a single item. 
•	XML Request Result
A request result that can be configured to put some or all xml tags into fields.
•	ZIP Request Result
A request result that generates other request results depending on the data inside the file. [Is there a better way to do this, like having a wrapped zip reader?]
•	Text Request Result
A request result which assumes the result as a single item. That is, not fielded data.
•	Row Request Result
A request result which can be configured to index data from all desired fields resulting in a sql query.
4.6	The Data Unit processing framework
In a queue where document objects are passed through the pipeline, a set of processing steps may be taken on the documents, depending on their types. That way a heterogeneous mass of documents (PDF, HTML, Word etc.) is transformed to Document objects consisting of name/value pairs with index terms for the search engine. Keeping this in a message pipeline (and not i.e. in a set of nested iterators that transform the documents on the fly) makes sure that this pipeline can always be broken up and be distributed on different nodes.

If a large switch/case block is to be avoided there must be at least a rudimentary way of dispatching a message to one of several MPs depending on a rule, i.e. the MIME type. The easiest implementation of that would be if each MP is asked if it wants to work on that message. Can we do like the config in tomcat? I’ll look, but think we can do it with reflection and dynamically handle this.

MessageDispatcher

MP 1

MP 2

MP 3

[rule]

Msg

4.7	Services
MPs may need shared services, which may be active or passive components on their own. MPs should tell which kind of services they rely on and should get a reference to them after the start-up. 
(this can be modelled well with the Avalon)