Lucene Retrieval Machine
Lucene Framework
Usage Scenarios Document

Revision: 5 (cmarschn, 2002-12-01)

Clemens Marschner - Otis Gospodnetic - Peter Carlson - Kelvin Tan

Usage Scenarios
1.1	File System Indexer
A file system indexer would work like the “Microsoft Index server”. It may consist of only one pipeline.
The Scheduler puts document locations (i.e. file URLs) in an asynchronous (Request-) pipeline. The first MP loads the document, replacing the file URL message by a document message, and tries to detect a MIME type. After that, the MessageDispatcher component dispatches the messages to different MPs, depending on the MIME type, that extract the text of each document. In the end, a LuceneStorage takes the resulting message and saves it in a Lucene index.
In an extension of that one MP first detects the Mime type, a second would check if that mime type can be handled by the application, the third then loads the doc, the fourth analyses the documents and the sixth saves them to a LuceneStorage.
In an incremental operation the Source is connected to the LuceneStorage and checks if documents have to be refreshed. An additional MP may check if the document loaded is newer than the one already indexed, and may discard the message if not.

1.2	Intranet Web Crawler
An intranet web crawler (only a few hosts) is not that different than the file system indexer, except that the loading process may be multithreaded and loads document over the net instead of the file system. In this case there have to be at least two MessagePipelines, since the crawling parts are again active components. An additional processing step extracts links from the loaded documents and puts them back into the queue. A URLSeenFilter (called URLVisitedFilter at this time) makes sure no URL is put into the pipeline twice. A  RobotExclusionFilter makes sure the robot exclusion standard is followed, and filters URLs that are marked to be “disallowed”. At the end there is again that LuceneStorage.
(This is how LARM is implemented right now. There are already some efforts made to put some of the data structures on hard drive)

1.3	Small WWW Web Crawler
If the system is supposed to scale to more than a few hosts, memory, efficiency and fault tolerance becomes a major concern, and it more and more becomes a matter of juggling with the system resources (network bandwidth, CPU time, RAM, hard drive space). If one of them becomes a bottleneck, the whole system may become very slow or (in case of RAM or HD shortage) may crash.
Suppose LuceneStorage is much slower than the crawler. Since the indexer is pretty much CPU bound it becomes necessary to distribute that on to two hosts. This can be done easily with the pipeline framework if it the pipeline is broken up into two parts and connected via JMS. The loaded document is put into a JMS topic which is configured such that the JMS messages are routed to one of the destinations in a round-robin manner. On the other side there are indexing components on different hosts that build Lucene indexes that are merged from time to time.
1.4	Large WWW Web Crawler
(todo)
If a web crawler is supposed to scale to the whole WWW a whole set of precautions have to be taken care of.
-	The URLSeen structure must scale to billions of URLs (i.e. constant memory usage) and must also be distributed
-	Crawlers and indexers must be distributed
-	Most of the data must be kept on disk
-	The server must be able to save its state on disk and recover after failures
-	Exchange of messages should take place in batch operation
-	special services, i.e. DNS resolvers, have to be installed to prevent bottlenecks

1.5	Database Indexer
A database indexer may consist of a Message source that is connected to the messaging mechanism of the database (i.e. triggers). It then reads the contents of changed database fields and puts them along to the indexer. That way Lucene may be integrated into HSSQL or even Oracle.
Problem here: Transaction safety may make it necessary to operate largely with disk based structures (i.e. transaction logs)
1.6	Single Subsystem Test Scenario
For testing purposes it may be viable to interface the SUT with dummy or utility components. That way, if the document processor subsystem is to be tested, the LuceneStorage may be replaced by a LogStorage, which does nothing but log everything it gets into log files. This log storage may also be placed between different processing steps.