2007-06-26 00:26:07 -04:00
|
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
2006-11-26 19:03:13 -05:00
|
|
|
<html>
|
|
|
|
|
<head>
|
|
|
|
|
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
|
|
|
|
<meta content="Apache Forrest" name="Generator">
|
2007-06-26 00:26:07 -04:00
|
|
|
<meta name="Forrest-version" content="0.8">
|
2009-06-24 12:20:00 -04:00
|
|
|
<meta name="Forrest-skin-name" content="lucene">
|
2006-11-26 19:03:13 -05:00
|
|
|
<title>
|
2006-11-29 15:11:38 -05:00
|
|
|
Apache Lucene - Index File Formats
|
|
|
|
|
</title>
|
2006-11-26 19:03:13 -05:00
|
|
|
<link type="text/css" href="skin/basic.css" rel="stylesheet">
|
|
|
|
|
<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
|
|
|
|
|
<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
|
|
|
|
|
<link type="text/css" href="skin/profile.css" rel="stylesheet">
|
|
|
|
|
<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
|
|
|
|
|
<link rel="shortcut icon" href="images/favicon.ico">
|
|
|
|
|
</head>
|
|
|
|
|
<body onload="init()">
|
|
|
|
|
<script type="text/javascript">ndeSetTextSize();</script>
|
|
|
|
|
<div id="top">
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|breadtrail
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="breadtrail">
|
2006-12-02 07:11:45 -05:00
|
|
|
<a href="http://www.apache.org/">Apache</a> > <a href="http://lucene.apache.org/">Lucene</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|header
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="header">
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|start group logo
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="grouplogo">
|
2006-12-02 07:11:45 -05:00
|
|
|
<a href="http://lucene.apache.org/"><img class="logoImage" alt="Lucene" src="http://www.apache.org/images/asf_logo_simple.png" title="Apache Lucene"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end group logo
|
|
|
|
|
+-->
|
|
|
|
|
<!--+
|
|
|
|
|
|start Project Logo
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="projectlogo">
|
|
|
|
|
<a href="http://lucene.apache.org/java/"><img class="logoImage" alt="Lucene" src="http://lucene.apache.org/images/lucene_green_300.gif" title="Apache Lucene is a high-performance, full-featured text search engine library written entirely in
|
|
|
|
|
Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform."></a>
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end Project Logo
|
|
|
|
|
+-->
|
|
|
|
|
<!--+
|
|
|
|
|
|start Search
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="searchbox">
|
2009-06-24 12:20:00 -04:00
|
|
|
<form action="http://search.lucidimagination.com/p:lucene" method="get" class="roundtopsmall">
|
2009-07-09 09:06:51 -04:00
|
|
|
<input onFocus="getBlank (this, 'Search the site with Lucene');" size="25" name="q" id="query" type="text" value="Search the site with Lucene">
|
2007-06-26 00:26:07 -04:00
|
|
|
<input name="Search" value="Search" type="submit">
|
2006-11-26 19:03:13 -05:00
|
|
|
</form>
|
2009-06-24 12:20:00 -04:00
|
|
|
<div style="position: relative; top: -5px; left: -10px">Powered by <a href="http://www.lucidimagination.com" style="color: #033268">Lucid Imagination</a>
|
|
|
|
|
</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end search
|
|
|
|
|
+-->
|
|
|
|
|
<!--+
|
|
|
|
|
|start Tabs
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<ul id="tabs">
|
|
|
|
|
<li class="current">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a class="selected" href="http://lucene.apache.org/java/docs/">Main</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
|
|
|
|
<li>
|
2007-06-26 00:26:07 -04:00
|
|
|
<a class="unselected" href="http://wiki.apache.org/lucene-java">Wiki</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
2008-01-11 18:21:01 -05:00
|
|
|
<li class="current">
|
2009-11-11 05:59:44 -05:00
|
|
|
<a class="selected" href="index.html">Lucene 3.0 Documentation</a>
|
2008-01-11 18:21:01 -05:00
|
|
|
</li>
|
2006-11-26 19:03:13 -05:00
|
|
|
</ul>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end Tabs
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
</div>
|
|
|
|
|
<div id="main">
|
|
|
|
|
<div id="publishedStrip">
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|start Subtabs
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div id="level2tabs"></div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end Endtabs
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<script type="text/javascript"><!--
|
2007-06-26 00:26:07 -04:00
|
|
|
document.write("Last Published: " + document.lastModified);
|
2006-11-26 19:03:13 -05:00
|
|
|
// --></script>
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|breadtrail
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="breadtrail">
|
2007-06-26 00:26:07 -04:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|start Menu, mainarea
|
|
|
|
|
+-->
|
|
|
|
|
<!--+
|
|
|
|
|
|start Menu
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div id="menu">
|
2008-01-11 18:21:01 -05:00
|
|
|
<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
|
|
|
|
|
<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="index.html">Overview</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2009-09-08 20:00:17 -04:00
|
|
|
<div onclick="SwitchMenu('menu_1.1.2', 'skin/')" id="menu_1.1.2Title" class="menutitle">Changes</div>
|
2008-01-11 18:21:01 -05:00
|
|
|
<div id="menu_1.1.2" class="menuitemgroup">
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="menuitem">
|
2009-09-08 20:00:17 -04:00
|
|
|
<a href="changes/Changes.html">Core</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
|
|
|
|
<a href="changes/Contrib-Changes.html">Contrib</a>
|
|
|
|
|
</div>
|
|
|
|
|
</div>
|
|
|
|
|
<div onclick="SwitchMenu('menu_1.1.3', 'skin/')" id="menu_1.1.3Title" class="menutitle">Javadocs</div>
|
|
|
|
|
<div id="menu_1.1.3" class="menuitemgroup">
|
|
|
|
|
<div class="menuitem">
|
2009-03-13 17:20:06 -04:00
|
|
|
<a href="api/all/index.html">All</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/core/index.html">Core</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/demo/index.html">Demo</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2009-09-08 20:00:17 -04:00
|
|
|
<div onclick="SwitchMenu('menu_1.1.3.4', 'skin/')" id="menu_1.1.3.4Title" class="menutitle">Contrib</div>
|
|
|
|
|
<div id="menu_1.1.3.4" class="menuitemgroup">
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-analyzers/index.html">Analyzers</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-07-23 14:32:12 -04:00
|
|
|
<a href="api/contrib-smartcn/index.html">Smart Chinese Analyzer</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-ant/index.html">Ant</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-bdb/index.html">Bdb</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-bdb-je/index.html">Bdb-je</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-benchmark/index.html">Benchmark</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-03-19 06:51:55 -04:00
|
|
|
<a href="api/contrib-collation/index.html">Collation</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-07-09 09:06:51 -04:00
|
|
|
<a href="api/contrib-fast-vector-highlighter/index.html">Fast Vector Highlighter</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-highlighter/index.html">Highlighter</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-10-11 13:28:23 -04:00
|
|
|
<a href="api/contrib-instantiated/index.html">Instantiated</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-lucli/index.html">Lucli</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-memory/index.html">Memory</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2007-12-04 07:56:22 -05:00
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-misc/index.html">Miscellaneous</a>
|
2007-12-04 07:56:22 -05:00
|
|
|
</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-queries/index.html">Queries</a>
|
2007-12-04 07:56:22 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-08-03 00:04:08 -04:00
|
|
|
<a href="api/contrib-queryparser/index.html">Query Parser Framework</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-regex/index.html">Regex</a>
|
2007-12-04 07:56:22 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-06-15 08:45:05 -04:00
|
|
|
<a href="api/contrib-remote/index.html">Remote</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-snowball/index.html">Snowball</a>
|
2007-12-04 07:56:22 -05:00
|
|
|
</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="menuitem">
|
2009-05-19 01:11:26 -04:00
|
|
|
<a href="api/contrib-spatial/index.html">Spatial</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-spellchecker/index.html">Spellchecker</a>
|
2006-12-02 07:11:45 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-surround/index.html">Surround</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-swing/index.html">Swing</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-13 13:36:22 -05:00
|
|
|
<a href="api/contrib-wikipedia/index.html">Wikipedia</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-wordnet/index.html">Wordnet</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="api/contrib-xml-query-parser/index.html">XML Query Parser</a>
|
2006-12-09 11:32:22 -05:00
|
|
|
</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-11-16 03:29:52 -05:00
|
|
|
<a href="systemrequirements.html">System Requirements</a>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="contributions.html">Contributions</a>
|
2007-02-18 09:24:11 -05:00
|
|
|
</div>
|
2007-06-19 04:00:51 -04:00
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="http://wiki.apache.org/lucene-java/LuceneFAQ">FAQ</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2008-01-11 18:21:01 -05:00
|
|
|
<div class="menupage">
|
|
|
|
|
<div class="menupagetitle">File Formats</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="gettingstarted.html">Getting Started</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2009-09-06 21:00:21 -04:00
|
|
|
<a href="lucene-contrib/index.html">Lucene Contrib</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="queryparsersyntax.html">Query Syntax</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="scoring.html">Scoring</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
<div class="menuitem">
|
2008-01-11 18:21:01 -05:00
|
|
|
<a href="http://wiki.apache.org/lucene-java">Wiki</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
</div>
|
|
|
|
|
<div id="credit"></div>
|
|
|
|
|
<div id="roundbottom">
|
|
|
|
|
<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|alternative credits
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div id="credit2"></div>
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end Menu
|
|
|
|
|
+-->
|
|
|
|
|
<!--+
|
|
|
|
|
|start content
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div id="content">
|
|
|
|
|
<div title="Portable Document Format" class="pdflink">
|
|
|
|
|
<a class="dida" href="fileformats.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
|
|
|
|
|
PDF</a>
|
|
|
|
|
</div>
|
|
|
|
|
<h1>
|
2006-11-29 15:11:38 -05:00
|
|
|
Apache Lucene - Index File Formats
|
|
|
|
|
</h1>
|
2006-11-26 19:03:13 -05:00
|
|
|
<div id="minitoc-area">
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Index File Formats">Index File Formats</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Definitions">Definitions</a>
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Inverted Indexing">Inverted Indexing</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Types of Fields">Types of Fields</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Segments">Segments</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Document Numbers">Document Numbers</a>
|
|
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Overview">Overview</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#File Naming">File Naming</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a href="#file-names">Summary of File Extensions</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
2006-11-26 19:03:13 -05:00
|
|
|
<a href="#Primitive Types">Primitive Types</a>
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Byte">Byte</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#UInt32">UInt32</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Uint64">Uint64</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#VInt">VInt</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Chars">Chars</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#String">String</a>
|
|
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
2009-05-25 16:07:24 -04:00
|
|
|
<a href="#Compound Types">Compound Types</a>
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
2009-05-27 10:02:56 -04:00
|
|
|
<a href="#MapStringString">Map<String,String></a>
|
2009-05-25 16:07:24 -04:00
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
2006-11-26 19:03:13 -05:00
|
|
|
<a href="#Per-Index Files">Per-Index Files</a>
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Segments File">Segments File</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
<a href="#Lock File">Lock File</a>
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Deletable File">Deletable File</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Compound Files">Compound Files</a>
|
|
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Per-Segment Files">Per-Segment Files</a>
|
|
|
|
|
<ul class="minitoc">
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Fields">Fields</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Term Dictionary">Term Dictionary</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Frequencies">Frequencies</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Positions">Positions</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Normalization Factors">Normalization Factors</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Term Vectors">Term Vectors</a>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Deleted Documents">Deleted Documents</a>
|
|
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</li>
|
|
|
|
|
<li>
|
|
|
|
|
<a href="#Limitations">Limitations</a>
|
|
|
|
|
</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</div>
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N1000C"></a><a name="Index File Formats"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Index File Formats</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
This document defines the index file formats used
|
2009-08-25 10:36:47 -04:00
|
|
|
in Lucene version 2.9. If you are using a different
|
2006-11-29 15:11:38 -05:00
|
|
|
version of Lucene, please consult the copy of
|
|
|
|
|
<span class="codefrag">docs/fileformats.html</span>
|
|
|
|
|
that was distributed
|
|
|
|
|
with the version you are using.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Apache Lucene is written in Java, but several
|
|
|
|
|
efforts are underway to write
|
2007-03-09 17:29:00 -05:00
|
|
|
<a href="http://wiki.apache.org/lucene-java/LuceneImplementations">versions
|
2006-11-29 15:11:38 -05:00
|
|
|
of Lucene in other programming
|
2006-11-26 19:03:13 -05:00
|
|
|
languages</a>. If these versions are to remain compatible with Apache
|
|
|
|
|
Lucene, then a language-independent definition of the Lucene index
|
|
|
|
|
format is required. This document thus attempts to provide a
|
2009-08-25 10:36:47 -04:00
|
|
|
complete and independent definition of the Apache Lucene 2.9 file
|
2006-11-26 19:03:13 -05:00
|
|
|
formats.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
As Lucene evolves, this document should evolve.
|
|
|
|
|
Versions of Lucene in different programming languages should endeavor
|
|
|
|
|
to agree on file formats, and generate new versions of this document.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Compatibility notes are provided in this document,
|
|
|
|
|
describing how file formats have changed from prior versions.
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>
|
|
|
|
|
In version 2.1, the file format was changed to allow
|
|
|
|
|
lock-less commits (ie, no more commit lock). The
|
|
|
|
|
change is fully backwards compatible: you can open a
|
|
|
|
|
pre-2.1 index for searching or adding/deleting of
|
|
|
|
|
docs. When the new segments file is saved
|
|
|
|
|
(committed), it will be written in the new file format
|
|
|
|
|
(meaning no specific "upgrade" process is needed).
|
|
|
|
|
But note that once a commit has occurred, pre-2.1
|
|
|
|
|
Lucene will not be able to read the index.
|
|
|
|
|
</p>
|
2007-07-04 11:16:38 -04:00
|
|
|
<p>
|
|
|
|
|
In version 2.3, the file format was changed to allow
|
|
|
|
|
segments to share a single set of doc store (vectors &
|
|
|
|
|
stored fields) files. This allows for faster indexing
|
|
|
|
|
in certain cases. The change is fully backwards
|
|
|
|
|
compatible (in the same way as the lock-less commits
|
|
|
|
|
change in 2.1).
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N1002B"></a><a name="Definitions"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Definitions</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
The fundamental concepts in Lucene are index,
|
|
|
|
|
document, field and term.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
An index contains a sequence of documents.
|
|
|
|
|
</p>
|
|
|
|
|
<ul>
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
A document is a sequence of fields.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
A field is a named sequence of terms.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
A term is a string.
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
<p>
|
|
|
|
|
The same string in two different fields is
|
|
|
|
|
considered a different term. Thus terms are represented as a pair of
|
|
|
|
|
strings, the first naming the field, and the second naming text
|
|
|
|
|
within the field.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N1004B"></a><a name="Inverted Indexing"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Inverted Indexing</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The index stores statistics about terms in order
|
|
|
|
|
to make term-based search more efficient. Lucene's
|
|
|
|
|
index falls into the family of indexes known as an <i>inverted
|
|
|
|
|
index.</i> This is because it can list, for a term, the documents that contain
|
|
|
|
|
it. This is the inverse of the natural relationship, in which
|
|
|
|
|
documents list terms.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10057"></a><a name="Types of Fields"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Types of Fields</h3>
|
|
|
|
|
<p>
|
|
|
|
|
In Lucene, fields may be <i>stored</i>, in which
|
|
|
|
|
case their text is stored in the index literally, in a non-inverted
|
|
|
|
|
manner. Fields that are inverted are called <i>indexed</i>. A field
|
|
|
|
|
may be both stored and indexed.</p>
|
|
|
|
|
<p>The text of a field may be <i>tokenized</i> into terms to be
|
|
|
|
|
indexed, or the text of a field may be used literally as a term to be indexed.
|
|
|
|
|
Most fields are
|
|
|
|
|
tokenized, but sometimes it is useful for certain identifier fields
|
|
|
|
|
to be indexed literally.
|
|
|
|
|
</p>
|
2009-08-26 16:15:09 -04:00
|
|
|
<p>See the <a href="api/core/org/apache/lucene/document/Field.html">Field</a> java docs for more information on Fields.</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10074"></a><a name="Segments"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Segments</h3>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
Lucene indexes may be composed of multiple sub-indexes, or
|
|
|
|
|
<i>segments</i>. Each segment is a fully independent index, which could be searched
|
2006-11-26 19:03:13 -05:00
|
|
|
separately. Indexes evolve by:
|
|
|
|
|
</p>
|
|
|
|
|
<ol>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Creating new segments for newly added documents.</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Merging existing segments.</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ol>
|
|
|
|
|
<p>
|
|
|
|
|
Searches may involve multiple segments and/or multiple indexes, each
|
|
|
|
|
index potentially composed of a set of segments.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10092"></a><a name="Document Numbers"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Document Numbers</h3>
|
|
|
|
|
<p>
|
|
|
|
|
Internally, Lucene refers to documents by an integer <i>document
|
|
|
|
|
number</i>. The first document added to an index is numbered zero, and each
|
|
|
|
|
subsequent document added gets a number one greater than the previous.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
|
|
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Note that a document's number may change, so caution should be taken
|
2006-11-29 15:11:38 -05:00
|
|
|
when storing these numbers outside of Lucene. In particular, numbers may
|
2006-11-26 19:03:13 -05:00
|
|
|
change in the following situations:
|
|
|
|
|
</p>
|
|
|
|
|
<ul>
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
The
|
|
|
|
|
numbers stored in each segment are unique only within the segment,
|
|
|
|
|
and must be converted before they can be used in a larger context.
|
|
|
|
|
The standard technique is to allocate each segment a range of
|
|
|
|
|
values, based on the range of numbers used in that segment. To
|
|
|
|
|
convert a document number from a segment to an external value, the
|
|
|
|
|
segment's <i>base</i> document
|
|
|
|
|
number is added. To convert an external value back to a
|
|
|
|
|
segment-specific value, the segment is identified by the range that
|
|
|
|
|
the external value is in, and the segment's base value is
|
|
|
|
|
subtracted. For example two five document segments might be
|
|
|
|
|
combined, so that the first segment has a base value of zero, and
|
|
|
|
|
the second of five. Document three from the second segment would
|
|
|
|
|
have an external value of eight.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
When documents are deleted, gaps are created
|
2006-11-29 15:11:38 -05:00
|
|
|
in the numbering. These are eventually removed as the index evolves
|
|
|
|
|
through merging. Deleted documents are dropped when segments are
|
|
|
|
|
merged. A freshly-merged segment thus has no gaps in its numbering.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N100B9"></a><a name="Overview"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Overview</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
Each segment index maintains the following:
|
|
|
|
|
</p>
|
|
|
|
|
<ul>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
|
|
|
|
<p>Field names. This
|
2006-11-26 19:03:13 -05:00
|
|
|
contains the set of field names used in the index.
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Stored Field
|
2006-11-29 15:11:38 -05:00
|
|
|
values. This contains, for each document, a list of attribute-value
|
|
|
|
|
pairs, where the attributes are field names. These are used to
|
2006-11-26 19:03:13 -05:00
|
|
|
store auxiliary information about the document, such as its title,
|
|
|
|
|
url, or an identifier to access a
|
|
|
|
|
database. The set of stored fields are what is returned for each hit
|
2006-11-29 15:11:38 -05:00
|
|
|
when searching. This is keyed by document number.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Term dictionary.
|
|
|
|
|
A dictionary containing all of the terms used in all of the indexed
|
2006-11-29 15:11:38 -05:00
|
|
|
fields of all of the documents. The dictionary also contains the
|
2006-11-26 19:03:13 -05:00
|
|
|
number of documents which contain the term, and pointers to the
|
|
|
|
|
term's frequency and proximity data.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Term Frequency
|
2006-11-29 15:11:38 -05:00
|
|
|
data. For each term in the dictionary, the numbers of all the
|
2006-11-26 19:03:13 -05:00
|
|
|
documents that contain that term, and the frequency of the term in
|
2008-08-05 13:17:42 -04:00
|
|
|
that document if omitTf is false.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Term Proximity
|
2006-11-29 15:11:38 -05:00
|
|
|
data. For each term in the dictionary, the positions that the term
|
2008-08-05 13:17:42 -04:00
|
|
|
occurs in each document. Note that this will
|
|
|
|
|
not exist if all fields in all documents set
|
|
|
|
|
omitTf to true.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Normalization
|
2006-11-29 15:11:38 -05:00
|
|
|
factors. For each field in each document, a value is stored that is
|
2006-11-26 19:03:13 -05:00
|
|
|
multiplied into the score for hits on that field.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
|
|
|
|
<p>Term Vectors. For each field in each document, the term vector
|
|
|
|
|
(sometimes called document vector) may be stored. A term vector consists
|
|
|
|
|
of term text and term frequency. To add Term Vectors to your index see the
|
2009-08-26 16:15:09 -04:00
|
|
|
<a href="api/core/org/apache/lucene/document/Field.html">Field</a>
|
2006-11-29 15:11:38 -05:00
|
|
|
constructors
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
</li>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Deleted documents.
|
|
|
|
|
An optional file indicating which documents are deleted.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
<p>Details on each of these are provided in subsequent sections.
|
|
|
|
|
</p>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N100FC"></a><a name="File Naming"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">File Naming</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
All files belonging to a segment have the same name with varying
|
2006-11-29 15:11:38 -05:00
|
|
|
extensions. The extensions correspond to the different file formats
|
2006-11-26 19:03:13 -05:00
|
|
|
described below. When using the Compound File format (default in 1.4 and greater) these files are
|
|
|
|
|
collapsed into a single .cfs file (see below for details)
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Typically, all segments
|
|
|
|
|
in an index are stored in a single directory, although this is not
|
|
|
|
|
required.
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>
|
|
|
|
|
As of version 2.1 (lock-less commits), file names are
|
|
|
|
|
never re-used (there is one exception, "segments.gen",
|
|
|
|
|
see below). That is, when any file is saved to the
|
|
|
|
|
Directory it is given a never before used filename.
|
|
|
|
|
This is achieved using a simple generations approach.
|
|
|
|
|
For example, the first segments file is segments_1,
|
|
|
|
|
then segments_2, etc. The generation is a sequential
|
|
|
|
|
long integer represented in alpha-numeric (base 36)
|
|
|
|
|
form.
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
2009-08-22 21:18:52 -04:00
|
|
|
|
|
|
|
|
<a name="N1010B"></a><a name="file-names"></a>
|
|
|
|
|
<h2 class="boxed">Summary of File Extensions</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>The following table summarizes the names and extensions of the files in Lucene:
|
|
|
|
|
<table class="ForrestTable" cellspacing="1" cellpadding="4">
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<th>Name</th>
|
|
|
|
|
<th>Extension</th>
|
|
|
|
|
<th>Brief Description</th>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Segments File">Segments File</a></td>
|
|
|
|
|
<td>segments.gen, segments_N</td>
|
|
|
|
|
<td>Stores information about segments</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Lock File">Lock File</a></td>
|
|
|
|
|
<td>write.lock</td>
|
|
|
|
|
<td>The Write lock prevents multiple IndexWriters from writing to the same file.</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Compound Files">Compound File</a></td>
|
|
|
|
|
<td>.cfs</td>
|
|
|
|
|
<td>An optional "virtual" file consisting of all the other index files for systems
|
|
|
|
|
that frequently run out of file handles.</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Fields">Fields</a></td>
|
|
|
|
|
<td>.fnm</td>
|
|
|
|
|
<td>Stores information about the fields</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#field_index">Field Index</a></td>
|
|
|
|
|
<td>.fdx</td>
|
|
|
|
|
<td>Contains pointers to field data</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#field_data">Field Data</a></td>
|
|
|
|
|
<td>.fdt</td>
|
|
|
|
|
<td>The stored fields for documents</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#tis">Term Infos</a></td>
|
|
|
|
|
<td>.tis</td>
|
|
|
|
|
<td>Part of the term dictionary, stores term info</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#tii">Term Info Index</a></td>
|
|
|
|
|
<td>.tii</td>
|
|
|
|
|
<td>The index into the Term Infos file</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Frequencies">Frequencies</a></td>
|
|
|
|
|
<td>.frq</td>
|
|
|
|
|
<td>Contains the list of docs which contain each term along with frequency</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Positions">Positions</a></td>
|
|
|
|
|
<td>.prx</td>
|
|
|
|
|
<td>Stores position information about where a term occurs in the index</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Normalization Factors">Norms</a></td>
|
2009-08-25 10:36:47 -04:00
|
|
|
<td>.nrm</td>
|
2009-08-22 21:18:52 -04:00
|
|
|
<td>Encodes length and boost factors for docs and fields</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#tvx">Term Vector Index</a></td>
|
|
|
|
|
<td>.tvx</td>
|
|
|
|
|
<td>Stores offset into the document data file</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#tvd">Term Vector Documents</a></td>
|
|
|
|
|
<td>.tvd</td>
|
|
|
|
|
<td>Contains information about each document that has term vectors</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#tvf">Term Vector Fields</a></td>
|
|
|
|
|
<td>.tvf</td>
|
|
|
|
|
<td>The field level info about term vectors</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td><a href="#Deleted Documents">Deleted Documents</a></td>
|
|
|
|
|
<td>.del</td>
|
|
|
|
|
<td>Info about what files are deleted</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
</div>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N101F5"></a><a name="Primitive Types"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Primitive Types</h2>
|
|
|
|
|
<div class="section">
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N101FA"></a><a name="Byte"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Byte</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The most primitive type
|
2006-11-29 15:11:38 -05:00
|
|
|
is an eight-bit byte. Files are accessed as sequences of bytes. All
|
2006-11-26 19:03:13 -05:00
|
|
|
other data types are defined as sequences
|
|
|
|
|
of bytes, so file formats are byte-order independent.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10203"></a><a name="UInt32"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">UInt32</h3>
|
|
|
|
|
<p>
|
|
|
|
|
32-bit unsigned integers are written as four
|
|
|
|
|
bytes, high-order bytes first.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
UInt32 --> <Byte><sup>4</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10212"></a><a name="Uint64"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Uint64</h3>
|
|
|
|
|
<p>
|
|
|
|
|
64-bit unsigned integers are written as eight
|
|
|
|
|
bytes, high-order bytes first.
|
|
|
|
|
</p>
|
|
|
|
|
<p>UInt64 --> <Byte><sup>8</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10221"></a><a name="VInt"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">VInt</h3>
|
|
|
|
|
<p>
|
|
|
|
|
A variable-length format for positive integers is
|
|
|
|
|
defined where the high-order bit of each byte indicates whether more
|
2006-11-29 15:11:38 -05:00
|
|
|
bytes remain to be read. The low-order seven bits are appended as
|
2006-11-26 19:03:13 -05:00
|
|
|
increasingly more significant bits in the resulting integer value.
|
|
|
|
|
Thus values from zero to 127 may be stored in a single byte, values
|
|
|
|
|
from 128 to 16,383 may be stored in two bytes, and so on.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>VInt Encoding Example</b>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<table class="ForrestTable" cellspacing="0" cellpadding="4" border="0">
|
|
|
|
|
|
|
|
|
|
<col width="64*">
|
|
|
|
|
|
|
|
|
|
<col width="64*">
|
|
|
|
|
|
|
|
|
|
<col width="64*">
|
|
|
|
|
|
|
|
|
|
<col width="64*">
|
|
|
|
|
|
|
|
|
|
<tr valign="TOP">
|
|
|
|
|
|
|
|
|
|
<td width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>Value</b>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>First byte</b>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>Second byte</b>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>Third byte</b>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="0" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">0
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="0" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
00000000
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">1
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="2" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">2
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
00000010
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td valign="TOP" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">...
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="127" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">127
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1111111" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
01111111
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="128" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">128
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
10000000
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="129" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">129
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000001" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
10000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="130" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">130
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000010" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
10000010
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td valign="TOP" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">...
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="16383" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">16,383
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="11111111" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
11111111
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1111111" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
01111111
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="16384" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">16,384
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
10000000
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
10000000
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr valign="BOTTOM">
|
|
|
|
|
|
|
|
|
|
<td sdnum="1033;0;#,##0" sdval="16385" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">16,385
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000001" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
10000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="10000000" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
10000000
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" sdval="1" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
|
|
|
|
00000001
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
|
|
|
|
|
|
|
<td valign="TOP" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT">...
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: 0.11cm; margin-right: 0.01cm">
|
|
|
|
|
|
|
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.07cm; margin-right: 0.01cm">
|
|
|
|
|
|
|
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
<td sdnum="1033;0;00000000" valign="BOTTOM" width="25%">
|
|
|
|
|
|
|
|
|
|
<p align="RIGHT" class="western" style="margin-left: -0.47cm; margin-right: 0.01cm">
|
|
|
|
|
|
|
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</td>
|
|
|
|
|
|
|
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
|
</table>
|
|
|
|
|
<p>
|
|
|
|
|
This provides compression while still being
|
|
|
|
|
efficient to decode.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10506"></a><a name="Chars"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Chars</h3>
|
|
|
|
|
<p>
|
|
|
|
|
Lucene writes unicode
|
2008-03-26 09:39:25 -04:00
|
|
|
character sequences as UTF-8 encoded bytes.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N1050F"></a><a name="String"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">String</h3>
|
|
|
|
|
<p>
|
2008-03-26 09:39:25 -04:00
|
|
|
Lucene writes strings as UTF-8 encoded bytes.
|
|
|
|
|
First the length, in bytes, is written as a VInt,
|
|
|
|
|
followed by the bytes.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
String --> VInt, Chars
|
|
|
|
|
</p>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N1051C"></a><a name="Compound Types"></a>
|
2009-05-25 16:07:24 -04:00
|
|
|
<h2 class="boxed">Compound Types</h2>
|
|
|
|
|
<div class="section">
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10521"></a><a name="MapStringString"></a>
|
2009-05-25 16:07:24 -04:00
|
|
|
<h3 class="boxed">Map<String,String></h3>
|
|
|
|
|
<p>
|
|
|
|
|
In a couple places Lucene stores a Map
|
|
|
|
|
String->String.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Map<String,String> --> Count<String,String><sup>Count</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10531"></a><a name="Per-Index Files"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Per-Index Files</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
The files in this section exist one-per-index.
|
|
|
|
|
</p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="N10539"></a><a name="Segments File"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Segments File</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The active segments in the index are stored in the
|
2006-11-29 15:11:38 -05:00
|
|
|
segment info file,
|
2007-01-23 20:28:53 -05:00
|
|
|
<tt>segments_N</tt>.
|
|
|
|
|
There may
|
2006-11-29 15:11:38 -05:00
|
|
|
be one or more
|
|
|
|
|
<tt>segments_N</tt>
|
|
|
|
|
files in the
|
|
|
|
|
index; however, the one with the largest
|
|
|
|
|
generation is the active one (when older
|
|
|
|
|
segments_N files are present it's because they
|
|
|
|
|
temporarily cannot be deleted, or, a writer is in
|
2007-03-27 07:28:15 -04:00
|
|
|
the process of committing, or a custom
|
2009-08-26 16:15:09 -04:00
|
|
|
<a href="api/core/org/apache/lucene/index/IndexDeletionPolicy.html">IndexDeletionPolicy</a>
|
2007-03-27 07:28:15 -04:00
|
|
|
is in use). This file lists each
|
2006-11-29 15:11:38 -05:00
|
|
|
segment by name, has details about the separate
|
|
|
|
|
norms and deletion files, and also contains the
|
|
|
|
|
size of each segment.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
As of 2.1, there is also a file
|
2007-01-23 20:28:53 -05:00
|
|
|
<tt>segments.gen</tt>.
|
|
|
|
|
This file contains the
|
2006-11-29 15:11:38 -05:00
|
|
|
current generation (the
|
|
|
|
|
<tt>_N</tt>
|
|
|
|
|
in
|
2007-01-23 20:28:53 -05:00
|
|
|
<tt>segments_N</tt>)
|
|
|
|
|
of the index. This is
|
2006-11-29 15:11:38 -05:00
|
|
|
used only as a fallback in case the current
|
|
|
|
|
generation cannot be accurately determined by
|
|
|
|
|
directory listing alone (as is the case for some
|
|
|
|
|
NFS clients with time-based directory cache
|
|
|
|
|
expiraation). This file simply contains an Int32
|
|
|
|
|
version header (SegmentInfos.FORMAT_LOCKLESS =
|
|
|
|
|
-2), followed by the generation recorded as Int64,
|
|
|
|
|
written twice.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<b>2.9</b>
|
2009-03-13 17:20:06 -04:00
|
|
|
Segments --> Format, Version, NameCounter, SegCount, <SegName, SegSize, DelGen, DocStoreOffset, [DocStoreSegment, DocStoreIsCompoundFile], HasSingleNormFile, NumField,
|
|
|
|
|
NormGen<sup>NumField</sup>,
|
2009-05-25 16:07:24 -04:00
|
|
|
IsCompoundFile, DeletionCount, HasProx, Diagnostics><sup>SegCount</sup>, CommitUserData, Checksum
|
2009-03-13 17:20:06 -04:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2008-08-05 13:17:42 -04:00
|
|
|
Format, NameCounter, SegCount, SegSize, NumField,
|
|
|
|
|
DocStoreOffset, DeletionCount --> Int32
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2008-02-11 13:56:09 -05:00
|
|
|
Version, DelGen, NormGen, Checksum --> Int64
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2009-05-25 16:07:24 -04:00
|
|
|
SegName, DocStoreSegment --> String
|
2006-11-29 15:11:38 -05:00
|
|
|
</p>
|
2009-05-25 16:07:24 -04:00
|
|
|
<p>
|
|
|
|
|
Diagnostics --> Map<String,String>
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>
|
2008-08-05 13:17:42 -04:00
|
|
|
IsCompoundFile, HasSingleNormFile,
|
2009-05-25 16:07:24 -04:00
|
|
|
DocStoreIsCompoundFile, HasProx --> Int8
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
CommitUserData --> Map<String,String>
|
2006-11-29 15:11:38 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2009-08-25 10:36:47 -04:00
|
|
|
Format is -9 (SegmentInfos.FORMAT_DIAGNOSTICS).
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Version counts how often the index has been
|
|
|
|
|
changed by adding or deleting documents.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
NameCounter is used to generate names for new segment files.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
SegName is the name of the segment, and is used as the file name prefix
|
|
|
|
|
for all of the files that compose the segment's index.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
SegSize is the number of documents contained in the segment index.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
DelGen is the generation count of the separate
|
|
|
|
|
deletes file. If this is -1, there are no
|
|
|
|
|
separate deletes. If it is 0, this is a pre-2.1
|
|
|
|
|
segment and you must check filesystem for the
|
|
|
|
|
existence of _X.del. Anything above zero means
|
|
|
|
|
there are separate deletes (_X_N.del).
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
NumField is the size of the array for NormGen, or
|
|
|
|
|
-1 if there are no NormGens stored.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
NormGen records the generation of the separate
|
|
|
|
|
norms files. If NumField is -1, there are no
|
|
|
|
|
normGens stored and they are all assumed to be 0
|
|
|
|
|
when the segment file was written pre-2.1 and all
|
|
|
|
|
assumed to be -1 when the segments file is 2.1 or
|
|
|
|
|
above. The generation then has the same meaning
|
|
|
|
|
as delGen (above).
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
IsCompoundFile records whether the segment is
|
|
|
|
|
written as a compound file or not. If this is -1,
|
|
|
|
|
the segment is not a compound file. If it is 1,
|
|
|
|
|
the segment is a compound file. Else it is 0,
|
|
|
|
|
which means we check filesystem to see if _X.cfs
|
|
|
|
|
exists.
|
|
|
|
|
</p>
|
2007-01-23 20:28:53 -05:00
|
|
|
<p>
|
|
|
|
|
If HasSingleNormFile is 1, then the field norms are
|
|
|
|
|
written as a single joined file (with extension
|
|
|
|
|
<tt>.nrm</tt>); if it is 0 then each field's norms
|
|
|
|
|
are stored as separate <tt>.fN</tt> files. See
|
|
|
|
|
"Normalization Factors" below for details.
|
|
|
|
|
</p>
|
2007-07-04 11:16:38 -04:00
|
|
|
<p>
|
|
|
|
|
DocStoreOffset, DocStoreSegment,
|
|
|
|
|
DocStoreIsCompoundFile: If DocStoreOffset is -1,
|
|
|
|
|
this segment has its own doc store (stored fields
|
|
|
|
|
values and term vectors) files and DocStoreSegment
|
|
|
|
|
and DocStoreIsCompoundFile are not stored. In
|
|
|
|
|
this case all files for stored field values
|
|
|
|
|
(<tt>*.fdt</tt> and <tt>*.fdx</tt>) and term
|
|
|
|
|
vectors (<tt>*.tvf</tt>, <tt>*.tvd</tt> and
|
|
|
|
|
<tt>*.tvx</tt>) will be stored with this segment.
|
|
|
|
|
Otherwise, DocStoreSegment is the name of the
|
|
|
|
|
segment that has the shared doc store files;
|
|
|
|
|
DocStoreIsCompoundFile is 1 if that segment is
|
|
|
|
|
stored in compound file format (as a <tt>.cfx</tt>
|
|
|
|
|
file); and DocStoreOffset is the starting document
|
|
|
|
|
in the shared doc store files where this segment's
|
|
|
|
|
documents begin. In this case, this segment does
|
|
|
|
|
not store its own doc store files but instead
|
|
|
|
|
shares a single set of these files with other
|
|
|
|
|
segments.
|
|
|
|
|
</p>
|
2008-02-11 13:56:09 -05:00
|
|
|
<p>
|
|
|
|
|
Checksum contains the CRC32 checksum of all bytes
|
|
|
|
|
in the segments_N file up until the checksum.
|
|
|
|
|
This is used to verify integrity of the file on
|
|
|
|
|
opening the index.
|
|
|
|
|
</p>
|
2008-08-05 13:17:42 -04:00
|
|
|
<p>
|
|
|
|
|
DeletionCount records the number of deleted
|
|
|
|
|
documents in this segment.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
HasProx is 1 if any fields in this segment have
|
|
|
|
|
omitTf set to false; else, it's 0.
|
|
|
|
|
</p>
|
2009-03-13 17:20:06 -04:00
|
|
|
<p>
|
2009-05-25 16:07:24 -04:00
|
|
|
CommitUserData stores an optional user-supplied
|
|
|
|
|
opaque Map<String,String> that was passed to
|
|
|
|
|
IndexWriter's commit or prepareCommit, or
|
|
|
|
|
IndexReader's flush methods.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
The Diagnostics Map is privately written by
|
|
|
|
|
IndexWriter, as a debugging aid, for each segment
|
|
|
|
|
it creates. It includes metadata like the current
|
|
|
|
|
Lucene version, OS, Java version, why the segment
|
|
|
|
|
was created (merge, flush, addIndexes), etc.
|
2009-03-13 17:20:06 -04:00
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N105BE"></a><a name="Lock File"></a>
|
2006-11-29 15:11:38 -05:00
|
|
|
<h3 class="boxed">Lock File</h3>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>
|
2007-03-09 17:29:00 -05:00
|
|
|
The write lock, which is stored in the index
|
|
|
|
|
directory by default, is named "write.lock". If
|
|
|
|
|
the lock directory is different from the index
|
|
|
|
|
directory then the write lock will be named
|
|
|
|
|
"XXXX-write.lock" where XXXX is a unique prefix
|
|
|
|
|
derived from the full path to the index directory.
|
2007-05-09 19:32:13 -04:00
|
|
|
When this file is present, a writer is currently
|
|
|
|
|
modifying the index (adding or removing
|
|
|
|
|
documents). This lock file ensures that only one
|
|
|
|
|
writer is modifying the index at a time.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N105C7"></a><a name="Deletable File"></a>
|
2006-11-29 15:11:38 -05:00
|
|
|
<h3 class="boxed">Deletable File</h3>
|
|
|
|
|
<p>
|
2009-08-25 10:36:47 -04:00
|
|
|
A writer dynamically computes
|
2006-11-29 15:11:38 -05:00
|
|
|
the files that are deletable, instead, so no file
|
|
|
|
|
is written.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N105D0"></a><a name="Compound Files"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Compound Files</h3>
|
|
|
|
|
<p>Starting with Lucene 1.4 the compound file format became default. This
|
2006-12-13 05:08:28 -05:00
|
|
|
is simply a container for all files described in the next section
|
|
|
|
|
(except for the .del file).</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Compound (.cfs) --> FileCount, <DataOffset, FileName>
|
|
|
|
|
<sup>FileCount</sup>
|
|
|
|
|
,
|
|
|
|
|
FileData
|
|
|
|
|
<sup>FileCount</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>FileCount --> VInt</p>
|
|
|
|
|
<p>DataOffset --> Long</p>
|
|
|
|
|
<p>FileName --> String</p>
|
|
|
|
|
<p>FileData --> raw file data</p>
|
|
|
|
|
<p>The raw file data is the data from the individual files named above.</p>
|
2007-07-04 11:16:38 -04:00
|
|
|
<p>Starting with Lucene 2.3, doc store files (stored
|
|
|
|
|
field values and term vectors) can be shared in a
|
|
|
|
|
single set of files for more than one segment. When
|
|
|
|
|
compound file is enabled, these shared files will be
|
|
|
|
|
added into a single compound file (same format as
|
|
|
|
|
above) but with the extension <tt>.cfx</tt>.
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N105F8"></a><a name="Per-Segment Files"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Per-Segment Files</h2>
|
|
|
|
|
<div class="section">
|
|
|
|
|
<p>
|
|
|
|
|
The remaining files are all per-segment, and are
|
|
|
|
|
thus defined by suffix.
|
|
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N10600"></a><a name="Fields"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Fields</h3>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>Field Info</b>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Field names are
|
|
|
|
|
stored in the field info file, with suffix .fnm.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
FieldInfos
|
2009-05-01 06:37:08 -04:00
|
|
|
(.fnm) --> FNMVersion,FieldsCount, <FieldName,
|
2006-11-29 15:11:38 -05:00
|
|
|
FieldBits>
|
|
|
|
|
<sup>FieldsCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
2009-05-01 06:37:08 -04:00
|
|
|
FNMVersion, FieldsCount --> VInt
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
FieldName --> String
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
FieldBits --> Byte
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<ul>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
The low-order bit is one for
|
|
|
|
|
indexed fields, and zero for non-indexed fields.
|
|
|
|
|
</li>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
The second lowest-order
|
|
|
|
|
bit is one for fields that have term vectors stored, and zero for fields
|
|
|
|
|
without term vectors.
|
|
|
|
|
</li>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<li>If the third lowest-order bit is set (0x04), term positions are stored with the term vectors.</li>
|
|
|
|
|
|
|
|
|
|
<li>If the fourth lowest-order bit is set (0x08), term offsets are stored with the term vectors.</li>
|
|
|
|
|
|
|
|
|
|
<li>If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field.</li>
|
2007-03-27 07:28:15 -04:00
|
|
|
|
|
|
|
|
<li>If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field.</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
</p>
|
2009-05-01 06:37:08 -04:00
|
|
|
<p>
|
|
|
|
|
FNMVersion (added in 2.9) is always -2.
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
Fields are numbered by their order in this file. Thus field zero is
|
2006-11-26 19:03:13 -05:00
|
|
|
the
|
2006-11-29 15:11:38 -05:00
|
|
|
first field in the file, field one the next, and so on. Note that,
|
2006-11-26 19:03:13 -05:00
|
|
|
like document numbers, field numbers are segment relative.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<b>Stored Fields</b>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<br>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Stored fields are represented by two files:
|
|
|
|
|
</p>
|
|
|
|
|
<ol>
|
|
|
|
|
|
|
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="field_index"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
The field index, or .fdx file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
This contains, for each document, a pointer to
|
|
|
|
|
its field data, as follows:
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
FieldIndex
|
2006-11-29 15:11:38 -05:00
|
|
|
(.fdx) -->
|
|
|
|
|
<FieldValuesPosition>
|
|
|
|
|
<sup>SegSize</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>FieldValuesPosition
|
|
|
|
|
--> Uint64
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>This
|
|
|
|
|
is used to find the location within the field data file of the
|
2006-11-29 15:11:38 -05:00
|
|
|
fields of a particular document. Because it contains fixed-length
|
|
|
|
|
data, this file may be easily randomly accessed. The position of
|
|
|
|
|
document
|
|
|
|
|
<i>n</i>
|
|
|
|
|
's
|
|
|
|
|
<i></i>
|
|
|
|
|
field data is the Uint64 at
|
|
|
|
|
<i>n*8</i>
|
|
|
|
|
in
|
2006-11-26 19:03:13 -05:00
|
|
|
this file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="field_data"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
The field data, or .fdt file.
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
This contains the stored fields of each document,
|
|
|
|
|
as follows:
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
FieldData (.fdt) -->
|
|
|
|
|
<DocFieldData>
|
|
|
|
|
<sup>SegSize</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>DocFieldData -->
|
|
|
|
|
FieldCount, <FieldNum, Bits, Value>
|
|
|
|
|
<sup>FieldCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>FieldCount -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VInt
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>FieldNum -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VInt
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Bits -->
|
2006-11-26 19:03:13 -05:00
|
|
|
Byte
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<ul>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>low order bit is one for tokenized fields</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>second bit is one for fields containing binary data</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>third bit is one for fields with compression option enabled
|
2006-11-29 15:11:38 -05:00
|
|
|
(if compression is enabled, the algorithm used is ZLIB)</li>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Value -->
|
2006-11-26 19:03:13 -05:00
|
|
|
String | BinaryValue (depending on Bits)
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>BinaryValue -->
|
2006-11-26 19:03:13 -05:00
|
|
|
ValueSize, <Byte>^ValueSize
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>ValueSize -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VInt
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ol>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N106A7"></a><a name="Term Dictionary"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Term Dictionary</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The term dictionary is represented as two files:
|
|
|
|
|
</p>
|
|
|
|
|
<ol>
|
|
|
|
|
|
|
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="tis"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
The term infos, or tis file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TermInfoFile (.tis)-->
|
2007-06-05 21:54:35 -04:00
|
|
|
TIVersion, TermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermInfos
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TIVersion -->
|
2006-11-26 19:03:13 -05:00
|
|
|
UInt32
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermCount -->
|
2006-11-26 19:03:13 -05:00
|
|
|
UInt64
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>IndexInterval -->
|
2006-11-26 19:03:13 -05:00
|
|
|
UInt32
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>SkipInterval -->
|
2006-11-26 19:03:13 -05:00
|
|
|
UInt32
|
|
|
|
|
</p>
|
|
|
|
|
|
2007-06-05 21:54:35 -04:00
|
|
|
<p>MaxSkipLevels -->
|
|
|
|
|
UInt32
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermInfos -->
|
|
|
|
|
<TermInfo>
|
|
|
|
|
<sup>TermCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermInfo -->
|
2006-11-26 19:03:13 -05:00
|
|
|
<Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Term -->
|
2006-11-26 19:03:13 -05:00
|
|
|
<PrefixLength, Suffix, FieldNum>
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Suffix -->
|
2006-11-26 19:03:13 -05:00
|
|
|
String
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>PrefixLength,
|
2006-11-29 15:11:38 -05:00
|
|
|
DocFreq, FreqDelta, ProxDelta, SkipDelta
|
|
|
|
|
<br>
|
|
|
|
|
--> VInt
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
2008-03-26 09:39:25 -04:00
|
|
|
<p>
|
|
|
|
|
This file is sorted by Term. Terms are
|
|
|
|
|
ordered first lexicographically (by UTF16
|
|
|
|
|
character code) by the term's field name,
|
|
|
|
|
and within that lexicographically (by
|
|
|
|
|
UTF16 character code) by the term's text.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>TIVersion names the version of the format
|
2009-08-25 10:36:47 -04:00
|
|
|
of this file and is equal to TermInfosWriter.FORMAT_CURRENT.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>Term
|
2006-11-29 15:11:38 -05:00
|
|
|
text prefixes are shared. The PrefixLength is the number of initial
|
2006-11-26 19:03:13 -05:00
|
|
|
characters from the previous term which must be pre-pended to a
|
2006-11-29 15:11:38 -05:00
|
|
|
term's suffix in order to form the term's text. Thus, if the
|
2006-11-26 19:03:13 -05:00
|
|
|
previous term's text was "bone" and the term is "boy",
|
|
|
|
|
the PrefixLength is two and the suffix is "y".
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>FieldNumber
|
|
|
|
|
determines the term's field, whose name is stored in the .fdt file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>DocFreq
|
|
|
|
|
is the count of documents which contain the term.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>FreqDelta
|
|
|
|
|
determines the position of this term's TermFreqs within the .frq
|
2006-11-29 15:11:38 -05:00
|
|
|
file. In particular, it is the difference between the position of
|
2006-11-26 19:03:13 -05:00
|
|
|
this term's data in that file and the position of the previous
|
|
|
|
|
term's data (or zero, for the first term in the file).
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>ProxDelta
|
|
|
|
|
determines the position of this term's TermPositions within the .prx
|
2006-11-29 15:11:38 -05:00
|
|
|
file. In particular, it is the difference between the position of
|
2006-11-26 19:03:13 -05:00
|
|
|
this term's data in that file and the position of the previous
|
2008-08-05 13:17:42 -04:00
|
|
|
term's data (or zero, for the first term in the file. For fields
|
|
|
|
|
with omitTf true, this will be 0 since
|
|
|
|
|
prox information is not stored.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>SkipDelta determines the position of this
|
2006-11-29 15:11:38 -05:00
|
|
|
term's SkipData within the .frq file. In
|
2006-11-26 19:03:13 -05:00
|
|
|
particular, it is the number of bytes
|
|
|
|
|
after TermFreqs that the SkipData starts.
|
|
|
|
|
In other words, it is the length of the
|
2007-06-05 21:54:35 -04:00
|
|
|
TermFreq data. SkipDelta is only stored
|
|
|
|
|
if DocFreq is not smaller than SkipInterval.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
|
|
|
|
|
|
|
|
|
<p>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="tii"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
The term info index, or .tii file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
2006-11-29 15:11:38 -05:00
|
|
|
This contains every IndexInterval
|
|
|
|
|
<sup>th</sup>
|
|
|
|
|
entry from the .tis
|
|
|
|
|
file, along with its location in the "tis" file. This is
|
2006-11-26 19:03:13 -05:00
|
|
|
designed to be read entirely into memory and used to provide random
|
|
|
|
|
access to the "tis" file.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
The structure of this file is very similar to the
|
|
|
|
|
.tis file, with the addition of one item per record, the IndexDelta.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TermInfoIndex (.tii)-->
|
2007-06-05 21:54:35 -04:00
|
|
|
TIVersion, IndexTermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermIndices
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>TIVersion -->
|
2006-11-29 15:11:38 -05:00
|
|
|
UInt32
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>IndexTermCount -->
|
2006-11-26 19:03:13 -05:00
|
|
|
UInt64
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>IndexInterval -->
|
2006-11-29 15:11:38 -05:00
|
|
|
UInt32
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>SkipInterval -->
|
2006-11-29 15:11:38 -05:00
|
|
|
UInt32
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermIndices -->
|
|
|
|
|
<TermInfo, IndexDelta>
|
|
|
|
|
<sup>IndexTermCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>IndexDelta -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VLong
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>IndexDelta
|
2006-11-29 15:11:38 -05:00
|
|
|
determines the position of this term's TermInfo within the .tis file. In
|
2006-11-26 19:03:13 -05:00
|
|
|
particular, it is the difference between the position of this term's
|
|
|
|
|
entry in that file and the position of the previous term's entry.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>SkipInterval is the fraction of TermDocs stored in skip tables. It is used to accelerate TermDocs.skipTo(int).
|
|
|
|
|
Larger values result in smaller indexes, greater acceleration, but fewer accelerable cases, while
|
2007-06-05 21:54:35 -04:00
|
|
|
smaller values result in bigger indexes, less acceleration (in case of a small value for MaxSkipLevels) and more
|
2006-11-26 19:03:13 -05:00
|
|
|
accelerable cases.</p>
|
2007-06-05 21:54:35 -04:00
|
|
|
|
|
|
|
|
<p>MaxSkipLevels is the max. number of skip levels stored for each term in the .frq file. A low value results in
|
|
|
|
|
smaller indexes but less acceleration, a larger value results in slighly larger indexes but greater acceleration.
|
|
|
|
|
See format of .frq file for more information about skip levels.</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ol>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N1072B"></a><a name="Frequencies"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Frequencies</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The .frq file contains the lists of documents
|
|
|
|
|
which contain each term, along with the frequency of the term in that
|
2008-08-05 13:17:42 -04:00
|
|
|
document (if omitTf is false).
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>FreqFile (.frq) -->
|
|
|
|
|
<TermFreqs, SkipData>
|
|
|
|
|
<sup>TermCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermFreqs -->
|
|
|
|
|
<TermFreq>
|
|
|
|
|
<sup>DocFreq</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermFreq -->
|
2008-08-05 13:17:42 -04:00
|
|
|
DocDelta[, Freq?]
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>SkipData -->
|
2007-06-05 21:54:35 -04:00
|
|
|
<<SkipLevelLength, SkipLevel>
|
|
|
|
|
<sup>NumSkipLevels-1</sup>, SkipLevel>
|
2006-11-29 15:11:38 -05:00
|
|
|
<SkipDatum>
|
2007-06-05 21:54:35 -04:00
|
|
|
</p>
|
|
|
|
|
<p>SkipLevel -->
|
|
|
|
|
<SkipDatum>
|
|
|
|
|
<sup>DocFreq/(SkipInterval^(Level + 1))</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>SkipDatum -->
|
2007-06-05 21:54:35 -04:00
|
|
|
DocSkip,PayloadLength?,FreqSkip,ProxSkip,SkipChildLevelPointer?
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2007-03-27 07:28:15 -04:00
|
|
|
<p>DocDelta,Freq,DocSkip,PayloadLength,FreqSkip,ProxSkip -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VInt
|
|
|
|
|
</p>
|
2007-06-05 21:54:35 -04:00
|
|
|
<p>SkipChildLevelPointer -->
|
|
|
|
|
VLong
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>TermFreqs
|
|
|
|
|
are ordered by term (the term is implicit, from the .tis file).
|
|
|
|
|
</p>
|
|
|
|
|
<p>TermFreq
|
|
|
|
|
entries are ordered by increasing document number.
|
|
|
|
|
</p>
|
2008-08-05 13:17:42 -04:00
|
|
|
<p>DocDelta: if omitTf is false, this determines both
|
|
|
|
|
the document number and the frequency. In
|
|
|
|
|
particular, DocDelta/2 is the difference between
|
|
|
|
|
this document number and the previous document
|
|
|
|
|
number (or zero when this is the first document in
|
|
|
|
|
a TermFreqs). When DocDelta is odd, the frequency
|
|
|
|
|
is one. When DocDelta is even, the frequency is
|
|
|
|
|
read as another VInt. If omitTf is true, DocDelta
|
|
|
|
|
contains the gap (not multiplied by 2) between
|
|
|
|
|
document numbers and no frequency information is
|
|
|
|
|
stored.
|
|
|
|
|
</p>
|
|
|
|
|
<p>For example, the TermFreqs for a term which occurs
|
|
|
|
|
once in document seven and three times in document
|
|
|
|
|
eleven, with omitTf false, would be the following
|
|
|
|
|
sequence of VInts:
|
|
|
|
|
</p>
|
|
|
|
|
<p>15, 8, 3
|
|
|
|
|
</p>
|
|
|
|
|
<p> If omitTf were true it would be this sequence
|
|
|
|
|
of VInts instead:
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
7,4
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>DocSkip records the document number before every
|
2006-11-29 15:11:38 -05:00
|
|
|
SkipInterval
|
|
|
|
|
<sup>th</sup>
|
|
|
|
|
document in TermFreqs.
|
2007-03-27 07:28:15 -04:00
|
|
|
If payloads are disabled for the term's field,
|
|
|
|
|
then DocSkip represents the difference from the
|
|
|
|
|
previous value in the sequence.
|
|
|
|
|
If payloads are enabled for the term's field,
|
|
|
|
|
then DocSkip/2 represents the difference from the
|
|
|
|
|
previous value in the sequence. If payloads are enabled
|
|
|
|
|
and DocSkip is odd,
|
|
|
|
|
then PayloadLength is stored indicating the length
|
|
|
|
|
of the last payload before the SkipInterval<sup>th</sup>
|
|
|
|
|
document in TermPositions.
|
|
|
|
|
FreqSkip and ProxSkip record the position of every
|
2006-11-29 15:11:38 -05:00
|
|
|
SkipInterval
|
|
|
|
|
<sup>th</sup>
|
|
|
|
|
entry in FreqFile and
|
|
|
|
|
ProxFile, respectively. File positions are
|
2006-11-26 19:03:13 -05:00
|
|
|
relative to the start of TermFreqs and Positions,
|
|
|
|
|
to the previous SkipDatum in the sequence.
|
|
|
|
|
</p>
|
|
|
|
|
<p>For example, if DocFreq=35 and SkipInterval=16,
|
|
|
|
|
then there are two SkipData entries, containing
|
2006-11-29 15:11:38 -05:00
|
|
|
the 15
|
|
|
|
|
<sup>th</sup>
|
|
|
|
|
and 31
|
|
|
|
|
<sup>st</sup>
|
|
|
|
|
document
|
|
|
|
|
numbers in TermFreqs. The first FreqSkip names
|
2006-11-26 19:03:13 -05:00
|
|
|
the number of bytes after the beginning of
|
2006-11-29 15:11:38 -05:00
|
|
|
TermFreqs that the 16
|
|
|
|
|
<sup>th</sup>
|
|
|
|
|
SkipDatum
|
2006-11-26 19:03:13 -05:00
|
|
|
starts, and the second the number of bytes after
|
2006-11-29 15:11:38 -05:00
|
|
|
that that the 32
|
|
|
|
|
<sup>nd</sup>
|
|
|
|
|
starts. The first
|
2006-11-26 19:03:13 -05:00
|
|
|
ProxSkip names the number of bytes after the
|
2006-11-29 15:11:38 -05:00
|
|
|
beginning of Positions that the 16
|
|
|
|
|
<sup>th</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
SkipDatum starts, and the second the number of
|
2006-11-29 15:11:38 -05:00
|
|
|
bytes after that that the 32
|
|
|
|
|
<sup>nd</sup>
|
|
|
|
|
starts.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>Each term can have multiple skip levels.
|
2007-06-05 21:54:35 -04:00
|
|
|
The amount of skip levels for a term is NumSkipLevels = Min(MaxSkipLevels, floor(log(DocFreq/log(SkipInterval)))).
|
|
|
|
|
The number of SkipData entries for a skip level is DocFreq/(SkipInterval^(Level + 1)), whereas the lowest skip
|
|
|
|
|
level is Level=0. <br>
|
|
|
|
|
Example: SkipInterval = 4, MaxSkipLevels = 2, DocFreq = 35. Then skip level 0 has 8 SkipData entries,
|
|
|
|
|
containing the 3<sup>rd</sup>, 7<sup>th</sup>, 11<sup>th</sup>, 15<sup>th</sup>, 19<sup>th</sup>, 23<sup>rd</sup>,
|
|
|
|
|
27<sup>th</sup>, and 31<sup>st</sup> document numbers in TermFreqs. Skip level 1 has 2 SkipData entries, containing the
|
|
|
|
|
15<sup>th</sup> and 31<sup>st</sup> document numbers in TermFreqs. <br>
|
|
|
|
|
The SkipData entries on all upper levels > 0 contain a SkipChildLevelPointer referencing the corresponding SkipData
|
|
|
|
|
entry in level-1. In the example has entry 15 on level 1 a pointer to entry 15 on level 0 and entry 31 on level 1 a pointer
|
|
|
|
|
to entry 31 on level 0.
|
|
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N107B3"></a><a name="Positions"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Positions</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The .prx file contains the lists of positions that
|
2008-08-05 13:17:42 -04:00
|
|
|
each term occurs at within documents. Note that
|
|
|
|
|
fields with omitTf true do not store
|
|
|
|
|
anything into this file, and if all fields in the
|
|
|
|
|
index have omitTf true then the .prx file will not
|
|
|
|
|
exist.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>ProxFile (.prx) -->
|
|
|
|
|
<TermPositions>
|
|
|
|
|
<sup>TermCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>TermPositions -->
|
|
|
|
|
<Positions>
|
|
|
|
|
<sup>DocFreq</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Positions -->
|
2007-03-27 07:28:15 -04:00
|
|
|
<PositionDelta,Payload?>
|
2006-11-29 15:11:38 -05:00
|
|
|
<sup>Freq</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2007-03-27 07:28:15 -04:00
|
|
|
<p>Payload -->
|
|
|
|
|
<PayloadLength?,PayloadData>
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>PositionDelta -->
|
2006-11-26 19:03:13 -05:00
|
|
|
VInt
|
|
|
|
|
</p>
|
2007-03-27 07:28:15 -04:00
|
|
|
<p>PayloadLength -->
|
|
|
|
|
VInt
|
|
|
|
|
</p>
|
|
|
|
|
<p>PayloadData -->
|
|
|
|
|
byte<sup>PayloadLength</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>TermPositions
|
|
|
|
|
are ordered by term (the term is implicit, from the .tis file).
|
|
|
|
|
</p>
|
|
|
|
|
<p>Positions
|
|
|
|
|
entries are ordered by increasing document number (the document
|
|
|
|
|
number is implicit from the .frq file).
|
|
|
|
|
</p>
|
|
|
|
|
<p>PositionDelta
|
2007-03-27 07:28:15 -04:00
|
|
|
is, if payloads are disabled for the term's field, the difference
|
|
|
|
|
between the position of the current occurrence in
|
2006-11-26 19:03:13 -05:00
|
|
|
the document and the previous occurrence (or zero, if this is the
|
|
|
|
|
first occurrence in this document).
|
2007-03-27 07:28:15 -04:00
|
|
|
If payloads are enabled for the term's field, then PositionDelta/2
|
|
|
|
|
is the difference between the current and the previous position. If
|
|
|
|
|
payloads are enabled and PositionDelta is odd, then PayloadLength is
|
|
|
|
|
stored, indicating the length of the payload at the current term position.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
For example, the TermPositions for a
|
|
|
|
|
term which occurs as the fourth term in one document, and as the
|
|
|
|
|
fifth and ninth term in a subsequent document, would be the following
|
2007-03-27 07:28:15 -04:00
|
|
|
sequence of VInts (payloads disabled):
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>4,
|
2006-11-26 19:03:13 -05:00
|
|
|
5, 4
|
|
|
|
|
</p>
|
2007-03-27 07:28:15 -04:00
|
|
|
<p>PayloadData
|
|
|
|
|
is metadata associated with the current term position. If PayloadLength
|
|
|
|
|
is stored at the current position, then it indicates the length of this
|
|
|
|
|
Payload. If PayloadLength is not stored, then this Payload has the same
|
|
|
|
|
length as the Payload at the previous position.
|
|
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N107EF"></a><a name="Normalization Factors"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Normalization Factors</h3>
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>There's a single .nrm file containing all norms:
|
2007-01-09 20:36:51 -05:00
|
|
|
</p>
|
|
|
|
|
<p>AllNorms
|
|
|
|
|
(.nrm) --> NormsHeader,<Norms>
|
|
|
|
|
<sup>NumFieldsWithNorms</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
<p>Norms
|
|
|
|
|
--> <Byte>
|
|
|
|
|
<sup>SegSize</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
<p>NormsHeader
|
|
|
|
|
--> 'N','R','M',Version
|
|
|
|
|
</p>
|
|
|
|
|
<p>Version
|
|
|
|
|
--> Byte
|
|
|
|
|
</p>
|
|
|
|
|
<p>NormsHeader
|
|
|
|
|
has 4 bytes, last of which is the format version for this file, currently -1.
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Each
|
2006-11-29 15:11:38 -05:00
|
|
|
byte encodes a floating point value. Bits 0-2 contain the 3-bit
|
2006-11-26 19:03:13 -05:00
|
|
|
mantissa, and bits 3-8 contain the 5-bit exponent.
|
|
|
|
|
</p>
|
|
|
|
|
<p>These
|
|
|
|
|
are converted to an IEEE single float value as follows:
|
|
|
|
|
</p>
|
|
|
|
|
<ol>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>If
|
|
|
|
|
the byte is zero, use a zero float.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Otherwise,
|
|
|
|
|
set the sign bit of the float to zero;
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>add
|
|
|
|
|
48 to the exponent and use this as the float's exponent;
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>map
|
|
|
|
|
the mantissa to the high-order 3 bits of the float's mantissa; and
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>set
|
|
|
|
|
the low-order 21 bits of the float's mantissa to zero.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ol>
|
2007-01-09 20:36:51 -05:00
|
|
|
<p>A separate norm file is created when the norm values of an existing segment are modified.
|
|
|
|
|
When field <em>N</em> is modified, a separate norm file <em>.sN</em>
|
|
|
|
|
is created, to maintain the norm values for that field.
|
|
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>Separate norm files are created (when adequate) for both compound and non compound segments.
|
2007-01-09 20:36:51 -05:00
|
|
|
</p>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N10840"></a><a name="Term Vectors"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Term Vectors</h3>
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>
|
|
|
|
|
Term Vector support is an optional on a field by
|
2009-03-18 17:42:32 -04:00
|
|
|
field basis. It consists of 3 files.
|
2008-02-01 20:00:04 -05:00
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<ol>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="tvx"></a>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>The Document Index or .tvx file.</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>For each document, this stores the offset
|
|
|
|
|
into the document data (.tvd) and field
|
|
|
|
|
data (.tvf) files.
|
2006-11-29 15:11:38 -05:00
|
|
|
</p>
|
|
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>DocumentIndex (.tvx) --> TVXVersion<DocumentPosition,FieldPosition>
|
2006-11-29 15:11:38 -05:00
|
|
|
<sup>NumDocs</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>TVXVersion --> Int (TermVectorsReader.CURRENT)</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>DocumentPosition --> UInt64 (offset in
|
|
|
|
|
the .tvd file)</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>FieldPosition --> UInt64 (offset in the
|
|
|
|
|
.tvf file)</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="tvd"></a>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>The Document or .tvd file.</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>This contains, for each document, the number of fields, a list of the fields with
|
2006-11-29 15:11:38 -05:00
|
|
|
term vector info and finally a list of pointers to the field information in the .tvf
|
|
|
|
|
(Term Vector Fields) file.</p>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>
|
2008-02-01 20:00:04 -05:00
|
|
|
Document (.tvd) --> TVDVersion<NumFields, FieldNums, FieldPositions>
|
2006-11-29 15:11:38 -05:00
|
|
|
<sup>NumDocs</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>TVDVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>NumFields --> VInt</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
|
|
|
|
<p>FieldNums --> <FieldNumDelta>
|
|
|
|
|
<sup>NumFields</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>FieldNumDelta --> VInt</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>FieldPositions --> <FieldPositionDelta>
|
|
|
|
|
<sup>NumFields-1</sup>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2008-02-01 20:00:04 -05:00
|
|
|
<p>FieldPositionDelta --> VLong</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>The .tvd file is used to map out the fields that have term vectors stored and
|
2006-11-29 15:11:38 -05:00
|
|
|
where the field information is in the .tvf file.</p>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<li>
|
2009-08-22 21:18:52 -04:00
|
|
|
<a name="tvf"></a>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>The Field or .tvf file.</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>This file contains, for each field that has a term vector stored, a list of
|
2006-12-16 15:23:30 -05:00
|
|
|
the terms, their frequencies and, optionally, position and offest information.</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-12-16 15:23:30 -05:00
|
|
|
<p>Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs>
|
2006-11-29 15:11:38 -05:00
|
|
|
<sup>NumFields</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<p>TVFVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>NumTerms --> VInt</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-12-16 15:23:30 -05:00
|
|
|
<p>Position/Offset --> Byte</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-12-16 15:23:30 -05:00
|
|
|
<p>TermFreqs --> <TermText, TermFreq, Positions?, Offsets?>
|
2006-11-29 15:11:38 -05:00
|
|
|
<sup>NumTerms</sup>
|
|
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>TermText --> <PrefixLength, Suffix></p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>PrefixLength --> VInt</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>Suffix --> String</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>TermFreq --> VInt</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-12-16 15:23:30 -05:00
|
|
|
<p>Positions --> <VInt><sup>TermFreq</sup>
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>Offsets --> <VInt, VInt><sup>TermFreq</sup>
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<br>
|
|
|
|
|
|
|
|
|
|
<p>Notes:</p>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
|
|
|
|
<li>Position/Offset byte stores whether this term vector has position or offset information stored.</li>
|
|
|
|
|
|
|
|
|
|
<li>Term
|
|
|
|
|
text prefixes are shared. The PrefixLength is the number of initial
|
|
|
|
|
characters from the previous term which must be pre-pended to a
|
|
|
|
|
term's suffix in order to form the term's text. Thus, if the
|
|
|
|
|
previous term's text was "bone" and the term is "boy",
|
|
|
|
|
the PrefixLength is two and the suffix is "y".
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li>Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position</li>
|
|
|
|
|
|
|
|
|
|
<li>Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</li>
|
2006-11-29 15:11:38 -05:00
|
|
|
|
2006-11-26 19:03:13 -05:00
|
|
|
</ol>
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N108DC"></a><a name="Deleted Documents"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h3 class="boxed">Deleted Documents</h3>
|
|
|
|
|
<p>The .del file is
|
2006-12-13 05:08:28 -05:00
|
|
|
optional, and only exists when a segment contains deletions.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
2006-12-13 05:08:28 -05:00
|
|
|
<p>Although per-segment, this file is maintained exterior to compound segment files.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Deletions
|
|
|
|
|
(.del) --> [Format],ByteCount,BitCount, Bits | DGaps (depending on Format)
|
|
|
|
|
</p>
|
|
|
|
|
<p>Format,ByteSize,BitCount -->
|
2006-11-26 19:03:13 -05:00
|
|
|
Uint32
|
|
|
|
|
</p>
|
2006-11-29 15:11:38 -05:00
|
|
|
<p>Bits -->
|
|
|
|
|
<Byte>
|
|
|
|
|
<sup>ByteCount</sup>
|
2006-11-26 19:03:13 -05:00
|
|
|
|
|
|
|
|
</p>
|
2006-12-13 05:08:28 -05:00
|
|
|
<p>DGaps -->
|
|
|
|
|
<DGap,NonzeroByte>
|
|
|
|
|
<sup>NonzeroBytesCount</sup>
|
|
|
|
|
|
|
|
|
|
</p>
|
|
|
|
|
<p>DGap -->
|
|
|
|
|
VInt
|
|
|
|
|
</p>
|
|
|
|
|
<p>NonzeroByte -->
|
|
|
|
|
Byte
|
|
|
|
|
</p>
|
|
|
|
|
<p>Format
|
|
|
|
|
is Optional. -1 indicates DGaps. Non-negative value indicates Bits, and that Format is excluded.
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
<p>ByteCount
|
2006-11-29 15:11:38 -05:00
|
|
|
indicates the number of bytes in Bits. It is typically
|
2006-11-26 19:03:13 -05:00
|
|
|
(SegSize/8)+1.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
BitCount
|
|
|
|
|
indicates the number of bits that are currently set in Bits.
|
|
|
|
|
</p>
|
|
|
|
|
<p>Bits
|
2006-11-29 15:11:38 -05:00
|
|
|
contains one bit for each document indexed. When the bit
|
2006-11-26 19:03:13 -05:00
|
|
|
corresponding to a document number is set, that document is marked as
|
2006-11-29 15:11:38 -05:00
|
|
|
deleted. Bit ordering is from least to most significant. Thus, if
|
2006-11-26 19:03:13 -05:00
|
|
|
Bits contains two bytes, 0x00 and 0x02, then document 9 is marked as
|
|
|
|
|
deleted.
|
|
|
|
|
</p>
|
2006-12-13 05:08:28 -05:00
|
|
|
<p>DGaps
|
|
|
|
|
represents sparse bit-vectors more efficiently than Bits.
|
|
|
|
|
It is made of DGaps on indexes of nonzero bytes in Bits,
|
|
|
|
|
and the nonzero bytes themselves. The number of nonzero bytes
|
|
|
|
|
in Bits (NonzeroBytesCount) is not stored.
|
|
|
|
|
</p>
|
|
|
|
|
<p>For example,
|
|
|
|
|
if there are 8000 bits and only bits 10,12,32 are set,
|
|
|
|
|
DGaps would be used:
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
(VInt) 1 , (byte) 20 , (VInt) 3 , (Byte) 1
|
|
|
|
|
</p>
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
2009-08-25 10:36:47 -04:00
|
|
|
<a name="N10916"></a><a name="Limitations"></a>
|
2006-11-26 19:03:13 -05:00
|
|
|
<h2 class="boxed">Limitations</h2>
|
|
|
|
|
<div class="section">
|
2009-04-04 06:18:05 -04:00
|
|
|
<p>
|
|
|
|
|
When referring to term numbers, Lucene's current
|
|
|
|
|
implementation uses a Java <span class="codefrag">int</span>, which means
|
|
|
|
|
the maximum number of unique terms in any single index
|
|
|
|
|
segment is 2,147,483,648. This is technically not a
|
|
|
|
|
limitation of the index file format, just of Lucene's
|
|
|
|
|
current implementation.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
Similarly, Lucene uses a Java <span class="codefrag">int</span> to refer
|
|
|
|
|
to document numbers, and the index file format uses an
|
|
|
|
|
<span class="codefrag">Int32</span> on-disk to store document numbers.
|
|
|
|
|
This is a limitation of both the index file format and
|
|
|
|
|
the current implementation. Eventually these should be
|
|
|
|
|
replaced with either <span class="codefrag">UInt64</span> values, or
|
|
|
|
|
better yet, <span class="codefrag">VInt</span> values which have no
|
|
|
|
|
limit.
|
2006-11-26 19:03:13 -05:00
|
|
|
</p>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end content
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="clearboth"> </div>
|
|
|
|
|
</div>
|
|
|
|
|
<div id="footer">
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|start bottomstrip
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
<div class="lastmodified">
|
|
|
|
|
<script type="text/javascript"><!--
|
2007-06-26 00:26:07 -04:00
|
|
|
document.write("Last Published: " + document.lastModified);
|
2006-11-26 19:03:13 -05:00
|
|
|
// --></script>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="copyright">
|
|
|
|
|
Copyright ©
|
|
|
|
|
2006 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
|
|
|
|
|
</div>
|
2007-06-26 00:26:07 -04:00
|
|
|
<!--+
|
|
|
|
|
|end bottomstrip
|
|
|
|
|
+-->
|
2006-11-26 19:03:13 -05:00
|
|
|
</div>
|
|
|
|
|
</body>
|
|
|
|
|
</html>
|
