-Plain text passed to Lucene for indexing goes through a process generally called tokenization – namely breaking of the -input text into small indexing elements – tokens. -The way input text is broken into tokens very -much dictates further capabilities of search upon that text. +Plain text passed to Lucene for indexing goes through a process generally called tokenization. Tokenization is the process +of breaking input text into small indexing elements – tokens. +The way input text is broken into tokens heavily influences how people will then be able to search for that text. For instance, sentences beginnings and endings can be identified to provide for more accurate phrase and proximity searches (though sentence identification is not provided by Lucene).
-In some cases simply breaking the input text into tokens is not enough – a deeper Analysis is needed, -providing for several functions, including (but not limited to): +In some cases simply breaking the input text into tokens is not enough – a deeper Analysis may be needed. +There are many post tokenization steps that can be done, including (but not limited to):