The AIO search index is built in a WebWorker on the browser from a set
of page information that is downloaded as a JSON file (`search-data.json`).
We want to keep this file as small as possible while providing enough
data to generate a useful index to query against.
Previously, we only included one copy of each (non-ignored) term from each
doc but this prevents more subtle ranking of query results, since the number
of occurences of a term in a doc is lost.
This commit changes the generated file in the following ways:
- All non-ignored terms are now included in the order in which they appear
in the doc.
- The terms are indexed into a dictonary to avoid the text of the term being
repeated in every doc that contains the term.
- Each term is pre-"stemmed" using the same Porter Stemming algorith that the
Lunr search engine uses.
The web-worker has been updated to decode the new format of the file.
Now that all terms are included, it may enable some level of phrase based
matching in the future.
The size of the generated file is considerably larger than previously, but
on production HTTP servers the data is sent compressed, which reduces the
size dramatically.
PR Close#41368
Pete Bacon Darwin