Adds new int8 scalar quantization for HNSW codec. This uses a new lucene9.9 format and auto quantizes floating point vectors into bytes on flush and merge.
Apache committers who opt-in (via authentication) can have their local build scans be submitted to ge.apache.org.
Co-authored-by: Clay Johnson <cjohnson@gradle.com>
The idea behind MAXSCORE is to run disjunctions as `+(essentialClause1 ...
essentialClauseM) nonEssentialClause1 ... nonEssentialClauseN`, moving more and
more clauses from the essential list to the non-essential list as the minimum
competitive score increases. For instance, a query such as `the book of life`
which I found in the Tantivy benchmark ends up running as `+book the of life`
after some time, ie. with one required clause and other clauses optional. This
is because matching `the`, `of` and `life` alone is not good enough for
yielding a match.
Here some statistics in that case:
- min competitive score: 3.4781857
- max_window_score(book): 2.8796153
- max_window_score(life): 2.037863
- max_window_score(the): 0.103848875
- max_window_score(of): 0.19427927
Actually if you look at these statistics, we could do better, because a match
may only be competitive if it matches both `book` and `life`, so this query
could actually execute as `+book +life the of`, which may help evaluate fewer
documents compared to `+book the of life`. Especially if you enable recursive
graph bisection.
This is what this PR tries to achieve: in the event when there is a single
essential clause and matching all clauses but the best non-essential clause
cannot produce a competitive match, then the scorer will only evaluate
documents that match the intersection of the essential clause and the best
non-essential clause.
It's worth noting that this optimization would kick in very frequently on
2-clauses disjunctions.
When operations are parallelized, like query rewrite, or search, or
createWeight, one of the tasks may throw an exception. In that case we
wait for all tasks to be completed before re-throwing the exception that
were caught. Tasks that were not started when the exception is captured
though can be safely skipped. Ideally we would also cancel ongoing tasks
but I left that for another time.
Currently, merge-on-full-flush only checks if merges need to run if changes
have been flushed to disk. This prevents from having different merging logic
for refreshes and commits, since the merge policy would not be checked upon
commit if no new documents got indexed since the previous refresh.
If the add/updateDocuments(List<>) API is used, lucene guarantees that
all documents are indexed in the same segment with consecutive document IDs.
This enables features like nested documents etc. This change records the usage
of this API in SegmentsInfo and preserves this property across merges.
Relates to #12665
* tweak comments; change if to switch
* remove old SOPs, minor comment styling, fixed silly performance bug on rehash using the wrong bitsRequired (count vs node)
* first raw cut; some nocommits added; some tests fail
* tests pass!
* fix silly fallback hash bug
* remove SOPs; add some temporary debugging metrics
* add temporary tool to test FST performance across differing NodeHash sizes
* remove (now deleted) shouldShareNonSingletonNodes call from Lucene90BlockTreeTermsWriter
* add simple tool to render results table to GitHub MD
* add simple temporary tool to iterate all terms from a provided luceneutil wikipedia index and build an FST from them
* first cut at using packed ints for hash t able again
* add some nocommits; tweak test_all_sizes.py to new RAM usage approach; when half of the double barrel is full, allocate new primary hash at full size to save cost of continuously rehashing for a large FST
* switch to limit suffix hash by RAM usage not count (more intuitive for users); clean up some stale nocommits
* switch to more intuitive approximate RAM (mb) limit for allowed size of NodeHash
* nuke a few nocommits; a few more remain
* remove DO_PRINT_HASH_RAM
* no more FST pruning
* remove final nocommit: randomly change allowed NodeHash suffix RAM size in TestFSTs.testRealTerms
* remove SOP
* tidy
* delete temp utility tools
* remove dead (FST pruning) code
* add CHANGES entry; fix one missed fst.addNode -> fstCompiler.addNode during merge conflict resolution
* remove a mal-formed nocommit
* fold PR feedback
* fold feedback
* add gradle help test details on how to specify heap size for the test JVM; fix bogus assert (uncovered by Test2BFST); add TODO to Test2BFST anticipating building massive FSTs in small bounded RAM
* suppress sysout checks for Test2BFSTs; add helpful comment showing how to run it directly
* tidy
Moved all the hairy allocSlice stuff as static method in TermsHashPerField and I introduce a BytesRefBlockPool to
encapsulate of the BytesRefHash write/read logic.
While working on the quantization codec & thinking about how merging will evolve, it became clearer that having merging attached directly to the vector writer is weird.
I extracted it out to its own class and removed the "initializedNodes" logic from the base class builder.
Also, there was on other refactoring around grabbing sorted nodes from the neighbor iterator, I just moved that static method so its not attached to the writer (as all bwc writers need it and all future HNSW writers will as well).
When we initially introduced support for dynamic pruning, we had an
implementation of WAND that would almost exclusively use `advance()`. Now that
we switched to MAXSCORE and rely much more on `nextDoc()`, it makes sense to
specialize nextDoc() as well.
This changes the following:
- fewer docs indexed in non-nightly runs,
- `QueryUtils#checkFirstSkipTo` uses the `ScorerSupplier` API to convey it
will only check one doc,
- `QueryUtils#checkFirstSkipTo` no longer relies on timing to run in a
reasonably amount of time.
This test sometimes fails because `SimpleText` has a non-deterministic size for
its segment info file, due to escape characters. The test now enforces the
default codec, and checks that segments have the expected size before moving
forward with forcemerge().
Closes#12648
Benjamin Trent
Nelson Osacky
Adrien Grand
Luca Cavanna
Shubham Chaudhary
Simon Willnauer
Uwe Schindler
Dzung Bui
Rob Muir
Michael McCandless
gf2121
Stéphane Campinas
iverase
Dawid Weiss
Patrick Zhai
Robert Muir
Greg Miller