The initial release of scalar quantization would periodically create a humongous allocation, which can put unwarranted pressure on the GC & on the heap usage as a whole.
This commit adjusts this by only allocating a float array of 20*dimensions and averaging the discovered quantiles from there.
Why does this work?
- Quantiles based on confidence intervals are (generally) unbiased and doing an average gives statistically good results
- The selector algorithm scales linearly, so the cost is just about the same
- We need to do more than `1` vector at a time to prevent extreme confidence intervals interacting strangely with edge cases
I noticed while experimenting with brute-force search that our visitation limit is EXACTLY the number of filtered docs to hit. Consequently, if we happen to do brute force search and visit that exact number of vectors, we will fall back again to do brute-force a second time. This struck me as weird.
This commit adjusts the visit limit threshold for approximate search to account for this.
Speedup concurrent multi-segment HNWS graph search by exchanging
the global top candidated collected so far across segments. These global top
candidates set the minimum threshold that new candidates need to pass
to be considered. This allows earlier stopping for segments that don't have
good candidates.
This change modernizes the BWC tests to leverage RandomizedRunners Parameterized Tests that allow us to have more structured and hopefully more extendible BWC tests in the future. This change doesn't add any new tests but tries to make the ones we have more structured and support growth down the road.
Basically, every index type got it's own Test class that doesn't require to loop over all the indices in each test. Each test case is run with all versions specified. Several sanity checks are applied in the base class to make individual tests smaller and much easier to read.
Co-authored-by: Michael McCandless <lucene@mikemccandless.com>
Co-authored-by: Adrien Grand <jpountz@gmail.com>
If the required clause doesn't contribute scores, which typically happens if
the required clause is a `FILTER` clause, then the minimum competitive score
can be propagated directly to the optional clause.
This change makes `PointRangeQuery` exit early when it knows that it doesn't
match a segment. In the case when this query is part of a conjunction, this
helps make sure `ScorerSupplier#get` doesn't get called on other required
clauses, which is sometimes an expensive operation (e.g. multi-term queries).
This is especially relevant for time-based data combined with
`LogByteSizePolicy`, which gives non-adjacent ranges of timestamps to each
segment, which in-turn increases the likelihood that some segments won't match
a range query at all.
Houston Putman
Benjamin Trent
Stefan Vodita
Christine Poerschke
Zhang Chao
Dawid Weiss
Mayya Sharipova
Uwe Schindler
Shubham Chaudhary
Dzung Bui
Dmitry Cherniachenko
Adrien Grand
Johannes Fredén
Simon Willnauer
ChrisHegarty
Shintaro Murakami