The exception happen because the tail postings list block, which encoding with GroupVInt, had a docID delta that was >= 1<<30, when the postings are also storing freqs.
We hit a Codec bug in Elasticsearch, but it went unnoticed because our
tests extend from BaseDocValuesFormatTestCase, which doesn't attempt to
read the doc-values of the same document twice. This change strengthens
BaseDocValuesFormatTestCase checks and randomly inserts that access
pattern.
In #13046 several changes broke the addBackcompatIndexes.py script
to properly add and test the unreleased version. This updates the
script to again properly add the new version.
Closes#13094
Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
The initial release of scalar quantization would periodically create a humongous allocation, which can put unwarranted pressure on the GC & on the heap usage as a whole.
This commit adjusts this by only allocating a float array of 20*dimensions and averaging the discovered quantiles from there.
Why does this work?
- Quantiles based on confidence intervals are (generally) unbiased and doing an average gives statistically good results
- The selector algorithm scales linearly, so the cost is just about the same
- We need to do more than `1` vector at a time to prevent extreme confidence intervals interacting strangely with edge cases
I noticed while experimenting with brute-force search that our visitation limit is EXACTLY the number of filtered docs to hit. Consequently, if we happen to do brute force search and visit that exact number of vectors, we will fall back again to do brute-force a second time. This struck me as weird.
This commit adjusts the visit limit threshold for approximate search to account for this.
Speedup concurrent multi-segment HNWS graph search by exchanging
the global top candidated collected so far across segments. These global top
candidates set the minimum threshold that new candidates need to pass
to be considered. This allows earlier stopping for segments that don't have
good candidates.
This change modernizes the BWC tests to leverage RandomizedRunners Parameterized Tests that allow us to have more structured and hopefully more extendible BWC tests in the future. This change doesn't add any new tests but tries to make the ones we have more structured and support growth down the road.
Basically, every index type got it's own Test class that doesn't require to loop over all the indices in each test. Each test case is run with all versions specified. Several sanity checks are applied in the base class to make individual tests smaller and much easier to read.
Co-authored-by: Michael McCandless <lucene@mikemccandless.com>
Co-authored-by: Adrien Grand <jpountz@gmail.com>
Zhang Chao
Nhat Nguyen
Adrien Grand
Simon Willnauer
Dawid Weiss
Uwe Schindler
Christine Poerschke
Houston Putman
Benjamin Trent
Stefan Vodita
Dzung Bui
Mayya Sharipova
Dmitry Cherniachenko
Johannes Fredén