The current logic for reordering splits a slice of doc IDs into a left side and
a right side, and for each document it computes the expected gain of moving to
the other side. Then it swaps documents from both sides as long as the sum of
the gain of moving the left doc to the right and the right doc to the left is
positive.
This works well, but I would like to extend BP reordering to also work with
blocks, and the swapping logic is challenging to modify as two parent documents
may have different numbers of children.
One of the follow-up papers on BP suggested to use a different logic, where one
would compute a bias for all documents that is negative when a document is
attracted to the left and positive otherwise. Then we only have to partition doc
IDs around the mid point, e.g. with quickselect.
A benefit of this change is that it will make it easier to generalize BP
reordering to indexes that have blocks, e.g. by using a stable sort on biases.
This commit updates the FieldInfosFormat translation of vector similarity functions to be independent of the VectorSimilartyFunction enum.
The VectorSimilartyFunction enum lives outside of the codec format, and the format should not inadvertently depend upon the declaration order or values in VectorSimilartyFunction. The format should be in charge of the translation of similarity function to format ordinal (and visa versa). In reality, and for now, the translation remains the same as the declaration order, but this may not be the case in the future.
Getting the access of a FeatureField#value is useful for deduplicating. If you have a sparse vector model and you want to handle multiple inputs from them, you want flexibility in how you de-duplicate the feature dimensions.
The original tests assume particular document orders & scores. To make the test more resilient to random flushes & merges, I adjusted the assertion conditions. Particularly, we verify matching ids -> score instead of relying on docIds.
closes: https://github.com/apache/lucene/issues/13057
In #13046 several changes broke the addBackcompatIndexes.py script
to properly add and test the unreleased version. This updates the
script to again properly add the new version.
Closes#13094
Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
This enables the optional parent field in BWC tests from 9.10 on.
This will need to be forward ported to main branch where the
parent field is required to these tests since they add document blocks
during tests.
Dmitry Cherniachenko
Dawid Weiss
Stefan Vodita
Mikhail Khludnev
Andrey Bozhko
Chris Hostetter
Shubham Chaudhary
Zhang Chao
Uwe Schindler
Adrien Grand
Chris Hegarty
Prabhat
Benjamin Trent
Simon Willnauer
Christine Poerschke
Houston Putman