There is a tricky race condition with DWPT threads. It is possible that a flush starts by advancing the deleteQueue (in charge of creating seqNo). Thus, the referenced deleteQueue, there should be a cap on the number of actions left.
However, it is possible after the advance, but before the DWPT are actually marked for flush, the DWPT gets freed and taken again to be used.
To replicate this extreme behavior, see: https://github.com/apache/lucene/compare/main...benwtrent:lucene:test-replicate-and-debug-13127?expand=1
This commit will prevent DWPT from being added back to the free list if their queue has been advanced. This is because the `maxSeqNo` for that queue was created accounting only for the current number of active threads. If the thread gets passed out again and still references the already advanced queue, it is possible that seqNo actually advances past the set `maxSeqNo`.
closes: https://github.com/apache/lucene/issues/13127
closes: https://github.com/apache/lucene/issues/13571
A number of functions in CandidateMatcher are protected or package-protected,
meaning that client code can't use them, which makes it difficult to build custom
wrapper matchers. This commit makes these functions public
When quantizing vectors in a COSINE vector space, we normalize them. However, there is a bug when building the quantizer quantiles and we didn't always use the normalized vectors. Consequently, we would end up with poorly configured quantiles and recall will drop significantly (especially in sensitive cases like int4).
closes: #13614
Implement Kmeans clustering algorithm for vectors.
Knn algorithms that further reduce memory usage of vectors (such as Product Quantization,
RaBitQ etc) require clustering of vectors. This implements KMeans clustering algorithm.
Co-authored-by: Jim Ferenczi jim.ferenczi@elastic.co
Opening for more pointed discussion. See latest discussion here: #13281
I was hoping to have a full answer for folks who use byte models by Lucene 10, but I just don't have that.
I still want to remove the internal cosine optimized methods. We can do this if we store magnitudes along side the raw vectors. This way we can remove all the internal optimized cosine code as its complicated.
I noticed that single-term readers are an edge case but not that
uncommon in Elasticsearch heap dumps. It seems quite common to have a
constant value for some field across a complete segment (e.g. a version
value that is repeated endlessly in logs).
Seems simple enough to deduplicate here to save a couple MB of heap.
Looking at how these instances are serialized to disk it appears
that the empty output in the FST metadata is always the same as the
rootCode bytes.
Without changing the serialization we could at least deduplicate here,
saving hundreds of MB in some high-segment count use cases I observed in
ES.
Make it so rejected tasks are execute right away on the caller thread.
Users of the API shouldn't have to worry about rejections when we don't
expose any upper limit to the task count that we put on the executor that
would help in sizing a queue for the executor.
This updates the postings format in order to inline skip data into postings. This format is generally similar to the current `Lucene99PostingsFormat`, e.g. it shares the same block encoding logic, but it has a few differences:
- Skip data is inlined into postings to make the access pattern more sequential.
- There are only 2 levels of skip data: on every block (128 docs) and every 32 blocks (4,096 docs).
In general, I found that the fact that skip data is inlined may slow down a bit queries that don't need skip data at all (e.g. `CountOrXXX` tasks that never advance of consult impacts) and speed up a bit queries that advance by small intervals. The fact that the greatest level only allows skipping 4096 docs at once means that we're slower at advancing by large intervals, but data suggests that it doesn't significantly hurt performance.
This adds a Directory wrapper that counts how many times we wait for I/O to
complete before doing something else, and adds tests that the default codec is
able to parallelize I/O for stored fields retrieval and term lookups.
There's a couple of places in the codebase where we extend `IndexSearcher` to customize
per leaf behaviour, and in order to do that, we need to override the entire search method
that loops through the leaves. A good example is `ScorerIndexSearcher`.
Adding a `searchLeaf` method that provides the per leaf behaviour makes those cases a little
easier to deal with.
It's been pointed multiple times that a difference between Tantivy and Lucene
is the fact that Tantivy uses windows of 4,096 docs when Lucene has a 2x
smaller window size of 2,048 docs and that this might explain part of the
performance difference. luceneutil suggests that bumping the window size to
4,096 does indeed improve performance for counting queries, but not for top-k
queries. I'm still suggesting to bump the window size across the board to keep
our disjunction scorer consistent.
Something I found in a heap dump. For large numbers of `FieldReader`
where the minimum term is an empty string, we allocate MBs worth of
empty `byte[]` in ES. Worth adding the conditional here I think.
This iterates on #13546 to further reduce the overhead of search concurrency by
caching whether the hit count threshold has been reached: once the threshold
has been reached, it cannot get "un-reached" again, so we don't need to pay the
cost of `LongAdder#longValue`.
* lazily write the FST padding byte
* Also write the pad byte when there is emptyOutput
* add comment
* Make Lucene90BlockTreeTermsWriter to write FST off-heap
* Add change log
* Tidy code & Add comments
* use temp IndexOutput for FST writing
* Use IOUtils to delete files
* Update CHANGES.txt
* Update CHANGES.txt
I analyzed a heap dump of Elasticsearch where FixedBitSet uses more than
1GB of memory. Most of these FixedBitSets are used by soft-deletes
reader wrappers, even though these segments have no deletes at all. I
believe these segments previously had soft-deletes, but these deletes
were pruned by merges. The reason we wrap soft-deletes is that the
soft-deletes field exists. Since these segments had soft-deletes
previously, we carried the field-infos into the new segment. Ideally, we
should have ways to check whether the returned docValues iterator is
empty or not so that we can avoid allocating FixedBitSet completely, or
we should prune fields without values after merges.
Ankit Jain
Michael Froh
Christine Poerschke
Michael Sokolov
Michael Sokolov
Benjamin Trent
bjacobowitz
Adrien Grand
Kaival Parikh
Mayya Sharipova
zhouhui
Jakub Slowinski
Zhang Chao
Greg Miller
Greg Miller
Egor Potemkin
Armin Braun
Luca Cavanna
Peter Gromov
Chris Hegarty
Dzung Bui
Michael McCandless
Ignacio Vera
Nhat Nguyen