Commit Graph

37794 Commits

Author SHA1 Message Date
Viliam Durina be66af249f
Fix changelog for GITHUB#14011 (#14018)
* Fix changelog for GITHUB#14011
2024-12-01 11:41:10 -05:00
Adrien Grand 06a320a53e Rewrite queries with no SHOULD clauses and minimumShouldMatch > 0 to a MatchNoDocsQuery.
Closes #14026
2024-11-29 14:47:44 +01:00
Adrien Grand f9869b54d5
Make inlining decisions a bit more predictable in our main queries. (#14023)
This implements a small contained hack to make sure that our compound scorers
like `MaxScoreBulkScorer`, `ConjunctionBulkScorer`,
`BlockMaxConjunctionBulkScorer`, `WANDScorer` and `ConjunctionDISI` only have
two concrete implementations of `DocIdSetIterator` and `Scorable` to deal with.

This helps because it makes calls to `DocIdSetIterator#nextDoc()`,
`DocIdSetIterator#advance(int)` and `Scorable#score()` bimorphic at most, and
bimorphic calls are candidate for inlining.

This should help speed up boolean queries of term queries at the expense of
boolean queries of other query types. This feels fair to me as it gives more
speedups than slowdowns in benchmarks, and that boolean queries of term queries
are extremely typical. Boolean queries that mix term queries and other types of
queries may get a slowdown or a speedup depending on whether they get more from
the speedup on their term clauses than they lose on their other clauses.
2024-11-29 13:27:49 +01:00
ChrisHegarty 70530a92d9 add missing changes log entry for 13998 2024-11-29 10:32:45 +00:00
Chris Hegarty 7dbbd0daa9
Add IndexInput isLoaded (#13998)
This commit adds IndexInput::isLoaded to help determine if the contents of an input is resident in physical memory.

The intent of this new method is to help build inspection and diagnostic infrastructure on top.
2024-11-29 10:28:32 +00:00
Adrien Grand 98c59a710e
Run filtered disjunctions with MaxScoreBulkScorer. (#14014)
Running filtered disjunctions with a specialized bulk scorer seems to yield a
good speedup. For what it's worth, I also tried to implement a MAXSCORE-based
scorer to see if it had to do with the `BulkScorer` specialization or the
algorithm, but it didn't help.

To work properly, I had to add a rewrite rule to inline disjunctions in a MUST
clause.

As a next step, it would be interesting to see if we can further optimize this
by loading the filter into a bitset and applying it like live docs.
2024-11-27 21:56:03 +01:00
Adrien Grand d9aa525c9e
Make WANDScorer compute scores on the fly. (#14021)
Currently, `WANDSCorer` considers that a hit is a match if the sum of maximum
scores across clauses is more than or equal to the minimum competitive score.
We can do better by computing scores of leading clauses on the fly. This helps
because scores are often lower than the score upper bound, so using actual
scores instead of score upper bounds can help skip advancing more clauses.

For reference, we are already doing the same trick in our conjunction (bulk)
scorers and in `MaxScoreBulkScorer` (bulk scorer for top-level disjunctions).
2024-11-26 16:00:30 +01:00
Jean-François BOEUF 534f815927
Improve checksum calculations (#13989)
Take advantage of the existing buffer in BufferedChecksum to speed up
reads for Longs, Ints, Shorts and Long arrays by avoiding byte-by-byte
reads.
2024-11-25 15:59:15 +01:00
Adrien Grand a5bf8a58b3
Stop using `SlowImpactsEnum` for terms whose `docFreq` is less than 128. (#14017)
We currently use `SlowImpactsEnum` for terms whose `docFreq` is less than 128
because it's convenient as these terms don't have impacts anyway. But a recent
slowdown on nightly benchmarks suggests that this contributes to making some
hot calls more polymorphic than we'd like, so this PR moves such terms back to
the regular impacts enums.
2024-11-25 15:48:27 +01:00
Stefan Vodita d19b8425dd Fix changelog for GITHUB#14008 2024-11-23 12:43:34 +00:00
Paul King adc1789b3b
Taxonomy counts are incorrect due to ordinal sorting (#14008) (#14010) 2024-11-23 12:03:23 +00:00
Sascha Szott ad2773b4ef
fix JavaDoc: TopDocs instead of Hits (#14015) 2024-11-22 15:56:58 -08:00
Adrien Grand de1feea401
Make CombinedFieldQuery eligible for WAND/MAXSCORE. (#13999)
`CombinedFieldQuery` currently returns an infinite maximum score. We can do
better by returning the maximum score that the sim scorer can return, which in
the case of BM25 is bounded by the IDF. This makes CombinedFieldQuery eligible
for WAND/MAXSCORE (not their block-max variants though, since we return the
same score upper bound for the whole index).
2024-11-22 18:18:51 +01:00
Adrien Grand 90fee7e13f
Only consider clauses whose cost is less than the lead cost to compute block boundaries in WANDScorer. (#14003)
WANDScorer implements block-max WAND and needs to recompute score upper bounds
whenever it moves to a different block. Thus it's important for these blocks to
be large enough to avoid re-computing score upper bounds over and over again.
With this commit, WANDScorer no longer uses clauses whose cost is higher than
the cost of the filter to compute block boundaries. This effectively makes
blocks larger when the filter is more selective.
2024-11-22 18:18:00 +01:00
Viliam Durina 532d267228
Reduce allocation rate in HNSW concurrent merge (#14011)
The PR removes the allocation of a new `LockedRow` for each locking operation in `HnswLock`. Even though the object was very quickly released, and JIT supports on-stack allocation, it didn't happen in my experiments on OpenJDK 21 - it's easy to avoid the allocation, rather than rely on the JIT.
2024-11-22 11:48:50 -05:00
Tejas Shah 46204f6b53
Introduces IndexInput#updateReadAdvice to change the ReadAdvice while merging vectors (#13985)
The commit updates the ReadAdvice.SEQUENTIAL before the merge starts and reverts it to ReadAdvice.RANDOM at the end of the merge for Lucene99FlatVectorsReader.
2024-11-22 12:14:49 +00:00
Paul King 5eeb760dae
Update ComplexPhraseQueryParser.java (#14005)
Fix typo
2024-11-21 09:13:23 +03:00
Adrien Grand 9b5edabf21 Revert "Only consider clauses whose cost is less than the lead cost to compute block boundaries in WANDScorer. (#14000)"
This reverts commit 5807ff1620.
2024-11-20 10:15:44 +01:00
Adrien Grand 4aeecdfebf
Speed up top-k retrieval of filtered disjunctions a bit. (#13996)
This moves work from `advance(int target)` to `TwoPhaseIterator#matches()` so
that we do less work on hits that do not match the filter.
2024-11-19 09:53:45 +01:00
Adrien Grand 5807ff1620
Only consider clauses whose cost is less than the lead cost to compute block boundaries in WANDScorer. (#14000)
WANDScorer implements block-max WAND and needs to recompute score upper bounds
whenever it moves to a different block. Thus it's important for these blocks to
be large enough to avoid re-computing score upper bounds over and over again.
With this commit, WANDScorer no longer uses clauses whose cost is higher than
the cost of the filter to compute block boundaries. This effectively makes
blocks larger when the filter is more selective.
2024-11-19 09:27:13 +01:00
Viswanath Kuchibhotla a991a9a861
Adding filter to the toString() method of KnnFloatVectorQuery (#13990)
* Adding filter to toString() of KnnFloatVectorQuery when it's present (addresses https://github.com/apache/lucene/issues/13983)

* addressing review comments

* adding knnbytevectorquery

* unit test improvements

* tidy

* adding changes entry for the bug fix
2024-11-18 14:44:26 -05:00
Adrien Grand 4400d55297
Speed up top-k retrieval on filtered conjunctions. (#13994)
A while back we added an optimized bulk scorer that implements block-max AND,
this yielded a good speedup on nightly benchmarks, see annotation `FP` at
https://benchmarks.mikemccandless.com/AndHighHigh.html. With this PR, filtered
conjunctions now also run through this optimized bulk scorer by doing two
things:
 - It flattens inner conjunctions. This makes queries initially written as
   something like `+(+term1 +term2) #filter` rewritten to
   `+term1 +term2 #filter`.
 - It evaluates queries that have a mix of MUST and FILTER clauses evaluated
   through `BlockMaxConjunctionBulkScorer` by treating FILTER clauses as
   scoring clauses that produce a score of 0.
2024-11-18 08:51:35 +01:00
Viswanath Kuchibhotla a0e1eeefeb
Suppressing "automatic module" warning for benchmark-jmh (#13997)
via Viswanath Kuchibhotla
2024-11-17 07:41:27 -05:00
Lu Xugang 300862bd20
Remove duplicate test code (#13982)
The only diff between doIterate1  and doIterate2  is bb < b.length() in doIterate1 , but this condition is always true, so we should remove it
2024-11-16 02:09:41 +08:00
Chris Hegarty 8698dd85d8
Allow easier verification of the Panama Vectorization provider with newer Java versions (#13986)
This commit allows easier verification of the Panama Vectorization provider with newer Java versions.

The upper bound Java version of the Vectorization provider is hardcoded to the version that has been tested and is known to work. This is a bit inflexible when experimenting with and verifying newer JDK versions. This change proposes to add a new system property that allows to set the upper bound of the range of Java versions supported.

With this change, and the accompanying small gradle change, then one can verify newer JDKs as follows:

CI=true; RUNTIME_JAVA_HOME=/Users/chegar/binaries/jdk-24.jdk-ea-b23/Contents/Home
./gradlew :lucene:core:test -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24

This change helps both testing and verifying with Early Access JDK builds, as well as allowing to override the upper bound when the JDK is known to work fine.
2024-11-14 09:02:15 +00:00
Adrien Grand 6fe8165cac
Simplify codec setup in vector-related tests. (#13970)
Many of vector-related tests set up a codec manually by extending the current
codec. This makes bumping the current codec a bit painful as all these files
need to be touched. This commit migrates to `TestUtil#alwaysKnnVectorsFormat`,
similarly to what we do for postings and doc values.
2024-11-12 10:38:54 +01:00
Ignacio Vera 65457224fb
Tessellator: Improve logic when two holes share the same vertex with the polygon (#13980)
This commit tries to improve the algorithm by:

1.- If there is only one vertex, then there is not further checks and that's the one used.

2.- if there is a common vertex, it first compute the signed area of the join, if they have different sign, 
it chooses the negative one as that's the convex union. If they have the same sign, it computes the angle 
of the join and chooses the smallest angle.
2024-11-11 13:28:09 +01:00
Adrien Grand 12ca4779b9
Fix TestPostingsUtil#testIntegerOverflow failure. (#13979)
The group vint logic is mistakenly using the long->int conversion logic for the
case when integers are being written rather than longs.

Closes #13978
2024-11-06 11:34:27 +01:00
Armin Braun a888af76b2
Simplify leaf slice calculation (#13893)
No need to go through the indirection of 2 wrapped functions, just put the logic in plain
methods. Also, we can just outright set the field if there's no executor.
2024-11-06 10:36:06 +01:00
Adrien Grand 539cf3c9a3 Adjust expectations of TestTopFieldCollectorEarlyTermination after #13943. 2024-11-05 23:52:56 +01:00
Adrien Grand bc18828fcc
Remove `supportsConcurrency` flag from top-docs collector managers. (#13977)
Our collector managers have a `supportsConcurrency` flag to optimize the case
when they are used in a single thread. This PR proposes to remove this flag now
that the optimization doesn't do much as a result of #13943.
2024-11-05 15:36:40 +01:00
Ignacio Vera 07955ff88d Avoid performance regression by constructing lazily the PointTree in NumericComparator (#13498) (#13877) 2024-11-05 12:07:41 +01:00
Luca Cavanna e37eaea11a
Revert "Disjunction as CompetitiveIterator for numeric dynamic pruning (#13221)" (#13857) (#13971)
This reverts commit 1ee4f8a111.

We have observed performance regressions that can be linked to #13221. We will need to revise the logic that such change introduced in main and branch_10x. While we do so, I propose that we bake it out of branch_10_0 and we release Lucene 10 without it.

Closes #13856
2024-11-05 12:00:45 +01:00
Greg Miller a3d56ea2ee
A few small tweaks to VectorUtil#findNextGEQ: (#13972)
1. Rearrange/rename the parameters to be more idiomatic (e.g., follow conventions of Arrays#... methods)
2. Add assert to ensure expected sortedness we may rely on in the future (so we're not trappy)
3. Migrate PostingsReader to call VectorUtil instead of VectorUtilSupport (so it benefits from the common assert)
2024-11-04 16:09:27 -08:00
Dzung Bui 13285279c2
Rename NodeHash to FSTSuffixNodeCache (#13259) 2024-11-01 18:14:17 -04:00
Adrien Grand cfdd20f5bc
Move postings back to int[] to take advantage of having more lanes per vector. (#13968)
In Lucene 8.4, we updated postings to work on long[] arrays internally. This
allowed us to workaround the lack of explicit vectorization (auto-vectorization
doesn't detect all the scenarios that we would like to handle) support in the
JVM by summing up two integers in one operation for instance.

With explicit vectorization now available, it looks like we can get more
benefits from the ability to compare multiple intetgers in one operations than
from summing up two integers in one operation. Moving back to ints helps
compare 2x more integers at once vs. longs.
2024-11-01 07:49:09 +01:00
panguixin 494b16063e
Replace Map<String,Object> with IntObjectHashMap for KnnVectorsReader (#13763) 2024-10-31 16:16:09 +01:00
Mayya Sharipova 26e0737e40
Account for 0 graph size when initializing HNSW graph (#13964)
When initializing a joint graph from one of the segments' graphs,
 we always assume that a segment's graph is present. But later we want
 to explore an option where some segments will not have graphs (#13447).

This change allows to account for missing graphs.
2024-10-31 10:22:30 -04:00
Adrien Grand 3041af7a94
Speed up advancing within a block, take 2. (#13958)
PR #13692 tried to speed up advancing by using branchless binary search, but while this yielded a speedup on my machine, this yielded a slowdown on nightly benchmarks.

This PR tries a different approach using vectorization. Experimentation suggests that it speeds up queries that advance to the next few doc IDs, such as `AndHighHigh`.
2024-10-30 12:51:36 +01:00
Adrien Grand 9359cfd32f
Speed up Lucene912PostingsReader nextDoc() impls. (#13963)
127 times out of 128, nextDoc() returns the next doc ID in the buffer.
Currently, we check if the current doc is equal to the last doc ID in the block
to know if we need to refill. We can do better by comparing the current index
in the block with the block size, which is a bit more efficient since the
latter is a constant.
2024-10-29 18:16:52 +01:00
panguixin 60ddd08c95
replace Map<String,Object> with IntObjectHashMap for DV producer (#13961) 2024-10-29 16:30:15 +01:00
Adrien Grand 937432acd8
Remove HitsThresholdChecker. (#13943)
`TopScoreDocCollectorManager` has a dependency on `HitsThresholdChecker`, which
is essentially a shared counter that is incremented until it reaches the total
hits threshold, when the scorer can start dynamically pruning hits.

A consequence of this removal is that dynamic pruning may start later, as soon
as:
 - either the current slice collected `totalHitsThreshold` hits,
 - or another slice collected `totalHitsThreshold` hits and the current slice
   collected enough hits (up to 1,024) to check the shared
   `MaxScoreAccumulator`.

So in short, it exchanges a bit more work globally in favor of a bit less
contention. A longer-term goal of mine is to stop specializing our
`CollectorManager`s based on whether they are going to be used concurrently or
not.
2024-10-28 15:50:53 +01:00
Adrien Grand 81ab3b9722 Move CHANGES entry to correct version. 2024-10-26 13:52:55 +02:00
Adrien Grand a3a00f3f58
Remove LeafSimScorer abstraction. (#13957)
`LeafSimScorer` is a specialization of a `SimScorer` for a given segment. It
doesn't add much value, but benchmarks suggest that it adds measurable overhead
to queries sorted by score.
2024-10-26 13:45:54 +02:00
ljak b12ee52999
Ensure stability of clause order for DisjunctionMaxQuery toString (#13944)
Co-authored-by: Laurent <jakubinal@lexum.com>
2024-10-25 18:21:58 +02:00
Adrien Grand 5c080156f5
Disable exchanging minimum scores across slices for exhaustive evaluation. (#13954)
When `totalHitsThreshold` is `Integer.MAX_VALUE`, dynamic pruning is never used
and all hits get evaluated. Thus, the minimum competitive score always stays at
zero, and there is nothing to exchange across slices.
2024-10-25 14:02:04 +02:00
Benjamin Trent 618bfa59a6
Ensure doc order for TestCommonTermsQuery#testMinShouldMatch (#13953) 2024-10-25 07:42:26 -04:00
Shubham Chaudhary bf69c3cf43
Make some BooleanQuery methods public and a new `#add(Collection)` method for BQ builder (#13950) 2024-10-25 13:31:34 +02:00
Adrien Grand 2268242412
Remove some useless code in TopScoreDocCollector. (#13955) 2024-10-25 13:30:49 +02:00
Adrien Grand 8362f609b2
Add MIGRATE entry about the fact that readVLong() may now read up to 10 bytes. (#13956)
This may be of interest for custom `DataInput`/`IndexInput` implementations
that extend `readVLong()`.
2024-10-25 13:30:29 +02:00