Commit Graph

37388 Commits

Author SHA1 Message Date
Chris Hegarty 80304802a5
Fix duplicate values in org.apache.lucene.analysis.ko.dict.UserDictionary (#13427)
Remove incorrect assertion in org.apache.lucene.analysis.ko.dict.UserDictionary, and replace with array copy if duplicate values are passed.
2024-05-27 17:53:01 +01:00
Chris Hegarty 944624b644
MemorySegment scorer should ensure that the values is of the correct type (#13423)
This commit updates the MemorySegment scorer so that it ensures the values is of the correct type.

The offset calculations for vectors in RandomAccessQuantizedByteVectorValues will be different than that of non-quantized. We can generalise the implementation for quantized vectors later, but for now, passing a quantised values indicated bug in wrapping or delegation.
2024-05-27 17:19:16 +01:00
Bruno Roustant f394c9418e
Remove the HPPC dependency from all modules and move the HPPC fork to internal. (#13422)
* Remove hppc dependency
* Change fork version to 0.10.0
* Add @lucene.internal
* Move hppc classes to oal.internal.hppc but export it.
* Delete hppc license since it's no longer a dependency.

---------

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
2024-05-27 12:09:25 +02:00
Adrien Grand ddf538d43e
Move bulkScorer() from Weight to ScorerSupplier (#13408)
This relates to #13359: we want to take advantage of the `Weight#scorerSupplier` call to start scheduling some I/O in the background in parallel across clauses. For this to work properly with top-level disjunctions, we need to move `#bulkScorer()` from `Weight` to `ScorerSupplier` as well, so that the disjunctive `BooleanQuery` first performs a call to `Weight#scorerSupplier()` on all inner clauses, and then `ScorerSupplier#bulkScorer` on all inner clauses.

`ScorerSupplier#get` and `ScorerSupplier#bulkScorer` only support being called once. This forced me to fix some inefficiencies in `bulkScorer()` implementations when we would pull scorers and then throw it away when realizing that the strategy we were planning on using was not optimal. This is why e.g. `ReqExclBulkScorer` now also supports prohibited clauses that produce a two-phase iterator.
2024-05-27 09:56:26 +02:00
Bruno Roustant 6e2a8fc9b7
Replace Map<Character> by CharObjectHashMap and Set<Character> by CharHashSet. (#13420)
Also optimize the character replacement in JapaneseKatakanaUppercaseFilter.
2024-05-27 08:44:23 +02:00
Bruno Roustant 444d4e7c42
Replace List<Integer> by IntArrayList and List<Long> by LongArrayList. (#13406) 2024-05-25 19:53:42 +02:00
David Smiley 90e07f6664
gradlew: no "--source 11" (#13404)
* avoid WrapperDownloader if have the JAR
* don't specify --source
More specific than needed, and some JDK/configs may complain about an incompatibility with --release.

From https://github.com/apache/solr/pull/2419
2024-05-24 18:19:53 -04:00
Christine Poerschke be513cf515 (fix precommit) lucene-monitor: remove unused import in ReadonlyQueryIndex 2024-05-24 18:14:44 +01:00
Christine Poerschke 40f674c5d3
lucene-monitor: replace wildcard imports (#13413) 2024-05-24 17:36:14 +01:00
Christine Poerschke 8dad226afa
lucene-monitor: remove now-unused Scorable in QueryIndex.DataValues (#13412) 2024-05-24 17:35:55 +01:00
Adrien Grand 9ecf566cca
Remove unchecked Scorable -> Scorer cast in lucene/monitor. (#13405)
While doing an unrelated refactoring, I got hit by this unchecked cast, which
is incorrecw when the presearcher query produces some specialized `BulkScorer`.
2024-05-24 09:36:38 +02:00
정승한(Seunghan Jung)/Search Platform 8773725ac0
UnifiedHighlighter: new passageSortComparator option (#13276)
new 'passageSortComparator' option to allow sorting other than offset order
2024-05-23 13:45:17 -04:00
Bruno Roustant d078fb774d
Replace Set<Integer> by IntHashSet and Set<Long> by LongHashSet (#13400)
Also rename lucene.document.LongHashSet to DocValuesLongHashSet.
2024-05-23 16:25:52 +02:00
Chris Hegarty 05f04aa08a
Add a MemorySegment Vector scorer - for scoring without copying on-heap (#13339)
Add a MemorySegment Vector scorer - for scoring without copying on-heap.

The vector scorer loads values directly from the backing memory segment when available. Otherwise, if the vector data spans across segments the scorer copies the vector data on-heap.

A benchmark shows ~2x performance improvement of this scorer over the default copy-on-heap scorer.

The scorer currently only operates on vectors with an element size of byte. We can evaluate if and how to support floats separately.
2024-05-21 17:34:37 +01:00
Bruno Roustant f70999980c
Replace Map<Long, Object> by primitive LongObjectHashMap. (#13392)
Add LongObjectHashMap and replace Map<Long, Object>.
Add LongIntHashMap and replace Map<Long, Int>.
Add HPPC dependency to join and spatial modules for primitive values float and double.
2024-05-21 17:11:34 +02:00
Adrien Grand 5b1a34bddd
Fix max score computation in BlockMaxConjunctionBulkScorer. (#13397)
It sums up max scores in a float when it should sum them up in a double like we
do for `Scorer#score()`. Otherwise, max scores may be returned that are less
than actual scores.

This bug was introduced in #13343, so it is not released yet.

Closes #13371
Closes #13396
2024-05-21 11:51:59 +02:00
Alessandro Benedetti da6bd22fd9 Revert "Vectors Format Refactor first draft"
This reverts commit 77cec44423.
2024-05-21 11:32:26 +02:00
Alessandro Benedetti 77cec44423 Vectors Format Refactor first draft 2024-05-21 11:30:13 +02:00
Alan Woodward 92420d345a
Add Intervals.noIntervals() method (#13389)
Parsers may sometimes want to create an IntervalsSource that returns no
intervals.  This adds a new factory method to `Intervals` that will create one,
and changes `IntervalBuilder` to use it in place of its custom empty intervals
source.
2024-05-21 09:22:44 +01:00
Adrien Grand aac856a831
Reduce the overhead of `IndexInput#prefetch` when data is cached in RAM. (#13381)
As Robert pointed out and benchmarks confirmed, there is some (small) overhead
to calling `madvise` via the foreign function API, benchmarks suggest it is in
the order of 1-2us. This is not much for a single call, but may become
non-negligible across many calls. Until now, we only looked into using
prefetch() for terms, skip data and postings start pointers which are a single
prefetch() operation per segment per term.

But we may want to start using it in cases that could result into more calls to
`madvise`, e.g. if we start using it for stored fields and a user requests 10k
documents. In #13337, Robert wondered if we could take advantage of `mincore()`
to reduce the overhead of `IndexInput#prefetch()`, which is what this PR is
doing via `MemorySegment#isLoaded()`.

`IndexInput#prefetch` tracks consecutive hits on the page cache and calls
`madvise` less and less frequently under the hood as the number of consecutive
cache hits increases.
2024-05-21 09:12:44 +02:00
expani1729 22d50be2ea
Make IntervalBuilder.NO_INTERVALS public (#13385)
This is generally useful for clients building their own Intervals
implementations.
2024-05-20 13:51:51 +01:00
gf2121 1ee4f8a111
Disjunction as CompetitiveIterator for numeric dynamic pruning (#13221)
// nightly-benchmarks-results-changed //
2024-05-20 15:00:09 +08:00
Bruno Roustant 7db9c8c9bd
Replace Map<Integer, Object> by primitive IntObjectHashMap. (#13368) 2024-05-18 14:40:40 +02:00
Tim Grein 24fd426d98
Fix TestHnswByteVectorGraph.testSortedAndUnsortedIndicesReturnSameResults (#13361)
Considering that the graphs of 2 indices are organized differently we need to explore a lot of candidates to ensure that both searchers find the same docs. Increasing beamWidth (number of nearest neighbor candidates to track while searching the graph for each newly inserted node) from 5 to 10 fixes the test.
2024-05-17 17:23:38 +01:00
Zhang Chao 22ddd481f0
Fix IntegerOverflow exception in postings encoding as group-varint (#13376)
The exception happen because the tail postings list block, which encoding with GroupVInt, had a docID delta that was >= 1<<30, when the postings are also storing freqs.
2024-05-17 23:43:23 +08:00
Christine Poerschke 2c81649e28
lucene-monitor: make TermFilteredPresearcher.ANYTOKEN[_FIELD] public (#13379) 2024-05-17 11:26:08 +01:00
Adrien Grand c5331df1c4
Use IndexInput#prefetch for postings, skip data and impacts (#13364)
This uses the `IndexInput#prefetch` API for postings. This relies on heuristics, as we don't know ahead of time what data we will need from a postings list:
 - Postings lists are prefetched entirely when they are short (< 16kB).
 - Impacts enums also prefetch the first page of skip data.
 - Postings enums prefetc skip data on the first call to advance().

Positions, offsets and payloads are never prefetched.

Putting the `IndexInput#prefetch` call in `TermsEnum#postings` and `TermsEnum#impacts` works well because `BooleanQuery` will first create postings/impacts enums for all clauses before it starts unioning/intersecting them. This allows the prefetching logic to run in parallel across all clauses of the same query on the same segment.
2024-05-17 09:07:07 +02:00
Chris Hegarty 3d671a0fbe
Fix bug in SQ when just a single vector present in a segment (#13374)
This commit fixes a corner case in the ScalarQuantizer when just a single vector is present. I ran into this when updating a test that previously passed successfully with Lucene 9.10 but fails in 9.x.

The score error correction is calculated to be NaN, as there are no score docs or variance.
2024-05-16 14:59:56 +01:00
Chris Hegarty 731cecf730
Fix points writing with no values (#13378)
This commit updates the writer to handle the case where there are no values.

Previously (before #13369), there was a check that there were some points values before trying to write, this is no longer the case. The code in writeFieldNDims has an assumption that the values is not empty - an empty values will result in calculating a negative number of splits, and a negate array size to hold the splits.

The fix is trivial, return null when values is empty - null is an allowable return value from this method. Note: writeField1Dim is able to handle an empty values.
2024-05-16 14:24:00 +01:00
Benjamin Trent b1d3c08619
Fix weird NRT bug #13353 (#13369)
The issue outlines the problem. When we have point value dimensions, segment core readers assume that there will be point files.

However, when allowing soft deletes and a document fails indexing failed before a point field could be written, this assumption fails. Consequently, the NRT fails to open. I settled on always flushing a point file if the field info says there are point fields, even if there aren't any docs in the buffer.

closes #13353
2024-05-15 09:37:19 -04:00
Adrien Grand 46f1f95ceb
Make `IndexInput#prefetch` take an offset. (#13363)
This makes `IndexInput#prefetch` take an offset instead of being relative to
the current position. This avoids requiring callers to seek only to call
`prefetch()`.
2024-05-14 18:43:21 +02:00
Sanjay Dutt 838b23ebed
Make Weight#scorerSupplier abstract, Weight#scorer final (#13319)
Co-authored-by: iamsanjay <sanjaydutt.india@yahoo.com>
2024-05-14 17:44:30 +02:00
Tim Grein 8e5409c9b4
Add sub query explanations in DisjunctionMaxQuery#explain on no-match (#13362) 2024-05-14 17:28:14 +02:00
Tim Grein dae7181a97
Align toString methods in geo module (#13302)
* Align toString methods in geo module
2024-05-14 16:31:34 +02:00
panguixin 37837133c3
Remove unnecessary bit conversion for IndexSorter (#13320) 2024-05-14 15:05:51 +02:00
Benjamin Trent 3aa25baac7
Protect against nan & inf values in quantizer and test with tiny vectors (#13366) 2024-05-14 07:30:38 -04:00
Bruno Roustant bcb62f56ad
Call ArrayUtil.copyArray instead of ArrayUtil.copySubArray for full array copy. (#13360) 2024-05-14 11:32:18 +02:00
Benjamin Trent fc12cc1847
Fix vector scorer interface consistency (#13365)
Follow up to: #13181

I noticed the quantized interface had a slightly different name.

Additionally, testing showed we are inconsistent when there aren't any vectors to score. This makes the response consistent (e.g. null when there aren't any vectors).
2024-05-13 13:02:38 -04:00
Benjamin Trent f10748cee7
Ensure negative scores aren not returned from scalar quantization scorer (#13356)
Depending on how we quantize and then scale, we can edge down below 0 for dotproduct scores.

This is exceptionally rare, I have only seen it in extreme circumstances in tests (with random data and low dimensionality).
2024-05-13 11:00:04 -04:00
Bruno Roustant 8c738ba010
Reduce memory usage of field maps in FieldInfos and BlockTree TermsReader. (#13327) 2024-05-13 15:49:41 +02:00
Chris Hegarty 25f1efd8eb
Fix default flat vector scorer supplier sharing backing array (#13355)
This commit fixes an issue in the default flat vector scorer supplier whereby subsequent scorers created by the supplier can affect previously created scorers.

The issue is that we're sharing the backing array from the vector values, and overwriting it in subsequent scorers. We just need to use the ordinal to protect the scorer instance from mutation.
2024-05-10 21:49:13 +01:00
zhongshanhao 701c75a412
Advoid the use of ImpactsDISI when no minimum competitive score has been set (#13343)
Co-authored-by: zhongshanhao <zhongshanhao@bilibili.com>
2024-05-10 18:27:00 +02:00
Adrien Grand 3003731aa9
Add IndexInput#prefetch. (#13337)
This adds `IndexInput#prefetch`, which is an optional operation that instructs
the `IndexInput` to start fetching bytes from storage in the background. These
bytes will be picked up by follow-up calls to the `IndexInput#readXXX` methods.
In the future, this will help Lucene move from a maximum of one I/O operation
per search thread to one I/O operation per search thread per `IndexInput`.
Typically, when running a query on two terms, the I/O into the terms dictionary
is sequential today. In the future, we would ideally do these I/Os in parallel
using this new API. Note that this will require API changes to some classes
including `TermsEnum`.

I settled on this API because it's simple and wouldn't require making all
Lucene APIs asynchronous to take advantage of extra I/O concurrency, which I
worry would make the query evaluation logic too complicated.

This change will require follow-ups to start using this new API when working
with terms dictionaries, postings, etc.

Relates #13179

Co-authored-by: Uwe Schindler <uschindler@apache.org>
2024-05-10 13:42:35 +02:00
boice 5ac88c750e
Performance improvements to use RWLock to access LRUQueryCache (#13306)
Elasticsearch (which based on lucene) can automatically infer types for users with its dynamic mapping feature. When users index some low cardinality fields, such as gender / age / status... they often use some numbers to represent the values, while ES will infer these fields as long, and ES uses BKD as the index of long fields. 

Just as #541 said, when the data volume grows, building the result set of low-cardinality fields will make the CPU usage and load very high even if we use a boolean query with filter clauses for low-cardinality fields. 

One reason is that it uses a ReentrantLock to limit accessing LRUQueryCache. QPS and costs of their queries are often high,  which often causes trying locking failures when obtaining the cache, resulting in low concurrency in accessing the cache.

So I replace the ReentrantLock with a ReentrantReadWriteLock. I only use the read lock when I need to get the cache for a query,
2024-05-10 07:04:26 -04:00
Adrien Grand f3a52113a4
Remove unused "implements Accountable". (#13330)
A number of file formats no longer implement `Accountable`. This allows
removing this interface on a number of internal classes as well.
2024-05-09 23:27:18 +02:00
Benjamin Trent b60e86c4b9
Add new VectorScorer interface to vector value iterators (#13181)
With quantized vectors, and with current vectors, we separate out the "scoring" vs. "iteration", requiring the user to always iterate the raw vectors and provide their own similarity function.

While this is flexible, it creates frustration in:

 - Just iterating and scoring, especially since the field already has a similarity function stored...Why can't we just know which one to use and use it!
 - Iterating and scoring quantized vectors. By default it would be good to be able to iterate and score quantized vectors (e.g. without going through the HNSW graph).

This significantly hampers support for true exact kNN search.

This commit extends the vector value iterators to be able to return a scorer given some vector value (what this PR demonstrates). The scorer contains a copy of the originating iterator and allows for iteration and scoring the most optimized way the provided codec can give. 

Users can still iterate vector values directly, read them on heap, and score any way they please.
2024-05-09 16:30:14 -04:00
Chris Hegarty 8d7e4174af
Add a separate option to allow running Panama Vectorization for all tests with suitable C2 defaults (#13351)
This commit adds a separate option, tests.defaultvectorization, to allow running Panama Vectorization for all tests with suitable C2 defaults.

For example:

./gradlew :lucene:core:test -Ptests.defaultvectorization=true
---------

Co-authored-by: Uwe Schindler <uschindler@apache.org>
2024-05-09 11:00:51 +01:00
Nhat Nguyen 06973f5723
Harden BaseDocValuesFormatTestCase (#13346)
We hit a Codec bug in Elasticsearch, but it went unnoticed because our 
tests extend from BaseDocValuesFormatTestCase, which doesn't attempt to
read the doc-values of the same document twice. This change strengthens
BaseDocValuesFormatTestCase checks and randomly inserts that access
pattern.
2024-05-07 09:04:25 -07:00
Michael Sokolov 30da7dabe0
gh-13340: Allow adding a parent field to an index with no fields (#13341) 2024-05-06 12:53:13 -04:00
Stefan Vodita 40cae087f7 Move lazy init facets from 10.0 to 9.11 2024-05-03 08:18:29 +00:00