Commit Graph

37789 Commits

Author SHA1 Message Date
정승한(Seunghan Jung)/Search Platform e3e3328a55
Fix UnifiedHighlighter DefaultPassageFormatter for non-offset order passages (#13832)
The ellipsis should have been inserted in more scenarios.
2024-10-03 02:00:09 -04:00
Luca Cavanna eaa6214daf
Adjust command to remove uploaded artifacts upon respin (#13853)
The command to remove uploaded artifacts from svn is missing a dash, hence it
fails as it does not match the name of the artifacts uploaded at the previous steps.
2024-10-03 00:18:54 +02:00
Luca Cavanna 2b45e3d8af
Reassign the knn values iterator in TestSortingCodecReader (#13852) 2024-10-02 23:43:28 +02:00
Luca Cavanna 1a746ecaac Add missing authors in Lucene 10 changelog 2024-10-02 20:46:10 +02:00
Benjamin Trent bf74233912
Expose more parameters in MultiLeafKnnCollector ctor (#13826)
There is currently no way to configure two parameters for the multi-leaf collector. For expert extensibility, this commit adds another ctor for advance usage:

closes: #13699
2024-10-02 13:58:16 -04:00
Michael Sokolov 3ede9de6aa
search on proper level when connecting graph components (#13846) 2024-10-02 12:47:49 -04:00
Michael Sokolov 3f6e91e87b
add infostream status messages to HnswGraphBuilder.connectComponents (#13849) 2024-10-02 11:27:48 -04:00
Chris Hegarty 4a1653c220
Add test for float vector values in FlatVectorsScorer impls (#13851)
This is a test only change that verifies the behaviour when float vector values are passed to our FlatVectorsScorer implementations. This would have caught the bug causing #13844, subsequently fixed by #13850.
2024-10-02 16:05:11 +01:00
Benjamin Trent 56e9468a6d
Fix bug where off-heap scorer would kick on even for float vectors (#13850)
introduced in the major refactor #13779

Off-heap scoring is only present for byte[] vectors, and it isn't enough to verify that the vector provider also satisfies the HasIndexSlice interface. The vectors need to be byte vectors otherwise, the slice iterations and scoring are completely nonsensical leading to HNSW graph building to run until the heat-death of the universe.
2024-10-02 09:27:41 -04:00
Chris Hegarty 19da754ee9
Lucene:11 Fix backward-compatibility tests and remove 9.x versions (#13848) 2024-10-02 09:57:12 +01:00
Luca Cavanna bb2d09f32e
Release wizard to split clean and check calls to separate calls (#13843)
While preparing Lucene 10 RC1, I had an issue running the release script from branch_10_0. It reproduces on branch_10x as well. The ./gradle clean check command fails with the following gradle error and some huge tasks dependency output:

Unable to make progress running work. There are items queued for execution but none of them can be started

I worked around this by splitting the clean and check into two separate calls, in which case everything works fine.am making this change at least until we have figured out what causes the issue and we have a fix.
2024-10-01 18:45:54 +02:00
Benjamin Trent 0c7163fc77 Fix simple text byte vector iteration (#13842) 2024-10-01 09:40:27 -04:00
Chris Hegarty cce7d36920 Override iterator in Empty off heap vector values (#13837)
This commit override the iterator method in the empty off-heap vector values. The implementation is just the dense iterator, which handles empty values just fine. We use it elsewhere for similar too.
2024-10-01 10:33:55 +01:00
Adrien Grand 4105c6e686
Bump the codec version to 10.0. (#13815)
Bump the codec version to 10.0.

Lucene100Codec is the exact same file format as Lucene912Codec. This codec
dance just makes things slightly easier to reason about since our backward
compatibility guarantees are aligned with major version: once we drop support
for 9.x indices, we can remove all `Lucene9XXCodec`s.
2024-10-01 08:56:21 +01:00
Armin Braun 15168ce5c1
Speedup GlobalHitsThresholdChecker a little (#13836)
Even though this field is not `volatile`, writing it isn't free and
causes needless cache thrashing at some frequency. We can speed things
up by only writing the `true` value and never the `false` value.
2024-10-01 04:13:36 +02:00
Uwe Schindler 94d3504359 Add changes entries for CVE-2024-45772 and related commits 2024-09-30 17:26:09 +02:00
ChrisHegarty 9e90fb5bf1 Fix backward compat in TestCustomAnalyzer 2024-09-30 14:54:02 +01:00
ChrisHegarty cf8c3c4f02 Fix backward compat tests in core 2024-09-30 14:52:47 +01:00
Michael Sokolov f2b2bfc414
CHANGES/MIGRATE entries for KnnVectorValues API change (#13833) 2024-09-30 07:04:37 -04:00
ChrisHegarty 4e1c8d20ce Add next minor version 10.1.0 2024-09-30 11:38:17 +01:00
ChrisHegarty 5ad0451666 Add next major version 11.0.0 2024-09-30 11:31:56 +01:00
Zhang Chao 8433352eec
Move DataInput.readGroupVInts into GroupVIntUtil (#13830) 2024-09-30 17:04:18 +08:00
ChrisHegarty 22ac47c07a Fix bwc tests by adding next minor version - even though there will never be a 9.13.0 2024-09-29 20:27:06 +01:00
zhouhui 7eea8fbd0d
Fix comment on compare suffix and target. (#13787) 2024-09-29 11:12:32 -04:00
Michael Sokolov 0a8604d908 Merge remote-tracking branch 'origin/main' into main 2024-09-28 17:45:05 -04:00
Michael Sokolov c8eeacb39a fix safety check in KnnVectorsWriter 2024-09-28 17:44:33 -04:00
Michael Sokolov f21ec8414c SlowCompositeCodecReaderWrapper KnnVectorValues handles binary search over array with repeated values 2024-09-28 17:43:01 -04:00
ChrisHegarty a38443bc6a Add back-compat indices for 9.12.0 2024-09-28 21:47:03 +01:00
ChrisHegarty dafe006652 DOAP changes for release 9.12.0 2024-09-28 21:14:49 +01:00
Michael Sokolov 6053e1e313
First-class random access API for KnnVectorValues (#13779) 2024-09-28 09:14:01 -04:00
ChrisHegarty 7b4b0238d7 Fix back compat 11.8.4 indices in main 2024-09-26 14:54:17 +01:00
ChrisHegarty ff57fa7b42 Add RDF entries for 8.11.4 2024-09-25 16:35:24 +01:00
Chris Hegarty 64371c14d6
Add 8.11.4 backward compat indices (#13824)
This commit add 8.11.4 back compat indices.
2024-09-25 16:20:54 +01:00
ChrisHegarty 53d1c2bd2f Test fix: make float vector dims even for SQ testing 2024-09-20 15:28:54 +01:00
Adrien Grand 73d71acedd Also fix formats that are only tested nightly. 2024-09-20 15:24:04 +02:00
Adrien Grand da1f954601
Improve testing of mismatched field numbers. (#13812)
This improves testing of mismatched field numbers by
 - improving `AssertingDocValuesProducer` to detect mismatched field numbers,
 - introducing a `MismatchedCodecReader` to actually test mismatched field
   numbers on `DocValuesProducer` (a `MismatchedLeafReader` wrapping a
`SlowCodecReaderWrapper` doesn't work since `SlowCodecReaderWrapper` implicitly
resolves the correct `FieldInfo` object),
 - introducing an explicit test for mismatched field numbers for doc values, points,
postings and knn vectors.

These new tests uncovered a bug when merging sorted doc values, which would
call the underlying doc values producer with the merged field info.

Closes #13805
2024-09-20 14:37:45 +02:00
Chris Hegarty 7ef7122eba
Revert "Replace Map<String,Object> with IntObjectHashMap for DV producer (#13686) (#13810)
Reverts "Replace Map<String,Object> with IntObjectHashMap for DV producer (#13686)

relates #13809
2024-09-20 11:06:43 +01:00
Christoph Büscher e4ac57746e
Add BytesRefIterator to TermInSetQuery (#13806)
TermInSetQuery used to have an accessor to its terms that was removed in #12173
to protect leaking internal encoding details. This introduces an accessor to the
term data in the query that doesn't expose internals but merely allows iterating
over the decoded BytesRef, making inspection of the querys content possible again.

Closes #13804
2024-09-19 11:51:42 +02:00
Benjamin Trent 6d987e1ce1
Disable intra-merge parallelism for all structures but kNN vectors (#13799)
After adjusting tests that truly exercise intra-merge parallelism, more issues have arisen. See: https://github.com/apache/lucene/issues/13798

To be risk adverse & due to the soon to be released/freezed Lucene 10 & 9.12, I am reverting all intra-merge parallelism, except for the parallelism when merging HNSW graphs.

Merging other structures was never really enabled in a release (we disabled it in a bugfix for Lucene 9.11). While this is frustrating as it seems like we leaving lots of perf on the floor, I am err'ing on the side of safety here. 

In Lucene 10, we can work on incrementally reenabling intra-merge parallelism.

closes: https://github.com/apache/lucene/issues/13798
2024-09-18 08:36:11 -04:00
Alan Woodward dbceba77a4
Correct Point file extensions in Codec javadocs (#13801) 2024-09-18 12:33:31 +01:00
Adrien Grand b59a357e58
Change docValuesSkipIndex from a boolean to an enum. (#13784)
At the moment, our skip indexes record min/max ordinal/value per range
of doc IDs. It would be natural to extend it to other pre-aggregated
data such as a sum and value count, which facets could take advantage
of. This change switches `docValuesSkipIndex` from a boolean to an enum
so that we could release such changes in the future in an additive
fashion, by adding constants to this enum and new methods to
`DocValuesSkipper`.
2024-09-17 14:35:30 +02:00
Armin Braun 644feeb02a
Cleanup redundant allocations and code around Comparator use (#13795)
Noticed some visible allocations in CompetitiveImpactAccumulator
during benchmarking and fixed the needless allocation for the comparator
in that class as well as a couple other similar spots where needless
classes and/or objects could easily be replaced by more lightweight
solutions.
2024-09-17 14:34:31 +02:00
Christine Poerschke a817426511
add RawTFSimilarity class (#13749) 2024-09-17 13:11:25 +01:00
Luca Cavanna a4c79c8d30
Remove CollectorManager#forSequentialExecution (#13790)
We have recently (see #13735) introduced this utility method that creates a
collector manager which only works when a searcher does not have an executor
set, otherwise it throws exception once we attempt to create a new collector
for more than one slice.

While we discussed it should be safe to use in some specific scenarios like the
monitor module, we should be careful exposing this utility publicly, because
while we'd like to ease migration from the search(Query, Collector) method, we
may end up making users like even worse, in that it exposes them to failures
whenever an executor is set and there are more than one slice created, which
is hard to follow and does not provide a good user experience.

My proposal is that we use a similar collector manager locally, where safe and
required, but we don't expose it to users. In most places, we should rather
expose collector managers that do support search concurrency, rather than working
around the lack of those.
2024-09-17 09:41:10 +02:00
Adrien Grand f4ebed2404
Make Automata#optional create simpler automata. (#13793)
In the common case when the input automaton has no transition to state 0, the
optional automaton can be created by marking state 0 as accepted.
2024-09-16 15:19:28 +02:00
Mike Pellegrini e7a6382089
Fix Flaky Test In TestBlockJoinBulkScorer (#13785) 2024-09-16 14:50:51 +02:00
Adrien Grand 73173a40c1 Revert "Speed up advancing within a block. (#13692)"
This reverts commit 79fd9fee97.
2024-09-16 10:39:21 +02:00
Dawid Weiss ebeb5c3281 Add --stacktrace to gradlew invocations for debugging odd issues #13792 2024-09-16 08:32:05 +02:00
Mike McCandless 3801d85978 #13767: fix test failure: compress can only be true when quantized bits=4 2024-09-15 16:11:00 -04:00
Mike McCandless a7332fb30a GITHUB#13519: move CHANGES entry to 9.12.0 2024-09-15 15:46:16 -04:00