Commit Graph

37662 Commits

Author SHA1 Message Date
easyice 2c65a616a8 Reduce allocations in BKDReaderDocIDSetIterator 2024-10-11 14:40:08 +08:00
Rene Groeschke 5f0d1fbd0c
Make generated archive files reproducible (#13835)
* Make generated archive files reproducible

This should ensure deterministic archive files and fix issues with changing checksums even
though the codebase has not changed
2024-10-10 11:49:41 +02:00
zhouhui ba72d2285d
Add reopen method in PerThreadPKLookup (#13596)
Co-authored-by: Adrien Grand <jpountz@gmail.com>
2024-10-10 10:30:24 +02:00
Dawid Weiss aaf5bc0907
Remove broken .toArray from IntObjectHashMap entirely (#13876) 2024-10-09 21:48:08 +02:00
Adrien Grand d9707609d3
Fix flakiness issues with TestTieredMergePolicy. (#13881)
The two seeds at #13818 had different root causes:
 - The test allows the number of segments to go above the limit, only if none
   of the merges are legal. But there are multiple reasons why a merge may be
   illegal: because it exceeds the max doc count or because it is too imbalanced.
   However these two things were checked independently, so you could run into
   cases when the test would think that there are legal merges from the doc count
   perspective and from the balance perspective, but all legal merges from the doc
   count perspective are illegal from the balance perspective and vice-versa. The
   test now checks that there are merges that are good wrt these two criteria
   at once.
 - `TieredMergePolicy` allows at least `targetSearchConcurrency` segments in an
   index. There was a bug in `TieredMergePolicy` where this condition is
   applied after "too big" segments have been removed, so it effectively allowed
   more segments than necessary in the index.

Closes #13818
2024-10-09 17:28:43 +02:00
Mayya Sharipova b609be99ec Add changes entry for #13859 2024-10-09 11:22:08 -04:00
Adrien Grand 396b7b1307
Disable CFS in TestDefaultCodecParallelizesIO. (#13875)
`SerialIODirectory` doesn't count reads to files that are open with
`ReadAdvice#RANDOM_PRELOAD` as these files are expected to be loaded in memory.
Unfortunately, we cannot detect such files on compound segments, so this test
now disables compound segments.

Closes #13854
2024-10-09 16:24:02 +02:00
Adrien Grand a103ab8aa7
Reuse Impacts instances across invocations. (#13878)
Our `ImpactsEnum` currently return a new object on every call to
`getImpacts()`, which shows up in nightly profiles.
2024-10-09 14:37:20 +02:00
Dawid Weiss b0762e9ef2 Adds changes entry and migration comments #13820. 2024-10-09 11:48:26 +02:00
Luca Cavanna 111fc6f078 Move changes entry for #13845 to 10.0.0 2024-10-09 11:37:45 +02:00
Armin Braun 25850923db
Reduce allocations in ByteBuffersDataOutput.writeString (#13863)
There's no need to allocate a byte array when serializing to heap
buffers and the string fits the remaining capacity without further bounds checks.
If it doesn't fit we could technically do better than the current
`writeLongString` and avoid one round of copying by chunking the string
but that might not be worth the complexity.
In either case we can calculate the utf8 length up-front.
While this costs extra cycles (in the small case) for iterating the string twice it saves
creating an oftentimes 3x oversized byte array, a `BytesRef`, field
reads from the `BytesRef`, copying from it to the buffer and the associated GC with cleaning it up.
Theory and some quick benchmarking suggests this version is likely faster for any string
length than the existing code.
2024-10-09 09:09:00 +02:00
Armin Braun e4092396d5
Misc cleanups postings codec (#13862)
Removing some obvious dead code, turning some fields into locals that don't need to be fields, making things static and deduplicating duplicate "scratch" field.
2024-10-08 19:01:11 +02:00
Armin Braun a14f9cd2eb
Speedup OrderedIntervalsSource (#13871)
Looking up the same indices out of a list is very costly, also better
keep things in local variables (likely registers) where possible.
2024-10-08 19:00:24 +02:00
Armin Braun 5d5dddd103
Make MaxScoreAccumulator use primitive long instead Object return (#13866)
An object return inside hot code like this is needlessly wasteful.
Escape analysis doesn't catch this one and we end up allocating many GB
of throwaway objects during benchmark runs. We might as well use two
utility methods and accumulate the raw value.
2024-10-08 12:50:16 +02:00
Armin Braun 7757d43f4c
Reduce TaskExecutor overhead (#13861)
The `TaskGroup` class is redundant, the futures list can be a local variable
shared by the tasks (this also removes any need for making it read-only).
2024-10-08 11:04:39 +02:00
Zhang Chao 50bf845757
Reduce long[] array allocation for bitset in readBitSetIterator (#13828) 2024-10-08 10:19:28 +08:00
Mayya Sharipova 970672ceab
Allow open-ended ranges in Intervals range (#13859)
Currently IntervalsSource.range function closed intervals.
This will allow open-ended ranges

Relates to #13562
2024-10-07 13:48:12 -04:00
Armin Braun 0fb7f9f89d
Speedup MaxScoreCache.computeMaxScore (#13865)
This shows up as allocating tens of GB for iterators in the nightly
benchmarks. We should go the zero-allocation route for RandomAccess
lists, which I'd expect 100% of them will be here for a bit of a speedup.
2024-10-07 14:39:22 +02:00
Uwe Schindler 486b3b4e4a Upgrade forbiddenapis to version 3.8 2024-10-07 12:25:26 +02:00
Christine Poerschke dab731175c
PR 13757 follow-up: add missing with-discountOverlaps Similarity constructor variants, CHANGES.txt entries (#13845) 2024-10-04 17:08:35 +01:00
Chris Hegarty a4a6cfb39c
Replace branch_9x references with branch_10x in workflows (#13855)
With the move of the main branch to 11, and the upcoming sunsetting of 9x, we need to replace branch_9x references with branch_10x in workflows.
2024-10-04 13:51:45 +01:00
David Smiley f4fcaa6423
Move CHANGES.txt entry 2024-10-03 08:46:44 -04:00
Jim Ferenczi 82b08045fd
Prevent flattening of ordered and unordered interval sources (#13819)
This commit removes the flattening of ordered and unordered interval sources, as it alters the gap visibility for parent intervals. For example, ordered("a", ordered("b", "c")) should result in a different gap compared to ordered("a", "b", "c").

Phrase/Block operators will continue to flatten their sub-sources since this does not affect the inner gap (which is always 0 in the case of blocks).
2024-10-03 12:21:43 +01:00
정승한(Seunghan Jung)/Search Platform e3e3328a55
Fix UnifiedHighlighter DefaultPassageFormatter for non-offset order passages (#13832)
The ellipsis should have been inserted in more scenarios.
2024-10-03 02:00:09 -04:00
Luca Cavanna eaa6214daf
Adjust command to remove uploaded artifacts upon respin (#13853)
The command to remove uploaded artifacts from svn is missing a dash, hence it
fails as it does not match the name of the artifacts uploaded at the previous steps.
2024-10-03 00:18:54 +02:00
Luca Cavanna 2b45e3d8af
Reassign the knn values iterator in TestSortingCodecReader (#13852) 2024-10-02 23:43:28 +02:00
Luca Cavanna 1a746ecaac Add missing authors in Lucene 10 changelog 2024-10-02 20:46:10 +02:00
Benjamin Trent bf74233912
Expose more parameters in MultiLeafKnnCollector ctor (#13826)
There is currently no way to configure two parameters for the multi-leaf collector. For expert extensibility, this commit adds another ctor for advance usage:

closes: #13699
2024-10-02 13:58:16 -04:00
Michael Sokolov 3ede9de6aa
search on proper level when connecting graph components (#13846) 2024-10-02 12:47:49 -04:00
Michael Sokolov 3f6e91e87b
add infostream status messages to HnswGraphBuilder.connectComponents (#13849) 2024-10-02 11:27:48 -04:00
Chris Hegarty 4a1653c220
Add test for float vector values in FlatVectorsScorer impls (#13851)
This is a test only change that verifies the behaviour when float vector values are passed to our FlatVectorsScorer implementations. This would have caught the bug causing #13844, subsequently fixed by #13850.
2024-10-02 16:05:11 +01:00
Benjamin Trent 56e9468a6d
Fix bug where off-heap scorer would kick on even for float vectors (#13850)
introduced in the major refactor #13779

Off-heap scoring is only present for byte[] vectors, and it isn't enough to verify that the vector provider also satisfies the HasIndexSlice interface. The vectors need to be byte vectors otherwise, the slice iterations and scoring are completely nonsensical leading to HNSW graph building to run until the heat-death of the universe.
2024-10-02 09:27:41 -04:00
Chris Hegarty 19da754ee9
Lucene:11 Fix backward-compatibility tests and remove 9.x versions (#13848) 2024-10-02 09:57:12 +01:00
Luca Cavanna bb2d09f32e
Release wizard to split clean and check calls to separate calls (#13843)
While preparing Lucene 10 RC1, I had an issue running the release script from branch_10_0. It reproduces on branch_10x as well. The ./gradle clean check command fails with the following gradle error and some huge tasks dependency output:

Unable to make progress running work. There are items queued for execution but none of them can be started

I worked around this by splitting the clean and check into two separate calls, in which case everything works fine.am making this change at least until we have figured out what causes the issue and we have a fix.
2024-10-01 18:45:54 +02:00
Benjamin Trent 0c7163fc77 Fix simple text byte vector iteration (#13842) 2024-10-01 09:40:27 -04:00
Chris Hegarty cce7d36920 Override iterator in Empty off heap vector values (#13837)
This commit override the iterator method in the empty off-heap vector values. The implementation is just the dense iterator, which handles empty values just fine. We use it elsewhere for similar too.
2024-10-01 10:33:55 +01:00
Adrien Grand 4105c6e686
Bump the codec version to 10.0. (#13815)
Bump the codec version to 10.0.

Lucene100Codec is the exact same file format as Lucene912Codec. This codec
dance just makes things slightly easier to reason about since our backward
compatibility guarantees are aligned with major version: once we drop support
for 9.x indices, we can remove all `Lucene9XXCodec`s.
2024-10-01 08:56:21 +01:00
Armin Braun 15168ce5c1
Speedup GlobalHitsThresholdChecker a little (#13836)
Even though this field is not `volatile`, writing it isn't free and
causes needless cache thrashing at some frequency. We can speed things
up by only writing the `true` value and never the `false` value.
2024-10-01 04:13:36 +02:00
Uwe Schindler 94d3504359 Add changes entries for CVE-2024-45772 and related commits 2024-09-30 17:26:09 +02:00
ChrisHegarty 9e90fb5bf1 Fix backward compat in TestCustomAnalyzer 2024-09-30 14:54:02 +01:00
ChrisHegarty cf8c3c4f02 Fix backward compat tests in core 2024-09-30 14:52:47 +01:00
Michael Sokolov f2b2bfc414
CHANGES/MIGRATE entries for KnnVectorValues API change (#13833) 2024-09-30 07:04:37 -04:00
ChrisHegarty 4e1c8d20ce Add next minor version 10.1.0 2024-09-30 11:38:17 +01:00
ChrisHegarty 5ad0451666 Add next major version 11.0.0 2024-09-30 11:31:56 +01:00
Zhang Chao 8433352eec
Move DataInput.readGroupVInts into GroupVIntUtil (#13830) 2024-09-30 17:04:18 +08:00
ChrisHegarty 22ac47c07a Fix bwc tests by adding next minor version - even though there will never be a 9.13.0 2024-09-29 20:27:06 +01:00
zhouhui 7eea8fbd0d
Fix comment on compare suffix and target. (#13787) 2024-09-29 11:12:32 -04:00
Michael Sokolov 0a8604d908 Merge remote-tracking branch 'origin/main' into main 2024-09-28 17:45:05 -04:00
Michael Sokolov c8eeacb39a fix safety check in KnnVectorsWriter 2024-09-28 17:44:33 -04:00
Michael Sokolov f21ec8414c SlowCompositeCodecReaderWrapper KnnVectorValues handles binary search over array with repeated values 2024-09-28 17:43:01 -04:00