Commit Graph

37672 Commits

Author SHA1 Message Date
Houston Putman d197f012ef Use multi-select instead of sort for Dynamic Ranges 2024-10-14 19:10:01 -05:00
Luca Cavanna 282945998d Update year in copyright 2024-10-14 17:59:52 +02:00
Armin Braun d5d73d060c
Dry up EverythingEnum and BlockDocsEnum in Lucene912PostingsReader (#13901)
It's in the title, extracting shared parts across both classes.
Almost exclusively mechanical changes with the exception of the introduction
of an array summing util.
2024-10-14 17:29:20 +02:00
Shubham Chaudhary 286b59a2db
Try using Murmurhash 3 for bloom filters (#12868) 2024-10-14 15:08:18 +02:00
Luca Cavanna e94ef1f315 DOAP changes for release 10.0.0 2024-10-14 14:57:01 +02:00
Prudhvi Godithi ada7eda517
IndexOrDocValuesQuery in query highlighting (#13902)
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
2024-10-14 14:19:24 +02:00
Nhat Nguyen 9d4bba3331
Avoid allocating liveDocs for no soft-deletes (#13895)
This is a continuation of #13588, where we avoided allocating liveDocs 
for segments that have the __soft_deletes field but no values in it.
However, that PR only addressed the reading side. This change fixes the
writing scenario with IndexWriter.

Relates #13588
2024-10-12 18:29:57 -07:00
Armin Braun 0368614d34
Dry up both the two ImpactsEnum Implementation ins Lucene912PostingsReader (#13892)
These two share a lot of code, in particular the impacts implementation is 100% identical.
We can save a lot of code and potentially some cycles for method invocations by
drying things up. The changes are just mechanical field movements with the following exceptions:

1. One of the two implementations was using a bytes ref builder, one a bytes ref for holding the
serialized impacts. The `BytesRef` variant is faster so I used that for both when extracting.
2. Some simple arithmetic simplifications around the levels that should be obvious.
3. Removed the the logic for an index without positions in `BlockImpactsPostingsEnum`, that was dead code,
we only set this thing up if there's positions.
2024-10-12 15:48:58 +02:00
Armin Braun 3ed1d1e96c
Lazy initialize ForDeltaUtil and ForUtil in Lucene912PostingsReader (#13885)
Lazy initialize these fields. They consume/cause a lot of memory/GC because they are
allocated frequently (~7% of all allocations in luceneutil's wikimedia medium run for me).
This does not cause any measurable slowdown as far as runtime is concerned and since these are not
even needed for all instances (in fact they are rarely used in the queries the benchmark explores)
save qutie a few CPU cycles for collecting and allocating them.
2024-10-12 12:42:29 +02:00
Armin Braun 48547f913c
Use RandomAccessInput instead of seeking in Lucene90DocValuesProducer (#13894)
We use manual seeking and reading instead of random access in this class.
That seems just unnecessarily complicated.
2024-10-12 10:31:51 +02:00
panguixin fa77d97c09
Remove broken .toArray from Long/CharObjectHashMap entirely (#13884) 2024-10-11 07:30:34 +02:00
Rene Groeschke 5f0d1fbd0c
Make generated archive files reproducible (#13835)
* Make generated archive files reproducible

This should ensure deterministic archive files and fix issues with changing checksums even
though the codebase has not changed
2024-10-10 11:49:41 +02:00
zhouhui ba72d2285d
Add reopen method in PerThreadPKLookup (#13596)
Co-authored-by: Adrien Grand <jpountz@gmail.com>
2024-10-10 10:30:24 +02:00
Dawid Weiss aaf5bc0907
Remove broken .toArray from IntObjectHashMap entirely (#13876) 2024-10-09 21:48:08 +02:00
Adrien Grand d9707609d3
Fix flakiness issues with TestTieredMergePolicy. (#13881)
The two seeds at #13818 had different root causes:
 - The test allows the number of segments to go above the limit, only if none
   of the merges are legal. But there are multiple reasons why a merge may be
   illegal: because it exceeds the max doc count or because it is too imbalanced.
   However these two things were checked independently, so you could run into
   cases when the test would think that there are legal merges from the doc count
   perspective and from the balance perspective, but all legal merges from the doc
   count perspective are illegal from the balance perspective and vice-versa. The
   test now checks that there are merges that are good wrt these two criteria
   at once.
 - `TieredMergePolicy` allows at least `targetSearchConcurrency` segments in an
   index. There was a bug in `TieredMergePolicy` where this condition is
   applied after "too big" segments have been removed, so it effectively allowed
   more segments than necessary in the index.

Closes #13818
2024-10-09 17:28:43 +02:00
Mayya Sharipova b609be99ec Add changes entry for #13859 2024-10-09 11:22:08 -04:00
Adrien Grand 396b7b1307
Disable CFS in TestDefaultCodecParallelizesIO. (#13875)
`SerialIODirectory` doesn't count reads to files that are open with
`ReadAdvice#RANDOM_PRELOAD` as these files are expected to be loaded in memory.
Unfortunately, we cannot detect such files on compound segments, so this test
now disables compound segments.

Closes #13854
2024-10-09 16:24:02 +02:00
Adrien Grand a103ab8aa7
Reuse Impacts instances across invocations. (#13878)
Our `ImpactsEnum` currently return a new object on every call to
`getImpacts()`, which shows up in nightly profiles.
2024-10-09 14:37:20 +02:00
Dawid Weiss b0762e9ef2 Adds changes entry and migration comments #13820. 2024-10-09 11:48:26 +02:00
Luca Cavanna 111fc6f078 Move changes entry for #13845 to 10.0.0 2024-10-09 11:37:45 +02:00
Armin Braun 25850923db
Reduce allocations in ByteBuffersDataOutput.writeString (#13863)
There's no need to allocate a byte array when serializing to heap
buffers and the string fits the remaining capacity without further bounds checks.
If it doesn't fit we could technically do better than the current
`writeLongString` and avoid one round of copying by chunking the string
but that might not be worth the complexity.
In either case we can calculate the utf8 length up-front.
While this costs extra cycles (in the small case) for iterating the string twice it saves
creating an oftentimes 3x oversized byte array, a `BytesRef`, field
reads from the `BytesRef`, copying from it to the buffer and the associated GC with cleaning it up.
Theory and some quick benchmarking suggests this version is likely faster for any string
length than the existing code.
2024-10-09 09:09:00 +02:00
Armin Braun e4092396d5
Misc cleanups postings codec (#13862)
Removing some obvious dead code, turning some fields into locals that don't need to be fields, making things static and deduplicating duplicate "scratch" field.
2024-10-08 19:01:11 +02:00
Armin Braun a14f9cd2eb
Speedup OrderedIntervalsSource (#13871)
Looking up the same indices out of a list is very costly, also better
keep things in local variables (likely registers) where possible.
2024-10-08 19:00:24 +02:00
Armin Braun 5d5dddd103
Make MaxScoreAccumulator use primitive long instead Object return (#13866)
An object return inside hot code like this is needlessly wasteful.
Escape analysis doesn't catch this one and we end up allocating many GB
of throwaway objects during benchmark runs. We might as well use two
utility methods and accumulate the raw value.
2024-10-08 12:50:16 +02:00
Armin Braun 7757d43f4c
Reduce TaskExecutor overhead (#13861)
The `TaskGroup` class is redundant, the futures list can be a local variable
shared by the tasks (this also removes any need for making it read-only).
2024-10-08 11:04:39 +02:00
Zhang Chao 50bf845757
Reduce long[] array allocation for bitset in readBitSetIterator (#13828) 2024-10-08 10:19:28 +08:00
Mayya Sharipova 970672ceab
Allow open-ended ranges in Intervals range (#13859)
Currently IntervalsSource.range function closed intervals.
This will allow open-ended ranges

Relates to #13562
2024-10-07 13:48:12 -04:00
Armin Braun 0fb7f9f89d
Speedup MaxScoreCache.computeMaxScore (#13865)
This shows up as allocating tens of GB for iterators in the nightly
benchmarks. We should go the zero-allocation route for RandomAccess
lists, which I'd expect 100% of them will be here for a bit of a speedup.
2024-10-07 14:39:22 +02:00
Uwe Schindler 486b3b4e4a Upgrade forbiddenapis to version 3.8 2024-10-07 12:25:26 +02:00
Christine Poerschke dab731175c
PR 13757 follow-up: add missing with-discountOverlaps Similarity constructor variants, CHANGES.txt entries (#13845) 2024-10-04 17:08:35 +01:00
Chris Hegarty a4a6cfb39c
Replace branch_9x references with branch_10x in workflows (#13855)
With the move of the main branch to 11, and the upcoming sunsetting of 9x, we need to replace branch_9x references with branch_10x in workflows.
2024-10-04 13:51:45 +01:00
David Smiley f4fcaa6423
Move CHANGES.txt entry 2024-10-03 08:46:44 -04:00
Jim Ferenczi 82b08045fd
Prevent flattening of ordered and unordered interval sources (#13819)
This commit removes the flattening of ordered and unordered interval sources, as it alters the gap visibility for parent intervals. For example, ordered("a", ordered("b", "c")) should result in a different gap compared to ordered("a", "b", "c").

Phrase/Block operators will continue to flatten their sub-sources since this does not affect the inner gap (which is always 0 in the case of blocks).
2024-10-03 12:21:43 +01:00
정승한(Seunghan Jung)/Search Platform e3e3328a55
Fix UnifiedHighlighter DefaultPassageFormatter for non-offset order passages (#13832)
The ellipsis should have been inserted in more scenarios.
2024-10-03 02:00:09 -04:00
Luca Cavanna eaa6214daf
Adjust command to remove uploaded artifacts upon respin (#13853)
The command to remove uploaded artifacts from svn is missing a dash, hence it
fails as it does not match the name of the artifacts uploaded at the previous steps.
2024-10-03 00:18:54 +02:00
Luca Cavanna 2b45e3d8af
Reassign the knn values iterator in TestSortingCodecReader (#13852) 2024-10-02 23:43:28 +02:00
Luca Cavanna 1a746ecaac Add missing authors in Lucene 10 changelog 2024-10-02 20:46:10 +02:00
Benjamin Trent bf74233912
Expose more parameters in MultiLeafKnnCollector ctor (#13826)
There is currently no way to configure two parameters for the multi-leaf collector. For expert extensibility, this commit adds another ctor for advance usage:

closes: #13699
2024-10-02 13:58:16 -04:00
Michael Sokolov 3ede9de6aa
search on proper level when connecting graph components (#13846) 2024-10-02 12:47:49 -04:00
Michael Sokolov 3f6e91e87b
add infostream status messages to HnswGraphBuilder.connectComponents (#13849) 2024-10-02 11:27:48 -04:00
Chris Hegarty 4a1653c220
Add test for float vector values in FlatVectorsScorer impls (#13851)
This is a test only change that verifies the behaviour when float vector values are passed to our FlatVectorsScorer implementations. This would have caught the bug causing #13844, subsequently fixed by #13850.
2024-10-02 16:05:11 +01:00
Benjamin Trent 56e9468a6d
Fix bug where off-heap scorer would kick on even for float vectors (#13850)
introduced in the major refactor #13779

Off-heap scoring is only present for byte[] vectors, and it isn't enough to verify that the vector provider also satisfies the HasIndexSlice interface. The vectors need to be byte vectors otherwise, the slice iterations and scoring are completely nonsensical leading to HNSW graph building to run until the heat-death of the universe.
2024-10-02 09:27:41 -04:00
Chris Hegarty 19da754ee9
Lucene:11 Fix backward-compatibility tests and remove 9.x versions (#13848) 2024-10-02 09:57:12 +01:00
Luca Cavanna bb2d09f32e
Release wizard to split clean and check calls to separate calls (#13843)
While preparing Lucene 10 RC1, I had an issue running the release script from branch_10_0. It reproduces on branch_10x as well. The ./gradle clean check command fails with the following gradle error and some huge tasks dependency output:

Unable to make progress running work. There are items queued for execution but none of them can be started

I worked around this by splitting the clean and check into two separate calls, in which case everything works fine.am making this change at least until we have figured out what causes the issue and we have a fix.
2024-10-01 18:45:54 +02:00
Benjamin Trent 0c7163fc77 Fix simple text byte vector iteration (#13842) 2024-10-01 09:40:27 -04:00
Chris Hegarty cce7d36920 Override iterator in Empty off heap vector values (#13837)
This commit override the iterator method in the empty off-heap vector values. The implementation is just the dense iterator, which handles empty values just fine. We use it elsewhere for similar too.
2024-10-01 10:33:55 +01:00
Adrien Grand 4105c6e686
Bump the codec version to 10.0. (#13815)
Bump the codec version to 10.0.

Lucene100Codec is the exact same file format as Lucene912Codec. This codec
dance just makes things slightly easier to reason about since our backward
compatibility guarantees are aligned with major version: once we drop support
for 9.x indices, we can remove all `Lucene9XXCodec`s.
2024-10-01 08:56:21 +01:00
Armin Braun 15168ce5c1
Speedup GlobalHitsThresholdChecker a little (#13836)
Even though this field is not `volatile`, writing it isn't free and
causes needless cache thrashing at some frequency. We can speed things
up by only writing the `true` value and never the `false` value.
2024-10-01 04:13:36 +02:00
Uwe Schindler 94d3504359 Add changes entries for CVE-2024-45772 and related commits 2024-09-30 17:26:09 +02:00
ChrisHegarty 9e90fb5bf1 Fix backward compat in TestCustomAnalyzer 2024-09-30 14:54:02 +01:00