Commit Graph

36729 Commits

Author SHA1 Message Date
Robert Muir 650a47e707
Merge branch 'main' into vector_512: stomping over any possible conflicting changes: if you made any, i erased them. git is garbage. 2023-10-15 08:20:43 -04:00
Robert Muir db746bdbc3
simple cleanups to vector code (#12680)
* simple cleanups to vector code

* cleanup cosine too, no perf impact
2023-10-15 08:12:14 -04:00
Uwe Schindler dbda33fb2f
Better detect vector module in non-default setups (e.g., custom module layers) (#12677) 2023-10-15 12:45:52 +02:00
Robert Muir a69cdeff56
fix html 2023-10-14 19:56:42 -04:00
Robert Muir 90c01c1e80
tidy 2023-10-14 19:54:47 -04:00
Robert Muir 94619e00e0
fix Panama provider to only have static final constants, but allow sysprops for testing.
Set those sysprops by the build when running tests: force integer vectors on always, and randomize vector size
2023-10-14 19:51:59 -04:00
Robert Muir cce3cf106a
stop using SPECIES_PREFERRED except at the top of this file 2023-10-14 18:08:33 -04:00
Robert Muir 3ec9c26d67
speedup all binary functions on avx256, speedup binary square on avx512 2023-10-14 16:28:26 -04:00
Robert Muir e93bd524cf
cleanup cosine too, no perf impact 2023-10-14 14:10:32 -04:00
Patrick Zhai bac4dd28e0
Fix unstable test TestVectorSimilarityValuesSource (#12678) 2023-10-14 11:04:29 -07:00
Robert Muir a72bf7ce68
simple cleanups to vector code 2023-10-14 13:08:36 -04:00
Robert Muir 872aee6d18
Speedup integer functions for 128-bit neon vectors (#12632) 2023-10-14 11:38:33 -04:00
Robert Muir a4ff129de8
migrate all vectorbench methods to lucene (#12667)
* migrate all vectorbench methods to lucene

Co-authored-by: Uwe Schindler <uschindler@apache.org>
2023-10-14 11:36:42 -04:00
Uwe Schindler 2b8c4b0781
Improve logging for vector support (#12676) 2023-10-13 22:04:05 +02:00
Adrien Grand fb674aa7f2
Fix TestSizeBoundedForceMerge. (#12673)
This test sometimes fails because `SimpleText` has a non-deterministic size for
its segment info file, due to escape characters. The test now enforces the
default codec, and checks that segments have the expected size before moving
forward with forcemerge().

Closes #12648
2023-10-13 18:05:27 +02:00
Greg Miller 7b7b0d2119
Ensure LeafCollector#finish is only called once on the main collector during drill-sideways (#12642) 2023-10-13 07:24:40 -07:00
Greg Miller 681975d6e8
Add javadoc note to LeafCollector#finish (#12643) 2023-10-13 07:03:24 -07:00
Adrien Grand afc990cd2b
Fix lazy decoding of frequencies in `BlockImpactsDocsEnum`. (#12668)
The code was written as if frequencies should be lazily decoding, except that
when refilling buffers freqs were getting eagerly decoded instead of lazily.
2023-10-13 10:42:34 +02:00
Dawid Weiss 06341ffe1d GITHUB#12655: upgrade jacoco aggregation plugin (failed with gradle 8.x), change html output property. 2023-10-12 20:42:17 +02:00
Dawid Weiss a97d7a3e35 Adding Benjamin's readme change #12663 2023-10-12 20:30:58 +02:00
Dawid Weiss 603cd41ca1
Added JMH micro-benchmarks submodule (#12663) 2023-10-12 20:25:34 +02:00
Shubham Chaudhary 52dfe50e8f
Ability to compute vector similarity scores with DoubleValuesSource (#12548)
### Description

This PR addresses the issue #12394. It adds an API **`similarityToQueryVector`** to `DoubleValuesSource` to compute vector similarity scores between the query vector and the `KnnByteVectorField`/`KnnFloatVectorField` for documents using the 2 new DVS implementations (`ByteVectorSimilarityValuesSource` for byte vectors and `FloatVectorSimilarityValuesSource` for float vectors). Below are the method signatures added to DVS in this PR:

- `DoubleValues similarityToQueryVector(LeafReaderContext ctx, float[] queryVector, String vectorField)` *(uses ByteVectorSimilarityValuesSource)*
- `DoubleValues similarityToQueryVector(LeafReaderContext ctx, byte[] queryVector, String vectorField)` *(uses FloatVectorSimilarityValuesSource)*

Closes #12394
2023-10-12 13:34:37 -04:00
Simon Willnauer 268dd54a86
Cleanup flushing logic in DocumentsWriter (#12647)
DocumentsWriter had some duplicate logic for iterating over
segments to be flushed. This change simplifies some of the loops
and moves common code in on place. This also adds tests to ensure
we actually freeze and apply deletes on segment flush.

Relates to #12572
2023-10-12 16:49:39 +02:00
Uwe Schindler fad6653495
Fix Gradle toolchain download with Gradle 8.4 (#12655) (#12662) 2023-10-12 13:22:48 +02:00
Dawid Weiss 2e12a35c87 GITHUB#12655: javac options via -J only when the compiler is in forked mode. 2023-10-12 09:05:03 +02:00
Kevin Risden a8fba38f16
GITHUB#12655: Add google java format upgrade tidy / regen to blame ignore 2023-10-11 16:15:42 -04:00
Kevin Risden de3b294be4
GITHUB#12655: gradle tidy after google java format update for jdk 21 and regen
* tidy whitespace changes from googleJavaFormat upgrade
* generateForUtil fixed and regened https://bugs.python.org/issue39350
* generateAntlr
* generateClassicTokenizer
* generateWikipediaTokenizer
2023-10-11 16:12:09 -04:00
Kevin Risden 2c42b8941a
GITHUB#12655: Update googleJavaFormat plugin 2023-10-11 16:12:01 -04:00
Kevin Risden 30d3eba933
GITHUB#12655: Upgrade to Gradle 8.4
See https://github.com/apache/lucene/pull/12650 for details

Closes GITHUB#12655
2023-10-11 16:11:53 -04:00
Benjamin Trent 05d26ac44d
Refactor Lucene95 to allow off heap vector reader reuse (#12629)
While going through: https://github.com/apache/lucene/pull/12582

I noticed that for a while now, our offheap vector readers haven't changed at all. We just keep copying them around for no reason.

To make adding a new vector codec simpler, this refactors the lucene95 codec to allow its offheap vector storage format (readers/writers) to be used. 

Additionally, it will handle writing the appropriate fields for sparse vectors (read/write) to a provided index output/inputs.

This should reduce the churn in new codecs significantly.
2023-10-10 14:53:54 -04:00
Dzung Bui 04f38dd288
Move addNode to FSTCompiler (#12646)
Currently FSTCompiler and FST have circular dependencies to each 
other. FSTCompiler creates an instance of FST, and on adding node 
(add(IntsRef input, T output)), it delegates to FST.addNode() and passes
itself as a variable. This introduces a circular dependency and mixes up 
the FST constructing and traversing code.

To make matter worse, this implies one can call FST.addNode with an 
arbitrary FSTCompiler (as it's a parameter), but in reality it should be 
the compiler which creates the FST.

This commit moves the addNode method to FSTCompiler instead.

Co-authored-by: Anh Dung Bui <buidun@amazon.com>
2023-10-10 13:34:57 +01:00
gf2121 65d2227f83
Avoid duplicate array fill in BPIndexReorderer (#12645) 2023-10-10 15:10:09 +08:00
gf2121 4f01de2a2d
Write MSB VLong for better outputs sharing in block tree index (#12631) 2023-10-10 14:00:21 +08:00
gf2121 e20e245f47
DeletedTerms#clear should reset ByteBlockPool (#12630) 2023-10-10 13:42:17 +08:00
Patrick Zhai 33a3af4260 Add a little bit more hint to releaseWizard 2023-10-09 17:05:23 -03:00
Piotrek Żygieło dfff1e6358
Avoid NPEx if the end of the stream has been reached without reading any characters (#12611)
e.g. by user responding with ^D
```
Press (n)ext page, (q)uit or enter number to jump to a page.
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "String.length()" because "line" is null
	at org.apache.lucene.demo.SearchFiles.doPagingSearch(SearchFiles.java:244)
	at org.apache.lucene.demo.SearchFiles.main(SearchFiles.java:152)
```

```
Press (p)revious page, (n)ext page, (q)uit or enter number to jump to a page.
n
Only results 1 - 50 of 104 total matching documents collected.
Collect more (y/n) ?
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "String.length()" because "line" is null
	at org.apache.lucene.demo.SearchFiles.doPagingSearch(SearchFiles.java:198)
	at org.apache.lucene.demo.SearchFiles.main(SearchFiles.java:152)
```

Co-authored-by: Piotrek Żygieło <pzygielo@users.noreply.github.com>
2023-10-09 08:11:59 +02:00
Robert Muir ce3a904465
add tests for vectorutils integer boundaries (#12634) 2023-10-08 11:26:33 -04:00
Kaival Parikh 2474940bff
Allow implementers of AbstractKnnVectorQuery to access final topK results (#12590)
* Make AbstractKnnVectorQuery#createRewrittenQuery protected

* Add explicit method to merge per-segment results

* Add javadocs explaining the new method

* Run gradlew tidy

---------

Co-authored-by: Kaival Parikh <kaivalnp@amazon.com>
2023-10-05 15:14:59 -04:00
Adrien Grand 91019d685a
Compute better windows in MaxScoreBulkScorer. (#12593)
MaxScoreBulkScorer computes windows based on the set of clauses that were
essential in the *previous* window. This usually works well as the set of
essential clauses tends to be stable over time, but there are cases when
clauses get swapped between essential and non-essential clauses, and computing
windows based on the previous window can lead to suboptimal choices.

This PR creates a first proposal for the next score window using essential
clauses from the previous window, and then creates a second proposal once
scorers have been partitioned and their max scores have been updated. If this
second proposal results in a smaller window, it gets used.

On one particular query (`the incredibles`) and a reordered index with BP
(which increases chances that scorers move from essential to non-essential or
vice-versa), this change yielded a 2.3x speedup.
2023-10-05 12:46:57 +02:00
gf2121 28f0885bdf
Use a MergeSorter taking advantage of extra storage for StableMSBRadixSorter (#12623) 2023-10-05 04:31:23 -05:00
Michael Peterson 7371493478
TaskExecutor waits for all tasks to complete before returning (#12523)
The TaskExecutor used to run concurrent operations may leave running tasks behind when an exception is thrown by one of the tasks. This commit ensures that it instead waits for all tasks to complete before it re-throws the exception. If there's more than one exception thrown, they are going to be added as suppressed exceptions to the first one that was caught.
2023-10-05 10:25:10 +02:00
iverase ba74da1dc0 Change CHANGES entry to 10 2023-10-04 16:46:11 +02:00
Ignacio Vera c4694c31fd
Add readBytes method to RandomAccessInput (#12600)
Adds a new method to RandomAccessInput tio bulk read bytes into a provided byte array.
2023-10-04 16:23:42 +02:00
gf2121 96052891e6
Reduce FST block size for BlockTreeTermsWriter (#12604) 2023-10-04 01:58:56 -05:00
Kevin Risden 75da33836b
avoid-circular-jar-checks (#12618) 2023-10-03 16:44:55 -04:00
Benjamin Trent 6b7d311c0c
Minor refactor for HNSW graph merging logic (#12616)
This is a minor refactor of HNSW graph merging logic.

Instead of directly checking the KnnVectorReader version, this commit adjusts the logic to see if a specific interface is satisfied for returning a view of the HnswGraph.
2023-10-03 14:28:10 -04:00
Grigoriy Troitskiy 1baae3629a
Make LRUQueryCache respect Accountable queries on eviction and consistency check (#12614)
Given a query that implements Accountable, the LRUQueryCache would increment
its internal accounting by the amount reported by Accountable.ramBytesUsed(), but
only decrement on eviction by the default used for all other queries.  This meant that 
the cache could eventually think it had run out of space, even if there were no queries 
in it at all.  This commit ensures that queries that implement Accountable are always 
accounted for correctly.
2023-10-03 14:52:12 +01:00
Adrien Grand 3f81f2f315 Fix excessive skipping in `BlockMaxConjunctionBulkScorer`. 2023-10-03 11:34:07 +02:00
Luca Cavanna 2106bf5172
Create a task executor when executor is not provided (#12606)
As we introduce more places where we add concurrency (there are
currently three) there is a common pattern around checking whether there
is an executor provided, and then going sequential on the caller thread
or parallel relying on the executor.

That can be improved by internally creating a TaskExecutor that relies
on an executor that executes tasks on the caller thread, which ensures
that the task executor is never null, hence the common conditional is no
longer needed, as the concurrent path that uses the task executor would
be the default and only choice for operations that can be parallelized.
2023-10-03 09:13:45 +02:00
Luca Cavanna 1dd05c89b0
Add missing create github release step to release wizard (#12607)
The "create github release" step was missing from the release wizard. We have forgotten about it a few times recently.

While at it, I also expanded the instructions around closing the current milestone and moved them after removing opened issues / PRs from the current milestone.
2023-10-02 19:59:23 +02:00