37231 Commits

Author SHA1 Message Date
Tim Grein
3f4413567d
Fix TestFloatVectorSimilarityQuery.testBoostQuery (#13176)
### Description

The boosted doc scores were the same due to floating point precision limitations, when multiplying the original scores with tiny differences with a fairly large multiplier.

My change removes the assertion expecting the exact order to be the same (AFAIU this won't be possible for every scores/boost configurations due to precision limitations), but rather asserts that the `original doc score * boost factor` and the `boosted doc score` are equals within a certain delta.

Closes https://github.com/apache/lucene/issues/13173.
2024-03-11 08:08:29 -04:00
panguixin
5b5815a26d
Fix NPE when LeafReader return null VectorValues (#13162)
### Description
`LeafReader#getXXXVectorValues` may return null value.

**Reproduction**:
```
public class TestKnnByteVectorQuery extends BaseKnnVectorQueryTestCase {
  public void testVectorEncodingMismatch() throws IOException {
    try (Directory indexStore =
        getIndexStore("field", new float[] {0, 1}, new float[] {1, 2}, new float[] {0, 0});
        IndexReader reader = DirectoryReader.open(indexStore)) {
      AbstractKnnVectorQuery query =
          new KnnFloatVectorQuery("field", new float[] {0, 1}, 10);
      IndexSearcher searcher = newSearcher(reader);
      searcher.search(query, 10);
    }
  }
}
```
**Output**:
```
java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.FloatVectorValues.size()" because the return value of "org.apache.lucene.index.LeafReader.getFloatVectorValues(String)" is null
```
2024-03-11 08:07:04 -04:00
zhouhui
6445bc0a14
Rename TestWildcard to TestWildcardQuery. (#13159) 2024-03-08 09:26:44 -05:00
Dawid Weiss
1c77e2315c
An eye-gouging way to limit suppressAccessChecks to just the three JARs that need them. (#13164) 2024-03-08 08:10:49 +01:00
Tim Grein
49db29e59f
Check whether document exists without relying on the id (#13160) 2024-03-06 05:50:59 -05:00
Benjamin Trent
012b959b05
Add mult-leaf optimizations for diversify children collector (#13121)
This adds multi-leaf optimizations for diversified children collector. This means as children vectors are collected within a block join, we can share information between leaves to speed up vector search.

To make this happen, I refactored the multi-leaf collector slightly. Now, instead of inheriting from TopKnnCollector, we inject a inner collector.
2024-03-05 09:02:49 -05:00
Peter Gromov
51122f8b2e
Hunspell: don't proceed with other suggestions if we found good REP ones (#13156)
* Hunspell: don't proceed with other suggestions if we found good REP ones

This makes "rep" and "ph" from TestHunspellRepositoryTestCases pass on latest revisions of the original Hunspell (after b88f9ea57b)
2024-03-05 14:12:01 +01:00
Peter Gromov
44d48df4b7
hunspell: allow ignoring exceptions on duplicate ICONV/OCONV mappings (#13155)
hunspell: allow ignoring exceptions on duplicate ICONV/OCONV mappings
2024-03-05 09:54:20 +01:00
Peter Gromov
193cc62c73
hunspell GeneratingSuggester: ensure there are never more than 100 roots to process (#13154)
hunspell GeneratingSuggester: ensure there are never more than 100 roots to process
2024-03-05 09:53:27 +01:00
guojialiang92
47792dfcd3
Fix DV update files referenced by merge will be deleted by concurrent flush (#13017) 2024-03-04 11:18:16 +00:00
Dawid Weiss
3ce9ba9fd5 Correct typo #13148 2024-03-01 07:12:57 +01:00
Uwe Schindler
08325ac3e8 Fix successful tests counting not working in Gradle build by adding ReflectPermission back (see ##13146) 2024-03-01 01:25:02 +01:00
Uwe Schindler
6910a4358c
Do not place Panama Java 21 class files in MR-JAR section of core.jar file (#13148) 2024-02-29 23:10:16 +01:00
Uwe Schindler
8ef60b3e72 Add changes entry (#13146) 2024-02-29 19:42:32 +01:00
Uwe Schindler
e446904c61
Remove ByteBufferIndexInput and update all Panama implementations (MMap and Vector) to Java 21 (#13146) 2024-02-29 19:38:37 +01:00
Uwe Schindler
dfce6ee8d2 Update the Javadoc package list to Java 21 2024-02-29 15:06:47 +01:00
Uwe Schindler
5aaaeaee39 Update link to javadocs for Java 21 2024-02-29 14:37:09 +01:00
Dawid Weiss
ba77cd4444 Make run-nightly-smoketester.yml run on java 21+ only 2024-02-29 13:03:42 +01:00
Uwe Schindler
62e2b08aa7 Remove empty file 2024-02-29 12:21:11 +01:00
Uwe Schindler
8f17f23acf Bump minimum required Java version to 21 (#12753)
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
Co-authored-by: Robert Muir <rmuir@apache.org>
2024-02-29 12:16:29 +01:00
Uwe Schindler
e7d2bd48a6 Revert "Merge branch 'java_21' of https://github.com/ChrisHegarty/lucene into main"
This reverts commit a356fc1e23ffc2f569b930f4c3431804df9a1e07, reversing
changes made to 7b01f2f516635eced934f3b950d58ad179bf0256.
2024-02-29 11:58:40 +01:00
Uwe Schindler
fe7382b253 Deprecate ByteBufferIndexInput and detect MemorySegmentIndexInput correctly in NRTSuggester (#13145) 2024-02-29 11:50:36 +01:00
Uwe Schindler
a356fc1e23 Merge branch 'java_21' of https://github.com/ChrisHegarty/lucene into main 2024-02-28 23:59:01 +01:00
Dmitry Cherniachenko
7b01f2f516
Make static final Set constants immutable (#13087) 2024-02-28 22:23:41 +01:00
Dawid Weiss
390c109e67
Add a nightly workflow to run and verify buildAndPushRelease.py and smokeTestRelease.py (#13141) 2024-02-28 11:49:39 +01:00
Uwe Schindler
bfa64b0725
Merge branch 'main' into java_21 2024-02-27 13:42:36 +01:00
Stefan Vodita
42269203cc
Allow multiple JDKs in smoke test (#13139)
Allow multiple JDKs in smoke test

Remove special handling for JDK 17/19

Consolidate base version into a single constant

Handle version checks and logs
2024-02-27 12:12:20 +00:00
Mikhail Khludnev
6c8be68f95
Update CHANGES.txt SynonymQuery.getField() (#13077) (#13136)
move #13077 under 9.11.0
2024-02-27 12:18:44 +03:00
Mikhail Khludnev
90e4c44b99
Log MockRandomMergePolicy reverse in verbose. (#13117) 2024-02-27 12:09:35 +03:00
Andrey Bozhko
6764a01805
Add getter for SynonymQuery#field (#13077)
---------

Co-authored-by: Andrey Bozhko <abozhko@apple.com>
2024-02-27 11:43:45 +03:00
Chris Hostetter
bf6f38665e Fix ByteKnnVectorFieldSource & FloatKnnVectorFieldSource to work correctly when a segment does not contain any docs with vectors (#13105) 2024-02-26 12:12:40 -07:00
Shubham Chaudhary
6eba1fb537
Fix test TestKnnByteVectorQuery.testToString (#13134)
Co-authored-by: Shubham Chaudhary <cshbha@amazon.com>
2024-02-26 02:31:49 -05:00
Zhang Chao
ca06693a16
Reduce ArrayUtil#grow in decompress (#12996) 2024-02-25 12:09:48 -08:00
Uwe Schindler
0ccb119495
Merge branch 'main' into java_21 2024-02-25 16:39:41 +01:00
Uwe Schindler
47021ae98f
Remove hardcoded "--release" from renderJavadoc task (#13132) 2024-02-25 16:32:30 +01:00
Robert Muir
738d661b94
fix smoketester 2024-02-24 15:11:24 -05:00
Robert Muir
3d7c88e695
fix release wizard 2024-02-24 15:08:11 -05:00
ChrisHegarty
54b6248a8d Merge branch 'main' into java_21 2024-02-23 17:07:54 +00:00
ChrisHegarty
e63df4d879 update changes.txt 2024-02-23 17:07:31 +00:00
Dmitry Cherniachenko
8f759d5bf6
Avoid allocating redundant Strings (#13085) 2024-02-23 10:41:53 +00:00
Adrien Grand
61f322905a
Change BP reordering logic to help support document blocks later on. (#13123)
The current logic for reordering splits a slice of doc IDs into a left side and
a right side, and for each document it computes the expected gain of moving to
the other side. Then it swaps documents from both sides as long as the sum of
the gain of moving the left doc to the right and the right doc to the left is
positive.

This works well, but I would like to extend BP reordering to also work with
blocks, and the swapping logic is challenging to modify as two parent documents
may have different numbers of children.

One of the follow-up papers on BP suggested to use a different logic, where one
would compute a bias for all documents that is negative when a document is
attracted to the left and positive otherwise. Then we only have to partition doc
IDs around the mid point, e.g. with quickselect.

A benefit of this change is that it will make it easier to generalize BP
reordering to indexes that have blocks, e.g. by using a stable sort on biases.
2024-02-23 09:09:19 +01:00
Chris Hegarty
17cbedccfc
FieldInfosFormat translation should be independent of VectorSimilartyFunction enum (#13119)
This commit updates the FieldInfosFormat translation of vector similarity functions to be independent of the VectorSimilartyFunction enum.

The VectorSimilartyFunction enum lives outside of the codec format, and the format should not inadvertently depend upon the declaration order or values in VectorSimilartyFunction. The format should be in charge of the translation of similarity function to format ordinal (and visa versa). In reality, and for now, the translation remains the same as the declaration order, but this may not be the case in the future.
2024-02-22 17:22:20 +00:00
Zhang Chao
6732d2b2fa
fix (#13122) 2024-02-22 14:11:07 +08:00
Adrien Grand
568c8eba97 Use NIO2 APIs. 2024-02-20 22:37:18 +01:00
Adrien Grand
75b26b1bee Bump min version with parent field. 2024-02-20 22:29:11 +01:00
Adrien Grand
dce66f1e3e Add back-compat indices for 9.10.0 2024-02-20 22:16:56 +01:00
Adrien Grand
13d561af1d Fix bw index generation logic. 2024-02-20 22:14:06 +01:00
Adrien Grand
47b02dba1e DOAP changes for release 9.10.0 2024-02-20 18:17:14 +01:00
Prabhat
ff59150934
Added queued flush check optimization to changes.txt (#13118)
Co-authored-by: Prabhat Sharma <ptsharma@amazon.com>
2024-02-20 17:21:05 +01:00
Prabhat
8a73cdef8c
Short circuit queued flush check when flush on update is disabled (#13115)
Co-authored-by: Prabhat Sharma <ptsharma@amazon.com>
2024-02-19 09:21:38 -05:00