Commit Graph

37383 Commits

Author SHA1 Message Date
Benjamin Trent b527e101e7
Fix potential race condition in DocumentsWriter & DocumentsWriterDeleteQueue (#13169)
* Fix potential race condition in DocumentsWriter & DocumentsWriterDeleteQueue

* add changes
2024-03-12 07:50:04 -04:00
Tim Grein 44fa35b65f
Adjust middle of sentence calculation and add test verifying correct behavior (#13170) 2024-03-11 13:21:22 +01:00
Tim Grein 3f4413567d
Fix TestFloatVectorSimilarityQuery.testBoostQuery (#13176)
### Description

The boosted doc scores were the same due to floating point precision limitations, when multiplying the original scores with tiny differences with a fairly large multiplier.

My change removes the assertion expecting the exact order to be the same (AFAIU this won't be possible for every scores/boost configurations due to precision limitations), but rather asserts that the `original doc score * boost factor` and the `boosted doc score` are equals within a certain delta.

Closes https://github.com/apache/lucene/issues/13173.
2024-03-11 08:08:29 -04:00
panguixin 5b5815a26d
Fix NPE when LeafReader return null VectorValues (#13162)
### Description
`LeafReader#getXXXVectorValues` may return null value.

**Reproduction**:
```
public class TestKnnByteVectorQuery extends BaseKnnVectorQueryTestCase {
  public void testVectorEncodingMismatch() throws IOException {
    try (Directory indexStore =
        getIndexStore("field", new float[] {0, 1}, new float[] {1, 2}, new float[] {0, 0});
        IndexReader reader = DirectoryReader.open(indexStore)) {
      AbstractKnnVectorQuery query =
          new KnnFloatVectorQuery("field", new float[] {0, 1}, 10);
      IndexSearcher searcher = newSearcher(reader);
      searcher.search(query, 10);
    }
  }
}
```
**Output**:
```
java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.FloatVectorValues.size()" because the return value of "org.apache.lucene.index.LeafReader.getFloatVectorValues(String)" is null
```
2024-03-11 08:07:04 -04:00
zhouhui 6445bc0a14
Rename TestWildcard to TestWildcardQuery. (#13159) 2024-03-08 09:26:44 -05:00
Dawid Weiss 1c77e2315c
An eye-gouging way to limit suppressAccessChecks to just the three JARs that need them. (#13164) 2024-03-08 08:10:49 +01:00
Tim Grein 49db29e59f
Check whether document exists without relying on the id (#13160) 2024-03-06 05:50:59 -05:00
Benjamin Trent 012b959b05
Add mult-leaf optimizations for diversify children collector (#13121)
This adds multi-leaf optimizations for diversified children collector. This means as children vectors are collected within a block join, we can share information between leaves to speed up vector search.

To make this happen, I refactored the multi-leaf collector slightly. Now, instead of inheriting from TopKnnCollector, we inject a inner collector.
2024-03-05 09:02:49 -05:00
Peter Gromov 51122f8b2e
Hunspell: don't proceed with other suggestions if we found good REP ones (#13156)
* Hunspell: don't proceed with other suggestions if we found good REP ones

This makes "rep" and "ph" from TestHunspellRepositoryTestCases pass on latest revisions of the original Hunspell (after b88f9ea57b)
2024-03-05 14:12:01 +01:00
Peter Gromov 44d48df4b7
hunspell: allow ignoring exceptions on duplicate ICONV/OCONV mappings (#13155)
hunspell: allow ignoring exceptions on duplicate ICONV/OCONV mappings
2024-03-05 09:54:20 +01:00
Peter Gromov 193cc62c73
hunspell GeneratingSuggester: ensure there are never more than 100 roots to process (#13154)
hunspell GeneratingSuggester: ensure there are never more than 100 roots to process
2024-03-05 09:53:27 +01:00
guojialiang92 47792dfcd3
Fix DV update files referenced by merge will be deleted by concurrent flush (#13017) 2024-03-04 11:18:16 +00:00
Dawid Weiss 3ce9ba9fd5 Correct typo #13148 2024-03-01 07:12:57 +01:00
Uwe Schindler 08325ac3e8 Fix successful tests counting not working in Gradle build by adding ReflectPermission back (see ##13146) 2024-03-01 01:25:02 +01:00
Uwe Schindler 6910a4358c
Do not place Panama Java 21 class files in MR-JAR section of core.jar file (#13148) 2024-02-29 23:10:16 +01:00
Uwe Schindler 8ef60b3e72 Add changes entry (#13146) 2024-02-29 19:42:32 +01:00
Uwe Schindler e446904c61
Remove ByteBufferIndexInput and update all Panama implementations (MMap and Vector) to Java 21 (#13146) 2024-02-29 19:38:37 +01:00
Uwe Schindler dfce6ee8d2 Update the Javadoc package list to Java 21 2024-02-29 15:06:47 +01:00
Uwe Schindler 5aaaeaee39 Update link to javadocs for Java 21 2024-02-29 14:37:09 +01:00
Dawid Weiss ba77cd4444 Make run-nightly-smoketester.yml run on java 21+ only 2024-02-29 13:03:42 +01:00
Uwe Schindler 62e2b08aa7 Remove empty file 2024-02-29 12:21:11 +01:00
Uwe Schindler 8f17f23acf Bump minimum required Java version to 21 (#12753)
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
Co-authored-by: Robert Muir <rmuir@apache.org>
2024-02-29 12:16:29 +01:00
Uwe Schindler e7d2bd48a6 Revert "Merge branch 'java_21' of https://github.com/ChrisHegarty/lucene into main"
This reverts commit a356fc1e23, reversing
changes made to 7b01f2f516.
2024-02-29 11:58:40 +01:00
Uwe Schindler fe7382b253 Deprecate ByteBufferIndexInput and detect MemorySegmentIndexInput correctly in NRTSuggester (#13145) 2024-02-29 11:50:36 +01:00
Uwe Schindler a356fc1e23 Merge branch 'java_21' of https://github.com/ChrisHegarty/lucene into main 2024-02-28 23:59:01 +01:00
Dmitry Cherniachenko 7b01f2f516
Make `static final Set` constants immutable (#13087) 2024-02-28 22:23:41 +01:00
Dawid Weiss 390c109e67
Add a nightly workflow to run and verify buildAndPushRelease.py and smokeTestRelease.py (#13141) 2024-02-28 11:49:39 +01:00
Uwe Schindler bfa64b0725
Merge branch 'main' into java_21 2024-02-27 13:42:36 +01:00
Stefan Vodita 42269203cc
Allow multiple JDKs in smoke test (#13139)
Allow multiple JDKs in smoke test

Remove special handling for JDK 17/19

Consolidate base version into a single constant

Handle version checks and logs
2024-02-27 12:12:20 +00:00
Mikhail Khludnev 6c8be68f95
Update CHANGES.txt SynonymQuery.getField() (#13077) (#13136)
move #13077 under 9.11.0
2024-02-27 12:18:44 +03:00
Mikhail Khludnev 90e4c44b99
Log MockRandomMergePolicy reverse in verbose. (#13117) 2024-02-27 12:09:35 +03:00
Andrey Bozhko 6764a01805
Add getter for SynonymQuery#field (#13077)
---------

Co-authored-by: Andrey Bozhko <abozhko@apple.com>
2024-02-27 11:43:45 +03:00
Chris Hostetter bf6f38665e Fix ByteKnnVectorFieldSource & FloatKnnVectorFieldSource to work correctly when a segment does not contain any docs with vectors (#13105) 2024-02-26 12:12:40 -07:00
Shubham Chaudhary 6eba1fb537
Fix test TestKnnByteVectorQuery.testToString (#13134)
Co-authored-by: Shubham Chaudhary <cshbha@amazon.com>
2024-02-26 02:31:49 -05:00
Zhang Chao ca06693a16
Reduce ArrayUtil#grow in decompress (#12996) 2024-02-25 12:09:48 -08:00
Uwe Schindler 0ccb119495
Merge branch 'main' into java_21 2024-02-25 16:39:41 +01:00
Uwe Schindler 47021ae98f
Remove hardcoded "--release" from renderJavadoc task (#13132) 2024-02-25 16:32:30 +01:00
Robert Muir 738d661b94
fix smoketester 2024-02-24 15:11:24 -05:00
Robert Muir 3d7c88e695
fix release wizard 2024-02-24 15:08:11 -05:00
ChrisHegarty 54b6248a8d Merge branch 'main' into java_21 2024-02-23 17:07:54 +00:00
ChrisHegarty e63df4d879 update changes.txt 2024-02-23 17:07:31 +00:00
Dmitry Cherniachenko 8f759d5bf6
Avoid allocating redundant Strings (#13085) 2024-02-23 10:41:53 +00:00
Adrien Grand 61f322905a
Change BP reordering logic to help support document blocks later on. (#13123)
The current logic for reordering splits a slice of doc IDs into a left side and
a right side, and for each document it computes the expected gain of moving to
the other side. Then it swaps documents from both sides as long as the sum of
the gain of moving the left doc to the right and the right doc to the left is
positive.

This works well, but I would like to extend BP reordering to also work with
blocks, and the swapping logic is challenging to modify as two parent documents
may have different numbers of children.

One of the follow-up papers on BP suggested to use a different logic, where one
would compute a bias for all documents that is negative when a document is
attracted to the left and positive otherwise. Then we only have to partition doc
IDs around the mid point, e.g. with quickselect.

A benefit of this change is that it will make it easier to generalize BP
reordering to indexes that have blocks, e.g. by using a stable sort on biases.
2024-02-23 09:09:19 +01:00
Chris Hegarty 17cbedccfc
FieldInfosFormat translation should be independent of VectorSimilartyFunction enum (#13119)
This commit updates the FieldInfosFormat translation of vector similarity functions to be independent of the VectorSimilartyFunction enum.

The VectorSimilartyFunction enum lives outside of the codec format, and the format should not inadvertently depend upon the declaration order or values in VectorSimilartyFunction. The format should be in charge of the translation of similarity function to format ordinal (and visa versa). In reality, and for now, the translation remains the same as the declaration order, but this may not be the case in the future.
2024-02-22 17:22:20 +00:00
Zhang Chao 6732d2b2fa
fix (#13122) 2024-02-22 14:11:07 +08:00
Adrien Grand 568c8eba97 Use NIO2 APIs. 2024-02-20 22:37:18 +01:00
Adrien Grand 75b26b1bee Bump min version with parent field. 2024-02-20 22:29:11 +01:00
Adrien Grand dce66f1e3e Add back-compat indices for 9.10.0 2024-02-20 22:16:56 +01:00
Adrien Grand 13d561af1d Fix bw index generation logic. 2024-02-20 22:14:06 +01:00
Adrien Grand 47b02dba1e DOAP changes for release 9.10.0 2024-02-20 18:17:14 +01:00