Commit Graph

36883 Commits

Author SHA1 Message Date
Mike McCandless 20d5de448a Revert "Copy directly between 2 ByteBlockPool to avoid double-copy (#12778)"
This reverts commit efd68f8165.
2023-11-08 11:14:32 -05:00
Dzung Bui efd68f8165
Copy directly between 2 ByteBlockPool to avoid double-copy (#12778)
* Copy directly between 2 ByteBlockPool to avoid double-copy

* Use TestUtil in test case

* Rename variables

* Rename methods
2023-11-08 11:09:45 -05:00
Jakub Slowinski f698f90171
Re-adding the backward_codecs.lucene90 TestPForUtil + TestForUtil (#12781)
Clean-up from adding the Lucene99PostingsFormat in https://github.com/apache/lucene/pull/12741
These test cases were moved to Lucene99 dir and I forgot to copy the unmodified versions for the backward_codecs.lucene90
2023-11-08 09:01:47 -05:00
Dawid Weiss c85529b2c0
TestIndexWriterOnVMError.testUnknownError times out (fixes potential IW deadlock on tragic exceptions). (#12751) 2023-11-08 09:47:19 +01:00
Jakub Slowinski f416b2b8b5
Fix nested link warning from renderSiteJavadoc (#12779) 2023-11-07 11:24:43 -05:00
Benjamin Trent 34d858fbbb Fix KnnQueryTestCase
Both testEuclidean and testExplain have vectors that result
in equal scores. Since we no longer tie break on vector ordinal
as it doesn't make sense when building the graph, the vectors returned
might be slightly different. This commit fixes the flaky nature of the
test.
2023-11-07 07:55:28 -05:00
Benjamin Trent e1ce1d692c
Stop exploring HNSW graph if scores are not getting better. (#12770)
I noticed while testing lower dimensionality and quantization, we would explore the HNSW graph way too much. I was stuck figuring out why until I noticed the searcher checks for distance equality (not just if the distance is better) when exploring neighbors-of-neighbors. This seems like a bad heuristic, but to double check I looked at what nmslib does. This pointed me back to this commit: nmslib/nmslib#106

Seems like this performance hitch was discovered awhile ago :).

This commit adjusts HNSW to only explore the graph layer if the distance is actually better.
2023-11-07 06:55:42 -05:00
Dzung Bui 3231028070
remove FST's BytesReader#reversed() (#12777) 2023-11-07 05:11:27 -05:00
Robert Muir eab3e502bb
disable vectors (and don't warn to add incubator module) for jvmci/graal (#12766) 2023-11-06 15:28:22 -05:00
Dzung Bui 0e9c23785e
Streamline FST constructors and make it fully read-only (#12758)
* Streamline FST constructors

* Let init return the FSTStore

* Change constructor visibility
2023-11-06 12:34:39 -05:00
Jakub Slowinski 8ae598bae5
Remove patching for doc blocks. (#12741)
* Change Postings back to using FOR in Lucene99PostingsFormat

We are still keeping PFOR for positions only.
This is a partial revert of https://github.com/apache/lucene/pull/69 which brings back ForDeltaUtil.

* fix merge commit

* Add forgotten forDeltaUtil calls to reader

* Addressing comments: adding Lucene90RWPostingsFormat + more

Also:
* Change to Changes.txt
* Removal of dead code which was only used in unit tests
* Removal of test code from PForUtil

* Changes.txt edit in right place now

* Apply suggestions from code review: `90 -> 99 refactoring`

Co-authored-by: gf2121 <52390227+gf2121@users.noreply.github.com>

* Remove decodeTo32 from ForUtil and regenerate

---------

Co-authored-by: gf2121 <52390227+gf2121@users.noreply.github.com>
2023-11-06 10:46:03 -05:00
Mike McCandless 3acc3c864a #12506: escape > in javadocs 2023-11-06 10:41:01 -05:00
Mike McCandless 2a31f2835c #12506: switch to TestUtil for random ints to be consistent with 9.x backport where Java 11 doesn't have this API 2023-11-06 08:41:06 -05:00
Stefan Vodita 6aed68f17a
Clean up ByteBlockPool (#12506)
* Clean up ByteBlockPool

1. Move slice functionality to a separate class, ByteSlicePool.
2. Add exception case if the requested slice size is larger than the
   block size.
3. Make pool buffers private instead of the comment asking not to modify it.
4. Consolidate setBytesRef methods with int offsets.
5. Simplify ramBytesUsed.
6. Update and expand comments and javadoc.

* Revert to long offsets in ByteBlockPool; Clean-up

* Remove slice functionality from TermsHashPerField

* First working test

* [Broken] Test interleaving slices

* Fixed randomized test

* Remove redundant tests

* Tidy

* Add CHANGES

* Move ByteSlicePool to oal.index
2023-11-06 08:12:16 -05:00
Chris Hegarty 8d4f9e50cb
Remove usage of deprecated java.util.Locale constructor (#12761)
This commit removes usages of the deprecated java.util.Locale constructor, replacing with Locale.Builder.
2023-11-06 08:06:45 +00:00
gf2121 1448015641
Use growNoCopy for SortingStoredFieldsConsumer#NO_COMPRESSION (#12733) 2023-11-06 15:41:07 +08:00
Uwe Schindler 382aa549a7
Limit vectorization API to Hotspot VMs (and rename some constants and fix Javadocs) (#12765) 2023-11-05 19:22:33 +01:00
Uwe Schindler 5358b7251e Fix OpenJ9 regression introduced by #12754 2023-11-05 10:53:49 +01:00
Robert Muir 40e55b0ce7
Speed up vectorutil float scalar methods, unroll properly, use fma where possible (#12737)
Co-authored-by: Uwe Schindler <uschindler@apache.org>
2023-11-04 19:25:58 -04:00
Dzung Bui b8a9b0ae29
Use value-based LRU cache in NodeHash (#12738)
* Use value-based LRU cache in NodeHash (#12714)

* tidy code

* Add a nocommit about OffsetAndLength

* Fix the readBytes method

* Use List<byte[]> instead of ByteBlockPool

* Move nodesEqual to PagedGrowableHash

* Add generic type

* Fix the count variable

* Fix the RAM usage measurement

* Use PagedGrowableWriter instead of HashMap

* Remove unused generic type

* Update the ramBytesUsed formula

* Retain the FSTCompiler.addNode signature

* Switch back to ByteBlockPool

* Remove the unnecessary assertion

* Remove fstHashAddress

* Add some javadoc

* Fix the address offset when reading from fallback table

* tidy code

* Address comments

* Add assertions
2023-11-04 11:24:49 -04:00
Chris Hegarty 5ef651fc4c
Replace usage of deprecated java.net.URL constructor with URI (#12755)
This commit replaces the usage of the deprecated java.net.URL constructor with URI, later converting toURL where necessary to interoperate with the URLConnection API.
2023-11-03 20:02:22 +00:00
Uwe Schindler a35573eed9
Refactor access to VM options and move some VM options to oal.util.Constants (#12754) 2023-11-03 20:55:36 +01:00
Dawid Weiss d6836d3d0e
tests.multiplier could be omitted in failed test reproduce line (#12752)
The default tests.multiplier passed from gradle was 1, but
LuceneTestCase tried to compute its default value from TESTS_NIGHTLY.
This could lead to subtle errors: nightly mode failures would not report
tests.multipler=1 and when started from the IDE, the tests.multiplier
would be set to 2 (leading to different randomness).
2023-11-03 17:05:17 +01:00
Mike McCandless 1f3f3ae14f fix comment: Test2BFST was exaggerating a bit about how big an FST it was really making 2023-11-03 07:45:45 -04:00
Adrien Grand 10ebd9025a Fix test after #12549 take 2. 2023-11-03 09:08:34 +01:00
Adrien Grand 43a568cb33
LUCENE-10560: Faster merging of TermsEnum (#1052)
Closes #11596
2023-11-03 08:55:15 +01:00
Michael Sokolov 4c9e241db3
Remove or repurpose obsolete JIRA tasks from release wizard (#11833) 2023-11-02 13:49:52 -04:00
Robert Muir 76aa38f7c0
Merge branch 'stabilize_benchmark' 2023-11-02 13:31:24 -04:00
zhouhui 2d50c345fe
Fix comment on decode method in PForUtil (#12495)
* Fix comment on decode method.

* Fix comment on decode method(lucene84).
2023-11-02 10:34:52 -04:00
twosom 30db217048
move CSVUtil to common from analyzer nori and kuromoji (#12390)
Closes #12389
2023-11-02 10:10:44 -04:00
Adrien Grand 5b87a31556
Speed up sorting on unique string fields. (#11903)
Since increasing the number of hits retrieved in nightly benchmarks from 10 to
100, the performance of sorting documents by title dropped back to the level it
had before introducing dynamic pruning. This is not too surprising given that
the `title` field is a unique field, so the optimization would only kick in
when the current 100th hit would have an ordinal that is less than 128 -
something that would only happen after collecting most hits.

This change increases the threshold to 1024, so that the optimization would
kick in when the current 100th hit has an ordinal that is less than 1024,
something that happens a bit sooner.
2023-11-02 14:16:11 +01:00
tang donghai 4b3f7662ce
unify exception thrown by regexp & check repetition range (#12277)
* unify exception thrown by regexp & check repetition range

* check m equals -1 or not

* needn't rethrow IllegalArgumentException
2023-11-02 08:29:22 -04:00
luyuncheng 96b5edd3bb
Remove unnecessary sort in writeFieldUpdates (#12273)
* Update ReadersAndUpdates.java

Reduce unnecessary sort in writeFieldUpdates

* Update ReadersAndUpdates.java

precommit import
2023-11-02 08:25:06 -04:00
Jan Høydahl 5f1c726807
ReleaseWizard - Upgrade 'consolemenu' dependency to v0.7.1 (#11855)
Ported from https://github.com/apache/solr/pull/1020
Also pin python versions in requirements.txt to avoid unexpected incompatibilties in the future

Co-authored-by: Jan Høydahl <janhoy@users.noreply.github.com>
2023-11-02 07:42:45 -04:00
xiaoshi d62fb5309e
LUCENE-10100: configuration items of the alg file are adapted to the 9.0 branch (#301) 2023-11-02 07:02:31 -04:00
lujiefsi cbb5b6e331
LUCENE-10144:fix resource leak due to Files.list (#354)
* LUCENE-10144:fix resource leak due to Files.list

* LUCENE-10144:fix resource leak due to Files.list
2023-11-02 06:53:48 -04:00
Dzung Bui cdc7d87fcc
Clean up UnCompiledNode.inputCount (#12735)
* Clean up inputCount

* Update CHANGES.txt
2023-11-02 04:34:36 -04:00
Dawid Weiss 8400f89a91
Fix javac task inputs so that they include modular dependencies #12742 (#12745)
Fix javac task inputs so that they include modular dependencies #12742
2023-11-02 08:49:41 +01:00
EC2 Default User 6bf2188b35 stabilize vectorutil benchmark 2023-11-02 04:09:26 +00:00
Adrien Grand 66324f763f Fix test failure.
The test expects that opening a writer on 5 segments doesn't cause merging, but
actually it does since randomization created a merge policy with a factor of 5.
2023-11-01 21:40:44 +01:00
Adrien Grand 8fa0de2743
Specialize the 2nd clause of conjunctions. (#12713)
This adds a bit more specialization to how we handle the 2nd clause in
conjunctions, which seems to help the JVM quite significantly.
2023-10-31 18:24:29 +01:00
Davis Cook bbf3221049
Fix NullPointerException in Monitor.getQuery when query is not present (#12736) 2023-10-31 15:05:31 +00:00
Adrien Grand 361f0a4524
Fix file handle leak in Lucene99ScalarQuantizedVectorsWriter. (#12739)
If `mergeQuantizedByteVectorValues` fails with an exception, the temp output
never gets closed. This was found by the test that throws random exceptions.
2023-10-31 11:35:05 +01:00
Adrien Grand 85f5d3bb0b Fix test after #12549. 2023-10-31 09:47:19 +01:00
Patrick Zhai 4576ae09e8 Do not close merge threadpool in Lucene99HnswVectorsWriter 2023-10-30 23:08:06 -07:00
Greg Miller a0887c7d26 CHANGES entry for GITHUB#12427 2023-10-30 12:53:44 -07:00
Shubham Chaudhary 9cffd4bcc6
Replace assert with IAE in StringsToAutomaton#build if data is not sorted (#12427)
Also update the API to accept more general Iterable<BytesRef> instead of Collection<BytesRef>
2023-10-30 12:47:56 -07:00
Robert Muir e292a5fe4f
Merge branch 'speed_float_cosine' 2023-10-30 12:29:01 -04:00
Robert Muir fcef1bfc1b
CHANGES 2023-10-30 12:25:48 -04:00
Adrien Grand 58b9352cc2
Add a specialized bulk scorer for regular conjunctions. (#12719)
PR #12382 added a bulk scorer for top-k hits on conjunctions that yielded a
significant speedup (annotation
[FP](http://people.apache.org/~mikemccand/lucenebench/AndHighHigh.html)). This
change proposes a similar change for exhaustive collection of conjunctive
queries, e.g. for counting, faceting, etc.
2023-10-30 16:11:19 +01:00