* Binary search the entries when all suffixes have the same length in a leaf block.
* add comment on allEqual.
* BackwardsCompatibility: keep the same logic on fillTerm and SeekStatus(NOT_FOUND, END).
* Update comments: modify scan to binary search.
* Add unit test for binarySearchTermLeaf.
* Format code.
* Assert the value of termsEnum.term() is correct after seeking.
* Add CHANGES entry.
* Clarify "leaf block _of the terms dict_"
* Set suffixesReader's position.
* Advance to the greater term If binary search ended at the less term.
* Assert termsEnum's position after seeking.
* Tidy.
* Advance to the greater term if binary search ended at the less term: nextEnt plus 1.
* Advance to the greater term if binary search ended at the less term and greater term exists.
* Add test case: target greater than the last entry of the matched block.
* Move test case that target greater than the last entry of the matched block to TestLucene90PostingsFormat.
* Move test case for target greater than the last entry of the matched block to TestLucene99PostingsFormat
* Clarify code.
* Replace ternary with verbose if.
* Replace seekExact with seekCeil.
* Replace division by 2 with logical right shift.
* Remove assert ste.termExists.
* Clarify code.
* Remove stale change entry.
* Fix comment.
---------
Co-authored-by: Adrien Grand <jpountz@gmail.com>
Correct estimate for singleton to return `0` and use custom accumulator in tests to
fix assertions. Tried not doing this in #13232 but turns out we need a little complexity here
since the singleton is recognized by the `RamUsageTester` in so far that is only counted
once if it shows up repeatedly in an array or so.
fixes#13249
I repeatably saw some test failures related to `TestParentBlockJoin[Byte|Float]KnnVectorQuery#testVectorEncodingMismatch`. This commit fixes those test failures and actually checks the field type.
Both the KnnWriters & FieldInfo keep track of the vector similarity used by a given field. This commit ensures they are the same and utilizes the FieldInfo one (which, while these are enums, are exactly the same).
Size 256 is very common here throguh the monotonic long values default
page size. In ES we're seing many MB O(10M) of duplicate instances of this
size relatively quickly.
=> adding a singleton for it to save some heap
This updates `IndexWriter` to only call `OneMerge#reorder` when it has a chance
to preserve the block structure, ie. either there are no blocks or blocks are
identified by a parent field.
Furthermore, `MockRandomMergePolicy` is updated to preserve the block structure
when a parent field is configured.
Merging `IOContext`s use a `SEQUENTIAL` `ReadAdvice`. However, some file
formats hardcode `IOContext.LOAD` for some of their files, which silences the
whole merging context, in particular the `SEQUENTIAL` `ReadAdvice`.
This PR switches file formats to
`ioContext.withReadAdvice(ReadAdvice.RANDOM_PRELOAD)` so that merges will use a
`SEQUENTIAL` `ReadAdvice` while searches will use a `RANDOM_PRELOAD`
`ReadAdvice`.
This is not a huge deal for `RANDOM_PRELOAD`, which is only used for very small
files. However, this change becomes more relevant for the new `RANDOM`
`ReadAdvice` as we would like merges to keep using a `SEQUENTIAL` `ReadAdvice`.
Having a single block of all zeros is a fairly common case that is using
a lot of heap for duplicate instances in some use-cases in ES.
=> read a singleton for it to save the duplication
Romanian uses s&t with commas (ș/ț), but for a long time those weren't available, so s&t with cedilla (ş/ţ) were used. Both are still in use, but the comma forms are much more common now. Both should be supported in stopword lists.
The variable-gaps terms format uses the legacy storage layout of storing
metadata at the end of the index file, and storing the start pointer of the
metadata as the last 8 bytes of the index files (just before the footer). This
forces an awkward access pattern at open time when we first need to seek
towards the end to check that a footer is present, then seek some more bytes
backwards to read metadata, and finally read the content of the index that sits
before metadata.
To fix this, meta data and index data are now split into different files. This
way, both files have a clean sequential and read-once access pattern, and can
take advantage of the `ChecksumIndexInput` abstraction for checksum validation.
This further helps clean up `IOContext` by removing the ability to set
`readOnce` to `true` on an existing `IOContext`.
* upgrade snowball to 26db1ab9adbf437f37a6facd3ee2aad1da9eba03
* add back-compat-hack to the factory, too
* remove open of irish package now that we don't have our own stopwords file here anymore
* CHANGES / MIGRATE
This replaces the `load`, `randomAccess` and `readOnce` flags with a
`ReadAdvice` enum, whose values are aligned with the allowed values to
(f|m)advise.
Closes#13211
The PerFieldDocValuesFormat and PerFieldPostingsFormat mutate and reset
the fieldInfos of the mergeState during merges. Consequently, other
running merge sub-tasks may fail to see some fieldInfos. This was
problematic since we introduced concurrency for merge sub-tasks.
Relates #13190
This commit adds a new interface to all MergeScheduler classes that allows the scheduler to provide an Executor for intra-merge parallelism. The first sub-class to satisfy this new interface is the ConcurrentMergeScheduler (CMS). In particular, the CMS will limit the amount of parallelism based on the current number of mergeThreads and the configured maxThread options.
Parallelism is now added by default to the SegmentMerger where each merging task is executed in parallel with the others.
Additionally, the Lucene99 HNSW codec will use the provided MergeScheduler executor if no executor is provided directly.
Relates to: https://github.com/apache/lucene/issues/12740
Relates to: https://github.com/apache/lucene/issues/9626
This is a take 2 of: https://github.com/apache/lucene/pull/13124
Currently, when performing KNN exact search, we consistently set the HitQueue size to k. However, there may be instances where the number of candidates is actually lower than k.
### Description
Sutegana (捨て仮名) is small letter of hiragana and katakana in Japanese. In the old Japanese text, sutegana (捨て仮名) is not used unlikely to modern one. For example:
- "ストップウォッチ" is written as "ストツプウオツチ"
- "ちょっとまって" is written as "ちよつとまつて"
So it's meaningful to normalize sutegana to normal (uppercase) characters if we search against the corpus which includes old Japanese text such as patents, legal documents, contract policies, etc.
This pull request introduces 2 token filters:
- JapaneseHiraganaUppercaseFilter for hiragana
- JapaneseKatakanaUppercaseFilter for katakana
so that user can use either one separately. Each. filter make all the sutegana (small characters) into normal kana (uppercase character) to normalize the token.
### Why it is needed
This transformation must be done as token filter. There have already been [MappingCharFilter](https://lucene.apache.org/core/8_0_0/analyzers-common/org/apache/lucene/analysis/charfilter/MappingCharFilter.html), but if we apply this character filter to normalize sutegana, it will impact to tokenization and it is not expected.
This commit adds a new interface to all MergeScheduler classes that allows the scheduler to provide an Executor for intra-merge parallelism. The first sub-class to satisfy this new interface is the ConcurrentMergeScheduler (CMS). In particular, the CMS will limit the amount of parallelism based on the current number of mergeThreads and the configured maxThread options.
Parallelism is now added by default to the SegmentMerger where each merging task is executed in parallel with the others.
Additionally, the Lucene99 HNSW codec will use the provided MergeScheduler executor if no executor is provided directly.
Relates to: https://github.com/apache/lucene/issues/12740
Relates to: https://github.com/apache/lucene/issues/9626
The current logic for reordering splits a slice of doc IDs into a left side and
a right side, and for each document it computes the expected gain of moving to
the other side. Then it swaps documents from both sides as long as the sum of
the gain of moving the left doc to the right and the right doc to the left is
positive.
This works well, but I would like to extend BP reordering to also work with
blocks, and the swapping logic is challenging to modify as two parent documents
may have different numbers of children.
One of the follow-up papers on BP suggested to use a different logic, where one
would compute a bias for all documents that is negative when a document is
attracted to the left and positive otherwise. Then we only have to partition doc
IDs around the mid point, e.g. with quickselect.
A benefit of this change is that it will make it easier to generalize BP
reordering to indexes that have blocks, e.g. by using a stable sort on biases.
zhouhui
Armin Braun
Robert Muir
Benjamin Trent
Kaival Parikh
Uwe Schindler
Adrien Grand
Tommaso Teofili
Trey Jones
Jean-François BOEUF
Zhang Chao
gf2121
Chris Hegarty
Nhat Nguyen
Tim Grein
Shintaro Murakami
panguixin
Dai Sugimori