Size 256 is very common here throguh the monotonic long values default
page size. In ES we're seing many MB O(10M) of duplicate instances of this
size relatively quickly.
=> adding a singleton for it to save some heap
This updates `IndexWriter` to only call `OneMerge#reorder` when it has a chance
to preserve the block structure, ie. either there are no blocks or blocks are
identified by a parent field.
Furthermore, `MockRandomMergePolicy` is updated to preserve the block structure
when a parent field is configured.
Merging `IOContext`s use a `SEQUENTIAL` `ReadAdvice`. However, some file
formats hardcode `IOContext.LOAD` for some of their files, which silences the
whole merging context, in particular the `SEQUENTIAL` `ReadAdvice`.
This PR switches file formats to
`ioContext.withReadAdvice(ReadAdvice.RANDOM_PRELOAD)` so that merges will use a
`SEQUENTIAL` `ReadAdvice` while searches will use a `RANDOM_PRELOAD`
`ReadAdvice`.
This is not a huge deal for `RANDOM_PRELOAD`, which is only used for very small
files. However, this change becomes more relevant for the new `RANDOM`
`ReadAdvice` as we would like merges to keep using a `SEQUENTIAL` `ReadAdvice`.
Having a single block of all zeros is a fairly common case that is using
a lot of heap for duplicate instances in some use-cases in ES.
=> read a singleton for it to save the duplication
Romanian uses s&t with commas (ș/ț), but for a long time those weren't available, so s&t with cedilla (ş/ţ) were used. Both are still in use, but the comma forms are much more common now. Both should be supported in stopword lists.
The variable-gaps terms format uses the legacy storage layout of storing
metadata at the end of the index file, and storing the start pointer of the
metadata as the last 8 bytes of the index files (just before the footer). This
forces an awkward access pattern at open time when we first need to seek
towards the end to check that a footer is present, then seek some more bytes
backwards to read metadata, and finally read the content of the index that sits
before metadata.
To fix this, meta data and index data are now split into different files. This
way, both files have a clean sequential and read-once access pattern, and can
take advantage of the `ChecksumIndexInput` abstraction for checksum validation.
This further helps clean up `IOContext` by removing the ability to set
`readOnce` to `true` on an existing `IOContext`.
* upgrade snowball to 26db1ab9adbf437f37a6facd3ee2aad1da9eba03
* add back-compat-hack to the factory, too
* remove open of irish package now that we don't have our own stopwords file here anymore
* CHANGES / MIGRATE
This replaces the `load`, `randomAccess` and `readOnce` flags with a
`ReadAdvice` enum, whose values are aligned with the allowed values to
(f|m)advise.
Closes#13211
The PerFieldDocValuesFormat and PerFieldPostingsFormat mutate and reset
the fieldInfos of the mergeState during merges. Consequently, other
running merge sub-tasks may fail to see some fieldInfos. This was
problematic since we introduced concurrency for merge sub-tasks.
Relates #13190
This commit adds a new interface to all MergeScheduler classes that allows the scheduler to provide an Executor for intra-merge parallelism. The first sub-class to satisfy this new interface is the ConcurrentMergeScheduler (CMS). In particular, the CMS will limit the amount of parallelism based on the current number of mergeThreads and the configured maxThread options.
Parallelism is now added by default to the SegmentMerger where each merging task is executed in parallel with the others.
Additionally, the Lucene99 HNSW codec will use the provided MergeScheduler executor if no executor is provided directly.
Relates to: https://github.com/apache/lucene/issues/12740
Relates to: https://github.com/apache/lucene/issues/9626
This is a take 2 of: https://github.com/apache/lucene/pull/13124
Currently, when performing KNN exact search, we consistently set the HitQueue size to k. However, there may be instances where the number of candidates is actually lower than k.
### Description
Sutegana (捨て仮名) is small letter of hiragana and katakana in Japanese. In the old Japanese text, sutegana (捨て仮名) is not used unlikely to modern one. For example:
- "ストップウォッチ" is written as "ストツプウオツチ"
- "ちょっとまって" is written as "ちよつとまつて"
So it's meaningful to normalize sutegana to normal (uppercase) characters if we search against the corpus which includes old Japanese text such as patents, legal documents, contract policies, etc.
This pull request introduces 2 token filters:
- JapaneseHiraganaUppercaseFilter for hiragana
- JapaneseKatakanaUppercaseFilter for katakana
so that user can use either one separately. Each. filter make all the sutegana (small characters) into normal kana (uppercase character) to normalize the token.
### Why it is needed
This transformation must be done as token filter. There have already been [MappingCharFilter](https://lucene.apache.org/core/8_0_0/analyzers-common/org/apache/lucene/analysis/charfilter/MappingCharFilter.html), but if we apply this character filter to normalize sutegana, it will impact to tokenization and it is not expected.
This commit adds a new interface to all MergeScheduler classes that allows the scheduler to provide an Executor for intra-merge parallelism. The first sub-class to satisfy this new interface is the ConcurrentMergeScheduler (CMS). In particular, the CMS will limit the amount of parallelism based on the current number of mergeThreads and the configured maxThread options.
Parallelism is now added by default to the SegmentMerger where each merging task is executed in parallel with the others.
Additionally, the Lucene99 HNSW codec will use the provided MergeScheduler executor if no executor is provided directly.
Relates to: https://github.com/apache/lucene/issues/12740
Relates to: https://github.com/apache/lucene/issues/9626
The current logic for reordering splits a slice of doc IDs into a left side and
a right side, and for each document it computes the expected gain of moving to
the other side. Then it swaps documents from both sides as long as the sum of
the gain of moving the left doc to the right and the right doc to the left is
positive.
This works well, but I would like to extend BP reordering to also work with
blocks, and the swapping logic is challenging to modify as two parent documents
may have different numbers of children.
One of the follow-up papers on BP suggested to use a different logic, where one
would compute a bias for all documents that is negative when a document is
attracted to the left and positive otherwise. Then we only have to partition doc
IDs around the mid point, e.g. with quickselect.
A benefit of this change is that it will make it easier to generalize BP
reordering to indexes that have blocks, e.g. by using a stable sort on biases.
### Description
The boosted doc scores were the same due to floating point precision limitations, when multiplying the original scores with tiny differences with a fairly large multiplier.
My change removes the assertion expecting the exact order to be the same (AFAIU this won't be possible for every scores/boost configurations due to precision limitations), but rather asserts that the `original doc score * boost factor` and the `boosted doc score` are equals within a certain delta.
Closes https://github.com/apache/lucene/issues/13173.
This adds multi-leaf optimizations for diversified children collector. This means as children vectors are collected within a block join, we can share information between leaves to speed up vector search.
To make this happen, I refactored the multi-leaf collector slightly. Now, instead of inheriting from TopKnnCollector, we inject a inner collector.
Armin Braun
Adrien Grand
Kaival Parikh
Tommaso Teofili
Robert Muir
Trey Jones
Uwe Schindler
Jean-François BOEUF
Zhang Chao
gf2121
Chris Hegarty
Nhat Nguyen
Tim Grein
Benjamin Trent
Shintaro Murakami
panguixin
Dai Sugimori
Dawid Weiss
zhouhui
Tim Grein