Today we have duplicated logic on how to convert a seqNo into a real
seqNo and process events based on this. This change consolidated the logic
into a single method.
* LUCENE-8962: Simplify test case
The testMergeOnCommit test case was trying to verify too many things
at once: basic semantics of merge on commit and proper behavior when
a bunch of indexing threads are writing and committing all at once.
Now we just verify basic behavior, with strict assertions on invariants, while
leaving it to MockRandomMergePolicy to enable merge on commit in existing
test cases to verify that indexing generally works as expected and no new
unexpected exceptions are thrown.
* LUCENE-8962: Only update toCommit if merge was committed
The code was previously assuming that if mergeFinished() was called and
isAborted() was false, then the merge must have completed successfully.
Instead, we should know for sure if a given merge was committed, and
only then update our pending commit SegmentInfos.
This commit makes ValueSourceScorer's costing algorithm also take the delegated FunctionValues's cost into consideration when calculating its cost. FunctionValues now exposes a cost method which is used by ValueSourceScorer's default matchCost method. In addition, ValueSourceScorer exposes a matchCost method which can be overridden to specify a custom costing mechanism
1. TestIndexWriterMergePolicy.testMergeOnCommit will fail if the last
commit (the one that should trigger the full merge) doesn't have any
pending changes (which could occur if the last indexing thread
commits at the end). We can fix that by adding one more document
before that commit.
2. The previous implementation was throwing IOException if the commit
thread gets interrupted while waiting for merges to complete. This
violates IndexWriter's documented behavior of throwing
ThreadInterruptedException.
* LUCENE-8962: Add ability to selectively merge on commit
This adds a new "findCommitMerges" method to MergePolicy, which can
specify merges to be executed before the
IndexWriter.prepareCommitInternal method returns.
If we have many index writer threads, they will flush their DWPT buffers
on commit, resulting in many small segments, which can be merged before
the commit returns.
* Add missing Javadoc
* Fix incorrect comment
* Refactoring and fix intermittent test failure
1. Made some changes to the callback to update toCommit, leveraging
SegmentInfos.applyMergeChanges.
2. I realized that we'll never end up with 0 registered merges, because
we throw an exception if we fail to register a merge.
3. Moved the IndexWriterEvents.beginMergeOnCommit notification to before
we call MergeScheduler.merge, since we may not be merging on another
thread.
4. There was an intermittent test failure due to randomness in the time
it takes for merges to complete. Before doing the final commit, we wait
for pending merges to finish. We may still end up abandoning the final
merge, but we can detect that and assert that either the merge was
abandoned (and we have > 1 segment) or we did merge down to 1 segment.
* Fix typo
* Fix/improve comments based on PR feedback
* More comment improvements from PR feedback
* Rename method and add new MergeTrigger
1. Renamed findCommitMerges -> findFullFlushMerges.
2. Added MergeTrigger.COMMIT, passed to findFullFlushMerges and to
MergeScheduler when merging on commit.
* Update renamed method name in strings and comments
Jan Høydahl
Joel Bernstein
iverase
Simon Willnauer
Tomoko Uchida
Bruno Roustant
Nhat Nguyen
Michael Sokolov
Paul Pazderski
David Smiley
noble
Robert Muir
Mike McCandless
Yannick Welsch
Michael Sokolov
Cassandra Targett
Atri Sharma
Tomas Fernandez Lobbe
Mike Drob
Adrien Grand
Christine Poerschke
Noble Paul
Pushkar Raste
msfroh
Nicholas Knize
Namgyu Kim