With this change, segments are more likely to be considered for merging until
they reach the max merge size. Before this change, LogMergePolicy would exclude
an entire window of `mergeFactor` segments from merging if this window had a
too large segment and other segments were on the same tier.
* Try to fix the gradle compilation in idea
* Try to detect sync and build phases within intellij and act accordingly to support both modes of compilation (gradle and intellij).
This gives implementations of `findFullFlushMerges` to `LogMergePolicy` and
`TieredMergePolicy` and enables merge-on-refresh with a default timeout of
500ms.
The idea behind the 500ms default is that it felt both high-enough to have time
to run merges of small segments, and low enough that the freshness of the data
wouldn't look badly affected for users who have high refresh rates (e.g.
refreshing every second).
In both cases, `findFullFlushMerges` delegates to `findMerges` and filters
merges whose segments are all below the min/floor size.
* remove commented-out/obselete AwaitsFix
All of these issues are fixed, but the AwaitsFix annotation is still there, just commented out. This causes confusion and makes it harder to keep an eye/review the AwaitsFix tests, e.g. false positives when running 'git grep AwaitsFix'
* Remove @AwaitsFix from TestMatchRegionRetriever. The problem has been fixed in LUCENE-10229.
Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
This is very noisy, can contain gradle status updates, various other tests.verbose prints from other threads, you name it.
It causes the check to be flaky, and randomly "miss" seeing a test that executed.
Instead, let's look at the zip files. We can still preserve the essence of what the test wants to do, but without any flakiness.
- MergeOnFlushMergePolicy doesn't try to avoid O(n^2) merges, so I'm disabling
the test on it for now.
- TestUpgradeIndexMergePolicy would sometimes wrap with a non-standard merge
policy like the alcoholic merge policy, I forced it to wrap a
TieredMergePolicy.
LogDocMergePolicy would previously always force-merge an index that has 10
segments of size 1 to 10, due to the min doc count. This is not the case
anymore, but the test was assuming that such an index would get merged, so I
fixed the test's expectations.
Also changed the merge policy to keep working when RAM buffers are flushed in
such a way that segments do not appear in decreasing size order using the same
logic as LogMergePolicy.
If a LogByteSizeMergePolicy is used, then it might decide to not merge the two
one-document segments if their on-disk sizes are too different. Using a
LogDocMergePolicy addresses the issue as both segments are always considered
the same size.
This updates TieredMergePolicy and Log(Doc|Size)MergePolicy to only ever
consider merges where the resulting segment would be at least 50% bigger than
the biggest input segment. While a merge that only grows the biggest segment by
50% is still quite inefficient, this constraint is good enough to prevent
pathological O(N^2) merging.
- Removed dependency on LineFileDocs to improve reproducibility.
- Relaxed the expected exception type: any exception is ok.
- Ignore rare cases when a file still appears to have a well-formed footer
after truncation.
Rushabh Shah
Adrien Grand
Craig Taverner
Sun WuQiang
Dawid Weiss
Patrick Zhai
Tomoko Uchida
Lu Xugang
Michael Sokolov
tang donghai
Yannick Welsch
Greg Miller
András Salamon
Alan Woodward
Robert Muir
Shai Erera