Apache
/
lucene
mirror of https://github.com/apache/lucene.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

LUCENE-8370: Reproducing TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failures

This commit is contained in:
Erick Erickson 2018-06-28 18:28:37 -07:00
parent 38c33de24c
commit c303c5f126
5 changed files with 22 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
lucene/CHANGES.txt
View File

@ -125,6 +125,10 @@ Bug Fixes:
via LUCENE-8165 since we are now checking for offset+length going out of bounds.
(Robert Muir, Nhat Nyugen, Simon Willnauer)
* LUCENE-8370: Reproducing
TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
failures (Erick Erickson)
Changes in Runtime Behavior:
* LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing

19
lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
View File

@ -60,15 +60,22 @@ import java.util.Set;
* LogMergePolicy}.
*
* <p><b>NOTE</b>: This policy always merges by byte size
* of the segments, always pro-rates by percent deletes,
* and does not apply any maximum segment size during
* forceMerge (unlike {@link LogByteSizeMergePolicy}).
* of the segments, always pro-rates by percent deletes
*
* <p><b>NOTE</b> As of Lucene 7.5, forceMerge (aka optimize)
* and expungeDeletes (findForcedMerges and
* findForcedDeletesMerges) respect the max segment
* <p><b>NOTE</b> Starting with Lucene 7.5, there are several changes:
*
* - findForcedMerges and findForcedDeletesMerges) respect the max segment
* size by default.
*
* - When findforcedmerges is called with maxSegmentCount other than 1,
* the resulting index is not guaranteed to have &lt;= maxSegmentCount segments.
* Rather it is on a "best effort" basis. Specifically the theoretical ideal
* segment size is calculated and a "fudge factor" of 25% is added as the
* new maxSegmentSize, which is respected.
*
* - findForcedDeletesMerges will not produce segments greater than
* maxSegmentSize.
*
* @lucene.experimental
*/

4
lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java
View File

@ -369,7 +369,9 @@ public class RandomIndexWriter implements Closeable {
System.out.println("RIW: doRandomForceMerge(" + limit + ")");
}
w.forceMerge(limit);
assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit: "limit=" + limit + " actual=" + w.getSegmentCount();
if (limit == 1 || (w.getConfig().getMergePolicy() instanceof TieredMergePolicy) == false) {
assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit : "limit=" + limit + " actual=" + w.getSegmentCount();
}
} else {
if (LuceneTestCase.VERBOSE) {
System.out.println("RIW: do random forceMergeDeletes()");

2
solr/solr-ref-guide/src/solr-upgrade-notes.adoc
View File

@ -30,7 +30,7 @@ Detailed steps for upgrading a Solr cluster are in the section <<upgrading-a-sol
=== Solr 7.5
When upgrading to Solr 7.4, users should be aware of the following major changes from v7.3:
* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. maxSegments > 1 are honored on a "best effort" basis. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
=== Solr 7.4

2
solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc
View File

@ -100,7 +100,7 @@ Default is `true`. Blocks until a new searcher is opened and registered as the m
WARNING: expungeDeletes is "less expensive" than optimize, but the same warnings apply.
`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Merges the segments down to no more than this number of segments. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Makes a "best effort" attempt to merge the segments down to no more than this number of segments but does not guarantee that the goal will be achieved. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
Here are examples of <commit> and <optimize> using optional attributes: