Apache
/
lucene
mirror of https://github.com/apache/lucene.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add support for intra-segment search concurrency (#13542) 

This commit introduces support for optionally creating slices that target leaf reader context partitions, which allow them to be searched concurrently. This is good to maximize resource usage when searching force-merged indices, or indices with rather big segments, by parallelizig search execution across subsets of segments being searched.
    
Note: this commit does not affect default generation of slices. Segments can be partitioned by overriding the `IndexSearcher#slices(List<LeafReaderContext>)` method to plug in ad-hoc slices creation. Moreover, the existing  `IndexSearcher#slices` static method now creates segment partitions when the additional `allowSegmentsPartitions` argument is set to `true`.
  
The overall design of this change is based on the existing search concurrency support that is based on `LeafSlice` and `CollectorManager`. A new `LeafReaderContextPartition` abstraction is introduced, that holds a reference to a `LeafReaderContext` and the range of doc ids it targets. A `LeafSlice` noew targets segment partitions, each identified by a `LeafReaderContext` instance and a range of doc ids. It is possible for a partition to target a whole segment, and for partitions of different segments to be combined into the same leaf slices freely, hence searched by the same thread. It is not possible for multiple partitions of the same segment to be added to the same leaf slice.
    
Segment partitions are searched concurrently leveraging the existing `BulkScorer#score(LeafCollector collector, Bits acceptDocs, int min, int max)` method, that allows to score a specific subset of documents for a provided
 `LeafCollector`, in place of the `BulkScorer#score(LeafCollector collector, Bits acceptDocs)` that would instead score all documents.

## Changes that require migration

The migrate guide has the following new clarifying items around the contract and breaking changes required to support intra-segment concurrency:
- `Collector#getLeafCollector` may be called multiple times for the same leaf across distinct `Collector` instances created by a `CollectorManager`. Logic that relies on `getLeafCollector` being called once per leaf per search needs updating.
- a `Scorer`, `ScorerSupplier` or `BulkScorer` may be requested multiple times for the same leaf
- `IndexSearcher#searchLeaf` change of signature to accept the range of doc ids
- `BulkScorer#score(LeafCollector, BitSet)` is removed in favour of `BulkScorer#score(LeafCollector, BitSet, int, int)`
-  static `IndexSearcher#slices` method changed to take a last boolean argument that optionally enables the creation of segment partitions
- `TotalHitCountCollectorManager` now requires that an array of `LeafSlice`s, retrieved via `IndexSearcher#getSlices`, 
is provided to its constructor
 
    
Note: `DrillSideways` is the only component that does not support intra-segment concurrency and needs considerable work to do so, due to its requirement that the entire set of docs in a segment gets scored in one go.
    
The default searcher slicing is not affected by this PR, but `LuceneTestCase` now randomly leverages intra-segment concurrency. An additional `newSearcher` method is added that takes a `Concurrency` enum as the last argument in place of the `useThreads` boolean flag. This is important to disable intra-segment concurrency for `DrillSideways` related tests that do support inter-segment concurrency but not intra-segment concurrency.

## Next step

While this change introduces support for intra-segment concurrency, it only sets up the foundations of it. There is still a performance penalty for queries that require segment-level computation ahead of time, such as points/range queries. This is an implementation limitation that we expect to improve in future releases, see #13745.

Additionally, we will need to decide what to do about the lack of support for intra-segment concurrency in `DrillSideways` before we can enable intra-segment slicing by default. See #13753 .

Closes #9721
This commit is contained in:
Luca Cavanna 2024-09-10 23:22:39 +02:00 committed by GitHub
parent 942065cc3e
commit fafd6af004
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
37 changed files with 957 additions and 211 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
lucene/CHANGES.txt
View File

@ -151,6 +151,14 @@ New Features
* GITHUB#13592: Take advantage of the doc value skipper when it is primary sort in SortedNumericDocValuesRangeQuery
and SortedSetDocValuesRangeQuery. (Ignacio Vera)
* GITHUB#13542: Add initial support for intra-segment concurrency. IndexSearcher now supports searching across leaf
reader partitions concurrently. This is useful to max out available resource usage especially with force merged
indices or big segments. There is still a performance penalty for queries that require segment-level computation
ahead of time, such as points/range queries. This is an implementation limitation that we expect to improve in
future releases, ad that's why intra-segment slicing is not enabled by default, but leveraged in tests when the
searcher is created via LuceneTestCase#newSearcher. Users may override IndexSearcher#slices(List) to optionally
create slices that target segment partitions. (Luca Cavanna)
Improvements
---------------------

47
lucene/MIGRATE.md
View File

@ -816,5 +816,48 @@ both `TopDocs` as well as facets results included in a reduced `FacetsCollector`
### `SearchWithCollectorTask` no longer supports the `collector.class` config parameter
`collector.class` used to allow users to load a custom collector implementation. `collector.manager.class`
replaces it by allowing users to load a custom collector manager instead.
`collector.class` used to allow users to load a custom collector implementation. `collector.manager.class`
replaces it by allowing users to load a custom collector manager instead.
### BulkScorer#score(LeafCollector collector, Bits acceptDocs) removed
Use `BulkScorer#score(LeafCollector collector, Bits acceptDocs, int min, int max)` instead. In order to score the
entire leaf, provide `0` as min and `DocIdSetIterator.NO_MORE_DOCS` as max. `BulkScorer` subclasses that override
such method need to instead override the method variant that takes the range of doc ids as well as arguments.
### CollectorManager#newCollector and Collector#getLeafCollector contract
With the introduction of intra-segment query concurrency support, multiple `LeafCollector`s may be requested for the
same `LeafReaderContext` via `Collector#getLeafCollector(LeafReaderContext)` across the different `Collector` instances
returned by multiple `CollectorManager#newCollector` calls. Any logic or computation that needs to happen
once per segment requires specific handling in the collector manager implementation. See `TotalHitCountCollectorManager`
as an example. Individual collectors don't need to be adapted as a specific `Collector` instance will still see a given
`LeafReaderContext` once, given that it is not possible to add more than one partition of the same segment to the same
leaf slice.
### Weight#scorer, Weight#bulkScorer and Weight#scorerSupplier contract
With the introduction of intra-segment query concurrency support, multiple `Scorer`s, `ScorerSupplier`s or `BulkScorer`s
may be requested for the same `LeafReaderContext` instance as part of a single search call. That may happen concurrently
from separate threads each searching a specific doc id range of the segment. `Weight` implementations that rely on the
assumption that a scorer, bulk scorer or scorer supplier for a given `LeafReaderContext` is requested once per search
need updating.
### Signature of IndexSearcher#searchLeaf changed
With the introduction of intra-segment query concurrency support, the `IndexSearcher#searchLeaf(LeafReaderContext ctx, Weight weight, Collector collector)`
method now accepts two additional int arguments to identify the min/max range of doc ids that will be searched in this
leaf partition`: IndexSearcher#searchLeaf(LeafReaderContext ctx, int minDocId, int maxDocId, Weight weight, Collector collector)`.
Subclasses of `IndexSearcher` that call or override the `searchLeaf` method need to be updated accordingly.
### Signature of static IndexSearch#slices method changed
The static `IndexSearcher#sslices(List<LeafReaderContext> leaves, int maxDocsPerSlice, int maxSegmentsPerSlice)`
method now supports an additional 4th and last argument to optionally enable creating segment partitions:
`IndexSearcher#slices(List<LeafReaderContext> leaves, int maxDocsPerSlice, int maxSegmentsPerSlice, boolean allowSegmentPartitions)`
### TotalHitCountCollectorManager constructor
`TotalHitCountCollectorManager` now requires that an array of `LeafSlice`s, retrieved via `IndexSearcher#getSlices`,
is provided to its constructor. Depending on whether segment partitions are present among slices, the manager can
optimize the type of collectors it creates and exposes via `newCollector`.

12
lucene/core/src/java/org/apache/lucene/search/BulkScorer.java
View File

@ -27,18 +27,6 @@ import org.apache.lucene.util.Bits;
*/
public abstract class BulkScorer {
/**
* Scores and collects all matching documents.
*
* @param collector The collector to which all matching documents are passed.
* @param acceptDocs {@link Bits} that represents the allowed documents to match, or {@code null}
* if they are all allowed to match.
*/
public void score(LeafCollector collector, Bits acceptDocs) throws IOException {
final int next = score(collector, acceptDocs, 0, DocIdSetIterator.NO_MORE_DOCS);
assert next == DocIdSetIterator.NO_MORE_DOCS;
}
/**
* Collects matching documents in a range and return an estimation of the next matching document
* which is on or after {@code max}.

7
lucene/core/src/java/org/apache/lucene/search/CollectorManager.java
View File

@ -18,6 +18,7 @@ package org.apache.lucene.search;
import java.io.IOException;
import java.util.Collection;
import org.apache.lucene.index.LeafReaderContext;
/**
* A manager of collectors. This class is useful to parallelize execution of search requests and has
@ -31,6 +32,12 @@ import java.util.Collection;
* fully collected.
* </ul>
*
* <p><strong>Note:</strong> Multiple {@link LeafCollector}s may be requested for the same {@link
* LeafReaderContext} via {@link Collector#getLeafCollector(LeafReaderContext)} across the different
* {@link Collector}s returned by {@link #newCollector()}. Any computation or logic that needs to
* happen once per segment requires specific handling in the collector manager implementation,
* because the collection of an entire segment may be split across threads.
*
* @see IndexSearcher#search(Query, CollectorManager)
* @lucene.experimental
*/

263
lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
View File

@ -21,8 +21,10 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.Executor;
import java.util.function.Function;
@ -233,7 +235,13 @@ public class IndexSearcher {
? leaves ->
leaves.isEmpty()
? new LeafSlice[0]
: new LeafSlice[] {new LeafSlice(new ArrayList<>(leaves))}
: new LeafSlice[] {
new LeafSlice(
new ArrayList<>(
leaves.stream()
.map(LeafReaderContextPartition::createForEntireSegment)
.toList()))
}
: this::slices;
leafSlicesSupplier = new CachingLeafSlicesSupplier(slicesProvider, leafContexts);
}
@ -319,21 +327,97 @@ public class IndexSearcher {
/**
* Expert: Creates an array of leaf slices each holding a subset of the given leaves. Each {@link
* LeafSlice} is executed in a single thread. By default, segments with more than
* MAX_DOCS_PER_SLICE will get their own thread
* MAX_DOCS_PER_SLICE will get their own thread.
*
* <p>It is possible to leverage intra-segment concurrency by splitting segments into multiple
* partitions. Such behaviour is not enabled by default as there is still a performance penalty
* for queries that require segment-level computation ahead of time, such as points/range queries.
* This is an implementation limitation that we expect to improve in future releases, see <a
* href="https://github.com/apache/lucene/issues/13745">the corresponding github issue</a>.
*/
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, MAX_DOCS_PER_SLICE, MAX_SEGMENTS_PER_SLICE);
return slices(leaves, MAX_DOCS_PER_SLICE, MAX_SEGMENTS_PER_SLICE, false);
}
/** Static method to segregate LeafReaderContexts amongst multiple slices */
/**
* Static method to segregate LeafReaderContexts amongst multiple slices. Creates slices according
* to the provided max number of documents per slice and max number of segments per slice. Splits
* segments into partitions when the last argument is true.
*
* @param leaves the leaves to slice
* @param maxDocsPerSlice the maximum number of documents in a single slice
* @param maxSegmentsPerSlice the maximum number of segments in a single slice
* @param allowSegmentPartitions whether segments may be split into partitions according to the
* provided maxDocsPerSlice argument. When <code>true</code>, if a segment holds more
* documents than the provided max docs per slice, it is split into equal size partitions that
* each gets its own slice assigned.
* @return the array of slices
*/
public static LeafSlice[] slices(
List<LeafReaderContext> leaves, int maxDocsPerSlice, int maxSegmentsPerSlice) {
List<LeafReaderContext> leaves,
int maxDocsPerSlice,
int maxSegmentsPerSlice,
boolean allowSegmentPartitions) {
// Make a copy so we can sort:
List<LeafReaderContext> sortedLeaves = new ArrayList<>(leaves);
// Sort by maxDoc, descending:
Collections.sort(
sortedLeaves, Collections.reverseOrder(Comparator.comparingInt(l -> l.reader().maxDoc())));
sortedLeaves.sort(Collections.reverseOrder(Comparator.comparingInt(l -> l.reader().maxDoc())));
if (allowSegmentPartitions) {
final List<List<LeafReaderContextPartition>> groupedLeafPartitions = new ArrayList<>();
int currentSliceNumDocs = 0;
List<LeafReaderContextPartition> group = null;
for (LeafReaderContext ctx : sortedLeaves) {
if (ctx.reader().maxDoc() > maxDocsPerSlice) {
assert group == null;
// if the segment does not fit in a single slice, we split it into maximum 5 partitions of
// equal size
int numSlices = Math.min(5, Math.ceilDiv(ctx.reader().maxDoc(), maxDocsPerSlice));
int numDocs = ctx.reader().maxDoc() / numSlices;
int maxDocId = numDocs;
int minDocId = 0;
for (int i = 0; i < numSlices - 1; i++) {
groupedLeafPartitions.add(
Collections.singletonList(
LeafReaderContextPartition.createFromAndTo(ctx, minDocId, maxDocId)));
minDocId = maxDocId;
maxDocId += numDocs;
}
// the last slice gets all the remaining docs
groupedLeafPartitions.add(
Collections.singletonList(
LeafReaderContextPartition.createFromAndTo(
ctx, minDocId, ctx.reader().maxDoc())));
} else {
if (group == null) {
group = new ArrayList<>();
groupedLeafPartitions.add(group);
}
group.add(LeafReaderContextPartition.createForEntireSegment(ctx));
currentSliceNumDocs += ctx.reader().maxDoc();
// We only split a segment when it does not fit entirely in a slice. We don't partition
// the
// segment that makes the current slice (which holds multiple segments) go over
// maxDocsPerSlice. This means that a slice either contains multiple entire segments, or a
// single partition of a segment.
if (group.size() >= maxSegmentsPerSlice || currentSliceNumDocs > maxDocsPerSlice) {
group = null;
currentSliceNumDocs = 0;
}
}
}
LeafSlice[] slices = new LeafSlice[groupedLeafPartitions.size()];
int upto = 0;
for (List<LeafReaderContextPartition> currentGroup : groupedLeafPartitions) {
slices[upto] = new LeafSlice(currentGroup);
++upto;
}
return slices;
}
final List<List<LeafReaderContext>> groupedLeaves = new ArrayList<>();
long docSum = 0;
@ -363,7 +447,12 @@ public class IndexSearcher {
LeafSlice[] slices = new LeafSlice[groupedLeaves.size()];
int upto = 0;
for (List<LeafReaderContext> currentLeaf : groupedLeaves) {
slices[upto] = new LeafSlice(currentLeaf);
slices[upto] =
new LeafSlice(
new ArrayList<>(
currentLeaf.stream()
.map(LeafReaderContextPartition::createForEntireSegment)
.toList()));
++upto;
}
@ -441,7 +530,7 @@ public class IndexSearcher {
return countTerm1 + countTerm2 - count(queries[2]);
}
}
return search(new ConstantScoreQuery(query), new TotalHitCountCollectorManager());
return search(new ConstantScoreQuery(query), new TotalHitCountCollectorManager(getSlices()));
}
/**
@ -658,11 +747,11 @@ public class IndexSearcher {
}
final List<Callable<C>> listTasks = new ArrayList<>(leafSlices.length);
for (int i = 0; i < leafSlices.length; ++i) {
final LeafReaderContext[] leaves = leafSlices[i].leaves;
final LeafReaderContextPartition[] leaves = leafSlices[i].partitions;
final C collector = collectors.get(i);
listTasks.add(
() -> {
search(Arrays.asList(leaves), weight, collector);
search(leaves, weight, collector);
return collector;
});
}
@ -674,7 +763,32 @@ public class IndexSearcher {
/**
* Lower-level search API.
*
* <p>{@link #searchLeaf(LeafReaderContext, Weight, Collector)} is called for every leaf
* <p>{@link #searchLeaf(LeafReaderContext, int, int, Weight, Collector)} is called for every leaf
* partition. <br>
*
* <p>NOTE: this method executes the searches on all given leaf partitions exclusively. To search
* across all the searchers leaves use {@link #leafContexts}.
*
* @param partitions the leaf partitions to execute the searches on
* @param weight to match documents
* @param collector to receive hits
* @throws TooManyClauses If a query would exceed {@link IndexSearcher#getMaxClauseCount()}
* clauses.
*/
protected void search(LeafReaderContextPartition[] partitions, Weight weight, Collector collector)
throws IOException {
collector.setWeight(weight);
for (LeafReaderContextPartition partition : partitions) { // search each subreader partition
searchLeaf(partition.ctx, partition.minDocId, partition.maxDocId, weight, collector);
}
}
/**
* Lower-level search API.
*
* <p>{@link #searchLeaf(LeafReaderContext, int, int, Weight, Collector)} is called for every leaf
* partition. <br>
*
* <p>NOTE: this method executes the searches on all given leaves exclusively. To search across
@ -685,7 +799,11 @@ public class IndexSearcher {
* @param collector to receive hits
* @throws TooManyClauses If a query would exceed {@link IndexSearcher#getMaxClauseCount()}
* clauses.
* @deprecated in favour of {@link #search(LeafReaderContextPartition[], Weight, Collector)} that
* provides the same functionality while also supporting segments partitioning. Will be
* removed once the removal of the deprecated {@link #search(Query, Collector)} is completed.
*/
@Deprecated
protected void search(List<LeafReaderContext> leaves, Weight weight, Collector collector)
throws IOException {
@ -695,7 +813,7 @@ public class IndexSearcher {
// threaded...? the Collector could be sync'd?
// always use single thread:
for (LeafReaderContext ctx : leaves) { // search each subreader
searchLeaf(ctx, weight, collector);
searchLeaf(ctx, 0, DocIdSetIterator.NO_MORE_DOCS, weight, collector);
}
}
@ -705,12 +823,15 @@ public class IndexSearcher {
* <p>{@link LeafCollector#collect(int)} is called for every document. <br>
*
* @param ctx the leaf to execute the search against
* @param minDocId the lower bound of the doc id range to search
* @param maxDocId the upper bound of the doc id range to search
* @param weight to match document
* @param collector to receive hits
* @throws TooManyClauses If a query would exceed {@link IndexSearcher#getMaxClauseCount()}
* clauses.
*/
protected void searchLeaf(LeafReaderContext ctx, Weight weight, Collector collector)
protected void searchLeaf(
LeafReaderContext ctx, int minDocId, int maxDocId, Weight weight, Collector collector)
throws IOException {
final LeafCollector leafCollector;
try {
@ -730,7 +851,7 @@ public class IndexSearcher {
scorer = new TimeLimitingBulkScorer(scorer, queryTimeout);
}
try {
scorer.score(leafCollector, ctx.reader().getLiveDocs());
scorer.score(leafCollector, ctx.reader().getLiveDocs(), minDocId, maxDocId);
} catch (
@SuppressWarnings("unused")
CollectionTerminatedException e) {
@ -879,7 +1000,8 @@ public class IndexSearcher {
/**
* A class holding a subset of the {@link IndexSearcher}s leaf contexts to be executed within a
* single thread.
* single thread. A leaf slice holds references to one or more {@link LeafReaderContextPartition}
* instances. Each partition targets a specific doc id range of a {@link LeafReaderContext}.
*
* @lucene.experimental
*/
@ -890,11 +1012,95 @@ public class IndexSearcher {
*
* @lucene.experimental
*/
public final LeafReaderContext[] leaves;
public final LeafReaderContextPartition[] partitions;
public LeafSlice(List<LeafReaderContext> leavesList) {
Collections.sort(leavesList, Comparator.comparingInt(l -> l.docBase));
this.leaves = leavesList.toArray(new LeafReaderContext[0]);
private final int maxDocs;
public LeafSlice(List<LeafReaderContextPartition> leafReaderContextPartitions) {
Comparator<LeafReaderContextPartition> docBaseComparator =
Comparator.comparingInt(l -> l.ctx.docBase);
Comparator<LeafReaderContextPartition> minDocIdComparator =
Comparator.comparingInt(l -> l.minDocId);
leafReaderContextPartitions.sort(docBaseComparator.thenComparing(minDocIdComparator));
this.partitions = leafReaderContextPartitions.toArray(new LeafReaderContextPartition[0]);
this.maxDocs =
Arrays.stream(partitions)
.map(leafPartition -> leafPartition.maxDocs)
.reduce(Integer::sum)
.get();
}
/**
* Returns the total number of docs that a slice targets, by summing the number of docs that
* each of its leaf context partitions targets.
*/
public int getMaxDocs() {
return maxDocs;
}
}
/**
* Holds information about a specific leaf context and the corresponding range of doc ids to
* search within. Used to optionally search across partitions of the same segment concurrently.
*
* <p>A partition instance can be created via {@link #createForEntireSegment(LeafReaderContext)},
* in which case it will target the entire provided {@link LeafReaderContext}. A true partition of
* a segment can be created via {@link #createFromAndTo(LeafReaderContext, int, int)} providing
* the minimum doc id (including) to search as well as the max doc id (excluding).
*
* @lucene.experimental
*/
public static final class LeafReaderContextPartition {
public final int minDocId;
public final int maxDocId;
public final LeafReaderContext ctx;
// we keep track of maxDocs separately because we use NO_MORE_DOCS as upper bound when targeting
// the entire segment. We use this only in tests.
private final int maxDocs;
private LeafReaderContextPartition(
LeafReaderContext leafReaderContext, int minDocId, int maxDocId, int maxDocs) {
if (minDocId >= maxDocId) {
throw new IllegalArgumentException(
"minDocId is greater than or equal to maxDocId: ["
+ minDocId
+ "] > ["
+ maxDocId
+ "]");
}
if (minDocId < 0) {
throw new IllegalArgumentException("minDocId is lower than 0: [" + minDocId + "]");
}
if (minDocId >= leafReaderContext.reader().maxDoc()) {
throw new IllegalArgumentException(
"minDocId is greater than than maxDoc: ["
+ minDocId
+ "] > ["
+ leafReaderContext.reader().maxDoc()
+ "]");
}
this.ctx = leafReaderContext;
this.minDocId = minDocId;
this.maxDocId = maxDocId;
this.maxDocs = maxDocs;
}
/** Creates a partition of the provided leaf context that targets the entire segment */
public static LeafReaderContextPartition createForEntireSegment(LeafReaderContext ctx) {
return new LeafReaderContextPartition(
ctx, 0, DocIdSetIterator.NO_MORE_DOCS, ctx.reader().maxDoc());
}
/**
* Creates a partition of the provided leaf context that targets a subset of the entire segment,
* starting from and including the min doc id provided, until and not including the provided max
* doc id
*/
public static LeafReaderContextPartition createFromAndTo(
LeafReaderContext ctx, int minDocId, int maxDocId) {
assert maxDocId != DocIdSetIterator.NO_MORE_DOCS;
return new LeafReaderContextPartition(ctx, minDocId, maxDocId, maxDocId - minDocId);
}
}
@ -1024,6 +1230,23 @@ public class IndexSearcher {
leafSlices =
Objects.requireNonNull(
sliceProvider.apply(leaves), "slices computed by the provider is null");
/*
* Enforce that there aren't multiple leaf partitions within the same leaf slice pointing to the
* same leaf context. It is a requirement that {@link Collector#getLeafCollector(LeafReaderContext)}
* gets called once per leaf context. Also, it does not make sense to partition a segment to then search
* those partitions as part of the same slice, because the goal of partitioning is parallel searching
* which happens at the slice level.
*/
for (LeafSlice leafSlice : leafSlices) {
Set<LeafReaderContext> distinctLeaves = new HashSet<>();
for (LeafReaderContextPartition leafPartition : leafSlice.partitions) {
distinctLeaves.add(leafPartition.ctx);
}
if (leafSlice.partitions.length != distinctLeaves.size()) {
throw new IllegalStateException(
"The same slice targets multiple leaf partitions of the same leaf reader context. A physical segment should rather get partitioned to be searched concurrently from as many slices as the number of leaf partitions it is split into.");
}
}
}
}
}

8
lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java
View File

@ -526,7 +526,9 @@ public class LRUQueryCache implements QueryCache, Accountable {
bitSet.set(doc);
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
return new CacheAndCount(new BitDocIdSet(bitSet, count[0]), count[0]);
}
@ -544,7 +546,9 @@ public class LRUQueryCache implements QueryCache, Accountable {
builder.add(doc);
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
RoaringDocIdSet cache = builder.build();
return new CacheAndCount(cache, cache.cardinality());
}

8
lucene/core/src/java/org/apache/lucene/search/TotalHitCountCollector.java
View File

@ -50,13 +50,17 @@ public class TotalHitCountCollector implements Collector {
totalHits += leafCount;
throw new CollectionTerminatedException();
}
return createLeafCollector();
}
protected final LeafCollector createLeafCollector() {
return new LeafCollector() {
@Override
public void setScorer(Scorable scorer) throws IOException {}
public void setScorer(Scorable scorer) {}
@Override
public void collect(int doc) throws IOException {
public void collect(int doc) {
totalHits++;
}

102
lucene/core/src/java/org/apache/lucene/search/TotalHitCountCollectorManager.java
View File

@ -18,6 +18,13 @@ package org.apache.lucene.search;
import java.io.IOException;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.util.ThreadInterruptedException;
/**
* Collector manager based on {@link TotalHitCountCollector} that allows users to parallelize
@ -28,17 +35,112 @@ import java.util.Collection;
*/
public class TotalHitCountCollectorManager
implements CollectorManager<TotalHitCountCollector, Integer> {
private final boolean hasSegmentPartitions;
/**
* Creates a new total hit count collector manager, providing the array of leaf slices that search
* targets, which can be retrieved via {@link IndexSearcher#getSlices()} for the searcher.
*
* @param leafSlices the slices that the searcher targets. Used to optimize the collection
* depending on whether segments have been partitioned into partitions or not.
*/
public TotalHitCountCollectorManager(IndexSearcher.LeafSlice[] leafSlices) {
this.hasSegmentPartitions = hasSegmentPartitions(leafSlices);
}
private static boolean hasSegmentPartitions(IndexSearcher.LeafSlice[] leafSlices) {
for (IndexSearcher.LeafSlice leafSlice : leafSlices) {
for (IndexSearcher.LeafReaderContextPartition leafPartition : leafSlice.partitions) {
if (leafPartition.minDocId > 0
|| leafPartition.maxDocId < leafPartition.ctx.reader().maxDoc()) {
return true;
}
}
}
return false;
}
/**
* Internal state shared across the different collectors that this collector manager creates. This
* is necessary to support intra-segment concurrency. We track leaves seen as an argument of
* {@link Collector#getLeafCollector(LeafReaderContext)} calls, to ensure correctness: if the
* first partition of a segment early terminates, count has been already retrieved for the entire
* segment hence subsequent partitions of the same segment should also early terminate without
* further incrementing hit count. If the first partition of a segment computes hit counts,
* subsequent partitions of the same segment should do the same, to prevent their counts from
* being retrieved from {@link LRUQueryCache} (which returns counts for the entire segment while
* we'd need only that of the current leaf partition).
*/
private final Map<Object, Future<Boolean>> earlyTerminatedMap = new ConcurrentHashMap<>();
@Override
public TotalHitCountCollector newCollector() throws IOException {
if (hasSegmentPartitions) {
return new LeafPartitionAwareTotalHitCountCollector(earlyTerminatedMap);
}
return new TotalHitCountCollector();
}
@Override
public Integer reduce(Collection<TotalHitCountCollector> collectors) throws IOException {
// Make the same collector manager instance reusable across multiple searches. It isn't a strict
// requirement but it is generally supported as collector managers normally don't hold state, as
// opposed to collectors.
assert hasSegmentPartitions || earlyTerminatedMap.isEmpty();
if (hasSegmentPartitions) {
earlyTerminatedMap.clear();
}
int totalHits = 0;
for (TotalHitCountCollector collector : collectors) {
totalHits += collector.getTotalHits();
}
return totalHits;
}
private static class LeafPartitionAwareTotalHitCountCollector extends TotalHitCountCollector {
private final Map<Object, Future<Boolean>> earlyTerminatedMap;
LeafPartitionAwareTotalHitCountCollector(Map<Object, Future<Boolean>> earlyTerminatedMap) {
this.earlyTerminatedMap = earlyTerminatedMap;
}
@Override
public LeafCollector getLeafCollector(LeafReaderContext context) throws IOException {
Future<Boolean> earlyTerminated = earlyTerminatedMap.get(context.id());
if (earlyTerminated == null) {
CompletableFuture<Boolean> firstEarlyTerminated = new CompletableFuture<>();
Future<Boolean> previousEarlyTerminated =
earlyTerminatedMap.putIfAbsent(context.id(), firstEarlyTerminated);
if (previousEarlyTerminated == null) {
// first thread for a given leaf gets to decide what the next threads targeting the same
// leaf do
try {
LeafCollector leafCollector = super.getLeafCollector(context);
firstEarlyTerminated.complete(false);
return leafCollector;
} catch (CollectionTerminatedException e) {
firstEarlyTerminated.complete(true);
throw e;
}
}
earlyTerminated = previousEarlyTerminated;
}
try {
if (earlyTerminated.get()) {
// first partition of the same leaf early terminated, do the same for subsequent ones
throw new CollectionTerminatedException();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new ThreadInterruptedException(e);
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
// first partition of the same leaf computed hit counts, do the same for subsequent ones
return createLeafCollector();
}
}
}

9
lucene/core/src/java/org/apache/lucene/search/Weight.java
View File

@ -113,7 +113,8 @@ public abstract class Weight implements SegmentCacheable {
* Optional method that delegates to scorerSupplier.
*
* <p>Returns a {@link Scorer} which can iterate in order over all matching documents and assign
* them a score.
* them a score. A scorer for the same {@link LeafReaderContext} instance may be requested
* multiple times as part of a single search call.
*
* <p><b>NOTE:</b> null can be returned if no documents will be scored by this query.
*
@ -135,7 +136,8 @@ public abstract class Weight implements SegmentCacheable {
/**
* Get a {@link ScorerSupplier}, which allows knowing the cost of the {@link Scorer} before
* building it.
* building it. A scorer supplier for the same {@link LeafReaderContext} instance may be requested
* multiple times as part of a single search call.
*
* <p><strong>Note:</strong> It must return null if the scorer is null.
*
@ -161,6 +163,9 @@ public abstract class Weight implements SegmentCacheable {
* scorerSupplier.setTopLevelScoringClause();
* return scorerSupplier.bulkScorer();
* </pre>
*
* A bulk scorer for the same {@link LeafReaderContext} instance may be requested multiple times
* as part of a single search call.
*/
public final BulkScorer bulkScorer(LeafReaderContext context) throws IOException {
ScorerSupplier scorerSupplier = scorerSupplier(context);

7
lucene/core/src/test/org/apache/lucene/index/TestForTooMuchCloning.java
View File

@ -78,9 +78,14 @@ public class TestForTooMuchCloning extends LuceneTestCase {
assertTrue(hits.totalHits.value > 0);
final int queryCloneCount = dir.getInputCloneCount() - cloneCount;
// System.out.println("query clone count=" + queryCloneCount);
// It is rather difficult to reliably predict how many query clone calls will be performed. One
// important factor is the number of segment partitions being searched, but it depends as well
// on the terms being indexed, and the distribution of the matches across the documents, which
// affects how the query gets rewritten and the subsequent number of clone calls it will
// perform.
assertTrue(
"too many calls to IndexInput.clone during TermRangeQuery: " + queryCloneCount,
queryCloneCount < 50);
queryCloneCount <= Math.max(s.getLeafContexts().size(), s.getSlices().length) * 5);
r.close();
dir.close();
}

334
lucene/core/src/test/org/apache/lucene/index/TestSegmentToThreadMapping.java
View File

@ -17,29 +17,24 @@
package org.apache.lucene.index;
import com.carrotsearch.randomizedtesting.RandomizedTest;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.apache.lucene.document.Document;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.KnnCollector;
import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.index.RandomIndexWriter;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.Bits;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.NamedThreadFactory;
import org.apache.lucene.util.Version;
public class TestSegmentToThreadMapping extends LuceneTestCase {
public LeafReader dummyIndexReader(final int maxDoc) {
private static LeafReader dummyIndexReader(final int maxDoc) {
return new LeafReader() {
@Override
public int maxDoc() {
@ -160,83 +155,230 @@ public class TestSegmentToThreadMapping extends LuceneTestCase {
};
}
public void testSingleSlice() {
LeafReader largeSegmentReader = dummyIndexReader(50_000);
LeafReader firstMediumSegmentReader = dummyIndexReader(30_000);
LeafReader secondMediumSegmentReader = dummyIndexReader(30__000);
LeafReader thirdMediumSegmentReader = dummyIndexReader(30_000);
private static List<LeafReaderContext> createLeafReaderContexts(int... maxDocs) {
List<LeafReaderContext> leafReaderContexts = new ArrayList<>();
for (int maxDoc : maxDocs) {
leafReaderContexts.add(new LeafReaderContext(dummyIndexReader(maxDoc)));
}
Collections.shuffle(leafReaderContexts, random());
return leafReaderContexts;
}
leafReaderContexts.add(new LeafReaderContext(largeSegmentReader));
leafReaderContexts.add(new LeafReaderContext(firstMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(secondMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(thirdMediumSegmentReader));
public void testSingleSlice() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(50_000, 30_000, 30_000, 30_000);
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(
leafReaderContexts, 250_000, RandomizedTest.randomIntBetween(4, 10), false);
assertEquals(1, resultSlices.length);
assertEquals(4, resultSlices[0].partitions.length);
}
IndexSearcher.LeafSlice[] resultSlices = IndexSearcher.slices(leafReaderContexts, 250_000, 5);
public void testSingleSliceWithPartitions() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(50_000, 30_000, 30_000, 30_000);
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(
leafReaderContexts, 250_000, RandomizedTest.randomIntBetween(4, 10), true);
assertEquals(1, resultSlices.length);
assertEquals(4, resultSlices[0].partitions.length);
}
assertTrue(resultSlices.length == 1);
public void testMaxSegmentsPerSlice() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(50_000, 30_000, 30_000, 30_000);
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 3, false);
assertEquals(2, resultSlices.length);
assertEquals(3, resultSlices[0].partitions.length);
assertEquals(110_000, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(30_000, resultSlices[1].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 2, false);
assertEquals(2, resultSlices.length);
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(80_000, resultSlices[0].getMaxDocs());
assertEquals(2, resultSlices[1].partitions.length);
assertEquals(60_000, resultSlices[1].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 1, false);
assertEquals(4, resultSlices.length);
assertEquals(1, resultSlices[0].partitions.length);
assertEquals(50_000, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(30_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(30_000, resultSlices[2].getMaxDocs());
assertEquals(1, resultSlices[3].partitions.length);
assertEquals(30_000, resultSlices[3].getMaxDocs());
}
}
final LeafReaderContext[] leaves = resultSlices[0].leaves;
assertTrue(leaves.length == 4);
public void testMaxSegmentsPerSliceWithPartitions() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(50_000, 30_000, 30_000, 30_000);
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 3, true);
assertEquals(2, resultSlices.length);
assertEquals(3, resultSlices[0].partitions.length);
assertEquals(110_000, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(30_000, resultSlices[1].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 2, true);
assertEquals(2, resultSlices.length);
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(80_000, resultSlices[0].getMaxDocs());
assertEquals(2, resultSlices[1].partitions.length);
assertEquals(60_000, resultSlices[1].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 1, true);
assertEquals(4, resultSlices.length);
assertEquals(1, resultSlices[0].partitions.length);
assertEquals(50_000, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(30_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(30_000, resultSlices[2].getMaxDocs());
assertEquals(1, resultSlices[3].partitions.length);
assertEquals(30_000, resultSlices[3].getMaxDocs());
}
}
public void testSmallSegments() {
LeafReader firstMediumSegmentReader = dummyIndexReader(10_000);
LeafReader secondMediumSegmentReader = dummyIndexReader(10_000);
LeafReader thirdMediumSegmentReader = dummyIndexReader(10_000);
LeafReader fourthMediumSegmentReader = dummyIndexReader(10_000);
LeafReader fifthMediumSegmentReader = dummyIndexReader(10_000);
LeafReader sixthMediumSegmentReader = dummyIndexReader(10_000);
LeafReader seventhLargeSegmentReader = dummyIndexReader(130_000);
LeafReader eigthLargeSegmentReader = dummyIndexReader(130_000);
List<LeafReaderContext> leafReaderContexts = new ArrayList<>();
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(10_000, 10_000, 10_000, 10_000, 10_000, 10_000, 130_000, 130_000);
leafReaderContexts.add(new LeafReaderContext(firstMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(secondMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(thirdMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(fourthMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(fifthMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(sixthMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(seventhLargeSegmentReader));
leafReaderContexts.add(new LeafReaderContext(eigthLargeSegmentReader));
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 5, false);
assertEquals(3, resultSlices.length);
IndexSearcher.LeafSlice[] resultSlices = IndexSearcher.slices(leafReaderContexts, 250_000, 5);
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(260_000, resultSlices[0].getMaxDocs());
assertEquals(5, resultSlices[1].partitions.length);
assertEquals(50_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(10_000, resultSlices[2].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 130_000, 5, false);
assertEquals(3, resultSlices.length);
// this is odd, because we allow two segments in the same slice with both size ==
// maxDocsPerSlice
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(260_000, resultSlices[0].getMaxDocs());
assertEquals(5, resultSlices[1].partitions.length);
assertEquals(50_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(10_000, resultSlices[2].getMaxDocs());
}
}
assertTrue(resultSlices.length == 3);
public void testSmallSegmentsWithPartitions() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(10_000, 10_000, 10_000, 10_000, 10_000, 10_000, 130_000, 130_000);
final LeafReaderContext[] firstSliceleaves = resultSlices[0].leaves;
final LeafReaderContext[] secondSliceleaves = resultSlices[1].leaves;
final LeafReaderContext[] thirdSliceleaves = resultSlices[2].leaves;
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 5, true);
assertEquals(3, resultSlices.length);
assertTrue(firstSliceleaves.length == 2);
assertTrue(secondSliceleaves.length == 5);
assertTrue(thirdSliceleaves.length == 1);
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(260_000, resultSlices[0].getMaxDocs());
assertEquals(5, resultSlices[1].partitions.length);
assertEquals(50_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(10_000, resultSlices[2].getMaxDocs());
}
{
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 130_000, 5, true);
assertEquals(3, resultSlices.length);
// this is odd, because we allow two segments in the same slice with both size ==
// maxDocsPerSlice
assertEquals(2, resultSlices[0].partitions.length);
assertEquals(260_000, resultSlices[0].getMaxDocs());
assertEquals(5, resultSlices[1].partitions.length);
assertEquals(50_000, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(10_000, resultSlices[2].getMaxDocs());
}
}
public void testLargeSlices() {
LeafReader largeSegmentReader = dummyIndexReader(290_900);
LeafReader firstMediumSegmentReader = dummyIndexReader(170_000);
LeafReader secondMediumSegmentReader = dummyIndexReader(170_000);
LeafReader thirdMediumSegmentReader = dummyIndexReader(170_000);
List<LeafReaderContext> leafReaderContexts = new ArrayList<>();
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(290_900, 170_000, 170_000, 170_000);
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 5, false);
leafReaderContexts.add(new LeafReaderContext(largeSegmentReader));
leafReaderContexts.add(new LeafReaderContext(firstMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(secondMediumSegmentReader));
leafReaderContexts.add(new LeafReaderContext(thirdMediumSegmentReader));
assertEquals(3, resultSlices.length);
assertEquals(1, resultSlices[0].partitions.length);
assertEquals(2, resultSlices[1].partitions.length);
assertEquals(1, resultSlices[2].partitions.length);
}
IndexSearcher.LeafSlice[] resultSlices = IndexSearcher.slices(leafReaderContexts, 250_000, 5);
public void testLargeSlicesWithPartitions() {
List<LeafReaderContext> leafReaderContexts =
createLeafReaderContexts(290_900, 170_000, 170_000, 170_000);
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(
leafReaderContexts, 250_000, RandomizedTest.randomIntBetween(5, 10), true);
assertTrue(resultSlices.length == 3);
assertEquals(4, resultSlices.length);
assertEquals(1, resultSlices[0].partitions.length);
assertEquals(145_450, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(145_450, resultSlices[1].getMaxDocs());
assertEquals(2, resultSlices[2].partitions.length);
assertEquals(340_000, resultSlices[2].getMaxDocs());
assertEquals(1, resultSlices[3].partitions.length);
assertEquals(170_000, resultSlices[3].getMaxDocs());
}
final LeafReaderContext[] firstSliceleaves = resultSlices[0].leaves;
final LeafReaderContext[] secondSliceleaves = resultSlices[1].leaves;
final LeafReaderContext[] thirdSliceleaves = resultSlices[2].leaves;
public void testSingleSegmentPartitions() {
List<LeafReaderContext> leafReaderContexts = createLeafReaderContexts(750_001);
IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(
leafReaderContexts, 250_000, RandomizedTest.randomIntBetween(1, 10), true);
assertTrue(firstSliceleaves.length == 1);
assertTrue(secondSliceleaves.length == 2);
assertTrue(thirdSliceleaves.length == 1);
assertEquals(4, resultSlices.length);
assertEquals(1, resultSlices[0].partitions.length);
assertEquals(187_500, resultSlices[0].getMaxDocs());
assertEquals(1, resultSlices[1].partitions.length);
assertEquals(187_500, resultSlices[1].getMaxDocs());
assertEquals(1, resultSlices[2].partitions.length);
assertEquals(187_500, resultSlices[2].getMaxDocs());
assertEquals(1, resultSlices[3].partitions.length);
assertEquals(187_501, resultSlices[3].getMaxDocs());
}
public void testExtremeSegmentsPartitioning() {
List<LeafReaderContext> leafReaderContexts = createLeafReaderContexts(2, 5, 10);
IndexSearcher.LeafSlice[] resultSlices = IndexSearcher.slices(leafReaderContexts, 1, 1, true);
assertEquals(12, resultSlices.length);
int i = 0;
for (IndexSearcher.LeafSlice leafSlice : resultSlices) {
if (i++ > 4) {
assertEquals(1, leafSlice.getMaxDocs());
} else {
assertEquals(2, leafSlice.getMaxDocs());
}
assertEquals(1, leafSlice.partitions.length);
}
}
public void testIntraSliceDocIDOrder() throws Exception {
@ -251,33 +393,54 @@ public class TestSegmentToThreadMapping extends LuceneTestCase {
IndexReader r = w.getReader();
w.close();
ExecutorService service =
new ThreadPoolExecutor(
4,
4,
0L,
TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
new NamedThreadFactory("TestSegmentToThreadMapping"));
IndexSearcher s = new IndexSearcher(r, service);
Query query = new MatchAllDocsQuery();
s.search(query, Integer.MAX_VALUE);
IndexSearcher s = new IndexSearcher(r, command -> {});
IndexSearcher.LeafSlice[] slices = s.getSlices();
assertNotNull(slices);
for (IndexSearcher.LeafSlice leafSlice : slices) {
LeafReaderContext[] leafReaderContexts = leafSlice.leaves;
int previousDocBase = leafReaderContexts[0].docBase;
int previousDocBase = leafSlice.partitions[0].ctx.docBase;
for (LeafReaderContext leafReaderContext : leafReaderContexts) {
assertTrue(previousDocBase <= leafReaderContext.docBase);
previousDocBase = leafReaderContext.docBase;
for (IndexSearcher.LeafReaderContextPartition leafReaderContextPartition :
leafSlice.partitions) {
assertTrue(previousDocBase <= leafReaderContextPartition.ctx.docBase);
previousDocBase = leafReaderContextPartition.ctx.docBase;
}
}
IOUtils.close(r, dir);
}
service.shutdown();
public void testIntraSliceDocIDOrderWithPartitions() throws Exception {
Directory dir = newDirectory();
RandomIndexWriter w = new RandomIndexWriter(random(), dir);
w.addDocument(new Document());
w.addDocument(new Document());
w.commit();
w.addDocument(new Document());
w.addDocument(new Document());
w.commit();
IndexReader r = w.getReader();
w.close();
IndexSearcher s =
new IndexSearcher(r, command -> {}) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
// force partitioning of segment with max docs per slice set to 1: 1 doc per partition.
return slices(leaves, 1, 1, true);
}
};
IndexSearcher.LeafSlice[] slices = s.getSlices();
assertNotNull(slices);
for (IndexSearcher.LeafSlice leafSlice : slices) {
int previousDocBase = leafSlice.partitions[0].ctx.docBase;
for (IndexSearcher.LeafReaderContextPartition leafReaderContextPartition :
leafSlice.partitions) {
assertTrue(previousDocBase <= leafReaderContextPartition.ctx.docBase);
previousDocBase = leafReaderContextPartition.ctx.docBase;
}
}
IOUtils.close(r, dir);
}
@ -291,9 +454,8 @@ public class TestSegmentToThreadMapping extends LuceneTestCase {
leafReaderContexts.add(
new LeafReaderContext(dummyIndexReader(random().nextInt((max - min) + 1) + min)));
}
IndexSearcher.LeafSlice[] resultSlices = IndexSearcher.slices(leafReaderContexts, 250_000, 5);
final IndexSearcher.LeafSlice[] resultSlices =
IndexSearcher.slices(leafReaderContexts, 250_000, 5, random().nextBoolean());
assertTrue(resultSlices.length > 0);
}
}

5
lucene/core/src/test/org/apache/lucene/search/TestBlockMaxConjunction.java
View File

@ -66,7 +66,10 @@ public class TestBlockMaxConjunction extends LuceneTestCase {
}
IndexReader reader = DirectoryReader.open(w);
w.close();
IndexSearcher searcher = newSearcher(reader);
// Disable search concurrency for this test: it requires a single segment, and no intra-segment
// concurrency for its assertions to always be valid
IndexSearcher searcher =
newSearcher(reader, random().nextBoolean(), random().nextBoolean(), false);
for (int iter = 0; iter < 100; ++iter) {
int start = random().nextInt(10);

4
lucene/core/src/test/org/apache/lucene/search/TestBooleanOr.java
View File

@ -259,7 +259,9 @@ public class TestBooleanOr extends LuceneTestCase {
matches.add(doc);
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
assertEquals(Arrays.asList(4000, 5000, 100000, 1000001, 1000051, 9999998, 9999999), matches);
}
}

40
lucene/core/src/test/org/apache/lucene/search/TestIndexSearcher.java
View File

@ -19,6 +19,7 @@ package org.apache.lucene.search;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Random;
import java.util.concurrent.ExecutorService;
@ -246,7 +247,7 @@ public class TestIndexSearcher extends LuceneTestCase {
// without executor
IndexSearcher.LeafSlice[] slices = new IndexSearcher(r).getSlices();
assertEquals(1, slices.length);
assertEquals(r.leaves().size(), slices[0].leaves.length);
assertEquals(r.leaves().size(), slices[0].partitions.length);
}
{
// force creation of multiple slices, and provide an executor
@ -254,12 +255,12 @@ public class TestIndexSearcher extends LuceneTestCase {
new IndexSearcher(r, Runnable::run) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, 1, 1);
return slices(leaves, 1, 1, false);
}
};
IndexSearcher.LeafSlice[] slices = searcher.getSlices();
for (IndexSearcher.LeafSlice slice : slices) {
assertEquals(1, slice.leaves.length);
assertEquals(1, slice.partitions.length);
}
assertEquals(r.leaves().size(), slices.length);
}
@ -280,7 +281,10 @@ public class TestIndexSearcher extends LuceneTestCase {
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
ArrayList<LeafSlice> slices = new ArrayList<>();
for (LeafReaderContext ctx : leaves) {
slices.add(new LeafSlice(Arrays.asList(ctx)));
slices.add(
new LeafSlice(
Collections.singletonList(
LeafReaderContextPartition.createForEntireSegment(ctx))));
}
return slices.toArray(new LeafSlice[0]);
}
@ -293,4 +297,32 @@ public class TestIndexSearcher extends LuceneTestCase {
IndexSearcher indexSearcher = new IndexSearcher(reader);
assertNotNull(indexSearcher.getTaskExecutor());
}
public void testSegmentPartitionsSameSlice() {
IndexSearcher indexSearcher =
new IndexSearcher(reader, Runnable::run) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
List<LeafSlice> slices = new ArrayList<>();
for (LeafReaderContext ctx : leaves) {
slices.add(
new LeafSlice(
new ArrayList<>(
List.of(
LeafReaderContextPartition.createFromAndTo(ctx, 0, 1),
LeafReaderContextPartition.createFromAndTo(
ctx, 1, ctx.reader().maxDoc())))));
}
return slices.toArray(new LeafSlice[0]);
}
};
assumeTrue(
"Needs at least 2 docs in the same segment",
indexSearcher.leafContexts.stream().allMatch(ctx -> ctx.reader().maxDoc() > 1));
IllegalStateException e = expectThrows(IllegalStateException.class, indexSearcher::getSlices);
assertEquals(
"The same slice targets multiple leaf partitions of the same leaf reader context. A physical segment should rather get partitioned to be searched concurrently from as many slices as the number of leaf partitions it is split into.",
e.getMessage());
}
}

18
lucene/core/src/test/org/apache/lucene/search/TestMaxScoreBulkScorer.java
View File

@ -146,7 +146,9 @@ public class TestMaxScoreBulkScorer extends LuceneTestCase {
}
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
}
}
}
@ -211,7 +213,9 @@ public class TestMaxScoreBulkScorer extends LuceneTestCase {
}
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
}
}
}
@ -289,7 +293,9 @@ public class TestMaxScoreBulkScorer extends LuceneTestCase {
}
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
}
}
}
@ -367,7 +373,9 @@ public class TestMaxScoreBulkScorer extends LuceneTestCase {
}
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
}
}
}
@ -503,7 +511,7 @@ public class TestMaxScoreBulkScorer extends LuceneTestCase {
assertEquals(1, i);
}
};
scorer.score(collector, liveDocs);
scorer.score(collector, liveDocs, 0, DocIdSetIterator.NO_MORE_DOCS);
collector.finish();
}

4
lucene/core/src/test/org/apache/lucene/search/TestReqExclBulkScorer.java
View File

@ -103,7 +103,9 @@ public class TestReqExclBulkScorer extends LuceneTestCase {
actualMatches.set(doc);
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
} else {
int next = 0;
while (next < maxDoc) {

46
lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java
View File

@ -17,14 +17,17 @@
package org.apache.lucene.search;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.NumericDocValuesField;
import org.apache.lucene.document.SortedDocValuesField;
@ -238,6 +241,7 @@ public class TestSortRandom extends LuceneTestCase {
private final List<BytesRef> docValues;
public final List<BytesRef> matchValues =
Collections.synchronizedList(new ArrayList<BytesRef>());
private final Map<LeafReaderContext, FixedBitSet> bitsets = new ConcurrentHashMap<>();
// density should be 0.0 ... 1.0
public RandomQuery(long seed, float density, List<BytesRef> docValues) {
@ -252,20 +256,34 @@ public class TestSortRandom extends LuceneTestCase {
return new ConstantScoreWeight(this, boost) {
@Override
public ScorerSupplier scorerSupplier(LeafReaderContext context) throws IOException {
Random random = new Random(context.docBase ^ seed);
final int maxDoc = context.reader().maxDoc();
final NumericDocValues idSource = DocValues.getNumeric(context.reader(), "id");
assertNotNull(idSource);
final FixedBitSet bits = new FixedBitSet(maxDoc);
for (int docID = 0; docID < maxDoc; docID++) {
assertEquals(docID, idSource.nextDoc());
if (random.nextFloat() <= density) {
bits.set(docID);
// System.out.println(" acc id=" + idSource.getInt(docID) + " docID=" + docID);
matchValues.add(docValues.get((int) idSource.longValue()));
}
}
FixedBitSet bits =
bitsets.computeIfAbsent(
context,
ctx -> {
Random random = new Random(context.docBase ^ seed);
final int maxDoc = context.reader().maxDoc();
try {
final NumericDocValues idSource =
DocValues.getNumeric(context.reader(), "id");
assertNotNull(idSource);
final FixedBitSet bitset = new FixedBitSet(maxDoc);
for (int docID = 0; docID < maxDoc; docID++) {
assertEquals(docID, idSource.nextDoc());
if (random.nextFloat() <= density) {
bitset.set(docID);
// System.out.println(" acc id=" + idSource.getInt(docID) + " docID=" +
// docID);
matchValues.add(docValues.get((int) idSource.longValue()));
}
}
return bitset;
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
// The bitset is built for the whole segment, the first time each leaf is seen. Every
// partition iterates through its own set of doc ids, using a separate iterator backed by
// the shared bitset.
final var scorer =
new ConstantScoreScorer(
score(), scoreMode, new BitSetIterator(bits, bits.approximateCardinality()));

4
lucene/core/src/test/org/apache/lucene/search/TestSynonymQuery.java
View File

@ -165,7 +165,7 @@ public class TestSynonymQuery extends LuceneTestCase {
}
public void testScores() throws IOException {
doTestScores(2);
doTestScores(1);
doTestScores(Integer.MAX_VALUE);
}
@ -211,7 +211,7 @@ public class TestSynonymQuery extends LuceneTestCase {
}
public void testBoosts() throws IOException {
doTestBoosts(2);
doTestBoosts(1);
doTestBoosts(Integer.MAX_VALUE);
}

4
lucene/core/src/test/org/apache/lucene/search/TestTaskExecutor.java
View File

@ -132,7 +132,7 @@ public class TestTaskExecutor extends LuceneTestCase {
new IndexSearcher(reader, executor) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, 1, 1);
return slices(leaves, 1, 1, false);
}
};
@ -206,7 +206,7 @@ public class TestTaskExecutor extends LuceneTestCase {
new IndexSearcher(reader, executor) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, 1, 1);
return slices(leaves, 1, 1, false);
}
};

4
lucene/core/src/test/org/apache/lucene/search/TestTermScorer.java
View File

@ -123,7 +123,9 @@ public class TestTermScorer extends LuceneTestCase {
return ScoreMode.COMPLETE;
}
},
null);
null,
0,
DocIdSetIterator.NO_MORE_DOCS);
assertTrue("docs Size: " + docs.size() + " is not: " + 2, docs.size() == 2);
TestHit doc0 = docs.get(0);
TestHit doc5 = docs.get(1);

3
lucene/core/src/test/org/apache/lucene/search/TestTopDocsCollector.java
View File

@ -89,12 +89,11 @@ public class TestTopDocsCollector extends LuceneTestCase {
public LeafCollector getLeafCollector(LeafReaderContext context) throws IOException {
final int base = context.docBase;
return new LeafCollector() {
private int idx = 0;
@Override
public void collect(int doc) {
++totalHits;
pq.insertWithOverflow(new ScoreDoc(doc + base, scores[context.docBase + idx++]));
pq.insertWithOverflow(new ScoreDoc(doc + base, scores[context.docBase + doc]));
}
@Override

4
lucene/core/src/test/org/apache/lucene/search/TestTopDocsMerge.java
View File

@ -51,13 +51,13 @@ public class TestTopDocsMerge extends LuceneTestCase {
}
public void search(Weight weight, Collector collector) throws IOException {
searchLeaf(ctx, weight, collector);
searchLeaf(ctx, 0, DocIdSetIterator.NO_MORE_DOCS, weight, collector);
}
public TopDocs search(Weight weight, int topN) throws IOException {
TopScoreDocCollector collector =
new TopScoreDocCollectorManager(topN, null, Integer.MAX_VALUE, false).newCollector();
searchLeaf(ctx, weight, collector);
searchLeaf(ctx, 0, DocIdSetIterator.NO_MORE_DOCS, weight, collector);
return collector.topDocs();
}

7
lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java
View File

@ -16,6 +16,7 @@
*/
package org.apache.lucene.search;
import com.carrotsearch.randomizedtesting.RandomizedTest;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.StringField;
@ -40,8 +41,10 @@ public class TestTotalHitCountCollector extends LuceneTestCase {
IndexReader reader = writer.getReader();
writer.close();
IndexSearcher searcher = newSearcher(reader, true, true, random().nextBoolean());
TotalHitCountCollectorManager collectorManager = new TotalHitCountCollectorManager();
Concurrency concurrency = RandomizedTest.randomFrom(Concurrency.values());
IndexSearcher searcher = newSearcher(reader, true, true, concurrency);
final TotalHitCountCollectorManager collectorManager =
new TotalHitCountCollectorManager(searcher.getSlices());
int totalHits = searcher.search(new MatchAllDocsQuery(), collectorManager);
assertEquals(5, totalHits);

8
lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java
View File

@ -82,12 +82,12 @@ public class FacetsCollector extends SimpleCollector {
public void collect(int doc) throws IOException {
docsBuilder.grow(1).add(doc);
if (keepScores) {
if (totalHits >= scores.length) {
float[] newScores = new float[ArrayUtil.oversize(totalHits + 1, 4)];
System.arraycopy(scores, 0, newScores, 0, totalHits);
if (doc >= scores.length) {
float[] newScores = new float[ArrayUtil.oversize(doc + 1, 4)];
System.arraycopy(scores, 0, newScores, 0, doc);
scores = newScores;
}
scores[totalHits] = scorer.score();
scores[doc] = scorer.score();
}
totalHits++;
}

81
lucene/facet/src/java/org/apache/lucene/facet/FacetsCollectorManager.java
View File

@ -17,8 +17,11 @@
package org.apache.lucene.facet;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Collection;
import java.util.List;
import java.util.HashMap;
import java.util.Map;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.search.CollectorManager;
import org.apache.lucene.search.FieldDoc;
import org.apache.lucene.search.IndexSearcher;
@ -32,6 +35,7 @@ import org.apache.lucene.search.TopFieldCollectorManager;
import org.apache.lucene.search.TopScoreDocCollectorManager;
import org.apache.lucene.search.TotalHitCountCollectorManager;
import org.apache.lucene.search.TotalHits;
import org.apache.lucene.util.DocIdSetBuilder;
/**
* A {@link CollectorManager} implementation which produces FacetsCollector and produces a merged
@ -75,12 +79,78 @@ public class FacetsCollectorManager implements CollectorManager<FacetsCollector,
ReducedFacetsCollector(final Collection<FacetsCollector> facetsCollectors, boolean keepScores) {
super(keepScores);
final List<MatchingDocs> matchingDocs = this.getMatchingDocs();
facetsCollectors.forEach(
facetsCollector -> matchingDocs.addAll(facetsCollector.getMatchingDocs()));
this.getMatchingDocs().addAll(reduceMatchingDocs(facetsCollectors));
}
}
/**
* Reduces matching docs held by the provided facets collectors, merging matching docs for the
* same leaf into a single matching docs instance
*
* @param facetsCollectors the facets collectors
* @return the reduced matching docs, with one instance per leaf reader context
*/
static Collection<FacetsCollector.MatchingDocs> reduceMatchingDocs(
final Collection<? extends FacetsCollector> facetsCollectors) {
// When a segment is split into partitions, each partition gets its own FacetsCollector that
// pulls doc_values independently, and builds a bitset of the size of the entire segment. When
// segments are partitioned, each partition will collect only the docs in its docid range, hence
// there will be multiple MatchingDocs pointing to the same LeafReaderContext. As part of the
// reduction we merge back partitions into a single MatchingDocs per segment.
Map<LeafReaderContext, FacetsCollector.MatchingDocs> matchingDocsMap = new HashMap<>();
for (FacetsCollector facetsCollector : facetsCollectors) {
for (FacetsCollector.MatchingDocs matchingDocs : facetsCollector.getMatchingDocs()) {
matchingDocsMap.compute(
matchingDocs.context,
(leafReaderContext, existing) -> {
if (existing == null) {
return matchingDocs;
}
return merge(existing, matchingDocs);
});
}
}
return matchingDocsMap.values();
}
private static FacetsCollector.MatchingDocs merge(
FacetsCollector.MatchingDocs matchingDocs1, FacetsCollector.MatchingDocs matchingDocs2) {
assert matchingDocs1.context == matchingDocs2.context;
final float[] scores;
// scores array is null when keepScores is true, and may be null when there are no matches for a
// segment partition, despite keepScores is true.
if (matchingDocs1.scores == null && matchingDocs2.scores == null) {
scores = new float[0];
} else {
if (matchingDocs2.scores == null) {
scores = matchingDocs1.scores;
} else if (matchingDocs1.scores == null) {
scores = matchingDocs2.scores;
} else {
int length = Math.max(matchingDocs1.scores.length, matchingDocs2.scores.length);
// merge the arrays if both have values, their size is bound to the highest collected docid
scores = new float[length];
for (int i = 0; i < length; i++) {
float firstScore = i < matchingDocs1.scores.length ? matchingDocs1.scores[i] : 0;
float secondScore = i < matchingDocs2.scores.length ? matchingDocs2.scores[i] : 0;
assert (firstScore > 0 && secondScore > 0) == false;
scores[i] = Math.max(firstScore, secondScore);
}
}
}
DocIdSetBuilder docIdSetBuilder = new DocIdSetBuilder(matchingDocs1.context.reader().maxDoc());
try {
docIdSetBuilder.add(matchingDocs1.bits.iterator());
docIdSetBuilder.add(matchingDocs2.bits.iterator());
} catch (IOException e) {
throw new UncheckedIOException(e);
}
int totalHits = matchingDocs1.totalHits + matchingDocs2.totalHits;
return new FacetsCollector.MatchingDocs(
matchingDocs1.context, docIdSetBuilder.build(), totalHits, scores);
}
/**
* Utility method, to search and also populate a {@code FacetsCollector} with hits. The provided
* {@code FacetsCollectorManager} will be used for creating/reducing {@code FacetsCollector}
@ -196,7 +266,8 @@ public class FacetsCollectorManager implements CollectorManager<FacetsCollector,
final TopDocs topDocs;
final FacetsCollector facetsCollector;
if (n == 0) {
TotalHitCountCollectorManager hitCountCollectorManager = new TotalHitCountCollectorManager();
TotalHitCountCollectorManager hitCountCollectorManager =
new TotalHitCountCollectorManager(searcher.getSlices());
MultiCollectorManager multiCollectorManager =
new MultiCollectorManager(hitCountCollectorManager, fcm);
Object[] result = searcher.search(q, multiCollectorManager);

5
lucene/facet/src/java/org/apache/lucene/facet/RandomSamplingFacetsCollector.java
View File

@ -288,9 +288,8 @@ public class RandomSamplingFacetsCollector extends FacetsCollector {
ReducedRandomSamplingFacetsCollector(
int sampleSize, long seed, Collection<RandomSamplingFacetsCollector> facetsCollectors) {
super(sampleSize, seed);
facetsCollectors.forEach(
facetsCollector ->
getOriginalMatchingDocs().addAll(facetsCollector.getOriginalMatchingDocs()));
this.getOriginalMatchingDocs()
.addAll(FacetsCollectorManager.reduceMatchingDocs(facetsCollectors));
}
}
}

6
lucene/facet/src/test/org/apache/lucene/facet/TestDrillSideways.java
View File

@ -142,7 +142,7 @@ public class TestDrillSideways extends FacetTestCase {
private IndexSearcher getNewSearcher(IndexReader reader) {
// Do not wrap with an asserting searcher, since DrillSidewaysQuery doesn't
// implement all the required components like Weight#scorer.
IndexSearcher searcher = newSearcher(reader, true, false, random().nextBoolean());
IndexSearcher searcher = newSearcher(reader, true, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
searcher.setTimeout(null);
@ -312,7 +312,9 @@ public class TestDrillSideways extends FacetTestCase {
baseScorer,
new DrillSidewaysScorer.DocsAndCost[] {docsAndCost},
scoreSubDocsAtOnce);
expectThrows(CollectionTerminatedException.class, () -> scorer.score(baseCollector, null));
expectThrows(
CollectionTerminatedException.class,
() -> scorer.score(baseCollector, null, 0, DocIdSetIterator.NO_MORE_DOCS));
// We've set things up so that our base collector with throw CollectionTerminatedException
// after collecting the first doc. This means we'll only collect the first indexed doc for

4
lucene/facet/src/test/org/apache/lucene/facet/range/TestRangeFacetCounts.java
View File

@ -469,7 +469,7 @@ public class TestRangeFacetCounts extends FacetTestCase {
final TaxonomyReader tr = new DirectoryTaxonomyReader(tw);
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);
@ -1652,7 +1652,7 @@ public class TestRangeFacetCounts extends FacetTestCase {
IndexReader r = writer.getReader();
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);

4
lucene/facet/src/test/org/apache/lucene/facet/rangeonrange/TestRangeOnRangeFacetCounts.java
View File

@ -492,7 +492,7 @@ public class TestRangeOnRangeFacetCounts extends FacetTestCase {
final TaxonomyReader tr = new DirectoryTaxonomyReader(tw);
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);
@ -605,7 +605,7 @@ public class TestRangeOnRangeFacetCounts extends FacetTestCase {
final TaxonomyReader tr = new DirectoryTaxonomyReader(tw);
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);

3
lucene/grouping/src/test/org/apache/lucene/search/grouping/TestGrouping.java
View File

@ -46,6 +46,7 @@ import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.valuesource.BytesRefFieldSource;
import org.apache.lucene.search.CachingCollector;
import org.apache.lucene.search.Collector;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.FieldDoc;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.MultiCollector;
@ -1581,7 +1582,7 @@ public class TestGrouping extends LuceneTestCase {
}
public void search(Weight weight, Collector collector) throws IOException {
searchLeaf(ctx, weight, collector);
searchLeaf(ctx, 0, DocIdSetIterator.NO_MORE_DOCS, weight, collector);
}
@Override

4
lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/TestRangeFacet.java
View File

@ -493,7 +493,7 @@ public class TestRangeFacet extends SandboxFacetTestCase {
final IndexReader r = w.getReader();
final TaxonomyReader tr = new DirectoryTaxonomyReader(tw);
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);
@ -1555,7 +1555,7 @@ public class TestRangeFacet extends SandboxFacetTestCase {
IndexReader r = writer.getReader();
IndexSearcher s = newSearcher(r, false, false);
IndexSearcher s = newSearcher(r, false, false, Concurrency.INTER_SEGMENT);
// DrillSideways requires the entire range of docs to be scored at once, so it doesn't support
// timeouts whose implementation scores one window of doc IDs at a time.
s.setTimeout(null);

4
lucene/suggest/src/java/org/apache/lucene/search/suggest/document/SuggestIndexSearcher.java
View File

@ -21,6 +21,7 @@ import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.search.BulkScorer;
import org.apache.lucene.search.CollectionTerminatedException;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.LeafCollector;
import org.apache.lucene.search.Weight;
@ -71,7 +72,8 @@ public class SuggestIndexSearcher extends IndexSearcher {
LeafCollector leafCollector = null;
try {
leafCollector = collector.getLeafCollector(context);
scorer.score(leafCollector, context.reader().getLiveDocs());
scorer.score(
leafCollector, context.reader().getLiveDocs(), 0, DocIdSetIterator.NO_MORE_DOCS);
} catch (
@SuppressWarnings("unused")
CollectionTerminatedException e) {

19
lucene/test-framework/src/java/org/apache/lucene/tests/search/AssertingBulkScorer.java
View File

@ -19,7 +19,6 @@ package org.apache.lucene.tests.search;
import com.carrotsearch.randomizedtesting.generators.RandomNumbers;
import java.io.IOException;
import java.util.Random;
import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.search.BulkScorer;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.LeafCollector;
@ -58,24 +57,6 @@ final class AssertingBulkScorer extends BulkScorer {
return in.cost();
}
@Override
public void score(LeafCollector collector, Bits acceptDocs) throws IOException {
assert max == 0;
collector = new AssertingLeafCollector(collector, 0, PostingsEnum.NO_MORE_DOCS);
if (random.nextBoolean()) {
try {
final int next = score(collector, acceptDocs, 0, PostingsEnum.NO_MORE_DOCS);
assert next == DocIdSetIterator.NO_MORE_DOCS;
} catch (
@SuppressWarnings("unused")
UnsupportedOperationException e) {
in.score(collector, acceptDocs);
}
} else {
in.score(collector, acceptDocs);
}
}
@Override
public int score(LeafCollector collector, Bits acceptDocs, int min, final int max)
throws IOException {

9
lucene/test-framework/src/java/org/apache/lucene/tests/search/AssertingIndexSearcher.java
View File

@ -80,6 +80,15 @@ public class AssertingIndexSearcher extends IndexSearcher {
assert assertingCollector.hasFinishedCollectingPreviousLeaf;
}
@Override
protected void search(LeafReaderContextPartition[] leaves, Weight weight, Collector collector)
throws IOException {
assert weight instanceof AssertingWeight;
AssertingCollector assertingCollector = AssertingCollector.wrap(collector);
super.search(leaves, weight, assertingCollector);
assert assertingCollector.hasFinishedCollectingPreviousLeaf;
}
@Override
public String toString() {
return "AssertingIndexSearcher(" + super.toString() + ")";

7
lucene/test-framework/src/java/org/apache/lucene/tests/search/CheckHits.java
View File

@ -654,6 +654,10 @@ public class CheckHits {
private final Weight weight;
private LeafReaderContext context;
int lastCheckedDoc = -1;
// with intra-segment concurrency, we may start from a doc id that isn't -1. We need to make
// sure that we don't go outside of the bounds of the current slice, meaning -1 can't be
// reliably used to signal that we are collecting the first doc for a given segment partition.
boolean collectedOnce = false;
public MatchesAsserter(Query query, IndexSearcher searcher) throws IOException {
this.weight = searcher.createWeight(searcher.rewrite(query), ScoreMode.COMPLETE_NO_SCORES, 1);
@ -671,7 +675,7 @@ public class CheckHits {
assertNotNull(
"Unexpected null Matches object in doc" + doc + " for query " + this.weight.getQuery(),
matches);
if (lastCheckedDoc != doc - 1) {
if (collectedOnce && lastCheckedDoc != doc - 1) {
assertNull(
"Unexpected non-null Matches object in non-matching doc"
+ doc
@ -679,6 +683,7 @@ public class CheckHits {
+ this.weight.getQuery(),
this.weight.matches(context, doc - 1));
}
collectedOnce = true;
lastCheckedDoc = doc;
}

10
lucene/test-framework/src/java/org/apache/lucene/tests/search/ScorerIndexSearcher.java
View File

@ -52,8 +52,16 @@ public class ScorerIndexSearcher extends IndexSearcher {
}
@Override
protected void searchLeaf(LeafReaderContext ctx, Weight weight, Collector collector)
protected void searchLeaf(
LeafReaderContext ctx, int minDocId, int maxDocId, Weight weight, Collector collector)
throws IOException {
// the default slices method does not create segment partitions, and we don't provide an
// executor to this searcher in our codebase, so we should not run into this problem. This class
// can though be used externally, hence it is better to provide a clear and hard error.
if (minDocId != 0 || maxDocId != DocIdSetIterator.NO_MORE_DOCS) {
throw new IllegalStateException(
"intra-segment concurrency is not supported by this searcher");
}
// we force the use of Scorer (not BulkScorer) to make sure
// that the scorer passed to LeafCollector.setScorer supports
// Scorer.getChildren

56
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
View File

@ -17,10 +17,12 @@
package org.apache.lucene.tests.util;
import static com.carrotsearch.randomizedtesting.RandomizedTest.frequently;
import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
import static com.carrotsearch.randomizedtesting.RandomizedTest.systemPropertyAsBoolean;
import static com.carrotsearch.randomizedtesting.RandomizedTest.systemPropertyAsInt;
import static org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS;
import static org.apache.lucene.search.IndexSearcher.LeafSlice;
import com.carrotsearch.randomizedtesting.JUnit4MethodProvider;
import com.carrotsearch.randomizedtesting.LifecycleScope;
@ -1922,8 +1924,32 @@ public abstract class LuceneTestCase extends Assert {
*/
public static IndexSearcher newSearcher(
IndexReader r, boolean maybeWrap, boolean wrapWithAssertions, boolean useThreads) {
if (useThreads) {
return newSearcher(r, maybeWrap, wrapWithAssertions, Concurrency.INTRA_SEGMENT);
}
return newSearcher(r, maybeWrap, wrapWithAssertions, Concurrency.NONE);
}
/** What level of concurrency is supported by the searcher being created */
public enum Concurrency {
/** No concurrency, meaning an executor won't be provided to the searcher */
NONE,
/**
* Inter-segment concurrency, meaning an executor will be provided to the searcher and slices
* will be randomly created to concurrently search entire segments
*/
INTER_SEGMENT,
/**
* Intra-segment concurrency, meaning an executor will be provided to the searcher and slices
* will be randomly created to concurrently search segment partitions
*/
INTRA_SEGMENT
}
public static IndexSearcher newSearcher(
IndexReader r, boolean maybeWrap, boolean wrapWithAssertions, Concurrency concurrency) {
Random random = random();
if (useThreads == false) {
if (concurrency == Concurrency.NONE) {
if (maybeWrap) {
try {
r = maybeWrapReader(r);
@ -1973,7 +1999,8 @@ public abstract class LuceneTestCase extends Assert {
new AssertingIndexSearcher(random, r, ex) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, maxDocPerSlice, maxSegmentsPerSlice);
return LuceneTestCase.slices(
leaves, maxDocPerSlice, maxSegmentsPerSlice, concurrency);
}
};
} else {
@ -1981,7 +2008,8 @@ public abstract class LuceneTestCase extends Assert {
new AssertingIndexSearcher(random, r.getContext(), ex) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, maxDocPerSlice, maxSegmentsPerSlice);
return LuceneTestCase.slices(
leaves, maxDocPerSlice, maxSegmentsPerSlice, concurrency);
}
};
}
@ -1990,7 +2018,8 @@ public abstract class LuceneTestCase extends Assert {
new IndexSearcher(r, ex) {
@Override
protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
return slices(leaves, maxDocPerSlice, maxSegmentsPerSlice);
return LuceneTestCase.slices(
leaves, maxDocPerSlice, maxSegmentsPerSlice, concurrency);
}
};
}
@ -2003,6 +2032,25 @@ public abstract class LuceneTestCase extends Assert {
}
}
/**
* Creates leaf slices according to the concurrency argument, that optionally leverage
* intra-segment concurrency by splitting segments into multiple partitions according to the
* maxDocsPerSlice argument.
*/
private static LeafSlice[] slices(
List<LeafReaderContext> leaves,
int maxDocsPerSlice,
int maxSegmentsPerSlice,
Concurrency concurrency) {
assert concurrency != Concurrency.NONE;
// Rarely test slices without partitions even though intra-segment concurrency is supported
return IndexSearcher.slices(
leaves,
maxDocsPerSlice,
maxSegmentsPerSlice,
concurrency == Concurrency.INTRA_SEGMENT && frequently());
}
/**
* Gets a resource from the test's classpath as {@link Path}. This method should only be used, if
* a real file is needed. To get a stream, code should prefer {@link #getDataInputStream(String)}.