Carry over version map size to prevent excessive resizing (#27516)
Today we create a new concurrent hash map everytime we refresh the internal reader. Under defaults this isn't much of a deal but once the refresh interval is set to `-1` these maps grow quite large and it can have a significant impact on indexing throughput. Under low memory situations this can cause up to 2x slowdown. This change carries over the map size as the initial capacity wich will be auto-adjusted once indexing stops. Closes #20498
This commit is contained in:
parent
c6724abe74
commit
17e9940fc1
|
@ -43,7 +43,14 @@ public abstract class ConcurrentCollections {
|
|||
* Creates a new CHM with an aggressive concurrency level, aimed at high concurrent update rate long living maps.
|
||||
*/
|
||||
public static <K, V> ConcurrentMap<K, V> newConcurrentMapWithAggressiveConcurrency() {
|
||||
return new ConcurrentHashMap<>(16, 0.75f, aggressiveConcurrencyLevel);
|
||||
return newConcurrentMapWithAggressiveConcurrency(16);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a new CHM with an aggressive concurrency level, aimed at high concurrent update rate long living maps.
|
||||
*/
|
||||
public static <K, V> ConcurrentMap<K, V> newConcurrentMapWithAggressiveConcurrency(int initalCapacity) {
|
||||
return new ConcurrentHashMap<>(initalCapacity, 0.75f, aggressiveConcurrencyLevel);
|
||||
}
|
||||
|
||||
public static <K, V> ConcurrentMap<K, V> newConcurrentMap() {
|
||||
|
|
|
@ -102,7 +102,7 @@ class LiveVersionMap implements ReferenceManager.RefreshListener, Accountable {
|
|||
// map. While reopen is running, any lookup will first
|
||||
// try this new map, then fallback to old, then to the
|
||||
// current searcher:
|
||||
maps = new Maps(ConcurrentCollections.<BytesRef,VersionValue>newConcurrentMapWithAggressiveConcurrency(), maps.current);
|
||||
maps = new Maps(ConcurrentCollections.newConcurrentMapWithAggressiveConcurrency(maps.current.size()), maps.current);
|
||||
|
||||
// This is not 100% correct, since concurrent indexing ops can change these counters in between our execution of the previous
|
||||
// line and this one, but that should be minor, and the error won't accumulate over time:
|
||||
|
@ -117,7 +117,7 @@ class LiveVersionMap implements ReferenceManager.RefreshListener, Accountable {
|
|||
// case. This is because we assign new maps (in beforeRefresh) slightly before Lucene actually flushes any segments for the
|
||||
// reopen, and so any concurrent indexing requests can still sneak in a few additions to that current map that are in fact reflected
|
||||
// in the previous reader. We don't touch tombstones here: they expire on their own index.gc_deletes timeframe:
|
||||
maps = new Maps(maps.current, ConcurrentCollections.<BytesRef,VersionValue>newConcurrentMapWithAggressiveConcurrency());
|
||||
maps = new Maps(maps.current, Collections.emptyMap());
|
||||
}
|
||||
|
||||
/** Returns the live version (add or delete) for this uid. */
|
||||
|
|
Loading…
Reference in New Issue