honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Reduce locking in prewarming (#61837) (#61967) 

During prewarming of a Lucene file a CacheFile is acquired and 
then locked for the duration of the prewarming, ie locked until all 
the part of the file has been downloaded and written to cache on 
disk. The locking (executed with CacheFile#fileLock()) is here to 
prevent the cache file to be evicted while it is prewarming.

But holding the lock may take a while for large files, specially since
 restoring snapshot files now respects the 
indices.recovery.max_bytes_per_sec setting of 40mb (#58658), 
and this can have bad consequences like preventing the CacheFile 
to be evicted, opened or closed. In manual tests this bug slow 
downs various requests like mounting a new searchable snapshot
 index or deleting an existing one that is still prewarming.

This commit reduces the time the lock is held during prewarming so
 that the read lock is only required when actively writing to the CacheFile.
This commit is contained in:
Tanguy Leroux 2020-09-04 15:06:50 +02:00 committed by GitHub
parent 3e6e81c993
commit 289b1f4ae7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 42 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
x-pack/plugin/searchable-snapshots/src/main/java/org/elasticsearch/index/store/cache/CachedBlobContainerIndexInput.java vendored
View File

@ -453,7 +453,6 @@ public class CachedBlobContainerIndexInput extends BaseSearchableSnapshotIndexIn
try { try {
final CacheFile cacheFile = getCacheFileSafe(); final CacheFile cacheFile = getCacheFileSafe();
try (Releasable ignored = cacheFile.fileLock()) {
final Tuple<Long, Long> range = cacheFile.getAbsentRangeWithin(partRange.v1(), partRange.v2()); final Tuple<Long, Long> range = cacheFile.getAbsentRangeWithin(partRange.v1(), partRange.v2());
if (range == null) { if (range == null) {
@ -481,8 +480,6 @@ public class CachedBlobContainerIndexInput extends BaseSearchableSnapshotIndexIn
cacheFileReference cacheFileReference
); );
final FileChannel fc = cacheFile.getChannel();
assert assertFileChannelOpen(fc);
final byte[] copyBuffer = new byte[toIntBytes(Math.min(COPY_BUFFER_SIZE, rangeLength))]; final byte[] copyBuffer = new byte[toIntBytes(Math.min(COPY_BUFFER_SIZE, rangeLength))];
long totalBytesRead = 0L; long totalBytesRead = 0L;
@ -502,6 +499,9 @@ public class CachedBlobContainerIndexInput extends BaseSearchableSnapshotIndexIn
// noinspection UnnecessaryLocalVariable // noinspection UnnecessaryLocalVariable
final Tuple<Long, Long> rangeToRead = rangeToWrite; final Tuple<Long, Long> rangeToRead = rangeToWrite;
try (Releasable ignored = cacheFile.fileLock()) {
assert assertFileChannelOpen(cacheFile.getChannel());
cacheFile.populateAndRead( cacheFile.populateAndRead(
rangeToWrite, rangeToWrite,
rangeToRead, rangeToRead,
@ -525,15 +525,14 @@ public class CachedBlobContainerIndexInput extends BaseSearchableSnapshotIndexIn
}, },
directory.cacheFetchAsyncExecutor() directory.cacheFetchAsyncExecutor()
).get(); ).get();
}
totalBytesRead += bytesRead; totalBytesRead += bytesRead;
remainingBytes -= bytesRead; remainingBytes -= bytesRead;
} }
final long endTimeNanos = stats.currentTimeNanos(); final long endTimeNanos = stats.currentTimeNanos();
stats.addCachedBytesWritten(totalBytesWritten.get(), endTimeNanos - startTimeNanos); stats.addCachedBytesWritten(totalBytesWritten.get(), endTimeNanos - startTimeNanos);
} }
assert totalBytesRead == rangeLength; assert totalBytesRead == rangeLength;
}
} catch (final Exception e) { } catch (final Exception e) {
throw new IOException("Failed to prefetch file part in cache", e); throw new IOException("Failed to prefetch file part in cache", e);
} }