Reduce contention in CacheFile.fileLock() method (#55662)

The CacheFile.fileLock() method is used to acquire a lock 
on a cache file so that the file can't be deleted (or its file 
handle closed) during the execution of a read or a write 
operation.

Today this lock is obtained by first acquiring the eviction 
lock (the write lock of the readwrite lock), then by checking 
if the cache file is evicted and the file channel still open, 
and finally by obtaining the file lock (the read lock of the 
readwrite lock). Acquiring the read lock while the eviction 
lock is held ensures that the cache file eviction cannot 
start in the meanwhile. But eviction starts (and terminations) 
also acquire the eviction lock; and this lock cannot be 
obtained while a read lock is held (the write lock of a 
readwrite lock is exclusive).

If we were acquiring a read lock and checking the eviction 
flag and file channel existence while holding the read lock 
we know that no eviction can start or finish until the 
read lock is released.

x-pack/plugin/searchable-snapshots/src/main/java/org/elasticsearch/index/store/cache/CacheFile.java

@@ -85,16 +85,20 @@ public class CacheFile {
     }
 
     ReleasableLock fileLock() {
-        try (ReleasableLock ignored = evictionLock.acquire()) {
+        boolean success = false;
+        final ReleasableLock fileLock = readLock.acquire();
+        try {
             ensureOpen();
-            // check if we have a channel under eviction lock
+            // check if we have a channel while holding the read lock
             if (channel == null) {
                 throw new AlreadyClosedException("Cache file channel has been released and closed");
             }
-            // acquire next read lock while holding the eviction lock
+            success = true;
-            // makes sure that channel won't be closed until this
+            return fileLock;
-            // read lock is released
+        } finally {
-            return readLock.acquire();
+            if (success == false) {
+                fileLock.close();
+            }
         }
     }