SOLR-13386: OverseerTaskQueue#remove should not throw an exception when no node exists after an exists check and the Overseer work loop should not allow free spinning the loop when it hits a KeeperException.

solr/CHANGES.txt

@@ -193,6 +193,10 @@ Bug Fixes
 * SOLR-13336: add maxBooleanClauses (default to 1024) setting to solr.xml, reverting previous effective
   value of Integer.MAX_VALUE-1, to restrict risk of pathalogical query expansion.  (hossman)
 
+  * SOLR-13386: OverseerTaskQueue#remove should not throw an exception when no node exists after an exists 
+  check and the Overseer work loop should not allow free spinning the loop when it hits a KeeperException.
+  (Mark Miller, Fernandez-Lobbe, Mike Drob)
+
 Improvements
 ----------------------
 

solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java

@@ -321,6 +321,15 @@ public class OverseerTaskProcessor implements Runnable, Closeable {
             return;
           }
           SolrException.log(log, "", e);
+          
+          // Prevent free-spinning this loop.
+          try {
+            Thread.sleep(1000);
+          } catch (InterruptedException e1) {
+            Thread.currentThread().interrupt();
+            return;
+          }
+          
         } catch (InterruptedException e) {
           Thread.currentThread().interrupt();
           return;

solr/core/src/java/org/apache/solr/cloud/OverseerTaskQueue.java

@@ -93,9 +93,11 @@ public class OverseerTaskQueue extends ZkDistributedQueue {
       String path = event.getId();
       String responsePath = dir + "/" + RESPONSE_PREFIX
           + path.substring(path.lastIndexOf("-") + 1);
-      if (zookeeper.exists(responsePath, true)) {
+
+      try {
         zookeeper.setData(responsePath, event.getBytes(), true);
-      } else {
+      } catch (KeeperException.NoNodeException ignored) {
+        // we must handle the race case where the node no longer exists
         log.info("Response ZK path: " + responsePath + " doesn't exist."
             + "  Requestor may have disconnected from ZooKeeper");
       }