MSQ WorkerImpl: Ignore ServiceClosedException on postCounters. (#14707)

* MSQ WorkerImpl: Ignore ServiceClosedException on postCounters.

A race can happen where postCounters is in flight while the controller
goes offline. When this happens, we should ignore the ServiceClosedException
and continue without posting counters.

* Fix style and logic.
This commit is contained in:
Gian Merlino 2023-08-01 18:30:10 -07:00 committed by GitHub
parent 4a31ae26f4
commit 72c151a192
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 12 additions and 1 deletions

View File

@ -132,6 +132,7 @@ import org.apache.druid.query.PrioritizedCallable;
import org.apache.druid.query.PrioritizedRunnable;
import org.apache.druid.query.QueryContext;
import org.apache.druid.query.QueryProcessingPool;
import org.apache.druid.rpc.ServiceClosedException;
import org.apache.druid.server.DruidNode;
import javax.annotation.Nullable;
@ -850,7 +851,17 @@ public class WorkerImpl implements Worker
final CounterSnapshotsTree snapshotsTree = getCounters();
if (controllerAlive && !snapshotsTree.isEmpty()) {
controllerClient.postCounters(id(), snapshotsTree);
try {
controllerClient.postCounters(id(), snapshotsTree);
}
catch (IOException e) {
if (e.getCause() instanceof ServiceClosedException) {
// Suppress. This can happen if the controller goes away while a postCounters call is in flight.
log.debug(e, "Ignoring failure on postCounters, because controller has gone away.");
} else {
throw e;
}
}
}
}