Apache
/
jclouds
mirror of https://github.com/apache/jclouds.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

JCLOUDS-460: Add jitter to avoid thundering herd 

When issuing many simultaneous requests to Synaptic Atmos I observed:

HTTP/1.1 failed with code 500, error: AtmosError
[code=1040, message=The server is busy. Please try again.]

Previously all clients slept for fixed intervals and thus retried
around the same time.  This commit adds a random delay which should
better distribute load on the provider.
This commit is contained in:
Andrew Gaul 2013-04-18 12:54:00 -07:00 committed by Andrew Gaul
parent a38cbd5803
commit fe79885663
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
core/src/main/java/org/jclouds/http/handlers/BackoffLimitedRetryHandler.java
View File

@ -19,6 +19,7 @@ package org.jclouds.http.handlers;
import static org.jclouds.http.HttpUtils.releasePayload;
import java.io.IOException;
import java.util.Random;
import javax.annotation.Resource;
import javax.inject.Named;
@ -126,6 +127,9 @@ public class BackoffLimitedRetryHandler implements HttpRetryHandler, IOException
public void imposeBackoffExponentialDelay(long period, long maxPeriod, int pow, int failureCount, int max,
String commandDescription) {
long delayMs = (long) (period * Math.pow(failureCount, pow));
// Add random delay to avoid thundering herd problem when multiple
// simultaneous failed requests retry after sleeping for the same delay.
delayMs += new Random().nextInt((int) (delayMs / 10));
delayMs = delayMs > maxPeriod ? maxPeriod : delayMs;
logger.debug("Retry %d/%d: delaying for %d ms: %s", failureCount, max, delayMs, commandDescription);
try {