Repeated SdkInterruptedException leading to filling up of logs
Describe the bug
Hi,
We recently made some changes in the code to paginate the autoscalingactivity calls instead of fetching top 10 records. Here is a sample code:
int count = 1;
while (count < 5) {
try{
DescribeScalingActivitiesRequest request = new DescribeScalingActivitiesRequest()
.withAutoScalingGroupName(asGroupName)
.withMaxRecords(maxRecords)
.withNextToken(nextToken);
DescribeScalingActivitiesResult result = client.describeScalingActivities(request);
recordRequestId(request);
return result;
} catch (Exception ase) {
// Retry in case of throttling exception
if (ase instanceof AmazonServiceException && isAwsThrottlingException((AmazonServiceException) ase)) {
// retry
count++;
} else {
logger.error(String.format("Error encountered while describeScalingActivities %s, %s", asGroupName,
ase.getMessage()), ase);
return null;
}
}
}
Almost all requests was going through fine, but suddenly we started running into the below error. The subsequent retries we made also failed with the same exception, filling up the logs and leading to a very high CPU usage:
[Error]Error encountered while describeScalingActivities awseb-e-ympup4jmfp-stack-AWSEBAutoScalingGroup-1WN2KTDE1R1JO,
com.amazonaws.AbortedException:
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:868)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:746)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:4837)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4804)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4793)
Caused by: com.amazonaws.http.timers.client.SdkInterruptedException
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:923) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:909) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1103) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~
Execution code was always hitting the below block once we started running into com.amazonaws.http.timers.client.SdkInterruptedException
else {
logger.error(String.format("Error encountered while describeScalingActivities %s, %s", asGroupName,
ase.getMessage()), ase);
return null;
}
Can you let us know why we are running into this error, even after retries which shouldn't be happening? We are using the latest version of aws-sdk for java and the decoupled version of AmazonHttpClient matches exactly with what we have inside the master branch.
Please let us know what needs to be done
Expected Behavior
Retries should work fine without SdkInterruptedException exception
Current Behavior
Fails with com.amazonaws.http.timers.client.SdkInterruptedException
Reproduction Steps
Use the above code and we have an SDK timeout of 5 seconds
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
1.11.x (internal)
JDK version used
1.8
Operating System and version
Linux