aws-sdk-java icon indicating copy to clipboard operation
aws-sdk-java copied to clipboard

Repeated SdkInterruptedException leading to filling up of logs

Open hariohmprasath opened this issue 3 years ago • 0 comments

Describe the bug

Hi,

We recently made some changes in the code to paginate the autoscalingactivity calls instead of fetching top 10 records. Here is a sample code:

int count = 1;
while (count < 5) {
    try{
        DescribeScalingActivitiesRequest request = new DescribeScalingActivitiesRequest()
                .withAutoScalingGroupName(asGroupName)
                .withMaxRecords(maxRecords)
                .withNextToken(nextToken);

        DescribeScalingActivitiesResult result = client.describeScalingActivities(request);
        recordRequestId(request);
        return result;
    } catch (Exception ase) {
        // Retry in case of throttling exception
        if (ase instanceof AmazonServiceException && isAwsThrottlingException((AmazonServiceException) ase)) {
            // retry
            count++;
        } else {
            logger.error(String.format("Error encountered while describeScalingActivities %s, %s", asGroupName,
                    ase.getMessage()), ase);
            return null;
        }
    }
}

Almost all requests was going through fine, but suddenly we started running into the below error. The subsequent retries we made also failed with the same exception, filling up the logs and leading to a very high CPU usage:

[Error]Error encountered while describeScalingActivities awseb-e-ympup4jmfp-stack-AWSEBAutoScalingGroup-1WN2KTDE1R1JO,
com.amazonaws.AbortedException:
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:868) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:746)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) 
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) 
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) 
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:4837) 
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4804) 
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4793) 

Caused by: com.amazonaws.http.timers.client.SdkInterruptedException
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:923) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:909) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1103) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~

Execution code was always hitting the below block once we started running into com.amazonaws.http.timers.client.SdkInterruptedException

else {
    logger.error(String.format("Error encountered while describeScalingActivities %s, %s", asGroupName,
            ase.getMessage()), ase);
    return null;
}

Can you let us know why we are running into this error, even after retries which shouldn't be happening? We are using the latest version of aws-sdk for java and the decoupled version of AmazonHttpClient matches exactly with what we have inside the master branch.

Please let us know what needs to be done

Expected Behavior

Retries should work fine without SdkInterruptedException exception

Current Behavior

Fails with com.amazonaws.http.timers.client.SdkInterruptedException

Reproduction Steps

Use the above code and we have an SDK timeout of 5 seconds

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

1.11.x (internal)

JDK version used

1.8

Operating System and version

Linux

hariohmprasath avatar May 14 '22 07:05 hariohmprasath