Memory usage growing on heap and remains after using SDK.

[Maknoos]

Describe the bug

When creating clients and doing requests with the AWS SDK for C++, memory usage on heap grows steadily and heap allocations do still exist even after calling Aws::ShutdownAPI. Is it possible that there is a memory leak in the SDK ?

Expected Behavior

There shouldn't be any remaining heap allocations related to the AWS SDK after clients go out of scope and ShutdownAPI has been called.

Current Behavior

I am a member of a team that develops an application that uses the AWS C++ SDK and creates a lot of clients to communicate with various different AWS services. We noticed that the memory usage in our application was growing steadily and memory profiling indicated that this was due to a lot of heap allocations for the AWS SDK.

We created the following C++ application to reproduce this:

#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/ec2/EC2Client.h>
#include <aws/ec2/model/DescribeVpcsRequest.h>

using namespace Aws::Auth;
using namespace Aws::EC2;

void useSDK() {
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
    Aws::InitAPI(options);

    AWSCredentials credentials;
    credentials.SetAWSAccessKeyId(Aws::String("Redacted-in-this-example"));
    credentials.SetAWSSecretKey(Aws::String("Redacted-in-this-example"));

    for (int i = 0; i < 100; i++) {
        auto client = EC2Client();
        const auto request{ Model::DescribeVpcsRequest() };
        const auto outcome = client.DescribeVpcs(request);
    }

    Aws::ShutdownAPI(options);
}

int main()
{
    useSDK();
    return 0;
}

Our expectation was that there wouldn't be a notable number of heap allocations after the call to useSDK() has finished, as all of the SDK logic is contained within that function and ShutdownAPI is called at the end. However heap snapshots show that there is still a lot of SDK allocated memory left when the program exits. We can also see that when we have a long running program that creates a lot of clients on stack, makes requests and discards them, the heap usage of the SDK keeps growing continuously instead of stopping at some point.

The image above shows how this looks in the Visual studio heap profiler.

The first snapshot is taken right before useSDK() is called and the second snapshot is taken before the program exits. Note the difference in the heap size.

When running the leak detection tool "Deleaker" on this program we also see that the tool is reporting potential memory leaks.

Reproduction Steps

The following code snippet can be run to reproduce the issue. Taking a heap snapshot before and after the call to the useSDK() function will show the heap allocations. The credentials in the snippet must of course be updated before running.

#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/ec2/EC2Client.h>
#include <aws/ec2/model/DescribeVpcsRequest.h>

using namespace Aws::Auth;
using namespace Aws::EC2;

void useSDK() {
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
    Aws::InitAPI(options);

    AWSCredentials credentials;
    credentials.SetAWSAccessKeyId(Aws::String("Redacted-in-this-example"));
    credentials.SetAWSSecretKey(Aws::String("Redacted-in-this-example"));

    for (int i = 0; i < 100; i++) {
        auto client = EC2Client();
        const auto request{ Model::DescribeVpcsRequest() };
        const auto outcome = client.DescribeVpcs(request);
    }

    Aws::ShutdownAPI(options);
}

int main()
{
    useSDK();
    return 0;
}

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.4

Compiler and Version used

Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31933 for x86

Operating System and version

Windows 10 Pro 21H2

[jmklix]

This might be caused by the same thing as this issue. Marking this as a duplicate for now.

[jmklix]

We fixed a memory leak here. You can get the latest updates from head or wait for tomorrow's release. Please let me know if you are still seeing any memory leaks/problems

[SergeyRyabinin]

Hi @Maknoos ,

In addition to @jmklix comments suggesting to update the SDK version that has a fix for a memory leak,

Please also wrap SDK handling code into curly braces { } In your example, destructor of credentials object

AWSCredentials credentials;

is being called after Aws::ShutdownAPI(options); ShutdownAPI effectively resets internal allocator pointers while AWSCredentials class encapsulates objects allocating memory using that allocator.

Best regards, Sergey

[Maknoos]

Hi

Thank you so much for the responses. I am going to update the SDK version and try wrapping the SDK code in curly braces as well. I will post an update here with the results afterwards.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[Maknoos]

I just updated to the latest SDK version and moved the curly braces. I am still seeing a lot of allocations remaining after calling ShutdownAPI. The heap size diff seems to grow bigger the more clients that are created. Here are screenshots of heap profiler info with snapshots taken before and after the call to useSDK() when creating 50,100 and 200 clients,

Is this intended behavior for the SDK ?

[jmklix]

If you are creating more clients, then it can be expected that the heap size will grow larger. We would encourage you to not create so many client, but rather reuse them for different operations that you want to do. Please let us know if you have any other questions.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.