Possible memory leaks in the SQSClient

[yehor-manzhula]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Describe the bug We are experiencing memory issues with the SQSClient. The issue is similar to one that was reported before Memory leaks in the DynamoDBClient, and DynamoDBDocumentClient but unfortunately it was closed with no solution. I've read the discussion in the issue and created a repository with an example that reproduces the issue we faced.

The repo contains simple polling functionality from test SQS queue in two variations: with forced garbage collection invocation on each request and without it. To be able to compare garbage collector behavior I've added test function that uses built-in http request wrapped in a promise to reach localstack internal API in same two variations.

I know that V8 has some internal optimizations around garbage collector invocation to archive better throughput performance but I notice a behavior that I found strange:

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

Clone example repo Make sure you have docker + docker-compose installed

Run npm install Run ./run.sh ./src/aws-sdk-gc.js

Observed Behavior

Once you started "http request" function with forced GC invocation it consumes around 5.5 Mb memory slowly growing with some point where memory consumption stays stable for some time till value up to 20.5 Mb and than memory gets cleaned to the startup value. As to me it looks like the memory got cleaned. (See screenshots)

But if you run polling from SQS queue with forced GC invocation you'll notice almost the same behavior except memory never returns back to initial values you saw on startup, continues slowly growing up to 44.8Mb and stays on the same level.

Expected Behavior

Memory should be cleaned at some point.

Possible Solution

No response

Additional Information/Context

No response

[kuhe]

note to maintainers:

code wrapper for debugging memory leaks

import { setFlagsFromString } from "v8";
import { runInNewContext } from "vm";

setFlagsFromString("--expose_gc");
const gc = runInNewContext("gc");

import { S3 } from "@aws-sdk/client-s3";
const s3 = new S3({
  region: "us-west-2",
});

setInterval(async () => {
  const mem = process.memoryUsage();
  console.log(mem.heapUsed.toLocaleString(), "of", mem.heapTotal.toLocaleString(), "bytes");

  // write code here you suspect causes a memory leak

  Array.from({ length: 250 }).map(async () => {
    //
  });

  gc();
}, 800);

[MajesticMug]

A possible solution to this was proposed on an old issue where limiting the max number of sockets on the HTTP modules prevents the excessive memory growth:

var http = require('http');
http.globalAgent.maxSockets = 30;
var https = require('https');
https.globalAgent.maxSockets = 30;

[smoke]

The very same case (issue) affects ALL of the @aws-sdk/client-* packages, so in most cases it is better to do client.destroy() as noted here and in other places https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-lambda/src/LambdaClient.ts#L692-L699

  /**
   * Destroy underlying resources, like sockets. It's usually not necessary to do this.
   * However in Node.js, it's best to explicitly shut down the client's agent when it is no longer needed.
   * Otherwise, sockets might stay open for quite a long time before the server terminates them.
   */
  destroy(): void

[RanVaknin]

Hi @yehor-manzhula ,

I have tested SQS polling for about 15 minutes and have observed that the memory does increase from about 9mb to around 11mb but stays steady at around that mark, and occasionally observe reduction in the heap size, indicating garbage collection. I also used the clinic memory profiling tool and see that the memory consumption is steady:

Also others on the thread have shared good pointers that may help get a more predictable memory usage from the SDK application.

If you have any data points for us to investigate, please let us know.

Thanks, Ran~

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.