aws
Lambda Node 22 coldstart latency regression
Checkboxes for prior research
- [x] I've gone through Developer Guide and API reference
- [x] I've checked AWS Forums and StackOverflow.
- [x] I've searched for previous similar issues and didn't find any solution.
Describe the bug
When I updated to Node 22 on Lambda using the AWS Javascript SDK V3 last year, it added at least 50 ms to my coldstart times. For awhile I assumed it was due to the edge caches not being warm, but recently the lambda perf benchmark was updated with Node 22 and showed no appreciable difference in coldstart times between Node 20 and Node 22. I investigated further and it appears it is only when using the AWS Javascript SDK V3. In Node 22, it loads the http request bits that add 50 ms to the coldstart even if they aren't used. Similar to #6144, adding some lazy loading logic on the http request and agent should keep the coldstart performance inline with Node 20.
Regression Issue
- [x] Select this option if this issue appears to be a regression.
SDK version number
@aws-sdk/client-sts 3.750
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
Node 22.11
Reproduction Steps
I've created a repro with instructions here.
Observed Behavior
Node 22 exhibits 50 additional ms of coldstart time over Node 20 in the Lambda environment.
Here is a snippet from the README in the repro:
- Runs 1 and 2 show the baseline without the AWS SDK.
- Runs 3 and 4 show the difference between Node 20 and Node 22 with the AWS SDK.
- Runs 5 and 6 show the difference between Node 20 and Node 22 with a patch that removes http from the AWS SDK.
These are samples from my runs, but the results were repeatable.
| run | node version | @initDuration | notes |
|---|---|---|---|
| 1 | 20.18 | 142.79 | No AWS V3 SDK |
| 2 | 22.11 | 142.26 | No AWS V3 SDK |
| 3 | 20.18 | 204.23 | With AWS V3 SDK |
| 4 | 22.11 | 255.26 | With AWS V3 SDK |
| 5 | 20.18 | 200.42 | With patched AWS V3 SDK |
| 6 | 22.11 | 203.62 | With patched AWS V3 SDK |
Expected Behavior
Similar coldstart performance between Node 20 and Node 22.
Possible Solution
Lazy load the http request and agent bits.
Additional Information/Context
No response
Hi @perpil - thanks for reporting and for your patience while I investigate. Based on my benchmarks, I was able to confirm the difference in coldstart times of ~15-20ms between Node 20 and 22. I went on to perform the benchmarks of http vs https between two Node versions and noticed similar performance. As you mentioned, we might consider to add lazy loading logic on the http request and agent to inline with Node 20. If you may find anything else that might be useful, please feel free to add. Thanks again!
Benchmarks
- ap-south-1
$ AwsSdkLambdaJsBenchmark> yarn benchmark
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 22.11.0, x86_64, 128 MB, │ init_duration (ms) │ 375.13 │ 401.87 │ 12.11 ║
║ ap-south-1]: Code (esm) with sts v3.758.0 │ │ │ │ ║
║ (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
$ AwsSdkLambdaJsBenchmark> yarn benchmark --node-versions 20
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 20.18.0, x86_64, 128 MB, │ init_duration (ms) │ 366.34 │ 403.38 │ 16.2 ║
║ ap-south-1]: Code (esm) with sts v3.758.0 │ │ │ │ ║
║ (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
- eu-west-1
$ AwsSdkLambdaJsBenchmark> yarn benchmark
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 22.11.0, x86_64, 128 MB, eu-west-1]: │ init_duration (ms) │ 407.18 │ 497.17 │ 35.76 ║
║ Code (esm) with sts v3.758.0 (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
$ AwsSdkLambdaJsBenchmark> yarn benchmark --node-versions 20
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 20.18.0, x86_64, 128 MB, eu-west-1]: │ init_duration (ms) │ 393.56 │ 466.05 │ 28.13 ║
║ Code (esm) with sts v3.758.0 (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
- eu-west-2
$ AwsSdkLambdaJsBenchmark> yarn benchmark
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 22.11.0, x86_64, 128 MB, eu-west-2]: │ init_duration (ms) │ 372.52 │ 384.89 │ 8.13 ║
║ Code (esm) with sts v3.758.0 (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
$ AwsSdkLambdaJsBenchmark> yarn benchmark --node-versions 20
╔════════════════════════════════════════════╤════════════════════╤════════╤════════╤════════╗
║ │ metric │ p50 │ p90 │ stdDev ║
╟────────────────────────────────────────────┼────────────────────┼────────┼────────┼────────╢
║ [node 20.18.0, x86_64, 128 MB, eu-west-2]: │ init_duration (ms) │ 356.89 │ 372.92 │ 10.57 ║
║ Code (esm) with sts v3.758.0 (2.27 MB) │ │ │ │ ║
╚════════════════════════════════════════════╧════════════════════╧════════╧════════╧════════╝
AwsSdkLambdaJsBenchmark nvm use 20
Now using node v20.18.3 (npm v10.8.2)
AwsSdkLambdaJsBenchmark node benchmark.mjs
┌──────────────────────┬──────────┬──────────┬──────────┐
│ (index) │ Avg (ms) │ Max (ms) │ Min (ms) │
├──────────────────────┼──────────┼──────────┼──────────┤
│ Full Stack Load Time │ '6.745' │ '65.768' │ '0.061' │
└──────────────────────┴──────────┴──────────┴──────────┘
Now using node v22.8.0 (npm v10.8.2)
AwsSdkLambdaJsBenchmark node benchmark.mjs
┌──────────────────────┬──────────┬──────────┬──────────┐
│ (index) │ Avg (ms) │ Max (ms) │ Min (ms) │
├──────────────────────┼──────────┼──────────┼──────────┤
│ Full Stack Load Time │ '7.899' │ '77.318' │ '0.070' │
└──────────────────────┴──────────┴──────────┴──────────┘
Thanks for the followup @aBurmeseDev
I'm surprised there is such a large discrepancy between your benchmark and mine. I'm seeing very different numbers in overall initDuration (~200 ms vs ~350+ ms), code size (315K vs 2.27 MB) and delta between node 20 and 22 (50 ms vs 15-20ms).
You can see my configuration here. I bundle, use esm, use 1767MB ram, arm64, remove unnecessary credentials providers, and run in us-east-2.
Are you able to share more about your benchmark, because 50 ms on 200 is a lot more interesting than 20 ms on 350 ms and you may not be replicating what I'm reporting. If you haven't can you try running my repro? At least one other person saw the same 50 ms delta between node 20 and 22 when using the v3 javascript sdk.
As an experiment, I decided to see what would happen if I didn't use the AWS Javascript SDK and just instantiated a http and https agent.
import { Agent as hAgent, request as hRequest } from "http";
import { Agent as hsAgent, request as hsRequest } from "https";
const httpAgent = new hAgent({ keepAlive: true, maxSockets: 50 });
const httpsAgent = new hsAgent({ keepAlive: true, maxSockets: 50 });
export const handler = async (event, context) => {
return {
statusCode: 200,
body: {
requestId: context.awsRequestId,
},
};
};
On Node 20, the coldstart time is: 157 ms and on Node 22, the coldstart time is: 224 ms. This was done with arm64/128MB/us-east-2.
Although some of the performance hit goes away if you only instantiate the https agent, this repro seems to point the finger at node.js or the lambda node runtime. I've reached out to the Lambda team to see if they have any thoughts on this.
Update 3/26:
This is a minimal repro, it's actually importing request at all which causes it:
import { request } from "http";
export const handler = async (event, context) => {
return {
statusCode: 200,
body: {
requestId: context.awsRequestId,
},
};
};
Although there is potential to lazy load the request class in the SDK, it might be worth waiting to see if it is node.js or lambda related first. If it is an issue with either of those, no action may be necessary with the SDK.
Edit: Sorry, I was confused. Unfortunately there is no improvement on this issue, yet :(
Original message:
According to the benchmarks, Node 22 is now on par with Node 20! https://maxday.github.io/lambda-perf/ 🎉
Last time I checked (last week I believe) it was still showing the difference in cold start time as described in this issue.
Hi @janpapenbrock
So there is no confusion here, the maxday benchmark does not initialize any javascript v3 sdk clients. You can see the code for node 22 here
As such, the regression in coldstart times is not visible in the benchmark. The smallest repro is to import request from the http module to simulate what the SDK does like so:
import { request } from "http";
export const handler = async (event, context) => {
return {
statusCode: 200,
body: {
requestId: context.awsRequestId,
},
};
};
With this, output with 1 run of Node 20 (140 ms):
INIT_START Runtime Version: nodejs:20.v70 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:c11827d1307a63ca6bbb996dbffba66628e1328ed4714070f4b891884a323e0b
REPORT RequestId: fb3848c4-efcb-440b-a0c4-09f553d55a44 Duration: 3.26 ms Billed Duration: 4 ms Memory Size: 1800 MB Max Memory Used: 67 MB Init Duration: 140.09 ms
Output with 1 run of Node 22 (250 ms):
INIT_START Runtime Version: nodejs:22.v48 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:ccd522aa46eeddade4be388ba28af972761953cf91d2745b89d3215c05b412c2
REPORT RequestId: e663aa02-7d05-4e6b-b1db-574944f1ae89 Duration: 3.63 ms Billed Duration: 4 ms Memory Size: 1800 MB Max Memory Used: 80 MB Init Duration: 250.22 ms