undici
undici copied to clipboard
nodejs
Memory leak with native fetch on Node v19+
Version
v20.2.0
Platform
Darwin Kamils-MacBook-Pro.local 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64
Subsystem
No response
What steps will reproduce the bug?
- Git clone kamilkisiela/node-fetch-leak repo
- Steps to reproduce the memory leak are in
README.md
How often does it reproduce? Is there a required condition?
Always
What is the expected behavior? Why is that the expected behavior?
Constant memory usage
What do you see instead?
Increase in memory (heap snapshots available)
Additional information
To reproduce the memory leak I created a simple node:http server that calls native fetch() on every incoming HTTP request.
I ran a load test against that server and measured the memory (heap snapshots available in the repository).
Next, I ran the same steps for Node v18.16 to compare the memory consumption.
It seems like Node v19, v20 are leaking but v18 seems stable.
| Requests made | v20.2 + fetch | v20.2 + undici v5.22.1 | v18.16 + fetch | v18.16 + undici v5.22.1 |
|---|---|---|---|---|
| ~300k | 16MB | 19MB | 9MB | 11MB |
| ~600k | 23MB | 26MB | 9MB | 11MB |
Node v20.2
Native fetch
| Requests made | Memory |
|---|---|
| 295k | 16MB |
| 575k | 23MB |
Undici v5.22.1
| Requests made | Memory |
|---|---|
| 365k | 19MB |
| 650k | 26MB |
Node v18.16
Native fetch
| Requests made | Memory |
|---|---|
| 360k | 9MB |
| 680k | 9MB |
Undici v5.22.1
| Requests made | Memory |
|---|---|
| 366k | 11MB |
| 678k | 11MB |
Related to nodejs/node#47936
Because it was moved to nodejs/undici I pushed a commit with numbers and heap snapshots for node + undici.
Does it eventually lead to a crash, or is it just an increase in memory usage?
I ran it only for 360s and the only thing I noticed is the increase of memory usage.
I also updated kamilkisiela/graphql-hive repo to Node 20 and we got to OOM pretty quickly there (but it’s another memory leak, related to http module of node, not fetch api but these two might be related).
Did you take only two snapshots of each case?
For the behavior... The memory grows exponentially, or did you notice drops in the consumption after X seconds?
I'll try to visit this one this week
I haven't went deep there, just noticed a memory leak and reproduced it. I only looked at the memory usage twice, right after sending low amount of traffic to warm the server and then right after sending huge traffic.
Based on the data we have, this doesn't look like a leak but rather an increase in memory usage. Given this happens across different versions of Node.js/V8, it would likely be a change in the garbage collector algorithm, and nothing something we are responsible for.
Note that fetch requires a few "bad" patterns that are hard on the GC (finalizers).
Let's say I run it now for 6 minutes (as I did), then 12 and 24 minutes and the memory keeps increasing. It would mean it's a leak no?
Can you try running node with for example --max-old-space-size=128? If there is a real memory leak, the process should crash when used memory reaches around 128MB.
Just to clarify, the reproduction is:
- Warm up the server:
k6 run k6.js --vus 5 --duration 10s - Run first set of requests:
k6 run k6.js --vus 5 --duration 360sand make the heap snapshot - Run second set of requests:
k6 run k6.js --vus 5 --duration 360sand make the heap snapshot
The memory goes up after each step. I could run 3rd step 10 more times and I believe the memory will goes up as well and at some point it will go to space with Elon (OOM).
Let's say I run it now for 6 minutes (as I did), then 12 and 24 minutes and the memory keeps increasing. It would mean it's a leak no?
No it does not mean it's a leak. When V8 is under pressure CPU-wise, it tends to not cleanup memory unless it is absolutely needed. This happens after memory is transferred to "old space".
The command @targos mentioned will definitely shows if there are issues.