tfjs
tfjs copied to clipboard
tensorflow
Memory leaks
System information
- Have I written custom code: yes
- OS Platform and Distribution: Windows 10
- TensorFlow.js installed from: npm (+ prebuild dll from https://www.tensorflow.org/install/lang_c#download_and_extract )
- TensorFlow.js version: node.js binding 4.22.0, libtensorflow dll 2.18.1
- Version: node.js 22.9.0, bun 1.2.17
Describe the current behavior
If you run training on a larger amount of data and a complex neural network model, then a memory leak causes a crash. Even if the Working Set stabilizes, Private Bytes continues to grow infinitely, reaching some limits
2025-06-26 03:02:12.840574: W tensorflow/core/framework/op_kernel.cc:1840] OP_REQUIRES failed at cwise_ops_common.h:124 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[32,16] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
============================================================
Bun v1.2.17 (282dda62) Windows x64
Windows v.win10_fe
CPU: sse42 avx avx2
Args: "C:\Users\****\.bun\bin\bun.exe" "src/index.ts"
Features: Bun.stderr Bun.stdout(3) jsc transpiler_cache(9) tsconfig(2) process_dlopen
Builtins: "bun:main" "bun:sqlite" "node:assert" "node:buffer" "node:child_process" "node:crypto" "node:events" "node:fs" "node:os" "node:path"
"node:stream" "node:url" "node:util"
Elapsed: 52508590ms | User: 54715984ms | Sys: 380281ms
RSS: 4.77GB | Peak: 4.77GB | Commit: 8.95GB | Faults: 80753124
panic(main thread): attempt to unwrap error: OutOfMemory
oh no: Bun has crashed. This indicates a bug in Bun, not your code.
Describe the expected behavior Private bytes does not grow and there is no out of memory (OOM) error
Standalone code to reproduce the issue
import tf from '@tensorflow/tfjs-node';
const width = 150;
const height = 10;
const forMemoryLeaks = [];
console.time('total time');
for (let i = 0; i < 5_000_000; i++) {
tf.tidy(() => {
const stateTensorTemp = tf.tensor2d([Array.from({ length: width * height }, () => Math.random())]);
const stateTensor = stateTensorTemp.reshape([1, width, height]);
});
if (i % 100_000 === 0) {
forMemoryLeaks.push(process.memoryUsage());
console.table(forMemoryLeaks);
}
}
console.timeEnd('total time');
Other info / logs
bun 1.2.17
┌────┬───────────┬───────────┬──────────┬──────────┬──────────────┐
│ │ rss │ heapTotal │ heapUsed │ external │ arrayBuffers │
├────┼───────────┼───────────┼──────────┼──────────┼──────────────┤
│ 0 │ 335286272 │ 9522176 │ 234333 │ 40781 │ 0 │
│ 1 │ 415936512 │ 25235456 │ 17000145 │ 4817217 │ 0 │
│ 2 │ 386093056 │ 31961088 │ 22685473 │ 4853361 │ 0 │
│ 3 │ 389959680 │ 37982208 │ 28366497 │ 4901505 │ 0 │
│ 4 │ 403562496 │ 44118016 │ 34018551 │ 4925559 │ 0 │
│ 5 │ 417083392 │ 50384896 │ 39662233 │ 4967721 │ 0 │
│ 6 │ 426631168 │ 56651776 │ 45272825 │ 5003865 │ 0 │
│ 7 │ 440102912 │ 62870528 │ 50923321 │ 5046009 │ 0 │
│ 8 │ 453619712 │ 69350400 │ 56565757 │ 5076189 │ 0 │
│ 9 │ 467386368 │ 75587584 │ 62185977 │ 5124441 │ 0 │
│ 10 │ 476299264 │ 80333824 │ 9441767 │ 2415351 │ 0 │
│ 11 │ 476356608 │ 80333824 │ 17333563 │ 2469675 │ 0 │
│ 12 │ 476372992 │ 80333824 │ 25366075 │ 2499819 │ 0 │
│ 13 │ 476381184 │ 80333824 │ 33327821 │ 2511837 │ 0 │
│ 14 │ 476381184 │ 80333824 │ 41325865 │ 2535945 │ 0 │
│ 15 │ 476393472 │ 80333824 │ 49374729 │ 2541945 │ 0 │
│ 16 │ 476565504 │ 80332800 │ 15177708 │ 2488108 │ 0 │
│ 17 │ 476585984 │ 80332800 │ 22753588 │ 2651044 │ 0 │
│ 18 │ 476585984 │ 80332800 │ 29741832 │ 2952808 │ 0 │
│ 19 │ 476585984 │ 80332800 │ 36739040 │ 3236464 │ 0 │
│ 20 │ 476651520 │ 80332800 │ 43679818 │ 3538138 │ 0 │
│ 21 │ 476921856 │ 80350208 │ 8513242 │ 2403786 │ 0 │
│ 22 │ 476938240 │ 80350208 │ 15338912 │ 2989152 │ 0 │
│ 23 │ 476966912 │ 80350208 │ 21742694 │ 3677094 │ 0 │
│ 24 │ 477151232 │ 80351232 │ 28637808 │ 4027056 │ 0 │
│ 25 │ 477560832 │ 80352256 │ 35541726 │ 4358910 │ 0 │
│ 26 │ 477822976 │ 80354304 │ 42451112 │ 4666728 │ 0 │
│ 27 │ 478085120 │ 80355328 │ 49594276 │ 4883988 │ 0 │
│ 28 │ 478470144 │ 80355328 │ 15196754 │ 2584738 │ 0 │
│ 29 │ 478470144 │ 80355328 │ 21831786 │ 3012682 │ 0 │
│ 30 │ 478470144 │ 80355328 │ 28328578 │ 3488914 │ 0 │
│ 31 │ 478535680 │ 80355328 │ 35007218 │ 3922930 │ 0 │
│ 32 │ 478535680 │ 80355328 │ 41238712 │ 4537288 │ 0 │
│ 33 │ 478703616 │ 80356352 │ 47458600 │ 5127880 │ 0 │
│ 34 │ 479059968 │ 80357376 │ 12294224 │ 2439856 │ 0 │
│ 35 │ 479059968 │ 80357376 │ 20219416 │ 2536360 │ 0 │
│ 36 │ 479068160 │ 80357376 │ 28099624 │ 2632936 │ 0 │
│ 37 │ 479068160 │ 80357376 │ 35324102 │ 2886358 │ 0 │
│ 38 │ 479068160 │ 80357376 │ 42561706 │ 3121690 │ 0 │
│ 39 │ 479068160 │ 80357376 │ 49670750 │ 3429454 │ 0 │
│ 40 │ 479072256 │ 80357376 │ 15236542 │ 2445838 │ 0 │
│ 41 │ 479076352 │ 80357376 │ 23327958 │ 2584630 │ 0 │
│ 42 │ 479141888 │ 80357376 │ 30321928 │ 2952808 │ 0 │
│ 43 │ 479141888 │ 80357376 │ 37340064 │ 3272608 │ 0 │
│ 44 │ 479141888 │ 80357376 │ 44238434 │ 3676786 │ 0 │
│ 45 │ 479211520 │ 80357376 │ 8989388 │ 2397676 │ 0 │
│ 46 │ 479227904 │ 80357376 │ 16379804 │ 2819980 │ 0 │
│ 47 │ 479227904 │ 80357376 │ 23996056 │ 2898376 │ 0 │
│ 48 │ 479232000 │ 80357376 │ 31619600 │ 2970736 │ 0 │
│ 49 │ 479232000 │ 80357376 │ 39494346 │ 3115402 │ 0 │
└────┴───────────┴───────────┴──────────┴──────────┴──────────────┘
node.js v22.9.0
┌─────────┬───────────┬───────────┬──────────┬──────────┬──────────────┐
│ (index) │ rss │ heapTotal │ heapUsed │ external │ arrayBuffers │
├─────────┼───────────┼───────────┼──────────┼──────────┼──────────────┤
│ 0 │ 116215808 │ 41435136 │ 25531400 │ 8048371 │ 22695 │
│ 1 │ 141303808 │ 60833792 │ 31878784 │ 8427025 │ 395399 │
│ 2 │ 141426688 │ 60833792 │ 33129896 │ 8475121 │ 443495 │
│ 3 │ 141529088 │ 60833792 │ 36780744 │ 8625421 │ 593795 │
│ 4 │ 141611008 │ 60833792 │ 38565640 │ 8697565 │ 665939 │
│ 5 │ 141742080 │ 60833792 │ 40001504 │ 8757685 │ 726059 │
│ 6 │ 141881344 │ 60833792 │ 25050832 │ 8144461 │ 112835 │
│ 7 │ 141946880 │ 60833792 │ 25870688 │ 8180533 │ 148907 │
│ 8 │ 142110720 │ 60833792 │ 26054456 │ 8186545 │ 154919 │
│ 9 │ 142159872 │ 60833792 │ 26052448 │ 8186545 │ 154919 │
│ 10 │ 142565376 │ 60833792 │ 26089056 │ 8186545 │ 154919 │
│ 11 │ 142565376 │ 60833792 │ 26089704 │ 8186545 │ 154919 │
│ 12 │ 142553088 │ 60833792 │ 26793592 │ 8216605 │ 184979 │
│ 13 │ 142557184 │ 60833792 │ 28897192 │ 8300773 │ 269147 │
│ 14 │ 142561280 │ 60833792 │ 30332528 │ 8360893 │ 329267 │
│ 15 │ 142708736 │ 60833792 │ 34644648 │ 8535241 │ 503615 │
│ 16 │ 142708736 │ 60833792 │ 35195320 │ 8559289 │ 527663 │
│ 17 │ 143118336 │ 60833792 │ 36665448 │ 8619409 │ 587783 │
│ 18 │ 143118336 │ 60833792 │ 38581752 │ 8697565 │ 665939 │
│ 19 │ 143171584 │ 60833792 │ 40239920 │ 8763697 │ 732071 │
│ 20 │ 143171584 │ 60833792 │ 25368712 │ 8156485 │ 124859 │
│ 21 │ 143609856 │ 60833792 │ 25325408 │ 8156485 │ 124859 │
│ 22 │ 143613952 │ 60833792 │ 27319824 │ 8234641 │ 203015 │
│ 23 │ 143613952 │ 60833792 │ 29305408 │ 8318809 │ 287183 │
│ 24 │ 143613952 │ 60833792 │ 29347608 │ 8318809 │ 287183 │
│ 25 │ 143613952 │ 60833792 │ 29452416 │ 8324821 │ 293195 │
│ 26 │ 143863808 │ 60833792 │ 33142576 │ 8475121 │ 443495 │
│ 27 │ 151642112 │ 60833792 │ 29757632 │ 8336845 │ 305219 │
│ 28 │ 150396928 │ 60833792 │ 36674872 │ 8619409 │ 587783 │
│ 29 │ 149626880 │ 60833792 │ 27989096 │ 8264701 │ 233075 │
│ 30 │ 148156416 │ 60833792 │ 36273696 │ 8601373 │ 569747 │
│ 31 │ 148353024 │ 60833792 │ 25047592 │ 8144461 │ 112835 │
│ 32 │ 148353024 │ 60833792 │ 25374664 │ 8156485 │ 124859 │
│ 33 │ 148488192 │ 60833792 │ 26228168 │ 8192557 │ 160931 │
│ 34 │ 148508672 │ 60833792 │ 26306128 │ 8192557 │ 160931 │
│ 35 │ 148942848 │ 60833792 │ 26726448 │ 8210593 │ 178967 │
│ 36 │ 148946944 │ 60833792 │ 27465200 │ 8240653 │ 209027 │
│ 37 │ 149377024 │ 60833792 │ 28015872 │ 8264701 │ 233075 │
│ 38 │ 149364736 │ 60833792 │ 30886048 │ 8378929 │ 347303 │
│ 39 │ 149368832 │ 60833792 │ 32767168 │ 8457085 │ 425459 │
│ 40 │ 149393408 │ 60833792 │ 35272896 │ 8559289 │ 527663 │
│ 41 │ 149393408 │ 60833792 │ 37634600 │ 8655481 │ 623855 │
│ 42 │ 149520384 │ 60833792 │ 25521776 │ 8162497 │ 130871 │
│ 43 │ 149524480 │ 60833792 │ 30265736 │ 8354881 │ 323255 │
│ 44 │ 149725184 │ 60833792 │ 32586352 │ 8451073 │ 419447 │
│ 45 │ 149725184 │ 60833792 │ 35386936 │ 8565301 │ 533675 │
│ 46 │ 149716992 │ 60833792 │ 37929784 │ 8667505 │ 635879 │
│ 47 │ 149803008 │ 60833792 │ 40542280 │ 8775721 │ 744095 │
│ 48 │ 149807104 │ 60833792 │ 26256176 │ 8192557 │ 160931 │
│ 49 │ 149803008 │ 60833792 │ 28464616 │ 8282737 │ 251111 │
└─────────┴───────────┴───────────┴──────────┴──────────┴──────────────┘
Private Bytes growth is especially noticeable through Process Explorer
I don't know how else to diagnose the problem, and whether it can be controlled from the js side by changing the code. So far it seems to me that the problem is on the c++ side, and GC js does not have the ability to clean the result of c++ work. But this is just a guess.
Sometime, in real sample i got this
┌────┬────────────┬───────────┬───────────┬───────────┬──────────────┐
│ │ rss │ heapTotal │ heapUsed │ external │ arrayBuffers │
├────┼────────────┼───────────┼───────────┼───────────┼──────────────┤
│ 59 │ 1055535104 │ 185823232 │ 130950109 │ 41608829 │ 834568 │
│ 60 │ 1061908480 │ 183415808 │ 134106732 │ 43019328 │ 1800884 │
│ 61 │ 1052352512 │ 184605696 │ 136818108 │ 44005848 │ 2766100 │
│ 62 │ 859553792 │ 183218176 │ 131028896 │ 41668052 │ 835168 │
│ 63 │ 859701248 │ 182607872 │ 134146287 │ 43035443 │ 1799984 │
│ 64 │ 859705344 │ 183825408 │ 136843983 │ 44019463 │ 2765700 │
│ 65 │ 859705344 │ 185381888 │ 139535688 │ 44998740 │ 3729816 │
but Private bytes not cleared!
I wouldn't rule out the possibility that the problem could be in the binding module itself.
interestingly, the tensorflow core written in c++ is used in python, right? but over the entire existence of the library, the problem in the core itself could have been fixed long ago. so I thought that the leak could be in the tfjs_binding.node file
I will check my simple example from this issue on the browser version of tfjs with different backends and if there is no leak, then the problem is definitely in c++ and binding
// import tf from '@tensorflow/tfjs-node';
import tf from '@tensorflow/tfjs';
const width = 150;
const height = 10;
console.time('total time');
for (let i = 0; i < 5_000_000; i++) {
tf.tidy(() => {
const stateTensorTemp = tf.tensor2d([Array.from({ length: width * height }, () => Math.random())]);
const stateTensor = stateTensorTemp.reshape([1, width, height]);
});
if (i % 100_000 === 0) {
console.log(i / 100_000, JSON.stringify(process.memoryUsage()));
}
}
console.timeEnd('total time');
============================
Hi, looks like you are running TensorFlow.js in Node.js. To speed things up dramatically, install our node backend, visit https://github.com/tensorflow/tfjs-node for more details.
============================
bun 1.2.17
0 {"rss":200781824,"heapTotal":6578176,"heapUsed":234333,"external":40781,"arrayBuffers":0}
1 {"rss":274120704,"heapTotal":16424960,"heapUsed":13888716,"external":3156108,"arrayBuffers":0}
2 {"rss":243421184,"heapTotal":18538496,"heapUsed":20757949,"external":3180093,"arrayBuffers":0}
3 {"rss":242053120,"heapTotal":20684800,"heapUsed":27723473,"external":3186209,"arrayBuffers":0}
4 {"rss":250589184,"heapTotal":22814720,"heapUsed":34553435,"external":3204347,"arrayBuffers":0}
5 {"rss":258985984,"heapTotal":25026560,"heapUsed":41463301,"external":3216485,"arrayBuffers":0}
6 {"rss":267370496,"heapTotal":27141120,"heapUsed":48346089,"external":3234601,"arrayBuffers":0}
7 {"rss":275787776,"heapTotal":29434880,"heapUsed":55407831,"external":3240695,"arrayBuffers":0}
8 {"rss":284241920,"heapTotal":31613952,"heapUsed":62300737,"external":3258833,"arrayBuffers":0}
9 {"rss":288673792,"heapTotal":33809408,"heapUsed":6675443,"external":1706755,"arrayBuffers":0}
10 {"rss":288694272,"heapTotal":33809408,"heapUsed":13252819,"external":1748947,"arrayBuffers":0}
11 {"rss":288694272,"heapTotal":33809408,"heapUsed":19789191,"external":1755063,"arrayBuffers":0}
12 {"rss":288710656,"heapTotal":33809408,"heapUsed":26380689,"external":1767201,"arrayBuffers":0}
13 {"rss":288718848,"heapTotal":33809408,"heapUsed":32890879,"external":1773295,"arrayBuffers":0}
14 {"rss":288718848,"heapTotal":33809408,"heapUsed":39488131,"external":1785411,"arrayBuffers":0}
15 {"rss":288718848,"heapTotal":33809408,"heapUsed":45987447,"external":1809527,"arrayBuffers":0}
16 {"rss":288718848,"heapTotal":33825792,"heapUsed":52605253,"external":1827621,"arrayBuffers":0}
17 {"rss":288886784,"heapTotal":33858560,"heapUsed":9404262,"external":1731222,"arrayBuffers":0}
18 {"rss":288903168,"heapTotal":33858560,"heapUsed":16608579,"external":1731315,"arrayBuffers":0}
19 {"rss":288907264,"heapTotal":33858560,"heapUsed":23337352,"external":1761464,"arrayBuffers":0}
20 {"rss":288911360,"heapTotal":33858560,"heapUsed":30382502,"external":1779558,"arrayBuffers":0}
21 {"rss":288911360,"heapTotal":33858560,"heapUsed":37440058,"external":1809674,"arrayBuffers":0}
22 {"rss":288911360,"heapTotal":33858560,"heapUsed":44668094,"external":1821790,"arrayBuffers":0}
23 {"rss":288911360,"heapTotal":33858560,"heapUsed":51889532,"external":1833884,"arrayBuffers":0}
24 {"rss":289095680,"heapTotal":33858560,"heapUsed":9044581,"external":1719189,"arrayBuffers":0}
25 {"rss":289103872,"heapTotal":33858560,"heapUsed":15524904,"external":1737304,"arrayBuffers":0}
26 {"rss":289107968,"heapTotal":33858560,"heapUsed":22083884,"external":1743420,"arrayBuffers":0}
27 {"rss":289112064,"heapTotal":33858560,"heapUsed":28764170,"external":1743514,"arrayBuffers":0}
28 {"rss":289112064,"heapTotal":33858560,"heapUsed":35346452,"external":1755652,"arrayBuffers":0}
29 {"rss":289112064,"heapTotal":33858560,"heapUsed":41891640,"external":1767768,"arrayBuffers":0}
30 {"rss":289112064,"heapTotal":33858560,"heapUsed":48314252,"external":1773884,"arrayBuffers":0}
31 {"rss":289116160,"heapTotal":33858560,"heapUsed":54795878,"external":1786022,"arrayBuffers":0}
32 {"rss":289218560,"heapTotal":33858560,"heapUsed":11492500,"external":1719156,"arrayBuffers":0}
33 {"rss":289222656,"heapTotal":33858560,"heapUsed":18762050,"external":1719250,"arrayBuffers":0}
34 {"rss":289226752,"heapTotal":33858560,"heapUsed":25895874,"external":1773410,"arrayBuffers":0}
35 {"rss":289226752,"heapTotal":33858560,"heapUsed":33190716,"external":1791548,"arrayBuffers":0}
36 {"rss":289226752,"heapTotal":33858560,"heapUsed":40504662,"external":1827686,"arrayBuffers":0}
37 {"rss":289226752,"heapTotal":33858560,"heapUsed":47760859,"external":1875835,"arrayBuffers":0}
38 {"rss":289226752,"heapTotal":33858560,"heapUsed":54913777,"external":1906017,"arrayBuffers":0}
39 {"rss":289308672,"heapTotal":33858560,"heapUsed":11874783,"external":1713167,"arrayBuffers":0}
40 {"rss":289308672,"heapTotal":33858560,"heapUsed":19211453,"external":1713261,"arrayBuffers":0}
41 {"rss":289312768,"heapTotal":33858560,"heapUsed":26601803,"external":1713355,"arrayBuffers":0}
42 {"rss":289312768,"heapTotal":33858560,"heapUsed":33973673,"external":1713449,"arrayBuffers":0}
43 {"rss":289312768,"heapTotal":33858560,"heapUsed":41305981,"external":1719565,"arrayBuffers":0}
44 {"rss":289312768,"heapTotal":33858560,"heapUsed":48756209,"external":1731681,"arrayBuffers":0}
45 {"rss":289316864,"heapTotal":33858560,"heapUsed":6077492,"external":1701156,"arrayBuffers":0}
46 {"rss":289325056,"heapTotal":33858560,"heapUsed":13081859,"external":1731315,"arrayBuffers":0}
47 {"rss":289325056,"heapTotal":33858560,"heapUsed":19965080,"external":1755464,"arrayBuffers":0}
48 {"rss":289325056,"heapTotal":33858560,"heapUsed":27015606,"external":1761558,"arrayBuffers":0}
49 {"rss":289325056,"heapTotal":33858560,"heapUsed":34040650,"external":1779674,"arrayBuffers":0}
[298.57s] total time
node.js v22.9.0
0 {"rss":60428288,"heapTotal":39301120,"heapUsed":23935808,"external":7898115,"arrayBuffers":22683}
1 {"rss":91873280,"heapTotal":58437632,"heapUsed":39319744,"external":8612115,"arrayBuffers":736643}
2 {"rss":92012544,"heapTotal":58437632,"heapUsed":37008424,"external":8516115,"arrayBuffers":640643}
3 {"rss":92045312,"heapTotal":58437632,"heapUsed":34629624,"external":8420115,"arrayBuffers":544643}
4 {"rss":92123136,"heapTotal":58437632,"heapUsed":32261592,"external":8324115,"arrayBuffers":448643}
5 {"rss":92254208,"heapTotal":58437632,"heapUsed":29909128,"external":8228115,"arrayBuffers":352643}
6 {"rss":92327936,"heapTotal":58437632,"heapUsed":27558792,"external":8132115,"arrayBuffers":256643}
7 {"rss":92491776,"heapTotal":58437632,"heapUsed":25240456,"external":8036115,"arrayBuffers":160643}
8 {"rss":92487680,"heapTotal":58437632,"heapUsed":39381560,"external":8612115,"arrayBuffers":736643}
9 {"rss":92594176,"heapTotal":58437632,"heapUsed":37034416,"external":8516115,"arrayBuffers":640643}
10 {"rss":92758016,"heapTotal":58437632,"heapUsed":34684160,"external":8420115,"arrayBuffers":544643}
11 {"rss":92803072,"heapTotal":58437632,"heapUsed":32322856,"external":8324115,"arrayBuffers":448643}
12 {"rss":92844032,"heapTotal":58437632,"heapUsed":29969568,"external":8228115,"arrayBuffers":352643}
13 {"rss":92942336,"heapTotal":58437632,"heapUsed":27619176,"external":8132115,"arrayBuffers":256643}
14 {"rss":93069312,"heapTotal":58437632,"heapUsed":25256192,"external":8036115,"arrayBuffers":160643}
15 {"rss":93200384,"heapTotal":58437632,"heapUsed":39395072,"external":8612115,"arrayBuffers":736643}
16 {"rss":93175808,"heapTotal":58437632,"heapUsed":37048384,"external":8516115,"arrayBuffers":640643}
17 {"rss":93270016,"heapTotal":58437632,"heapUsed":34697992,"external":8420115,"arrayBuffers":544643}
18 {"rss":93286400,"heapTotal":58437632,"heapUsed":32329688,"external":8324115,"arrayBuffers":448643}
19 {"rss":93470720,"heapTotal":58437632,"heapUsed":29976368,"external":8228115,"arrayBuffers":352643}
20 {"rss":93556736,"heapTotal":58437632,"heapUsed":27625976,"external":8132115,"arrayBuffers":256643}
21 {"rss":93589504,"heapTotal":58437632,"heapUsed":25262992,"external":8036115,"arrayBuffers":160643}
22 {"rss":93868032,"heapTotal":58437632,"heapUsed":39401872,"external":8612115,"arrayBuffers":736643}
23 {"rss":93933568,"heapTotal":58437632,"heapUsed":37048552,"external":8516115,"arrayBuffers":640643}
24 {"rss":93933568,"heapTotal":58437632,"heapUsed":34698160,"external":8420115,"arrayBuffers":544643}
25 {"rss":94064640,"heapTotal":58437632,"heapUsed":32329856,"external":8324115,"arrayBuffers":448643}
26 {"rss":94130176,"heapTotal":58437632,"heapUsed":30010328,"external":8228115,"arrayBuffers":352643}
27 {"rss":94334976,"heapTotal":58437632,"heapUsed":27626144,"external":8132115,"arrayBuffers":256643}
28 {"rss":94334976,"heapTotal":58437632,"heapUsed":25306616,"external":8036115,"arrayBuffers":160643}
29 {"rss":94400512,"heapTotal":58437632,"heapUsed":39402040,"external":8612115,"arrayBuffers":736643}
30 {"rss":94535680,"heapTotal":58437632,"heapUsed":37048720,"external":8516115,"arrayBuffers":640643}
31 {"rss":94535680,"heapTotal":58437632,"heapUsed":34698328,"external":8420115,"arrayBuffers":544643}
32 {"rss":94760960,"heapTotal":58437632,"heapUsed":32330024,"external":8324115,"arrayBuffers":448643}
33 {"rss":94670848,"heapTotal":58437632,"heapUsed":29976704,"external":8228115,"arrayBuffers":352643}
34 {"rss":94736384,"heapTotal":58437632,"heapUsed":27626288,"external":8132115,"arrayBuffers":256643}
35 {"rss":94793728,"heapTotal":58437632,"heapUsed":25263304,"external":8036115,"arrayBuffers":160643}
36 {"rss":94887936,"heapTotal":58437632,"heapUsed":39402184,"external":8612115,"arrayBuffers":736643}
37 {"rss":95285248,"heapTotal":58437632,"heapUsed":37048864,"external":8516115,"arrayBuffers":640643}
38 {"rss":95281152,"heapTotal":58437632,"heapUsed":34698856,"external":8420115,"arrayBuffers":544643}
39 {"rss":95420416,"heapTotal":58437632,"heapUsed":32330552,"external":8324115,"arrayBuffers":448643}
40 {"rss":95420416,"heapTotal":58437632,"heapUsed":29977208,"external":8228115,"arrayBuffers":352643}
41 {"rss":95555584,"heapTotal":58437632,"heapUsed":27626816,"external":8132115,"arrayBuffers":256643}
42 {"rss":95555584,"heapTotal":58437632,"heapUsed":25263832,"external":8036115,"arrayBuffers":160643}
43 {"rss":95686656,"heapTotal":58437632,"heapUsed":39402712,"external":8612115,"arrayBuffers":736643}
44 {"rss":95686656,"heapTotal":58437632,"heapUsed":37049392,"external":8516115,"arrayBuffers":640643}
45 {"rss":95686656,"heapTotal":58437632,"heapUsed":34699000,"external":8420115,"arrayBuffers":544643}
46 {"rss":95875072,"heapTotal":58437632,"heapUsed":32330696,"external":8324115,"arrayBuffers":448643}
47 {"rss":95875072,"heapTotal":58437632,"heapUsed":29977376,"external":8228115,"arrayBuffers":352643}
48 {"rss":96010240,"heapTotal":58437632,"heapUsed":27626984,"external":8132115,"arrayBuffers":256643}
49 {"rss":96010240,"heapTotal":58437632,"heapUsed":25264000,"external":8036115,"arrayBuffers":160643}
total time: 14:02.745 (m:ss.mmm)
Without using a browser, it was possible to prove that Private Bytes is stable and not growing.
Look at Heap Total after 9 iterations on Bun - more stable than with c++ backend.
Node js is incredibly stable, there is a small increase within the margin of error.
Do I understand correctly that it will take a lot of time to fix the problem in c++ and binding?
import tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-backend-wasm';
await tf.setBackend('wasm');
const width = 150;
const height = 10;
console.time('total time');
for (let i = 0; i < 5_000_000; i++) {
tf.tidy(() => {
const stateTensorTemp = tf.tensor2d([Array.from({ length: width * height }, () => Math.random())]);
const stateTensor = stateTensorTemp.reshape([1, width, height]);
stateTensorTemp.dispose();
stateTensor.dispose();
});
if (i % 100_000 === 0) {
console.log(i / 100_000, JSON.stringify(process.memoryUsage()));
}
}
console.timeEnd('total time');
bun 1.2.18
0 {"rss":216272896,"heapTotal":7141376,"heapUsed":7886474,"external":19970826,"arrayBuffers":17208098}
1 {"rss":286535680,"heapTotal":16714752,"heapUsed":56770335,"external":52659983,"arrayBuffers":33955840}
2 {"rss":247152640,"heapTotal":16714752,"heapUsed":65073238,"external":61573046,"arrayBuffers":42862096}
3 {"rss":237338624,"heapTotal":16714752,"heapUsed":73493676,"external":70801820,"arrayBuffers":52146304}
4 {"rss":237441024,"heapTotal":16714752,"heapUsed":44122332,"external":43138172,"arrayBuffers":24482656}
5 {"rss":237424640,"heapTotal":16714752,"heapUsed":56741900,"external":53062460,"arrayBuffers":34406944}
6 {"rss":237457408,"heapTotal":16714752,"heapUsed":65097144,"external":63011144,"arrayBuffers":44355616}
7 {"rss":237633536,"heapTotal":16714752,"heapUsed":73579964,"external":72843980,"arrayBuffers":54188464}
8 {"rss":237633536,"heapTotal":16714752,"heapUsed":48277212,"external":45308348,"arrayBuffers":26652832}
9 {"rss":237658112,"heapTotal":16714752,"heapUsed":56718772,"external":55385060,"arrayBuffers":36729520}
10 {"rss":237670400,"heapTotal":16714752,"heapUsed":69399884,"external":65242268,"arrayBuffers":46586752}
11 {"rss":237666304,"heapTotal":16714752,"heapUsed":73494552,"external":37907804,"arrayBuffers":19252288}
12 {"rss":237678592,"heapTotal":16714752,"heapUsed":48277564,"external":47850380,"arrayBuffers":29194864}
13 {"rss":237694976,"heapTotal":16714752,"heapUsed":60843548,"external":57933164,"arrayBuffers":39277648}
14 {"rss":237694976,"heapTotal":16714752,"heapUsed":69313724,"external":68259788,"arrayBuffers":49604272}
15 {"rss":237694976,"heapTotal":16714752,"heapUsed":44177884,"external":40577852,"arrayBuffers":21922336}
16 {"rss":237694976,"heapTotal":16714752,"heapUsed":52468708,"external":50611892,"arrayBuffers":31956352}
17 {"rss":237703168,"heapTotal":16714752,"heapUsed":65097284,"external":60627620,"arrayBuffers":41972080}
18 {"rss":237707264,"heapTotal":16714752,"heapUsed":73511004,"external":70344620,"arrayBuffers":51689104}
19 {"rss":237719552,"heapTotal":16714752,"heapUsed":44104412,"external":43034540,"arrayBuffers":24379024}
20 {"rss":237740032,"heapTotal":16714752,"heapUsed":56691668,"external":52946660,"arrayBuffers":34291120}
21 {"rss":237744128,"heapTotal":16714752,"heapUsed":65095036,"external":63261068,"arrayBuffers":44605552}
22 {"rss":237760512,"heapTotal":16714752,"heapUsed":73433156,"external":35676668,"arrayBuffers":17021152}
23 {"rss":237776896,"heapTotal":16714752,"heapUsed":48380340,"external":45704612,"arrayBuffers":27049072}
24 {"rss":237776896,"heapTotal":16714752,"heapUsed":56777164,"external":55439900,"arrayBuffers":36784384}
25 {"rss":237776896,"heapTotal":16714752,"heapUsed":69264044,"external":65376380,"arrayBuffers":46720864}
26 {"rss":237776896,"heapTotal":16714752,"heapUsed":73494552,"external":37926092,"arrayBuffers":19270576}
27 {"rss":237776896,"heapTotal":16714752,"heapUsed":48302972,"external":47752844,"arrayBuffers":29097328}
28 {"rss":237776896,"heapTotal":16714752,"heapUsed":60915500,"external":58012412,"arrayBuffers":39356896}
29 {"rss":237776896,"heapTotal":16714752,"heapUsed":69302148,"external":67936724,"arrayBuffers":49281184}
30 {"rss":237776896,"heapTotal":16714752,"heapUsed":44126900,"external":40187732,"arrayBuffers":21532192}
31 {"rss":237776896,"heapTotal":16714752,"heapUsed":52576284,"external":50307068,"arrayBuffers":31651552}
32 {"rss":237776896,"heapTotal":16714752,"heapUsed":60917260,"external":60371564,"arrayBuffers":41716048}
33 {"rss":237785088,"heapTotal":16714752,"heapUsed":73400748,"external":70442156,"arrayBuffers":51786640}
34 {"rss":237785088,"heapTotal":16714752,"heapUsed":44147068,"external":42632204,"arrayBuffers":23976688}
35 {"rss":237801472,"heapTotal":16714752,"heapUsed":56720044,"external":52617452,"arrayBuffers":33961936}
36 {"rss":237883392,"heapTotal":16714752,"heapUsed":65106236,"external":62407628,"arrayBuffers":43752112}
37 {"rss":237887488,"heapTotal":16714752,"heapUsed":73529100,"external":72520892,"arrayBuffers":53865376}
38 {"rss":237887488,"heapTotal":16714752,"heapUsed":48277132,"external":45046220,"arrayBuffers":26390704}
39 {"rss":237887488,"heapTotal":16714752,"heapUsed":56718700,"external":55000988,"arrayBuffers":36345472}
40 {"rss":237887488,"heapTotal":16714752,"heapUsed":69251860,"external":65144756,"arrayBuffers":46489216}
41 {"rss":237887488,"heapTotal":16714752,"heapUsed":73492648,"external":37590812,"arrayBuffers":18935296}
42 {"rss":237887488,"heapTotal":16714752,"heapUsed":48307996,"external":47405372,"arrayBuffers":28749856}
43 {"rss":237887488,"heapTotal":16714752,"heapUsed":60767052,"external":57555212,"arrayBuffers":38899696}
44 {"rss":237887488,"heapTotal":16714752,"heapUsed":69276460,"external":67686764,"arrayBuffers":49031248}
45 {"rss":237887488,"heapTotal":16714752,"heapUsed":44107340,"external":40059692,"arrayBuffers":21404176}
46 {"rss":237887488,"heapTotal":16714752,"heapUsed":52519548,"external":50282684,"arrayBuffers":31627168}
47 {"rss":237887488,"heapTotal":16714752,"heapUsed":60998268,"external":60219164,"arrayBuffers":41563648}
48 {"rss":237891584,"heapTotal":16714752,"heapUsed":73607820,"external":69923996,"arrayBuffers":51268480}
49 {"rss":237891584,"heapTotal":16714752,"heapUsed":44124636,"external":42223772,"arrayBuffers":23568256}
[289.04s] total time
It's hard to say if there is a leak, because Bun ate the maximum 4.8gb of Private Bytes. But the Working Set did not grow, I did not see a memory leak. And look at the incredibly stable heapTotal.
node.js v22.9.0
0 {"rss":63725568,"heapTotal":39686144,"heapUsed":25844536,"external":25141349,"arrayBuffers":447314}
1 {"rss":91910144,"heapTotal":56991744,"heapUsed":38562136,"external":25472726,"arrayBuffers":778651}
2 {"rss":91828224,"heapTotal":56991744,"heapUsed":36206920,"external":25376726,"arrayBuffers":682651}
3 {"rss":92004352,"heapTotal":56991744,"heapUsed":33851184,"external":25280726,"arrayBuffers":586651}
4 {"rss":92123136,"heapTotal":56991744,"heapUsed":31498432,"external":25184726,"arrayBuffers":490651}
5 {"rss":92266496,"heapTotal":56991744,"heapUsed":29147992,"external":25088726,"arrayBuffers":394651}
6 {"rss":92479488,"heapTotal":56991744,"heapUsed":26786992,"external":24992726,"arrayBuffers":298651}
7 {"rss":92381184,"heapTotal":56991744,"heapUsed":24445112,"external":24896726,"arrayBuffers":202651}
8 {"rss":92692480,"heapTotal":56991744,"heapUsed":38618312,"external":25472726,"arrayBuffers":778651}
9 {"rss":92594176,"heapTotal":56991744,"heapUsed":36263144,"external":25376726,"arrayBuffers":682651}
10 {"rss":92831744,"heapTotal":56991744,"heapUsed":33943864,"external":25280726,"arrayBuffers":586651}
11 {"rss":92311552,"heapTotal":56991744,"heapUsed":31588816,"external":25184726,"arrayBuffers":490651}
12 {"rss":92880896,"heapTotal":56991744,"heapUsed":29210968,"external":25088726,"arrayBuffers":394651}
13 {"rss":92876800,"heapTotal":56991744,"heapUsed":26842792,"external":24992726,"arrayBuffers":298651}
14 {"rss":93167616,"heapTotal":56991744,"heapUsed":24489664,"external":24896726,"arrayBuffers":202651}
15 {"rss":93036544,"heapTotal":56991744,"heapUsed":38633672,"external":25472726,"arrayBuffers":778651}
16 {"rss":93036544,"heapTotal":56991744,"heapUsed":36270592,"external":25376726,"arrayBuffers":682651}
17 {"rss":93302784,"heapTotal":56991744,"heapUsed":33951064,"external":25280726,"arrayBuffers":586651}
18 {"rss":93245440,"heapTotal":56991744,"heapUsed":31568384,"external":25184726,"arrayBuffers":490651}
19 {"rss":93466624,"heapTotal":56991744,"heapUsed":29217800,"external":25088726,"arrayBuffers":394651}
20 {"rss":93417472,"heapTotal":56991744,"heapUsed":26849592,"external":24992726,"arrayBuffers":298651}
21 {"rss":93757440,"heapTotal":56991744,"heapUsed":24496464,"external":24896726,"arrayBuffers":202651}
22 {"rss":93728768,"heapTotal":56991744,"heapUsed":38640472,"external":25472726,"arrayBuffers":778651}
23 {"rss":93728768,"heapTotal":56991744,"heapUsed":36277392,"external":25376726,"arrayBuffers":682651}
24 {"rss":93732864,"heapTotal":56991744,"heapUsed":33921680,"external":25280726,"arrayBuffers":586651}
25 {"rss":93863936,"heapTotal":56991744,"heapUsed":31568552,"external":25184726,"arrayBuffers":490651}
26 {"rss":94007296,"heapTotal":56991744,"heapUsed":29217968,"external":25088726,"arrayBuffers":394651}
27 {"rss":94007296,"heapTotal":56991744,"heapUsed":26849760,"external":24992726,"arrayBuffers":298651}
28 {"rss":94142464,"heapTotal":56991744,"heapUsed":24496632,"external":24896726,"arrayBuffers":202651}
29 {"rss":94142464,"heapTotal":56991744,"heapUsed":38640640,"external":25472726,"arrayBuffers":778651}
30 {"rss":94146560,"heapTotal":56991744,"heapUsed":36321112,"external":25376726,"arrayBuffers":682651}
31 {"rss":94412800,"heapTotal":56991744,"heapUsed":33921848,"external":25280726,"arrayBuffers":586651}
32 {"rss":94339072,"heapTotal":56991744,"heapUsed":31568720,"external":25184726,"arrayBuffers":490651}
33 {"rss":94547968,"heapTotal":56991744,"heapUsed":29218136,"external":25088726,"arrayBuffers":394651}
34 {"rss":94576640,"heapTotal":56991744,"heapUsed":26849904,"external":24992726,"arrayBuffers":298651}
35 {"rss":94650368,"heapTotal":56991744,"heapUsed":24496776,"external":24896726,"arrayBuffers":202651}
36 {"rss":94806016,"heapTotal":56991744,"heapUsed":38671840,"external":25472726,"arrayBuffers":778651}
37 {"rss":95068160,"heapTotal":56991744,"heapUsed":36277704,"external":25376726,"arrayBuffers":682651}
38 {"rss":95072256,"heapTotal":56991744,"heapUsed":33922376,"external":25280726,"arrayBuffers":586651}
39 {"rss":95072256,"heapTotal":56991744,"heapUsed":31569248,"external":25184726,"arrayBuffers":490651}
40 {"rss":95334400,"heapTotal":56991744,"heapUsed":29218640,"external":25088726,"arrayBuffers":394651}
41 {"rss":95334400,"heapTotal":56991744,"heapUsed":26850432,"external":24992726,"arrayBuffers":298651}
42 {"rss":95338496,"heapTotal":56991744,"heapUsed":24530904,"external":24896726,"arrayBuffers":202651}
43 {"rss":95404032,"heapTotal":56991744,"heapUsed":38672368,"external":25472726,"arrayBuffers":778651}
44 {"rss":95404032,"heapTotal":56991744,"heapUsed":36278232,"external":25376726,"arrayBuffers":682651}
45 {"rss":95678464,"heapTotal":56991744,"heapUsed":33958704,"external":25280726,"arrayBuffers":586651}
46 {"rss":95678464,"heapTotal":56991744,"heapUsed":31602992,"external":25184726,"arrayBuffers":490651}
47 {"rss":95744000,"heapTotal":56991744,"heapUsed":29218808,"external":25088726,"arrayBuffers":394651}
48 {"rss":95744000,"heapTotal":56991744,"heapUsed":26850600,"external":24992726,"arrayBuffers":298651}
49 {"rss":95817728,"heapTotal":56991744,"heapUsed":24497472,"external":24896726,"arrayBuffers":202651}
total time: 14:15.252 (m:ss.mmm)
I'm confused, looks like slow growth. Process Explorer also shows a slow increase in Private Bytes and Working Set. Maybe this is normal?
More experiments led me to the reshape function. It may be the cause of the memory leaks.
It seems incredible, but I observe memory leak problems when initializing tensors with shape
tf.tensor4d(new Float32Array([1, 2, 3, ..., N]), [1, H, W, C])
Why does this surprise me? Because without knowing the source code in JS or C++, I think it's use to the reshape function, since it's changing the shape from a flat structure to a 4D shape.
I don't see such problems if the incoming data form is already correctly shaped
tf.tensor4d([[[[...]]]])
What do you think?