Possible memory leak with longStackTraces

[jordanrastrick]

1. What version of bluebird is the issue happening on?

3.5.0

2. What platform and version? (For example Node.js 0.12 or Google Chrome 32)

Node 6.11.0

3. Did this issue happen with earlier version of bluebird?

(Unknown)

Description:

We had a process under development that was continually crashing with an out of memory error and after some investigation discovered the problem seemed to be longStackTraces, which we usually have switched on for ease of debugging.

I understand it is expected behaviour that longStackTraces causes substantially increased memory use when enabled, however it appeared to be causing the allocated heap size to grow without bound in our case where I suspect it is probably not intended to do so.

The minimal example to replicate that I have so far (N.B. I have only made brief attempts to investigate the problem absent use of the mysql library, where it didn't seem to occur).

'use strict';

var _ = require('lodash');
var Promise = require('bluebird');
Promise.config({
  longStackTraces: true,
});

var mysql = require('mysql');
Promise.promisifyAll(mysql);

var connection = mysql.createConnection({
  host: '<insert endpoint here>',
  user: 'user',
  password: 'password',
  database: 'dev',
  multipleStatements: false,
});

Promise.promisifyAll(connection);

connection.connectAsync()
    .then(function() {
      return Promise.each(_.range(20001), function(i) {
        if (i % 1000 === 0) {
          console.log(i, 'iterations', process.memoryUsage().heapUsed, 'memory');
        } 
        return connection.queryAsync('SELECT * FROM Testing LIMIT 200');
      });
    });

Typical output looks like:

0 'iterations' 33959512 'memory'
1000 'iterations' 45899712 'memory'
2000 'iterations' 57277592 'memory'
3000 'iterations' 80591528 'memory'
4000 'iterations' 91669376 'memory'
5000 'iterations' 102731272 'memory'
6000 'iterations' 113795168 'memory'
7000 'iterations' 124875248 'memory'
8000 'iterations' 135929400 'memory'
9000 'iterations' 146991888 'memory'
10000 'iterations' 158073192 'memory'
11000 'iterations' 182009640 'memory'
12000 'iterations' 192961040 'memory'
13000 'iterations' 204009848 'memory'
14000 'iterations' 215088672 'memory'
15000 'iterations' 226137936 'memory'
16000 'iterations' 237189504 'memory'
17000 'iterations' 248241968 'memory'
18000 'iterations' 259293296 'memory'
19000 'iterations' 282598992 'memory'
20000 'iterations' 293650240 'memory'

In contrast, with longStackTraces: false, the output does not show unbounded growth:

0 'iterations' 14797992 'memory'
1000 'iterations' 28803472 'memory'
2000 'iterations' 33358200 'memory'
3000 'iterations' 50655120 'memory'
4000 'iterations' 24730408 'memory'
5000 'iterations' 20349328 'memory'
6000 'iterations' 15804040 'memory'
7000 'iterations' 13750176 'memory'
8000 'iterations' 29893576 'memory'
9000 'iterations' 26801824 'memory'
10000 'iterations' 23831392 'memory'
11000 'iterations' 20059536 'memory'
12000 'iterations' 18392912 'memory'
13000 'iterations' 17217504 'memory'
14000 'iterations' 33311472 'memory'
15000 'iterations' 29926456 'memory'
16000 'iterations' 28783016 'memory'
17000 'iterations' 25796360 'memory'
18000 'iterations' 24840016 'memory'
19000 'iterations' 24532224 'memory'
20000 'iterations' 27535432 'memory'

Thanks,

Jordan

[axetroy]

I think I have meet same proble with you. Possible memory leak

[benjamingr]

@petkaantonov cc

[petkaantonov]

.each forms a chain of the promises in the array that behaves exactly like a normal manually formed promise chain. With long stack traces the entire chain is always kept in memory so that the chain steps can be traced back. So the memory use is not technically unbounded, it is the same memory use you would get from chaining 20k promises manually.

I am not sure this is fixable because any other implementation would not make cancellation propagate through the chain properly.

[Cleod9]

Possibly related: https://github.com/petkaantonov/bluebird/issues/1478

[Strandedpirate]

Confirmed. Don't use long stack traces with bluebird. We had to disable it in all of our production apps.

[Shingyx]

Has this improved since 3.5.2, which includes https://github.com/petkaantonov/bluebird/pull/1544? I fixed a similar issue where sometimes old stack traces were unnecessarily occupying memory.

EDIT: Never mind, I think the issue I had is unrelated.

[sdd]

I've also experienced this. We have some heavy promise use and were leaking ~5Mb/API request(!) Setting BLUEBIRD_LONG_STACK_TRACES=0 resloved the problem.