Memory leak on large files with streaming middleware union

[zckevin]

I'm trying to serve large files(1 GiB) using http-server, the memory usage is abnormally high, so I got the heap profile and saw lots of 65535 KiB Buffer without being GCed.

But when I switch to connect instead of union, everything is all right then, so I'm pretty sure that something wrong is happening in union.

http-server v0.7.5 node v0.10.32

[stianeikeland]

I'm seeing the same thing here. Requesting/streaming a large file (ex: a movie) will make it consume Gbs of memory. If I request the same file again the memory usage will double. Doesn't seem to be released, in the end the node process is killed by the OS when it runs out of available memory.

[roccomuso]

+1 I'm having the same issue when trying to transfer big files from devices with low-memory (like the Raspberry Pi).

Couldn't be related to this the issue? http://stackoverflow.com/questions/15001822/sending-large-image-data-over-http-in-node-js

[zckevin]

Pull request https://github.com/flatiron/union/pull/55, but seems this project is not maintained any more, a new dependency should be chosen.

[indexzero]

Yes. union is not maintained anymore on purpose since it predates streams2 in Node core. A PR that removes it would be welcome.

[derhuerst]

@indexzero

union is not maintained anymore […]. A PR that removes it would be welcome.

I'd be happy to do this. One solution might be to just use Express for the middleware-like behavior. A more lightweight solution might be to used middl. What do you think?

[hdf]

Problem persists on http-server 0.9.0 and node 6.7.0. Which is too bad, as otherwise this is a great little tool. Have to use HFS instead (http://www.rejetto.com/hfs/?f=dl). :( (For some reason HFS is faster as well, by about 20%.)

[jcrugzz]

This should be fixed with in [email protected]

[thornjad]

Is this confirmed fixed? Can this issue be closed?

[tremby]

I just tried transferring a huge file after running http-server 0.11.1 with no special command-line arguments and the memory usage stayed reasonable throughout. I don't know whether this answers the question, since there is no example code in this ticket and I don't know if union is being used by default.

[thornjad]

union is still being used (though maybe it shouldn't be), but this issue looks to have been fixed with flatiron/union#54

[thornjad]

We will eventually be removing union (see #483), so hopefully this issue will go away.

[github-actions[bot]]

This issue has been inactive for 180 days