Hight memory usage, process doesn't shut down properly

[ddcovery]

I am studying using vibe.d in place of node and I begun to do some tests

My platform:

• Ubuntu 20.04,
• dmd64, v2.090.1
• dub 1.19.0

The first one is to "stress" the "hello world" example:

• I run server process executing dub (vibe-d 0.8.6, vibe-core 1.9.2 and dependencies are installed and then process starts running)

• I navigate to localhost:8080 on firefox and then I hold CTRL+F5 to force continuous refreshing. Less than 2 minutes aftger, the process is consuming about 11GB of memory... If I stop refreshing the "hello world" page, memory doesn't recover (continues growing about 10 seconds).

Finally, If I stop runing dub (CTRL+C) and consoles logs "Received signal 2. Shutting down.", but process continues running and I need to kill it (kill -s 9 ...).

If I run the compiled one (without dub) and I try to finish then using CTRL+C (without any firefox request), process logs on console "Received signa 2. Shutting down.", but it doesn't finish properly... I can press CTRL+C repetidely and message repetidely is shown on terminal... but process doesn't stop (and memory continues without recovering).

Any sugestion about how to solve the problem?

[rikkimax]

Vibe.d by itself does not interact with the GC (from a quick search).

import core.memory; Basically GC.collect; and a GC.minimize; will get it to decrease in memory usage reflected by the OS.

[ddcovery]

Thanks, How exactly have I to include GC in this code ? I imported the library

import vibe.vibe;
import core.memory;

void main()
{
        auto settings = new HTTPServerSettings;
        settings.port = 8080;
        settings.bindAddresses = ["::1", "127.0.0.1"];
        listenHTTP(settings, &hello);

        logInfo("Please open http://127.0.0.1:8080/ in your browser.");
        runApplication();
}

void hello(HTTPServerRequest req, HTTPServerResponse res)
{
        res.writeBody("Hello, World!");
}

But memory leak continues...

• Is there any way to use vibe.d without GC.
• How GC must be imported and used in the example app to work properly? (explicit call to GC.collect after each res.writeBody() doesn't solve de problem.

Thank you

[rikkimax]

Is there any way to use vibe.d without GC.

No, that would only make the situation worse.

How GC must be imported and used in the example app to work properly? (explicit call to GC.collect after each res.writeBody() doesn't solve de problem.

Inside of a timer use:

import core.memory;
GC.collect;
GC.minimize;

The one that matters is the GC.minimize; which will tell the GC to give back memory to the OS.

Make sure it has a fairly high delay between each execution, to minimize performance cost to doing this.

[ddcovery]

Thankou rikkimax, but memory usage continues growing...

This is the code now (I suppose that timer must be stopped in some moment, but this is not important for the first test):

import vibe.vibe;
import std.stdio;
void main()
{
  Timer  timer = setTimer(250.msecs, {
    import core.memory;
    GC.collect;
    GC.minimize;
    writeln("Timer tick");
  }, true);

  auto settings = new HTTPServerSettings;
  settings.port = 8080;
  settings.bindAddresses = ["::1", "127.0.0.1"];
  listenHTTP(settings, &hello);

  logInfo("Please open http://127.0.0.1:8080/ in your browser.");
  runApplication();
}

void hello(HTTPServerRequest req, HTTPServerResponse res)
{
        res.writeBody("Hello, World!");
}

After 5-10 seconds of CTRL+F5 on firefox the application memory starts growing and never decreases.

[wilzbach]

Could you try the newly released 0.9.0 version of vibe.d to see whether the issue still persists?

[ddcovery]

The memory problem persists.

This is a gif showing what happens just refreshing pages (clicking on refresh button)... all works properly (firefox/chrome). (note: The "timer tick" message is a timer running GC each 250ms)

And this is the result if I hold CTRL+F5 on Firefox

[FeepingCreature]

Can confirm this issue. We have a Vibe-based service and if I even just hammer a static route with repeated wgets, the memory usage quickly creeps up.

This is a GC failure. If the GC was operating correctly, it should just be reusing the same memory over and over, collect/minimize be damned.

edit: May have hit some sort of steady state around 750MB.. gonna leave it hammering a bit and see if it fills up again.

edit: It's once again creeping upwards. I think sometimes it manages to free a large GC area, and then it spends some time filling it up again. Small leaks adding up by keeping large data structures alive?

edit: Poking at the memory dump with strings seems to show it's largely a long list of HTTP request bodies. May be reasonable if the GC is reusing that memory.

edit: Hm. May be a memory fragmentation issue?

[ddcovery]

May be a problem with SIGPIPE management?

Using CTRL+F5 repetidely causes Firefox to abandon sockets causing server to receive a SIGPIPE when tries to write the response.

vibe.d manages this signal manually (to avoid process termination) but... -Is it possible that something is not correctly managed (i.e. some callback) when a SIGPIPE is raised and function managing the response doesn't end nicely?

edit: From linux write man page https://linux.die.net/man/2/write

EPIPE fd is connected to a pipe or socket whose reading end is closed. When this happens the writing process will also receive a SIGPIPE signal. (Thus, the write return value is seen only if the program catches, blocks or ignores this signal.)

[ddcovery]

Hi wilzbach

I tested today with last version and there is not memory problem. May be I didn't test correctly the version 0.9.0 25 days ago.

I suppose Issue can be closed.

Thanks a lot

[FeepingCreature]

It may be helpful if there was a way, mode or flag to overwrite a request with BADFOOD or whatever on completion, or force-free the region, or maybe mprotect it as PROT_NONE to salt the earth. I suspect we have string pointers into the request object keeping the entire request alive. A "crash on access to old request memory" mode would be handy to ferret them out.

[kookman]

I wonder if #2484 is the root problem here? Would be interesting to re-test with this now fixed.