Unbounded actors created for log when console is stuck

[neeraj9]

While using lager 06f676aad8ed823b7838a70288cd4d13b5a25ede one of the production cluster node started consuming a lot of memory. Upon detailed inspection, which is covered as part of my bug report https://bugs.erlang.org/browse/ERL-710 it was found that lager (https://github.com/erlang-lager/lager/blob/bf60290101306b3cc26544d0d181a5cff8931e82/include/lager.hrl#L137L156) spawns unbounded actors. This is an issue when gen_event:which_handlers(lager_event) is stuck, which happened in this particular case.

-define(INT_LOG(Level, Format, Args),
    Self = self(),
    %% do this in a spawn so we don't cause a deadlock calling gen_event:which_handlers
    %% from a gen_event handler
    spawn(fun() ->
            case catch(gen_event:which_handlers(lager_event)) of

Although the real issue is still under investigation (as part of https://bugs.erlang.org/browse/ERL-710), but it is better to protect against such issues by keeping spawn under control and bounded. There should be a mechanism to break the circuit when io write is stuck instead of allowing unbounded actors to be spawned for each log.

[Vagabond]

Yeah, that's fair. Usually I'd recommend disabling console logging in production because the console is generally a problematic device to work with because of exactly this issue.

[jadeallenx]

It's a fair point that lager ought to use a proxy process rather than a blind spawn in this situation.

[dingdongnigetou]

any improvment ? we aslo get this issue :(