elixir_agent
elixir_agent copied to clipboard
[Bug/Request] Change or clearly communicate risks with default logging behaviour
First of all I just wanted to say thank you for creating and maintaining this package. In our testing so far it's worked beautifully and manual instrumentation APIs are very neat and handy!
Onto the bug/request!
Describe the bug
- Elixir 1.9 OTP 23
- Install new_relic_agent version 1.12.7
- Setup barebones config in
releases.exs
config :new_relic_agent,
app_name: System.get_env("NEW_RELIC_APP_NAME"),
license_key: System.get_env("NEW_RELIC_API_KEY")
- Deploy and run application, integration works perfectly 👍
- After a couple of weeks the application dies with an erlang/BEAM exception, something about being out of memory (sorry we didn't capture the original error)
- Upon investigating the server we found the log file (
/tmp/new_relic.log
) grows infinitely? big, or at least consumed all the remaining HDD space on the server (~ 10GB) - Nothing in the README mentions that this might happen, and it wasn't immediately apparent in the hexdocs (although it's kinda implied behaviour in the logger)
I'm not 100% sure if this is intentional behaviour or if we've just stumbled upon some sort of logging bug?
If it is intentional, then in my opinion I think the default behaviour is risky as it has the potential to crash production systems. Personally I would prefer making the risks clear in the documentation somewhere, or even changing the default logging behaviour.
Possible resolutions I fully understand some of these might be tricky as they could be a breaking change, and as I'm not familiar with the internals of new relic or the agent I could be proposing something that isn't feasible.
- Find and fix the logging bug (if that's what this is?)
- Update the README and/or hexdocs to make it clear that the default logging behaviour is risky in production
- Limit the size of the log file (rolling log?) and have a "reasonable" default size
- Change the default logging mechanism
- Add compile/run time warnings when logging configuration could result in large log files
I'm more than happy to get my hands dirty and submit a PR if there's a preferred way to tackle this.