pg-bifrost
pg-bifrost copied to clipboard
If DATADOG_HOST points to a socket file, and the file goes away, bifrost never notices..
If you set DATADOG_HOST to point to a unix socket file (ie, /var/run/dogstatsd/dsd.sock) and for some reason that file goes away (lets say you upgrade the datadog agent and it restarts the process) - bifrost never notices that it went away and silently fails forever. Because of this silent failure, you lose all your metrics .. but you don't know why.
We shouldn't crash when this happens .. but, we should at some point try to get that socket back. Maybe every 30s? 60s? Then we should throw an error in the logs to help the user diagnose the issue.