pyroscope-java
pyroscope-java copied to clipboard
pyroscope-io
Be able to set an agent log format with a configuration option
Our scenario
Hello. We are using docker as the underlying technology to deploy apps and the observability stack. For all the logs we decide to use JSON format that is sent to our Loki backend. We are trying to adopt the pyroscope Java Otel extension (that implicitly has the Java Agent) to send profiling data to our Pyroscope server.
The issue
During our testing, when we are sending the Pyroscope backend server down, all Java apps using the extension are producing lot of logs that don't align with the rest of echosystem. In our case is JSON. This is the kind of output we are getting from the agent logger in the docker console:
2024-03-21 11:37:15.292 [ERROR] Error uploading snapshot: 502 <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
Currently the agent only has this log parameter PYROSCOPE_LOG_LEVEL which is not enough for this case.
Possible solutions
- As discussed in this thread a custom logger interface implementation can be done:
https://github.com/grafana/pyroscope-java/blob/f73ebfaae1fcdfdf61d3418a78807c5b2b0[…]/agent/src/main/java/io/pyroscope/javaagent/PyroscopeAgent.java https://github.com/grafana/pyroscope-java/blob/82b1daba0dc05b1851e6b96f2d760ab7901[…]0c49/agent/src/main/java/io/pyroscope/javaagent/api/Logger.java
However, that would require initializing the profiler outside the otel extension, and this is not available when it's running as a javagent.
- Add a flag
PYROSCOPE_LOG_FORMATto set the logger format. it could be JSON or LOGFTM to align with rest of Grafana stack.
This might be related to https://github.com/grafana/pyroscope-java/issues/126.
