spark-monitoring icon indicating copy to clipboard operation
spark-monitoring copied to clipboard

Adding newline option to JSONLayout

Open ginaluca opened this issue 2 years ago • 0 comments

Hi all,

We use spark-monitoring in our Databricks workspaces, but in a slightly non-standard way:

  • Metrics are collected and pushed to Log Analytics, as per project design;
  • Logs, on the other side, follow a different route: we want them formatted in JSON, but they are written to a shared storage (due to our overall enterprise architecture) and then eventually exported to our logging backend.

For this reason, we only need the JSONLayout piece of spark-monitoring when it comes to logs. We can do this by setting

log4j.appender.publicFile.layout=com.microsoft.pnp.logging.JSONLayout

in log4j.properties, but we have an additional requirement on JSONLayout: it must be configurable to add a newline at the end of the serialised string because the software which parses our logs expects one logging even per line.

We added a OneLogEventPerLine property to JSONLayout in our private fork of the project, it's a simple change and it's working well. Would you be interested in porting this to your project?

ginaluca avatar Aug 16 '22 09:08 ginaluca