spark-monitoring icon indicating copy to clipboard operation
spark-monitoring copied to clipboard

Add newline option to JSONLayout

Open ginaluca opened this issue 2 years ago • 2 comments

PR related to issue https://github.com/mspnp/spark-monitoring/issues/191:

Hi all,

We use spark-monitoring in our Databricks workspaces, but in a slightly non-standard way:

Metrics are collected and pushed to Log Analytics, as per project design; Logs, on the other side, follow a different route: we want them formatted in JSON, but they are written to a shared storage >(due to our overall enterprise architecture) and then eventually exported to our logging backend. For this reason, we only need the JSONLayout piece of spark-monitoring when it comes to logs. We can do this by setting

log4j.appender.publicFile.layout=com.microsoft.pnp.logging.JSONLayout

in log4j.properties, but we have an additional requirement on JSONLayout: it must be configurable to add a newline at >the end of the serialised string because the software which parses our logs expects one logging even per line.

We added a OneLogEventPerLine property to JSONLayout in our private fork of the project, it's a simple change and it's >working well. Would you be interested in porting this to your project?

ginaluca avatar Aug 22 '22 11:08 ginaluca

CLA assistant check
All CLA requirements met.

ghost avatar Aug 22 '22 11:08 ghost

@florent-brosse - is this something you could review and merge if appropriate?

hallihan avatar Sep 22 '23 18:09 hallihan