spark-monitoring
spark-monitoring copied to clipboard
Adding newline option to JSONLayout
Hi all,
We use spark-monitoring in our Databricks workspaces, but in a slightly non-standard way:
- Metrics are collected and pushed to Log Analytics, as per project design;
- Logs, on the other side, follow a different route: we want them formatted in JSON, but they are written to a shared storage (due to our overall enterprise architecture) and then eventually exported to our logging backend.
For this reason, we only need the JSONLayout
piece of spark-monitoring when it comes to logs. We can do this by setting
log4j.appender.publicFile.layout=com.microsoft.pnp.logging.JSONLayout
in log4j.properties
, but we have an additional requirement on JSONLayout: it must be configurable to add a newline at the end of the serialised string because the software which parses our logs expects one logging even per line.
We added a OneLogEventPerLine
property to JSONLayout in our private fork of the project, it's a simple change and it's working well. Would you be interested in porting this to your project?