Machine memory usage (JVM+py)

[tonkolviktor]

Hi,

would it be possible to get the total memory used of a databricks job across all nodes and on the OS level.

As far as I can work out all memory metrics logged are from the JVM. That's nice, however for heavy pyspark jobs it's enough. For our use-case we actually do not need this data only at the end of the job. So ideally the following metrics would be available:

• job_run_id
• os total memory
• os used memory
• node_id

it's related to: https://github.com/mspnp/spark-monitoring/issues/189

[hallihan]

Please reach out to the contact listed in the README if you still need assistance. https://github.com/mspnp/spark-monitoring/tree/l4jv2#monitoring-azure-databricks-in-an-azure-log-analytics-workspace