promitor icon indicating copy to clipboard operation
promitor copied to clipboard

Published 20 hours ago • verified publisher icon tomkerkhove

Provide scraper for Azure DataBricks

Open gburton1 opened this issue 2 years ago • 2 comments

Proposal

@tomkerkhove do you have any thoughts on whether there could be a generalized, viable way to monitor Azure DataBricks with Promitor? If there was a clean pathway, we could contribute. Azure officially offers this approach: https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/

We are exploring alternatives as well, such as exposing Prometheus metrics directly on the Spark nodes.

Component

Scraper

Contact Details

No response

gburton1 avatar Sep 01 '22 16:09 gburton1

Unfortunately it does not look like Databricks is providing metrics in Azure Monitor - https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported

tomkerkhove avatar Sep 02 '22 07:09 tomkerkhove

Yes, Databricks service is using the new way to export the metrics via Azure Log Analytics. This might help - https://github.com/tomkerkhove/promitor/issues/1076

Related discussion: https://github.com/tomkerkhove/promitor/discussions/1975

locmai avatar Sep 02 '22 07:09 locmai