[FEATURE]: Integration of external-health-monitor with Observability

[atye]

Describe the solution you'd like

When volume health monitoring is enabled for the drivers, we see volume metadata logs in the csi-external-health-monitor-controller sidecar. See the log below for PowerFlex containing volume and storage pool name, id, etc.

I1017 03:25:29.441242       1 connection.go:200] GRPC response: {"status":{"volume_condition":{"message":"Volume is in good condition"}},"volume":{"capacity_bytes":8589934592,"volume_context":{"CreationTime":"2023-10-05 13:31:02 +0000 UTC","InstallationID":"1281c03634933b94","Name":"k8s-3d2d3ea88f","StoragePoolID":"87f310a300000000","StoragePoolName":"mypool","StorageSystem":"78d250af6a330a0f"},"volume_id":"78d250af6a330a0f-d5dab71100000008"}}

This information is also collected by Observability. According to the features for the external-health-monitor, an endpoint for prometheus metrics can be exposed via the metrics-path argument.

Perhaps some work is being duplicated in Observability and we can integrate this health monitoring functionality with Observability.

[csmbot]

@atye: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at [email protected].

[sharmilarama]

Thanks @atye, this feature is not high on our list of priorities at this time, but community driven contributions to enable this feature are welcome.