Add a controller for Ceph storage

[medkbadri]

Hello,

We're currently using Rook ceph for Kubernetes storage. I experimented with this controller and it works like magic! Except that for my case, ceph OSD PVCs have block volumeMode: Thus, kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes metrics for OSD PVCs are unavailable in Prometheus.

The idea is to:

• add specific annotations for ceph PVCs or cluser definition manifest (CephCluster CRD)
• use ceph metrics instead: ceph_cluster_total_bytes and ceph_cluster_total_used_bytes
• To be decided: update the PVC or CephCluster CRD (safer)

I'd be more than happy to contribute to the implementation of the idea if it is worth experimenting with.

++