kadalu
kadalu copied to clipboard
kadalu
[Bug]: kadalu-operator can't response prometheus metrics from `/metrics`, except `/metrics.json`
Environment
- kubernetes: 1.22
- gluster-fs: version 11
- kadalu: 1.2.0
- kadalu-operator: install by Helm,
External Glusterfs mode Volume
This feature is available，k8s StorageClass is working, PVC is fine, PV is OK, quote is work.
[Problem] How to scrape the export Prometheus Metrics ?
api interface kadalu-operator:8050/metrics.json is work:
>_ curl -s http://localhost:8050/metrics.json | python3 -m json.tool
{
"operator": {
"memory_usage_in_bytes": 255156224,
"cpu_usage_in_nanoseconds": 47963487575
},
"storages": [
{
"name": "kadalu-ext",
"type": "External",
"total_capacity_bytes": 858993459200,
"free_capacity_bytes": 855966322688,
"used_capacity_bytes": 3027136512,
"total_inodes": 314572800,
"free_inodes": 314572154,
"used_inodes": 646,
"pvc": [
{
"pvc_name": "pvc-ebd301a4-e664-431a-a545-64ea176c708f",
"total_pvc_capacity_bytes": 4080218112,
"free_pvc_capacity_bytes": 1932734464,
"used_pvc_capacity_bytes": 2147483648,
"total_pvc_inodes": 314572800,
"free_pvc_inodes": 314572175,
"used_pvc_inodes": 625
},
{
"pvc_name": "pvc-57dd3321-3701-4100-9574-d7a868507b1b",
"total_pvc_capacity_bytes": 40802189312,
"free_pvc_capacity_bytes": 40362192896,
"used_pvc_capacity_bytes": 439996416,
"total_pvc_inodes": 314572800,
"free_pvc_inodes": 314572175,
"used_pvc_inodes": 625
},
{
"pvc_name": "pvc-c912a6f4-2f73-4c94-bb41-1c1d3ac49cbc",
"total_pvc_capacity_bytes": 40802189312,
"free_pvc_capacity_bytes": 40362557440,
"used_pvc_capacity_bytes": 439631872,
"total_pvc_inodes": 314572800,
"free_pvc_inodes": 314572175,
"used_pvc_inodes": 625
}
],
"bricks": ""
..............
But, the Prometheus Metrics interface dones't work :
>_ curl -s http://localhost:8051/metrics
Internal Server Error
I found the source code, this api don't work https://github.com/kadalu/kadalu/blob/ebe8797c10aff2dd35b114ee291986deb87b2cd9/kadalu_operator/exporter.py#L32-L56
def get_pod_data():
""" Get pod and container info of all Pods in kadalu namespace """
cmd = ["kubectl", "get", "pods", "-l", "app.kubernetes.io/part-of=kadalu",
"--field-selector=status.phase==Running", "-nkadalu", "-ojson"]
It's not useful, you should use Kubernetes api-server in the get_pod_data() and get_storage_config_data() .
Now, when i get api kadalu-operator:8050/metrics/, response all of the Prometheus HELP content:
>_ curl http://localhost:8051/metrics/
..................
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP kadalu_memory_usage_in_bytes Kadalu Memory Usage in Bytes
# TYPE kadalu_memory_usage_in_bytes gauge
# HELP kadalu_cpu_usage_in_ns Kadalu Memory Usage in Nanoseconds
# TYPE kadalu_cpu_usage_in_ns gauge
# HELP kadalu_total_number_of_containers Kadalu Total Number Of Containers
# TYPE kadalu_total_number_of_containers gauge
# HELP kadalu_total_number_of_ready_containers Kadalu Total Number Of Ready Containers
# TYPE kadalu_total_number_of_ready_containers gauge
# HELP kadalu_storage_total_capacity_bytes Kadalu Total Storage Capacity
# TYPE kadalu_storage_total_capacity_bytes gauge
# HELP kadalu_storage_used_capacity_bytes Kadalu Total Storage Used Capacity
# TYPE kadalu_storage_used_capacity_bytes gauge
# HELP kadalu_storage_free_capacity_bytes Kadalu Total Storage Free Capacity
# TYPE kadalu_storage_free_capacity_bytes gauge
# HELP kadalu_storage_total_inodes Kadalu Total Storage Inodes
# TYPE kadalu_storage_total_inodes gauge
# HELP kadalu_storage_used_inodes Kadalu Total Storage Inodes Used
# TYPE kadalu_storage_used_inodes gauge
# HELP kadalu_storage_free_inodes Kadalu Total Storage Inodes Free
# TYPE kadalu_storage_free_inodes gauge
# HELP kadalu_pvc_total_capacity_bytes Kadalu Total PVC Capacity
# TYPE kadalu_pvc_total_capacity_bytes gauge
# HELP kadalu_pvc_used_capacity_bytes Kadalu Total PVC Used Capacity
# TYPE kadalu_pvc_used_capacity_bytes gauge
# HELP kadalu_pvc_free_capacity_bytes Kadalu Total PVC Free Capacity
# TYPE kadalu_pvc_free_capacity_bytes gauge
# HELP kadalu_pvc_total_inodes Kadalu Total Total PVC Inodes
# TYPE kadalu_pvc_total_inodes gauge
# HELP kadalu_pvc_used_inodes Kadalu Total Used PVC Inodes
# TYPE kadalu_pvc_used_inodes gauge
# HELP kadalu_pvc_free_inodes Kadalu Total Free PVC Inodes
# TYPE kadalu_pvc_free_inodes gauge
My fault, I forgot set kadalu-operator label app.kubernetes.io/part-of: kadalu.
Now It's worked.
>_ curl http://kadalu-operator-svc:8050/metrics/
.......
# HELP kadalu_pvc_total_inodes Kadalu Total Total PVC Inodes
# TYPE kadalu_pvc_total_inodes gauge
kadalu_pvc_total_inodes{name="pvc-52a25f51-ef95-4798-91e7-e501039d0b1b"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-887080a6-10c9-43ae-b03c-546df6b3e081"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-92417ec0-0106-477a-bb56-239824a49082"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-c1f104e8-0d57-4c5c-a488-0c4ed146f46c"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-58933b72-d077-4091-ad18-a598f498c65a"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-ebd301a4-e664-431a-a545-64ea176c708f"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-57dd3321-3701-4100-9574-d7a868507b1b"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-c912a6f4-2f73-4c94-bb41-1c1d3ac49cbc"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-72d74f25-cbde-431d-a229-33986f854aea"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-7026ba3d-2acd-492e-93ee-61ef82eac83c"} 3.145728e+08
kadalu_pvc_total_inodes{name="pvc-0aa84e17-c10b-440a-a240-2b2389a51e3e"} 3.145728e+08
# HELP kadalu_pvc_used_inodes Kadalu Total Used PVC Inodes
# TYPE kadalu_pvc_used_inodes gauge
kadalu_pvc_used_inodes{name="pvc-52a25f51-ef95-4798-91e7-e501039d0b1b"} 43425.0
kadalu_pvc_used_inodes{name="pvc-887080a6-10c9-43ae-b03c-546df6b3e081"} 44022.0
kadalu_pvc_used_inodes{name="pvc-92417ec0-0106-477a-bb56-239824a49082"} 43425.0
kadalu_pvc_used_inodes{name="pvc-c1f104e8-0d57-4c5c-a488-0c4ed146f46c"} 44022.0
kadalu_pvc_used_inodes{name="pvc-58933b72-d077-4091-ad18-a598f498c65a"} 44022.0
kadalu_pvc_used_inodes{name="pvc-ebd301a4-e664-431a-a545-64ea176c708f"} 43425.0
kadalu_pvc_used_inodes{name="pvc-57dd3321-3701-4100-9574-d7a868507b1b"} 43425.0
kadalu_pvc_used_inodes{name="pvc-c912a6f4-2f73-4c94-bb41-1c1d3ac49cbc"} 43425.0
kadalu_pvc_used_inodes{name="pvc-72d74f25-cbde-431d-a229-33986f854aea"} 44022.0
kadalu_pvc_used_inodes{name="pvc-7026ba3d-2acd-492e-93ee-61ef82eac83c"} 44022.0
kadalu_pvc_used_inodes{name="pvc-0aa84e17-c10b-440a-a240-2b2389a51e3e"} 43425.0
# HELP kadalu_pvc_free_inodes Kadalu Total Free PVC Inodes
# TYPE kadalu_pvc_free_inodes gauge
kadalu_pvc_free_inodes{name="pvc-52a25f51-ef95-4798-91e7-e501039d0b1b"} 3.14529375e+08
kadalu_pvc_free_inodes{name="pvc-887080a6-10c9-43ae-b03c-546df6b3e081"} 3.14528778e+08
kadalu_pvc_free_inodes{name="pvc-92417ec0-0106-477a-bb56-239824a49082"} 3.14529375e+08
kadalu_pvc_free_inodes{name="pvc-c1f104e8-0d57-4c5c-a488-0c4ed146f46c"} 3.14528778e+08
kadalu_pvc_free_inodes{name="pvc-58933b72-d077-4091-ad18-a598f498c65a"} 3.14528778e+08
kadalu_pvc_free_inodes{name="pvc-ebd301a4-e664-431a-a545-64ea176c708f"} 3.14529375e+08
kadalu_pvc_free_inodes{name="pvc-57dd3321-3701-4100-9574-d7a868507b1b"} 3.14529375e+08
kadalu_pvc_free_inodes{name="pvc-c912a6f4-2f73-4c94-bb41-1c1d3ac49cbc"} 3.14529375e+08
kadalu_pvc_free_inodes{name="pvc-72d74f25-cbde-431d-a229-33986f854aea"} 3.14528778e+08
kadalu_pvc_free_inodes{name="pvc-7026ba3d-2acd-492e-93ee-61ef82eac83c"} 3.14528778e+08
kadalu_pvc_free_inodes{name="pvc-0aa84e17-c10b-440a-a240-2b2389a51e3e"} 3.14529375e+08