pmproxy and libvirt PMDA

[Tualua]

Hello! Some of the metrics is missing when querying via pmproxy

# pmval libvirt.dominfo.name

metric:    libvirt.dominfo.name
host:      hv16.aaa.local
semantics: discrete instantaneous value
units:     none
samples:   all
full label for instance[0]: 545d60cf-916c-4944-88d6-194758efc074
full label for instance[1]: 4dee0f75-ad9e-4054-9de2-62a9771e0f30
full label for instance[2]: d666696a-87a2-475b-9b4e-3dfc1df3a5d4

545d60cf-916c-4944-88 4dee0f75-ad9e-4054-9d d666696a-87a2-475b-9b
               "vm46"                "vm48"                "vm47"

When I try to get it via proxy

# curl http://localhost:44322/metrics?names=libvirt.dominfo.name
#

Check all available metric under libvirt

# PCP5 libvirt.dominfo.memory.current 140.1.10 u64 140.0 instant Kbyte
# HELP libvirt_dominfo_memory_current VM memory, current
# TYPE libvirt_dominfo_memory_current gauge
libvirt_dominfo_memory_current{hostname="hv16.aaa.local",instname="545d60cf-916c-4944-88d6-194758efc074",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="9"} 16777216
libvirt_dominfo_memory_current{hostname="hv16.aaa.local",instname="4dee0f75-ad9e-4054-9de2-62a9771e0f30",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="10"} 16777216
libvirt_dominfo_memory_current{hostname="hv16.aaa.local",instname="d666696a-87a2-475b-9b4e-3dfc1df3a5d4",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="11"} 16777216
# PCP5 libvirt.dominfo.memory.boot 140.1.9 u64 140.0 instant Kbyte
# HELP libvirt_dominfo_memory_boot VM memory, at boot
# TYPE libvirt_dominfo_memory_boot gauge
libvirt_dominfo_memory_boot{hostname="hv16.aaa.local",instname="545d60cf-916c-4944-88d6-194758efc074",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="9"} 16777216
libvirt_dominfo_memory_boot{hostname="hv16.aaa.local",instname="4dee0f75-ad9e-4054-9de2-62a9771e0f30",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="10"} 16777216
libvirt_dominfo_memory_boot{hostname="hv16.aaa.local",instname="d666696a-87a2-475b-9b4e-3dfc1df3a5d4",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="11"} 16777216
# PCP5 libvirt.dominfo.vcpu.max 140.1.8 u32 140.0 instant count
# HELP libvirt_dominfo_vcpu_max VM vCPUs, maximum
# TYPE libvirt_dominfo_vcpu_max gauge
libvirt_dominfo_vcpu_max{hostname="hv16.aaa.local",instname="545d60cf-916c-4944-88d6-194758efc074",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="9"} 8
libvirt_dominfo_vcpu_max{hostname="hv16.aaa.local",instname="4dee0f75-ad9e-4054-9de2-62a9771e0f30",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="10"} 8
libvirt_dominfo_vcpu_max{hostname="hv16.aaa.local",instname="d666696a-87a2-475b-9b4e-3dfc1df3a5d4",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="11"} 8
# PCP5 libvirt.dominfo.vcpu.current 140.1.7 u32 140.0 instant count
# HELP libvirt_dominfo_vcpu_current VM vCPUs, current
# TYPE libvirt_dominfo_vcpu_current gauge
libvirt_dominfo_vcpu_current{hostname="hv16.aaa.local",instname="545d60cf-916c-4944-88d6-194758efc074",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="9"} 8
libvirt_dominfo_vcpu_current{hostname="hv16.aaa.local",instname="4dee0f75-ad9e-4054-9de2-62a9771e0f30",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="10"} 8
libvirt_dominfo_vcpu_current{hostname="hv16.aaa.local",instname="d666696a-87a2-475b-9b4e-3dfc1df3a5d4",machineid="2ef228af221c43239d2e5a534435be1c",domainname="localdomain",instid="11"} 8

Many metrics are missing.

OS: Oracle Linux 7.9, libvirt 5.7.0

[natoscott]

The OpenMetrics format does not support string-valued metrics ... are all of the missing metrics strings?

pminfo -d libvirt

will help answer this question.

[Tualua]

Yes. Mostly I need vm names, because my vms are recreated after each user session, so uuids are different each time Is there any way to include vm names along with uuod in metrics as in libvirt-exporter?

[natoscott]

@tualua @myllynen perhaps the libvirt PMDA could be modified to use metric labels to represent these entities? that would make them available via pmproxy /metrics endpoint anyway.

[Tualua]

@natoscott but metrics are exported using pmproxy, right? I thought that pmproxy should be modified. Libvirt is using Python, so may be I can draft a patch. Could you point me where to look?

[natoscott]

@Tualua sure, the agent will need a labels callback function which can provide these strings - it doesn't currently have one, so the libvirt metrics just get the default labels for the host.

Have a look at some of the other python agents that use labels for example pmdasimple, pmdaopenvswitch, pmdarabbitmq, pmdaopenmetrics, pmdanetcheck, pmdamongodb, or pmdabcc.

[myllynen]

The patch should help with this issue, another user was happy with the result. If time permits @Tualua please give it a try (just copy the latest file from git to your system and reinstall the PMDA). Thanks.

[myllynen]

After some libpcp fixes we have now hopefully proper labels in place, see https://github.com/performancecopilot/pcp/commit/1e9b86a5bef6dacdceed9353c035f62ef1cd9017. Closing, but let us know if you see any issues or room for improvement. Thanks.