xen-exporter icon indicating copy to clipboard operation
xen-exporter copied to clipboard

Exporter brakes after "unsuccessful" VM migration to another pool

Open al-indigo opened this issue 6 months ago • 0 comments

I've faced the following just after start (added some debug output):

['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_latency
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_read'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_read
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_read_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_read_latency
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_write'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_write
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_write_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_write_latency
['vm', '7ab09e90-de79-190e-1109-d1fae4690189', 'cpu0'] vm 7ab09e90-de79-190e-1109-d1fae4690189 cpu0
Traceback (most recent call last):
  File "/app/xen-exporter.py", line 265, in do_GET
    metric_output = collect_metrics().encode("utf-8")
  File "/app/xen-exporter.py", line 191, in collect_metrics
    vm = get_or_set(vms, collector, lookup_vm_name, xen)
  File "/app/xen-exporter.py", line 72, in get_or_set
    d[key] = func(key, *args)
  File "/app/xen-exporter.py", line 29, in lookup_vm_name
    return session.xenapi.VM.get_name_label(session.xenapi.VM.get_by_uuid(vm_uuid))
  File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 306, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 187, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 281, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.XenAPI.Failure: ['UUID_INVALID', 'VM', '7ab09e90-de79-190e-1109-d1fae4690189']

I've diagnosed the problem: exactly the same VM (with this UUID) exists in another Xen pool where I moved it several days ago. I noticed that this VM was migrated with and error in XCP, but didn't realise that after migration it exists in both pools simultiniously.

XCP-ng still stores metrics for this machine I suppose, but from the point of view of XCP-ng it's not a real VM (and it actually does nothing). Although, I was able to catch its metrics when I disabled exception.

That's not the problem of your exporter, I suppose. But maybe it would be better to catch that situation for generating some alerts.

And the solution of the problem is to reboot pool-master of the poll which was the source of migration by the way. After that this false VM has disappeared

al-indigo avatar Aug 27 '24 00:08 al-indigo