xen-exporter
xen-exporter copied to clipboard
Exporter brakes after "unsuccessful" VM migration to another pool
I've faced the following just after start (added some debug output):
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_latency
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_read'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_read
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_read_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_read_latency
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_write'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_write
['vm', 'd5de7e57-b1a6-4e14-c334-5729395dade3', 'vbd_xvda_write_latency'] vm d5de7e57-b1a6-4e14-c334-5729395dade3 vbd_xvda_write_latency
['vm', '7ab09e90-de79-190e-1109-d1fae4690189', 'cpu0'] vm 7ab09e90-de79-190e-1109-d1fae4690189 cpu0
Traceback (most recent call last):
File "/app/xen-exporter.py", line 265, in do_GET
metric_output = collect_metrics().encode("utf-8")
File "/app/xen-exporter.py", line 191, in collect_metrics
vm = get_or_set(vms, collector, lookup_vm_name, xen)
File "/app/xen-exporter.py", line 72, in get_or_set
d[key] = func(key, *args)
File "/app/xen-exporter.py", line 29, in lookup_vm_name
return session.xenapi.VM.get_name_label(session.xenapi.VM.get_by_uuid(vm_uuid))
File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 306, in __call__
return self.__send(self.__name, args)
File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 187, in xenapi_request
result = _parse_result(getattr(self, methodname)(*full_params))
File "/usr/local/lib/python3.10/site-packages/XenAPI/XenAPI.py", line 281, in _parse_result
raise Failure(result['ErrorDescription'])
XenAPI.XenAPI.Failure: ['UUID_INVALID', 'VM', '7ab09e90-de79-190e-1109-d1fae4690189']
I've diagnosed the problem: exactly the same VM (with this UUID) exists in another Xen pool where I moved it several days ago. I noticed that this VM was migrated with and error in XCP, but didn't realise that after migration it exists in both pools simultiniously.
XCP-ng still stores metrics for this machine I suppose, but from the point of view of XCP-ng it's not a real VM (and it actually does nothing). Although, I was able to catch its metrics when I disabled exception.
That's not the problem of your exporter, I suppose. But maybe it would be better to catch that situation for generating some alerts.
And the solution of the problem is to reboot pool-master of the poll which was the source of migration by the way. After that this false VM has disappeared