kubevirt
kubevirt copied to clipboard
kubevirt
vGPU mount bug : PCI_RESOURCE_NVIDIA_COM_GRID_P40-1Q not set for resource nvidia.com/GRID_P40-1Q
What happened: When I used kubevirt to mount vgpu, the mount was successful, but the pod log indicated that the resource was not needed. "PCI_RESOURCE_NVIDIA_COM_GRID_P40-1Q not set for resource nvidia.com/GRID_P40-1Q“ The same device ID can be used normally when using a KVM virtual machine.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Additional context:
pod logs:
{"component":"virt-launcher-monitor","level":"error","msg":"could not create up serial console term file: /var/run/kubevirt-private/c3a21246-cf20-4120-afc2-788d959b08db/virt-serial0-log-sigTerm","pos":"virt-launcher-monitor.go:72","reason":"open /var/run/kubevirt-private/c3a21246-cf20-4120-afc2-788d959b08db/virt-serial0-log-sigTerm: no such file or directory","timestamp":"2024-02-02T09:27:49.021625Z"}
{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:86","timestamp":"2024-02-02T09:27:49.071111Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:89","timestamp":"2024-02-02T09:27:49.071212Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu+unix:///session?socket=/var/run/libvirt/virtqemud-sock","pos":"libvirt.go:563","timestamp":"2024-02-02T09:27:49.071649Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory')","pos":"libvirt.go:571","timestamp":"2024-02-02T09:27:49.072271Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 9.5.0, package: 6.el9 ([email protected], 2023-08-25-08:53:56, )","subcomponent":"libvirt","thread":"43","timestamp":"2024-02-02T09:27:49.110000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: vgpu","subcomponent":"libvirt","thread":"43","timestamp":"2024-02-02T09:27:49.110000Z"}
{"component":"virt-launcher","level":"error","msg":"Unable to open /dev/kvm: Permission denied","pos":"virHostCPUGetCPUID:1422","subcomponent":"libvirt","thread":"43","timestamp":"2024-02-02T09:27:49.110000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get session bus connection: Cannot spawn a message bus without a machine-id: Unable to load /var/lib/dbus/machine-id or /etc/machine-id: Failed to open file “/var/lib/dbus/machine-id”: No such file or directory","pos":"virGDBusGetSessionBus:126","subcomponent":"libvirt","thread":"43","timestamp":"2024-02-02T09:27:49.110000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get system bus connection: Could not connect: No such file or directory","pos":"virGDBusGetSystemBus:99","subcomponent":"libvirt","thread":"43","timestamp":"2024-02-02T09:27:49.110000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:579","timestamp":"2024-02-02T09:27:49.576362Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:562","timestamp":"2024-02-02T09:27:49.582928Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:75","timestamp":"2024-02-02T09:27:49.583307Z"}
{"component":"virt-launcher","level":"warning","msg":"PCI_RESOURCE_NVIDIA_COM_GRID_P40-1Q not set for resource nvidia.com/GRID_P40-1Q","pos":"addresspool.go:51","timestamp":"2024-02-02T09:27:54.423068Z"}
{"component":"virt-launcher","level":"info","msg":"host-devices created: [334852fe-079b-11ee-9fc7-77463608f467]","pos":"hostdev.go:98","timestamp":"2024-02-02T09:27:54.423160Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Executing PreStartHook on VMI pod environment","name":"vgpu","namespace":"default","pos":"manager.go:684","timestamp":"2024-02-02T09:27:54.423855Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Starting PreCloudInitIso hook","name":"vgpu","namespace":"default","pos":"manager.go:705","timestamp":"2024-02-02T09:27:54.423915Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"info","msg":"Found nameservers in /etc/resolv.conf: \n`\u0000\n","pos":"network.go:171","timestamp":"2024-02-02T09:27:54.425297Z"}
{"component":"virt-launcher","level":"info","msg":"Found search domains in /etc/resolv.conf: default.svc.cluster.local svc.cluster.local cluster.local","pos":"network.go:172","timestamp":"2024-02-02T09:27:54.425343Z"}
{"component":"virt-launcher","level":"info","msg":"Starting SingleClientDHCPServer","pos":"server.go:65","timestamp":"2024-02-02T09:27:54.425537Z"}
{"component":"virt-launcher","level":"info","msg":"Driver cache mode for /var/run/kubevirt-private/vmi-disks/harddrive/disk.img set to none","pos":"converter.go:496","timestamp":"2024-02-02T09:27:54.451966Z"}
{"component":"virt-launcher","level":"info","msg":"Driver cache mode for /var/run/kubevirt-ephemeral-disks/disk-data/virtiocontainerdisk/disk.qcow2 set to none","pos":"converter.go:496","timestamp":"2024-02-02T09:27:54.486004Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Domain XML generated. Base64 dump PGRvbWFpbiB0eXBlPSJrdm0iIHhtbG5zOnFlbXU9Imh0dHA6Ly9saWJ2aXJ0Lm9yZy9zY2hlbWFzL2RvbWFpbi9xZW11LzEuMCI+Cgk8bmFtZT5kZWZhdWx0X3ZncHU8L25hbWU+Cgk8bWVtb3J5IHVuaXQ9ImIiPjY0MDAwMDAwMDAwPC9tZW1vcnk+Cgk8b3M+CgkJPHR5cGUgYXJjaD0ieDg2XzY0IiBtYWNoaW5lPSJxMzUiPmh2bTwvdHlwZT4KCQk8c21iaW9zIG1vZGU9InN5c2luZm8iPjwvc21iaW9zPgoJPC9vcz4KCTxzeXNpbmZvIHR5cGU9InNtYmlvcyI+CgkJPHN5c3RlbT4KCQkJPGVudHJ5IG5hbWU9InV1aWQiPjAxODYxNWMwLWZhZDUtNTIyOS1iMTJlLTEwNTM3OTg0MDMyYzwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJtYW51ZmFjdHVyZXIiPkt1YmVWaXJ0PC9lbnRyeT4KCQkJPGVudHJ5IG5hbWU9ImZhbWlseSI+S3ViZVZpcnQ8L2VudHJ5PgoJCQk8ZW50cnkgbmFtZT0icHJvZHVjdCI+Tm9uZTwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJza3UiPjwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJ2ZXJzaW9uIj48L2VudHJ5PgoJCTwvc3lzdGVtPgoJCTxiaW9zPjwvYmlvcz4KCQk8YmFzZUJvYXJkPjwvYmFzZUJvYXJkPgoJCTxjaGFzc2lzPjwvY2hhc3Npcz4KCTwvc3lzaW5mbz4KCTxkZXZpY2VzPgoJCTxpbnRlcmZhY2UgdHlwZT0iZXRoZXJuZXQiPgoJCQk8c291cmNlPjwvc291cmNlPgoJCQk8dGFyZ2V0IGRldj0idGFwMCIgbWFuYWdlZD0ibm8iPjwvdGFyZ2V0PgoJCQk8bW9kZWwgdHlwZT0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPjwvbW9kZWw+CgkJCTxtYWMgYWRkcmVzcz0iMGU6ZWE6YWQ6MzA6NmM6NzUiPjwvbWFjPgoJCQk8bXR1IHNpemU9IjE0ODAiPjwvbXR1PgoJCQk8YWxpYXMgbmFtZT0idWEtZGVmYXVsdCI+PC9hbGlhcz4KCQkJPHJvbSBlbmFibGVkPSJubyI+PC9yb20+CgkJPC9pbnRlcmZhY2U+CgkJPGludGVyZmFjZSB0eXBlPSJldGhlcm5ldCI+CgkJCTxzb3VyY2U+PC9zb3VyY2U+CgkJCTx0YXJnZXQgZGV2PSJwbGFjZWhvbGRlci0wIiBtYW5hZ2VkPSJubyI+PC90YXJnZXQ+CgkJCTxtb2RlbCB0eXBlPSJ2aXJ0aW8tbm9uLXRyYW5zaXRpb25hbCI+PC9tb2RlbD4KCQk8L2ludGVyZmFjZT4KCQk8aW50ZXJmYWNlIHR5cGU9ImV0aGVybmV0Ij4KCQkJPHNvdXJjZT48L3NvdXJjZT4KCQkJPHRhcmdldCBkZXY9InBsYWNlaG9sZGVyLTEiIG1hbmFnZWQ9Im5vIj48L3RhcmdldD4KCQkJPG1vZGVsIHR5cGU9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj48L21vZGVsPgoJCTwvaW50ZXJmYWNlPgoJCTxpbnRlcmZhY2UgdHlwZT0iZXRoZXJuZXQiPgoJCQk8c291cmNlPjwvc291cmNlPgoJCQk8dGFyZ2V0IGRldj0icGxhY2Vob2xkZXItMiIgbWFuYWdlZD0ibm8iPjwvdGFyZ2V0PgoJCQk8bW9kZWwgdHlwZT0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPjwvbW9kZWw+CgkJPC9pbnRlcmZhY2U+CgkJPGNoYW5uZWwgdHlwZT0idW5peCI+CgkJCTx0YXJnZXQgbmFtZT0ib3JnLnFlbXUuZ3Vlc3RfYWdlbnQuMCIgdHlwZT0idmlydGlvIj48L3RhcmdldD4KCQk8L2NoYW5uZWw+CgkJPGhvc3RkZXYgdHlwZT0ibWRldiIgbWFuYWdlZD0iIiBtb2RlPSJzdWJzeXN0ZW0iIG1vZGVsPSJ2ZmlvLXBjaSIgZGlzcGxheT0ib24iIHJhbWZiPSJvbiI+CgkJCTxzb3VyY2U+CgkJCQk8YWRkcmVzcyB0eXBlPSIiIGJ1cz0iIiB1dWlkPSIzMzQ4NTJmZS0wNzliLTExZWUtOWZjNy03NzQ2MzYwOGY0NjciPjwvYWRkcmVzcz4KCQkJPC9zb3VyY2U+CgkJCTxhbGlhcyBuYW1lPSJ1YS1ncHUtZ3B1MSI+PC9hbGlhcz4KCQk8L2hvc3RkZXY+CgkJPGNvbnRyb2xsZXIgdHlwZT0idXNiIiBpbmRleD0iMCIgbW9kZWw9Im5vbmUiPjwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJzY3NpIiBpbmRleD0iMCIgbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj48L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0idmlydGlvLXNlcmlhbCIgaW5kZXg9IjAiIG1vZGVsPSJ2aXJ0aW8tbm9uLXRyYW5zaXRpb25hbCI+PC9jb250cm9sbGVyPgoJCTx2aWRlbz4KCQkJPG1vZGVsIHR5cGU9InZnYSIgaGVhZHM9IjEiIHZyYW09IjE2Mzg0Ij48L21vZGVsPgoJCTwvdmlkZW8+CgkJPGdyYXBoaWNzIHR5cGU9InZuYyI+CgkJCTxsaXN0ZW4gdHlwZT0ic29ja2V0IiBzb2NrZXQ9Ii92YXIvcnVuL2t1YmV2aXJ0LXByaXZhdGUvYzNhMjEyNDYtY2YyMC00MTIwLWFmYzItNzg4ZDk1OWIwOGRiL3ZpcnQtdm5jIj48L2xpc3Rlbj4KCQk8L2dyYXBoaWNzPgoJCTxtZW1iYWxsb29uIG1vZGVsPSJ2aXJ0aW8tbm9uLXRyYW5zaXRpb25hbCIgZnJlZVBhZ2VSZXBvcnRpbmc9Im9uIj4KCQkJPHN0YXRzIHBlcmlvZD0iMTAiPjwvc3RhdHM+CgkJPC9tZW1iYWxsb29uPgoJCTxkaXNrIGRldmljZT0iZGlzayIgdHlwZT0iZmlsZSIgbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj4KCQkJPHNvdXJjZSBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC1wcml2YXRlL3ZtaS1kaXNrcy9oYXJkZHJpdmUvZGlzay5pbWciPjwvc291cmNlPgoJCQk8dGFyZ2V0IGJ1cz0idmlydGlvIiBkZXY9InZkYSI+PC90YXJnZXQ+CgkJCTxkcml2ZXIgY2FjaGU9Im5vbmUiIGVycm9yX3BvbGljeT0ic3RvcCIgbmFtZT0icWVtdSIgdHlwZT0icmF3IiBkaXNjYXJkPSJ1bm1hcCI+PC9kcml2ZXI+CgkJCTxhbGlhcyBuYW1lPSJ1YS1oYXJkZHJpdmUiPjwvYWxpYXM+CgkJCTxib290IG9yZGVyPSIxIj48L2Jvb3Q+CgkJCTxmaWxlc3lzdGVtT3ZlcmhlYWQ+MC4wNTU8L2ZpbGVzeXN0ZW1PdmVyaGVhZD4KCQkJPGNhcGFjaXR5Pjg1ODk5MzQ1OTIwPC9jYXBhY2l0eT4KCQk8L2Rpc2s+CgkJPGRpc2sgZGV2aWNlPSJjZHJvbSIgdHlwZT0iZmlsZSI+CgkJCTxzb3VyY2UgZmlsZT0iL3Zhci9ydW4va3ViZXZpcnQtZXBoZW1lcmFsLWRpc2tzL2Rpc2stZGF0YS92aXJ0aW9jb250YWluZXJkaXNrL2Rpc2sucWNvdzIiPjwvc291cmNlPgoJCQk8dGFyZ2V0IGJ1cz0ic2F0YSIgZGV2PSJzZGEiIHRyYXk9ImNsb3NlZCI+PC90YXJnZXQ+CgkJCTxkcml2ZXIgY2FjaGU9Im5vbmUiIGVycm9yX3BvbGljeT0ic3RvcCIgbmFtZT0icWVtdSIgdHlwZT0icWNvdzIiIGRpc2NhcmQ9InVubWFwIj48L2RyaXZlcj4KCQkJPHJlYWRvbmx5PjwvcmVhZG9ubHk+CgkJCTxhbGlhcyBuYW1lPSJ1YS12aXJ0aW9jb250YWluZXJkaXNrIj48L2FsaWFzPgoJCQk8YmFja2luZ1N0b3JlIHR5cGU9ImZpbGUiPgoJCQkJPGZvcm1hdCB0eXBlPSJyYXciPjwvZm9ybWF0PgoJCQkJPHNvdXJjZSBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC9jb250YWluZXItZGlza3MvZGlza18xLmltZyI+PC9zb3VyY2U+CgkJCTwvYmFja2luZ1N0b3JlPgoJCQk8Ym9vdCBvcmRlcj0iMiI+PC9ib290PgoJCTwvZGlzaz4KCQk8c2VyaWFsIHR5cGU9InVuaXgiPgoJCQk8dGFyZ2V0IHBvcnQ9IjAiPjwvdGFyZ2V0PgoJCQk8c291cmNlIG1vZGU9ImJpbmQiIHBhdGg9Ii92YXIvcnVuL2t1YmV2aXJ0LXByaXZhdGUvYzNhMjEyNDYtY2YyMC00MTIwLWFmYzItNzg4ZDk1OWIwOGRiL3ZpcnQtc2VyaWFsMCI+PC9zb3VyY2U+CgkJCTxsb2cgZmlsZT0iL3Zhci9ydW4va3ViZXZpcnQtcHJpdmF0ZS9jM2EyMTI0Ni1jZjIwLTQxMjAtYWZjMi03ODhkOTU5YjA4ZGIvdmlydC1zZXJpYWwwLWxvZyIgYXBwZW5kPSJvbiI+PC9sb2c+CgkJPC9zZXJpYWw+CgkJPGNvbnNvbGUgdHlwZT0icHR5Ij4KCQkJPHRhcmdldCB0eXBlPSJzZXJpYWwiIHBvcnQ9IjAiPjwvdGFyZ2V0PgoJCTwvY29uc29sZT4KCQk8c291bmQgbW9kZWw9ImljaDkiPgoJCQk8YWxpYXMgbmFtZT0idWEtc291bmQxIj48L2FsaWFzPgoJCTwvc291bmQ+Cgk8L2RldmljZXM+Cgk8bWV0YWRhdGE+CgkJPGt1YmV2aXJ0IHhtbG5zPSJodHRwOi8va3ViZXZpcnQuaW8iPgoJCQk8dWlkPjwvdWlkPgoJCTwva3ViZXZpcnQ+Cgk8L21ldGFkYXRhPgoJPGZlYXR1cmVzPgoJCTxhY3BpPjwvYWNwaT4KCTwvZmVhdHVyZXM+Cgk8Y3B1IG1vZGU9Imhvc3QtbW9kZWwiPgoJCTx0b3BvbG9neSBzb2NrZXRzPSIxIiBjb3Jlcz0iMTYiIHRocmVhZHM9IjEiPjwvdG9wb2xvZ3k+Cgk8L2NwdT4KCTx2Y3B1IHBsYWNlbWVudD0ic3RhdGljIj4xNjwvdmNwdT4KCTxpb3RocmVhZHM+MTwvaW90aHJlYWRzPgo8L2RvbWFpbj4=","name":"vgpu","namespace":"default","pos":"libvirt_helper.go:133","timestamp":"2024-02-02T09:27:54.494193Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"error","msg":"XML error: Invalid value for attribute 'managed' in element 'hostdev': ''.","pos":"virXMLPropEnumInternal:393","subcomponent":"libvirt","thread":"29","timestamp":"2024-02-02T09:27:54.495000Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 65 with status 0","pos":"virt-launcher-monitor.go:192","timestamp":"2024-02-02T09:27:54.570214Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 67 with status 0","pos":"virt-launcher-monitor.go:192","timestamp":"2024-02-02T09:27:54.778452Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 73 with status 0","pos":"virt-launcher-monitor.go:192","timestamp":"2024-02-02T09:27:55.010506Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 75 with status 0","pos":"virt-launcher-monitor.go:192","timestamp":"2024-02-02T09:27:55.065838Z"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event Domain event=\"defined\" detail=\"added\" with event id 0 reason 0 received","pos":"client.go:469","timestamp":"2024-02-02T09:27:55.405325Z"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Shutoff(5):Unknown(0)","pos":"client.go:296","timestamp":"2024-02-02T09:27:55.410352Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Domain XML generated. Base64 dump PGRvbWFpbiB0eXBlPSJrdm0iIHhtbG5zOnFlbXU9Imh0dHA6Ly9saWJ2aXJ0Lm9yZy9zY2hlbWFzL2RvbWFpbi9xZW11LzEuMCI+Cgk8bmFtZT5kZWZhdWx0X3ZncHU8L25hbWU+Cgk8bWVtb3J5IHVuaXQ9ImIiPjY0MDAwMDAwMDAwPC9tZW1vcnk+Cgk8b3M+CgkJPHR5cGUgYXJjaD0ieDg2XzY0IiBtYWNoaW5lPSJxMzUiPmh2bTwvdHlwZT4KCQk8c21iaW9zIG1vZGU9InN5c2luZm8iPjwvc21iaW9zPgoJPC9vcz4KCTxzeXNpbmZvIHR5cGU9InNtYmlvcyI+CgkJPHN5c3RlbT4KCQkJPGVudHJ5IG5hbWU9InV1aWQiPjAxODYxNWMwLWZhZDUtNTIyOS1iMTJlLTEwNTM3OTg0MDMyYzwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJtYW51ZmFjdHVyZXIiPkt1YmVWaXJ0PC9lbnRyeT4KCQkJPGVudHJ5IG5hbWU9ImZhbWlseSI+S3ViZVZpcnQ8L2VudHJ5PgoJCQk8ZW50cnkgbmFtZT0icHJvZHVjdCI+Tm9uZTwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJza3UiPjwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJ2ZXJzaW9uIj48L2VudHJ5PgoJCTwvc3lzdGVtPgoJCTxiaW9zPjwvYmlvcz4KCQk8YmFzZUJvYXJkPjwvYmFzZUJvYXJkPgoJCTxjaGFzc2lzPjwvY2hhc3Npcz4KCTwvc3lzaW5mbz4KCTxkZXZpY2VzPgoJCTxlbXVsYXRvcj4vdXNyL2xpYmV4ZWMvcWVtdS1rdm08L2VtdWxhdG9yPgoJCTxpbnRlcmZhY2UgdHlwZT0iZXRoZXJuZXQiPgoJCQk8c291cmNlPjwvc291cmNlPgoJCQk8dGFyZ2V0IGRldj0idGFwMCIgbWFuYWdlZD0ibm8iPjwvdGFyZ2V0PgoJCQk8bW9kZWwgdHlwZT0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPjwvbW9kZWw+CgkJCTxtYWMgYWRkcmVzcz0iMGU6ZWE6YWQ6MzA6NmM6NzUiPjwvbWFjPgoJCQk8bXR1IHNpemU9IjE0ODAiPjwvbXR1PgoJCQk8YWxpYXMgbmFtZT0idWEtZGVmYXVsdCI+PC9hbGlhcz4KCQkJPHJvbSBlbmFibGVkPSJubyI+PC9yb20+CgkJPC9pbnRlcmZhY2U+CgkJPGNoYW5uZWwgdHlwZT0idW5peCI+CgkJCTx0YXJnZXQgbmFtZT0ib3JnLnFlbXUuZ3Vlc3RfYWdlbnQuMCIgdHlwZT0idmlydGlvIj48L3RhcmdldD4KCQk8L2NoYW5uZWw+CgkJPGhvc3RkZXYgdHlwZT0ibWRldiIgbWFuYWdlZD0ibm8iIG1vZGU9InN1YnN5c3RlbSIgbW9kZWw9InZmaW8tcGNpIiBkaXNwbGF5PSJvbiIgcmFtZmI9Im9uIj4KCQkJPHNvdXJjZT4KCQkJCTxhZGRyZXNzIHR5cGU9IiIgYnVzPSIiIHV1aWQ9IjMzNDg1MmZlLTA3OWItMTFlZS05ZmM3LTc3NDYzNjA4ZjQ2NyI+PC9hZGRyZXNzPgoJCQk8L3NvdXJjZT4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDA4IiBzbG90PSIweDAwIiBmdW5jdGlvbj0iMHgwIj48L2FkZHJlc3M+CgkJCTxhbGlhcyBuYW1lPSJ1YS1ncHUtZ3B1MSI+PC9hbGlhcz4KCQk8L2hvc3RkZXY+CgkJPGNvbnRyb2xsZXIgdHlwZT0idXNiIiBpbmRleD0iMCIgbW9kZWw9Im5vbmUiPjwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJzY3NpIiBpbmRleD0iMCIgbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDA1IiBzbG90PSIweDAwIiBmdW5jdGlvbj0iMHgwIj48L2FkZHJlc3M+CgkJPC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InZpcnRpby1zZXJpYWwiIGluZGV4PSIwIiBtb2RlbD0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDYiIHNsb3Q9IjB4MDAiIGZ1bmN0aW9uPSIweDAiPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0ic2F0YSIgaW5kZXg9IjAiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDAiIHNsb3Q9IjB4MWYiIGZ1bmN0aW9uPSIweDIiPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0icGNpIiBpbmRleD0iMCIgbW9kZWw9InBjaWUtcm9vdCI+PC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InBjaSIgaW5kZXg9IjEiIG1vZGVsPSJwY2llLXJvb3QtcG9ydCI+CgkJCTxhZGRyZXNzIHR5cGU9InBjaSIgZG9tYWluPSIweDAwMDAiIGJ1cz0iMHgwMCIgc2xvdD0iMHgwMiIgZnVuY3Rpb249IjB4MCI+PC9hZGRyZXNzPgoJCTwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJwY2kiIGluZGV4PSIyIiBtb2RlbD0icGNpZS1yb290LXBvcnQiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDAiIHNsb3Q9IjB4MDIiIGZ1bmN0aW9uPSIweDEiPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0icGNpIiBpbmRleD0iMyIgbW9kZWw9InBjaWUtcm9vdC1wb3J0Ij4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDAwIiBzbG90PSIweDAyIiBmdW5jdGlvbj0iMHgyIj48L2FkZHJlc3M+CgkJPC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InBjaSIgaW5kZXg9IjQiIG1vZGVsPSJwY2llLXJvb3QtcG9ydCI+CgkJCTxhZGRyZXNzIHR5cGU9InBjaSIgZG9tYWluPSIweDAwMDAiIGJ1cz0iMHgwMCIgc2xvdD0iMHgwMiIgZnVuY3Rpb249IjB4MyI+PC9hZGRyZXNzPgoJCTwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJwY2kiIGluZGV4PSI1IiBtb2RlbD0icGNpZS1yb290LXBvcnQiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDAiIHNsb3Q9IjB4MDIiIGZ1bmN0aW9uPSIweDQiPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0icGNpIiBpbmRleD0iNiIgbW9kZWw9InBjaWUtcm9vdC1wb3J0Ij4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDAwIiBzbG90PSIweDAyIiBmdW5jdGlvbj0iMHg1Ij48L2FkZHJlc3M+CgkJPC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InBjaSIgaW5kZXg9IjciIG1vZGVsPSJwY2llLXJvb3QtcG9ydCI+CgkJCTxhZGRyZXNzIHR5cGU9InBjaSIgZG9tYWluPSIweDAwMDAiIGJ1cz0iMHgwMCIgc2xvdD0iMHgwMiIgZnVuY3Rpb249IjB4NiI+PC9hZGRyZXNzPgoJCTwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJwY2kiIGluZGV4PSI4IiBtb2RlbD0icGNpZS1yb290LXBvcnQiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDAiIHNsb3Q9IjB4MDIiIGZ1bmN0aW9uPSIweDciPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPGNvbnRyb2xsZXIgdHlwZT0icGNpIiBpbmRleD0iOSIgbW9kZWw9InBjaWUtcm9vdC1wb3J0Ij4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDAwIiBzbG90PSIweDAzIiBmdW5jdGlvbj0iMHgwIj48L2FkZHJlc3M+CgkJPC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InBjaSIgaW5kZXg9IjEwIiBtb2RlbD0icGNpZS1yb290LXBvcnQiPgoJCQk8YWRkcmVzcyB0eXBlPSJwY2kiIGRvbWFpbj0iMHgwMDAwIiBidXM9IjB4MDAiIHNsb3Q9IjB4MDMiIGZ1bmN0aW9uPSIweDEiPjwvYWRkcmVzcz4KCQk8L2NvbnRyb2xsZXI+CgkJPHZpZGVvPgoJCQk8bW9kZWwgdHlwZT0idmdhIiBoZWFkcz0iMSIgdnJhbT0iMTYzODQiPjwvbW9kZWw+CgkJPC92aWRlbz4KCQk8Z3JhcGhpY3MgdHlwZT0idm5jIj4KCQkJPGxpc3RlbiB0eXBlPSJzb2NrZXQiIHNvY2tldD0iL3Zhci9ydW4va3ViZXZpcnQtcHJpdmF0ZS9jM2EyMTI0Ni1jZjIwLTQxMjAtYWZjMi03ODhkOTU5YjA4ZGIvdmlydC12bmMiPjwvbGlzdGVuPgoJCTwvZ3JhcGhpY3M+CgkJPG1lbWJhbGxvb24gbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIiBmcmVlUGFnZVJlcG9ydGluZz0ib24iPgoJCQk8c3RhdHMgcGVyaW9kPSIxMCI+PC9zdGF0cz4KCQkJPGFkZHJlc3MgdHlwZT0icGNpIiBkb21haW49IjB4MDAwMCIgYnVzPSIweDA5IiBzbG90PSIweDAwIiBmdW5jdGlvbj0iMHgwIj48L2FkZHJlc3M+CgkJPC9tZW1iYWxsb29uPgoJCTxkaXNrIGRldmljZT0iZGlzayIgdHlwZT0iZmlsZSIgbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj4KCQkJPHNvdXJjZSBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC1wcml2YXRlL3ZtaS1kaXNrcy9oYXJkZHJpdmUvZGlzay5pbWciPjwvc291cmNlPgoJCQk8dGFyZ2V0IGJ1cz0idmlydGlvIiBkZXY9InZkYSI+PC90YXJnZXQ+CgkJCTxkcml2ZXIgY2FjaGU9Im5vbmUiIGVycm9yX3BvbGljeT0ic3RvcCIgbmFtZT0icWVtdSIgdHlwZT0icmF3IiBkaXNjYXJkPSJ1bm1hcCI+PC9kcml2ZXI+CgkJCTxhbGlhcyBuYW1lPSJ1YS1oYXJkZHJpdmUiPjwvYWxpYXM+CgkJCTxib290IG9yZGVyPSIxIj48L2Jvb3Q+CgkJCTxhZGRyZXNzIHR5cGU9InBjaSIgZG9tYWluPSIweDAwMDAiIGJ1cz0iMHgwNyIgc2xvdD0iMHgwMCIgZnVuY3Rpb249IjB4MCI+PC9hZGRyZXNzPgoJCTwvZGlzaz4KCQk8ZGlzayBkZXZpY2U9ImNkcm9tIiB0eXBlPSJmaWxlIj4KCQkJPHNvdXJjZSBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC1lcGhlbWVyYWwtZGlza3MvZGlzay1kYXRhL3ZpcnRpb2NvbnRhaW5lcmRpc2svZGlzay5xY293MiI+PC9zb3VyY2U+CgkJCTx0YXJnZXQgYnVzPSJzYXRhIiBkZXY9InNkYSI+PC90YXJnZXQ+CgkJCTxkcml2ZXIgY2FjaGU9Im5vbmUiIGVycm9yX3BvbGljeT0ic3RvcCIgbmFtZT0icWVtdSIgdHlwZT0icWNvdzIiIGRpc2NhcmQ9InVubWFwIj48L2RyaXZlcj4KCQkJPHJlYWRvbmx5PjwvcmVhZG9ubHk+CgkJCTxhbGlhcyBuYW1lPSJ1YS12aXJ0aW9jb250YWluZXJkaXNrIj48L2FsaWFzPgoJCQk8YmFja2luZ1N0b3JlIHR5cGU9ImZpbGUiPgoJCQkJPGZvcm1hdCB0eXBlPSJyYXciPjwvZm9ybWF0PgoJCQkJPHNvdXJjZSBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC9jb250YWluZXItZGlza3MvZGlza18xLmltZyI+PC9zb3VyY2U+CgkJCTwvYmFja2luZ1N0b3JlPgoJCQk8Ym9vdCBvcmRlcj0iMiI+PC9ib290PgoJCQk8YWRkcmVzcyB0eXBlPSJkcml2ZSIgYnVzPSIwIiBjb250cm9sbGVyPSIwIiB0YXJnZXQ9IjAiIHVuaXQ9IjAiPjwvYWRkcmVzcz4KCQk8L2Rpc2s+CgkJPGlucHV0IHR5cGU9Im1vdXNlIiBidXM9InBzMiI+PC9pbnB1dD4KCQk8aW5wdXQgdHlwZT0ia2V5Ym9hcmQiIGJ1cz0icHMyIj48L2lucHV0PgoJCTxzZXJpYWwgdHlwZT0idW5peCI+CgkJCTx0YXJnZXQgcG9ydD0iMCI+PC90YXJnZXQ+CgkJCTxzb3VyY2UgbW9kZT0iYmluZCIgcGF0aD0iL3Zhci9ydW4va3ViZXZpcnQtcHJpdmF0ZS9jM2EyMTI0Ni1jZjIwLTQxMjAtYWZjMi03ODhkOTU5YjA4ZGIvdmlydC1zZXJpYWwwIj48L3NvdXJjZT4KCQkJPGxvZyBmaWxlPSIvdmFyL3J1bi9rdWJldmlydC1wcml2YXRlL2MzYTIxMjQ2LWNmMjAtNDEyMC1hZmMyLTc4OGQ5NTliMDhkYi92aXJ0LXNlcmlhbDAtbG9nIiBhcHBlbmQ9Im9uIj48L2xvZz4KCQk8L3NlcmlhbD4KCQk8Y29uc29sZSB0eXBlPSJ1bml4Ij4KCQkJPHRhcmdldCB0eXBlPSJzZXJpYWwiIHBvcnQ9IjAiPjwvdGFyZ2V0PgoJCQk8c291cmNlIG1vZGU9ImJpbmQiIHBhdGg9Ii92YXIvcnVuL2t1YmV2aXJ0LXByaXZhdGUvYzNhMjEyNDYtY2YyMC00MTIwLWFmYzItNzg4ZDk1OWIwOGRiL3ZpcnQtc2VyaWFsMCI+PC9zb3VyY2U+CgkJPC9jb25zb2xlPgoJCTx3YXRjaGRvZyBtb2RlbD0iaXRjbyIgYWN0aW9uPSJyZXNldCI+PC93YXRjaGRvZz4KCQk8c291bmQgbW9kZWw9ImljaDkiPgoJCQk8YWxpYXMgbmFtZT0idWEtc291bmQxIj48L2FsaWFzPgoJCTwvc291bmQ+Cgk8L2RldmljZXM+Cgk8bWV0YWRhdGE+CgkJPGt1YmV2aXJ0IHhtbG5zPSJodHRwOi8va3ViZXZpcnQuaW8iPgoJCQk8dWlkPjwvdWlkPgoJCTwva3ViZXZpcnQ+Cgk8L21ldGFkYXRhPgoJPGZlYXR1cmVzPgoJCTxhY3BpPjwvYWNwaT4KCTwvZmVhdHVyZXM+Cgk8Y3B1IG1vZGU9Imhvc3QtbW9kZWwiPgoJCTx0b3BvbG9neSBzb2NrZXRzPSIxIiBjb3Jlcz0iMTYiIHRocmVhZHM9IjEiPjwvdG9wb2xvZ3k+Cgk8L2NwdT4KCTx2Y3B1IHBsYWNlbWVudD0ic3RhdGljIj4xNjwvdmNwdT4KCTxpb3RocmVhZHM+MTwvaW90aHJlYWRzPgo8L2RvbWFpbj4=","name":"vgpu","namespace":"default","pos":"libvirt_helper.go:133","timestamp":"2024-02-02T09:27:55.411056Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"info","msg":"Successfully connected to domain notify socket at /var/run/kubevirt/domain-notify-pipe.sock","pos":"client.go:170","timestamp":"2024-02-02T09:27:55.416778Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: default_vgpu","pos":"client.go:423","timestamp":"2024-02-02T09:27:55.425314Z"}
{"component":"virt-launcher","level":"info","msg":"Monitoring loop: rate 1s start timeout 4m50s","pos":"monitor.go:209","timestamp":"2024-02-02T09:27:55.427819Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Domain defined.","name":"vgpu","namespace":"default","pos":"manager.go:1076","timestamp":"2024-02-02T09:27:55.459915Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event Domain event=\"defined\" detail=\"updated\" with event id 0 reason 1 received","pos":"client.go:469","timestamp":"2024-02-02T09:27:55.460118Z"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Shutoff(5):Unknown(0)","pos":"client.go:296","timestamp":"2024-02-02T09:27:55.463810Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: default_vgpu","pos":"client.go:423","timestamp":"2024-02-02T09:27:55.465295Z"}
{"component":"virt-launcher","level":"info","msg":"GuestAgentLifecycle event state 2 with reason 1 received","pos":"client.go:545","timestamp":"2024-02-02T09:27:55.953403Z"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Paused(3):StartingUp(11)","pos":"client.go:296","timestamp":"2024-02-02T09:27:55.957230Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: default_vgpu","pos":"client.go:423","timestamp":"2024-02-02T09:27:55.960522Z"}
{"component":"virt-launcher","level":"info","msg":"2024-02-02 09:27:55.474+0000: starting up libvirt version: 9.5.0, package: 6.el9 ([email protected], 2023-08-25-08:53:56, ), qemu version: 8.0.0qemu-kvm-8.0.0-13.el9, kernel: 5.15.0-92-generic, hostname: vgpu","subcomponent":"qemu","timestamp":"2024-02-02T09:27:56.075998Z"}
{"component":"virt-launcher","level":"info","msg":"LC_ALL=C \\PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\HOME=/home/qemu \\XDG_CACHE_HOME=/var/run/kubevirt-private/libvirt/qemu/lib/domain-1-default_vgpu/.cache \\/usr/libexec/qemu-kvm \\-name guest=default_vgpu,debug-threads=on \\-S \\-object '{\"qom-type\":\"secret\",\"id\":\"masterKey0\",\"format\":\"raw\",\"file\":\"/var/run/kubevirt-private/libvirt/qemu/lib/domain-1-default_vgpu/master-key.aes\"}' \\-machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on \\-accel kvm \\-cpu IvyBridge-IBRS,ss=on,vmx=on,pdcm=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on \\-m size=62500864k \\-object '{\"qom-type\":\"memory-backend-ram\",\"id\":\"pc.ram\",\"size\":64000884736}' \\-overcommit mem-lock=off \\-smp 16,sockets=1,dies=1,cores=16,threads=1 \\-object '{\"qom-type\":\"iothread\",\"id\":\"iothread1\"}' \\-uuid 018615c0-fad5-5229-b12e-10537984032c \\-smbios type=1,manufacturer=KubeVirt,product=None,uuid=018615c0-fad5-5229-b12e-10537984032c,family=KubeVirt \\-no-user-config \\-nodefaults \\-chardev socket,id=charmonitor,fd=20,server=on,wait=off \\-mon chardev=charmonitor,id=monitor,mode=control \\-rtc base=utc \\-no-shutdown \\-boot strict=on \\-device '{\"driver\":\"pcie-root-port\",\"port\":16,\"chassis\":1,\"id\":\"pci.1\",\"bus\":\"pcie.0\",\"multifunction\":true,\"addr\":\"0x2\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":17,\"chassis\":2,\"id\":\"pci.2\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x1\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":18,\"chassis\":3,\"id\":\"pci.3\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x2\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":19,\"chassis\":4,\"id\":\"pci.4\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x3\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":20,\"chassis\":5,\"id\":\"pci.5\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x4\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":21,\"chassis\":6,\"id\":\"pci.6\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x5\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":22,\"chassis\":7,\"id\":\"pci.7\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x6\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":23,\"chassis\":8,\"id\":\"pci.8\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x7\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":24,\"chassis\":9,\"id\":\"pci.9\",\"bus\":\"pcie.0\",\"multifunction\":true,\"addr\":\"0x3\"}' \\-device '{\"driver\":\"pcie-root-port\",\"port\":25,\"chassis\":10,\"id\":\"pci.10\",\"bus\":\"pcie.0\",\"addr\":\"0x3.0x1\"}' \\-device '{\"driver\":\"virtio-scsi-pci-non-transitional\",\"id\":\"scsi0\",\"bus\":\"pci.5\",\"addr\":\"0x0\"}' \\-device '{\"driver\":\"virtio-serial-pci-non-transitional\",\"id\":\"virtio-serial0\",\"bus\":\"pci.6\",\"addr\":\"0x0\"}' \\-blockdev '{\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-private/vmi-disks/harddrive/disk.img\",\"node-name\":\"libvirt-3-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-3-format\",\"read-only\":false,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"raw\",\"file\":\"libvirt-3-storage\"}' \\-device '{\"driver\":\"virtio-blk-pci-non-transitional\",\"bus\":\"pci.7\",\"addr\":\"0x0\",\"drive\":\"libvirt-3-format\",\"id\":\"ua-harddrive\",\"bootindex\":1,\"write-cache\":\"on\",\"werror\":\"stop\",\"rerror\":\"stop\"}' \\-blockdev '{\"driver\":\"file\",\"filename\":\"/var/run/kubevirt/container-disks/disk_1.img\",\"node-name\":\"libvirt-2-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-2-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"raw\",\"file\":\"libvirt-2-storage\"}' \\-blockdev '{\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-ephemeral-disks/disk-data/virtiocontainerdisk/disk.qcow2\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-1-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":\"libvirt-2-format\"}' \\-device '{\"driver\":\"ide-cd\",\"bus\":\"ide.0\",\"drive\":\"libvirt-1-format\",\"id\":\"ua-virtiocontainerdisk\",\"bootindex\":2,\"write-cache\":\"on\",\"werror\":\"stop\",\"rerror\":\"stop\"}' \\-netdev '{\"type\":\"tap\",\"fd\":\"21\",\"vhost\":true,\"vhostfd\":\"23\",\"id\":\"hostua-default\"}' \\-device '{\"driver\":\"virtio-net-pci-non-transitional\",\"host_mtu\":1480,\"netdev\":\"hostua-default\",\"id\":\"ua-default\",\"mac\":\"0e:ea:ad:30:6c:75\",\"bus\":\"pci.1\",\"addr\":\"0x0\",\"romfile\":\"\"}' \\-add-fd set=0,fd=19,opaque=serial0-log \\-chardev socket,id=charserial0,fd=17,server=on,wait=off,logfile=/dev/fdset/0,logappend=on \\-device '{\"driver\":\"isa-serial\",\"chardev\":\"charserial0\",\"id\":\"serial0\",\"index\":0}' \\-chardev socket,id=charchannel0,fd=18,server=on,wait=off \\-device '{\"driver\":\"virtserialport\",\"bus\":\"virtio-serial0.0\",\"nr\":1,\"chardev\":\"charchannel0\",\"id\":\"channel0\",\"name\":\"org.qemu.guest_agent.0\"}' \\-audiodev '{\"id\":\"audio1\",\"driver\":\"none\"}' \\-vnc vnc=unix:/var/run/kubevirt-private/c3a21246-cf20-4120-afc2-788d959b08db/virt-vnc,audiodev=audio1 \\-device '{\"driver\":\"VGA\",\"id\":\"video0\",\"vgamem_mb\":16,\"bus\":\"pcie.0\",\"addr\":\"0x1\"}' \\-device '{\"driver\":\"ich9-intel-hda\",\"id\":\"ua-sound1\",\"bus\":\"pcie.0\",\"addr\":\"0x1b\"}' \\-device '{\"driver\":\"hda-duplex\",\"id\":\"ua-sound1-codec0\",\"bus\":\"ua-sound1.0\",\"cad\":0,\"audiodev\":\"audio1\"}' \\-global ICH9-LPC.noreboot=off \\-watchdog-action reset \\-device '{\"driver\":\"vfio-pci-nohotplug\",\"id\":\"ua-gpu-gpu1\",\"sysfsdev\":\"/sys/bus/mdev/devices/334852fe-079b-11ee-9fc7-77463608f467\",\"display\":\"on\",\"ramfb\":true,\"bus\":\"pci.8\",\"addr\":\"0x0\"}' \\-device '{\"driver\":\"virtio-balloon-pci-non-transitional\",\"id\":\"balloon0\",\"free-page-reporting\":true,\"bus\":\"pci.9\",\"addr\":\"0x0\"}' \\-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \\-msg timestamp=on","subcomponent":"qemu","timestamp":"2024-02-02T09:27:56.076403Z"}
{"component":"virt-launcher","level":"info","msg":"2024-02-02T09:27:55.762423Z qemu-kvm: -device {\"driver\":\"vfio-pci-nohotplug\",\"id\":\"ua-gpu-gpu1\",\"sysfsdev\":\"/sys/bus/mdev/devices/334852fe-079b-11ee-9fc7-77463608f467\",\"display\":\"on\",\"ramfb\":true,\"bus\":\"pci.8\",\"addr\":\"0x0\"}: warning: vfio 334852fe-079b-11ee-9fc7-77463608f467: Could not enable error recovery for the device","subcomponent":"qemu","timestamp":"2024-02-02T09:27:56.076540Z"}
{"component":"virt-launcher","level":"info","msg":"Found PID for default_vgpu: 86","pos":"monitor.go:170","timestamp":"2024-02-02T09:27:56.428525Z"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event Domain event=\"resumed\" detail=\"unpaused\" with event id 4 reason 0 received","pos":"client.go:469","timestamp":"2024-02-02T09:27:56.596148Z"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event Domain event=\"started\" detail=\"booted\" with event id 2 reason 0 received","pos":"client.go:469","timestamp":"2024-02-02T09:27:56.601873Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Domain started.","name":"vgpu","namespace":"default","pos":"manager.go:1104","timestamp":"2024-02-02T09:27:56.610794Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"vgpu","namespace":"default","pos":"server.go:208","timestamp":"2024-02-02T09:27:56.614460Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:296","timestamp":"2024-02-02T09:27:56.614691Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: default_vgpu","pos":"client.go:423","timestamp":"2024-02-02T09:27:56.617594Z"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:296","timestamp":"2024-02-02T09:27:56.621526Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: default_vgpu","pos":"client.go:423","timestamp":"2024-02-02T09:27:56.623874Z"}
{"component":"virt-launcher","level":"warning","msg":"PCI_RESOURCE_NVIDIA_COM_GRID_P40-1Q not set for resource nvidia.com/GRID_P40-1Q","pos":"addresspool.go:51","timestamp":"2024-02-02T09:27:57.054391Z"}
{"component":"virt-launcher","level":"info","msg":"host-devices created: [334852fe-079b-11ee-9fc7-77463608f467]","pos":"hostdev.go:98","timestamp":"2024-02-02T09:27:57.054471Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"vgpu","namespace":"default","pos":"server.go:208","timestamp":"2024-02-02T09:27:57.057065Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
{"component":"virt-launcher","level":"warning","msg":"PCI_RESOURCE_NVIDIA_COM_GRID_P40-1Q not set for resource nvidia.com/GRID_P40-1Q","pos":"addresspool.go:51","timestamp":"2024-02-02T09:27:57.123540Z"}
{"component":"virt-launcher","level":"info","msg":"host-devices created: [334852fe-079b-11ee-9fc7-77463608f467]","pos":"hostdev.go:98","timestamp":"2024-02-02T09:27:57.123612Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"vgpu","namespace":"default","pos":"server.go:208","timestamp":"2024-02-02T09:27:57.126359Z","uid":"c3a21246-cf20-4120-afc2-788d959b08db"}
device info:
<device>
<name>mdev_334852fe_079b_11ee_9fc7_77463608f467_0000_03_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:03.0/0000:03:00.0/334852fe-079b-11ee-9fc7-77463608f467</path>
<parent>pci_0000_03_00_0</parent>
<driver>
<name>nvidia-vgpu-vfio</name>
</driver>
<capability type='mdev'>
<type id='nvidia-46'/>
<uuid>334852fe-079b-11ee-9fc7-77463608f467</uuid>
<iommuGroup number='76'/>
</capability>
</device>
kubevirt-cr.yaml
...
configuration:
permittedHostDevices:
mediatedDevices:
- mdevNameSelector: "GRID P40-1Q"
resourceName: "nvidia.com/GRID_P40-1Q"
nodeMediatedDeviceTypes:
- nodeSelector:
kubernetes.io/hostname: cx106
mediatedDevicesTypes:
- nvidia-46
developerConfiguration:
featureGates:
- DisableMDEVConfiguration
- GPU
...
node info:
...
Addresses:
InternalIP: 10.11.0.106
Hostname: cx106
Capacity:
cpu: 40
devices.kubevirt.io/kvm: 1k
devices.kubevirt.io/tun: 1k
devices.kubevirt.io/vhost-net: 1k
ephemeral-storage: 236103552Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 131940700Ki
nvidia.com/GRID_P40-1Q: 1
pods: 110
Allocatable:
cpu: 40
devices.kubevirt.io/kvm: 1k
devices.kubevirt.io/tun: 1k
devices.kubevirt.io/vhost-net: 1k
ephemeral-storage: 217593033163
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 131838300Ki
nvidia.com/GRID_P40-1Q: 1
pods: 110
System Info:
Machine ID: 91740dbc4cc54723a5b127a2fafb0dbc
System UUID: f20619e0-5bcc-11d9-b3a1-d850e60a5ac2
Boot ID: 528ce537-2841-4dcc-9eec-e132c7f19fd6
Kernel Version: 5.15.0-92-generic
OS Image: Ubuntu 22.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://25.0.2
Kubelet Version: v1.23.3
Kube-Proxy Version: v1.23.3
PodCIDR: 10.244.3.0/24
PodCIDRs: 10.244.3.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default virt-launcher-vgpu-mzhnd 1606m (4%) 100m (0%) 65614326336 (48%) 65678326336 (48%) 31m
kube-system calico-node-l6zvh 250m (0%) 0 (0%) 0 (0%) 0 (0%) 40m
kube-system kube-proxy-h2mmh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 40m
kubevirt virt-handler-gkk9s 10m (0%) 0 (0%) 357Mi (0%) 0 (0%) 38m
rook-ceph csi-cephfsplugin-6hsbz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 38m
rook-ceph csi-rbdplugin-mx5xz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 38m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1866m (4%) 100m (0%)
memory 65988667968 (48%) 65678326336 (48%)
ephemeral-storage 100M (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
devices.kubevirt.io/kvm 1 1
devices.kubevirt.io/tun 1 1
devices.kubevirt.io/vhost-net 1 1
nvidia.com/GRID_P40-1Q 1 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 39m kube-proxy
Warning listen tcp4 :30690: bind: address already in use 39m kube-proxy can't open port "nodePort for kubevirt/virtvnc" (:30690/tcp4), skipping it
Normal NodeHasSufficientMemory 40m (x2 over 40m) kubelet Node cx106 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 40m (x2 over 40m) kubelet Node cx106 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 40m (x2 over 40m) kubelet Node cx106 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 40m kubelet Updated Node Allocatable limit across pods
Normal Starting 40m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 39m kubelet Node cx106 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 39m kubelet Node cx106 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 39m kubelet Updated Node Allocatable limit across pods
Normal NodeHasNoDiskPressure 39m kubelet Node cx106 status is now: NodeHasNoDiskPressure
Normal Starting 39m kubelet Starting kubelet.
Normal NodeReady 38m kubelet Node cx106 status is now: NodeReady
VirtualMachine:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vgpu
spec:
running: false
template:
metadata:
labels:
kubevirt.io/domain: win10
spec:
domain:
cpu:
cores: 16
devices:
sound:
name: sound1
# autoattachGraphicsDevice: false #禁止vm自带显示 如果新装系统可以开机装完显卡驱动再加这个参数
gpus:
- deviceName: nvidia.com/GRID_P40-1Q
name: gpu1
disks:
- bootOrder:
cdrom:
bus: sata
#name: cdromiso
- bootOrder: 2
disk:
bus: virtio
name: harddrive
- bootOrder: 2
cdrom:
bus: sata
name: virtiocontainerdisk
machine:
type: q35
resources:
limits:
memory: 64G
nodeSelector:
kubernetes.io/hostname: cx106
volumes:
- name: cdromiso
containerDisk:
image: win10-iso:v1
- name: harddrive
persistentVolumeClaim:
claimName: vgpuhd-clone
- containerDisk:
image: quay.io/kubevirt/virtio-container-disk
name: virtiocontainerdisk
Environment:
- KubeVirt version (use
virtctl version): v1.1.1 - Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:09:57Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
- VM or VMI specifications: VMS
- Cloud provider or hardware configuration: N/A
- OS (e.g. from /etc/os-release): ubuntu 22.04
- Kernel (e.g.
uname -a): Linux cx103 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux - Install tools: N/A
- Others: N/A
@zhangchi6414 the messaging can indeed be improved in virt-handler. The message in your report speaks about PCI_RESOURCE_ while you are assigning an MDEV - a mediated device (NOT PCI) so you can safely ignore this.
Logging can be improved...
@zhangchi6414 the messaging can indeed be improved in virt-handler. The message in your report speaks about
PCI_RESOURCE_while you are assigning anMDEV- a mediated device (NOT PCI) so you can safely ignore this.Logging can be improved...
I don't think it's a useless message, the Windows virtual machine running in virt-lander mounts this device, and the system will also say that the device resource is unavailable. I have no problem using this device in a virtual machine in KVM, is there a good way to fix this error?
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close
@kubevirt-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity. Reopen the issue with
/reopen. Mark the issue as fresh with/remove-lifecycle rotten./close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Author
Are you resolved, i encountered the same issue when using RTX A5000.
