cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard

enhancement: add instance info as Libvirt metadata

Open phsm opened this issue 5 months ago • 1 comments
trafficstars

Description

This PR adds <metadata> section to the domain XML of the virtual machine.

It is useful when the additional instance info is needed from within the compute host, for example to use it with a monitoring exporter that would label the virtual machine with the additional information.

The method that generates the metadata object is defined in the HypervisorGuruBase class as protected, so it can be used in all its children classes. Currently I've only used it in the KVMGuru because I do not have any other hypervisor kind to test this on.

Additionally, some dead code that looks to be half-finished, was removed, namely LibvirtVMDef.java: public class MetadataDef and the references to it. Kindly check this part, perhaps I accidentally remove something that was obscurely used by something.

Fixes: #6695

Types of changes

  • [ ] Breaking change (fix or feature that would cause existing functionality to change)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] Enhancement (improves an existing feature and functionality)
  • [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
  • [ ] build/CI
  • [ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • [ ] Major
  • [x] Minor

Bug Severity

  • [ ] BLOCKER
  • [ ] Critical
  • [ ] Major
  • [ ] Minor
  • [ ] Trivial

Screenshots (if appropriate):

The metadata has the following format:

  <metadata>
    <cloudstack:instance xmlns:cloudstack="http://cloudstack.apache.org/instance">
      <cloudstack:zone uuid="e2b3416d-d21a-4907-8865-a3caeeae6a82">ZoneName</cloudstack:zone>
      <cloudstack:pod uuid="962722db-62eb-4ad4-8485-254d87c3050c">PodName</cloudstack:pod>
      <cloudstack:cluster uuid="aff7c9e2-aba8-4e8e-bcec-425d1c5e08ee">clusterName</cloudstack:cluster>
      <cloudstack:name>testname</cloudstack:name>
      <cloudstack:internal_name>i-64-320-VM</cloudstack:internal_name>
      <cloudstack:display_name>testdisplayname</cloudstack:display_name>
      <cloudstack:uuid>365622ff-5902-4c7b-beda-1790f99162f9</cloudstack:uuid>
      <cloudstack:service_offering>
        <cloudstack:name>serviceOfferingName</cloudstack:name>
        <cloudstack:cpu>2</cloudstack:cpu>
        <cloudstack:memory>4096</cloudstack:memory>
        <cloudstack:host_tags>
          <cloudstack:tag>hosttag1</cloudstack:tag>
          <cloudstack:tag>hosttag2</cloudstack:tag>
        </cloudstack:host_tags>
      </cloudstack:service_offering>
      <cloudstack:created_at>2025-06-19T13:13:43</cloudstack:created_at>
      <cloudstack:started_at>2025-06-19T13:17:08</cloudstack:started_at>
      <cloudstack:owner>
        <cloudstack:domain uuid="06fe1128-8252-11ef-8f6c-00163e3b404b">ROOT</cloudstack:domain>
        <cloudstack:account uuid="04c2e6cd-eb39-4619-9dbb-670fe27f2807">PrjAcct-test-1</cloudstack:account>
        <cloudstack:project uuid="0c437d02-7429-47b4-b5a4-19fb4bb452ff">test</cloudstack:project>
      </cloudstack:owner>
      <cloudstack:resource_tags>
        <cloudstack:resource_tag key="restag2">resvalue2</cloudstack:resource_tag>
        <cloudstack:resource_tag key="restag1">resvalue1</cloudstack:resource_tag>
      </cloudstack:resource_tags>
    </cloudstack:instance>
  </metadata>

How Has This Been Tested?

I specifically put an emphasis on compatibility with the previous versions, so having a mgmt server with this feature/agent without this feature and vice versa will not crash the VM startup. These are the tests that I preformed:

Both the Mgmt servers and the Nodes contain this patch:

  • Start a VM on a random host: works
  • Start a VM on a specific host: works
  • Start a VM with and without resource tags: works, the resource_tags becomes an empty array <cloudstack:resource_tags/>
  • Start a VM with and without service offering host tags: works, the host_tags becomes an empty array <cloudstack:host_tags/>
  • VM does not belong to a project: works, the project becomes empty <cloudstack:project uuid=""/>
  • VM belongs to a project: works, the project info is shown.
  • Livemigrate a VM with metadata between the hosts having this patch: works

Mgmt server contains the patch, the node does not contain the patch:

  • Start a VM: works, the metadata section is absent
  • Migrate a VM from a non-patched host to a patched host: works, the metadata is abesnt on the destination domain XML (because it is only formed during the VM start)
  • Migrate a VM from a patched host to a non-patched host: works, the metadata is present on the destination domain XML

Mgmt server does not contain the patch, the host contains the patch:

  • Start a VM: works, the metadata section is absent because createMetadataDef() returns null, and createVMFromSpec() does not try to append add this component to the LibvirtVMDef object.
  • Migrate a VM from a patched host to a non-patched host, and then migrate back: works

How did you try to break this feature and the system with this change?

I specifically checked for the empty values in my tests: service offering without tags, no resource tags, no project etc as those look to be the most dangerous parts. Plus, the code checks for null value everywhere, and uses "unknown" default value when null is occurred.

phsm avatar Jun 19 '25 13:06 phsm

Codecov Report

:x: Patch coverage is 1.64474% with 299 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 17.37%. Comparing base (b0c7719) to head (386890a). :warning: Report is 122 commits behind head on main.

Files with missing lines Patch % Lines
...om/cloud/hypervisor/kvm/resource/LibvirtVMDef.java 0.00% 116 Missing :warning:
...m/cloud/agent/api/to/VirtualMachineMetadataTO.java 0.00% 91 Missing :warning:
.../java/com/cloud/hypervisor/HypervisorGuruBase.java 0.00% 83 Missing :warning:
.../java/com/cloud/agent/api/to/VirtualMachineTO.java 16.66% 5 Missing :warning:
...ervisor/kvm/resource/LibvirtComputingResource.java 57.14% 1 Missing and 2 partials :warning:
...er/src/main/java/com/cloud/hypervisor/KVMGuru.java 0.00% 1 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #11061      +/-   ##
============================================
- Coverage     17.38%   17.37%   -0.01%     
+ Complexity    15282    15281       -1     
============================================
  Files          5891     5892       +1     
  Lines        526356   526631     +275     
  Branches      64270    64307      +37     
============================================
- Hits          91526    91523       -3     
- Misses       424488   424764     +276     
- Partials      10342    10344       +2     
Flag Coverage Δ
uitests 3.61% <ø> (ø)
unittests 18.42% <1.64%> (-0.02%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Jun 19 '25 18:06 codecov[bot]

@blueorangutan package

harikrishna-patnala avatar Jun 20 '25 06:06 harikrishna-patnala

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Jun 20 '25 06:06 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13855

blueorangutan avatar Jun 20 '25 08:06 blueorangutan

@blueorangutan test

DaanHoogland avatar Jun 23 '25 08:06 DaanHoogland

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan avatar Jun 23 '25 08:06 blueorangutan

[SF] Trillian test result (tid-13587) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 62552 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11061-t13587-kvm-ol8.zip Smoke tests completed. 139 look OK, 2 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestAccounts>:setup Error 0.00 test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup Error 0.00 test_accounts.py
test_DeleteDomain Error 12.74 test_accounts.py
test_forceDeleteDomain Failure 12.91 test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup Error 15.14 test_accounts.py
ContextSuite context=TestTemplateHierarchy>:setup Error 1532.60 test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup Error 0.00 test_affinity_groups_projects.py

blueorangutan avatar Jun 24 '25 02:06 blueorangutan

@weizhouapache @DaanHoogland @rohityadavcloud can this be merged?

dalax01 avatar Aug 22 '25 08:08 dalax01

@weizhouapache @DaanHoogland @rohityadavcloud can this be merged?

It needs 2 approvals and manual test

weizhouapache avatar Aug 22 '25 08:08 weizhouapache

Just FYI: this code is now used on ~500 hosts across multiple Cloudstack platforms in our production. So it at least doesn't crash :)

phsm avatar Aug 22 '25 08:08 phsm

Just FYI: this code is now used on ~500 hosts across multiple Cloudstack platforms in our production. So it at least doesn't crash :)

ok @phsm since this is an enhancement, I change the target branch to main

can you fix the conflicts @phsm ?

weizhouapache avatar Aug 22 '25 10:08 weizhouapache

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions[bot] avatar Aug 22 '25 14:08 github-actions[bot]

@blueorangutan package

DaanHoogland avatar Aug 26 '25 11:08 DaanHoogland

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Aug 26 '25 12:08 blueorangutan

Just FYI: this code is now used on ~500 hosts across multiple Cloudstack platforms in our production. So it at least doesn't crash :)

ok @phsm since this is an enhancement, I change the target branch to main

can you fix the conflicts @phsm ?

The conflicts have been fixed. Should be alright now.

phsm avatar Aug 26 '25 12:08 phsm

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14734

blueorangutan avatar Aug 26 '25 13:08 blueorangutan

@blueorangutan test

weizhouapache avatar Aug 26 '25 13:08 weizhouapache

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan avatar Aug 26 '25 13:08 blueorangutan

[SF] Trillian test result (tid-14118) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 54501 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11061-t14118-kvm-ol8.zip Smoke tests completed. 146 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File

blueorangutan avatar Aug 27 '25 05:08 blueorangutan

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions[bot] avatar Sep 24 '25 09:09 github-actions[bot]

@DaanHoogland This is cool... it's possible to have it in next release?

@phsm I like it but i think some merge conflict should be fixed.

thanks

tanganellilore avatar Sep 24 '25 17:09 tanganellilore

Hi, The conflict is resolved, it was just 1 more import from upstream.

phsm avatar Sep 25 '25 07:09 phsm

@blueorangutan package

rosi-shapeblue avatar Oct 13 '25 14:10 rosi-shapeblue

@rosi-shapeblue a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Oct 13 '25 14:10 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15439

blueorangutan avatar Oct 13 '25 15:10 blueorangutan

Summary Report - Test Cases Covered

1. Baseline Test (No Metadata): VM deployed on non-patched host.

Verified domain XML → no metadata section present. Status: Pass.

2. Start VM on Patched Host: VM deployed on patched mgmt + patched agent.

Verified metadata section present in domain XML. All expected fields populated (name, internal_name, uuid, zone, cluster, account, service offering, etc.). Empty project field shown as <cloudstack:project uuid=""/>. Status: Pass.

3. Special Scenarios

  • With/without resource tags → metadata updated correctly.

  • With/without host tags in service offering → cloudstack:host_tags populated or empty as expected.

  • VM deployed with project → project name and UUID serialized correctly.

  • VM deployed without project → empty project element as expected.

  • Live migration patched → patched → metadata preserved.

  • Live migration patched → non-patched → metadata persisted (as expected due to XML transfer). Status: Pass.

4. Compatibility Check

Mgmt patched + agent unpatched → VM started successfully, no metadata generated. Status: Pass.

5. Negative / Edge

  • Default deploy without tags or project → empty but valid metadata elements.
  • Stop / start / migrate tested → no crash, no malformed XML.
  • No fatal errors in logs. Status: Pass.

6. Cleanup: VMs destroyed and expunged.

  • virsh list --all confirmed no remaining domains.
  • No leftover XML or disk files on hosts. Status: Pass.

Detailed Test Report

Baseline Test (No Metadata) - Start a VM before applying the patch

  • Expected Result: No metadata section in the domain XML
  • Actual Result: No metadata in the domain XML
image

Start VM on Patched Host: Deploy a new VM after patching both mgmt and agent

  • Expected Result:

    • A metadata section in the domain XML;
    • It contains cloudstack:instance, name, internal_name, uuid, zone, cluster, account, service offering, etc.
    • No crash or error in the XML structure / Empty fields show unknown or empty (no crash)
  • Actual Result: A metadata section is present in the domain XML; It contains all necessary fields; No crash/error

image

Without/with Resource Tags: Add resource tag to a VM and deploy/start it on a patched host

  • Expected Result:

    • Before a tag is created - the cloudstack:resource_tags/ element present but empty
    • After a tag is added -> metadata section includes cloudstack:resource_tags with key–value pairs.; No crash or malformed XML.
  • Actual Result: Section is empty before a tag is created. Once it is added -> the section contains the added tag; XML structure is valid, no crash or errors observed.

Without Resource Tag image

With Resource Tag image

With Service Offering Host Tags: Deploy/start a VM with a service offering that has host tags.**

Steps:

  1. Create a Service Offering with hosttags=abc.
  2. Update a Host to have the same tag (abc).
  3. Deploy a VM using the tagged Service Offering.
  4. Check the VM’s domain XML on the KVM host:

Expected Result:

  • cloudstack:host_tags section should be present in the VM’s libvirt domain XML.
  • It should contain the tag from the service offering.
  • Metadata should reflect the service offering host tag with no errors or malformed XML.

Actual Result:

  • cloudstack:host_tags section is present in the domain XML.
  • It contains the expected tag abc.
  • Service offering, host, and VM all correctly reference the tag.
  • No crash, errors, or malformed XML.

Before Host Tag

image

After Host Tag

(localcloud) 🐱 > list serviceofferings id=5b2d77be-4c08-4451-b84f-5343e0290c18  | grep hosttags
      "hosttags": "abc",
(localcloud) 🐱 > 
(localcloud) 🐱 > list hosts id=05132830-5e67-4c59-8a4f-dcab8117a95c | grep hosttags
      "explicithosttags": "abc",
      "hosttags": "abc",
(localcloud) 🐱 > 
(localcloud) 🐱 > list virtualmachines id=e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f | grep serviceofferingid
      "serviceofferingid": "5b2d77be-4c08-4451-b84f-5343e0290c18",
(localcloud) 🐱 > 
(localcloud) 🐱 > list virtualmachines id=e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f | grep hostid
      "hostid": "05132830-5e67-4c59-8a4f-dcab8117a95c",
(localcloud) 🐱 >  
image

VM with Project: Deploy/start a VM under a project.

Expected Result:

  • Before deploying in a project:

    • cloudstack:project element is present with an empty UUID and no name (<cloudstack:project uuid=""/>).
    • No crash or malformed XML.
  • After deploying in a project:

    • cloudstack:project element contains the project name and UUID.
    • No crash or malformed XML.

Actual Result:

  • Before deploying in a project:
image
  • After deploying in a project:
image

Live Migration: Live migrate the VM between two patched hosts

Expected Result:

  • Live migration completes successfully.
  • metadata section is intact on the destination host.
  • All fields (including tags, project, zone, etc.) remain unchanged.
  • No crash or malformed XML.

Actual Result:

  • metadata block is fully preserved.
  • All expected fields are intact: zone, pod, cluster, name, uuid, service offering, created/started timestamps, owner info.
  • cloudstack:host_tags/ is still present (empty as in the original).
  • <cloudstack:project uuid=""/> correctly indicates no project association.
  • cloudstack:resource_tags with your env=test tag is preserved.
  • No malformed XML, no missing sections.
localcloud) 🐱 > 
(localcloud) 🐱 > list virtualmachines name=post-patch-instance | grep hostid
      "hostid": "1a6fcc0b-47e7-4813-8c69-481f9ecf493a",
(localcloud) 🐱 > 
(localcloud) 🐱 > list hosts id=1a6fcc0b-47e7-4813-8c69-481f9ecf493a | grep name
      "clustername": "p1-c1",
      "managementservername": "ref-trl-9679-k-mol8-rositsa-kyuchukova-mgmt1.sofia.shapeblue.com",
      "name": "ref-trl-9679-k-Mol8-rositsa-kyuchukova-kvm2",
      "podname": "Pod1",
      "zonename": "ref-trl-9679-k-Mol8-rositsa-kyuchukova"
(localcloud) 🐱 > 
(localcloud) 🐱 > migrate virtualmachine virtualmachineid=eecb752c-159a-43ff-a1b9-c118caebcc5f hostid=05132830-5e67-4c59-8a4f-dcab8117a95c 
{
  "virtualmachine": {
    "account": "admin",
    "affinitygroup": [],
    "arch": "x86_64",
    "cpunumber": 1,
    "cpuspeed": 500,
    "cpuused": "13.9%",
    "created": "2025-10-14T05:00:28+0000",
    "deleteprotection": false,
    "details": {
      "Message.ReservedCapacityFreed.Flag": "false",
      "cpuOvercommitRatio": "2.0"
    },
    "diskioread": 0,
    "diskiowrite": 4,
    "diskkbsread": 0,
    "diskkbswrite": 24,
    "displayname": "post-patch-instance",
    "displayvm": true,
    "domain": "ROOT",
    "domainid": "c7f20973-a8b5-11f0-857f-1e00e4000412",
    "domainpath": "ROOT",
    "guestosid": "c8448805-a8b5-11f0-857f-1e00e4000412",
    "haenable": false,
    "hasannotations": false,
    "hostcontrolstate": "Enabled",
    "hostid": "05132830-5e67-4c59-8a4f-dcab8117a95c",
    "hostname": "ref-trl-9679-k-Mol8-rositsa-kyuchukova-kvm1",
    "hypervisor": "KVM",
    "id": "eecb752c-159a-43ff-a1b9-c118caebcc5f",
    "instancename": "i-2-3-VM",
    "isdynamicallyscalable": false,
    "jobid": "00d25461-2e81-4d29-8b04-c3cfe589a193",
    "jobstatus": 0,
    "lastupdated": "2025-10-14T07:08:02+0000",
    "memory": 512,
    "memoryintfreekbs": -1,
    "memorykbs": 524288,
    "memorytargetkbs": 524288,
    "name": "post-patch-instance",
    "networkkbsread": 0,
    "networkkbswrite": 0,
    "nic": [
      {
        "broadcasturi": "vlan://1934",
        "deviceid": "0",
        "extradhcpoption": [],
        "id": "64cc9d5d-fce6-4b87-b360-de0cd0402628",
        "isdefault": true,
        "isolationuri": "vlan://1934",
        "macaddress": "02:01:00:cc:00:01",
        "networkid": "92de5793-fef5-4ae6-9128-9104b14a8330",
        "networkname": "test",
        "secondaryip": [],
        "traffictype": "Guest",
        "type": "L2"
      }
    ],
    "osdisplayname": "CentOS 5.5 (64-bit)",
    "ostypeid": "c8448805-a8b5-11f0-857f-1e00e4000412",
    "passwordenabled": false,
    "pooltype": "NetworkFilesystem",
    "receivedbytes": 0,
    "rootdeviceid": 0,
    "rootdevicetype": "ROOT",
    "securitygroup": [],
    "sentbytes": 0,
    "serviceofferingid": "2b4c988f-d7b2-4455-a89d-dee618c49aa1",
    "serviceofferingname": "Small Instance",
    "state": "Running",
    "tags": [
      {
        "account": "admin",
        "domain": "ROOT",
        "domainid": "c7f20973-a8b5-11f0-857f-1e00e4000412",
        "key": "env",
        "resourceid": "eecb752c-159a-43ff-a1b9-c118caebcc5f",
        "resourcetype": "UserVm",
        "value": "test"
      }
    ],
    "templatedisplaytext": "CentOS 5.5(64-bit) no GUI (KVM)",
    "templateformat": "QCOW2",
    "templateid": "c7fd765d-a8b5-11f0-857f-1e00e4000412",
    "templatename": "CentOS 5.5(64-bit) no GUI (KVM)",
    "templatetype": "BUILTIN",
    "userid": "21e941c7-a8b6-11f0-857f-1e00e4000412",
    "username": "admin",
    "zoneid": "3550556c-5e4f-4bee-8582-5c4db3c98511",
    "zonename": "ref-trl-9679-k-Mol8-rositsa-kyuchukova"
  }
}
[root@ref-trl-9679-k-Mol8-rositsa-kyuchukova-kvm1 ~]# virsh dumpxml i-2-3-VM | sed -n '/<metadata>/,/<\/metadata>/p'
  <metadata>
    <cloudstack:instance xmlns:cloudstack="http://cloudstack.apache.org/instance">
      <cloudstack:zone uuid="3550556c-5e4f-4bee-8582-5c4db3c98511">ref-trl-9679-k-Mol8-rositsa-kyuchukova</cloudstack:zone>
      <cloudstack:pod uuid="078866e4-119e-40b9-9eec-b0431903f6f9">Pod1</cloudstack:pod>
      <cloudstack:cluster uuid="1ed93f29-0ada-43b2-b07d-007f1eeff3b9">p1-c1</cloudstack:cluster>
      <cloudstack:name>post-patch-instance</cloudstack:name>
      <cloudstack:internal_name>i-2-3-VM</cloudstack:internal_name>
      <cloudstack:display_name>post-patch-instance</cloudstack:display_name>
      <cloudstack:uuid>eecb752c-159a-43ff-a1b9-c118caebcc5f</cloudstack:uuid>
      <cloudstack:service_offering>
        <cloudstack:name>Small Instance</cloudstack:name>
        <cloudstack:cpu>1</cloudstack:cpu>
        <cloudstack:memory>512</cloudstack:memory>
        <cloudstack:host_tags/>
      </cloudstack:service_offering>
      <cloudstack:created_at>2025-10-14T05:00:28</cloudstack:created_at>
      <cloudstack:started_at>2025-10-14T05:16:31</cloudstack:started_at>
      <cloudstack:owner>
        <cloudstack:domain uuid="c7f20973-a8b5-11f0-857f-1e00e4000412">ROOT</cloudstack:domain>
        <cloudstack:account uuid="21e83ed9-a8b6-11f0-857f-1e00e4000412">admin</cloudstack:account>
        <cloudstack:project uuid=""/>
      </cloudstack:owner>
      <cloudstack:resource_tags>
        <cloudstack:resource_tag key="env">test</cloudstack:resource_tag>
      </cloudstack:resource_tags>
    </cloudstack:instance>
  </metadata>
[root@ref-trl-9679-k-Mol8-rositsa-kyuchukova-kvm1 ~]# 

Live Migration: Live migrate the VM between patched and non-patched host

(localcloud) 🐱 > migrate virtualmachine virtualmachineid=e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f hostid=77a38c46-5c5d-4bba-a2be-1bf58116c838 
{
  "virtualmachine": {
    "account": "admin",
    "affinitygroup": [],
    "arch": "x86_64",
    "cpunumber": 1,
    "cpuspeed": 500,
    "cpuused": "13.79%",
    "created": "2025-10-14T05:32:30+0000",
    "deleteprotection": false,
    "details": {
      "cpuOvercommitRatio": "2.0"
    },
    "diskioread": 0,
    "diskiowrite": 4,
    "diskkbsread": 0,
    "diskkbswrite": 24,
    "displayname": "host-tagged-instance-abc",
    "displayvm": true,
    "domain": "ROOT",
    "domainid": "c7f20973-a8b5-11f0-857f-1e00e4000412",
    "domainpath": "ROOT",
    "guestosid": "c8448805-a8b5-11f0-857f-1e00e4000412",
    "haenable": false,
    "hasannotations": false,
    "hostcontrolstate": "Enabled",
    "hostid": "77a38c46-5c5d-4bba-a2be-1bf58116c838",
    "hostname": "host3",
    "hypervisor": "KVM",
    "id": "e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f",
    "instancename": "i-2-6-VM",
    "isdynamicallyscalable": false,
    "jobid": "f4059d47-b81a-4d5c-909d-14c989d96d66",
    "jobstatus": 0,
    "lastupdated": "2025-10-14T07:50:18+0000",
    "memory": 512,
    "memoryintfreekbs": -1,
    "memorykbs": 524288,
    "memorytargetkbs": 524288,
    "name": "host-tagged-instance-abc",
    "networkkbsread": 0,
    "networkkbswrite": 0,
    "nic": [
      {
        "broadcasturi": "vlan://1934",
        "deviceid": "0",
        "extradhcpoption": [],
        "id": "03e2f2b9-cdf7-4432-a484-c0dd492362c9",
        "isdefault": true,
        "isolationuri": "vlan://1934",
        "macaddress": "02:01:00:cc:00:04",
        "networkid": "92de5793-fef5-4ae6-9128-9104b14a8330",
        "networkname": "test",
        "secondaryip": [],
        "traffictype": "Guest",
        "type": "L2"
      }
    ],
    "osdisplayname": "CentOS 5.5 (64-bit)",
    "ostypeid": "c8448805-a8b5-11f0-857f-1e00e4000412",
    "passwordenabled": false,
    "pooltype": "NetworkFilesystem",
    "receivedbytes": 0,
    "rootdeviceid": 0,
    "rootdevicetype": "ROOT",
    "securitygroup": [],
    "sentbytes": 0,
    "serviceofferingid": "5b2d77be-4c08-4451-b84f-5343e0290c18",
    "serviceofferingname": "TaggedSO",
    "state": "Running",
    "tags": [],
    "templatedisplaytext": "CentOS 5.5(64-bit) no GUI (KVM)",
    "templateformat": "QCOW2",
    "templateid": "c7fd765d-a8b5-11f0-857f-1e00e4000412",
    "templatename": "CentOS 5.5(64-bit) no GUI (KVM)",
    "templatetype": "BUILTIN",
    "userid": "21e941c7-a8b6-11f0-857f-1e00e4000412",
    "username": "admin",
    "zoneid": "3550556c-5e4f-4bee-8582-5c4db3c98511",
    "zonename": "ref-trl-9679-k-Mol8-rositsa-kyuchukova"
  }
}

XML is preserved

[root@ref-trl-9676-k-Mol8-rositsa-kyuchukova-kvm1 ~]# virsh list --all
 Id   Name       State
--------------------------
 1    s-1-VM     running
 3    i-2-4-VM   running

[root@ref-trl-9676-k-Mol8-rositsa-kyuchukova-kvm1 ~]# virsh list --all
 Id   Name       State
--------------------------
 1    s-1-VM     running
 3    i-2-4-VM   running
 4    i-2-6-VM   running

[root@ref-trl-9676-k-Mol8-rositsa-kyuchukova-kvm1 ~]# virsh dumpxml i-2-6-VM
<domain type='kvm' id='4'>
  <name>i-2-6-VM</name>
  <uuid>e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f</uuid>
  <description>CentOS 5.5 (64-bit)</description>
  <metadata>
    <cloudstack:instance xmlns:cloudstack="http://cloudstack.apache.org/instance">
      <cloudstack:zone uuid="3550556c-5e4f-4bee-8582-5c4db3c98511">ref-trl-9679-k-Mol8-rositsa-kyuchukova</cloudstack:zone>
      <cloudstack:pod uuid="078866e4-119e-40b9-9eec-b0431903f6f9">Pod1</cloudstack:pod>
      <cloudstack:cluster uuid="1ed93f29-0ada-43b2-b07d-007f1eeff3b9">p1-c1</cloudstack:cluster>
      <cloudstack:name>host-tagged-instance-abc</cloudstack:name>
      <cloudstack:internal_name>i-2-6-VM</cloudstack:internal_name>
      <cloudstack:display_name>host-tagged-instance-abc</cloudstack:display_name>
      <cloudstack:uuid>e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f</cloudstack:uuid>
      <cloudstack:service_offering>
        <cloudstack:name>TaggedSO</cloudstack:name>
        <cloudstack:cpu>1</cloudstack:cpu>
        <cloudstack:memory>512</cloudstack:memory>
        <cloudstack:host_tags>
          <cloudstack:tag>abc</cloudstack:tag>
        </cloudstack:host_tags>
      </cloudstack:service_offering>
      <cloudstack:created_at>2025-10-14T05:32:30</cloudstack:created_at>
      <cloudstack:started_at>2025-10-14T05:32:31</cloudstack:started_at>
      <cloudstack:owner>
        <cloudstack:domain uuid="c7f20973-a8b5-11f0-857f-1e00e4000412">ROOT</cloudstack:domain>
        <cloudstack:account uuid="21e83ed9-a8b6-11f0-857f-1e00e4000412">admin</cloudstack:account>
        <cloudstack:project uuid=""/>
      </cloudstack:owner>
      <cloudstack:resource_tags/>
    </cloudstack:instance>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>250</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Apache Software Foundation</entry>
      <entry name='product'>CloudStack KVM Hypervisor</entry>
      <entry name='serial'>e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f</entry>
      <entry name='uuid'>e5fc1cd8-6ea0-4c8b-a04f-0e2e3835005f</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <topology sockets='1' dies='1' cores='1' threads='1'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
  </cpu>
  <clock offset='utc'>
    <timer name='kvmclock'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/4a6ffbeb-b5f3-3e7d-a3a7-699b384a0bb0/f5984422-65d1-4361-940f-5f72c359ee68' index='2'/>
      <backingStore type='file' index='3'>
        <format type='qcow2'/>
        <source file='/mnt/4a6ffbeb-b5f3-3e7d-a3a7-699b384a0bb0/c7fd765d-a8b5-11f0-857f-1e00e4000412'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>f598442265d14361940f</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='02:01:00:cc:00:04'/>
      <source bridge='breth1-1934'/>
      <bandwidth>
        <inbound average='25000' peak='25000'/>
        <outbound average='25000' peak='25000'/>
      </bandwidth>
      <target dev='vnet5'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/i-2-6-VM.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5901' autoport='yes' listen='10.0.33.96'>
      <listen type='address' address='10.0.33.96'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <watchdog model='i6300esb' action='none'>
      <alias name='watchdog0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

Cleanup: Destroy VMs, verify no unexpected libvirt leftovers

Steps

  1. Delete/expunge VM
  2. Verify VM is not present on the host / no leftovers

Actual Result

VM expunged, removed from Host

image

rosi-shapeblue avatar Oct 14 '25 08:10 rosi-shapeblue

@rosi-shapeblue thanks for testing the PR.

@phsm the PR changes looks good, but I've a small concern, can you please help me with it. Is it fine to turn on this feature by default ? as existing VM instances XML will be updated after upgrading to this version and I'm not sure if it will cause any potential issues.

Do you think we can add feature toggle to switch on and off with a global setting ? cc @DaanHoogland

harikrishna-patnala avatar Oct 14 '25 12:10 harikrishna-patnala

Well, the Libvirt metadata is specifically designed to not influence the VM behavior, so it will be effectively invisible to the VM. Therefore, it is technically safe: an existing VM gets the metadata tag after stop/start but the behavior won't change.

However, the final word has to be spoken by actual Cloudstack maintainers.

phsm avatar Oct 14 '25 12:10 phsm

Well, the Libvirt metadata is specifically designed to not influence the VM behavior, so it will be effectively invisible to the VM. Therefore, it is technically safe: an existing VM gets the metadata tag after stop/start but the behavior won't change.

However, the final word has to be spoken by actual Cloudstack maintainers.

Thanks for the reply @phsm I get it what you are trying to say, but all operators may not want the data to be exposed on the hypervisor. Do you think we can simply solve this by introducing a global setting ?

harikrishna-patnala avatar Oct 14 '25 12:10 harikrishna-patnala

I think we need a ON/OFF global setting, some operators might not want the extra information visible at the hypervisor level.

alexandremattioli avatar Oct 14 '25 12:10 alexandremattioli