[cloud_usage] vm_instance_id is null for volume usage (usage_type = 6)

[Yuliia7-1]

problem

When tracking volume usage in the cloud_usage table (usage_type = 6), the vm_instance_id field is NULL, even if the volume is currently attached to a VM. This leads to potential inaccuracies in historical usage data, especially when volumes are detached and reattached to different VMs.

Expected Behavior: ACS should record the correct vm_instance_id associated with a volume when usage occurs and store it in the cloud_usage table.

versions

The ACS version is 4.19.1.1

The steps to reproduce the bug

Case 1:

1. Create and attach a volume to VM.
2. Use the VM for some time (allow usage data to be generated).
3. Observe that cloud_usage.vm_instance_id is still NULL for this VM.

Case 2:

1. Create and attach a volume to VM A.
2. Use the VM for some time (allow usage data to be generated).
3. Detach the volume and attach it to VM B.
4. Trigger a reprocessing of usage data for the original time period.
5. Observe that cloud_usage.vm_instance_id is still NULL.

What to do about it?

Consider enhancing the ACS UsageManagerImpl or submitting a patch upstream to ensure vm_instance_id is consistently captured and stored when usage is recorded.

My goal is to accurately track historical usage of volumes in relation to the VMs they were attached to. To achieve this and mitigate the issue of missing vm_instance_id values for usage_type = 6 in the cloud_usage table.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[sudo87]

@DaanHoogland we have similar ticket https://github.com/apache/cloudstack/issues/7399 and based on prior discussion on that ticket, it will require functional definition.

[DaanHoogland]

I see @sudo87 , but I see no transcript of such discussion. There is a draft PR out by @sureshanaparti . Do you think we can move this forward?

[sudo87]

Hi @DaanHoogland, in Draft PR we are fetching vmId (cloud db) in usage parser which isn't visible to it. Fix will require more work on usage event table and event generation logic. cc: @sureshanaparti

[abh1sar]

Will something like this make sense for listUsageRecords?

Scenario:

1. Volume vol1 is created at 18:00:00
2. vol1 is attached to instance-1 at 18:20:00
3. vol1 is detached from instance-1 at 18:39:50 and attached to instances-2 at 18:40:00

Currently the usage is not tracked per VM, so it look like this

    {
      "account": "admin",
      "accountid": "50fa8874-8270-11f0-b91c-dd78c265b147",
      "description": "Volume usage for vol1 (7f63db70-901c-4fa2-863a-779ee1dd02e6) with disk offering Small (8f899760-b55d-46df-be83-2d98cfc38f9e) and size (5.00 GB) 5368709120",
      "domain": "ROOT",
      "domainid": "50fa5f98-8270-11f0-b91c-dd78c265b147",
      "domainpath": "/",
      "enddate": "2025-08-27T18:59:59+0530",
      "offeringid": "8f899760-b55d-46df-be83-2d98cfc38f9e",
      "rawusage": "1",
      "size": 5368709120,
      "startdate": "2025-08-27T18:00:00+0530",
      "tags": [],
      "usage": "1 Hrs",
      "usageid": "7f63db70-901c-4fa2-863a-779ee1dd02e6",
      "usagetype": 6,
      "zoneid": "b1136f3c-e578-4016-9b81-23f319d272fa"
    }

But after per VM tracking, we will get 3 different entries

  {
     "account": "admin",
     "accountid": "50fa8874-8270-11f0-b91c-dd78c265b147",
     "description": "Volume usage for vol1 (7f63db70-901c-4fa2-863a-779ee1dd02e6) with disk offering Small (8f899760-b55d-46df-be83-2d98cfc38f9e) and size (5.00 GB) 5368709120",
     "domain": "ROOT",
     "domainid": "50fa5f98-8270-11f0-b91c-dd78c265b147",
     "domainpath": "/",
     "enddate": "2025-08-27T18:59:59+0530",
     "offeringid": "8f899760-b55d-46df-be83-2d98cfc38f9e",
     "rawusage": "1",
     "size": 5368709120,
     "startdate": "2025-08-27T18:00:00+0530",
     "tags": [],
     "usage": "1 Hrs",
     "usageid": "7f63db70-901c-4fa2-863a-779ee1dd02e6",
     "usagetype": 6,
     "zoneid": "b1136f3c-e578-4016-9b81-23f319d272fa"
   }
   {
     "account": "admin",
     "accountid": "50fa8874-8270-11f0-b91c-dd78c265b147",
     "description": "Volume usage for vol1 (7f63db70-901c-4fa2-863a-779ee1dd02e6) attached to instance-1 (5f51357b-824e-421b-9d01-907f52e726f9) with disk offering Small (8f899760-b55d-46df-be83-2d98cfc38f9e) and size (5.00 GB) 5368709120",
     "domain": "ROOT",
     "domainid": "50fa5f98-8270-11f0-b91c-dd78c265b147",
     "domainpath": "/",
     "enddate": "2025-08-27T18:39:50+0530",
     "offeringid": "8f899760-b55d-46df-be83-2d98cfc38f9e",
     "rawusage": "0.33333",
     "size": 5368709120,
     "startdate": "2025-08-27T18:20:00+0530",
     "tags": [],
     "usage": "0.33333 Hrs",
     "usageid": "7f63db70-901c-4fa2-863a-779ee1dd02e6",
     "usagetype": 6,
     “virtualmachineid”: “5f51357b-824e-421b-9d01-907f52e726f9”,
     "zoneid": "b1136f3c-e578-4016-9b81-23f319d272fa"
   }
   {
     "account": "admin",
     "accountid": "50fa8874-8270-11f0-b91c-dd78c265b147",
     "description": "Volume usage for vol1 (7f63db70-901c-4fa2-863a-779ee1dd02e6) attached to instance-2 (3445j27b-824e-421b-9d01-907f52e726f9) with disk offering Small (8f899760-b55d-46df-be83-2d98cfc38f9e) and size (5.00 GB) 5368709120",
     "domain": "ROOT",
     "domainid": "50fa5f98-8270-11f0-b91c-dd78c265b147",
     "domainpath": "/",
     "enddate": "2025-08-27T18:59:59+0530",
     "offeringid": "8f899760-b55d-46df-be83-2d98cfc38f9e",
     "rawusage": "0.33333",
     "size": 5368709120,
     "startdate": "2025-08-27T18:40:00+0530",
     "tags": [],
     "usage": "0.33333 Hrs",
     "usageid": "7f63db70-901c-4fa2-863a-779ee1dd02e6",
     "usagetype": 6,
     “virtualmachineid”: “5f51357b-824e-421b-9d01-907f52e726f9”,
     "zoneid": "b1136f3c-e578-4016-9b81-23f319d272fa"
   }

The first entry doesn't contain virtualmachineid and tracks the cumulative volume usage to be consistent with the current behaviour. The second and third entry contains per instance volume usage for more granular tracking.

Your thoughts please @weizhouapache @DaanHoogland @sureshanaparti @sudo87?

[DaanHoogland]

Makes sense to me. How about you @Yuliia7-1 ?

[Yuliia7-1]

Exactly what I was looking for, thanks

[weizhouapache]

@abh1sar looks good

[DaanHoogland]

fixed by #11531