apache
Create Instance from Backup Failed
problem
While I was trying to create an instance from a backup, it failed.
I have tried both NFS and Linstor storage as target pools when restoring from an NFS backup, and encountered this error.
2025-12-10 09:17:21,196 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-53:[ctx-29561268, job-857]) (logid:0ad3bb95) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.CreateVMFromBackupCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to create Instance [i-2-65-VM] from backup [{"externalId":"i-2-64-VM\/2025.12.10.08.53.42","name":"nfs-0-nfs-0","uuid":"3acd3982-663f-4d2f-874c-364c5bb22a4a"}] due to: Unable to restore contents from the backup volume [0c3785e2-9473-4c5f-89d7-7c9e449e8bc7]..
I suspect that during the restoration process, the file name being looked for may not match the actual file name stored in the backup, which could be causing the failure to locate the backup file.
Here is a summary of some information. I will try to upload the relevant logs for this part.
| No. | Storage Type | Backup Stroage | Volume ID | Volume Name in backup | Backup info in database |
|---|---|---|---|---|---|
| 1 | Linstor | NFS | d9283f2c-26e9-4adf-9816-a54537b6e81a | root.drbd1039.qcow2 | [{"uuid":"d9283f2c-26e9-4adf-9816-a54537b6e81a","type":"ROOT","size":107374182400,"path":"d9283f2c-26e9-4adf-9816-a54537b6e81a","deviceId":0,"diskOfferingId":"5081ea5d-87ec-4903-a30d-7fab9c378ce8"}] |
| 2 | NFS | NFS | 0c3785e2-9473-4c5f-89d7-7c9e449e8bc7 | root.0c3785e2-9473-4c5f-89d7-7c9e449e8bc7.qcow2 | [{"uuid":"0c3785e2-9473-4c5f-89d7-7c9e449e8bc7","type":"ROOT","size":107374182400,"path":"0c3785e2-9473-4c5f-89d7-7c9e449e8bc7","deviceId":0,"diskOfferingId":"880ba876-defd-4bf3-b28b-1a973c9dff53"}] |
versions
4.22.0
The steps to reproduce the bug
- Provision virtual machines using LINSTOR and NFS as the primary storage types respectively.
- Back up the VMs to the NFS-based backup repository.
- Attempt to restore the backup to a new virtual machine → Restore fails.
- Shut down the original virtual machine, then restore the same backup to it → Restore succeeds.
What to do about it?
I tried modifying the file name after backup, and when I performed the restore, it prompted that the file could not be found. It seems the issue is not related to the file name as I previously guessed.
@leo79901 can you share the agent log from kvm host while restoring from the backup (for NFS and Linstor storages)? I think, for Linstor the restore volume path might be incorrect, it shouldn't be /mnt..., correct?
{
"org.apache.cloudstack.backup.RestoreBackupCommand": {
"vmName": "i-2-65-VM",
"backupPath": "i-2-64-VM/2025.12.10.08.53.42",
"backupRepoType": "nfs",
"backupRepoAddress": " 10.89.52.101:/nfs_with_vdo/acs-dev",
"backupVolumesUUIDs": [
"0c3785e2-9473-4c5f-89d7-7c9e449e8bc7"
],
"restoreVolumePools": [
{
"uuid": "732a881b-916f-4601-bb16-c5da58831047",
"name": "Linstor",
"id": "1",
"poolType": "Linstor",
"host": "http://10.89.52.111:3370",
"path": "cloudstack",
"port": "3370",
"url": "Linstor://http://10.89.52.111:3370/cloudstack/?ROLE=Primary&STOREUUID=732a881b-916f-4601-bb16-c5da58831047",
"isManaged": "false"
}
],
"restoreVolumePaths": [
"/mnt/732a881b-916f-4601-bb16-c5da58831047/d16d7121-1dc7-4588-9c3d-d141581c485a"
],
"vmExists": "true",
"vmState": "Restoring",
"mountTimeout": "30",
"wait": "0",
"bypassHostMainten ance": "false"
}
}
I think the issue is not with the linstor volume path, but the backup file path itself.
@leo79901 could you please perform a "Create instance from backup" command and a "Restore backup to the same vm" command back to back using the same backup and share the management server log.
Done. Logs of 3 management servers was here:
management-server.node1.log management-server.node2.log management-server.node3.log
The error message on the web is : Failed to create Instance [i-2-74-VM] from backup [{"externalId":"i-2-64-VM/2025.12.11.09.42.11","name":"nfs-0-2025-12-11T01:42:11+0000","uuid":"b712b318-75e3-4e3a-b309-e8353dded853"}] due to: Unable to restore contents from the backup volume [0c3785e2-9473-4c5f-89d7-7c9e449e8bc7]..
Thanks @leo79901 In the above logs, the source vm is on an NFS pool and the destination vm is created on the Linstor Pool.
Earlier you mentioned that you tried both NFS and Linstor as target pools, but it failed.
- Can you please try to create the new instance from backup on the NFS pool and see if it fails. And also share the logs from that.
- Does backup creation and restore to the same VM work if the VM is on Linstor?
I'm not entirely sure about the content you need me to work on, so I conducted a full test. Here are the test steps, results, and logs:
Backed up the Linstor instance, which uses Linstor storage. The backup was successful, with the backup name: linstor-2025-12-12T07:21:22+0000.
Created a new instance using this backup without specifying a storage solution. The creation failed, with the following web information:
Failed to create Instance [i-2-93-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..
Created a new instance using this backup, specifying NFS storage. The creation failed, with the following web information:
Failed to create Instance [i-2-94-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..
Created a new instance using this backup, specifying Linstor storage. The creation failed, with the following web information:
Failed to create Instance [i-2-95-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..
Shut down the virtual machine and performed a self-restore of the virtual machine. The restore failed, with the following log:
(linstor-2025-12-12T07:21:22+0000) Error restoring VM from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549","vmId":91}].
Backed up the NFS instance, which uses NFS storage. The backup was successful, with the backup name: nfs-0.
Created a new instance using this backup without specifying a storage solution. The creation failed, with the following web information:
Failed to create Instance [i-2-96-VM] from backup [{"externalId":"i-2-92-VM/2025.12.12.15.29.17","name":"nfs-0","uuid":"b20c9561-438a-442e-90e8-62161f04de82"}] due to: Unable to restore contents from the backup volume [26acbb35-5b85-4610-b2e9-6f6f8e84d205]..
Created a new instance using this backup, specifying NFS storage. The creation was successful.
Shut down the virtual machine and performed a self-restore of the virtual machine. The restore was successful.
management-server-node1.log management-server-node2.log management-server-node3.log
@leo79901 thanks for the detailed info.
There are 3 notable cases:
- Source VM on Linstor and Destination VM on NFS -> Failed with Backup File doesn't exist.
- Source VM on NFS and Destination VM on Linstor -> Backup file exists but copying the backup file to the destination volume path failed.
- Source VM on NFS and Destination VM on NFS -> This was successful
NAS Bnr plugin currently only supports VMs running on NFS, Ceph, Shared mount point or the local storage. The plugin is treating the Linstor pool like an NFS pool and is creating the backup and restore path incorrectly for it.
Could you execute these two commands please? These will tell us how the volume path in Linstor differs from what NAS BnR is using and what we need to do so that the NAS BnR plugin supports Linstor as well.
- On the kvm host:
virsh -c qemu:///system domblklist i-2-91-VM --details - On the NAS backup repository
10.89.52.101:ls /nfs_with_vdo/acs-dev/i-2-91-VM/2025.12.12.15.21.22
The result:
[root@yf-acs-hci-dev-3 ~]# virsh -c qemu:///system domblklist i-2-91-VM --details
Type Device Target Source
------------------------------------------
block disk vda /dev/drbd1030
file cdrom hdc
[root@yf-acs-nfs ~]# ls /nfs_with_vdo/acs-dev/i-2-91-VM/2025.12.12.15.21.22
domain-config.xml domblklist.xml domiflist.xml dominfo.xml root.drbd1030.qcow2
Linstor names volumes based on the mirror it is currently using. So for example /dev/drbd1030. The NAS plugin backs up the volume to the path <backup_dir>/root.drbd1030.qcow2.
Then during restore, it doesn't know what was the backup file path (it is not constructed using the volume uuid). Also, the actual volume could now be pointing to a different mirror e.g, /dev/drbd1029.
@rp- Is my understanding correct? Could you share some ideas about how to support Backups of Linstor Instances using the NAS BnR plugin? We basically want two things
- Create the backup path and volume path (to restore) in the code
- Rsync should work from the backup path to the volume path.
Linstor names volumes based on the mirror it is currently using. So for example /dev/drbd1030. The NAS plugin backs up the volume to the path <backup_dir>/root.drbd1030.qcow2.
Then during restore, it doesn't know what was the backup file path (it is not constructed using the volume uuid). Also, the actual volume could now be pointing to a different mirror e.g, /dev/drbd1029.
@rp- Is my understanding correct? Could you share some ideas about how to support Backups of Linstor Instances using the NAS BnR plugin? We basically want two things
- Create the backup path and volume path (to restore) in the code
- Rsync should work from the backup path to the volume path.
/dev/drbd1030 is what we write as device path in the libvirt xml, that is the path the DRBD device is exposing for usage (there would be also mapped paths by resource name in /dev/drbd/by-res/, those minor numbers are stable as long the linstor resources weren't completely deleted and recreated.
I'm not sure how the restore works with the NAS backup plugin, can you only restore/revert the same VM? or is a completely new VM instance created and the backup applied? (but than you would need to restore the same paths as the backup had?)
@rp- You can restore to the same VM or to a new VM instance. Restore is basically rsync from the backup file to the instance volume. So, for restore we just need to know the backup file path and the path of the instance volume (same vm or different).
In this case the backup file path is backup_repo:/<backup_dir>/root.drbd1030.qcow2 (NAS adds the root and qcow2 prefix and suffix incorrectly for Linstor which needs to be changed)
But let's say we make some change to the code and the backup path is now <backup_dir>drbd1030.
And let's say the path that the DRBD device is exposing to the user for the volume is /dev/drbd1030.
Will rsync <backup_di>/drbd1030 /dev/drbd1030 work for restoring the backup?
If yes, we need a way to detect the destination path (/dev/drbd1030) from within cloudstack. Is domainxml the only way?
Also, I think we would need to store the backup_path in some metadata?
Will
rsync <backup_di>/drbd1030 /dev/drbd1030work for restoring the backup?If yes, we need a way to detect the destination path (
/dev/drbd1030) from within cloudstack. Is domainxml the only way? Also, I think we would need to store the backup_path in some metadata?
Yes that rsync would work.
I'm afraid right now the only way is the path from domainxml, as we provide that in the kvm plugin as KVMPhysical disk in the agent code. We only store the path uuid, which is the prefixless resource name of the linstor resource.
What we could do in the Linstor plugin is, that we change the paths to /dev/drbd/by-res/cs-{volume.path} instead of /dev/drbd1xxx.
What we could do in the Linstor plugin is, that we change the paths to
/dev/drbd/by-res/cs-{volume.path}instead of/dev/drbd1xxx.
Thanks @rp-. That would be great.
Just to confirm, after this change domblklist will now give /dev/drbd/by-res/cs-{volume.path} instead of /dev/drbdxxxx?
And is the volume_path same as the one which is stored in the cloudstack db?
@rp- Thank you. Will you or someone from Linstor be making these changes? I'll be happy to make the changes required to the NAS plugin.
@abh1sar see:
https://github.com/apache/cloudstack/pull/12300
The full path is:
/dev/drbd/by-res/cs-{volume.path}/0