DIMSI-IS
Error when creating backup of a VM
Hello, an error occurs when creating a backup copy of a virtual machine.
Version cloudstack packages:
ii cloudstack-common 4.19.0.1-shapeblue0 all A common package which contains files which are shared by several CloudStack packages ii cloudstack-management 4.19.0.1-shapeblue0 all CloudStack server library
Logs(docker container):
core | INFO: 10.10.10.2:52938 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:52954 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:42296 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:42300 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:40772 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:40780 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:40780 - "POST /api/v1/tasks/singlebackup/43deaea7-10db-4df4-891b-52cb638ed853 HTTP/1.1" 202 Accepted
core | INFO: 10.10.10.2:40772 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core | INFO: 10.10.10.2:40780 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
worker | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker | "class": algorithms.Blowfish,
worker | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker | absolutely not recommended!
worker |
worker | Please specify a different user using the --uid option.
worker |
worker | User information: uid=0 euid=0 gid=0 egid=0
worker |
worker | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 43deaea7-10db-4df4-891b-52cb638ed853
worker | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 80c1d07c-a890-44a6-9dff-1b75b33f5515
worker | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 616efeb5-9675-4082-8fc0-b14c0e7b431d
worker | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 43deaea7-10db-4df4-891b-52cb638ed853
worker-lrt | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker-lrt | "class": algorithms.Blowfish,
worker-lrt | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker-lrt | absolutely not recommended!
worker-lrt |
worker-lrt | Please specify a different user using the --uid option.
worker-lrt |
worker-lrt | User information: uid=0 euid=0 gid=0 egid=0
worker-lrt |
worker-lrt | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker-lrt | [2024-06-23 13:44:00,143: WARNING/ForkPoolWorker-1] {'id': 39, 'uuid': '43deaea7-10db-4df4-891b-52cb638ed853', 'name': 'i-2-32-VM', 'mem': 1048576, 'cpus': 1, 'host': 'acc91e74-0a23-460b-87a1-2effc730a8f1', 'host_tag': 'pbx', 'cloudstack_instance': True, 'state': 'Running'}
worker-lrt | [2024-06-23 13:44:00,160: WARNING/ForkPoolWorker-1] [i-2-32-VM] Gathering data...
worker-lrt | [2024-06-23 13:44:00,160: WARNING/ForkPoolWorker-1] [i-2-32-VM] Disk(s) found : 1
worker-lrt | [2024-06-23 13:44:00,161: WARNING/ForkPoolWorker-1] [i-2-32-VM] Pre-Flight checks incoming.
worker-lrt | [2024-06-23 13:44:00,346: WARNING/ForkPoolWorker-1] [i-2-32-VM] A snapshot has been detected !
worker-lrt | [2024-06-23 13:44:00,346: WARNING/ForkPoolWorker-1] [i-2-32-VM] VM is currently under snapshot. Checking disk files...
worker-lrt | [2024-06-23 13:44:00,531: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshot i-2-32-VM.snap has been successfully deleted
worker-lrt | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshot deleted.
worker-lrt | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Virtual Machine is now in clean condition.
worker-lrt | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Pre-Flight checks done...
worker-lrt | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshotting virtual machines disks
worker-lrt | [2024-06-23 13:44:00,787: WARNING/ForkPoolWorker-1] [i-2-32-VM] A Borg repository for this VM has been found
worker-lrt | [2024-06-23 13:44:00,788: WARNING/ForkPoolWorker-1] [i-2-32-VM] Borg repository setup is OK
worker-lrt | [2024-06-23 13:44:00,788: WARNING/ForkPoolWorker-1] [i-2-32-VM] Checking borg repository lock status
worker-lrt | [2024-06-23 13:44:01,393: WARNING/ForkPoolWorker-1] [i-2-32-VM] Borg repository is unlocked, job will continue
worker-lrt | [2024-06-23 13:44:01,810: WARNING/ForkPoolWorker-1] [i-2-32-VM] Disk vda has been successfully blockcommited
worker-lrt | [2024-06-23 13:44:01,846: ERROR/ForkPoolWorker-1] Task Single_VM_Backup[e2c3d8ba-3a98-4839-93c4-1272cc102b1c] raised unexpected: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
worker-lrt | Traceback (most recent call last):
worker-lrt | File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
worker-lrt | R = retval = fun(*args, **kwargs)
worker-lrt | File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__
worker-lrt | return self.run(*args, **kwargs)
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 161, in single_vm_backup
worker-lrt | raise backup_error
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 154, in single_vm_backup
worker-lrt | raise startbackup_error
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 150, in single_vm_backup
worker-lrt | backup_result = backup_creation(virtual_machine_info)
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 137, in backup_creation
worker-lrt | raise sequence_error
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 133, in backup_creation
worker-lrt | return backup_sequence(info, host_info)
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 126, in backup_sequence
worker-lrt | raise backup_error
worker-lrt | File "/usr/src/app/app/backup_tasks/single_backup.py", line 93, in backup_sequence
worker-lrt | backup_job.manage_backing_file(disk)
worker-lrt | File "/usr/src/app/app/borg/borg_core.py", line 155, in manage_backing_file
worker-lrt | qemu_img_info = json.loads(qemu_img_info)
worker-lrt | File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
worker-lrt | return _default_decoder.decode(s)
worker-lrt | File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
worker-lrt | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
worker-lrt | File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
worker-lrt | raise JSONDecodeError("Expecting value", s, err.value) from None
worker-lrt | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
worker | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker | "class": algorithms.Blowfish,
worker | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker | absolutely not recommended!
worker |
worker | Please specify a different user using the --uid option.
worker |
worker | User information: uid=0 euid=0 gid=0 egid=0
worker |
worker | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker | [2024-06-23 13:44:04,810: WARNING/ForkPoolWorker-16] DEBUG TASK Failure: text: ({'id': 39, 'uuid': '43deaea7-10db-4df4-891b-52cb638ed853', 'name': 'i-2-32-VM', 'mem': 1048576, 'cpus': 1, 'host': 'acc91e74-0a23-460b-87a1-2effc730a8f1', 'host_tag': 'pbx', 'cloudstack_instance': True, 'state': 'Running'},)
worker | [2024-06-23 13:44:04,894: ERROR/ForkPoolWorker-16] Task Handle task failure[e2e5a3d6-8bb4-4b69-b1a2-7db3403218ff] raised unexpected: ValueError(NoResultFound('No row was found when one was required'))
worker | Traceback (most recent call last):
worker | File "/usr/src/app/app/routes/external_hooks.py", line 59, in filter_external_hook_by_id
worker | pool = results.one()
worker | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/result.py", line 1408, in one
worker | return self._only_one_row(
worker | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/result.py", line 562, in _only_one_row
worker | raise exc.NoResultFound(
worker | sqlalchemy.exc.NoResultFound: No row was found when one was required
worker |
worker | The above exception was the direct cause of the following exception:
worker |
worker | Traceback (most recent call last):
worker | File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
worker | R = retval = fun(*args, **kwargs)
worker | File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__
worker | return self.run(*args, **kwargs)
worker | File "/usr/src/app/app/task_handler.py", line 199, in handle_task_failure
worker | hook = hook_route.filter_external_hook_by_id(policy.externalhook)
worker | File "/usr/local/lib/python3.8/dist-packages/celery/local.py", line 188, in __call__
worker | return self._get_current_object()(*a, **kw)
worker | File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 735, in __protected_call__
worker | return orig(self, *args, **kwargs)
worker | File "/usr/local/lib/python3.8/dist-packages/celery/app/task.py", line 392, in __call__
worker | return self.run(*args, **kwargs)
worker | File "/usr/src/app/app/routes/external_hooks.py", line 64, in filter_external_hook_by_id
worker | raise ValueError(exc) from exc
worker | ValueError: No row was found when one was required
Task logs (i-2-32-VM):
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__ return self.run(*args, **kwargs) File "/usr/src/app/app/backup_tasks/single_backup.py", line 161, in single_vm_backup raise backup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 154, in single_vm_backup raise startbackup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 150, in single_vm_backup backup_result = backup_creation(virtual_machine_info) File "/usr/src/app/app/backup_tasks/single_backup.py", line 137, in backup_creation raise sequence_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 133, in backup_creation return backup_sequence(info, host_info) File "/usr/src/app/app/backup_tasks/single_backup.py", line 126, in backup_sequence raise backup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 93, in backup_sequence backup_job.manage_backing_file(disk) File "/usr/src/app/app/borg/borg_core.py", line 155, in manage_backing_file qemu_img_info = json.loads(qemu_img_info) File "/usr/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/usr/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Hi virsibl,
Could you try the following please? As you are using Cloudstack, the VMs storage mount must be mounted under /mnt/PR_STORAGE_ID to follow cloudstack naming.
I completed the documentation, I apologize if it was unclear previously.
#If you are mounting a Cloudstack Primary storage please respect the Cloudstack format such as: /mnt/PR_STORAGE_ID
# Create directory under /mnt/ corresponding to your CS primary storage
mkdir /mnt/138338fb-xxxx-xxxx-b219-ff968ca3ed3d
# Mount using NFS, to make the mount persistent, edit fstab with corresponding values
mount -v -t nfs -o nolock NFS_server:/nfs_shareCS1 /mnt/138338fb-xxxx-xxxx-b219-ff968ca3ed3d
Hello JoffreyLuang, Thanks for your support.
The backup copy was created successfully. But I have some questions, sorry off topic:
1) When turning off the VM, for some reason it is not possible to make a backup copy. It is not present in the vm lists.
2) As far as I understand, you store all the information about the state and recovery VM in a file, it is not entirely clear why this is done, given the presence of the mariadb database.
# file database.db
database.db: Berkeley DB (Hash, version 9, native byte-order)
3) If I deleted the VM in cloudstack, then I can no longer restore it from the web interface.
4) When restoring the VM, the time of the copy creation is not visible.
Hello @virsibl,
I am closing the issue because you have managed to backup your VM.
For points 1 and 3 you can open new issues or start discussions here https://github.com/DIMSI-IS/BackROLL/discussions/categories/q-a as for point 2.
The point 4 has been done in a branch but we still need to review it before merging. Feel free to submit other ideas here https://github.com/DIMSI-IS/BackROLL/discussions/categories/ideas.
