cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard

Published 20 hours ago • verified publisher icon apache

Fix data volume import / export functionality when using swift as secondary storage

Open skattoju4 opened this issue 3 years ago • 39 comments

Description

This PR is a fix for the data volume import and export functionality when using swift as secondary storage.

Background: Secondary storage is used for templates, snapshots and iso's. A data disk is a volume and is meant to reside on primary storage. However, when importing a volume it is first uploaded to secondary storage and then copied to over to primary storage when attached to a VM. Similarly when exporting a volume, it is copied to secondary storage before it is made available to the user for download. When using swift as a secondary storage, an intermediate staging nfs store is needed when copying volumes between primary and secondary storage. This staging nfs store is leveraged for the volume import and export functionality since volumes need not have a footprint on secondary storage.

Currently this functionality does not work:

When importing a volume the following error is observed:

org.apache.cloudstack.storage.to.VolumeObjectTO cannot be cast to org.apache.cloudstack.storage.to.TemplateObjectTO

Further investigation reveals that there is no provision in the code to handle import of volumes. Currently only template import is supported.

When exporting a volume a URL to the secondary storage is generated but it clicking it does not initiate a download of the exported volume. Further investigation revealed that the url points to a broken symlink which points to an invalid path on the SSVM

Use Cases

  1. User should be able to import a volume from a URL when using swift as secondary storage and XenServer as the hypervisor.
  2. User should be able to export a volume to a secondary storage VM URL when using swift as secondary storage and XenServer as the hypervisor.

Types of changes

  • [ ] Breaking change (fix or feature that would cause existing functionality to change)
  • [ ] New feature (non-breaking change which adds functionality)
  • [x] Bug fix (non-breaking change which fixes an issue)
  • [ ] Enhancement (improves an existing feature and functionality)
  • [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

This addresses https://github.com/apache/cloudstack/issues/4874

Feature/Enhancement Scale

  • [ ] Major
  • [*] Minor

Bug Severity

  • [ ] BLOCKER
  • [ ] Critical
  • [ ] Major
  • [*] Minor
  • [ ] Trivial

How Has This Been Tested?

  1. Create data volume (non-managed storage)
  2. Attach it to a VM.
  3. Create a file system and a test file on the new data volume.
  4. Export volume.
  5. Upload exported volume to a bucket and generate a URL.
  6. Import volume back into CloudStack from URL.
  7. Attach imported disk to a VM.
  8. Verify previously created test file.

skattoju4 avatar Mar 25 '21 18:03 skattoju4

@syed @khos2ow could you take a look at this if you have some time ?

skattoju4 avatar Apr 21 '21 22:04 skattoju4

@skattoju4 is this ready, how do we test it or can you provide test results?

rohityadavcloud avatar May 10 '21 17:05 rohityadavcloud

@rhtyd it should be good to go. A CloudStack setup that uses swift as secondary storage is needed to test. This was tested manually with the following high level steps:

  1. Create data volume (only non-managed storage is supported for now)
  2. Attach it to a VM.
  3. Create a file system and a test file on the new data volume.
  4. Export volume. (tests volume export feature)
  5. Upload exported volume to a bucket and generate a URL.
  6. Import volume back into CloudStack from URL. (tests volume import feature)
  7. Attach imported disk to a VM.
  8. Verify previously created test file.

Its been in production for some time with no side effects so far @pdion891 can attest.

skattoju4 avatar May 10 '21 23:05 skattoju4

The travis ci failures are intermittent.

skattoju4 avatar May 10 '21 23:05 skattoju4

@blueorangutan package

rohityadavcloud avatar Jun 18 '21 01:06 rohityadavcloud

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan avatar Jun 18 '21 01:06 blueorangutan

Packaging result: :heavy_multiplication_x: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 268

blueorangutan avatar Jun 18 '21 02:06 blueorangutan

@blueorangutan package

nvazquez avatar Jun 23 '21 14:06 nvazquez

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan avatar Jun 23 '21 14:06 blueorangutan

Packaging result: :heavy_check_mark: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 339

blueorangutan avatar Jun 23 '21 15:06 blueorangutan

@blueorangutan test

nvazquez avatar Jun 23 '21 17:06 nvazquez

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan avatar Jun 23 '21 17:06 blueorangutan

Trillian test result (tid-1071) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 35011 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4875-t1071-kvm-centos7.zip Intermittent failure detected: /marvin/tests/smoke/test_usage.py Smoke tests completed. 87 look OK, 1 have error(s) Only failed tests results shown below:

Test Result Time (s) Test File
test_01_volume_usage Error 127.92 test_usage.py

blueorangutan avatar Jun 24 '21 03:06 blueorangutan

Let's try again @blueorangutan package

rohityadavcloud avatar Jul 30 '21 05:07 rohityadavcloud

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan avatar Jul 30 '21 05:07 blueorangutan

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian. SL-JID 705

blueorangutan avatar Jul 30 '21 05:07 blueorangutan

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian. SL-JID 713

blueorangutan avatar Jul 30 '21 15:07 blueorangutan

@blueorangutan package

nvazquez avatar Aug 07 '21 03:08 nvazquez

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan avatar Aug 07 '21 03:08 blueorangutan

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian. SL-JID 792

blueorangutan avatar Aug 07 '21 04:08 blueorangutan

Ping @skattoju4 cc @syed @swill - can you check and fix the build failure? Is this PR ready for reviewing/merging?

rohityadavcloud avatar Aug 09 '21 08:08 rohityadavcloud

Ping @skattoju4

rohityadavcloud avatar Sep 08 '21 04:09 rohityadavcloud

Ping @skattoju4 can you fix conflicts and address comments?

rohityadavcloud avatar Sep 14 '21 06:09 rohityadavcloud

Ping @skattoju4

rohityadavcloud avatar Sep 21 '21 06:09 rohityadavcloud

Hi @${author}, your pull request has merge conflicts. Can you fix the conflicts and sync your branch with the base branch?

github-actions[bot] avatar Apr 07 '22 05:04 github-actions[bot]

Found UI changes, kicking a new UI QA build @blueorangutan ui

acs-robot avatar Jul 19 '22 09:07 acs-robot

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan avatar Jul 19 '22 09:07 blueorangutan

@blueorangutan package

shwstppr avatar Jul 19 '22 09:07 shwstppr

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Jul 19 '22 09:07 blueorangutan

UI build: :heavy_check_mark: Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/4875 (SL-JID-1980)

blueorangutan avatar Jul 19 '22 09:07 blueorangutan