cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard

Add support to RBD erasure code pools

Open BryanMLima opened this issue 1 year ago • 4 comments

Description

This PR adds support for Ceph erasure code pools, allowing users to specify the data pool required for using this type of Ceph pool.

Types of changes

  • [ ] Breaking change (fix or feature that would cause existing functionality to change)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [x] Enhancement (improves an existing feature and functionality)
  • [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
  • [ ] build/CI
  • [ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • [ ] Major
  • [x] Minor

Screenshots (if appropriate):

Details of the data pool in the UI

image

How Has This Been Tested?

The following tests were executed to verify this patch; tests were executed on replicated pools as well to check if this patch would interfere in it.

  • Deploy of VM in a Ceph replicated pool
  • Deploy of VM in a Ceph erasure code pool
  • Volume resize in a Ceph replicated pool
  • Volume resize in a Ceph erasure code pool
  • Live migration of VM with volume allocated in a Ceph replicated pool
  • Live migration of VM with volume allocated in a Ceph erasure code pool
  • Attach, mount and utilization of volume in a Ceph replicated pool
  • Attach, mount and utilization of volume in a Ceph erasure code pool
  • Template creation from a volume in a Ceph replicated pool
  • Deploy of VM from the template based on volume in a Ceph replicated pool
  • Template creation from a volume in a Ceph erasure code pool
  • Deploy of VM from the template based on volume in a Ceph erasure code pool
  • Volume migration from/to Ceph replicated pool to/from NFS
  • Volume migration from/to Ceph replicated pool to/from iSCSI (SharedMountPoint)
  • Volume migration from/to Ceph erasure code pool to/from NFS
  • Volume migration from/to Ceph erasure code pool to/from iSCSI (SharedMountPoint)
  • Volume migration from/to Ceph erasure code pool to/from Ceph replicated pool
  • Snapshot of a volume from Ceph replicated pool
  • Revert snapshot of a volume from Ceph replicated pool
  • Snapshot of a volume from Ceph erasure code pool
  • Revert snapshot of a volume from Ceph erasure code pool

How did you try to break this feature and the system with this change?

I tried to migrate volumes from different type of pool types (NFS, iSCSI and RBD) to each other, verifying that the migration was successful.

BryanMLima avatar Oct 16 '24 13:10 BryanMLima

Codecov Report

Attention: Patch coverage is 6.42202% with 102 lines in your changes missing coverage. Please review.

Project coverage is 16.18%. Comparing base (81e052c) to head (fce6e14). Report is 360 commits behind head on main.

Files with missing lines Patch % Lines
...hypervisor/kvm/storage/StoragePoolInformation.java 0.00% 35 Missing :warning:
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java 0.00% 18 Missing :warning:
...ain/java/com/cloud/api/query/QueryManagerImpl.java 0.00% 16 Missing :warning:
.../hypervisor/kvm/storage/KVMStoragePoolManager.java 0.00% 10 Missing :warning:
...e/cloudstack/api/response/StoragePoolResponse.java 0.00% 6 Missing :warning:
...torage/datastore/provider/DefaultHostListener.java 0.00% 4 Missing :warning:
...oud/hypervisor/kvm/storage/LibvirtStoragePool.java 0.00% 4 Missing :warning:
.../cloud/hypervisor/kvm/storage/KVMPhysicalDisk.java 70.00% 1 Missing and 2 partials :warning:
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java 0.00% 2 Missing :warning:
...CreatePrivateTemplateFromVolumeCommandWrapper.java 0.00% 1 Missing :warning:
... and 3 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9808      +/-   ##
============================================
- Coverage     16.18%   16.18%   -0.01%     
+ Complexity    13046    13045       -1     
============================================
  Files          5645     5646       +1     
  Lines        494795   494841      +46     
  Branches      59955    59963       +8     
============================================
+ Hits          80065    80069       +4     
- Misses       405897   405937      +40     
- Partials       8833     8835       +2     
Flag Coverage Δ
uitests 4.01% <ø> (-0.01%) :arrow_down:
unittests 17.03% <6.42%> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Oct 16 '24 13:10 codecov[bot]

@blueorangutan package

BryanMLima avatar Oct 16 '24 16:10 BryanMLima

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Oct 16 '24 16:10 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11365

blueorangutan avatar Oct 16 '24 17:10 blueorangutan

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions[bot] avatar Jan 08 '25 13:01 github-actions[bot]

@blueorangutan package

BryanMLima avatar Jan 22 '25 17:01 BryanMLima

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Jan 22 '25 17:01 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12174

blueorangutan avatar Jan 22 '25 18:01 blueorangutan

@DaanHoogland @sureshanaparti @JoaoJandre, I think we can merge this one, right?

BryanMLima avatar Mar 20 '25 10:03 BryanMLima

@blueorangutan package

JoaoJandre avatar Mar 20 '25 12:03 JoaoJandre

@DaanHoogland @sureshanaparti @JoaoJandre, I think we can merge this one, right?

@BryanMLima For formality's sake, we should run the CI still

JoaoJandre avatar Mar 20 '25 12:03 JoaoJandre

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Mar 20 '25 12:03 blueorangutan

@blueorangutan test

DaanHoogland avatar Mar 20 '25 13:03 DaanHoogland

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan avatar Mar 20 '25 13:03 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12848

blueorangutan avatar Mar 20 '25 14:03 blueorangutan

@blueorangutan test

DaanHoogland avatar Mar 24 '25 09:03 DaanHoogland

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan avatar Mar 24 '25 09:03 blueorangutan

[SF] Trillian test result (tid-12807) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 69190 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12807-kvm-ol8.zip Smoke tests completed. 135 look OK, 6 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
test_01_create_template Error 1.15 test_templates.py
test_CreateTemplateWithDuplicateName Error 1.14 test_templates.py
test_02_create_template_with_checksum_sha1 Error 65.61 test_templates.py
test_03_create_template_with_checksum_sha256 Error 65.65 test_templates.py
test_04_create_template_with_checksum_md5 Error 65.64 test_templates.py
test_05_create_template_with_no_checksum Error 65.62 test_templates.py
ContextSuite context=TestTemplates>:setup Error 270.58 test_templates.py
ContextSuite context=TestISOUsage>:setup Error 0.00 test_usage.py
test_01_snapshot_usage Error 24.76 test_usage.py
test_06_download_detached_volume Error 314.07 test_volumes.py
test_13_migrate_volume_and_change_offering Error 124.18 test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup Error 0.00 test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py

blueorangutan avatar Mar 25 '25 05:03 blueorangutan

@blueorangutan package

JoaoJandre avatar Mar 31 '25 12:03 JoaoJandre

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan avatar Mar 31 '25 12:03 blueorangutan

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12934

blueorangutan avatar Mar 31 '25 13:03 blueorangutan

@DaanHoogland could we try the CI again?

JoaoJandre avatar Apr 01 '25 11:04 JoaoJandre

@blueorangutan test

DaanHoogland avatar Apr 01 '25 17:04 DaanHoogland

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan avatar Apr 01 '25 17:04 blueorangutan

[SF] Trillian test result (tid-12877) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 52440 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12877-kvm-ol8.zip Smoke tests completed. 141 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File

blueorangutan avatar Apr 02 '25 08:04 blueorangutan