Add support for Ceph RGW Object Store

[wido]

trafficstars

Description

This PR adds support for the Ceph RGW as an Object Store Driver in CloudStack

Types of changes

• [x] New feature (non-breaking change which adds functionality)

Screenshots (if appropriate):

How Has This Been Tested?

I have a local Ceph environment (IPv6 only) running with the RADOS Gateway (S3 server) running against which I tested this PR.

It also has SSL enabled so it could verify that this works as expected.

How did you try to break this feature and the system with this change?

I manually tried to create and remove buckets and that worked as expected.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 4.21%. Comparing base (cd8442a) to head (d036e02).

:exclamation: There is a different number of reports uploaded between BASE (cd8442a) and HEAD (d036e02). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (cd8442a)	HEAD (d036e02)
unittests	1	0

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #8389       +/-   ##
============================================
- Coverage     15.54%   4.21%   -11.34%     
============================================
  Files          5495     373     -5122     
  Lines        481076   30463   -450613     
  Branches      62235    5399    -56836     
============================================
- Hits          74785    1283    -73502     
+ Misses       398028   29035   -368993     
+ Partials       8263     145     -8118     

Flag	Coverage Δ
uitests	4.21% <ø> (ø)
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8356

[rohityadavcloud]

@wido I'll help kick tests again - can you check for build failures?

@blueorangutan package

[blueorangutan]

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8524

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8548

[weizhouapache]

test failure found

Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.059 s <<< FAILURE! - in org.apache.cloudstack.storage.datastore.provider.CephObjectStoreProviderImplTest
Error:  testGetName(org.apache.cloudstack.storage.datastore.provider.CephObjectStoreProviderImplTest)  Time elapsed: 0.01 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<Ceph[ RGW]> but was:<Ceph[]>
	at org.apache.cloudstack.storage.datastore.provider.CephObjectStoreProviderImplTest.testGetName(CephObjectStoreProviderImplTest.java:41)

cc @wido

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8554

[rohityadavcloud]

@wido can you have a look at the build failure?

[wido]

I am going to look at the build failure and improving the logging as requested. Give me some time.

[wido]

I just rebased and implemented the new logger way of working.

[wido]

Code wise this PR is now good? We are just waiting for the tests to proceed, correct?

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[weizhouapache]

Code wise this PR is now good? We are just waiting for the tests to proceed, correct?

@wido it is great. I will setup a ceph environment and do some testing (probably next week).

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9781

[DaanHoogland]

@wido the builds fail with

20:25:32 [ERROR] Errors: 
20:25:32 [ERROR]   UriUtilsTest.validateUrl:285 ? IllegalArgument Unable to resolve ipv6.google.c...

I don't understand the lint failure in the GHA, as the file reported does not seem to be touched in this PR.

[wido]

Code wise this PR is now good? We are just waiting for the tests to proceed, correct?

@wido it is great. I will setup a ceph environment and do some testing (probably next week).

Ok! Keep in mind that getting the object browser to work in the UI will be difficult. This is due to CORS and outside the scope of this PR.

That is something we need to fix regardless of this PR

[wido]

@wido the builds fail with

20:25:32 [ERROR] Errors: 
20:25:32 [ERROR]   UriUtilsTest.validateUrl:285 ? IllegalArgument Unable to resolve ipv6.google.c...

I don't understand the lint failure in the GHA, as the file reported does not seem to be touched in this PR.

I am touchting the UriUtils.java file and removing an IPv6 check there. I'll have to verify why this fails then

[DaanHoogland]

I don't understand the lint failure in the GHA, as the file reported does not seem to be touched in this PR.

this was about the lint failure, but that passes now.

[weizhouapache]

Code wise this PR is now good? We are just waiting for the tests to proceed, correct?

@wido it is great. I will setup a ceph environment and do some testing (probably next week).

Ok! Keep in mind that getting the object browser to work in the UI will be difficult. This is due to CORS and outside the scope of this PR.

That is something we need to fix regardless of this PR

Noted Thanks @wido

[wido]

Can I ping somebody to get this one merged?