Quota custom tariffs

[GutoVeronezi]

Description

This PR solves issue #5891.

The main focus of this PR is to provide a flexible manner for operators to charge and bill their cloud resources. Issue #5891 has more details and flowcharts of all the mapped and affected/changed workflows.

The UI compatibility was not addressed in this PR. However, it's mapped to be the next steps.

Types of changes

• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] Enhancement (improves an existing feature and functionality)
• [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• [x] Major
• [ ] Minor

How Has This Been Tested?

I added unit tests to several methods, tested the whole CRUD, and the usage calculation.

To test the usage calculation I created some activation rules based on the cases described in issue #5891 (doing minor changes, such as changing the UUID to be validated). Then, I manually calculated the results and verified if the output is the same as I obtained in ACS.

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2389

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2395

[GutoVeronezi]

quite a bit @GutoVeronezi ;) is it ready for/do you want review?

@DaanHoogland, just a little =3.

There some details to adjust, like billing of CPU_CLOCK_RATE/CPU_NUMBER/MEMORY; however, reviews are welcome.

[DaanHoogland]

@GutoVeronezi I got to go through only half yesterday, code looks good so far. I'll continue further down the line.

[GutoVeronezi]

Thanks @DaanHoogland & @RodrigoDLopez for the reviews!

[GutoVeronezi]

Issue #5891 was updated addressing the changes regarding CPU_SPEED, CPU_NUMBER and MEMORY quota types.

[nvazquez]

@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2754

[nvazquez]

@blueorangutan package

[nvazquez]

@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2758

[nvazquez]

Thanks @GutoVeronezi, just a comment for future reference: as explained in the PR description we can fix the packaging issue on CentOS7 only by upgrading GCC to version 9.3:

[ERROR] Tests run: 23, Failures: 0, Errors: 23, Skipped: 0, Time elapsed: 5.135 s <<< FAILURE! - in org.apache.cloudstack.utils.jsinterpreter.JsInterpreterTest
[ERROR] injectNullVariableTestInjectNullVariableResultTrue(org.apache.cloudstack.utils.jsinterpreter.JsInterpreterTest)  Time elapsed: 0.001 s  <<< ERROR!
com.cloud.utils.exception.CloudRuntimeException: Unable to instantiate JS interpreter [V8] due to [java.lang.IllegalStateException: J2V8 native library not loaded]
	at org.apache.cloudstack.utils.jsinterpreter.JsInterpreterTest.<init>(JsInterpreterTest.java:53)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: J2V8 native library not loaded
	at org.apache.cloudstack.utils.jsinterpreter.JsInterpreterTest.<init>(JsInterpreterTest.java:53)
Caused by: java.lang.IllegalStateException: J2V8 native library not loaded
Caused by: java.lang.UnsatisfiedLinkError: 
Could not load J2V8 library. Reasons: 
	no libj2v8_linux_x86_64.so in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
	no j2v8_linux_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
	/root/libj2v8_linux_x86_64.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /root/libj2v8_linux_x86_64.so)

[nvazquez]

@GutoVeronezi have tried adding the specific gcc needed version on the packaging/<DIST>/cloud.spec files (or at least for centos7)?

[NuxRo]

@GutoVeronezi @nvazquez Operationally if one must depart from the stock gcc packages which should only be done as a last resort, at least do it the RedHat way and look at employing the devtoolset-7 (or greater) from the CentOS Software Collections. This way everybody gets a proper environment. /IMHO

[rohityadavcloud]

Hi @GutoVeronezi this sounds like a bigger undertaking, I'm still trying to understand why we need a JS interpreter? Pardon me but I don't remember that being used in the original quota feature so that's why asking.

[GutoVeronezi]

Hi @rohityadavcloud, as explained in the spec (issue #5891 - here and here), we're introducing a way to allow operators to define (or not) activation rules for the tariffs. These rules will be written in JS code; therefore, we'll need a JS interpreter to interpret them.

[rohityadavcloud]

Thanks for sharing @GutoVeronezi I'll get back in sometime after reviewing your spec as time allows. As the original co-author of the quota feature (https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS) I'm not sure if defining rules in JavaScript is a good idea but let me review the idea/spec; we've as with all other features a standard way for admin/users to work with CloudStack via APIs and configuration (offering or global settings). Most users/operators may not be JS developers/users. Considering this, you may (if not already) start a thread on users@ to build support on your idea (i.e. who may want to use this, esp the rule definition in JS - is it general-purpose, or just specific users at scclouds?).

[GutoVeronezi]

@rohityadavcloud, the idea behind the use of activation rules is to allow operators to decide when a tariff should be applied and customize the values according to their business requirements. JavaScript (JS) was the best option we found to create and process them, due to its simplicity (when compared with other languages, like Python, C, C++, and Java) and people's understanding of JSON.

The whole design of the feature was done considering that some users/operators could not use it; therefore, the activation rules are not mandatory and the previous behavior is kept. These changes are an enrichment of the quota plugin and have backward compatibility; users can decide to use it or not.

The idea came from some use cases that we have, indeed, however it was designed and developed to address generic needs. Furthermore, before proposing anything, we looked at OpenStack to see how it was handling rating/billing; we noticed that they use rules to decide when one tariff should be applied, and based on those ideas we extended the quota plugin in ACS to make it even more flexible and dynamic than OpenStack.

Regarding opening a thread on users@, the change is a new feature/improvement (allowing users to set activation rules to tariffs) and has backward compatibility. Therefore, I do not see the necessity in this case.

[github-actions[bot]]

Hi @${author}, your pull request has merge conflicts. Can you fix the conflicts and sync your branch with the base branch?

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[GutoVeronezi]

As explained in #5891, we choose J2V8 to be our JS engine based on its relevance, performance and implementation; However, we faced a dependency on a GCC version higher or equal to 4.9. CentOS7, by default, is shipped with GCC 4.8, therefore, to build the packages we would need to build a newer GCC version and install it. In order to avoid this barrier, we searched for another JS engine and found Nashorn Engine (standalone) (which has a similar performance compared to J2V8).

"The Nashorn JavaScript engine was first incorporated into JDK 8 via JEP 174 as a replacement for the Rhino scripting engine" (JEP 372). Due to the rapid pace of development of ECMAScript, the community found Nashorn challenging to maintain. Then, in JDK 11, they started the engine deprecation (JEP 335). After that, the OpenJDK community created a standalone version of the Nashorn Engine, which is available for JDK 11 and after.

Currently, ACS is developed under JDK 11, which still has the engine as built-in; however, looking at future compatibility, we decided to use the standalone Nashorn Engine version.

Nashorn Engine (standalone) will attend our necessities with JS processing inside Java, allowing users to write rules for Quota tariffs, equally as we achieved with J2V8.

[DaanHoogland]

SonarCloud Quality Gate failed.

4 Bugs 0 Vulnerabilities 2 Security Hotspots 64 Code Smells

42.3% Coverage 0.5% Duplication

@GutoVeronezi I marked the security hotspots as not relevant. Can you please look at the bugs and maybe at the code smells. The bugs look serious. The code smells usually aren´t but they tend to be easily fixable. thanks.

[acs-robot]

Found UI changes, kicking a new UI QA build @blueorangutan ui

[blueorangutan]

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.