Fix memory stats for KVM

[joseflauzino]

Description

Using KVM hypervisor, the free memory stats of the user VMs that are returned by ACS do not correspond to those observed directly in the VMs (when using the free -m command, for example).

It was identified that the problem was in the memory stats collection process performed by Libvirt. This happens because the period that Libvirt would get memory stats updates for each VM was never set. Thus, even if ACS periodically requested the stats for the hosts, the data obtained were always the same.

This PR solves the mentioned problem. A new Agent configuration parameter called vm.memballoon.stats.period was created. This parameter allows operatos to set the time interval in which Libvirt will get memory stats updates. The default value of this parameter is 60 seconds, the same default value as the vm.stats.interval parameter. However, in a cloud containing multiple Management Servers, operators can set the value of vm.memballoon.stats.period lower than the vm.stats.interval, so that each Management Server always gets updated memory stats for all user VMs (since each Management Server can request stats at different instants).

Types of changes

• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] New feature (non-breaking change which adds functionality)
• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] Enhancement (improves an existing feature and functionality)
• [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• [ ] BLOCKER
• [ ] Critical
• [ ] Major
• [x] Minor
• [ ] Trivial

How Has This Been Tested?

In a local lab, I tested multiple combinations for all the related Agent properties (vm.memballoon.stats.period, vm.stats.interval, and vm.memballoon.disable). In all cases tested, the memory stats of the user VMs were obtained properly. Also, I added unit tests.

[joseflauzino]

@blueorangutan package

[blueorangutan]

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3358

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[acs-robot]

Found UI changes, kicking a new UI QA build @blueorangutan ui

[joseflauzino]

I moved the code to a more appropriate location. Also, I adjusted the code to get the VM list (Libvirt domains) by using libvirt-java instead of a virsh command. Thanks for the reviews.

[acs-robot]

Found UI changes, kicking a new UI QA build @blueorangutan ui

[blueorangutan]

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

[blueorangutan]

UI build: :heavy_check_mark: Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6358 (SL-JID-1609)

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[joseflauzino]

@blueorangutan package

[blueorangutan]

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[sonarqubecloud[bot]]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
11 Code Smells

70.9% Coverage
0.0% Duplication

[blueorangutan]

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3525

[rohityadavcloud]

ping @joseflauzino can you address the review comments?

[joseflauzino]

@weizhouapache, thanks for the review. I addressed the comments with the most straightforward solutions and will address the remaining ASAP.

[joseflauzino]

Thanks for the review, @GabrielBrascher. I will address your comments as well. I will be pushing a new version of the code including all the revisions in the next few days.

[joseflauzino]

Thanks for the review, @weizhouapache and @GabrielBrascher.

I have just committed the new version of the code. These are the relevant changes:

1. Now the stats parameter of the memballoon tag will be set to 0 when vm.memballoon.disable=true.

2. Virsh commands have been replaced by methods from the Domain class. The only exception is the command to set the stats parameter of the memballoon tag (virsh dommemstat <vm-name-or-id> --period <value-in-seconds> --live), which cannot be replaced by the memoryStats(int number) method. The reason is that this method only allows us to get the memory stats and not to set the new gathering period in the VM XML (as desired).

With that said, I will resolve all related threads.

[acs-robot]

Found UI changes, kicking a new UI QA build @blueorangutan ui

[blueorangutan]

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

[blueorangutan]

UI build: :heavy_check_mark: Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6358 (SL-JID-2130)

[joseflauzino]

@blueorangutan package

[blueorangutan]

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov[bot]]

Codecov Report

Merging #6358 (6a7f177) into main (1ee58ec) will increase coverage by 0.02%. The diff coverage is 92.85%.

@@             Coverage Diff              @@
##               main    #6358      +/-   ##
============================================
+ Coverage     10.85%   10.87%   +0.02%     
- Complexity     7106     7117      +11     
============================================
  Files          2485     2485              
  Lines        245417   245499      +82     
  Branches      38326    38334       +8     
============================================
+ Hits          26631    26699      +68     
- Misses       215516   215530      +14     
  Partials       3270     3270              

Impacted Files	Coverage Δ
...ypervisor/kvm/resource/LibvirtDomainXMLParser.java	50.61% <91.66%> (+2.11%)	:arrow_up:
...om/cloud/hypervisor/kvm/resource/LibvirtVMDef.java	67.65% <92.30%> (+0.61%)	:arrow_up:
...ervisor/kvm/resource/LibvirtComputingResource.java	17.03% <93.47%> (+1.34%)	:arrow_up:
...rg/apache/cloudstack/quota/QuotaStatementImpl.java	36.28% <0.00%> (-3.99%)	:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[blueorangutan]

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3964

[sonarqubecloud[bot]]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
22 Code Smells

70.9% Coverage
0.0% Duplication

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian :heavy_multiplication_x: suse15. SL-JID 4003

[blueorangutan]

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 4014