Automatus tests fail when rules marked with 'platform: machine' are run in a container environment

[mpurg]

Description of problem:

Automatus tests fail with "Environment failed to prepare" when rules, which are not applicable to containers (e.g. platform is machine,system_with_kernel,systemd,...), are tested in a container environment, and which try to manipulate a component which is not applicable to containers (kernel modules, sysctl, systemd services, apparmor profiles, ...).

An example rule which tests fine in a VM but fails to test in a container is all_apparmor_profiles_in_enforce_complain_mode

$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

ERROR - Rule 'all_apparmor_profiles_in_enforce_complain_mode' test setup script 'correct_all_apparmor_profiles_in_enforce.pass.sh' failed with exit code 1
ERROR - Environment failed to prepare, skipping test

By adding the following to the test scenario, the issue is resolved:

if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then
    exit 0
fi

$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

WARNING - Script correct_all_apparmor_profiles_in_enforce.pass.sh using profile (all) notapplicable
WARNING - Rule xccdf_org.ssgproject.content_rule_all_apparmor_profiles_in_enforce_complain_mode evaluation resulted in notapplicable

Reason:

The ssg test suite does not take into account the various applicability definitions in the rule, when deciding which tests to run. The evaluation of applicability (machine vs container) is performed only after the test environment is already prepared, which can result in the test failing if the environment is not setup correctly in a non-supported environment.

SCAP Security Guide Version:

master branch (https://github.com/ComplianceAsCode/content/commit/a11711d14d3252adc05916d5a9419eabd9cce10c)

Operating System Version:

Steps to Reproduce:

1. Add platform: machine to rule all_apparmor_profiles_in_enforce_complain_mode
2. ./build_product ubuntu2204
3. podman build --build-arg "CLIENT_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)" -t ssg_test_suite -f Dockefiles/test_suite-ubuntu2204
4. tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

Actual Results:

See error above

Expected Results:

The expected result is that the tests pass when inside a container because the rule should not be evaluated.

Additional Information/Debugging Steps:

Possible solutions that come to mind:

• the tests should be removed from the set of applicable tests in a container environment
• scenarios are appended with the conditional to return 0 if running in a container environment