[BUG] "resource_policies" (and 4 additional) test crashes

[martin-mat]

Describe the bug The upstream Kubescape test has been renamed causing the testsuite resource_policies test to fail

"resource_policies" test crashes with an error:

$ ./cnf-testsuite resource_policies -l debug
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape_framework_download
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: install_kubescape
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: scan command: /home/ubuntu/.cnf-testsuite/tools/kubescape/kubescape scan framework nsa --use-from /home/ubuntu/.cnf-testsuite/tools/kubescape/nsa.json --exclude-namespaces kube-system,kube-public,kube-node-lease,local-path-storage,litmus,cnf-testsuite --format json --output kubescape_results.json
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: output:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: stderr: {"level":"info","ts":"2024-04-23T13:42:07Z","msg":"ARMO security scanner starting"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"current version 'v2.0.158' is not updated to the latest release: 'v3.0.0'"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"Kubernetes cluster nodes scanning is disabled. This is required to collect valuable data for certain controls. You can enable it using  the --enable-host-scan flag"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"Deprecated format version","run":"--format-version=v2"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Downloading/Loading policy definitions"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Downloaded/Loaded policy"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Accessing Kubernetes objects"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Accessed to Kubernetes objects"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Scanning","cluster":"cnf-setup"}
{"level":"error","ts":"2024-04-23T13:42:16Z","msg":"in 'runRegoOnSingleRule', failed to compile rule, name: linux-hardening, reason: 1 error occurred: linux-hardening:23: rego_parse_error: functions must use = operator (not := operator)\n\tis_unsafe_obj(obj) := fix_paths {\n\t                   ^"}
{"level":"error","ts":"2024-04-23T13:42:16Z","msg":"in 'runRegoOnSingleRule', failed to compile rule, name: linux-hardening, reason: 1 error occurred: linux-hardening:23: rego_parse_error: functions must use = operator (not := operator)\n\tis_unsafe_obj(obj) := fix_paths {\n\t                   ^"}
{"level":"info","ts":"2024-04-23T13:42:16Z","msg":"Done scanning","cluster":"cnf-setup"}

Overall risk-score (0- Excellent, 100- All failed): 7

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the '--submit' flag
Sign up for free: https://portal.armo.cloud/account/sign-up?utm_source=GitHub&utm_medium=CLI&utm_campaign=no_submit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🕵️  Run with '--verbose'/'-v' flag for detailed resources view


I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: task_runner args: #<Sam::Args:0x7fbc5997c480 @arr=[], @named_args={}>
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: Results.file created: results/cnf-testsuite-results-20240423-134216-579.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_installed?  true
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7fbc5997c480 @arr=[], @named_args={}>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: check_cnf_config cnf:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: CNF configs found: 1
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: single_task_runner args: #<Sam::Args:0x7fbc5902ada0 @arr=[], @named_args={"cnf-config" => "cnfs/coredns/cnf-testsuite.yml"}>
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: parse_config_yml config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: airgapped: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_and_set_release_name
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_and_set_release_name config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: airgapped mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: src_helm_directory:
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: predefined_release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7fbc5c283640>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory :  parameters:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: helm_directory:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: manifest_directory:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: Building helm_directory and manifest_directory full paths
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: full_helm_directory: /home/ubuntu/cnf-testsuite/cnfs/coredns/ exists? true
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: full_manifest_directory: /home/ubuntu/cnf-testsuite/cnfs/coredns/ exists? true
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install type count install_type: helm_chart
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install_type: helm_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install_type: manifest_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: installation_type_count: 1
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_destination_dir config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: cnf_destination_dir parsed_config_file config: #<Totem::Config:0x7fbc5c283460>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf destination dir: /home/ubuntu/cnf-testsuite/cnfs/coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: USING EXPORTED CHART PATH
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite-resource_policies: Starting test
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite-resource_policies: cnf_config: #<CNFManager::Config:0x7fbc5c2898c0>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape parse
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape test_by_test_name
E, [2024-04-23 13:42:16 +00:00 #1388680] ERROR -- cnf-testsuite: Cast from Array(JSON::Any) to Hash(K, V) failed, at /usr/share/crystal/src/json/any.cr:274:5:274
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/json/any.cr:273:3 in 'as_h'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/kubescape.cr:69:7 in 'parse'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/kubescape.cr:51:5 in 'parse_test_report'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/workload/security.cr:314:19 in '->'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/log/log.cr:36:3 in 'all_cnfs_task_runner'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/task.cr:38:9 in 'task_runner:task'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/workload/security.cr:311:3 in '->'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam/task.cr:54:39 in 'call'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam/execution.cr:20:7 in 'invoke'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:35:5 in 'invoke'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:53:7 in 'process_tasks'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/cnf-testsuite.cr:132:3 in '__crystal_main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:129:5 in 'main_user_code'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:115:7 in 'main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:141:3 in 'main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '??'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: ./cnf-testsuite in '_start'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: ???
D, [2024-04-23 13:42:17 +00:00 #1388680] DEBUG -- cnf-testsuite: update_yml results: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 0, "items" => []}
D, [2024-04-23 13:42:17 +00:00 #1388680] DEBUG -- cnf-testsuite: update_yml parsed_new_yml: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 2, "items" => []}
I, [2024-04-23 13:42:17 +00:00 #1388680]  INFO -- cnf-testsuite: exception with skipped exit code
I, [2024-04-23 13:42:17 +00:00 #1388680]  INFO -- cnf-testsuite: results yaml: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 2, "items" => []}
$

To Reproduce

• get latest main of cnf-testsuite, compile
• remove any old downloaded kubescape rm -rf ~/.cnf-testsuite/tools/kubescape use coredns sample ./cnf-testsuite cnf_setup cnf-config=./sample-cnfs/sample-coredns-cnf
• run the test ./cnf-testsuite resource_policies

Expected behavior the tests passes

Note1: The test may pass in case that old/workinging kubescape is not wiped. This may be also a reason why github actions did not detect this.

Note2: the issue was most probably introduced by this change. #1992

Note3: "resource_policies" is an essential certification test.

[horecoli]

I quickly analyzed this issue, and it seems that the issue is caused by this change: https://github.com/cnti-testcatalog/testsuite/pull/1992

I compared the old nsa.json with the new nsa.json taken after #1992, and unfortunately, the new nsa.json does not define a test called "Resource policies." As a result, the parsing is failing because there are no results for that test in the output JSON from Kubescape.

[taylor]

@HashNuke @martin-mat @wavell ~~this test was removed upstream in this PR https://github.com/kubescape/regolibrary/pull/586~~

PR to remove was closed without merging. Test renamed. See @HashNuke comments below regarding

• https://github.com/cnti-testcatalog/testsuite/blob/main/docs/LIST_OF_TESTS.md#resource-policies
• https://hub.armosec.io/docs/c-0009

[HashNuke]

Looks like the test was renamed to Resource limits.

• This is the old file - https://github.com/kubescape/regolibrary/blob/b9fff54c7e692384a24e5b83d0f05b8195d182ab/controls/resourcepolicies.json
• This is the new file - https://github.com/kubescape/regolibrary/blob/master/controls/C-0009-resourcelimits.json

The file was renamed twice. The history can be tracked via the new file's link.

controls/resourcepolicies.json -> controls/resourcelimits.json
controls/resourcelimits.json -> controls/C-0009-resourcelimits.json

[lixuna]

Documentation for resource limits at: https://hub.armosec.io/docs/c-0009

[taylor]

See https://hub.armosec.io/docs/c-0009 which is linked from https://github.com/cnti-testcatalog/testsuite/blob/main/docs/LIST_OF_TESTS.md#resource-policies

[martin-mat]

resource_policies is not the only test impacted. Additional:

hostpath_mounts (also cert/essential) removed by this commit: https://github.com/kubescape/regolibrary/commit/e909c92840bce4cf8cbde9b2ad0f8819a7023d05

and 3 platform security tests: control_plane_hardening cluster_admin exposed_dashboard

[horecoli]

In the new version of nsaframework.json for kubescape (available at https://github.com/armosec/regolibrary/releases/download/v1.0.316/nsa ), there is not even a Resource limits test. It appears that this test has been split into two separate tests:

{ "controlID": "C-0270", "patch": { "name": "Ensure CPU limits are set" } }, { "controlID": "C-0271", "patch": { "name": "Ensure memory limits are set" } }

The split was done for that here: https://github.com/kubescape/regolibrary/pull/594

So, I suggest splitting it in the testsuite as well. This will make it more consistent and understandable for users to know which limits are missing, whether it is for CPU or memory.

[HashNuke]

Got some fixes in this branch that resolves this issue - https://github.com/cnti-testcatalog/testsuite/tree/kubescape-version-check

I'll let the build run and see if there is anything else to resolve.

[horecoli]

Okay, @HashNuke, I have seen your changes, so I will let you fix this issue inside that branch.

[HashNuke]

The changes required to fix the kubescape-related issues are in this PR - https://github.com/cnti-testcatalog/testsuite/pull/2004

Without these combined fixes, the main branch build would just fail. So they all have to go in together.

[taylor]

(copying from https://github.com/cnti-testcatalog/testsuite/pull/2004)

resource_policies test is split into: cpu_limits and memory_limits tests

• The Resource Policies control has been split in the Kubescape NSA framework.
• The test has been split in the testsuite accordingly as cpu_limits and memory_limits tests.
• The automated test have also been updated appropriately.
• These new tests have been updated in points.yml and rationale doc and the usage doc.

Because the resouce_policies was an essential test the parts (memory and CPU) that are now two tests have been tagged as essential tests.

@martin-mat @horecoli @HashNuke

[kosstennbl]

Resource limits: Splitting the test seems like a reasonable decision, we were discussing this inside the team and came to the same solution (before Akash's PR).

platform:exposed_dashboard removal: I'm not sure if we should look for ways to replace it later or not. From deleted entry in Rationale.md - this test was existing to cover for a vulnerability in old versions of dashboard. If so - this test might be not needed anymore at all.

Using control id to run tests: Nice solution, didn't even knew that there's a possibility to do it like that in Kubescape. Following question: maybe we could move all tests to using control ID's instead of NSA framework? We require only single test results for each testsuite task, is there a reason to run the whole framework each time?