[BUG] `open_metrics` test is being skipped

[HashNuke]

Describe the bug

Issue reported by @iElephant on #1623 in the comments.

Another side effect observed is that the open_metrics test case will be always skipped as the prometheus_traffic is a dependency task of open_metrics.

To reproduce

[HashNuke]

[HashNuke]

• The error has something to do with the metric names. openmetricsvalidator does not like metrics which have the same name as another metric without the suffix.
• The metrics output should also end with # EOF as the last line.

[HashNuke]

The above issues with the invalid metrics output needs to be fixed in coredns itself.

[HashNuke]

Verified coredns metrics output with the OpenMetrics standards doc.

[HashNuke]

@iElephant This issue stumped me a bit, but I have an answer.

As of now, the cnf-testsuite can only run multiple tests if the task names are separated by @. This is a limitation of the library we use (sam.cr). Found this today.

Example that works:

./cnf-testsuite prometheus_traffic @ open_metrics

Based on a quick check that I ran, we haven't documented this anywhere in the testsuite. I wouldn't recommend documenting the above usage this this isn't make-like and might confuse users.

If there is anything else, please do let us know. Else our team will proceed to close this ticket in a few days.

Notes on different scenarios

./cnf-testsuite prometheus_traffic open_metrics
# Runs only prometheus_traffic test because open_metrics is passed as task args to the prometheus_traffic test (which is ignored).

./cnf-testsuite open_metrics prometheus_traffic
# Runs prometheus_traffic test because open_metrics depends on prometheus_traffic test
# prometheus_traffic is passed as task args to the open_metrics test, which is ignored.

Notes for team

The relevant code block that parses args in sam.cr is here. Notice that everything after the task name is assigned to task_args. The readme of the library also mentions using @ as task separator.

[HashNuke]

@agentpoyo @lixuna No code changes to be made for this issue. I would recommending moving to close this after peer review.

[agentpoyo]

Acceptance Criteria

• [x] Setup k8s cluster
• [x] Setup cnf-testsuite using INSTALL instructions (pull latest main branch from source install that hold these updates)
• [x] Run ./cnf-testsuite cnf_setup cnf-path=sample-cnfs/sample_coredns
• [x] Run ./cnf-testsuite prometheus_traffic @ open_metrics
• [x] Verify that both tests run and open_metrics does not fail as false positive.
• [x] I can see results screenshot here.

[agentpoyo]