[Feature] Run CLI tests via the go test runner and record coverage

[sambhav]

Problem Statement

Currently we run the CLI tests by compiling the CLI and then invoking kyverno test/apply on the compiled CLI. We should instead just call these commands programatically from go test code so that we can also capture the coverage metrics from our vast array of CLI tests.

Solution Description

We need to refactor the CLI to not call os.Exit at random places in the code base and instead return appropriate error. Once this is done, we can use test the cobra commands programatically in go test code. We should also include these tests when capturing coverage metrics.

Alternatives

n/a

Additional Context

Currently our coverage metrics do not account for coverage of our code base due to CLI tests. We should also capture this.

Slack discussion

No response

Research

• [X] I have read and followed the documentation AND the troubleshooting guide.
• [X] I have searched other issues in this repository and mine is not recorded.

[kranurag7]

@samj1912 I would love to work on this. Can you please assign me?

[kranurag7]

@samj1912 I have worked on the issue. First I have removed all the os.Exit(1) under the cmd/cli/kubectl-kyverno directory. Now, the default error is getting passed through fmt.Print(err) mostly. One more thing is that I need to pass the files that the test in depending on while executing the command, otherwise it's throwing an error. It's similar for other two generate and report I'm not sure if this is what we are trying to achieve. Please guide me further.

[SD-13]

Hey, @samj1912 I wanna give it a try. Can you please assign it to me? Thanks,

[JimBugwadia]

This Go 1.20 feature may be worth looking into:

https://www.mgasch.com/2023/02/go-e2e/?s=31

[sidjaiswal2003]

can you please explain the issue?

[coder12git]

@JimBugwadia i would like to work on this issue, I am hoping that this is still not resolved

[coder12git]

/assign