ci: After the test run the kata configuration is in an 'inconsistent' state

[stevenhorsman]

Description of problem

When trying to add some new CC tests at the end of the containerd test run step: https://github.com/kata-containers/tests/blob/2158ccf1823b80016a5453374c7122764f9e131f/.ci/run.sh#L40 they fail because the kata configuration isn't consistent with it's 'normal' state.

We saw this problem in CC testing, but it's relevant to main as well, it will happen for any test suite that runs after the tracing tests https://github.com/kata-containers/tests/blob/2158ccf1823b80016a5453374c7122764f9e131f/.ci/run.sh#L62 and tries to set the hypervisor.qemu.enable_debug setting through the 'usual' means.

The 'usual' means to enable debug in the code base is to use sed to uncomment the default #enable_debug = true line in the kata configuration toml: e.g. https://github.com/kata-containers/tests/blob/2158ccf1823b80016a5453374c7122764f9e131f/.ci/install_runtime.sh#L116

Adding a test that runs at the end and sets hypervisor.qemu.enable_debug by uncommenting the line in the kata config toml results in.

time="2022-04-27T16:46:08Z" level=fatal msg="run pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: Near line 333 (last key parsed 'hypervisor.qemu.enable_debug'): Key 'hypervisor.qemu.enable_debug' has already been defined.: unknown"

After adding some debug I found that this is due to a new entry enable_debug = false being added in the kata configuration toml after the line valid_entropy_sources = ["/dev/urandom","/dev/random",""]. This means that something is adding in an extra enable_debug = false rather than uncommenting the existing one and therefore when the next test uncomments that there are two entries and the kata runtime can't process it.

I think the problems in with the tracing tests that looks like it might create this entry rather than uncomment the existing one with the call in toggle_debug: https://github.com/kata-containers/tests/blob/3a698f7f022f60dcdf7ae08e3560491fc573b46e/tracing/test-agent-shutdown.sh#L512