CLI needs integration/e2e testing

[janpio]

We should add a suite of tests that test the different use cases of the CLI.

Currently this is only done implicitly via things like paramedic (which is used by all plugin CI) or mobile-spec (which is run manually on releases), which use the CLI to do things. But I doubt this really covers all the cases and probably mostly also only uses already release CLI versions.