elemental-toolkit
elemental-toolkit copied to clipboard
rancher
Rethink upgrade tests
Currently in our tests we are not really testing upgrades, but downgrades, which also has a value in its own, but these cannot be considered upgrades.
To test upgrades I'd suggest to use some pre-published vbox image created with the CI and simply run it to upgrade to the in PR built system image. Probably we could do releases including some vbox image artifact, so it can be downloaded in a workflow, boot from there and then upgrade to image built in opensuse-build step. This way we would do always a real upgrade, from a previous release to bleeding edge.
It sounds good to me, but I'd add that looks like an additional suite/scenario, our current tests are making sure that the introduced changes to the cos-* utils are not breaking the upgrade/recovery mechanism to whatever that version is - so I see them more as functional.
The suggested scenario sounds instead: "Can I upgrade from an old version X (latest master) to the one produced in the PR", which sounds more like an "integration test"
The suggested scenario sounds instead: "Can I upgrade from an old version X (latest master) to the one produced in the PR", which sounds more like an "integration test"
Yes it is an integration test, agree, however our current upgrade test are also kind of integration tests, since the upgrade mechanism relies on rebooting to other systems. Probably we could think of simplified upgrade tests (no need to reboot to the upgraded image and mostly relay on return code) and think of an upgrade/downgrade scenario in which we validate we can go back and forth.
My point is trying to reduce the scope of failures to the actual changes and have a clear view and scenario for tests that include older sources or binaries.
So imagine in an upgrade tests we upgrade to some image and then reboot fails, it is likely that the problem is not the upgrade procedure (upgrade.sh code), but the deployed old image or some backward incompatibility.