Backwards/forwards compatibility needs for swtpm state vs QEMU live migration

[berrange]

Describe the bug This issue arises from the comments leading up to https://bugzilla.redhat.com/show_bug.cgi?id=2090219#c37

I wanted to clarify the needs around backwards/forwards compatibility for swtpm state and how it impacts on QEMU live migration compatibility, as I'm not sure that current swtpm approach to its state format is compatible with what QEMU deployments need.

On the QEMU side, let us consider two arbitrary point versions 6.1.0 and 6.2.0, which each defined a new machine type 'pc-q35-6.1' and 'pc-q35-6.2' respectively. From the QEMU POV the following migration scenarios apply

• QEMU 6.1.0 with --machine pc-q35-6.1 migrating to QEMU 6.1.0 with --machine pc-q35-6.1 => PASS
• QEMU 6.1.0 with --machine pc-q35-6.1 migrating to QEMU 6.2.0 with --machine pc-q35-6.1 => PASS
• QEMU 6.2.0 with --machine pc-q35-6.2 migrating to QEMU 6.1.0 with --machine pc-q35-6.1 => FAIL (expected, the tgt machine type is older)
• QEMU 6.2.0 with --machine pc-q35-6.2 migrating to QEMU 6.1.0 with --machine pc-q35-6.2 => FAIL (expected, t6.1.0 doesn't support a pc-q35-6.2 machine type)
• QEMU 6.2.0 with --machine pc-q35-6.1 migrating to QEMU 6.1.0 with --machine pc-q35-6.1 => PASS
• QEMU 6.2.0 with --machine pc-q35-6.1 migrating to QEMU 6.2.0 with --machine pc-q35-6.1 => PASS

IOW, after a QEMU VM is launched, it is expected to be possible to live migrate it to arbitrarily newer or older QEMU versions, provided they all support the machine type version the VM was initially launched with.

This guarantee applies transitively for as long as the VM is running. ie cold boot on QEMU 6.1.0, live migrate to 6.2.0 and then live migrate a second time to 6.1.0 is explicitly supported.

Note that in addition to live migration, this all applies to save/restore-to-disk, live snapshots/checkpoints too, as they all leverage QEMU's migration logic for serializing VM state.

Management applications such as oVirt, OpenStack typically set the machine type version globally in a deployment, allowing them to achieve live migration compatibility across a pool of hosts, even if they have different QEMU versions installed.

Based on what was explained in bug 2090219 linked above, I does not sound like swtpm is compatible with that in two ways.

• If a VM is launched with QEMU 6.1.0 provisioning vTPM with swtpm NN, and live migrate to a host with QEMU 6.2.0 and swtpm NN+1, the vTPM state on the target host is liable to be "upgraded" to a new format. This would prevent the migration from QEMU 6.2.0 + swtpm NN+1, back to QEMU 6.1.0 + swtpm NN. There is no way to get back to the swtpm NN state format

• If a VM is luanched with QEMU 6.2.0 but machine type pc-q35-6.1.0 and swtpm NN+1, if there was new swtpm state format in NN+1 be impossible to live migrate back to a host with QEMU 6.1.0 and swtpm NN, despite having intentionally selected a compatible machine type. There is no way to request that swtpm NN+1 creates a vTPM state that is compatible with swtpm NN

Today this may not have much impact on real world QEMU deployments, because usage of vTPM is still relatively rare. With the drive towards considering TPMs and UEFI as mandatory features, the usage of vTPMs is likely to increased significantly in the relatively near term. As such this state format limitation from swtpm is likely to become a significant problem for mgmt of QEMU deployments, by restricting the administrators freedom in usage of live migration.

At the absolute bare minimum IMHO, swtpm needs to be able to preserve the state format that was used when the VM was initially priovisioned for the lifetime of the VM. ie it needs to support round tripping a VM from version NN forwards to version NN+1 and then backwards again to NN. There must be no silent state format alternations as a side effect of live migrating. This should be achievable without any conscious user interaction in config settings - swtpm simply needs to honour the state format it finds.

To be able to fully match QEMU requirements for live migration though, the would require more advanced features. The ability to report on a list of supported state formats per swtpm binary, and provide an explicit choice of format version at time of provisioning the vTPM with swtpm_setup. This would allow for provisioning a brand new VM with QEMU 6.2.0 and swtpm NN+1, and then live migrating them backwards to QEMU 6.1.0 and swtpm NN.

[stefanberger]

The issue with the TPM 2 is that it's a crypto/security device with a complex implementation of multiple thousands of lines of code and more than 100 distinct commands of which some of them influence the state of the device. The testing of this device is crucial and I rely on existing test suites. Just supporting FIPS mode for example and disabling algorithms and key sized causes failures in those test suites and whole subtest don't work anymore.

So here's a discussion about the issue from here;

Future of libtpms: Downgrading (of state for support of migration)

I think this is an important discussion to have since downgrading of state has come up in some discussions and it's not quite clear how to go about.

First, downgrading of state means that the state of a TPM 2.0 can be migrated for example from libtpms version 0.12 to version 0.9, or more general from version (n+m) to version n and with the latest release being version 0.9, n can only possibly be version 0.9 since previous versions do not allow downgrading anyway. There is at least the following to consider for downgrading:

Applications cannot just loose functionality: Assuming a new crypto algorithm was added to v0.12 and we now want to migrate to a machine that only has libtpms v0.9 without the crypto algorithm, applications or kernel drivers would not be able to use the crypto algorithm anymore and en- or decrypt data. So just loosing functionality when downgrading is generally not an option.

The solution may be to extend the code in such a way that one can run libtpms v0.12 'at the level' of libtpms v0.9. This means that all additional algorithms that were added between v0.9 and v0.12 can be made inaccessible and invisible, meaning applications neither can use these algorithms nor that the TPM 2.0 would advertise these algorithms as available. But this also means that a version of libtpms later than v0.9 must be able to write the state in the format understandable to the previous version, so libtpms v0.12 needs to be able to write the state understandable for v0.9. And of course, for this all to work the user or cloud admin has to know at which level to run libtpms so that this whole downgrading story has a chance to work.

A couple of concerns that arise out of this:

• What effects does this have on the code when future extensions need to be made accessible or invisible? I cannot predict how the TPM 2 code will evolve but it has algorithm agility and could theoretically be extended in many ways. futureTPM tries to add quantum-safe crypto to TPM 2 for example.
• How does one test this then?

Committing to downgrading of state considering unknown future evolution of the TPM 2 code is 'difficult' also considering the fact that it's a complicated device already and testing just its plain functionality is quite an involved process just to begin with. Now we would need to put emphasis on testing of downgrading of state...

So, the following options exist:

• Libtpms v0.9 is the last version. It's currently more advanced than any TPM 2.0 out there and implements the latest revision of the spec, revision 164. Eventually all systems will be on libtpms v0.9 level and downgrading won't be an issue.
• Libtpms gets further developed as before but it is the sole responsibility of the distros to package future versions of libtpms and ensure that their migration requirements are met. Libtpms will not commit to downgrading of state.
• We (additional maintainers required) commit to downgrade-ability of libtpms as outlined above (unless of course someone else comes up with a better definition). **

Also, from what I had heared last year further TPM 2 development would come to an end.

At the absolute bare minimum IMHO, swtpm needs to be able to preserve the state format that was used when the VM was initially priovisioned for the lifetime of the VM. ie it needs to support round tripping a VM from version NN forwards to version NN+1 and then backwards again to NN. There must be no silent state format alternations as a side effect of live migrating. This should be achievable without any conscious user interaction in config settings - swtpm simply needs to honour the state format it finds.

I don't know what this means if the project was to commit to this and what the future development of TPM 2 may or may not bring, if it happens at all. New commands could be added to the TPM 2 that extend the state of the device. Migrating from a previous version to this future version would have to be able to disable those commands along with the state that those commands may be influencing so that that state isn't written out. This may go well for a few commands (if they are going to be defined at all) but may not go well if there are multiple rounds of new commands to be defined.

[berrange]

The issue with the TPM 2 is that it's a crypto/security device with a complex implementation of multiple thousands of lines of code and more than 100 distinct commands of which some of them influence the state of the device. The testing of this device is crucial and I rely on existing test suites. Just supporting FIPS mode for example and disabling algorithms and key sized causes failures in those test suites and whole subtest don't work anymore.

Everything you say here applies to QEMU too. I'm not claiming backwards compat is easy, on the contrary it is hard. It is none the less necessary to support what users want to do with QEMU.

Applications cannot just loose functionality: Assuming a new crypto algorithm was added to v0.12 and we now want to migrate to a machine that only has libtpms v0.9 without the crypto algorithm, applications or kernel drivers would not be able to use the crypto algorithm anymore and en- or decrypt data. So just loosing functionality when downgrading is generally not an option.

This is precisely why we need the crypto algorithm agility at swtpm_setup time, rather than compile time. Anything that was exposed to the guest from the time swtpm_setup first runs, needs to remain exposed to the guest forever more. If migrating forwards, we must ensure we don't expose new things. If migrating backwards, we must reject the migration if the old version can't support what was currently in use. If the selection of algorithms is encoded in the state, then at swtpm startup time it can validate this and abort startup if finding some cipher that's unsupported. This should in turn cause the migration to abort.

A couple of concerns that arise out of this:

• What effects does this have on the code when future extensions need to be made accessible or invisible? I cannot predict how the TPM 2 code will evolve but it has algorithm agility and could theoretically be extended in many ways. futureTPM tries to add quantum-safe crypto to TPM 2 for example.

There needs to be a mechanism to track things which can affect the vTPM state and encode whether they're in use or not, so this can be validated at smtpm startup time,to ensure anything activated at smtpm_setup time was still available.

• How does one test this then?

Testing is the biggest burden wrt live migration supportability. The testing matrix grows significantly over time. QEMU upstream does NOT have comprehensive testing, and often downstream distros add value here in doing more thorough testing. Conceptually though, this is all about guest ABI, and so at a high level testing involves a means to dump info about the guest ABI features exposed by the TPM, and comparing against known previously recorded dumps to validate if anything unexpected changes.

So, the following options exist:

• Libtpms v0.9 is the last version. It's currently more advanced than any TPM 2.0 out there and implements the latest revision of the spec, revision 164. Eventually all systems will be on libtpms v0.9 level and downgrading won't be an issue.

Do these kind of things ever really stop evolving ? I'm sceptical, particularly if there's increasing interest in quantum-safe crypto.

• Libtpms gets further developed as before but it is the sole responsibility of the distros to package future versions of libtpms and ensure that their migration requirements are met. Libtpms will not commit to downgrading of state.

Distros can't magically solve this on their own, unless they never deploy newer libtpms which doesn't sound viable.

• We (additional maintainers required) commit to downgrade-ability of libtpms as outlined above (unless of course someone else comes up with a better definition).

[stefanberger]

Applications cannot just loose functionality: Assuming a new crypto algorithm was added to v0.12 and we now want to migrate to a machine that only has libtpms v0.9 without the crypto algorithm, applications or kernel drivers would not be able to use the crypto algorithm anymore and en- or decrypt data. So just loosing functionality when downgrading is generally not an option.

This is precisely why we need the crypto algorithm agility at swtpm_setup time, rather than compile time. Anything that was

It's NOT just the selection of crypto algoritms and key sizes but it's also selection of TPM 2 commands to enable and knowing which commands influence which part of the state of the TPM. At least this is what it would come down to if we wanted to have the flexibility of being able to further develop the device and adding new commands to it that create new device-internal state or extend existing one in some form. I cannot predict the future but this could become a hairy undertaking considering the complexity of this device.

At compile-time everything should be enabled so that users who want to use everything have the chance to do so.

A couple of concerns that arise out of this:

• What effects does this have on the code when future extensions need to be made accessible or invisible? I cannot predict how the TPM 2 code will evolve but it has algorithm agility and could theoretically be extended in many ways. futureTPM tries to add quantum-safe crypto to TPM 2 for example.

There needs to be a mechanism to track things which can affect the vTPM state and encode whether they're in use or not, so this can be validated at smtpm startup time,to ensure anything activated at smtpm_setup time was still available

Other than knowing at development time what commands influence what part of the TPM 2 state I wouldn't know what it 'tracking' would mean on this device.

• How does one test this then?

Testing is the biggest burden wrt live migration supportability. The testing matrix grows significantly over time. QEMU upstream does NOT have comprehensive testing, and often downstream distros add value here in doing more thorough testing. Conceptually though, this is all about guest ABI, and so at a high level testing involves a means to dump info about the guest ABI features exposed by the TPM, and comparing against known previously recorded dumps to validate if anything unexpected changes.

And my concern with the TPM 2 is the existing complexity and its testablity and what a commitment to this type of flexibility would mean to the future of the device and its testability. If we ever make a mistake with how the state is written out the whole forward/backward migration falls apart because older versions will not be able to read the fixed state blob then where something needed to be added that was forgotten - at least I wouldn't know how that would work.

[stefanberger]

FYI: FIPS mode on CentOS 9 (and Fedora 35) disables RSS-PSS signing scheme. Per this TCG spec [Table 2] RSS-PSS is mandatory. We can expect failing applications (and test suites will report failures) inside VMs if people wanted to run existing TPM 2 compliant applications with this type of algorithm restriction of libtpms's TPM 2 on the host. These are then expected failures that may confuse users but should not be unnecessarily reported because there's nothing we can do then.

[dagrh]

FYI: FIPS mode on CentOS 9 (and Fedora 35) disables RSS-PSS signing scheme. Per this TCG spec [Table 2] RSS-PSS is mandatory. We can expect failing applications (and test suites will report failures) inside VMs if people wanted to run existing TPM 2 compliant applications with this type of algorithm restriction of libtpms's TPM 2 on the host. These are then expected failures that may confuse users but should not be unnecessarily reported because there's nothing we can do then.

I think that's a separate question to the general backwards/forwards compat thing - I wouldn't expect compatibiltiy between a vTPM configured in FIPS mode and one configured in normality.

Getting back to normal migration compat problems; in Qemu we do keep bugcompatibility when operating in old 'machine types' to keep the migration stream compatible; I agree it's a pain to do (and test).

[stefanberger]

FYI: FIPS mode on CentOS 9 (and Fedora 35) disables RSS-PSS signing scheme. Per this TCG spec [Table 2] RSS-PSS is mandatory. We can expect failing applications (and test suites will report failures) inside VMs if people wanted to run existing TPM 2 compliant applications with this type of algorithm restriction of libtpms's TPM 2 on the host. These are then expected failures that may confuse users but should not be unnecessarily reported because there's nothing we can do then.

I think that's a separate question to the general backwards/forwards compat thing - I wouldn't expect compatibiltiy between a vTPM configured in FIPS mode and one configured in normality.

This is just FYI because it is in relation to the flexibility that we require of the TPM 2 now as part of the bigger picture and the side effects that this has, in particular here in FIPS mode.

[stefanberger]

After a long time I have been doing some work on profiles and the language/verbs for the profiles that libtpms understands and that give control over enabling & disabling algorithms and (ECC) curves. Any support for backwards compatibility and FIPS ( will have to be based on these profiles since there will only be one version of libtpms with all algorithms compiled in and a profile lets one disable the undesired algorithms...