SEV-SNP Machines on AWS potentially hand out broken attestation reports

[msanft]

Issue Description

We recently observed virtual machines on AWS with AMD SEV-SNP enabled to not reliably contain a (functioning) SEV-SNP device. Machines where this is the case will not be able to join or bootstrap a Constellation cluster, as they are not able to hand out a valid attestation report. Therefore, the issue is not impacting Constellation's security guarantees.

The issue may show different symptoms, depending on which part of a Constellation cluster the broken VM is.

• When a machine trying to bootstrap the Constellation cluster is broken, the CLI will show an error stating that an invalid attestation report has been supplied when trying to apply the initial Constellation cluster configuration on it.
• When a machine trying to join a Constellation cluster, be that within a cluster in its bootstrapping process or a cluster being upgraded, the machine will be rejected by Constellation's join-service, as it is not able to supply a valid attestation report. When bootstrapping a cluster, this will lead to the node simply not being able to join the cluster. On an upgrade, where Kubernetes operators manage the VM lifecycle, this rejection will lead to nodes being re-provisioned until a VM with a working device is received.

The issue has already been reported to the AWS team and they are working on fixing it.

Possible Workarounds

The issue is not present on all machines, so it is still possible to create a functioning Constellation cluster in most cases. If you should run into the issue on a machine, the following workarounds can help.

• Try to provision another VM. It is recommended to provision VMs on the same region until you get a working one, and then terminate all non-working VMs to not receive the same machine again when re-provisioning. The same can be achieved by provisioning a VM in another region, but as AWS does not provide SEV-SNP machines on all regions, you might run into availability issues, depending on which region is used. To do so, you can navigate to the constellation-terraform directory in your Constellation workspace (or the directory containing the infrastructure configuration, if not using the Constellation CLI) to destroy and re-apply the instance group, which contains the VMs, and apply the Constellation configuration again.

  cd constellation-terraform
  terraform destroy
  terraform apply
  constellation apply
  

• If the deployment is non-production, you can also use AWS NitroTPM attestation instead of SEV-SNP. To do so, remove the attestation.awsSEVSNP block from constellation-conf.yaml and insert the following the following block instead:

  awsNitroTPM:
    measurements: {}
  

  After that, destroy the cluster, fetch the measurements for machines with NitroTPM attestation, and recreate the cluster.

  constellation terminate
  constellation config fetch-measurements
  constellation create
  constellation apply