Running an unencrypted container with `--pem-path` should fail

[Flamefire]

Version of Singularity

singularity-ce version 3.11.3-jammy

Describe the bug

When building a container --pem-path implies --encrypt. However running a container with --pem-path doesn't imply the key is actually used.

To Reproduce

$ sudo singularity build ubuntu.sif docker://ubuntu:22.04
$ singularity exec --pem-path=rsa_pri.pem ubuntu.sif bash -c "echo Hello" 

Expected behavior

Not using the key file due to the container not being encrypted should fail so that one can be sure the container is "secure".

Not sure if this is a bug or rather a feature request. But IMO it is reasonable to be able to use the container encryption also for verifying the authenticity of a container, i.e. prevent a container in a shared filesystem from being modified.

[dtrudg]

Hello. If I recall correctly, this behaviour was intentional. Singularity flags such as pem-path are also linked to environment variables, and there are valid workflows where someone would want to avoid specifying the pem-path each time... but also want to run a mix of encrypted and unencrypted containers.

I'd suggest perhaps raising the issue in GitHub discussions / Slack / the Google Group to see if there is more general support for a behaviour change.

But IMO it is reasonable to be able to use the container encryption also for verifying the authenticity of a container, i.e. prevent a container in a shared filesystem from being modified.

Generally we'd advise using sign and verify for this. Encryption applies only to the root filesystem, while signatures are over the whole SIF... so signing and verifying can protect against other changes.

[Flamefire]

Singularity flags such as pem-path are also linked to environment variables, and there are valid workflows where someone would want to avoid specifying the pem-path each time... but also want to run a mix of encrypted and unencrypted containers.

For building you'd need to specify --encrypt when using the env variable, so something similar might make sense, e.g. a --decrypt enabled automatically on --pem-path

[dtrudg]

Singularity flags such as pem-path are also linked to environment variables, and there are valid workflows where someone would want to avoid specifying the pem-path each time... but also want to run a mix of encrypted and unencrypted containers.

For building you'd need to specify --encrypt when using the env variable, so something similar might make sense, e.g. a --decrypt enabled automatically on --pem-path

Could you perhaps write out, long form, how you envisage this working? E.g. if I do, or do not, set the SINGULARITY_ENCRYPTION_PEM_PATH environment variable... what flags are necessary / implied?

If there's a concrete proposal then we can circulate it for discussion.

I think the idea that encryption is useful for verification of a container isn't true, so shouldn't be considered relevant here, but there is a valid debate between...

• Encryption material is provided at runtime, to be used if & when it is required. -and-
• Encryption material is provided at runtime, indicating that we must be running an encrypted container using that material.

[Flamefire]

Could you perhaps write out, long form, how you envisage this working? E.g. if I do, or do not, set the SINGULARITY_ENCRYPTION_PEM_PATH environment variable... what flags are necessary / implied?

• Introduce a new flag --decrypt for the run/exec/shell etc commands
• --pem-path on those commands implies --decrypt on those
• if --decrypt is set (explicitly or implicitly) then Singularity will fail if the container is not encrypted
• If the container is encrypted then Singularity will use --pem-path or SINGULARITY_ENCRYPTION_PEM_PATH in this order whichever is not empty

So for the 2 use cases:

1. Encryption material is provided at runtime, to be used if & when it is required. Provide that via SINGULARITY_ENCRYPTION_PEM_PATH which is then only used when the container is encrypted
2. Encryption material is provided at runtime, indicating that we must be running an encrypted container using that material.
   Provide it either via --pem-path or SINGULARITY_ENCRYPTION_PEM_PATH together with --decrypt

So in summary: I propose the same handling for running a container as for building a container by using a flag to enforce en/decryption which is set implicitly by one of the protection flags (i.e. --passphrase or similar should also imply --decrypt). For backwards compatibility the env variable is ignored unless --decrypt is given or the container is encrypted

[dtrudg]

@Flamefire thanks. I'll point this issue out in other spaces closer to our next community call.

@tri-adam / @preminger - any thoughts on this? (at your convenience)

[tri-adam]

So for the 2 use cases:

1. Encryption material is provided at runtime, to be used if & when it is required. Provide that via SINGULARITY_ENCRYPTION_PEM_PATH which is then only used when the container is encrypted
2. Encryption material is provided at runtime, indicating that we must be running an encrypted container using that material. Provide it either via --pem-path or SINGULARITY_ENCRYPTION_PEM_PATH together with --decrypt

So in summary: I propose the same handling for running a container as for building a container by using a flag to enforce en/decryption which is set implicitly by one of the protection flags (i.e. --passphrase or similar should also imply --decrypt). For backwards compatibility the env variable is ignored unless --decrypt is given or the container is encrypted

It seems to me that these two use cases are both valuable; they address two different needs/expectations that users may have. --pem-path (and the SINGULARITY_ENCRYPTION_PEM_PATH env var) currently address the first case only. @Flamefire I believe your proposal would make it possible to address the second case, using the method that key material was provided to determine the expected behaviour. This would be a behaviour change for the --pem-path case, and it does seem relatively subtle. I worry that it won't be obvious that supplying key material one way or the other gives different behaviour for unencrypted containers.

We could also make this explicit, through a separate option (perhaps --must-decrypt?) That would make it possible to utilize key material provided in either fashion, while specifying the desired behaviour, without requiring a behaviour change.

With an unencrypted ubuntu.sif, then, the first use case is as it behaves today:

$ singularity exec --pem-path=rsa_pri.pem ubuntu.sif // OK
$ SINGULARITY_ENCRYPTION_PEM_PATH=rsa_pri.pem singularity exec ubuntu.sif // OK

And the second use case:

$ singularity exec --must-decrypt --pem-path=rsa_pri.pem ubuntu.sif // error
$ SINGULARITY_ENCRYPTION_PEM_PATH=rsa_pri.pem singularity exec --must-decrypt ubuntu.sif // error

[Flamefire]

Thanks for you analysis

This would be a behaviour change for the --pem-path case, and it does seem relatively subtle. I worry that it won't be obvious that supplying key material one way or the other gives different behaviour for unencrypted containers.

My intention was to make it less subtile/confusing by matching the behavior during build

An -e|--encrypt flag to singularity build is used to indicate that the container needs to be encrypted.

And:

The -e|--encrypt flag is implicitly set when the --passphrase or --pem-path flags are passed with the build command. If multiple encryption related flags and/or environment variables are set, the following precedence is respected.

--pem-path --passphrase SINGULARITY_ENCRYPTION_PEM_PATH SINGULARITY_ENCRYPTION_PASSPHRASE

So as far as I understand setting SINGULARITY_ENCRYPTION_PEM_PATH is different to --pem-path in that it doesn't imply --encrypt and hence is by default ignored. So I wanted to match that with the reasoning that an explicitely passed cmdline option shouldn't be ignored while the environment variable(s) may be set globally somewhere else.

Hence:

$ singularity exec --pem-path=rsa_pri.pem ubuntu.sif // ~OK~ error $ SINGULARITY_ENCRYPTION_PEM_PATH=rsa_pri.pem singularity exec ubuntu.sif // OK

$ singularity exec --must-decrypt --pem-path=rsa_pri.pem ubuntu.sif // error $ SINGULARITY_ENCRYPTION_PEM_PATH=rsa_pri.pem singularity exec --must-decrypt ubuntu.sif // error

Anyway if that still seems to be to confusing any force-decrypt flag would help. IMO --decrypt is specific enough already: Requesting decryption which is not happening is reasonably an error

@dtrudg For my understanding re

Generally we'd advise using sign and verify for this. Encryption applies only to the root filesystem, while signatures are over the whole SIF... so signing and verifying can protect against other changes.

1. There currently isn't any mechanism in Singularity itself for signing containers, so this means e.g. gpg would need to be used, correct?
2. Which other changes are possible here that can be potentially problematic? I imagine altering the runscript could be an issue if that is in the metadata but not anything that could interfere with singularity exec-usecases.

[preminger]

With regard to —

1. There currently isn't any mechanism in Singularity itself for signing containers, so this means e.g. gpg would need to be used, correct?

I might have misunderstood your question, here, but isn't this the mechanism in question...?

[Flamefire]

I might have misunderstood your question, here, but isn't this the mechanism in question...?

Indeed this is what I was looking for. Especially the ecl.toml looks useful in allowing only containers signed with specific keys.

As for a current --require-decrypt I found allow container in the singularity.conf where I'd need something like allow container * no & allow container encrypted yes but for the lack of a match-all I'd need a "no" entry for each type except "encrypted".
Downside is of course that this disallows running any unencrypted container but that may work in our use case.