signature: add OpenPGP signing mechanism based on Sequoia

[ueno]

This adds a new OpenPGP signing mechanism based Sequoia[1]. As Sequoia is written in Rust and doesn't provide a stable C FFI, this integration uses a minimal shared library as a "point solution".

To build, first follow the instruction at [2] and install libpodman_sequoia.so* into the library path, and then build with the following command from the top-level directory:

  $ make BUILDTAGS="btrfs_noversion libdm_no_deferred_remove containers_image_sequoia"

Note also that, for testing on Fedora, crypto-policies settings might need to be modified to explictly allow SHA-1 and 1024 bit RSA, as the testing keys in signature/fixtures are using those legacy algorithms.

1. https://sequoia-pgp.org/
2. https://github.com/ueno/podman-sequoia

[cgwalters]

(I'm not an "official" reviewer here, FWIW)

I skimmed your code and I have to say it looks generally great.

But very similarly to your PR in https://github.com/ostreedev/ostree/pull/3278 it'd be quite nice I think to list out why we're making this change. Especially questions like: Do we still need the openpgp backend too going forward?

https://github.com/ueno/podman-sequoia

I do wonder whether it makes sense for such a thing to live separately from this repository. Would it have a stable API/ABI? Of course adding it to this repository suddenly grows the scope of things here from "Go" to "Go, Rust and C" which is not at all a small thing.

But, I personally also do like the idea of opening up to thinking about how we use Rust code in this stack too. (To be clear, I am not speaking in any way for the other people who have done 99% of the work in this repository that are not me)

[mtrmac]

Hey @ueno , I’m very sorry I couldn’t get back to this discussion since … April?, due to a different RHEL priority (… possibly relevant here: sigstore signatures, using the Go standard-library crypto implementation).

The code in this PR is all good.

---

We do need to figure out packaging, and RH package ownership.

Do we still need the openpgp backend too going forward?

There’s a product discussion motivating the inclusion of this — and I assume product concerns would drive the continued existence of OpenPGP for a number of years. The default … should be kept up to date, but also needs to be somewhat practical to use.

Of course adding it to this repository suddenly grows the scope of things here from "Go" to "Go, Rust and C" which is not at all a small thing.

This repository is currently consumed by referencing it in go.mod in Podman & friends, and using go mod vendor; i.e. (apart from some configuration files), it mostly does not directly participate in builds.

*shrug* That means that adding an entirely new built artifact into this repo would probably not hurt any existing packaging.

But one other implication is that different versions of “Podman & friends” can be concurrently installed and include different versions of c/image, i.e. the C-API shared library (in a single system-wide location) will have to maintain a stable ABI to support all of those versions; directly including the Rust+C part in the c/image repo would not allow us to develop the server and client in immediate lockstep, so there’s no direct benefit of using one repo for all.

One thing I’m unsure about is assumptions of the Go tooling about repository contents — we have a go.mod at the top level of this Git repo, it’s conceivable that compilers/linters/… are going to assume that subdirectories primarily (exclusively?) contain Go code.

---

Podman etc. developers who don’t directly work on signing do need some reasonably convenient way to get an environment which doesn’t panic during process start. For GPGME, we tell them to install existing packages, on both Linux and macOS: https://github.com/containers/image/tree/main?tab=readme-ov-file#building . Something at about that level of hassle (or less) should exist for https://github.com/containers/image/tree/main?tab=readme-ov-file#building , or those developers are going to start sticking containers_image_openpgp in Makefiles.

[ueno]

[...] it'd be quite nice I think to list out why we're making this change. Especially questions like: Do we still need the openpgp backend too going forward?

In the discussion in April, a couple of motivations raised are:

• Sequoia integrates better in the OS, as it respects crypto-policies
• Troubleshooting GPGME related issues is inherently difficult (see https://github.com/containers/container-libs/issues/223), as the library internally spawns GnuPG components and communicate with them (which could fail in many ways)

As for the future of the openpgp backend, I agree with @mtrmac that it still has some value to maintain it as a legacy/standalone backend, even if sigstore signatures are preferred.

Podman etc. developers who don’t directly work on signing do need some reasonably convenient way to get an environment which doesn’t panic during process start. For GPGME, we tell them to install existing packages, on both Linux and macOS: https://github.com/containers/image/tree/main?tab=readme-ov-file#building . Something at about that level of hassle (or less) should exist for https://github.com/containers/image/tree/main?tab=readme-ov-file#building , or those developers are going to start sticking containers_image_openpgp in Makefiles.

Not sure if this is an ideal solution, but one option might be to merge mechanism_sequoia.go into mechanism_gpgme.go (rename it accordingly), and make it use the existing GPGME backend as a fallback if it fails to load libpodman_sequoia.so.0. The current Go binding uses dlwrap to enable this kind of run-time detection of Sequoia through sequoia.Init.

[mtrmac]

Not sure if this is an ideal solution, but one option might be to merge mechanism_sequoia.go into mechanism_gpgme.go (rename it accordingly), and make it use the existing GPGME backend as a fallback if it fails to load libpodman_sequoia.so.0.

That’s interesting … that would certainly eliminate concerns about developers not being able to work on other things. OTOH we’d then need some other mode to run in tests, and in production, to ensure we are using the right backend.

[ueno]

@mtrmac I've reworked this and incorporated all the required code into the same tree, instead of vendoring an external repository. The rust code is now hosted in signatures/sequoia/rust, which can be packaged something like rust-rpm-sequoia; we (RHEL Crypto team) are happy to own the downstream package.

As for the CI coverage, the base CI image needs to be updated to include the Rust toolchain and OpenSSL development files. Could you help with that?

[ueno]

To test it locally, I currently do the following on Fedora 41:

cd signature/sequoia/rust
PREFIX=/usr LIBDIR="\${prefix}/lib64" cargo build --release
cd -
sudo update-crypto-policies --set LEGACY
LD_LIBRARY_PATH=$PWD/signature/sequoia/rust/target/release make BUILDTAGS="btrfs_noversion containers_image_sequoia"

[mtrmac]

Cc: @jnovy, I’ll probably come asking questions.

[mtrmac]

To test it locally, I currently do the following on Fedora 41:

macOS:

(cd signature/internal/sequoia/rust; DYLD_FALLBACK_LIBRARY_PATH="$(xcode-select --print-path)/Toolchains/XcodeDefault.xctoolchain/usr/lib/" LIBDIR="$(pwd)" cargo build --release)

where DYLD_FALLBACK_LIBRARY_PATH is required to find a libclang.dylib

… and more modifications are required in the caller.

[mtrmac]

A high-level note: during a personal discussion at DevConf.CZ, it was agreed that the preferred design is to use Sequoia for all signature verifications; but for signing, to expose signing using GnuPG-maintained secret keys, and Sequoia-maintained secret keys, as two separate features in the UI.

That way, existing GnuPG keys, and existing gpg-agent-reliant workflows will continue to work, while the Sequoia implementation, including its new algorithm support, can be used by those who generate new keys or can migrate; and the c/image implementation will not need to worry about transparently/automatically migrating users’ secret keys.

[ueno]

@nwalfield pointed out that, sq can transparently use gpg-agent, so the migration of user's secret keys shouldn't be needed as long as gpg-agent is installed. The feature is implemented in the sequoia-keystore crate, which this PR already uses, but currently it excludes gpg-agent support. Maybe we could enable it and remove all the manual migration logic from this PR, which might be simpler than exposing two separate UI for signing. @mtrmac What do you think?

[mtrmac]

@ueno I think that might well make sense to do, but not as the first step.

I don’t think full transparency is achievable: if we want to support PQC algorithms using the Sequoia implementations, users should be manually generating keys using Sequoia directly, without GnuPG. So, users will need to use sq and deal with Sequoia key storage mechanisms directly; asking them to use GNUPGHOME or some other GnuPG-primary mechanism to point “not-actually-GnuPG” at the Sequoia directory would only be confusing.

So, I think the best UI will be to add new CLI features (skopeo --sign-by-sq or something like that), with the Sequoia key store configuration a new option, not interacting with GNUPGHOME at all. By using a new CLI option, users explicitly opt-in into a behavior change, so we don’t need to try to interact with gpg-agent in that flow.

In the first version of this, it would be less work to keep the existing gpgme backend (for signing, not verification — hopefully all verification can be done using Sequoia) for the currently-existing CLI flows: that will guarantee full compatibility.

And afterwards, after this time-sensitive work is time-sensitive is done, we can consider replacing the gpgme calls with the Sequoia gpg-agent. Certainly removing the dependency on GnuPG would be nice, but if it is transparent, it’s probably not as urgent. And I think it is somewhat blocked on understanding of things like https://gitlab.com/sequoia-pgp/sequoia/-/issues/1194 — it’s quite possible that this would not be an issue if the signing happened in gpg-agent, but let’s see what that investigation shows.

---

@ueno I can work on the Go / c/image-side work to introduce the new Sequoia-only signing features, and handle the other small comments from the earlier review. (I’d be also happy to work on improvements to the Rust side, with the disclaimer that I’m very new to Rust and completely unfamiliar with the Sequoia codebase and API.)

The primary thing I’d need help with is for you and @jnovy to come to an agreement on packaging / ownership. Does the offer that the RHEL Crypto team could own and maintain the Rust-side package still stand?

In that case, it would probably make most sense for the Rust code to live in its own repository (with c/image containing just a static copy of the generated headers and CGo stubs); and I think it would be simpler for Podman/Buildah/Skopeo to link directly to a shared library, without dlopen, so that the usual linker dependencies and tools like nm can help with symbol lookups.

[ueno]

@ueno I can work on the Go / c/image-side work to introduce the new Sequoia-only signing features, and handle the other small comments from the earlier review. (I’d be also happy to work on improvements to the Rust side, with the disclaimer that I’m very new to Rust and completely unfamiliar with the Sequoia codebase and API.)

That would be great! I just pushed some fixes (not all) to the issues pointed in the review, FWIW.

The primary thing I’d need help with is for you and @jnovy to come to an agreement on packaging / ownership. Does the offer that the RHEL Crypto team could own and maintain the Rust-side package still stand?

Yes, it does.

In that case, it would probably make most sense for the Rust code to live in its own repository (with c/image containing just a static copy of the generated headers and CGo stubs); and I think it would be simpler for Podman/Buildah/Skopeo to link directly to a shared library, without dlopen, so that the usual linker dependencies and tools like nm can help with symbol lookups.

That would be simpler, though I actually chose this dlopen design in case you want to keep both GnuPG and Sequoia interfaces (and do not always want to link to Sequoia, as the binary size is not small). If it is not a problem, we can certainly make it as a shared library.

[mtrmac]

I actually chose this dlopen design in case you want to keep both GnuPG and Sequoia interfaces (and do not always want to link to Sequoia, as the binary size is not small). If it is not a problem, …

I don’t know. @jnovy ?

[mtrmac]

Note also that, for testing on Fedora, crypto-policies settings might need to be modified to explictly allow SHA-1 and 1024 bit RSA, as the testing keys in signature/fixtures are using those legacy algorithms.

That’s not a blocker but fairly inconvenient. I tried to avoid that by regenerating keys in https://github.com/containers/image/pull/2875 ; that works with gpgme, but fails for v3 signature packets, I’m still looking into why.

[mtrmac]

I tried to avoid that by regenerating keys in containers/image#2875 ; that works with gpgme, but fails for v3 signature packets, I’m still looking into why.

My fault, the signatures use SHA1.

[ueno]

I tried to avoid that by regenerating keys in containers/image#2875 ; that works with gpgme, but fails for v3 signature packets, I’m still looking into why.

My fault, the signatures use SHA1.

If the primary goal is to work around the CI issues with legacy fixture data, it might be simpler to override it with the SEQUOIA_CRYPTO_POLICY envvar in GitHub workflows, rather than hard-coding it.

[mtrmac]

My fault, the signatures use SHA1.

I have updated https://github.com/containers/image/pull/2875 , re-generating signatures to use SHA256; now the Sequoia code is passing tests on macOS with the default (= built-in) policy and no modifications.