failed to verify signature for keyfile [email protected]:

[marcofranssen]

I'm using apko to build the base image for some of our dagger modules. Since yesterday we fail to update our base images with following error:

2025/11/19 09:55:44 WARN failed to verify signature for keyfile [email protected]: verify PKCS1v15 signature: crypto/rsa: verification error
index https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: parsing https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: signature verification failed for repository index, for all provided keys

See here our apko configuration file:

contents:
 repositories:
 - https://dl-cdn.alpinelinux.org/alpine/v3.22/main
 - https://dl-cdn.alpinelinux.org/alpine/v3.22/community
 packages:
 - alpine-baselayout
 - ca-certificates-bundle
 - alpine-keys
 - aws-cli
 - opentofu
archs: [x86_64, aarch64]
environment:
 PATH: /usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin
entrypoint:
 command: tofu
cmd: version

To isolate the problem and make it reproduceable without involving Dagger I did following test.

$ docker run --rm -it cgr.dev/chainguard/wolfi-base

Then inside the container do following:

$ echo 'contents:
 repositories:
 - https://dl-cdn.alpinelinux.org/alpine/v3.22/main
 - https://dl-cdn.alpinelinux.org/alpine/v3.22/community
 packages:
 - alpine-baselayout
 - ca-certificates-bundle
 - alpine-keys
 - aws-cli
 - opentofu
archs: [x86_64, aarch64]
environment:
 PATH: /usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin
entrypoint:
 command: tofu
cmd: version
' > apko.yaml
$ apk add --no-cache apko
$ apko build apko.yaml ttl.sh/stuff/test:latest .
2025/11/19 09:55:44 WARN failed to verify signature for keyfile [email protected]: verify PKCS1v15 signature: crypto/rsa: verification error
2025/11/19 09:55:44 WARN failed to verify signature for keyfile [email protected]: verify PKCS1v15 signature: crypto/rsa: verification error
Error: locking config: resolving apk packages: for arch "arm64": error getting repository indexes: reading index https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: parsing https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: signature verification failed for repository index, for all provided keys
for arch "amd64": getting indexes for "arm64" sibling: reading index https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: parsing https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: signature verification failed for repository index, for all provided keys
2025/11/19 09:55:44 INFO error during command execution: locking config: resolving apk packages: for arch "arm64": error getting repository indexes: reading index https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: parsing https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: signature verification failed for repository index, for all provided keys
for arch "amd64": getting indexes for "arm64" sibling: reading index https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: parsing https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz: signature verification failed for repository index, for all provided keys

It happens for aarch64 architectures. When removing aarch64 from the apko.yaml configuration it succeeds.

Potential thing that could have caused breaking it:

• https://github.com/chainguard-dev/apko/commit/7918e99fb3750d57cf4cdb3a67ca9a9bb76e4cfd

[cpanato]

cc @xnox

[xnox]

@javacruft fun, but i am also confused about "amd64" and "arm64" cause these are not the right arch names.

[marcofranssen]

Indeed I also noticed it, it looks like some things are getting mixed up.

[BobyMCbobs]

I'm having the same errors!

[javacruft]

The introduction of the caching client for offline build usage uses etag for indexing - however:

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub" | grep etag
etag: "639a4604-320"

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-6165ee59.rsa.pub" | grep etag
etag: "639a4604-320"

which means we end up with a cache conflct as they resolve to the same etag key.

[marcofranssen]

The introduction of the caching client for offline build usage uses etag for indexing - however:

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub" | grep etag
etag: "639a4604-320"

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-6165ee59.rsa.pub" | grep etag
etag: "639a4604-320"

which means we end up with a cache conflct as they resolve to the same etag key.

Good find, now a fix. Let me know if help required, I might be able to have a look tomorrow.

[xnox]

https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10862

So alpine says not to rely on etags, as they use nginx with time + size (basically etag is based on stat).

[xnox]

The introduction of the caching client for offline build usage uses etag for indexing - however:

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub" | grep etag
etag: "639a4604-320"

$ curl -sI "https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-6165ee59.rsa.pub" | grep etag
etag: "639a4604-320"

which means we end up with a cache conflct as they resolve to the same etag key.

Good find, now a fix. Let me know if help required, I might be able to have a look tomorrow.

We are currently travelling / flying, but our UK engineers will be able to fix this early next week.

Until then, please use previous tag, it is not worth withdrawing this version right now.

[javacruft]

We are currently travelling / flying, but our UK engineers will be able to fix this early next week.

Until then, please use previous tag, it is not worth withdrawing this version right now.

I found some time while travelling - take a look at the linked PR