GPG error "public key is not available" in Ubuntu 20.04 CUDA 11.4.0 image while building

[wilkesreid]

1. Issue or feature description

The following Dockerfile does not build today (April 28, 2022), even though it built successfully yesterday:

FROM    nvidia/cuda:11.4.0-runtime-ubuntu20.04
RUN     apt-get update

The error is the following:

GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.

2. Steps to reproduce the issue

Create the above dockerfile and attempt to docker build it.

3. Information to attach (optional if deemed irrelevant)

I am running Docker version 20.10.14, build a224086 on WSL 2 Ubuntu 20.04 on Windows 10 Pro, Version 21H2, OS Build 19044.1645

nvidia-smi on my host machine:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.65       Driver Version: 471.96       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0A:00.0  On |                  N/A |
|  0%   55C    P0    61W / 250W |   1515MiB /  8192MiB |    ERR!      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[wilkesreid]

Possible duplicate of https://github.com/NVIDIA/nvidia-docker/issues/1631

[klueska]

https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

[wilkesreid]

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=

[memray]

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=

Tried following the methods in announcement and didn't work either. Well done Nvidia.

[memray]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

[jtran1999]

I noticed different keys available to day for the repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

[zestrells]

This is what I used in the dockerfile to fix this issue.

    && apt-key del 7fa2af80 \
    && curl -L -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
    && dpkg -i cuda-keyring_1.0-1_all.deb \

[mirekphd]

None of these hacks above are sufficiently reliable yet, as NVIDIA is still working on the changes. Some latest CUDA and Ubuntu versions are already working (images such as CUDA 11.6 for Ubuntu 20.04 can be rebuild from their code at Gitlab), but others (older CUDA/Ubuntu versions such as CUDA 11.2) may still fail.

So given that CUDA 11.1 .. 11.6 toolkits are compatible with the same drivers version (>=450.80.02) it should be possible to adapt NVIDIA's Dockerfile with the latest CUDA version (11.6) and your OS of choice: https://gitlab.com/nvidia/container-images/cuda/-/tree/master/dist/11.6.2

More info: https://gitlab.com/nvidia/container-images/cuda/-/issues/158

[372046933]

nvidia/cuda:11.2.1-base-ubuntu20.04 is updated this afternoon. I pull the updated image and the pub key problem disappeared

[kinue00]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

seems working for me (kaldi's official dockerfile ubuntu1804)

[VictorZuanazzi]

We are having the same problem with the image nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 and 11.3. Any overview of which images are fixed?

[zjuPeco]

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

[Co0perator]

This may add nothing productive to the conversation but I thought it worth mentioning Most people seem to be getting this error in Ubuntu 20.04 Docker containers, I'm getting it on Ubuntu 20.04 Desktop.

[AmanG96]

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

This worked for me as well.

[eijynagai]

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

This didn't work for me. Using Ubuntu server 20.04 and docker image.

I got this error W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is not signed.

[hammer-wang]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks for the magic!

[zabir-nabil]

RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list

worked like a charm

[ramosmarco]

I noticed different keys available to day for the repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

You're right, the nvidia instrucction (https://developer.nvidia.com/cuda-11.2.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=2004&target_type=debnetwork) set to add this key sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub however that doesn't match with the repository key, but using the one you mentioned (sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub), that it match, and allow to update apt-get.

Thanks

[riponcse10]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks! Works like a charm!

[LordNex]

having the same issue on a 4gb Jetson Nano

[zanedurante]

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

[aycaecemgul]

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

This is the only solution that worked for ubuntu 20.04 desktop.