nvidia-docker
nvidia-docker copied to clipboard
NVIDIA
Public GPG key error
Where's the public GPG key? Was working fine for weeks. Today this error.
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is not signed.
https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772
@klueska I'm starting from the tensorflow/tensorflow:2.7.0-gpu image, I tried following this blog and so I added the following lines at the start of my file (after FROM tensorflow...)
RUN apt-get install -y --no-install-recommends wget RUN apt-key del 7fa2af80 RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb RUN apt-get update
When running apt-get update, I have the following error :
#11 [ 8/13] RUN apt-get update #11 sha256:c1afeb0121b560d8f8cf7247423fec777c038dd0f561407a5757db6c7e71ce82 #11 0.503 E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != #11 0.503 E: The list of sources could not be read. #11 ERROR: executor failed running [/bin/bash -c apt-get update]: exit code: 100
In order to add as much information as possible, here is the content of the /etc/apt/sources.list.d folder :
- cuda.list
- nvidia-ml.list
I tried deleting both these files before running the previous lines, then I get the following error :
#12 7.636 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443] #12 7.636 Hashes of expected file: #12 7.636 - Filesize:481481 [weak] #12 7.636 - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b #12 7.636 - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak] #12 7.636 - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak] #12 7.636 Release file created at: Mon, 25 Apr 2022 23:27:19 +0000 #12 7.636 E: Some index files failed to download. They have been ignored, or old ones used instead.
I have the exact same issue. Added this to my Dockerfile:
WORKDIR /tmp RUN apt-key del 7fa2af80 && \ apt install wget && \ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \ dpkg -i cuda-keyring_1.0-1_all.deb
And then, when running apt update and apt install:
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != E: The list of sources could not be read.
https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772
I rotated my keys, but still getting errors.
Commands in Dockerfile:
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb
RUN apt update && apt install -y cuda-compat-11-4
Error:
#8 7.228 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.195.19.142 443]
#8 7.228 Hashes of expected file:
#8 7.228 - Filesize:481481 [weak]
#8 7.228 - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
#8 7.228 - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
#8 7.228 - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
#8 7.228 Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
It's pretty disappointing that hardly a day's notice is considered enough warning before breaking everyone's ability to install.
Argggg, borked build club member signin
this seems to work for me RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-get update && apt-get install -y --no-install-recommends wget RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb
it seems the keys changed today.
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda
This is what I used in the Dockerfile to fix this
&& apt-key del 7fa2af80 \
&& curl -L -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
&& dpkg -i cuda-keyring_1.0-1_all.deb \
This worked for me for a dockerfile, I used the alternative approach on the official notice but I had to remove old source files and I also needed to fetch 2 keys, one for "cuda" and one for "machine learning" Put these after the FROM command
RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub
Make sure you change ubuntu1804/x86_64 to your distro/arch. Also, note the first two lines are necessary as the base image might have bad keys (you might get an error The list of sources could not be read without them)
RUN rm /etc/apt/sources.list.d/cuda.list
I can confirm. This is also the line that I needed to add to get it to work. Only the alternative method from the official announcement seems to work in a Docker build.
This is what I added at the start of my Dockerfile: (after FROM)
RUN rm /etc/apt/sources.list.d/cuda.list && \ apt-key del 7fa2af80 && \ apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
I still think it's strange for NVIDIA to break everyone's build pipelines with only a few hours notice.
Same here, ubuntu2004 repo is broken because the previous key 7fa2af80.pub stopped working.
The right one can be found here ubuntu1804/x86_64/3bf863cc.pub 😐
This worked for me, I used the alternative approach on the official notice but I had to fetch another 2 keys, one for "cuda" and one for "machine learning"
RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub
This worked for me, but remember to change the arguments of the command depending on the version of your distribution. Thanks!
This may solved for me (base image: nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04)
RUN \
# Update nvidia GPG key
rm /etc/apt/sources.list.d/cuda.list && \
rm /etc/apt/sources.list.d/nvidia-ml.list && \
apt-key del 7fa2af80 && \
apt-get update && apt-get install -y --no-install-recommends wget && \
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
dpkg -i cuda-keyring_1.0-1_all.deb && \
apt-get update
This one worked for me here
RUN rm /etc/apt/sources.list.d/cuda.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
Argggg, borked build club member signin
this seems to work for me RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-get update && apt-get install -y --no-install-recommends wget RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb
This solved it for me. Thanks!
Argggg, borked build club member signin
this seems to work for me RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-get update && apt-get install -y --no-install-recommends wget RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb
It works for me. Thank you!
The files /etc/apt/sources.list.d/cuda.list and /etc/apt/sources.list.d/nvidia-ml.list should be removed before installing cuda-keyring, otherwise it raises the error The list of sources could not be read.
"If you are unable to install the cuda-keyring package, you can optionally install the new signing key manually (although this [apt-key] is not the recommended method)." [source]
In a nutshell: NVIDIA wants you to download their proprietary key(ring) management tool in favor of existing Ubuntu built-in solution (apt-key), because the latter has become deprecated. But why not contribute such enhancements to the official distro to make the open-source ecosystem stronger?
I am having the same issue. None of the solutions above worked for me.... :-(
It looks like the cuda images for ubuntu 20.04 has been updated 1 day after 18.04. So now ubuntu 20.04 + cuda >= 11.4 images are fine. Same for cuda <= 10.2.
However cuda 11.0, 11.1, and 11.2 on ubuntu are still outdated (only on ubuntu - centos, ubi etc have been updated reently so they are fune) will these be updated any time soon?
thanks!
The solutions all of you provided worked for me for about a day or two and now it gives me the same problem :( Anyone can offer a salvation? 🤣😭
"If you are unable to install the cuda-keyring package, you can optionally install the new signing key manually (although this [apt-key] is not the recommended method)." [source]
In a nutshell: NVIDIA wants you to download their proprietary key(ring) management tool in favor of existing Ubuntu built-in solution (
apt-key), because the latter has become deprecated. But why not contribute such enhancements to the official distro to make the open-source ecosystem stronger?
The question is, why is this Nvidia CUDA key ring not included into the base image?
For those older CUDA versions the cuda-keyring installation method seems to work:
sudo dpkg -i cuda-keyring_1.0-1_all.deb
Yesterday I managed to revive all of our CUDA 11.2-based images this way. Here's the relevant part of the Dockerfile for mirekphd/cuda-11.2-devel-ubuntu20.04 base image:
ENV UBUNTU_VER_STR 2004
ENV CPU_ARCH x86_64
# download and install NVIDIA's public key
RUN apt-get update && apt-get install -y --no-install-recommends \
wget ca-certificates gnupg2 && \
# caution: we have to switch to NVIDIA's new key management tool
# instead of the deprecated apt-key
# (see https://github.com/NVIDIA/nvidia-docker/issues/1631#issuecomment-1114208147)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu$UBUNTU_VER_STR/$CPU_ARCH/cuda-keyring_1.0-1_all.deb && \
dpkg -i cuda-keyring_1.0-1_all.deb && \
apt-get purge --autoremove -y wget && \
rm -rf /var/lib/apt/lists/*
It looks like the cuda images for ubuntu 20.04 has been updated 1 day after 18.04. So now ubuntu 20.04 + cuda >= 11.4 images are fine. Same for cuda <= 10.2.
However cuda 11.0, 11.1, and 11.2 on ubuntu are still outdated (only on ubuntu - centos, ubi etc have been updated reently so they are fune) will these be updated any time soon?
thanks!
FWIW, here was our dockerfile. The primary way listed in the NVIDIA notice wasn't straight forward because wget wasn't installed, so we went with the alternative method.
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
# Update NVIDIA's apt-key
# Announcement: https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772
ENV DISTRO ubuntu1804
ENV CPU_ARCH x86_64
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$DISTRO/$CPU_ARCH/3bf863cc.pub
RUN apt-get update && ...
Getting:
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.
Using:
FROM nvidia/cuda:10.0-devel-ubuntu18.04
ENV DISTRO ubuntu1804
ENV CPU_ARCH x86_64
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$DISTRO/$CPU_ARCH/3bf863cc.pub
RUN apt-get update
any clue? :(
When running:
uname -a
I'm getting:
Linux *******IP*** 5.4.0-1068-aws #72~18.04.1-Ubuntu SMP Thu Mar 3 08:49:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
I tried:
` RUN apt-get install wget -y
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/$DISTRO/$CPU_ARCH/cuda-keyring_1.0-1_all.deb &&
dpkg -i cuda-keyring_1.0-1_all.deb
`
also tried:
RUN apt-get update && apt-get install -y --no-install-recommends \
wget ca-certificates gnupg2
that didn't change anything
Looks like NVIDIA did not test their new keyring method for CUDA versions below 10.2...
See which versions got the "Centos7/UBI7/UBI8 key update" commits in their nvidia/container-images/cuda Gitlab repo... try upgrading CUDA to at least 10.2 or wait for the update for older CUDAs which was promised for "next week" 3 days ago here.
FROM nvidia/cuda:10.0-devel-ubuntu18.04
ENV DISTRO ubuntu1804 ENV CPU_ARCH x86_64
