gpu-operator

gpu-operator copied to clipboard

nvidia-device-plugin-validator fails after node reboot (with MIG enabled)

16

# System Running on bare-metal - Ubuntu 20.04.4 - Kubernetes v1.24.3 - Containerd 1.6.7 - GPU-Operator v1.11.1 # Setup GPU-Operator is installed with: ``` helm install --wait --debug --generate-name --create-namespace...

dasantonym

nvidia-driver-daemonset, nvidia-container-toolkit-daemonset and nvidia-device-plugin-daemonset not added.

4

_The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense._...

nonpolarity

DCGM-Expoter msg="Could not retrieve ConfigMap ..."

3

I am running a cluster with a number of nvidia gpu. I'm also monitoring gpu using dcgm-exporter. However, sometimes the dcgm-exporter fails to give metrics with the logs below. ```...

devnjw

### 1. Issue or feature description I would love to have ability to specify my own set of labels on pods. My organization uses them for pod ownership (resource cost...

boniek83

Nvidia operator doesn't schedule required nvidia driver pods on the new GPU machines

1

_The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense._...

arpitsharma-vw

The GPU Operator driver build fails on GCP when using Ubuntu 22.04.

21

### 1. Quick Debug Information * OS/Version: Ubuntu 22.04 * Kernel Version: ``` Linux version 5.19.0-1030-gcp (buildd@bos03-amd64-050) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.1.0-2ubuntu1~22.04) 12.1.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #32~22.04.1-Ubuntu SMP...

uniit

Which driver image version is best suitable for gpu nvidia rtx A4000 and A2000 in redhat8.6

2

In driver image container, the latest driver version is 525.85.12-rhel8.6. After I did some reseach and found only 'nvcr.io/nvidia/driver:515.86.01-rhel8.6' can both support these two card, nvidia rtx A4000 and A2000....

carlwang87

Device plugin does not start on MIG-enabled host due to insufficient permissions

2

### 1. Quick Debug Information * OS/Version: Ubuntu20.04: * Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd * K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): EKS 1.23 ### 2. Issue or...

yunfeng-scale

Use gpu-operator v23.9.2 version, when pulling nvcr.io/nvidia/driver:550.54.14 and nvcr.io/nvidia/cloud-native/nvidia-fs:2.17.5, the error "manifest unknown" always seems to be reported "

2

Hi, I want to install gpu-operator(v23.9.2 version) in my kubernetes cluster, but when pulling nvcr.io/nvidia/driver:550.54.14 and nvcr.io/nvidia/cloud-native/nvidia-fs:2.17.5, the error "manifest unknown" always seems to be reported ", What did I...

sunwuyan

Fatal error: concurrent map read and map write - CrashLoopBackOff

1

### 1. Quick Debug Information * OS/Version - Ubuntu22.04 * Kernel Version: 5.15.0-1045-gke * Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd * K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): GKE...

ujjwal