GPU isolation not working after setting default runtime to nvidia

[carlosleocadio]

Issue description

I'm running a bare metal k8s cluster (kube_version: v1.14.1), with nvidia-gpu-device-plugin pods on each one of my GPU servers to enable GPU resources allocation. I'm deploying the k8s cluster using Kubespray, and I have two GPU enabled servers - (let's call them servers A and B).

On server A I've installed nvidia-docker2 and edited the file /etc/docker/daemon.json setting the default-runtime to nvidia. But now, when I schedule a pod on Server A with a request for nvidia.com/gpu resources, all the available GPUs on the server are visible inside the container, as per the outputs below.

For some reason, setting the runtime to nvidia as default prevents the gpu isolation feature to work as expected. On Server B I haven't installed nvidia-docker2, and the only runtime available for docker is runc and gpu isolation works fine.

Any idea what I'm doing wrong? I appreciate any help. Please, let me know if extra info is needed.

gpu-pod at Server A, with 'nvidia' as Default Runtime:

root@gpu-pod:/# nvidia-smi 
Thu Jul 18 10:08:20 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.74       Driver Version: 418.74       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:0A:00.0 Off |                  N/A |
|  0%   38C    P8    20W / 300W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:41:00.0 Off |                  N/A |
| 35%   35C    P8    25W / 260W |      0MiB / 10981MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

gpu-pod at Server B, with runc as Default Runtime:

root@gpu-pod:/# nvidia-smi 
Thu Jul 18 10:04:40 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.74       Driver Version: 418.74       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:63:00.0 Off |                  N/A |
|  0%   29C    P8     8W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Details from Server A:

 Kernel Version:             4.15.0-55-generic
 OS Image:                   Ubuntu 18.04.2 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.5
 Kubelet Version:            v1.14.1
 Kube-Proxy Version:         v1.14.1
PodCIDR:                     10.233.67.0/24
Non-terminated Pods:         (7 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                              ------------  ----------  ---------------  -------------  ---
  default                    gpu-pod-node5                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         60s
  kube-system                kube-flannel-88l8w                150m (0%)     300m (1%)   64M (0%)         500M (0%)      27m
  kube-system                kube-proxy-gn6n6                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         26m
  kube-system                nginx-proxy-node5                 25m (0%)      0 (0%)      32M (0%)         0 (0%)         28m
  kube-system                nodelocaldns-x6x5w                100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     26m
  kube-system                nvidia-driver-installer-fjpwz     150m (0%)     0 (0%)      0 (0%)           0 (0%)         26m
  kube-system                nvidia-gpu-device-plugin-sbg77    50m (0%)      50m (0%)    30Mi (0%)        30Mi (0%)      26m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests        Limits
  --------           --------        ------
  cpu                475m (1%)       350m (1%)
  memory             200857600 (0%)  709715200 (0%)
  ephemeral-storage  0 (0%)          0 (0%)
  nvidia.com/gpu     1               1

gpu-pod.yaml - Kubernetes manifest used for test purposes

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
    - name: cuda-container
      image: tensorflow/tensorflow:1.13.2-gpu-py3
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          nvidia.com/gpu: 1
  nodeSelector:
    kubernetes.io/hostname: serverA
    #kubernetes.io/hostname: serverB

[bleachzk]

get the same problem

[RenaudWasTaken]

Hello!

Sorry for the very late reply. I'm confused as to where these nvidia-smi commands are coming from? A container on server A and B?

Can you detail a bit more your setup, the commands you are running and the result you are expecting. Also note that without default runtime you can't run the device plugin and hence can't schedule GPU pods.

[RenaudWasTaken]

Also note that isolation is still a work in progress with the community.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.

[github-actions[bot]]

This issue was automatically closed due to inactivity.