The resource requests and limits are not being applied to the pod as expected.

[IndhumithaR]

Gpu operator version: v24.6.1
driver.version: 535.154.05 device plugin verion: v0.16.2-ubi8

Kubernetes distribution EKS

Kubernetes version v1.27.0

Hi,

We attempted to install the Nvidia driver directly on our node's base image instead of using the GPU operator. However, after doing so, the resource requests and limits set for the pods are no longer effective, and all containers within the pods are able to access all the GPUs.

Sample pod spec

apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi-pod-3
spec:

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - g5.48xlarge
  containers:
  - name: nvidia-smi-container
    image: nvidia/cuda:12.6.2-cudnn-devel-ubuntu20.04
    command: ["sleep", "infinity"]
    resources:
      limits:
        nvidia.com/gpu: 5
      requests:
        nvidia.com/gpu: 5
 
    securityContext:
      capabilities:
        add:
        - SYS_NICE
      privileged: true
  tolerations:
  - key: "nvidia.com/gpu"
    value: "true"
    effect: "NoSchedule"

Here I am trying to set request and limit to 5. But when I enter into the container and check, I am able to see all the 8 gpus.

However, we tested running the same pod in a different environment where the same driver version was installed using the GPU operator (instead of directly in the base image), and it worked as expected.

What could be the problem? Is there a way to fix it?

[badaldavda8]

Any update on this issue?

[Nimbus318]

@badaldavda8 @IndhumithaR Privileged Pods have direct access to the host's devices—they share the host's device namespace and can directly access everything under the /dev directory. This basically bypasses the container's device isolation.

[badaldavda8]

@badaldavda8 @IndhumithaR Privileged Pods have direct access to the host's devices—they share the host's device namespace and can directly access everything under the /dev directory. This basically bypasses the container's device isolation.

Thanks a lot for your response.

The thing is, we did have same configuration of our pods even before. But only with the change of EC2 installed nvidia drivers vs GPU Operator Nvidia Driver, we have seen this difference

[IndhumithaR]

But when we install nvidia driver with gpu operator , and it worked as expected. I mean limits and request are getting set properly. How is installing driver directly and via gpu-operator different?

Can we restrict the limit and request inspite of running privileged pod?

[Nimbus318]

The thing is, we did have same configuration of our pods even before. But only with the change of EC2 installed nvidia drivers vs GPU Operator Nvidia Driver, we have seen this difference

The driver installation method itself isn't likely the primary cause of resource limitation differences. The key factor is whether the Pod is running in privileged mode, which fundamentally impacts resource control and isolation.

How is installing driver directly and via gpu-operator different?

Direct GPU driver installation requires manual configuration on each machine, suitable for small-scale environments but complex to manage. GPU Operator provides automated, cluster-wide driver management, offering more efficient and consistent resource deployment for large-scale Kubernetes environments.

Can we restrict the limit and request inspite of running privileged pod?

Privileged pods are granted extensive host capabilities, including direct access to devices in the /dev directories. This broad range of permissions can bypass certain Kubernetes security restrictions, making it difficult to enforce standard resource limitations and management mechanisms effectively.

[badaldavda8]

Context - We wanted to set ecc=0 for which we already had to install nvidia driver on the host EC2 instance. We wanted to avoid the time it took to uninstall the driver and then re-install using gpu-operator. Therefore now we are not installing drivers using GPU Operator.

The strange part is that when we had uninstalled and reinstalled with GPU-operator, the pods got allocated GPUs perfectly even with privileged mode enabled. But now that we are not installing it with GPU operator, its allocating all the resources to the pods with privileged mode.

The thing is, we did have same configuration of our pods even before. But only with the change of EC2 installed nvidia drivers vs GPU Operator Nvidia Driver, we have seen this difference

The driver installation method itself isn't likely the primary cause of resource limitation differences. The key factor is whether the Pod is running in privileged mode, which fundamentally impacts resource control and isolation.

The pod was running in privileged mode even before with GPU operator.

How is installing driver directly and via gpu-operator different?

Direct GPU driver installation requires manual configuration on each machine, suitable for small-scale environments but complex to manage. GPU Operator provides automated, cluster-wide driver management, offering more efficient and consistent resource deployment for large-scale Kubernetes environments.

We are using EKS and using Image Builder Pipeline to create an AMI with driver already installed so the scalability is handled here. We wanted to check how is the gpu operator installation changing such that even with privileged pods, its not allocating all the gpus. And how is installing it directly on the AMI making gpus allocate to a single pod?

[DougTea]

any updates?my cluster occurs same problem.Limits of gpu has no effect whether privileged or not.We use self install driver too.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[shivamerla]

Context - We wanted to set ecc=0 for which we already had to install nvidia driver on the host EC2 instance. We wanted to avoid the time it took to uninstall the driver and then re-install using gpu-operator. Therefore now we are not installing drivers using GPU Operator.

The strange part is that when we had uninstalled and reinstalled with GPU-operator, the pods got allocated GPUs perfectly even with privileged mode enabled. But now that we are not installing it with GPU operator, its allocating all the resources to the pods with privileged mode.

The thing is, we did have same configuration of our pods even before. But only with the change of EC2 installed nvidia drivers vs GPU Operator Nvidia Driver, we have seen this difference

The driver installation method itself isn't likely the primary cause of resource limitation differences. The key factor is whether the Pod is running in privileged mode, which fundamentally impacts resource control and isolation.

The pod was running in privileged mode even before with GPU operator.

How is installing driver directly and via gpu-operator different?

Direct GPU driver installation requires manual configuration on each machine, suitable for small-scale environments but complex to manage. GPU Operator provides automated, cluster-wide driver management, offering more efficient and consistent resource deployment for large-scale Kubernetes environments.

We are using EKS and using Image Builder Pipeline to create an AMI with driver already installed so the scalability is handled here. We wanted to check how is the gpu operator installation changing such that even with privileged pods, its not allocating all the gpus. And how is installing it directly on the AMI making gpus allocate to a single pod?

Regardless of whether the NVIDIA driver is installed by the GPU Operator or pre-installed on the node, running a pod in privileged mode exposes all GPU device nodes inside the container (e.g., /dev/nvidia0, /dev/nvidia1, etc.). Because of that, it’s unclear how it was determined that the pods were “not seeing” all GPUs. One possibility is that CUDA_VISIBLE_DEVICES was set to a restricted set of GPUs, this would allow nvidia-smi to display all GPUs, while CUDA applications would only see the subset defined by that variable.

Another scenario where this can occur (for non-privileged pods) is when NVIDIA_VISIBLE_DEVICES=all is set but the pod does not request the nvidia.com/gpu resource. There are however knobs in the container toolkit to prevent this by default.