[Trimaran] Support GPU resource

[Telemaco019]

Hello everyone,

It would be nice if Trimaran supported load-aware scheduling based on GPU utilization.

The load-watcher service supports Prometheus sources, so the metrics exposed by the NVIDIA DCGM Exporter (described here) could be used for determining GPU usage.

The feature could either be implemented as additional plugins (e.g. something like GPUTargetLoadPacking and GPULoadVariationRiskBalancing) or as an extension of the existing ones.

Example

The following is a simple use case I would like Trimaran to support.

Scenario:

• two nodes (Node A and Node B)
• each node has a single NVIDIA GPU
• the NVIDIA k8s-device-plugin is configured for partitioning the GPUs of the two nodes using time-slicing (thus the compute resources of the GPUs are shared among multiple Pods).
• Node A and Node B both expose the resource nvidia.com/gpu=3
• Node A and Node B are both running a Pod requesting nvidia.com/gpu=1

Use case:

• overall average GPU usage of Node A is 10%
• overall average GPU usage of Node B is 80%
• a new Pod requesting nvidia.com/gpu=1 is created
• the Pod could be hosted by both Node A and Node B, but the scheduler should prefer Node A since its average GPU usage is lower

[Huang-Wei]

cc @atantawi @zorro786 @wangchen615

[wangchen615]

@Huang-Wei , thanks for tagging us. We will take a look.

[wangchen615]

@Telemaco019, thanks for raising this issue. If you are interested in contributing, let's have a follow-up discussion on k8s slack.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.