[Feature] Run different worker types with different images

[ddelange]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

I would like to run multiple worker groups with individual docker images, e.g. one general-purpose CPU worker using the same image as head (rayproject/ray:latest), and one GPU worker using the rayproject/ray-ml:latest-gpu image.

In the current ray helm chart, one can specify an arbitrary number of worker types. I modified that helm chart to allow individual images per worker type.

I wasn't able to find documentation on how to achieve multiple worker types (with multiple docker images) using the kuberay chart: there is only one worker entry.

Use case

My concrete use-case of values for the ray helm chart:

image: rayproject/ray:1.11.0
namespacedOperator: true
podTypes:
  rayHeadType:
    memory: 1Gi
    CPU: 1
  rayWorkerType:
    memory: 1Gi
    CPU: 1
    minWorkers: 0
    maxWorkers: 4
  rayWorkerTypeGPU:
    image: rayproject/ray-ml:1.11.0-gpu
    GPU: 1
    memory: 15Gi  # g4dn.xlarge has 16GiB and 4vCPU
    CPU: 3800m    # kube-system daemonsets reserve 200m on each node
    rayResources: # trick Ray into still using all 4 cores (as long as cpu.limits is disabled ref https://github.com/ddelange/ray/blob/ce528d4d/deploy/charts/ray/templates/raycluster.yaml#L70)
      CPU: 4      # for kuberay equivalent see #167
    minWorkers: 0
    maxWorkers: 4
    nodeSelector:
      kops.k8s.io/gpu: "1"
    tolerations:
    - effect: NoSchedule
      key: nvidia.com/gpu
      operator: Exists

Related issues

A similar use case (multiple workers with multiple images) but different feature request: #167

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

[Nintorac]

Does it make sense to have RayWorker as it's own custom resource here?

[kevin85421]

Same as #1291. Close this one.