[Feature] Add default volumes and volumeMounts for `/tmp/ray` to Ray Pods

[kevin85421]

trafficstars

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

In most KubeRay YAML manifests, you will find volumes and volumeMounts for /tmp/ray, as illustrated below. This enables users to access Ray logs even after the Ray container restarts. I am unsure whether it is common for some Kubernetes clusters to reject Pods with volumeMounts or not. If it is rare, we can add the volumes and volumeMounts for users automatically.

volumeMounts:
  - mountPath: /tmp/ray
    name: ray-logs

volumes:
  - name: ray-logs
    emptyDir: {}

Note: If Ray Autoscaler is enabled, the volume and volumeMount will be set automatically. See code for more details.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[kevin85421]

cc @andrewsykim Would you mind commenting on this issue? Without your comment, I am unable to assign the issue to you. Thanks!

[andrewsykim]

+1 I think just automating the volume mounting is easier.

[andrewsykim]

Actually, one thing to consider here is that users may not want to use emptyDir for the ray logs. They may want to persist them with remote storage or local disk on nodes. We might make that harder to do if we automatically add these volumes and mounts

[kevin85421]

Actually, one thing to consider here is that users may not want to use emptyDir for the ray logs. They may want to persist them with remote storage or local disk on nodes. We might make that harder to do if we automatically add these volumes and mounts

This makes sense. I am wondering if is it common for users to use volumes to persist logs. In my experience, most users utilize logging tools like FluentBit to stream logs to remote storage. Since my data points are quite limited, would you mind sharing your experiences with volume/volumeMount usage in Kubernetes (not limited to KubeRay)?

• Case 1: If it's common for users to set up volume/volumeMount themselves, we should avoid automatically injecting volume/volumeMount for them. Instead, we just need to write the docs to suggest users to add volume for /tmp/ray.
• Case 2: If it's rare, we can inject volume/volumeMount for users with a feature gate to enable users to disable the injection.

[andrewsykim]

From my experience, I find the sidecar route with fluentbit to be the most common, but I don't think case 2) is rare.

Another option we can take is to always apply the default volume mounts unless we see an existing mount from the user with a prefix /tmp/ray. Do you know if ray logs are always stored in /tmp/ray? If so this might be a reliable way to know if we should automatically add log volumes

[kevin85421]

I checked with the Ray team.

1. Ray logs are not always stored in /tmp/ray. It is configurable by setting _temp_dir. See this doc for more details.
2. Most users use the default path /tmp/ray.
3. My colleague said that 3 independent users have tried to configure the path. All of them configured it because they were running Ray alongside an external system that ingests logs written to a specific directory.

[ravishtiwari]

agree with @andrewsykim - case 2 is not rare and I have been using volume mounts to store logs for smaller PoC deployments. However, since this is configurable as standard Kubernetes volume mounts, people can use them while creating RayService or RayCluster deployment configuration YAML.

Are we looking here to add one such relevant example @kevin85421 ?