[KubeRay, dashboard] Clarify that the users can use persistent volumes for log_dir and ray dashboard can read from it.

[scottsun94]

Description

https://ray-distributed.slack.com/archives/C01DLHZHRBJ/p1683569842407549?thread_ts=1682522893.195049&cid=C01DLHZHRBJ

A user suggests that we should clarify that the users can use persistent volumes for log_dir and ray dashboard can read from it.

Quote from the user

should be that one way to do NFS is using PVs with ReadWriteMany capability, and a little example of where to mount the PVs so that the dashboard loads them correctly (I imagine it could be anywhere for NFS but for the dashboard specifically there should be one specific path, but I suggest checking this internally)

Link

No response

[richardliaw]

Thanks for filing this!

This should be documented in KubeRay right?

We shouldn't have infra specific configuration concepts in Tune, and we also already have a guide for using Tune with an NFS: https://docs.ray.io/en/latest/tune/tutorials/tune-storage.html

[scottsun94]

@richardliaw Not necessarily in kuberay doc This is something that the Tune users care about. I think that there are at least two things we should improve to the tune docs:

1. Talk about how to visualize the ray results in ray dashboard if they want. (I believe if the ray results are saved into the log directory, they'll be rendered automatically?)
2. Maybe add a link to the guide for using Tune with an NFS about how to use persistent volumes as NFS

[richardliaw]

No, again this does not have anything to do with hyperparameter tuning but rather is a sysadmin / kuberay responsibility. I think there's a conflation of issues here:

ray results in ray dashboard

This isn't related to the thread right? The user is asking about Ray logs, not result visualization (which is currently a responsibility of external system like w&b/mlflow, not Ray). Ray log locations should be configured at the Ray start level (which in this case, is in kuberay configs).

Maybe add a link to the guide for using Tune with an NFS about how to use persistent volumes as NFS

I couldn't quite understand why we need this, per the thread...

[scottsun94]

You are right. I probably misread the thread since the user linked the ray tune doc.

[cszhu]

This P2 issue has seen no activity in the past 2 years. It will be closed in 2 weeks as part of ongoing cleanup efforts.

Please comment and remove the pending-cleanup label if you believe this issue should remain open.

Thanks for contributing to Ray!