Clarify URI syntax for connecting to ray cluster
Describe the bug
Ray exposes two ports, 6379 and 10001, depending on which port is used, the syntax for daft.context.set_runner_ray is different.
For example: Both of these works
daft.context.set_runner_ray(address="127.0.0.1:6379")
daft.context.set_runner_ray(address="ray://127.0.0.1:10001")
But this does not
daft.context.set_runner_ray(address="ray://127.0.0.1:6379")
This might be a Ray thing, but I think it'll make sense to clarify it in the Daft docs. Esp, since there are lots of examples using the ray:// prefix https://github.com/search?q=repo%3AEventual-Inc%2FDaft+%22ray%3A%2F%2F%22&type=code
To Reproduce Code above
Expected behavior As a user, I would want Ray to work regardless of which "style" I specify.
daft.context.set_runner_ray(address="127.0.0.1:6379")
daft.context.set_runner_ray(address="127.0.0.1:10001")
daft.context.set_runner_ray(address="ray://127.0.0.1:6379")
daft.context.set_runner_ray(address="ray://127.0.0.1:10001")
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
assigning @raunakab!
Hey @kevinjqliu, I think this issue was fixed recently.
I ran all three of the commands that you gave me, and each one succeeded. When you were running the final one (i.e., daft.context.set_runner_ray(address="ray://127.0.0.1:6379")), were you receiving python exceptions?
@raunakab I think it is fixed, I couldn't reproduce the issue.
But I also cannot reproduce the issue using old versions of the library. I remember using https://github.com/Eventual-Inc/Daft/blob/main/tutorials/intro.ipynb for testing. I edited this cell with the above options
## Use the Python multithreaded local runner (default behavior)
# daft.context.set_runner_py()
## Connect to a Ray cluster and use the Ray runner
# daft.context.set_runner_ray(address="ray://...")