Daft icon indicating copy to clipboard operation
Daft copied to clipboard

Clarify URI syntax for connecting to ray cluster

Open kevinjqliu opened this issue 1 year ago • 1 comments

Describe the bug Ray exposes two ports, 6379 and 10001, depending on which port is used, the syntax for daft.context.set_runner_ray is different.

For example: Both of these works

daft.context.set_runner_ray(address="127.0.0.1:6379")
daft.context.set_runner_ray(address="ray://127.0.0.1:10001")

But this does not

daft.context.set_runner_ray(address="ray://127.0.0.1:6379")

This might be a Ray thing, but I think it'll make sense to clarify it in the Daft docs. Esp, since there are lots of examples using the ray:// prefix https://github.com/search?q=repo%3AEventual-Inc%2FDaft+%22ray%3A%2F%2F%22&type=code

To Reproduce Code above

Expected behavior As a user, I would want Ray to work regardless of which "style" I specify.

daft.context.set_runner_ray(address="127.0.0.1:6379")
daft.context.set_runner_ray(address="127.0.0.1:10001")
daft.context.set_runner_ray(address="ray://127.0.0.1:6379")
daft.context.set_runner_ray(address="ray://127.0.0.1:10001")

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context Add any other context about the problem here.

kevinjqliu avatar Sep 25 '24 21:09 kevinjqliu

assigning @raunakab!

samster25 avatar Sep 26 '24 00:09 samster25

Hey @kevinjqliu, I think this issue was fixed recently.

I ran all three of the commands that you gave me, and each one succeeded. When you were running the final one (i.e., daft.context.set_runner_ray(address="ray://127.0.0.1:6379")), were you receiving python exceptions?

raunakab avatar Oct 23 '24 20:10 raunakab

@raunakab I think it is fixed, I couldn't reproduce the issue.

But I also cannot reproduce the issue using old versions of the library. I remember using https://github.com/Eventual-Inc/Daft/blob/main/tutorials/intro.ipynb for testing. I edited this cell with the above options

## Use the Python multithreaded local runner (default behavior)
# daft.context.set_runner_py()

## Connect to a Ray cluster and use the Ray runner
# daft.context.set_runner_ray(address="ray://...")

kevinjqliu avatar Oct 23 '24 21:10 kevinjqliu