streamz icon indicating copy to clipboard operation
streamz copied to clipboard

Published 20 hours ago • verified publisher icon python-streamz

Ray.io

Open dioptre opened this issue 4 years ago • 7 comments

What's required to get streams working on Ray? Anyone started working on it?

dioptre avatar May 14 '20 02:05 dioptre

I am not aware of any work in this direction. Integration would maybe look similar to the current dask interaction code.

martindurant avatar May 14 '20 12:05 martindurant

You might also take a look at how this was done in https://github.com/xpdAcq/rapidz/blob/master/rapidz/clients.py and https://github.com/xpdAcq/rapidz/blob/master/rapidz/parallel.py

CJ-Wright avatar May 14 '20 12:05 CJ-Wright

What is rapidz? And why isn't it in streamz? :) @CJ-Wright

dioptre avatar May 14 '20 21:05 dioptre

Rapidz (https://github.com/xpdAcq/rapidz/) is different than Rapids (https://rapids.ai). @jsmaupin I believe you were referring to Rapids as opposed to Rapidz.

kkraus14 avatar May 14 '20 21:05 kkraus14

Well its also exciting streamz is going to support arrow through rapids too?! @kkraus14

dioptre avatar May 14 '20 21:05 dioptre

Well its also exciting streamz is going to support arrow through rapids too?! @kkraus14

At that point Arrow is more of an implementation detail for GPU execution, but yes we're very excited about stream processing with GPUs + Arrow 😄

kkraus14 avatar May 14 '20 22:05 kkraus14

@dioptre rapidz was my friendly fork of streamz when I was using it to build software for x-ray sciences and needed a faster bug fix cycle. Rapidz also was built to support arbitrary Executor likes (formally arbitrary dask.Client likes, but there is a function for monkey patching Executors into Clients) for parallel execution (for example ThreadPoolExecutor). If Ray supports that kind of interface then it may be possible to use the rapidz code as a blueprint for streamz.

CJ-Wright avatar May 14 '20 23:05 CJ-Wright