Add watermark-based scheduling to the Ray Runner

[pabloem]

The Ray Runner currently works by topologically sorting the pipeline graph, and executing stage by stage until the whole pipeline has been executed. This means that it only supports batch mode, and it can't execute multiple stages in parallel.

By implementing watermark-based scheduling, and by executing any bundle that is ready for execution, we can start gaining parallelism, and move towards streaming support.

This work is somewhat involved, because it requires changing the whole execution logic for the pipeline, however it should increase our parallelism, which will be great (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py#L420-L487)

[pabloem]

Useful literature: https://s.apache.org/beam-fn-api

[wilsonwang371]

what is our plan to add watermark-based scheduling? any reference implementation that we need to discuss? for example: flink?

[rkenmi]

Hi @pabloem @iasoon , is this PR https://github.com/ray-project/ray_beam_runner/pull/24 associated to this issue? I might be interested in picking this up.