wally icon indicating copy to clipboard operation
wally copied to clipboard

Add coalescing of stateless computations into state steps

Open jtfmumm opened this issue 8 years ago • 5 comments

We currently coalesce stateless computations into the same steps since (given that we only have linear pipelines) this leads to less message passing and better performance with correct results. There's no reason in principle why we shouldn't coalesce a series of stateless computations in a preceding state computation, but this functionality is not currently supported.

In order to do this, we would need a way to look up the correct sequence of stateless computations given where the state computation output should be routed to. One possibility is a kind of internal router that maps from target id to Runner-Router chain. To do this, we would have to find a way to assign this target id both to the chain of computations and the router on the step before the state step.

jtfmumm avatar Dec 15 '16 14:12 jtfmumm

@jtfmumm do we need to do this before we open source or can it come afterwards?

SeanTAllen avatar Jan 31 '17 02:01 SeanTAllen

This would be invisible to the app developer. It's a performance optimization. So I think we can probably open source first.

jtfmumm avatar Jan 31 '17 16:01 jtfmumm

@jtfmumm is this done?

SeanTAllen avatar Aug 15 '17 02:08 SeanTAllen

No

jtfmumm avatar Aug 15 '17 02:08 jtfmumm

It's not clear we should do this. See #2633.

jtfmumm avatar Nov 07 '18 17:11 jtfmumm