noleme-flow icon indicating copy to clipboard operation
noleme-flow copied to clipboard

[API] Flow helper `runAs*` methods simplification

Open eledhwen opened this issue 2 years ago • 0 comments

Consider moving away from the runAsPipeline, runAsParallel methods, and instead provide compiler helpers :

So instead of:

var flow = /* Flow.from(x).pipe(etc)... */
Flow.runAsPipeline(flow);

Decompose it so that:

var flow = /* Flow.from(x).pipe(etc)... */
Flow.createPipeline(flow).run();

This also helps with a common pattern when building flow-based reusable classes, where we compile the graph and delay and/or reuse the runtime several times. As it currently stands, the developer has to explicitly create a compiler of the appropriate type:

var path = Input.key(String.class);
var flow = /* Flow.from(path).pipe(etc)... */
var runtime = new PipelineCompiler().compile(flow);
/* somewhere else */
runtime.run(Input.of(path, "/some/path");

instead, we could handle it more gracefully (by virtue of being aligned with instant run scenarios):

var path = Input.key(String.class);
var flow = /* Flow.from(path).pipe(etc)... */
var runtime = Flow.createPipeline(flow);
/* somewhere else */
runtime.run(Input.of(path, "/some/path");

Of note, this would help with having richer run arguments, like a decomposed Input overload for instance. If we keep the runAs methods we can't keep up with the diversity of arguments required for several different compilation and runtime setups (eg. nodes + parallelism + pool management + Input vs. nodes + parallelism + decomposed Input).

eledhwen avatar Nov 22 '22 10:11 eledhwen