noleme-flow
noleme-flow copied to clipboard
[API] Flow helper `runAs*` methods simplification
Consider moving away from the runAsPipeline
, runAsParallel
methods, and instead provide compiler helpers :
So instead of:
var flow = /* Flow.from(x).pipe(etc)... */
Flow.runAsPipeline(flow);
Decompose it so that:
var flow = /* Flow.from(x).pipe(etc)... */
Flow.createPipeline(flow).run();
This also helps with a common pattern when building flow-based reusable classes, where we compile the graph and delay and/or reuse the runtime several times. As it currently stands, the developer has to explicitly create a compiler of the appropriate type:
var path = Input.key(String.class);
var flow = /* Flow.from(path).pipe(etc)... */
var runtime = new PipelineCompiler().compile(flow);
/* somewhere else */
runtime.run(Input.of(path, "/some/path");
instead, we could handle it more gracefully (by virtue of being aligned with instant run scenarios):
var path = Input.key(String.class);
var flow = /* Flow.from(path).pipe(etc)... */
var runtime = Flow.createPipeline(flow);
/* somewhere else */
runtime.run(Input.of(path, "/some/path");
Of note, this would help with having richer run
arguments, like a decomposed Input
overload for instance. If we keep the runAs
methods we can't keep up with the diversity of arguments required for several different compilation and runtime setups (eg. nodes + parallelism + pool management + Input
vs. nodes + parallelism + decomposed Input
).