Jonathan Bingham

In this case, I think of Funnel as 3 distinct components that could be refactored into 3 separate github repos: 1. a robust stateless TES implementation with support for multiple...

@buchanae -- by backends, I meant job schedulers like local execution, SGE, Slurm, Pipelines API, AWS Batch, Azure Batch. A lightweight stateless TES implementation would be a straight pass-through to...

The simplest version would be to not anything about moving files. Just leave it up to each backend what kind of file URLs it accepts. For SGE, that means only...

I see. Does Funnel maintain its own queue in order to reuse VMs? For task dependency, so you can wait until earlier tasks complete? Or because there might not be...

Got it. On Wed, Mar 22, 2017 at 11:28 AM Alex Buchanan wrote: Yes, it maintains a queue. There is no task dependency. If a job can't be scheduled to...

Cool. Are you imagining the TES/Funnel client being a service, library, or CLI?

Great! I'm imagining exactly the same. There could be a library, with all of the backends included. Then the client could call any backend. There could be a lightweight TES...

Yeah, client side is where we're starting. Status is harder because you have a large batch. Eg, you submit 100 tasks in parallel to do a parameter sweep. You can...

Not 100k tasks, but 1000. We currently do this client side. The users create a TSV file of input parameters, one row per task. Then they run a single command-line,...

Cases where client-side isn't ideal: - Close laptop (have to run the client on a VM at addl cost) - Complex graph where a blocking funnel task wastes cpu cycles...