Support for Container Linking & Services

[ChetanBhasin]

Description I'm not entirely sure if it's already possible, but I couldn't find any evidence of it, but it would be great to have container linking and running services. For example, for running tests that require a database or external cache, it's very useful to be able to have a long-running task run alongside another. It's also useful for running integration tests where one service communicates with the other.

Alternatives considered I've considered two alternatives:

• Docker Compose: This is nice and very useful, but it does not have the DAG like toast does. Neither is the caching on Docker Compose good enough for complex workflows, particularly in CI.
• Running parallel toast jobs: Running another toast job alongside one, but linking them with a virtual network so the applications/services can talk to each other.

[stepchowfun]

Hi @ChetanBhasin, thanks for this feature request. It seems like it would be a useful feature, though it would be quite a departure from the way Toast works today. We'd have to think through what it would look like and what the implications would be (if any). For now, I'll leave this issue open for further discussion.

[stepchowfun]

Here is an idea for how it might work:

• Tasks will run in parallel if and only if (a) they have cache: false and (b) they do not have a common descendant in the dependency graph (or the dependency subgraph containing only the tasks relevant for the current execution?).
• For simplicity, tasks would run on the default bridge network, as they do today. This means parallel tasks can talk to each other, but they can also talk to other Docker containers that are not related to Toast (not great, but I'm on the fence about having Toast creating a user-defined bridge network).

This would likely be a large change, so if anyone is interested in taking this on, please discuss it with me first. I might not have the bandwidth to review such a change.

@ChetanBhasin For now, I think running multiple Toast jobs in parallel should "just work", as they would run on Docker's default bridge network and should be able to talk to each other (even without declaring any ports in the toastfile).

[ChetanBhasin]

@stepchowfun I like your suggestion. There are also some interesting considerations to keep in mind when running tasks in parallel. What if the parallel running tasks are interactive? And do we output the logs from all of them in the console? That could be quite annoying, IMO.

Anyway, I'm happy to brainstorm a solution and try implementing it.

[stepchowfun]

Those are great points. Right now Toast does not really support interactive tasks, because it does not attach stdin (except for the --shell feature—speaking of that, we'd have to think of what that does in this hypothetical parallel world).

For output, Toast currently reuses its own stdout/stderr for the container. But we'd probably want to change that to do some kind of streaming, prefixing each chunk with some kind of line/header indicating which task it's from (I think I recall Docker Compose doing something like this, but I haven't used it in a while).

I am worried this could get quite complex unless we are careful. For example, maybe users would want to see only the output from one of their containers, and have everything else be logged to files. Or maybe users would want to customize how the network is configured. Etc...

FWIW, as a workaround for not having parallel tasks, I sometimes just run multiple processes in parallel within a single task. It's not ideal, but it works well enough for my simple use cases.