run -local mode doesn't kill all spawned processes when canceled

[olgabot]

Hello, I killed a reflow run -local job with Ctrl-C and a bunch of the processes are still running so I have to manually kill them. Is there a way to get Reflow to look for its spawned processes? Warmest, Olga

[mariusae]

Since -local still uses Docker, the spawned processes aren't children of Reflow and thus are not killed by the operating system when Reflow is killed. One interesting thing: if you kill Reflow like this, and then restart it, you'll find that it'll just attach itself to already running Docker executions, so that they do not need to be restarted.

We have discussed changing -local mode to use a reflowlet (Reflow server) as well: so when you run reflow -local, it would look for an existing reflow server instance or start one anew. In this case, when you kill the original reflow the reflowlet would notice that it's gone and then dispose of the running tasks.

cc: @swami-m since this is also interesting for another change he's been working on

[olgabot]

Ah I see, that explains some of the issues I was having! I'd Ctrl-C the reflow process, git pull for the latest workflow and then get the exact same error, and only by running on a fresh EC2 instance could I get the configurations to refresh.

[mariusae]

you can also clear the local state by clearing /tmp/flow or using a different local dir: reflow run -local -localdir=/tmp/xxx ...

[mariusae]

This is by design currently, but we are working on changing the way local mode works so that this will not happen.