How about background jobs?

[FrontMage]

Sometimes a job can take a long time or just a daemonized job. Wouldn't it be great to have something like pm2 to manage background jobs?

$ corgi bg

┌──────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬───────┐
│ Job name     │ id │ mode │ pid   │ status │ restart │ uptime │ cpu │ mem       │ user  │
├──────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼───────┤
│ job_hi       │ 0  │ fork │ 2273  │ online │ 1       │ 6D     │ 0%  │ 10.3 MB   │ sport │
└──────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴───────┘

[junyu-w]

hmm I'm a little bit hesitant to add this to corgi cuz as you mentioned there's dedicated tools like pm2 that does the job well already, is there a specific use case that makes you think this could be a good addition to corgi?

[FrontMage]

I mean pm2 and other stuff are good at process managing, but they can not start a command, and they don't have that script management like corgi does.

Sometimes I need to map local ports to server ports using ssh tunnelling, and kill them after I'm done. With pm2 and all that, I need to have a folder of scripts, and I have to change them if the ports are changed.

If I have corgi with background job management, I can simply write one corgi script, then use template to pass ports and all that, kill them after I'm done.

[junyu-w]

ok I see, so what you requested are actually two parts:

1. allow corgi to start job in the background
2. allow corgi to manage/monitor background process

I think point 1 is actually a good addition since starting corgi directly in the background with & won't prompt you to enter your templates!

As for managing background jobs, do you think it'll be the same if you just do corgi exec -bg <snippet> (after it's implemented) and use pm2 for managing?

[FrontMage]

It will not be the same, pm2 is delicate to process management, log rotate.

corgi has the template while pm2 have to rewrite the corresponding script. corgican start a command while pm2 have to write it as pm2.json and remember its location when starting.

That's why pm2 is not the man for the job.

While pm2 has a daemon in the background, monitoring all process status, corgi need only remember the pid of that job, so that after everything is done, the user can terminate that process.

[junyu-w]

lol that was a mis-click.

I'll research a little bit more, since this feature doesn't seem to be trivial I'll see if it can fit into the 0.3.0 release

[FrontMage]

@DrakeW After trying to make a pull request about this, I met 2 problems:

• With the -bg flag, does this make all steps background jobs or something else?
• What to do about crashed jobs, stopped jobs and all their logs?

For the -bg flag I think the best way is to define background jobs them in corgi new, for each step users can specify if it is a background job.

About the logs and all that, I'm thinking to write an independent package just to manage those background jobs. With corgi ps list all jobs and corgi logs <id> to get logs, pretty much like the docker command line tool.

Appreciate some advice.

[junyu-w]

@FrontMage - thanks for your effort in making this happen My input on this:

• for -bg flag, all steps should be made as background jobs, and there will be a background job id returned for the snippet that can be used later to check on progress & logs
• and regarding crashed/stopped jobs, similar to how corgi handles step today, the execution will halt upon failure. And their logs should be saved in some temporary file that could be displayed with the corgi logs <id> command

[FrontMage]

@DrakeW If all steps a made background job, there will be more than one id returned. So are they grouped together by some namespace, or just bunch of sequence ids?

[junyu-w]

the id I referred to was not the pid of a running process, but a unique id (a layer of abstraction) that corgi creates that provide easy access to the underlying running pids. So it's more like a namespace.