flux-core icon indicating copy to clipboard operation
flux-core copied to clipboard

flux-mini: interactive alloc jobs stick around even when original flux-job attach is terminated

Open grondo opened this issue 2 years ago • 0 comments

Problem: When a user uses flux mini alloc to grab an interactive, single-user instance, the instance will continue running even if the original flux job attach process is terminated, i.e. if the users original interactive session to the system is terminated due to VPN shutdown, or any other reason.

Since a job run with -o pty.interactive acts a little bit like screen or tmux, this is expected behavior, but it may not be the right behavior, since it can lead to wasted resources (The user could log back in and reattach to their job with flux job attach JOBID, but it is perhaps unlikely the user will do that before the job timelimit).

According to @garlick, flux job attach should get SIGHUP in this case, so we could have flux job attach terminate an instance by default if it is attached to an interactive pty when it gets SIGHUP. However, there really isn't a way for flux job attach to tell if there is another process attached to the instance pty, a flux proxy session making use of the instance, or if the user started some jobs which could actually run to completion if the instance was left alone, so this approach could end up being annoying.

Another idea would be to add an idle-timeout to these interactive instances, though that might be tricky since by "idle" we'd probably mean: no activity on pty and no jobs in the queue or running. Maybe the shell pty plugin could be extended to subscribe to events from the job manager or something on "idle" (last client detaches)?

grondo avatar Aug 22 '22 21:08 grondo