flux-core
flux-core copied to clipboard
flux-mini: interactive alloc jobs stick around even when original flux-job attach is terminated
Problem: When a user uses flux mini alloc
to grab an interactive, single-user instance, the instance will continue running even if the original flux job attach
process is terminated, i.e. if the users original interactive session to the system is terminated due to VPN shutdown, or any other reason.
Since a job run with -o pty.interactive
acts a little bit like screen
or tmux
, this is expected behavior, but it may not be the right behavior, since it can lead to wasted resources (The user could log back in and reattach to their job with flux job attach JOBID
, but it is perhaps unlikely the user will do that before the job timelimit).
According to @garlick, flux job attach
should get SIGHUP
in this case, so we could have flux job attach
terminate an instance by default if it is attached to an interactive pty when it gets SIGHUP
. However, there really isn't a way for flux job attach
to tell if there is another process attached to the instance pty, a flux proxy
session making use of the instance, or if the user started some jobs which could actually run to completion if the instance was left alone, so this approach could end up being annoying.
Another idea would be to add an idle-timeout to these interactive instances, though that might be tricky since by "idle" we'd probably mean: no activity on pty and no jobs in the queue or running. Maybe the shell pty plugin could be extended to subscribe to events from the job manager or something on "idle" (last client detaches)?