deepforge icon indicating copy to clipboard operation
deepforge copied to clipboard

Disconnect/reconnect to/from long-running interactive sessions

Open brollb opened this issue 4 years ago • 2 comments

Especially given the upcoming interactive editor for training neural networks (#1816), it is likely that users will want to train long running jobs that they can later monitor and check in on. To that end, interactive sessions should be able to run without a connected browser connection and resume them at a later date.

brollb avatar Sep 21 '20 13:09 brollb

This actually raises a bigger question about the current framework for interactive editors/sessions. Currently, the browser connects to the compute instance using DeepForge as a proxy (via the InteractiveCompute router). An alternative approach would be to add a server-side plugin as shown below (on the right) and to persist the data in the model from the start (probably with locks so only one user can open a node at a time): IMG_20201110_105505

In this case, the plugin would need to expose an editor-specific API to the browser and project edits would need to be sent to the plugin. The plugin would manage communication between the compute instance and the project. This would make reconnecting/suspending sessions quite simple as the plugin would remain running when the browser has been closed.

The biggest challenge of this approach lies in detecting abandoned sessions (so we don't have essentially a "session leak") and locking the nodes so they only have a single editor at a time.

brollb avatar Nov 10 '20 18:11 brollb

It is also unclear if artifact explorers should behave the same way... It is more likely that they will not be quite as long running though it certainly is possible.

brollb avatar Nov 10 '20 20:11 brollb