Restoring computation output after disconnect in Notebook

[hadim]

Following this discussion https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jupyter/8hyGKoBY6O0/RyEDDyOZAQAJ, I'd like to open an issue to track down progress on this feature.

The idea would be to add the capability to restore output of a running kernel after client reconnection.

Feel free to ask me to move this issue in a more appropriate jupyter project if needed.

[rshpeley]

Hey Carreau. I noticed on the google forum you said the code would need a significant refactor, and MinRK said it involved storing outputs on the server. In my case I'm ok with losing the outputs during the disconnect. I just need to be able to reconnect the browser session to the kernel to get new outputs and to get at the data and code I've loaded. Kernel/Reconnect doesn't do it. I've also tried closing the tab and opening a new one from the Jupyter base page.

This is fairly important for me. I live in a rural area with a poor internet connection and my notebook takes about 8 hours to load. I'm pretty well guaranteed to have an interruption during that time.

[rshpeley]

Still looking into this. This is currently just a problem for me on Jupyter, but I'm more interested in getting it solved for JupyterHub. I'm planning to use it to instruct a class at the university and students losing access to their assignment work through a disrupted connection is not an option.

mosh have solved this problem on terminals and it seems to me that using mosh's SSP (State Synchronization Protocol) with Speculation adapted to Jupyter Notebook requirements would be the best solution to this problem (see the mosh paper).

If you want to hand this problem off to JupyterHub that's ok with me, but others may like to see a solution for Jupyter as well.

[minrk]

This isn't really a JupyterHub problem. It should be fixed in the notebook (specifically JupyterLab), hopefully before too long.

[post2web]

This feature would be great!

[manuelsh]

Loosing the output of a Jupyter notebook if you are not connected to the hosting computer is a problem, although you can do some hacks. Having the capacity of recovering it would be great.

[ghost]

Just a side note: Apache Zeppelin notebook with Python interpreter doesn't have this problem, as it handles disconnects or multiple connects during tasks executions transparently. But it has its own problems: it loses interactive output for a running cell after a disconnect, although when a task is done it eventually shows all its output.

[louismartin]

Is there any update on this issue?

[jasongrout]

Not a huge amount beyond the discussion and issues linked above, that I know of.

There is (slow, contributors welcome!) work going on in JupyterLab around real-time collaboration, which also involves moving notebook state to the server, which solves this issue. @rgbkrk, @captainsafia and others have also experimented with moving state to the server in the nteract project. And @williamstein's cocalc implementation of a notebook frontend does have this feature - it saves state on the server, so you can reconnect whenever you want and get the full state.

So there have been some initial experiments (jlab, jupyter_server, nteract), and at least one full implementation tied to a platform (cocalc).

[cossio]

Any news on this? Is there an issue / PR that one can check to track progress on this?

[rgbkrk]

Follow this repository for the related effort: https://github.com/jupyterlab/rtc

[apophenist]

Hi friends,

I have this problem often, when the output buffer stops (I use Atom/Hydrogen), and I lose visibility of long running processes.

I have found a bit of a workaround which may be helpful to others (but it requires this to be done upfront, i.e. there is no way that I have found to resume output from existing processes).

The solution requires SSH access, and involves redirecting logging to a file as follows:

    import sys
    log_file = 'results.log'
    sys.stdout=(open(log_file,"w"))
    clf = RandomizedSearchCV(RandomForestRegressor(random_state = 0), param_distributions = param_grid, n_iter = 200, cv=3, random_state = 42, verbose = 51, n_jobs=8)
    sys.stdout.close()

My example is arbitrarily for a RandomizedSearchCV.

However, buffering in std.out proved to be an issue, so Magnus Lycka's referenced answer proved helpful in overriding this:

class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def writelines(self, datas):
       self.stream.writelines(datas)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr) 

and replace above sys.stdout=(open(log_file,"w")) with:
sys.stdout=Unbuffered(open(log_file,"w"))

Now, you can ssh into the machine, run docker exec -it image_name bash and tail -f the log file.

Hope this helps, it seems to be the best solution for now.

[banderlog]

Just met this issue with losing 5h computations on GPU. Used AI notebook in GCP. Thought that it will do plots with a closed browser, but I thought wrong :(

[astrojuanlu]

There is (slow, contributors welcome!) work going on in JupyterLab around real-time collaboration, which also involves moving notebook state to the server, which solves this issue.

To my understanding, Real Time Collaboration is now supported in JupyterLab https://github.com/jupyterlab/jupyterlab/issues/5382

@jasongrout as I read in https://blog.jupyter.org/how-we-made-jupyter-notebooks-collaborative-with-yjs-b8dff6a9d8af

Currently, we still send HTTP PUT requests to save a document to the file system via the Jupyter Server. Once we have the Yjs CRDT working in Python, we will create a Python implementation of the shared notebook model which will allow the Jupyter Server to directly access the collaborative state and synchronize that with the file system.

It is not clear to me if the notebook state is now handled by the server and therefore this issue is fixed, or if it is still present.

[jasongrout]

To my understanding, Real Time Collaboration is now supported in JupyterLab jupyterlab/jupyterlab#5382

This will be released in the upcoming 3.1 release, behind a command-line flag.

It is not clear to me if the notebook state is now handled by the server and therefore this issue is fixed, or if it is still present.

This is still present for now, but the infrastructure needed to move things to the server is being put in place by the RTC work. Moving state to the server is another big project that builds on the RTC work being done now.

[cossio]

What issue / repo / PR should one follow to get relevant updates on this issue? Will it be resolved on this repo or on jupyterlab?

[tgpfeiffer]

Related to https://github.com/jupyter/notebook/issues/641#issuecomment-872469554: I have tried the Jupyter collaboration feature in the latest release (3.3.3) and I found out that if user A starts a long-running computation, then the output is synced across browsers to all collaborators and they can see it in real-time, reload their browser windows etc., but if A does it, then we are back at the place of this issue, the output goes into the void and cannot be accessed later.

So as written by @jasongrout above this still seems to be a WIP. I'd be interested to contribute some work here, where is it happening? Any main issue, meetings etc. where this feature is driven forwards?

[Wh1isper]

Related to #641 (comment): I have tried the Jupyter collaboration feature in the latest release (3.3.3) and I found out that if user A starts a long-running computation, then the output is synced across browsers to all collaborators and they can see it in real-time, reload their browser windows etc., but if A does it, then we are back at the place of this issue, the output goes into the void and cannot be accessed later.

So as written by @jasongrout above this still seems to be a WIP. I'd be interested to contribute some work here, where is it happening? Any main issue, meetings etc. where this feature is driven forwards?

I am also interested in this issue, which is a problem of the overall Jupyter architecture. Jupyter puts all file-related operations on the front end, which disappears after closing the tab page, naturally leading to the problem of missing output

Looking at the behavior of JupyterLab, I found that JupyterLab itself has some problems with the kernel state loop. For example, when reconnecting to a kernel that is performing a computation task, JupyterLab will lose the ability to track its output, such as the following code.

import time 
while True:
  time.sleep(1)
  print(1)

I think the reconnection problem should be solved first, and the loss of output during front-end shutdown is probably tolerable.

Second, the best solution is to leave the output of the computation task to the backend and return it to the frontend appropriately.

[jasongrout]

I am also interested in this issue, which is a problem of the overall Jupyter architecture.

There is active work in JupyterLab to correct this architecture issue. This builds on work done over the last year in real-time collaboration. @davidbrochart, is there a top-level issue in JupyterLab tracking the work to move state to the server side?

[jasongrout]

It looks like https://github.com/jupyterlab/jupyterlab/issues/11434 contains useful discussion about the current state of the RTC work, and https://github.com/jupyterlab/jupyterlab/issues/2833 (also noted above) is another good issue to follow.

[davidbrochart]

Besides the issues @jasongrout mentioned, https://github.com/jupyterlab/jupyterlab/pull/12360 is the PR that will probably solve this issue. I am actively working on it, and it also depends on the development of https://github.com/y-crdt/ypy. Give us one more month or so!

[cossio]

Now that https://github.com/jupyterlab/jupyterlab/pull/12360 is merged, any updates on this issue? Thanks!

[davidbrochart]

Seems to work: https://user-images.githubusercontent.com/4711805/168849750-7a1a3a29-807f-457e-b9cc-471afa30960c.mp4

[davidbrochart]

But it already works with Jupyter 3.4.2, so this is not related to https://github.com/jupyterlab/jupyterlab/pull/12360.

[cossio]

@davidbrochart Wowwww had no idea! So if I just upgrade to latest Jupyterlab I will get this fix?

EDIT: Does it work with other languages besides Python?

[davidbrochart]

To be honest I'm as surprised as you, nothing has been done to fix this issue, and it's not related to RTC at all. Maybe it has always been working in JupyterLab? It's not specific to Python, it should work with other languages too.

[cossio]

Hmm doesn't seem to be working for me (I'm using Julia).

What version of the Jupyter server you are using?

[davidbrochart]

Could you post a screencast?

[cossio]

https://user-images.githubusercontent.com/10637482/168861679-aa562fe5-5f78-444b-9531-7baf3b807c05.mp4

@davidbrochart I tried to replicate your video, with Python. Notice here that the outputs keep coming in even after the Throttling setting is set to Offline, so I'm not sure this is a good experiment?

[davidbrochart]

Outputs are updating event when you're offline :smile: I saw that too, maybe Google Chrome's dev tool is broken?

[cossio]

Maybe. If I try to refresh the notebook browser window (hit F5), however, I lose all output :disappointed: