Updating view in the background with threads

[katsar0v]

I am not sure if this is possible, but it definitely works in JupyterLab 1.x flawlessly. It does not in Voila, if there is a working way to do it, I will be happy to try it out and maybe make a PR in the documentation about it.

I have two notebooks, one with asyncio, one with threading.

Asyncio

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import time
import asyncio
import random
from IPython.display import display, clear_output
%matplotlib inline

fig = plt.figure()
ax = plt.axes()
x = np.linspace(0, random.randint(5,10), random.randint(100,1000))
plt.plot(x, np.sin(x))
plt.show()

async def do():
    while True:
        await asyncio.sleep(0.5)
        clear_output()
        fig = plt.figure()
        ax = plt.axes()
        x = np.linspace(0, random.randint(5,10), random.randint(100,1000))
        plt.plot(x, np.sin(x))
        display(plt.gcf())
        
    
print("Starting async thread")
asyncio.create_task(do())

Threading

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import time
import threading
import random
from IPython.display import display, clear_output
%matplotlib inline

fig = plt.figure()
ax = plt.axes()
x = np.linspace(0, random.randint(5,10), random.randint(100,1000))
plt.plot(x, np.sin(x))
plt.show()

def do():
    while True:
        time.sleep(0.5)
        clear_output()
        fig = plt.figure()
        ax = plt.axes()
        x = np.linspace(0, random.randint(5,10), random.randint(100,1000))
        plt.plot(x, np.sin(x))
        display(plt.gcf())
        
    
print("Starting async thread")
t = threading.Thread(target=do)
t.start()

Here I have preview (with gifs) what happens:

JupyterLab with asyncio:

JupyterLab with threading:

JupyterLab and voila with asyncio:

JupyterLab and voila with threading:

I would like to make an example in the Voila Gallery about a self-updating dashboard, but I need the threads working for this. If you could help me out with this, I will be really thankful.

PS: I tried with %matplotlib widget and %matplotlib ipympl, they did not refresh even in JupyterLab.

[maartenbreddels]

Hi Kristiyan,

could you try if you get this to work with: https://ipywidgets.readthedocs.io/en/latest/examples/Output%20Widget.html Also, clear_output(wait=True) helps reducing flicker.

cheers,

Maarten

[katsar0v]

Hi Kristiyan,

could you try if you get this to work with: https://ipywidgets.readthedocs.io/en/latest/examples/Output%20Widget.html Also, clear_output(wait=True) helps reducing flicker.

cheers,

Maarten

Could you maybe give me an example?

Tried:

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

Which works in JupyterLab but not in Voila :)

Edit: it did not work in the JupyterLab preview, but worked running voila Untitled.ipynb. The examples I gave above do not work with voila Untitled.ipynb in the console.

PS: Another example that works in JupyterLab but not in Voila:

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import ipywidgets as widgets
from IPython.display import display, clear_output
import numpy as np
import threading
import time
import random

out = widgets.Output()
def do(out):
    while True:
        display(out)
        data1 = pd.DataFrame(np.random.normal(size = random.randint(50,5000)))
        fig1, axes1 = plt.subplots()
        data1.hist(ax = axes1)
        plt.show(fig1)
        time.sleep(0.5)
        clear_output()
        out = widgets.Output()

    
with out:
    t = threading.Thread(target=do, args=(out,))
    t.start()

[jtpio]

@katsar0v how about something like this (using Python 3.7+):

import asyncio

%matplotlib widget
import ipympl

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import random

SLEEP = 1

all_tasks = asyncio.all_tasks()
for task in all_tasks:
    task.cancel()


fig1, axes1 = plt.subplots()
plt.show(fig1)

async def do():
    while True:
        data1 = pd.DataFrame(np.random.normal(size = random.randint(50,5000)))
        axes1.cla()
        data1.hist(ax = axes1)
        plt.draw()
        await asyncio.sleep(SLEEP)

asyncio.create_task(do());

[katsar0v]

That's great! Thanks @jtpio

It works with your code :) you used plt.draw() which works in voila, but plt.show() and clear_output() do not. What was the mistake I made and what can we add in the docs to avoid this in the future?

Also why display(widgets.Output()) together with clear_output() had no effect. I might be wrong becayse I do not understand the ipython way of visualizing things, but I think clear_output() does not work?

Still I'm going to make a repo with an interactive dashboard to use as a demo and boilerplate, so others do not struggle with these display problems like me :)

One last question @jtpio: currently it works with this plot, what if I want to make it with a widget (tabs or something like this) or just want to display a new YouTube video every 3 seconds?

[jtpio]

Also why display(widgets.Output()) together with clear_output() had no effect. I might be wrong becayse I do not understand the ipython way of visualizing things, but I think clear_output() does not work?

out.clear_output() should work with the Output widget.

what if I want to make it with a widget (tabs or something like this) or just want to display a new YouTube video every 3 seconds?

ipywidgets has a Tab widget: https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html#Tabs, which can be updated in the do function used in the examples above.

Also you might want to consider using bqplot and its Figure widget (lots of example on Binder)

[maartenbreddels]

Tried:

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

Which works in JupyterLab but not in Voila :)

This works for me, which version are you using?

[katsar0v]

Tried:

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

Which works in JupyterLab but not in Voila :)

This works for me, which version are you using?

This one works, sorry for the confusion, tested again. It seems clear_output() and other lines in the examples (gifs) I posted above do not work in Voila.

[jtpio]

@katsar0v do you still want to keep this issue open for further discussions, now that your self-updating example is working?

[katsar0v]

We might close this issue, although it is not clear why it is not working with normal threads and why the code I posted above does not work

[anjackson]

I can get the append-only example to work, but if I try to clear the output, it doens't work!?

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.clear_output() # <<< THIS DOES NOTHING
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

[Sedom9]

Hello!

I did thread, where plot updated every 1 second. In Jupyter all right. But in voila, I haven't updated plot.

Code below:

def learning_curves(stop_button):

        metrics_graphic_buffer.append(f_score[0])
        metrics_graphic_buffer1.append(f_score[1])
        metrics_graphic_buffer2.append(f_score[2])

        stop_button.on_click(stoping)
        
        with output_learning_curves:
            clear_output()
    #             print("Metrics graphic buffer", metrics_graphic_buffer)
            plt.figure(figsize=(12, 12))
            plt.axis([0, 20, 0, 1.1])
            plt.plot(metrics_graphic_buffer, 'r--', label = 'Tap', 
                     linewidth = 1.0)
            plt.plot(metrics_graphic_buffer1, 'g--', label = 'Toilet', 
                     linewidth = 3.0)
            plt.plot(metrics_graphic_buffer2, 'y--', label = 'Shower', 
                     linewidth = 5.0)
            plt.plot(avr, 'black', label = 'Average f-score', 
                     linewidth = 4.0)
            plt.legend(loc='upper right', prop={'size': 12})
            plt.draw()
            time.sleep(1)

        if flag == False:
            print ('Status flag1', flag)
            break

Initial thread : thread = threading.Thread(target=learning_curves, args=(stop_button, ))

Start thread :

def on_clicked_Learn_button(state):
    with output_learning_curves:
        thread.start()

Thanx for your help !

[cailiang9]

Tried:

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

Which works in JupyterLab but not in Voila :)

This works for me, which version are you using?

This one works, sorry for the confusion, tested again. It seems clear_output() and other lines in the examples (gifs) I posted above do not work in Voila.

seems ipw.HTML can work as a replacement:

def ipwHTMLstr(df, max_rows=20, max_cols=20):
    return '<div class="jp-RenderedHTMLCommon jp-RenderedHTML jp-OutputArea-output">%s</div>'%df.to_html(escape=True, max_rows=max_rows, max_cols=max_cols)

# in thread foo:
ht.value=ipwHTMLstr(df)

[cailiang9]

setting out.output=[] can fix out.clear_output not working issue. see github for reference: https://github.com/jupyter-widgets/ipywidgets/blob/master/ipywidgets/widgets/widget_output.py#L159

I can get the append-only example to work, but if I try to clear the output, it doens't work!?

import threading
from IPython.display import display, HTML
import ipywidgets as widgets
import time

def thread_func(something, out):
    for i in range(1, 5):
        time.sleep(0.3)
        out.clear_output() # <<< THIS DOES NOTHING
        out.append_stdout('{} {} {}\n'.format(i, '**'*i, something))
    out.append_display_data(HTML("<em>All done!</em>"))

display('Display in main thread')
out = widgets.Output()
# Now the key: the container is displayed (while empty) in the main thread
display(out)

thread = threading.Thread(
    target=thread_func,
    args=("some text", out))
thread.start()

[rht]

Encountered this problem again in #1341.

[richy1996]

Encountered this problem again in #1341.

@rht did you ever get this to work?

[rht]

Yes, I did: https://github.com/voila-dashboards/voila/issues/1341#issuecomment-1608976452. Though we (Mesa) has since been using Solara instead.