ipywidgets

ipywidgets copied to clipboard

Hi, I'm writing a simple Binance screener in JupyterLab. In the program I connect to Binance API using asynchronous approach. I create multiple tasks using Semaphore, then each of these tasks connects to API, get information from it and update one mutual dataframe (pumps). Like this (here and further I simplify and omit most of the code, if you need more I can share it):

async def pump_screener():
    #...
	semaphore = asyncio.Semaphore(20)
	tasks = [asyncio.create_task(process_klines(client, symbol, futures_ticker, semaphore)) for symbol in futures_filtered]
	await asyncio.gather(*tasks)

async def process_klines(client, symbol, futures_ticker, semaphore):
    async with semaphore:
        global pumps
        #...
        if (price_current > 1.04 * price_median or price_current < 0.96 * price_median) and volume_current > 4 * volume_median:
            pumps.loc[len(pumps)] = #...

            update_pumps()

Then I create two widgets (using ipywidgets), one to display the dataframe, and one to add additional column of checkboxes that correspond to a specific column of my dataframe. Like this:

def update_pumps():
    global pumps, pumps_widget, checkbox_box, combo_box
    checkboxes = []

    for row in range(len(pumps)):
        checkbox = widgets.Checkbox(value=bool(pumps.loc[row, 'Hidden']), indent=False)
        checkbox.observe(lambda change, index=row: on_checkbox_change(change, index), names='value')
        checkboxes.append(checkbox)

    if pumps_widget is None:
        pumps_widget = widgets.HTML(pumps.to_html())
    else:
        pumps_widget.value = pumps.to_html()

    if checkbox_box is None:
        checkbox_box = widgets.VBox(checkboxes)
    else:
        checkbox_box.children = checkboxes

    if combo_box is None:
        combo_box = widgets.HBox([pumps_widget, checkbox_box])
        display(combo_box)
    else:
        combo_box.children = [pumps_widget, checkbox_box]

def on_checkbox_change(change, index):
    pumps.loc[index, 'Hidden'] = change['new']

Then I create the final loop function:

async def main_loop():
    while True:
        try:
            await pump_screener()
            pause = 6
            for i in range(pause):
                await asyncio.sleep(1)
        except asyncio.CancelledError:
            print('Cancelled')
            break

And in the next cell I initiate the loop:

pumps_widget = checkbox_box = combo_box = iterator = gap = counter_dh = None
main_task = asyncio.ensure_future(main_loop())

To stop the asynchronous main_loop() I use the following command in the next cell:

main_task.cancel()

Here's how it looks like in the cell output:

The problem is when the number of dataframe rows exceeds 20 or so, and the dataframe updates happen relatively fast (it can happen with "spikes" of 2-3 updates per second) the overall process starts slowing down. I can see it by slowing down the tqdm progress bar updates and by my secondary simple progress counter. If I intentionally lower the condition threshold in the process_klines() for testing purposes, the dataframe starts growing relatively fast and the updates also happens noticeably faster. And after just about a minute of script execution it's getting really slow and in the next few seconds completely stops. The stop command doesn't work (the cell is not executed (*) despite asynchrony). And the only way to stop the script is to shut down the kernel.

The strange thing is even when I stop the script (before complete hanging), clear the cell output and the main dataframe, even then when I start the script again I face the same slow speed of updating when I stopped the script for the first time. And if I wait a little longer the script also hangs (on the second run with empty dataframe). It seems like some buffer in the kernel gets overflowed. But it also seems that my conditions do not correspond to this: I have enough RAM (32GB) and CPU (Intel Core i7-4790), the dataframe is relatively small (in the normal flow the slowing down starts at 20 or so rows and more), the updates speed is also relatively slow, it can be "spikes" with 3-4 updates per second maximum.

On the other side If I use pure display() to output the dataframe (but without interactive checkboxes) there is almost no slowing down during updates even with more than a hundred rows in the dataframe and relatively fast dataframe updates.

Also I recorded a video. Recording it I additionally noticed that the slowing down is happend only when the widgets update. If there are no updates, the speed returns to normal (or close to that). Here's the video:

What could be the cause for this slowing down and how can I avoid it? Perhaps I need some additional options for widgets or shoud I restructure some methods.