dash-extensions icon indicating copy to clipboard operation
dash-extensions copied to clipboard
Published 4 months ago • verified publisher icon emilhe

Serverside + Backgroundmanager + Windows == Error

Open noxthot opened this issue 1 year ago • 4 comments

With recent versions, the example from https://www.dash-extensions.com/transforms/serverside_output_transform#a-serversideoutputtransform works nicely on both, Windows and Ubuntu.

A slight modification introducing a background-manager with Diskcache plus Progress bar breaks the app on Windows, while it is still running on Ubuntu.

The code which is not working (also check diff before):

import time
import plotly.express as px
import diskcache

from dash import DiskcacheManager
from dash_extensions.enrich import DashProxy, Output, Input, State, Serverside, html, dcc, \
    ServersideOutputTransform

cache = diskcache.Cache("./cache")

# Background callbacks require a cache manager
background_callback_manager = DiskcacheManager(cache)

app = DashProxy(transforms=[ServersideOutputTransform()], background_callback_manager=background_callback_manager)
app.layout = html.Div(
    [
        html.Button("Query data", id="btn"),
        dcc.Dropdown(id="dd"),
        html.P("0%", id="progress-component"),
        dcc.Graph(id="graph"),
        dcc.Store(id="store"),
    ]
)

@app.callback(
    Output("store", "data"),
    Input("btn", "n_clicks"),
    background=True,
    progress=[
        Output("progress-component", "children"),
    ],
    prevent_initial_call=True
)
def query_data(set_progress, _):
    total = 3

    for i in range(total):
        # Simulating a process step
        time.sleep(1)  # emulate slow database operation

        set_progress([(i + 1) / total])  # Update progress using set_progress
    return Serverside(px.data.gapminder())  # no JSON serialization here

@app.callback(Output("dd", "options"),  Output("dd", "value"), Input("store", "data"), prevent_initial_call=True)
def update_dd(df):
    options = [{"label": column, "value": column} for column in df["year"]]   # no JSON de-serialization here
    return options, options[0]['value']

@app.callback(Output("graph", "figure"), [Input("dd", "value"), State("store", "data")], prevent_initial_call=True)
def update_graph(value, df):
    df = df.query("year == {}".format(value))  # no JSON de-serialization here
    return px.sunburst(df, path=["continent", "country"], values="pop", color="lifeExp", hover_data=["iso_alpha"])

if __name__ == "__main__":
    app.run_server()

Installed and updated packages (for completeness; note, that this is from a "larger" project): https://pastebin.com/56Nitkmk

Stacktrace (Microsoft Windows; Version 23H2 (Build 22631.3737); this is a Windows 11 version)

PS C:\Users\anonymized\anonymizedproject> pdm run .\drafts\tmp\test_serverside.py
Dash is running on http://127.0.0.1:8050/
 
 * Serving Flask app 'test_serverside'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8050
Press CTRL+C to quit
[2024-06-21 12:17:48,303] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\dash.py", line 1373, in dispatch
    ctx.run(
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\_callback.py", line 389, in add_context
    job = callback_manager.call_job_fn(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\long_callback\managers\diskcache_manager.py", line 126, in call_job_fn
    proc.start()
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\popen_spawn_win32.py", line 95, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\reduction.py", line 63, in dump
    ForkingPickler(file, protocol, *args, **kwds).dump(obj)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 420, in dump
    StockPickler.dump(self, obj)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 481, in dump
    self.save(obj)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 597, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 711, in save_reduce
    save(state)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 966, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 990, in _batch_setitems
    save(v)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1985, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1117, in _save_with_postproc
    pickler.save_reduce(*reduction)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 686, in save_reduce
    save(args)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 881, in save_tuple
    save(element)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1985, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1112, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 990, in _batch_setitems
    save(v)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 572, in save
    rv = reduce(self.proto)
         ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle '_contextvars.ContextVar' object
127.0.0.1 - - [21/Jun/2024 12:17:48] "POST /_dash-update-component HTTP/1.1" 500 -
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 289, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 444, in load
    obj = StockUnpickler.load(self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input

Please note that using DiskCache Manager, background manager and progress bar without ServerSide works without problem.

Any idea what is going on? Thanks!

noxthot avatar Jun 21 '24 10:06 noxthot

Update: 1.0.17 shows the same behaviour.

noxthot avatar Jul 02 '24 06:07 noxthot

I would suggest to use WSL, if you are on Windows. I believe that would solve the issue?

emilhe avatar Jul 02 '24 07:07 emilhe

I suppose this should work and this is actually our plan B, but I hoped to see the same behaviour with both OS.

I am personally using Ubuntu; this error occurs on another dev's Windows machine. Will keep you updated.

noxthot avatar Jul 02 '24 09:07 noxthot

fyi: works when using WSL

noxthot avatar Jul 09 '24 12:07 noxthot