Support background callbacks

[BeastyBlacksmith]

Would be nice if we could support background callbacks on the julia side also.

Related to https://github.com/plotly/Dash.jl/issues/110

[etpinard]

Current dash docs articles on background callbacks

• https://dash.plotly.com/background-callbacks
• https://dash.plotly.com/background-callback-caching

History lesson

please let me know if I'm missing anything :smile:

Background callbacks in python dash were first called "long" callback. They were added via the @long_callback decorator in https://github.com/plotly/dash/pull/1702, released as part of python dash v2.0.0 on 2021-08-03.

Long callbacks were then improved and effectively renamed "background" callbacks in https://github.com/plotly/dash/pull/2039 and https://github.com/plotly/dash/pull/2116, released in python dash v2.6.0 on 2022-07-14. This PR also refactored most of the functionality: the traditional callback function (normally used as a @callback decorator) gets a background=True keyword argument. The @long_callback decorator still exist to this day, but now is just a wrapper around callback(..., background=True).

Subsequent fixes and tweaks to background callbacks were made in:

• https://github.com/plotly/dash/pull/2226
• https://github.com/plotly/dash/pull/2415
• https://github.com/plotly/dash/pull/2473
• https://github.com/plotly/dash/pull/2599
• https://github.com/plotly/dash/pull/2625

High-level design

The python dash callback function has many keyword arguments to configure the "background" behaviour.

If background=True, the extra callback registration logic is here and here. In brief, using a cache manager, we (1) build the cache key, (2) terminate old job if any, (3) call the callback function if cache miss. There's also logic for cancelling and an optional progress bar. This logic will have to be rewritten in Julia!

The supported cache managers are: Celery (paired with Redis) and Diskcache. That is, the python dash cache managers are wrappers around established python caching / queuing libraries. We should survey caching package in the Julia ecosystem that would work well for this use case. After a quick search, Dagger.jl look like a promising Celery-like package.

The extra clientside logic mostly is here and here (refer to .ts file modifications in https://github.com/plotly/dash/pull/2039 for the details) . That part should just work :smile: but we'll need to make sure we send the appropriate JSON payload.

Dash.jl API

see base python example

import time
import os
from dash import Dash, DiskcacheManager, Input, Output, html, callback

import diskcache
cache = diskcache.Cache("./cache")
background_callback_manager = DiskcacheManager(cache)

app = Dash(__name__)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
    ]
)

@callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    background=True,
    manager=background_callback_manager,
)
def update_clicks(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run(debug=True)

would translate to

see Dash.jl translation

using Dash
import SomeCachingPkg

app = dash()
cache = SomeCachingPkg.Cache("./cache")
manager = DiskcacheManager(cache)

app.layout = html_div() do
  html_div() do
    html_p("Button not clicked"; id="paragraph_id")
  end,
  html_button("Run Job!"; id="button_id")
end

callback!(app, 
          Output("paragraph_id", "children"),
          Input("button_id", "n_clicks");
          background=true,
          manager) do n_clicks
  sleep(2)
  return "Clicked $n_clicks times"       
end

run_server(app)