Cubed Dashboard

[tomwhite]

Suggested by @alimanfoo at the Zarr Summit. It would be useful to see the progress of a computation in a web dashboard (like Dask), and would make for great demos.

I would start with a dynamic version of the timeline plots (e.g.):

I think max actual memory usage per task would be useful too:

Is Bokeh a good choice for this, or is there another recommendation?

[TomNicholas]

Another idea from that discussion was to animate the Plan itself.

• The arrays themselves could be squares like in the image above, or potentially even cubes like in the xarray html repr
  • Though then it's not clear what to do for >3D arrays
• Each array in the plan could turn green when fully complete
• Progress towards computing each array could be indicated in a few ways:
  • The vector linking the two array nodes could fill up with green like a progress bar
  • The target array could incrementally become more and more green
    • This could be done by splitting the image for that array into n_chunks number of tiny squares, each of which turns green once that task is complete
    • Then you want a big checkmark once the final one is done
  • Or both
• Mousing over the arrays could show more information about them
  • Maybe that's the place to put the multi-dimensional HTML repr

I think each of the 3 ideas above could be different pages of a dashboard, like how dask has many pages. We might also imagine executors able to define their own additional pages. (e.g. a dask/coiled executor could contain a link/embed of the corresponding dask dashboard.)

Is Bokeh a good choice for this, or is there another recommendation?

I have no idea but I do think the dashboard should be built in a way that would be robust to an absurd number of tasks. e.g. above X total tasks it changes the way it plots to not even try to render anything which scales with the number of tasks.

[alimanfoo]

I've not used it myself, but Grafana might be an option to get something up and running quickly, apparently it has built-in views for trace data and you can push via web sockets and Grafana Live.

[tomwhite]

Another idea from that discussion was to animate the Plan itself.

Thanks for adding that. The plan is created using GraphViz, so rendering it using another library that can do custom animation is a challenge! (Even finding such a library is a challenge...)

The target array could incrementally become more and more green

• This could be done by splitting the image for that array into n_chunks number of tiny squares, each of which turns green once that task is complete

I originally thought that the arrays in the plan would be represented like in the xarray html repr, but that has problems with 3D due to occlusion, and >3D (as you said). So just having a 2D grid of squares that acts as an abstract progress bar would be better.

[tomwhite]

I re-wrote the plan visualization using a combination of Cytoscape.js and Dash, and then added a very simple animation that changes the fill colour of the operations from white to purple as they progress:

https://github.com/user-attachments/assets/9e8173cb-2a9d-4a19-8fdd-2b7c6a13da97

This is just a proof of concept, but it should be possible to add better animations (maybe using Cytoscape.js's UI extensions) as discussed above.

[TomNicholas]

Here's the version from today

[alimanfoo]

This is very cool!