Update a cell's outputs property with the widget manager

[flekschas]

I am developing a custom widget for Jupyter Lab/Notebook and I am trying to find out if it's possible to updates the outputs property of a cells such that it's saved in the .ipynb file.

For context, the widget I am developing is an data visualization that supports interactive view changes via panning and zooming. For accessibility reasons it'd be great to be awesome if I could add a screenshot as a base64 encode string.

I know that this works in general via the image/png property. And from within Python I can realize this by implementing a custom _repr_mimebundle_ method. The problem is that the rendering is happening in the JavaScript kernel asynchronously and that the view might change as the user pans and zooms. Hence, I would like to be able to update the outputs object from within JavaScript.

I noticed that the JavaScript widget has access to the WidgetManager via this.model.widget_manager and I was able to actually retrieve and update a cell's outputs representation using the WidgetManager as follows.

Very hacky example of how I attempt to change a cell's outputs property:

demoFunction: function demoFunction() {
  // First I am retrieving the cell of interest (Apologies, I know this is super hacky...)
  let i = 0;
  let cell;
  this.model.widget_manager._context.model.cells._cellMap._map.forEach((c) => {
    // I know the cell of my interest is cell number 3
    if (3 === i++) {
      cell = c
    }
  });

  // Having the cell, I can change the `outputs` property as follows
  cell._outputs.list._array[0]._data.set('text/plain', 'test test');
  cell._outputs.list._array[0]._data.set('image/png', this.getCanvas().toDataURL());

  // I can confirm that the cells representation changes as expected
  console.log(cell._outputs.list._array[0]._data.set('image/png');
  // ==> data:image/png;base64,iVBORw0KGgo...
}

However, it seems that the change is only reflected in JavaScript and not synchronized with the Python kernel. Is there any way to sync the changes to a cells outputs property back to Python?

Beyond the special case of interactive data visualizations, this would be super useful for any kind of widget that uses JavaScript to render a data visualization as _repr_mimebundle_ is synchronous and, thus, by the time it's called the JavaScript rendering was not even triggered. One can update a cell via update_display() but that is not very useful either as it recreates the entire JS widget, which leads to a flicker. Besides that it also does not trigger the remove function of the previously instantiated JS widget, which leads to memory issues.

This ticket is a follow up on #2682