Function hooks when executing cells / notebooks?

[choldgraf]

I'm trying to find a way to achieve scrapbook-like functionality but without asking users to write new code. Instead, I'd like them to be able to tag cells with metadata (either tags like glue_varname, or other metadata) and to programatically embed extra outputs in the cell depending on that metadata.

I think to do this with nbclient, we could allow either a notebook-level or cell-level hook that could be run alongside the call to execute(). Basically, a function that runs in the context of the kernel that is run with the notebook. I'd have a preference for the ability to do this in each cell, but notebook-level could work too.

It seems like this could potentially be useful for other kinds of usecases too, so I wonder what folks think about supporting something like this. Alternatively, can anybody think of a way to support the usecase I described at the top?

[MSeal]

I think you could do this in papermill by extending https://github.com/nteract/papermill/blob/master/papermill/engines.py#L231-L285 and calling a post-execution hook. Course you'd need to run code in the kernel to support it properly. Maybe a hook that saves the final line of each cell as a scrap on the notebook?

def cell_complete(self, cell, cell_index=None, **kwargs):
    post_cell = """
    result = _
    try:
      import scrapbook as sb
    except ImportError:
      print("scrapbook not installed, no outcomes saved")
    else:
      try:
        if result:
          sb.glue(f"__result_{cell_index}", result)
      except sb.ScrapbookException:
        print("Could not save result of cell {cell_index}")
    """
    self.execute_cell(post_cell, -1)
    self.super(cell, cell_index, **kwargs)

I think we'd want to expose call-backs for cell-complete and other events to the engine in papermill so you can just register an engine that supported it.

[choldgraf]

yes! that is the kind of pattern that I had in mind. You think papermill would be a better home for this kind of functionality than nbclient?

Agreed that the nicer way to handle this for our use-case (since I am still not sure what the "right" UX is here) is to just make it possible to extend the back that papermill uses along w/ access to the kernel

[MSeal]

I think papermill is a more established place for adding execution hooks in managed execution patterns, yeah. Keeping nbclient simple and low level makes it easier to understand and less opinionated about how you execute notebook code.

Want me to transfer this issue to papermill to continue the discussion about making it easier to add these particular hook patterns?

[choldgraf]

happy to do so, thanks for the explanation