papermill

papermill copied to clipboard

Currently, the execute_notebook function assumes a path to an input and output notebook. Sometimes, I'd like to manipulate the notebook that's already in memory, or output a notebook that's in-memory in order to do other stuff.

It seems like this would be pretty straightforward to do (e.g. if input_path were an instance of NotebookNode, then skip the read input line, and if output_path were None, then return the NotebookNode object instead of writing it to disk here.

Is this something folks have considered before? I'd be happy to make a PR if it is a simple as it seems to be...

Mostly it hadn't been done because people usually used nbconvert for in-memory execution in the past. But I'm not opposed to the idea of making it modular.

Another option would be to make a new function for executing in-memory to avoid overloading too heavily on CLI called function. Then the CLI called execute_notebook could call the shared code for that and the in-memory executor method (execute_notebook_in_mem?).

Good point re: nbconvert - perhaps what we need is a high-level function there, as opposed to in papermill? I believe we also discussed moving the execution stuff into a separate module, so maybe that's where such a function would exist...

Yeah that probably makes the most sense for general execution. Papermill will basically be applying the parameter injection and sourcing/sinking above that library.

It would be great if execute_notebook could take a NotebookNode object directly. I like to use jupytext and not track ipynb - I then use jupytext.read to get a NotebookNode. Right now I have to write an intermediate ipynb to use execute_notebook, so I second @choldgraf's proposal.

Hi, I found this really interesting as I wanted to execute Notebooks and handle them in memory, rather than with files. Have you consider this option at the end? I see this conversation has been opened since 2019.