Notebook as scripts

[hamelsmu]

From @seeM

[seeM]

Here's a quick implementation that I've started using:

# nbscript.py
from fastcore.script import *
from nbprocess.export import nb_export
from nbprocess.processors import _hide_dirs
from pathlib import Path
from runpy import run_path
from tempfile import TemporaryDirectory

def keep_code(cell):
    "Remove cells that aren't exported or hidden (opposite of `rm_export`)"
    if not (cell.directives_ and (cell.directives_.keys() & _hide_dirs) and cell.cell_type == 'code'): del(cell['source'])

@call_parse(nested=True)
def nbscript(nb_path:str, # Notebook path to execute 
             script_name:str=None): # Name of the script used in the help message (default: `nb_path`  with special chars replaced by `_`)
    "Run notebooks as scripts"
    nb_path = Path(nb_path)
    script_name = script_name or nb_path.stem.replace('-','_') # TODO: should replace all special chars
    with TemporaryDirectory() as td:
        lib_path = Path(td)/script_name
        nb_export(nb_path, lib_path)
        run_path(str(lib_path), run_name='__main__')

[seeM]

Since this is such a thin wrapper, it might be move convenient if this could be achieved by piping nbprocess_export to python, e.g:

nbprocess_export —stdout script.ipynb | python -

(The --stdout option doesn't exist yet.)

[hamelsmu]

That seems very doable!

[dleen]

I was also thinking about this for a bit and came up with this which involves parametrizing and scripts:

Have a cell in your notebook that essentially is a docment:

#|params
a:int=1 # This is parameter a
b:str="foo" # This is another parameter...
c:int=my_func() # Call a function to get the default value?

You can have a markdown cell before the params which acts as the helpstring for the script e.g.

"This is the rest of the documentation for the script?"

As usual a cell after the #|params cell is injected with the values given by the user.

This notebook could be run via a new function like:

def exec_nb_params(
    ...
    *args
    **kwargs
)

where args and kwargs are the positional and optional arguments. The injection with inject_idx etc is taken care of by this function .

Because we are using docments and essentially the call_parse mechanism used elsewhere in fastai then we can get a CLI script

exec_nb_params my_nb.ipynb --help

which would print

This is the rest of the documentation for the script

[jph00]

The cell could just have the docstring in it directly:

#|params
a:int=1 # This is parameter a
b:str="foo" # This is another parameter...
c:int=my_func() # Call a function to get the default value?
"This is the rest of the documentation for the script?"

Just to confirm my understanding - the purpose of this is to get a help message (and param error checking) for the "script"? And also to have any values passed at the command line injected into the appropriate cell?

Perhaps you could also specific the cell index of the place where the params will be injected? e.g. to inject into cell 2:

#|params 2

[dleen]

Just to confirm my understanding - the purpose of this is to get a help message (and param error checking) for the "script"? And also to have any values passed at the command line injected into the appropriate cell?

Correct

Perhaps you could also specific the cell index of the place where the params will be injected? e.g. to inject into cell 2:

Makes sense

Mostly just exchanging ideas here, haven't implemented anything yet

[MichaelJFishmanBA]

I was just looking for a tool to run a notebook with parameters and save the resulting notebook. Papermill does this, but its parameters rely on tagging Jupyter cells, which I can't do easily in VSCode.

[seeM]

I don't think it would be too hard to get a working prototype of this with execnb. If anyone would like to take a shot, I'm happy to share pointers. I unfortunately won't be able to get around to it myself just yet