Importing local files from input notebook fails with execute_notebook

[fonnesbeck]

I have a set of plotting utility functions that I keep alongside a set of production notebooks, importing them via (for example):

from plotting import plot_heatmap

However, when I run this from a script in the same directory that I am using to iterate over input parameters, this import fails:

However, this import does not fail when run interactively from the notebook.

Running papermill 1.0.1 on Linux (via GCS) and Python 3.7.

[MSeal]

Check your whereis papermill. I am guessing your python paths aren't consistent and your kernel is running in a different environment. You can also add a print statement to list the python path and see what python the kernel is using.

[glass-ships]

Versions: 
anaconda3 1.7.2 - python3.7
papermill - 1.2.1 from /opt/anaconda3/lib/python3.7/site-packages/papermill/cli.py (3.7.3)

Can confirm this issue, at least in the context of a Docker container with the software in question appropriately installed.

Also confirmed works in a jupyter-lab notebook on the same image and host machine.

I checked my python and papermill paths as suggested, but they are consistent.

Any thoughts or suggested workaround? This issue is unfortunately roadblocking the testing I'm trying to setup for my collaboration.

[mpacer]

In the case where a module can't be found but is generally installed, it could be a couple of things. Most likely, depending on the name of your kernel the kernel that papermill finds may be different than the one you are trying to run it in.

Run jupyter kernelspec list to see which kernels you have installed.

In the case of a module in a local file, I would check whether papermill is running in the same directory as the local file. The kernel's Python path will include the local directory which should make it possible to find local modules.

[MSeal]

For the second printed error, it looks like the local variable inside your notebook is None when you expected to to have a value, which means it's likely unrelated to papermill, which is simply reporting the failure in the 5th cell of the notebook execution.

[glass-ships]

Thanks for your responses!

@MSeal to reiterate, this notebook performs as expected, without error, when being run in a jupyter notebook. The AttributeError only occurs when trying to papermill the notebook (as well as when trying to use my simplified notebook testing script.

@mpacer

$ jupyter kernelspec list
Available kernels: 
    root          /opt/root6.18/etc/notebook/kernels/root
    python3   /opt/anaconda3/share/jupyter/kernels/python3

I've attempted to run papermill in the directory containing the notebook, as well as the main directory of the package in question, but the error persists.

The successful in-jupyter-lab run is done in the python3 kernel as far as I can tell, although it's possible that the ROOT one is being referenced, since the cell containing import statements doesn't error out. There's also a line which imports functions from the package, so if it's not spitting out an error my assumption is paths are consistent.

I admit I'm at something of a loss here. Perhaps someone can suggest further ways I can try to help identify the issue?

[MSeal]

Right, but the error message is still isolated to inside your code. That looks like the right kernel provided your notebook is using a python3 kernel in it's metadata. Given that it's more likely your environment is different causing your python code to execute differently inside the container / papermill. Perhaps you rely on a particular environment variable or a file path which isn't available in the execution context, leading to executing a code path in your notebook which has a bug? I'd trace back why ev is set to None until you find a difference to figure out what's missing.

[glass-ships]

Ahh I see what you mean, that makes sense! I had a closer look at the parameter I was passing and there was indeed a slight typo.

Thanks again, I greatly appreciate your help!

[GPla]

I encountered the same problem when running pm.execute_notebook and needed to explicitly specify the correct kernel. As stated by @mpacer the available kernels can be listed with jupyter kernelspec list. Then the correct kernel can be passed pm.execute_notebook(..., kernel_name='kernel name').