Need to enhance notebook_util.py to fit papermill

[jinchihe]

/kind bug

What steps did you take and what happened:

While I execute the notebook using papermill, the below problem happened.

(python36) [root@jinchi1 testing]# python execute_notebook.py
/tmp/tmpu433eq_r
Executing:  48%|████████████████████████████████████████████▎                                                | 20/42 [00:39<00:43,  1.99s/cell]
Traceback (most recent call last):
  File "execute_notebook.py", line 38, in <module>
    run_notebook_test(NOTEBOOK_ABS_PATH, EXPECTED_MGS)
  File "execute_notebook.py", line 18, in run_notebook_test
    output_path = execute_notebook(notebook_path, parameters=parameters)
  File "execute_notebook.py", line 14, in execute_notebook
    parameters=parameters)
  File "/opt/kubeflow/python3/python36/lib/python3.6/site-packages/papermill/execute.py", line 104, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/opt/kubeflow/python3/python36/lib/python3.6/site-packages/papermill/execute.py", line 188, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [11]":
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-ac262374af2b> in <module>
      1 from fairing.builders import cluster
----> 2 preprocessor = ConvertNotebookPreprocessorWithFire("ModelServe")
      3
      4 if not preprocessor.input_files:
      5     preprocessor.input_files = set()


▽
/opt/kubeflow/python3/python36/lib/python3.6/site-packages/fairing/preprocessors/converted_notebook.py in __init__(self, class_name, notebook_file, notebook_preprocessor, executable, command, path_prefix, output_map, overwrite)
    101             command=command,

▽
import json
    102             path_prefix=path_prefix,
--> 103             output_map=output_map)
    104
    105         self.class_name = class_name

/opt/kubeflow/python3/python36/lib/python3.6/site-packages/fairing/preprocessors/converted_notebook.py in __init__(self, notebook_file, notebook_preprocessor, executable, command, path_prefix, output_map, overwrite)
     62
     63         if notebook_file is None and notebook_util.is_in_notebook():
---> 64             notebook_file = notebook_util.get_notebook_name()
     65
     66         self.notebook_file = notebook_file

/opt/kubeflow/python3/python36/lib/python3.6/site-packages/fairing/notebook/notebook_util.py in get_notebook_name()
     17     kernel_id = re.search('kernel-(.*).json',
---> 18                           ipykernel.connect.get_connection_file()).group(1)
     19     servers = list_running_servers()
     20     for ss in servers:

AttributeError: 'NoneType' object has no attribute 'group'

In fact, the connection file name is /tmp/tmpcn0jj17i.json, the re.search('kernel-(.*).json', cannot match that.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.94. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[xauthulei]

@jinchihe ， I want to have a try on this issue, Could you share me more about your env? how could I reproduce your case ? or what 's the notes here ? Thanks

[jinchihe]

wow! That's great, @xauthulei

I reproduced the issue while running the below notebook using papermill. https://github.com/kubeflow/examples/blob/master/xgboost_synthetic/build-train-deploy.ipynb

[xauthulei]

@jinchihe , In our common practice, when we run ConvertNotebookPreprocessorWithFire function without defining the notebook_file value, but we could get its value by its current kernal ID (from its connection file name)compared with running jupyter server API to search the notebook file path. However, papermill is differnet and it doesn't start such notebook sever(even if we could get the connection file name liking /tmp/tmpcn*.json), so it is not easy for us to detect its notebook file path.

So, in my points, we have these options when using the papermill:

1. Specify the notebook_file in ConvertNotebookPreprocessorWithFire function firstly, if not, we could alert message to define it .
2. Parameterize notebook_file's value , then assign a value to this parameter when executing papermill.

[jinchihe]

I think we can Specify the notebook_file in ConvertNotebookPreprocessorWithFire to work around this now :-)

[jtfogarty]

/area jupyter /priority p2