LabelBot continuous training is failing - notebook fails to run

[jlewi]

Notebook is failing to run. Its choking trying to install papermill and some other dependencies

+ pip install --upgrade --user papermill jupyter-client==6.1.2
Collecting papermill
  Downloading papermill-2.2.0-py3-none-any.whl (35 kB)
Collecting jupyter-client==6.1.2
  Downloading jupyter_client-6.1.2-py3-none-any.whl (106 kB)
Collecting nbclient>=0.2.0
  Downloading nbclient-0.5.1-py3-none-any.whl (65 kB)
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-17xnjq7h/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
       cwd: None
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.6/dist-packages/pip/__main__.py", line 16, in <module>
      from pip._internal.cli.main import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/main.py", line 5, in <module>
      import locale
    File "/usr/lib/python3.6/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------

Looks like we aren't pinning papermill so I suspect papermill upgraded

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/jupyter	1.00
kind/bug	0.89

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[jlewi]

This is coming from here: https://github.com/kubeflow/code-intelligence/blob/428406b8fc052d671b2e621f1602ea84082e7215/tekton/tasks/run-notebook-task.yaml#L47

[jlewi]

image is gcr.io/kubeflow-images-public/tensorflow-1.15.2-notebook-cpu@sha256:1aac550b0d76c23d6400f9426542fe2948203bfa95cbd61fee23d8df674b404a

I can repo the bug bu running inside that docker image

pip install --upgrade --user papermill
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting papermill
  Downloading papermill-2.2.0-py3-none-any.whl (35 kB)
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 2.7 MB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yzpaqkdf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
       cwd: None
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.6/dist-packages/pip/__main__.py", line 16, in <module>
      from pip._internal.cli.main import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/main.py", line 5, in <module>
      import locale
    File "/usr/lib/python3.6/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yzpaqkdf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
WARNING: You are using pip version 20.0.2; however, version 20.2.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

[jlewi]

python --version
Python 3.6.9

[jlewi]

python -m pip install --user --upgrade pip
Installing collected packages: pip
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/jovyan/.local'
Check the permissions.

[jlewi]

Installing papermill works on tensorflow/tensorflow:2.1.2-jupyter even thought is python 3.6.9