rocker-versioned2 icon indicating copy to clipboard operation
rocker-versioned2 copied to clipboard

Published 20 hours ago • verified publisher icon rocker-org

How to use with JupyterHub

Open ktaletsk opened this issue 2 years ago • 3 comments

I am looking at instructions on how to use this image with JupyterHub (+ KubeSpawner). I am able to run this image standalone on my machine and get to RStudio, I am able to launch it in Binder and get to RStudio, but not in JupyterHub. I added the image built from this repo to the list of profiles in KubeSpawner in jupyterhub-config.py

c.KubeSpawner.profile_list.extend(
    [
        {
            "display_name": "RStudio",
            "slug": "rstudio",
            "kubespawner_override": {
                "image": "ktaletsk/rstudio:latest",
                "url": "/rstudio",
            },
        },
    ]
)

When I launch this profile, container is successfully pulled and launched, however in browser JupyterHub redirects to a broken page image

This is the log from the K8s pod:

[I 17:25:46.340 NotebookApp] Writing notebook server cookie secret to /home/rstudio/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2022-08-12 17:25:47.173 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-08-12 17:25:47.173 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2022-08-12 17:25:47.183 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/dist-packages/jupyterlab
[I 2022-08-12 17:25:47.183 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 17:25:47.189 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 17:25:47.189 NotebookApp] Jupyter Notebook 6.4.12 is running at:
[I 17:25:47.189 NotebookApp] http://<K8s_pod_name>:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:47.189 NotebookApp]  or http://127.0.0.1:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:47.189 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 17:25:47.193 NotebookApp] No web browser found: could not locate runnable browser.
[C 17:25:47.193 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/rstudio/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://<pod_name>:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
     or http://127.0.0.1:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:49.745 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.630000ms
[I 17:25:49.976 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.620000ms
[W 17:25:50.112 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 43.370000ms referer=https://<deployment_url>/hub/spawn-pending/<jupyterhub_username>
[W 17:26:05.159 NotebookApp] 404 GET /user/<jupyterhub_username>/lab (192.168.21.10) 1.790000ms referer=None
[I 18:14:45.678 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.880000ms
[W 18:14:45.767 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 1.670000ms referer=None
[I 18:54:46.715 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.590000ms
[W 18:54:46.804 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 1.940000ms referer=None
[W 19:02:17.283 NotebookApp] 404 GET /user/<jupyterhub_username>/rsutdio (192.168.21.10) 1.710000ms referer=None

Furthermore, I can connect to my pod by running proxy and verify that RStudio web UI is accessible

kubectl port-forward pod/<pod_name> 8080:8888

image

I can conclude that either is something is wrong with the image or JupyterHub configuration. For my normal JupyterLab image, I use the following profile:

"kubespawner_override": {
        "cmd": ["jupyter-labhub"],
        "args": ["--collaborative"],
        "url": "/lab",
    },

which requires JupyterHub to be installed in the spawned image. I tried to install JupyterHub in this repo's image, but encountered errors with pip:

pip install log

Full log:

rstudio@5fb651f7f41c:~$ pip install jupyterhub
Collecting jupyterhub
Downloading jupyterhub-2.3.1-py3-none-any.whl (3.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 16.0 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.8.2)
Collecting SQLAlchemy>=1.1
Downloading SQLAlchemy-1.4.40-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 29.2 MB/s eta 0:00:00
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.28.0)
Collecting oauthlib>=3.0
Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 22.3 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (21.3)
Collecting jupyter-telemetry>=0.1.0
Downloading jupyter_telemetry-0.1.0-py3-none-any.whl (7.1 kB)
Collecting pamela
Downloading pamela-1.0.0-py2.py3-none-any.whl (5.8 kB)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.4)
Requirement already satisfied: traitlets>=4.3.2 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (5.3.0)
Requirement already satisfied: prometheus-client>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.14.1)
Requirement already satisfied: tornado>=5.1 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (6.1)
Collecting certipy>=0.1.2
Downloading certipy-0.1.3-py3-none-any.whl (22 kB)
Collecting async-generator>=1.9
Downloading async_generator-1.10-py3-none-any.whl (18 kB)
Collecting alembic>=1.4
Downloading alembic-1.8.1-py3-none-any.whl (209 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 kB 23.5 MB/s eta 0:00:00
Requirement already satisfied: jinja2>=2.11.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (3.1.2)
Collecting Mako
Downloading Mako-1.2.1-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.6/78.6 kB 9.4 MB/s eta 0:00:00
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (5.8.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (4.11.4)
Collecting pyopenssl
Downloading pyOpenSSL-22.0.0-py2.py3-none-any.whl (55 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 12.1 MB/s eta 0:00:00
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2>=2.11.0->jupyterhub) (2.1.1)
Collecting python-json-logger
Downloading python_json_logger-2.0.4-py3-none-any.whl (7.8 kB)
Requirement already satisfied: jsonschema in /usr/local/lib/python3.8/dist-packages (from jupyter-telemetry>=0.1.0->jupyterhub) (4.6.0)
Collecting ruamel.yaml
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 kB 18.1 MB/s eta 0:00:00
Collecting greenlet!=0.4.17
Downloading greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.6/156.6 kB 23.3 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->jupyterhub) (3.0.9)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil->jupyterhub) (1.16.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (2.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (2022.6.15)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->alembic>=1.4->jupyterhub) (3.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (21.4.0)
Collecting cryptography>=35.0
Downloading cryptography-37.0.4-cp36-abi3-manylinux_2_24_x86_64.whl (4.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 31.1 MB/s eta 0:00:00
Collecting ruamel.yaml.clib>=0.2.6
Downloading ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl (570 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.4/570.4 kB 32.5 MB/s eta 0:00:00
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.8/dist-packages (from cryptography>=35.0->pyopenssl->certipy>=0.1.2->jupyterhub) (1.15.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.12->cryptography>=35.0->pyopenssl->certipy>=0.1.2->jupyterhub) (2.21)
Installing collected packages: pamela, ruamel.yaml.clib, python-json-logger, oauthlib, Mako, greenlet, async-generator, SQLAlchemy, ruamel.yaml, cryptography, pyopenssl, jupyter-telemetry, alembic, certipy, jupyterhub
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/bin/mako-render'
Consider using the `--user` option or check the permissions.

--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 177, in emit
  self.console.print(renderable, overflow="ignore", crop=False, style=style)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1752, in print
  extend(render(renderable, render_options))
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1390, in render
  for render_output in iter_render:
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 134, in __rich_console__
  for line in lines:
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/segment.py", line 245, in split_lines
  for segment in segments:
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1368, in render
  renderable = rich_cast(renderable)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/protocol.py", line 36, in rich_cast
  renderable = cast_method()
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/self_outdated_check.py", line 130, in __rich__
  pip_cmd = get_best_invocation_for_this_pip()
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/entrypoints.py", line 58, in get_best_invocation_for_this_pip
  if found_executable and os.path.samefile(
File "/usr/lib/python3.8/genericpath.py", line 101, in samefile
  s2 = os.stat(f2)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip3.8'
Call stack:
File "/usr/local/bin/pip", line 8, in <module>
  sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/main.py", line 70, in main
  return command.main(cmd_args)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 101, in main
  return self._main(args)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 223, in _main
  self.handle_pip_version_check(options)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 148, in handle_pip_version_check
  pip_self_version_check(session, options)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/self_outdated_check.py", line 237, in pip_self_version_check
  logger.info("[present-rich] %s", upgrade_prompt)
File "/usr/lib/python3.8/logging/__init__.py", line 1446, in info
  self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log
  self.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle
  self.callHandlers(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
  hdlr.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
  self.emit(record)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 179, in emit
  self.handleError(record)
Message: '[present-rich] %s'
Arguments: (UpgradePrompt(old='22.1.2', new='22.2.2'),)

If anyone has experience running this image in JupyterHub and/or can provide a minimal working example of that, I would really appreciate it and will help to update the documentation on this repo to help others.

ktaletsk avatar Aug 12 '22 19:08 ktaletsk

@yuvipanda Do you know anything about this? Thanks.

eitsupi avatar Aug 13 '22 14:08 eitsupi

@eitsupi @yuvipanda any hints?

ktaletsk avatar Aug 29 '22 18:08 ktaletsk

@ktaletsk Sorry, but I don't use k8s or JupyterHub, so I have nothing to say about this. You might be more likely to get an answer if you ask this in the Jupyter community e.g. https://github.com/jupyterhub/jupyter-rsession-proxy

eitsupi avatar Aug 30 '22 10:08 eitsupi

I think the issue is routing. JupyterHub in my configuration sends user to /user/< jupyterhub_username >/rstudio, but the only path that works in container is /rstudio (or /lab for that matter). I wonder if there is a configuration in JupyterHub I can change to make it work.

Binder seem to have no issues with this URL scheme: user/rocker-org-binder-0g8a6gu4/rstudio/

ktaletsk avatar Sep 12 '22 15:09 ktaletsk