rules_python

rules_python copied to clipboard

I'd like to run ipython or jupyter notebook and be able to load my bazel libraries. How can I do that?

Unfortunately, this is about the hardest use case possible, assuming you need things with extension modules (pure Python is fine). To set expectations, this won't be directly supported or worked on for quite a while.

I was able to get it working by creating a notebook.pywith this content:

from notebook import notebookapp
notebookapp.main()

And BUILD:

load("@pip_packages//:requirements.bzl", "all_requirements")
 
py_binary(
  name = "notebook",
  srcs = ["notebook.py"],
  deps = all_requirements,
)

I had to clean up my workspace then rebuild in a python3 virtualenv (as in https://github.com/bazelbuild/rules_python/issues/64)

I quickly put together a simple, kinda hacky, rendition of semi-generalized rules: https://github.com/EricCousineau-TRI/repro/tree/52e484e3d68e29bfb8a8e66362c60838161f0dfa/bazel/jupyter_integration

This allows interactively editing the notebook, using some .runfiles hackery (which I guess may break in upcoming Bazel version) via the ./run script (which perhaps could be replaced via bazel run --direct?). You can also run the notebook via bazel run and bazel test.

Granted, this uses the system jupyter; it focuses on the content of notebooks, not necessarily using a Bazel version of IPython, but I'm sure that could be easy to patch in.

I did something similar to @fortuna

notebooks/jupyter.py

# -*- coding: utf-8 -*-
import re
import sys

from notebook.notebookapp import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

BUILD

load("@notebooks//:requirements.bzl", "requirement")

py_binary(
    name = "jupyter",
    srcs = ["jupyter.py"],
    deps = [
        "//yourlibrary",
        "//yourotherlibrary:foo",
        requirement("notebook"),
    ],
)

requirements.txt

notebook>=5.6.0

Then just bazel run notebooks:juypter and the browser should open and you can create a new notebook with access to all your dep libraries.

(I also created a small repo with a running example, https://github.com/michael-quinlan/jupyter-bazel)

@duggelz as a year has past since your last comment, what is your opinion today on how far out this is on the roadmap?

Similar to the two jupyter solutions, I created a reusable macro for the iPython shell

Macro defined at bazel_helpers/ipython.bzl

load("@ipython_deps//:requirements.bzl", "all_requirements")

def ipython(name, deps, **kwargs):
    native.py_binary(
        name = name,
        srcs = ["//bazel_helpers:shell.py"],
        deps = deps + all_requirements,
        python_version = "PY3",
    )

Entrypoint at bazel_helpers/shell.py

from IPython import start_ipython

if __name__ == '__main__':
    start_ipython()

Since I put this in a separate package from my code, added a bazel_helpers/BUILD:

exports_files(["shell.py"])

bazel_helpers/requirements.txt (using --hash pinning):

ipython==7.8.0 \
  --hash=sha256:c4ab005921641e40a68e405e286e7a1fcc464497e14d81b6914b4fd95e5dee9b
# and other dependencies

In the BUILD for the package I want the iPython shell (mypkg/BUILD):

load("//bazel_helpers:ipython.bzl", "ipython")

py_library(
    name = "mylib",
    srcs = glob([
        "*.py",
    ]),
    deps = [...],
)

ipython(
    name = "shell",
    deps = [
        ":mylib",
    ],
)

Can run via bazel run //mypkg:shell

Nice!

To follow up on my prior post, we have put the Jupyter Bazel stuff into Drake master: https://github.com/RobotLocomotion/drake/blob/master/tools/jupyter/README.md (permalink) This permits unittesting the notebooks (ensuring code works and that nominal usage of GUI elements doesn't cause errors, etc.).

Granted, this may be noisy for other people, and the current solution in Drake only uses system-provided Jupyter and does not try to use it through Bazel (like John's solution above).

Thanks guys for the starting point. A more recent attempt of mine: https://github.com/dayfine/xlab/pull/6

Basically my setup allows me to use Google Colab with the local libraries (proto, etc) I build with Bazel

@dayfine I think this is a good setup. I tried the same at work and ran well, but we strictly avoid pulling platform-dependent packages and notebook uses appnope==0.1.0 which is OSX-only.

I believe this issue should be closed. There are numerous example approaches and macros mentioned that show how to implement this.

I used @fortuna's method. In addition, add -- --notebook-dir=/path/to/working/directory/ to your bazel run command to start the notebook server in a useful directory, not deep in the runfiles

For jupyter notebook 7, one needs to replace from notebook.notebookapp import main with from notebook.app import main.

Unfortunately I'm then getting errors of the form

[W 2024-02-08 11:53:01.777 JupyterNotebookApp] Missing or misshapen translation settings schema:
    HTTP 404: Not Found (Schema not found: [...]/share/jupyter/lab/schemas/@jupyterlab/translation-extension/plugin.json)

when launching the notebook.

Has anyone successfully upgraded to notebook 7?