tutor
tutor copied to clipboard
overhangio
Tutor codebase Python 3.12 upgrade
- [ ] Ensure tutor codebase is compatible with Python 3.12
- [ ] Update Python to 3.12 in Dockerfile for nightly and ensure the image builds as expected. The changes would be merged to nightly. This task would rely on upgrade of edx-platform to Python 3.12
See https://github.com/overhangio/tutor/issues/1004 for the context.
CI checks, docs fixup -- https://github.com/overhangio/tutor/pull/1008
Attempted building Open edX docker file on Python 3.12. I had to make the following changes /tutor/templates/build/openedx/Dockerfile
L27-L30 (Python and Pyenv)
ARG PYTHON_VERSION=3.12.2
ENV PYENV_ROOT /opt/pyenv
RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.36 --depth 1
L96 - L102 (setuptools and pip update)
RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
pip install \
# https://pypi.org/project/setuptools/
# https://pypi.org/project/pip/
# https://pypi.org/project/wheel/
setuptools==69.1.1 pip==24.0 wheel==0.41.2
However, the build failed when installing Open edX base requirements with the error:
Downloading numpy-1.22.4.zip (11.5 MB)
#40 115.6 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 1.6 MB/s eta 0:00:00
#40 116.3 Installing build dependencies: started
#40 124.2 Installing build dependencies: finished with status 'done'
#40 124.2 Getting requirements to build wheel: started
#40 124.4 Getting requirements to build wheel: finished with status 'error'
#40 124.4 error: subprocess-exited-with-error
#40 124.4
#40 124.4 × Getting requirements to build wheel did not run successfully.
#40 124.4 │ exit code: 1
#40 124.4 ╰─> [33 lines of output]
#40 124.4 Traceback (most recent call last):
#40 124.4 File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#40 124.4 main()
#40 124.4 File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#40 124.4 json_out['return_val'] = hook(**hook_input['kwargs'])
#40 124.4 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4 File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
#40 124.4 backend = _build_backend()
#40 124.4 ^^^^^^^^^^^^^^^^
#40 124.4 File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
#40 124.4 obj = import_module(mod_path)
#40 124.4 ^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4 File "/opt/pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py", line 90, in import_module
#40 124.4 return _bootstrap._gcd_import(name[level:], package, level)
#40 124.4 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4 File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#40 124.4 File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#40 124.4 File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
#40 124.4 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#40 124.4 File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#40 124.4 File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#40 124.4 File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#40 124.4 File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#40 124.4 File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#40 124.4 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#40 124.4 File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
#40 124.4 import setuptools.version
#40 124.4 File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
#40 124.4 import pkg_resources
#40 124.4 File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
#40 124.4 register_finder(pkgutil.ImpImporter, find_on_path)
#40 124.4 ^^^^^^^^^^^^^^^^^^^
#40 124.4 AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
#40 124.4 [end of output]
At first, I thought this was due to setuptools version. It is sort-of due to setuptools but as an indirect dependency of numpy. The numpy version in quince is 1.22.4. Numpy version less than 1.26.0 do not work out of box with Python 3.12 due to pinned setuptools. This has been fixed by numpy as part of https://github.com/numpy/numpy/issues/23808. However, because the requirements in edx-platform base.txt are pinned, I suspect the docker image build will fail until the underlying code in platform is updated to 3.12.
- https://github.com/pypa/setuptools/issues/3935
- https://github.com/numpy/numpy/issues/23808
Adding some links for reference (When upgrading to slim-bookworm image)
- https://levelup.gitconnected.com/alpine-slim-stretch-bookworm-bullseye-buster-jessie-which-docker-image-should-i-choose-500f8c15c8cf
- ☝🏽 This article explains the difference between various Python image types beautifully.
- https://web.archive.org/web/20230323021946/http://jaredmarkell.com/docker-and-locales/
- https://stackoverflow.com/questions/28405902/how-to-set-the-locale-inside-a-debian-ubuntu-docker-container/28406007#28406007
- https://askubuntu.com/questions/1043900/package-libmysqlclient-dev-has-no-installation-candidate
- https://stackoverflow.com/questions/57048428/e-package-mysql-client-has-no-installation-candidate-in-php-fpm-image-build-u
- https://wiki.debian.org/DebianReleases
The upstream has a PR to upgrade edx-platform to Python 3.11 (https://github.com/openedx/edx-platform/pull/34374). I have created a draft PR to upgrade Dockerfile to use Python 3.11 by upgrading pyenv https://github.com/overhangio/tutor/pull/1056.
Régis had suggested using bookworm image on tutor-discovery (https://github.com/overhangio/tutor-discovery/pull/67#discussion_r1533458049). In tutor-notes, the slim-bookworm image has been working as expected (https://github.com/overhangio/tutor-notes/pull/36/files). For tutor, I am testing python:3.11-slim-bookworm but I don't expect it to be a blocker in releasing 3.11 changes in Tutor.
FYI, I've just merged the edx-platform Python 3.11 upgrade PR so you should be all set to update the edx-platform Tutor images.
Closing this issue. The slim-bookworm update will be handled in a follow-up PR.