docker-library
Python images broke - can we get a changelog?
We recently upgraded our base images from, for example,
python:3.11-bookworm@sha256:f7543d9969bdc112dd9819ca642e14433fdacfe857f170f6b803392fc7e451ad
to
python:3.11-bookworm@sha256:157a371e60389919fe4a72dff71ce86eaa5234f59114c23b0b346d0d02c74d39
and that broke all our Python images. I'm trying to understand what exactly changed (and then how to fix it). I saw in the issues that setuptools and wheel were drop as well as something with ensurepip?
What I've noticed in the image itself so far is that the name of the python binary seems to have changed - my symlink to /usr/local/bin/python3.11 is dead, there's only /usr/local/bin/python.
I'm getting build errors for an already-deployed, successfully building commitd, since yesterday.
Base image is pinned to :3.12-slim.
Googling the error mostly pop setuptools issues, but nothing about that on my Dockerfile, AFAIK (at least no change on that). So, @black-snow's guess looks reasonable to me (within my knowledge).
The error pops when building Django staticfiles, no code nor deps changed:
#12 [stage-0 7/9] RUN DEFAULT_FROM_EMAIL=dummy FEEDBACK_EMAIL=dummy EMAIL_URL=consolemail:// DATABASE_URL=sqlite://:memory: SECRET_KEY=dummy STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage python manage.py collectstatic --no-input --verbosity 2
#12 0.426 Traceback (most recent call last):
#12 0.426 File "/app/manage.py", line 15, in <module>
#12 0.426 execute_from_command_line(sys.argv)
#12 0.426 File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
#12 0.427 utility.execute()
#12 0.427 File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
#12 0.427 django.setup()
#12 0.427 File "/usr/local/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
#12 0.427 apps.populate(settings.INSTALLED_APPS)
#12 0.427 File "/usr/local/lib/python3.12/site-packages/django/apps/registry.py", line 91, in populate
#12 0.427 app_config = AppConfig.create(entry)
#12 0.427 ^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427 File "/usr/local/lib/python3.12/site-packages/django/apps/config.py", line 193, in create
#12 0.427 import_module(entry)
#12 0.427 File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
#12 0.427 return _bootstrap._gcd_import(name[level:], package, level)
#12 0.427 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427 File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#12 0.427 File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#12 0.427 File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#12 0.427 File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#12 0.427 File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#12 0.427 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#12 0.427 File "/usr/local/lib/python3.12/site-packages/rest_framework_simplejwt/__init__.py", line 1, in <module>
#12 0.427 from pkg_resources import DistributionNotFound, get_distribution
#12 0.427 ModuleNotFoundError: No module named 'pkg_resources'
#12 ERROR: process "/bin/sh -c DEFAULT_FROM_EMAIL=dummy FEEDBACK_EMAIL=dummy EMAIL_URL=consolemail:// DATABASE_URL=sqlite://:memory: SECRET_KEY=dummy STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage python manage.py collectstatic --no-input --verbosity 2" did not complete successfully: exit code: 1
------
In addition, same commit builds successfully with :3.9-slim.
Trying from :3.12-slim (which equals to :3.12.6-slim as of now) downwards, worked for me with :3.12.4-slim.
Fair enough.
Regarding my particular issue I'm not quite sure where it stems from as there's a bunch of moving pieces involved (python base image, minimal debian base, copying things around, poetry, venvs, ...) - but for now I'll just fix it with
RUN ln -s /usr/local/bin/python /usr/local/bin/python3.11.
I'm still curious about what exactly changed. Could we perhaps get a changelog we could check before bumping the digest? It's rather difficult to manually diff what's changed in between versions ;)
See also https://github.com/docker-library/python/issues/957 (and other linked/related issues/PRs) :eyes:
Hi
These are two completely separate issues (in general it's best to open separate issues rather than re-using an existing open issue when the symptoms aren't the same):
- @black-snow your issue is presumably due to the subtle change in shebang lines described in #967, that was a side-effect of #955, and changed in the images of all supported Python versions. Most apps will work with both the old and new shebang lines, but if you are performing unusual manual symlinking/moving files around (like the use-case in #967 was), then you may need to adjust your symlinks/manual steps. The path you mentioned (
/usr/local/bin/python3.11) isn't in the images published from this repo, so it must be something you are creating manually. - @pataquets your issue is due to setuptools no longer being installed in the Python 3.12+ images as of #954. This removal will only affect broken packages that don't correctly specify their dependencies. It seems you are using an old version of
djangorestframework-simplejwtthat doesn't include the fix from https://github.com/jazzband/djangorestframework-simplejwt/commit/19cf38e4a440fd648e82a94a7998ac857d48a28d. Upgrade to a newerdjangorestframework-simplejwtversion or else manually install the setuptools package in the meantime.
Setuptools was distributes with 3.12.1-4 but adruptedly stopped in 3.12.5 ... Is this something that permanent or is it just something that was done in the middle of a version chain by accident?
@klausmyrseth This was answered/linked to already above. Please read:
- https://github.com/docker-library/python/pull/954
ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.
The reason I linked to that PR, is that your message asked if this change was done by accident - and the PR title and description make it clear that it wasn't by accident. It sounds that you are instead meaning "why was this change made deliberately" (which is a very reasonable question to ask, but it's still a different question), in which case there's more detail on the reasoning why in: https://github.com/docker-library/python/issues/964#issuecomment-2340617118
Thanks @edmorley !
Breaking changes suck and they are rather hard to avoid when the tag doesn't reflect a SemVer of the image but the underlying Python version (and who would want a 3.11-bookworm-1.17?).
So the remaining open question for me is if we could start providing a changelog. I pin all my digests and updates are deliberate, but it's veeeery cumbersome to find out what changed between two digests. I'll rename the issue to reflect that.
I agree that normally a changelog is a must for a published project. However, I'm not sure a changelog would be very easy to implement in a useful way for this repository, due to it not being a typical type of package with versioned releases. How would you list the changes, when there isn't a version? By date? Also, the rate of commits on this repo is generally very low, so skim reading https://github.com/docker-library/python/commits/master/ should typically be sufficient to figure out exactly what changed in the last week or so - and any changes outside of version bumps are so rare the changelog will be redundant 99% of the time.
In general if you ever have a problem with any open source project, I'd recommend checking the GitHub releases section, followed by the commit history, then searching recently updated issues and PRs. In this case a search for the word "symlink" would have found the issue filed 5 days ago, describing exactly your issue: https://github.com/docker-library/python/issues?q=is%3Aissue+symlink
What about bringing the links and names to the pull requests into a changelog that is easy accessible.
For me that would help me to see what's going on. For my sake I was very aware of the change python wise) but it can be hard to see what's going on with the docker. We had some code refering distutils in the old libs so I knew what to look for when the services refused to start suddently, but the first time I met the error it was hard to find the reason.
Also our thought process to remove the change when it was in v3.12.1 we thought you would not remove it until 13 as the version association brings that association to most users (like me)
@edmorley I disagree with a bunch of that ;)
Ofc I found similar issues, but do I want to rely on that someone else has already run into a similar problem and filed an issue for me to find before I bump the digest? Certainly not.
Do I want to identify all diff commits and read them in order to see if anything relevant might have changed? Nope.
I get that its somewhat difficult and many docker images are in a rather bad shape regarding this, yet writing a simple CHANGELOG.md when substantial things change seems easy enough to me.
ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.
Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments
ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.
Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments
At least you shouldn't roll 'em out without proper testing and scanning.