Make Podman support more prominent in the documentation

[rgaiacs]

By

• using "container" instead of Docker when possible
• mention Podman configuration

[rgaiacs]

Please correct me if I got something wrong.

Before https://github.com/jupyterhub/repo2docker/pull/1402, repo2docker used Docker SDK for Python for all operations with Docker. The Docker client was created using from_env() that read information from the DOCKER_HOST environment variable. This allowed users to configure DOCKER_HOST to point to their Podman installation, see procedures.

After https://github.com/jupyterhub/repo2docker/pull/1402, repo2docker did not work with Podman. Changes in DockerEngine included in this pull request, restore the support to Podman.

Regarding container technology, the big players are

• Docker
• Podman (backed by Red Hat)
• Singularity (from Sylabs)
• Apptainer (fork of Singularity and under the umbrella of the Linux Foundation)

Podman keeps some level of compatibility with Docker. As far as I know, Singularity / Apptainer are incompatible with Docker.

My first question is, should repo2docker also support Podman as first class container engine? I mean: repo2docker documents how to use Podman and run tests with Podman.

If the answer is "yes", a follow up question is what would be the road map for repo2podman, a plugin for repo2docker that @manics developed.

[minrk]

My understanding was that podman was supported as long as it the engine backed by the docker cli (e.g. I use colima, not docker engine), but podman cli is not supported first-class in this package. Is that accurate, @manics?

I agree generally that pointing most language to OCI terms is good, but I don't know what commitment we are making in terms of cli compatibility assumptions.

[manics]

Originally repo2docker should've worked fine with Podman when run as a service since podman has a docker compatible API:

podman system service --time 0
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

After https://github.com/jupyterhub/repo2docker/pull/1402 we're now using the Docker CLI. In general the Podman CLI should be compatible, but I haven't looked at how good buildx support is. If it currently works I've no objection to supporting if, and ensuring that any future changes to repo2docker only use buildx features that podman supports.

[minrk]

If we've got tests running that verify it (to a reasonable degree), I'm happy to make the support official.

[rgaiacs]

After

• change the code to do base the logic in DOCKER_HOST
• add container_cli as a GitHub Actions matrix dimension

most of the tests using Podman fails. One of the tests fails in https://github.com/jupyterhub/repo2docker/blob/358888c6a93839dd4c60859737d886e49a4a1247/repo2docker/app.py#L633-L634 When I try to investigate it locally, I observe

>>> import docker
>>> client = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
>>> image = client.inspect_image("localhost/r2d-tests-dockerfile-2fsimple-1759752459")
>>> image.config
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    import platform
    ^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'config'
>>> dir(image)
['__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__ior__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__ror__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'copy', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']
>>> image.keys()
dict_keys(['Id', 'RepoTags', 'RepoDigests', 'Parent', 'Comment', 'Created', 'ContainerConfig', 'DockerVersion', 'Author', 'Config', 'Architecture', 'Os', 'Size', 'VirtualSize', 'GraphDriver', 'RootFS', 'Metadata', 'Container'])
>>> image["Config"]
{'Hostname': '', 'Domainname': '', 'User': '1000', 'AttachStdin': False, 'AttachStdout': False, 'AttachStderr': False, 'Tty': False, 'OpenStdin': False, 'StdinOnce': False, 'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LANG=C.UTF-8', 'GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D', 'PYTHON_VERSION=3.10.18', 'PYTHON_SHA256=ae665bc678abd9ab6a6e1573d2481625a53719bc517e9a634ed2b9fefae3817f', 'HOME=/tmp'], 'Cmd': ['/bin/sh', '-c', '"/bin/sh"'], 'Image': '', 'Volumes': None, 'WorkingDir': '', 'Entrypoint': None, 'OnBuild': None, 'Labels': {'io.buildah.version': '1.40.0', 'repo2docker.ref': 'None', 'repo2docker.repo': 'local', 'repo2docker.version': '2025.08.0+61.gced04a5'}}
>>> image["Config"]["WorkingDir"]
''

@manics do you know what might be happening?

[rgaiacs]

In 48e9243, tests with Podman were failing because Podman's service was not activate. This was fixed in 4e7ed9e and documented in ef36a9c.

Test of Dockerfile configuration with Podman continues to fail in https://github.com/jupyterhub/repo2docker/blob/358888c6a93839dd4c60859737d886e49a4a1247/repo2docker/app.py#L632-L634 I can reproduce the error locally with

import docker

a = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
i = a.inspect_image("docker.io/library/httpd:latest")
i.config

that returns

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    import platform
    ^^^^^^^^
AttributeError: 'dict' object has no attribute 'config'

but I can access the information using

i["Config"]["WorkingDir"]

I tested the

import docker
a = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
i = a.inspect_image("docker.io/library/httpd:latest")
i["Config"]["WorkingDir"]

with Docker and it works.

[rgaiacs]

A few tests are failing.

WorkingDir is missing

• [x] tests/dockerfile/binder-dir/verify::check-tmp
• [x] tests/dockerfile/simple/verify::check-tmp

in test (24.04, 3.13, podman, dockerfile).

I wrongly assume that in https://github.com/jupyterhub/repo2docker/blob/358888c6a93839dd4c60859737d886e49a4a1247/repo2docker/app.py#L632-L634 the client was APIClient when, in fact, the client is DockerEngine.

DockerEngine's inspect_image looks OK but there are cases where docker image inspect and podman image inspect produce different outputs.

~I will try to investigate this upstream.~ I asked https://github.com/containers/podman/discussions/27313.

buildx support

• [x] tests/unit/test_app.py::test_extra_buildx_build_args
• [x] tests/unit/test_connect_url.py::test_connect_url
• [x] tests/unit/test_editable.py::test_editable
• [x] tests/unit/test_editable.py::test_editable_by_host
• [x] tests/unit/test_users.py::test_user
• [x] tests/unit/test_volumes.py::test_volume_abspath
• [x] tests/unit/test_volumes.py::test_volume_relpath

in test (24.04, 3.13, podman, unit).

My suspicious is that the test fails because of some "obscure" difference between Podman and Docker.

[rgaiacs]

With commit efe313d256ffba5626a26c748a8ce3774ad3c5b7, when I run

pytest -s tests/unit/test_editable.py::test_editable

the log shows

[Warning] one or more build args were not consumed: [NB_USER]

Why does the above warning happened? Is --user-id not supported for Dockerfile configuration?

[manics]

DockerEngine's inspect_image looks OK but there are cases where docker image inspect and podman image inspect produce different outputs.

I repo2podman I found passing the format arg --format '{{json .}}' worked: https://github.com/manics/repo2podman/blob/b772bb818dbdc8f513a01bbb0c02eed4848d371d/repo2podman/podman.py#L259-L264

[rgaiacs]

This is ready for a review and I would suggest to avoid increase it more.

The tests now run and pass for Podman as for Docker. Because Podman is, by default, rootless, I configured tests related to bind mount to be skipped, read https://github.com/jupyterhub/repo2docker/issues/1483 for details. I also skipped the tests related with the registry as they have some bind mount, see https://github.com/jupyterhub/repo2docker/blob/443b85127d2f745090dd2354cb77daae0c41bcae/tests/norun/test_registry.py#L42-L43

The GitHub Action runner ubuntu-24.04 includes Podman 4.9.3 that is a good number of versions older than the latest Podman release (5.6.2). In another pull request, we could have miniconda installing podman from conda-forge.

[rgaiacs]

@minrk wrote

If we've got tests running that verify it (to a reasonable degree), I'm happy to make the support official.

Excepted for some tests mentioned in https://github.com/jupyterhub/repo2docker/pull/1471#issuecomment-3491389862, we have all existing tests passing for Podman and Docker.

@manics wrote

I don't think we need to run all tests with Podman, just a subset

and I agree that we could avoid run all the tests a second time for Podman. Before I amend this pull request, do you have a strong opinion of which or how many tests should be run with Podman?

Instead of using @manics suggestion to select the subset of tests for Podman with include:, I prefer to go down one level and use pytest.mark to annotate the tests that we want to run with Podman.

[manics]

How about:

• The UI test https://github.com/jupyterhub/repo2docker/tree/main/tests/ui
• Unit tests since they should be quick https://github.com/jupyterhub/repo2docker/tree/main/tests/unit
• one of https://github.com/jupyterhub/repo2docker/tree/main/tests/dockerfile
• https://github.com/jupyterhub/repo2docker/tree/main/tests/conda/py-r-postbuild-file since it involves R, Python and Postbuild
• https://github.com/jupyterhub/repo2docker/tree/main/tests/pipfile/setup-py-explicit-in-binder-dir ?

If you're using pytest.mark perhaps mark them as something like "quick/quicktest/fast/etc" instead of "podman" since it's more generally useful if you don't want to run the full test suite?

[rgaiacs]

Thanks @manics for the suggestion. I will incorporate the suggestion later this week.

[rgaiacs]

The way that the tests are organised with pytest_collect_file and make_test_func makes hard to use pytest's built-in mark.

GitHub Actions has include and exclude. I think the include hard to understand and the exclude easier. Although more verbose, I'm amending this PR with exclude.

[rgaiacs]

@manics and @yuvipanda are you OK for this to be merged?

[rgaiacs]

Thanks for the review. I improved the PR based on the comments.