gsc build fails for any distro workload on a Ubuntu 24.04 host system

[anjalirai-intel]

Description of the problem

gsc build fails for any distro workload on a Ubuntu 24.04 host system

As per gsc requirement pip3 install docker jinja2 tomli tomli-w pyyaml when we try to install these libraries, it fails with below errors.

pip3 install docker jinja2 tomli tomli-w pyyaml
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Ubuntu adopted PEP 668 from 23.04 , which dictates that pip can no longer install packages managed by the distro's general-purpose package manager, hence we use apt-get. Above listed requirement are installed via apt, versions are mentioned below

docker: 5.0.3 jinja2 : 3.1.2 tomli : 2.0.1 tomli-w : 1.0.0 pyyaml : 6.0.1

Steps to reproduce

https://github.com/gramineproject/gsc/tree/master/test#building-sample-images

Expected results

./gsc build -nc --insecure-args bash-test test/ubuntu20.04-bash.manifest
Building unsigned graminized Docker image `gsc-bash-test-unsigned` from original application image `bash-test`...
Warning: Duplicate key `loader.env.PATH`. Concatenating values from `<merged test/ubuntu20.04-bash.manifest and ubuntu/entrypoint.manifest.template>` and `<bash-test image env>`.
Step 1/29 : FROM ubuntu:20.04 AS gramine

 ---> 2abc4dfd8318
Step 2/29 : RUN env DEBIAN_FRONTEND=noninteractive apt-get update     && env DEBIAN_FRONTEND=noninteractive apt-get install -y         autoconf         bison         build-essential         coreutils         curl         gawk         git         libprotobuf-c-dev         linux-headers-generic         nasm         ninja-build         pkg-config         protobuf-c-compiler         protobuf-compiler         python3         python3-cryptography         python3-protobuf         wget            python3-pip            && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'

 ---> Running in 6b1a9fa7af23
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3608 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3669 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1205 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4143 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1502 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.5 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3758 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 31.5 MB in 6s (5035 kB/s)
Reading package lists...

Reading package lists...

Building dependency tree...

Reading state information...
coreutils is already the newest version (8.30-3ubuntu2).
The following additional packages will be installed:
  automake autotools-dev binutils binutils-common binutils-x86-64-linux-gnu
  ca-certificates cpp cpp-9 dirmngr dpkg-dev fakeroot file g++ g++-9 gcc gcc-9
  .....
  .....
  .....
  ....
  ...
  Step 29/29 : ENTRYPOINT ["/bin/bash", "/gramine/app_files/apploader.sh"]

 ---> Running in 93e57c7507b4
 ---> d734a23fb3fc
Successfully built d734a23fb3fc
Successfully tagged gsc-bash-test-unsigned:latest
Successfully built an unsigned graminized Docker image `gsc-bash-test-unsigned` from original application image `bash-test`.

Actual results

Error:

./gsc build -nc --insecure-args bash-test test/ubuntu20.04-bash.manifest
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
TypeError: HTTPConnection.request() got an unexpected keyword argument 'chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc", line 12, in <module>
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc.py", line 680, in main
    return args.command(args)
           ^^^^^^^^^^^^^^^^^^
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc.py", line 284, in gsc_build
    docker_socket = docker.from_env()
                    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/usr/lib/python3/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'

[marchuk-vlad]

Same problem. Install requests version 2.31.0 until it will be fixed pip3 install requests==2.31.0 docker jinja2 tomli tomli-w pyyaml

[anjalirai-intel]

We can no longer install packages via pip

[mkow]

Does it work if you install these packages from apt?

[anjalirai-intel]

Does it work if you install these packages from apt?

All the packages are installed via apt

[jkr0103]

Root cause of the issue:

GSC uses Docker SDK for python for building the docker images which internally makes use of requests and urllib3 for connections with docker service.

Ubuntu 24.04 installs python3-urllib3=2.0.7 and python3-requests=2.31.0 versions which are not compatible as described here. This compatibility issue is fixed in Docker SDK for python version 6.1.0 with PR 3116 but Ununtu 24.04 installs Docker SDK for python version 5.0.3 which doesn't have the fix in it, hence we see the issue.

[jkr0103]

There are three approaches to fix this issue, @dimakuv @mkow, Please suggest which one suits better:

1. Force docker sdk for python installation with pip using command: pip3 install docker --break-system-packages which installs the python3-docker version >=6.1.0.
2. Build/install manually from source code:

git clone --branch 7.0.0 https://github.com/docker/docker-py.git
cd docker-py
python3 setup.py build
sudo python3 setup.py install

Test:

$ python3
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker;
>>> client = docker.from_env()

Note: setup.py is not available after version 7.0.0 which break above commands (build/install). I created an issue here for the same.

3. Wait for the python3-docker package upgrade to >=6.1.0 until Gramine announses support for ubuntu 24.04. This would automatically fix the issue. I have created a bug for ubuntu here which is in confirmed state now.

[dimakuv]

Gramine and GSC currently don't claim official support for Ubuntu 24.04. So I think we can safely wait for option 3 -- this bug should be fixed in Ubuntu itself, and looks like there is no easy workaround for GSC at the moment.

So my suggestion is to sit and wait for ~2 months. If nothing happens in this time frame, then we should revisit this issue.

[anjalirai-intel]

@dimakuv @aneessahib @jkr0103

As we approach the Gramine 1.8 release and prepare for the official support of Ubuntu 24.04, we have been anticipating some resolution from the Ubuntu package manager regarding this issue. However, since I am still able to reproduce the problem, should we start considering alternative solutions to address this issue?

[dimakuv]

@anjalirai-intel The fix is currently in progress:

• https://bugs.launchpad.net/ubuntu/+source/python-docker/+bug/2065348
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1073196

Given this, I would still prefer for this to be fixed in Ubuntu 24.04. From my understanding, the fix is very close to be available in Ubuntu 24.04.

[jkr0103]

There is duplicate bug reported here which have current status about the bug:

• Fixed in the development version that is the Oracular
• Fix have been uploaded for Noble and is now in the SRU review queue

[dimakuv]

The bug is not being fixed as of yet, and the release date of Gramine v1.8 and GSC v1.8 is getting closer.

I propose to continue with solutions 1 (pip install) or 2 (manual install) from here: https://github.com/gramineproject/gsc/issues/202#issuecomment-2134551321

@woju What could you recommend? Maybe there's another solution to circumvent the bug in Ubuntu 24.04 with respect to this package?

[woju]

The "correct" thing to do from Python POV would be to package gsc as a Python distribution, then instruct people to install it (and it's requirements that can't be installed from Linux distro's package manager) into virtual environment (virtualenv or venv). "Correct" is scarequoted, because this would require full reorganisation of the repository and God forbid if someone would be packaging a Python app, then user will be dealing with two different venvs at the same time, and that would almost guarantee that people will confuse those (unless they'd have expert knowledge about what package goes where, which they probably don't have because gsc is meant to be easy).

So people would do something like this:

python3 -m venv --system-site-packages ./venv
./venv/bin/activate
pip3 install .
gsc ...

There's also a hybrid approach that we can do right now: instruct people to get a venv, but run the script as we currectly do:

python3 -m venv --system-site-packages ./venv
./venv/bin/activate
pip3 install 'docker>=6.1.0'
./gsc ...

This will be also confusing to two kinds of people: 1) those who don't know python: wtf is venv; and 2) those who know python: why we don't install this package into venv.

Of those available ways, I suggest in order: 1) We wait as long as possible for Ubuntu to release those packages. If they can't make it before we want to release gsc, then 2) for now write in README that people install docker-py into venv and run gsc from activated venv. That's the least amount of work for us. If someone objects against that and say they want to install gsc into venv, then 3) we reorganise the GSC repo. That'd be some work (mainly because we also need to package templates), but overall doable and not very complicated. Worse, it'd be yet more work downstream if someone maintains internal GSC fork because IDK, maybe they have some changes to the templates or something like that.

[dimakuv]

Thanks @woju. The hybrid approach (where Python's venv inherits all system packages but only installs docker package via pip3) seems like the simplest workaround for now.

@anjalirai-intel @jkr0103 Could you test this approach? It should be enough to execute Woju's snippet to test it:

python3 -m venv --system-site-packages ./venv
./venv/bin/activate
pip3 install 'docker>=6.1.0'
./gsc ...

If this indeed works fine, then I propose to add a section on How to run on Ubuntu 24.04 in README and ReadTheDocs, with this bash snippet and short explanation (and a link to the to-be-fixed bug description). For ReadTheDocs, we have the Limitations section, so should be reasonable to add there: https://gramine.readthedocs.io/projects/gsc/en/latest/#limitations

[woju]

If you add such a section, can you title it "... on Ubuntu 24.04 (and any recent distro compliant with PEP 668)"? It's not Ubuntu problem per se, eventually all distros (their recent versions) will need this or similar procedure.

[jkr0103]

For ReadTheDocs, we have the Limitations section, so should be reasonable to add there

I feel users might miss it if we add it to the limitations section as it's towards the end of document, also this is still a temporary workaround until fix is upstreamed.

[anjalirai-intel]

Thanks @woju. The hybrid approach (where Python's venv inherits all system packages but only installs docker package via pip3) seems like the simplest workaround for now.

@anjalirai-intel @jkr0103 Could you test this approach? It should be enough to execute Woju's snippet to test it:

python3 -m venv --system-site-packages ./venv
./venv/bin/activate
pip3 install 'docker>=6.1.0'
./gsc ...

If this indeed works fine, then I propose to add a section on How to run on Ubuntu 24.04 in README and ReadTheDocs, with this bash snippet and short explanation (and a link to the to-be-fixed bug description). For ReadTheDocs, we have the Limitations section, so should be reasonable to add there: https://gramine.readthedocs.io/projects/gsc/en/latest/#limitations

@dimakuv We tried the venv experiment with Ubuntu 20.04 and Ubuntu 24.04 distro for bash workload, we were able to run the workloads, No issue seen with venv environment

[ezekieldas]

Just adding comment here, FWIW. We were eager to adopt 24.04. All our systems are provisioned via Ansible. The failure of a critical application using Docker SDK Python led me here. To workaround this issue I chose to use as minimal steps as necessary. Additionally, all of our Python libs are applied via apt (this now being an exception). With Ansible (ansible [core 2.16.8]):

    - name: install python items - pip
      pip:
        name: docker
        extra_args: --break-system-packages

I'll be keeping a close eye on the status of this issue as we'll want to remove this workaround as soon as the proper Docker SDK Python version (docker>=6.1.0) is available via apt.

Thanks for all the attention on this issue!

[akx]

then user will be dealing with two different venvs at the same time, and that would almost guarantee that people will confuse those

TBH, with a properly packaged application, they would be able to install gsc with pipx (as the externally-managed-environment error message says): pipx install gsc. It would live in a pipx-managed isolated venv, and users wouldn't need to know about venvs at all.

[jkr0103]

3. Wait for the python3-docker package upgrade until Gramine announses support for ubuntu 24.04. This would automatically fix the issue. I have created a bug for ubuntu here which is in confirmed state now.

Fix is upstreamed now for nobel. We can take back the venv workaround pushed with PR#214 and instruct to do sudo apt update && sudo apt install python3-docker --upgrade

[dimakuv]

@jkr0103 Feel free to submit a corresponding PR.