podman-py
podman-py copied to clipboard
containers
podman client hangs when starting container
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
When trying to run a container (with possibly invalid configuration) via podman-py, causes the system service to hang. Killing the client call will still leave the service spinning.
❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)'
^CTraceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run
container.start()
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 350, in start
response = self.client.post(
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 304, in post
return self._request(
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 402, in _request
self.request(
File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.9/http/client.py", line 1345, in getresponse
response.begin()
File "/usr/lib/python3.9/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
❯ podman run --mount=type=bind,source=tests/,destination=tests123 -d registry.fedoraproject.org/fedora
Error: invalid container path "tests123", must be an absolute path
❯ podman run --mount=type=bind,source=tests/,destination=/tests123 -t -d registry.fedoraproject.org/fedora
4b2829c8e699643859b55369d2d558198dc17f04d5d7c1597a4ac791258d62c6
❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "/tests123", "source": "tests/", "mode": "bind"}], tty=True)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run
container.start()
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 353, in start
response.raise_for_status()
File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 65, in raise_for_status
raise APIError(cause, response=self._response, explanation=message)
podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (invalid mount type for `/tests123`: OCI runtime error)
Steps to reproduce the issue:
mkdir xyz && cd xyz/
python -m venv venv; source venv/bin/activate
podman system service -t 0 &
pip install podman
podman pull registry.fedoraproject.org/fedora
python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="registry.fedoraproject.org/fedora", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)'
Describe the results you received: the podman client will hang while the podman service spins and stresses out the system.
Describe the results you expected: Either incorrect configuration or a successful start + detach
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Version: 3.1.2
API Version: 3.1.2
Go Version: go1.16.4
Git Commit: 51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
Built: Sat May 22 16:22:34 2021
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.20.1
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.0.28-1
path: /usr/bin/conmon
version: 'conmon version 2.0.28, commit: 6b18f7e0f2e4cd7f7b016b88141e82210d370008'
cpus: 24
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: autumn
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.12.7-arch1-1
linkmode: dynamic
memFree: 57026449408
memTotal: 67362938880
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 0.19.1-1
path: /usr/bin/crun
version: |-
crun version 0.19.1
commit: 1535fedf0b83fb898d449f9680000f729ba719f5
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
selinuxEnabled: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.10-1
version: |-
slirp4netns version 1.1.10
commit: baa2bc5ff12fe6db646c1f4f3f966526c0eba5a0
libslirp: 4.5.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 1073737728
swapTotal: 1073737728
uptime: 10h 29m 51.81s (Approximately 0.42 days)
registries: {}
store:
configFile: /home/jbpratt/.config/containers/storage.conf
containerStore:
number: 2
paused: 0
running: 0
stopped: 2
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/jbpratt/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 5
runRoot: /run/user/1000/containers
volumePath: /home/jbpratt/.local/share/containers/storage/volumes
version:
APIVersion: 3.1.2
Built: 1621718554
BuiltTime: Sat May 22 16:22:34 2021
GitCommit: 51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
GoVersion: go1.16.4
OsArch: linux/amd64
Version: 3.1.2
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.): physical
@jbpratt: The label(s) kind/bug cannot be applied, because the repository doesn't have them.
In response to this:
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
When trying to run a container (with possibly invalid configuration) via
podman-py, causes the system service to hang. Killing the client call will still leave the service spinning. When running directly from podman, I get the correct error that:❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)' ^CTraceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run container.start() File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 350, in start response = self.client.post( File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 304, in post return self._request( File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 402, in _request self.request( File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.9/http/client.py", line 1345, in getresponse response.begin() File "/usr/lib/python3.9/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.9/http/client.py", line 268, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) KeyboardInterrupt ❯ podman run --mount=type=bind,source=tests/,destination=tests123 -d registry.fedoraproject.org/fedora Error: invalid container path "tests123", must be an absolute path ❯ podman run --mount=type=bind,source=tests/,destination=/tests123 -t -d registry.fedoraproject.org/fedora 4b2829c8e699643859b55369d2d558198dc17f04d5d7c1597a4ac791258d62c6 ❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "/tests123", "source": "tests/", "mode": "bind"}], tty=True)' Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run container.start() File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 353, in start response.raise_for_status() File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 65, in raise_for_status raise APIError(cause, response=self._response, explanation=message) podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (invalid mount type for `/tests123`: OCI runtime error)Steps to reproduce the issue:
mkdir xyz && cd xyz/ python -m venv venv; source venv/bin/activate podman system service -t 0 & pip install podman podman pull registry.fedoraproject.org/fedora python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="registry.fedoraproject.org/fedora", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)'Describe the results you received: the podman client will hang while the podman service spins and stresses out the system.
Describe the results you expected: Either incorrect configuration or a successful start + detach
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:(paste your output here)Output of
podman info --debug:host: arch: amd64 buildahVersion: 1.20.1 cgroupManager: systemd cgroupVersion: v2 conmon: package: /usr/bin/conmon is owned by conmon 1:2.0.28-1 path: /usr/bin/conmon version: 'conmon version 2.0.28, commit: 6b18f7e0f2e4cd7f7b016b88141e82210d370008' cpus: 24 distribution: distribution: arch version: unknown eventLogger: journald hostname: autumn idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.12.7-arch1-1 linkmode: dynamic memFree: 57026449408 memTotal: 67362938880 ociRuntime: name: crun package: /usr/bin/crun is owned by crun 0.19.1-1 path: /usr/bin/crun version: |- crun version 0.19.1 commit: 1535fedf0b83fb898d449f9680000f729ba719f5 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL os: linux remoteSocket: exists: true path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true selinuxEnabled: false slirp4netns: executable: /usr/bin/slirp4netns package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.10-1 version: |- slirp4netns version 1.1.10 commit: baa2bc5ff12fe6db646c1f4f3f966526c0eba5a0 libslirp: 4.5.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.1 swapFree: 1073737728 swapTotal: 1073737728 uptime: 10h 29m 51.81s (Approximately 0.42 days) registries: {} store: configFile: /home/jbpratt/.config/containers/storage.conf containerStore: number: 2 paused: 0 running: 0 stopped: 2 graphDriverName: overlay graphOptions: {} graphRoot: /home/jbpratt/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 5 runRoot: /run/user/1000/containers volumePath: /home/jbpratt/.local/share/containers/storage/volumes version: APIVersion: 3.1.2 Built: 1621718554 BuiltTime: Sat May 22 16:22:34 2021 GitCommit: 51b8ddbc22cf5b10dd76dd9243924aa66ad7db39 GoVersion: go1.16.4 OsArch: linux/amd64 Version: 3.1.2Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.): physical
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi @mrbazzan , I don't think I found a workaround and eventually just switched to using the podman CLI. Sorry :frowning_face:
@rhatdan @jbpratt
Looking over the source code, I noticed that this line might have something to do with this error(i.e container hanging).
Using the PodmanClient().images.run method, whenever I run commands like ls, /bin/ls in the container; the STATUS goes from CREATED to EXITED without passing through the RUNNING state(or maybe it did...). And the container.wait(condition="running") is expecting the RUNNING status.
Commands such as /bin/sh -c ls works fine (until it throws TypeError as stated in #184 number 3).
Thoughts?
@rhatdan @jwhonce
Please what's the return type of PodmanClient.containers.wait()?
Looks like this issue was fixed by https://github.com/containers/podman-py/pull/185. closing the issue, but please re-open if it is not actually fixed.
