podman commit on containers with a large files throws error: io: read/write on closed pipe

[Rubusch]

trafficstars

Issue Description

My original motivation is creating dockerized development environments keeping installations of Xilinx Vivado and tooling. This Package is huge ~100GBs+. Migrating this setup from docker-compose/docker to podman, I saw this problem using podman build. Manually I tried to stop building at half of the image, then login to the container, and install Vivado manually which worked. Outside then I tried a commit of the container, (to my understanding this is what a RUN instruction in a Dockerfile is performing). This produced the exact same error: ...:io: read/write on closed pipe.

I found a similarly described issue here: 8845. But my case seems to be different: In the issue huge files are copied into the container. In my case the container contains a huge file and I'd like to commit it (assumed it is my issue with installing Vivado). Anyway I also tried to play with systemd notification w/o success here. As mentioned, the setup worked actually with docker, so I assume a bug here. Thus, here to reproduce what probably provokes my error is the described approach down below.

If there is anything I missed out which makes this setup work, pls don't hesitate to let me know! Perhaps I'm wrong with my assumption and the commit. Also, I highly appreciate any hints to make this run.

Steps to reproduce the issue

Steps to reproduce the issue
(this is the 'sudo' version to rule out permission related things, but rootless shows the same result):

1. Create Dockerfile with:

FROM alpine:3.1

2. Execute sudo podman build -t test-large-container.
3. Run the container and login, then create a "huge file"

$ sudo podman run --rm -it localhost/test-large-container /bin/sh
/ # dd if=/dev/zero of=large-file-1gb.txt count=102400 bs=1048576

4. In a different shell, try to commit:

$ sudo podman commit 529e4d221b0c localhost/test-large-container
ERRO[0438] Can't add file /var/lib/containers/storage/overlay/24d7d336d6402609a3889f4aed6b120daee1c30881f1cd6da069ca260779f569/diff/large-file-1gb.txt to tar: io: read/write on closed pipe
ERRO[0438] io: read/write on closed pipe
ERRO[0438] Can't close tar writer: io: read/write on closed pipe
Error: copying layers and metadata for container "529e4d221b0c0220b44ef0acaa879f54406411695a1c2a65128169239a40fd2a": initializing source containers-storage:flamboyant_wu: storing layer "24d7d336d6402609a3889f4aed6b120daee1c30881f1cd6da069ca260779f569" to file: io: read/write on closed pip

Describe the results you received

I receive the described error:

Can't add file /var/lib/containers/storage/overlay/.../large-file-1gb.txt to tar: io: read/write on closed pipe

Describe the results you expected

Doing the Steps to reproduce the issue with 10GB works. The same I would expect for >100G.

$ sudo podman commit 0318214e6d99 localhost/test-large-container
Getting image source signatures
Copying blob d4c261b2e248 skipped: already exists
Copying blob 587bd0c5f5ab skipped: already exists
Copying blob 67bdf04dbb3d done   |
Copying config e6289a249d done   |
Writing manifest to image destination
e6289a249d18b2b8b9567d464dab4b4ab297636507ad694b225d45c42dcc4e85

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 92.13
    systemPercent: 0.97
    userPercent: 6.89
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2046
  hostname: MENDOZA
  idMappings:   
    gidmap: null
    uidmap: null
  kernel: 6.8.0-31-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 11812769792
  memTotal: 33547034624
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: runc  
    package: runc_1.1.12-0ubuntu3_amd64
    path: /usr/bin/runc
    version: |- 
      runc version 1.1.12-0ubuntu3
      spec: 1.0.2-dev
      go: go1.22.2
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |  
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket: 
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:  
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |- 
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 90112
  swapTotal: 1023406080
  uptime: 842h 60m 33.00s (Approximately 35.08 days)
  variant: ""   
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough 
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}  
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 2   
    paused: 0   
    running: 2  
    stopped: 0  
  graphDriverName: overlay
  graphOptions: 
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 4465355571200
  graphRootUsed: 2422932537344
  graphStatus:  
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:   
    number: 89  
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: "" 
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Additional information

No response

[rhatdan]

Any chance you could try this with a newer podman version 5.*

[Rubusch]

I would like to. I tried downloading podman-remote-static-linux_amd64.tar.gz 5.1.1 but was not sure how to use it. After unpacking I simply placed it at /usr/bin/podman. Then - following https://podman.io/docs/installation somehow - I tried podman machine init, followed by podman machine start. I received an error, the old .sock file was there, no connection. (Feeling lucky) I simply removed the .sock. Then it complained about gvproxy not being there or /and I should register some dns at [engline] in the container.conf. I felt a bit in the weeds and gave that up. Is there something I can do to stop the existing podman, and replace it by this statically linked version?

[rhatdan]

That is probably not what you wanted to do. Getting an updated version of Podman onto Ubuntu is a bit painful, which is why I use Fedora...

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[MagicRB]

just ran into this, for an extremely basic container, no where near 100GB distro: NixOS

[reukiodo]

Is there no workaround for this?

EDIT: nevermind, I'll just use docker proper

[royteeuwen]

same here, we have a container that is a few GB that we are building on a macOS with podman desktop, it stops after a while with the following error:

Error: committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/aemc-cli/1.8.4 container=oci AEM_CLI_EXECUTABLE=/opt/aemc-cli/1.8.4/aem] 
Command:run Args:[ansible-playbook -i "localhost," -c local /ansible/aem-forms.yml] Flags:[] Attrs:map[] Message:RUN ansible-playbook -i "localhost," -c local /ansible/aem-forms.yml Heredocs:[] 
Original:RUN ansible-playbook -i "localhost," -c local /ansible/aem-forms.yml}: 
copying layers and metadata for container "d679aab69b5fbef05a5c5d0039e1a3685cadb487068bfe4236314d1467728c12": 
initializing source containers-storage:2f4323ee7f4e-working-container: storing layer "ee77cb8e2da039b044ad3e4ad720f1dca03969c5e75a89c9c3123f5abab6b346" to file: 
io: read/write on closed pipe

What could be done to mitigate this?

[rhatdan]

Could be memory issues, perhaps increase memory available to podman machine?

[royteeuwen]

I deleted the podman machine, added more cpu and ram and that indeed did seem to solve it :)

[rhatdan]

I think podman machine set can reset this without needing to blow away the machine. You would need to restart it though.