Can't build images and some images have issues with OverlayFS on ZFS dataset (NO zfs storage driver - rootless)

[DistantThunder]

System information

Operating System: Arch Linux Kernel Version: 5.14.9-zen2-1-zen (64-bit) Processors: 16 × AMD Ryzen 7 5800X 8-Core Processor Memory: 62.7 Gio of RAM Graphics Processor: AMD Radeon RX 6900 XT

Also reproduced on:

Fedora 34 Workstation 5.14.11

• ZFS-on-Linux version: 2.1.1

How to reproduce

• Set up an update- to-date Fedora environment though I suspect you can reproduce on practically all existing Linux distributions
• Install ZFS latest stable at this date: 2.1.1
• Create a ZFS dataset on mount it on Podman userspace directory hierarchy:
  • sudo zpool create my_pool <my_device> (where device can be a loop device backed by a file)
  • sudo zfs create -o mountpoint=/home/my_user/.local/share/containers my_pool/my_zfs_container_fs
  • sudo chown my_user:my_group ~/.local/share/containers
  • podman run --rm -it ubuntu:latest
    • apt update -y

Expected Output

Current Output

podman run --rm -it ubuntu:latest bash
Resolved "ubuntu" as an alias (/home/USER/.cache/containers/short-name-aliases.conf) Trying to pull docker.io/library/ubuntu:latest... Getting image source signatures Copying blob 7b1a6ab2e44d done
Copying config ba6acccedd done
Writing manifest to image destination Storing signatures root@c673eaee0123:/# apt update -y Err:1 http://archive.ubuntu.com/ubuntu focal InRelease Could not resolve 'archive.ubuntu.com' Err:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease Could not resolve 'archive.ubuntu.com' Err:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease Could not resolve 'archive.ubuntu.com' Err:4 http://security.ubuntu.com/ubuntu focal-security InRelease Could not resolve 'security.ubuntu.com' Reading package lists... Done
Building dependency tree
Reading state information... Done All packages are up to date. W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.orGYYt - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.ygZHHv - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.f5WYux - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.l3rrUt - IsAccessibleBySandboxUser (95: Operation not supported) W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Could not resolv

When trying to build:

podman build -f podman_ubuntu.dockerfile
STEP 1/2: FROM ubuntu:latest STEP 2/2: RUN apt update -y

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Err:1 http://archive.ubuntu.com/ubuntu focal InRelease Could not resolve 'archive.ubuntu.com' Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease Could not resolve 'security.ubuntu.com' Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease Could not resolve 'archive.ubuntu.com' Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease Could not resolve 'archive.ubuntu.com' Reading package lists... Building dependency tree... Reading state information... All packages are up to date. W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.z2PRDl - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.3uswOm - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.qbKIRj - IsAccessibleBySandboxUser (95: Operation not supported) W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.pYa7Km - IsAccessibleBySandboxUser (95: Operation not supported) W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Could not resolve 'archive.ubuntu.com' W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Could not resolve 'security.ubuntu.com' W: Some index files failed to download. They have been ignored, or old ones used instead. COMMIT Error: error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[apt update -y] Flags:[] Attrs:map[] Message:RUN apt update -y Original:RUN apt update -y}: error copying layers and metadata for container "4ce3e7abda649095263443f92cefda89ffbc24acb4fcaaba80337cfdff9a825a": initializing source containers-storage:ubuntu-working-container: error extracting layer "ead3b6dacf7bb14d766f280bb52ba3c519f6682bd4a8a86bf8390bc2f7bbb233": open /home/USER/.local/share/containers/storage/overlay/ead3b6dacf7bb14d766f280bb52ba3c519f6682bd4a8a86bf8390bc2f7bbb233/merged/var/lib/apt/lists/auxfiles: operation not supported

Since it is not possible at the moment to use ZFS storage driver in a non-root setup, would it be possible at least to have overlayfs use it as a simple "flat" filesystem?

I'm not exactly sure what's the problem here but it appears as if it should work?