containers
podman import failed with error 'layer 0 <...> does not match config's DiffID'
Issue Description
Importing a tar.xz archive as a container fail with the following error :
Error: writing blob: layer 0 (blob "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"/""/"sha256:ebccf21abea5bba2d53df5158901c325b33a5cb1a17c44ddd6cc97c6036a9dcc") does not match config's DiffID "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"
I tested with an archive that the size is ~45MB without problem.
Steps to reproduce the issue
Steps to reproduce the issue
-
wget https://mirror.init7.net/gentoo//releases/amd64/autobuilds/current-stage3-amd64-musl-hardened/stage3-amd64-musl-hardened-20250309T170330Z.tar.xz -
podman import stage3-amd64-musl-hardened-20250309T170330Z.tar.xz gentoo-musl:latest
Describe the results you received
Error: writing blob: layer 0 (blob "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"/""/"sha256:ebccf21abea5bba2d53df5158901c325b33a5cb1a17c44ddd6cc97c6036a9dcc") does not match config's DiffID "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"
Describe the results you expected
Import task successfull (with podman-5.3.2):
Getting image source signatures
Copying blob 57eca825bd92 done |
Copying config 0b14cb6fdc done |
Writing manifest to image destination
sha256:0b14cb6fdc54455ab9964482dff111d02a696a3846df2d4db5bdf1dae6ce0582
podman info output
host:
arch: amd64
buildahVersion: 1.39.2
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: cgroupfs
cgroupVersion: v2
conmon:
package: app-containers/conmon-2.1.13
path: /usr/bin/conmon
version: 'conmon version 2.1.13, commit: v2.1.13'
cpuUtilization:
idlePercent: 99.89
systemPercent: 0.06
userPercent: 0.05
cpus: 16
databaseBackend: sqlite
distribution:
distribution: gentoo
version: "2.17"
eventLogger: file
freeLocks: 2045
hostname: masha
idMappings:
gidmap: null
uidmap: null
kernel: 6.12.16-gentoo
linkmode: dynamic
logDriver: k8s-file
memFree: 48935530496
memTotal: 67151851520
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: app-containers/aardvark-dns-1.12.2-r1
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.12.2
package: app-containers/netavark-1.12.2-r1
path: /usr/libexec/podman/netavark
version: netavark 1.12.2
ociRuntime:
name: crun
package: app-containers/crun-1.20
path: /usr/bin/crun
version: |-
crun version 1.20
commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
rundir: /run/crun
spec: 1.0.0
+SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: net-misc/passt-2025.01.21
version: |
pasta 2025.01.21
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: app-containers/slirp4netns-1.2.0
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.6.0
swapFree: 0
swapTotal: 0
uptime: 406h 4m 29.00s (Approximately 16.92 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 0
stopped: 1
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /vol/containers/storage
graphRootAllocated: 21474836480
graphRootUsed: 5049495552
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 120
runRoot: /run/containers/storage
transientStore: false
volumePath: /vol/containers/storage/volumes
version:
APIVersion: 5.4.1
Built: 1741719773
BuiltTime: Tue Mar 11 20:02:53 2025
GitCommit: ""
GoVersion: go1.23.6
Os: linux
OsArch: linux/amd64
Version: 5.4.1
Podman in a container
No
Privileged Or Rootless
None
Upstream Latest Release
Yes
Additional environment details
Additional environment details
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
also see https://github.com/containers/podman/issues/18193, importing via xz is not supported
Was the resulting image on 5.3 before that actually usable?
cc @mtrmac
… and https://github.com/containers/podman/pull/25465 afterwards.
I think the image might have previously mostly worked: until recently, Podman basically didn’t care about the DiffIDs values in the config (which is why the reproducer in #18193 requires import+save+load, it’s only the load step that relies on DiffIDs being accurate).
But now we require DiffIDs to match, so this fails early.
Otherwise, #18193 still applies: it would be ~possible but awkward to support non-OCI-recognized compression formats, or c/image should detect non-gzip compression formats, and reject them if they are not implemented.
@Luap99 @mtrmac Thanks you for the precise responses. I confirm : the images are working well with podman-5.3.2.
podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/kubler-gentoo/stage3-amd64-hardened-openrc latest f5e1eb40577b 2 days ago 1.19 GB
localhost/kubler-gentoo/stage3-amd64-hardened-openrc 20250309T170330Z f5e1eb40577b 2 days ago 1.19 GB
localhost/kubler-gentoo/stage3-amd64-musl-hardened latest 631fd823c282 2 days ago 1 GB
localhost/kubler-gentoo/stage3-amd64-musl-hardened 20250309T170330Z 631fd823c282 2 days ago 1 GB
localhost/kubler-gentoo/portage 20250309 38a1f7a739ab 3 days ago 341 MB
localhost/kubler-gentoo/portage latest 38a1f7a739ab 3 days ago 341 MB
Reading the various tickets, I understand that load and import no longer support bzip2 and xz formats, but this seems to me to be a break with docker compatibility. I don't think theses archive formats are exotic. Won't this impact the adoption of podman?
If we were able to import xz images fine then we should continue to support that as not doing so it clearly a breaking change. My assumption was the the resulting image was not usable already, but if we were able to run containers with that then this seems clearly unacceptable to break in a minor release. Docker compat is also certainly a factor for us.
@mtrmac Why does the compression format matter here at all? Import only takes a tarball and there is no actual image and/or layer data so why would c/image or c/storage care since whatever we store is the uncompress representation as single layer from the full tar content? Why would we keep any compression info from the tar used to import? If we decompress the tar outside of podman then load it it seems to work fine.
The problematic piece of code in the tarball transport turns the input into a single image layer, and writes (what it thinks is) the uncompressed digest into the DiffID metadata field. And that is:
- disconnected from the “primary consumer” that extracts the layer on disk; that consumer can handle other formats, so the import mostly works, except for the metadata.
- only supporting gzip
- not all that robust, assuming that anything not gzip is uncompressed. (To be fair, this happens throughout the codebase — we have heuristics to recognize compression formats, but not a heuristic to validate that the outcome is a tar. Usually that’s unnecessary because parsing as a tar immediately follows anyway.)
Anyway, the breakage should be easy to fix. The incorrect metadata on the imported tarball (no way to express "xz") would remain, but, meh.
Anyway, the breakage should be easy to fix. The incorrect metadata on the imported tarball (no way to express "xz") would remain, but, meh.
Which would be fine as it would not be a new regression. I mean we could work around it by decompression the stream before handing to c/image I suppose if that would help? I don't think we need or want to actually express a xz image if it has no defined media types. But on a podman push I would think it should be able to compress using gzip or zstd to publish it somewhere.
I don’t think working around this at the Podman level would be any better — the tarball transports exists ~exclusively for {buildah,podman} import.
What needs to happen is something like https://github.com/containers/image/pull/2795 , just as a demo.
Does anyone have time to test that, confirm that it does fixes this failure, and that a subsequent podman push / podman save produces reasonable output, particularly MIME types?
Looks reasonable to me at least, though I don't pretend I know all the mime type/layout specifics so see for yourself:
$ bin/podman import ~/testimage.tar.xz test
Getting image source signatures
Copying blob 5329b0732ae3 skipped: already exists
Copying config 801aac52ee done |
Writing manifest to image destination
sha256:801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7
$ bin/podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/test latest 801aac52ee1d 26 hours ago 12.2 MB
$ bin/podman push test oci:/tmp/test
Getting image source signatures
Copying blob cecadf112837 done |
Copying config 801aac52ee done |
Writing manifest to image destination
$ tree /tmp/test/
/tmp/test/
├── blobs
│ └── sha256
│ ├── 3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad
│ ├── 801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7
│ └── eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915
├── index.json
└── oci-layout
3 directories, 5 files
$ jq . /tmp/test/index.json
{
"schemaVersion": 2,
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915",
"size": 348
}
]
}
$ jq . /tmp/test/blobs/sha256/eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7",
"size": 419
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad",
"size": 5595088
}
]
}
$ file /tmp/test/blobs/sha256/3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad
/tmp/test/blobs/sha256/3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad: gzip compressed data, original size modulo 2^32 12175360
$ podman save test -o test.tar
Copying blob cecadf112837 done |
Copying config 801aac52ee done |
Writing manifest to image destination
$ tar -tf test.tar
cecadf112837f146298c7433211a515d0df853a8dbe21fb08f1327152c951be6.tar
801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7.json
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/layer.tar
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/VERSION
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/json
manifest.json
repositories
$ bin/podman load --input test.tar
Getting image source signatures
Copying blob cecadf112837 skipped: already exists
Copying config 801aac52ee done |
Writing manifest to image destination
Loaded image: localhost/test:latest
Thanks! That looks correct, and I think that as long as we are not doing worse than Podman 5.3, that’s ~by definition good enough, at least for the 5.4 release branch. I just don’t want to take on more hard commitments for the next ~2 weeks.
