cc-oci-runtime
cc-oci-runtime copied to clipboard
intel
qemu segfault (free(): invalid pointer) running dnf install
When dnf is used to install a pakcage in a container it breaks all the container.
sudo docker run -ti fedora:25 dnf install rpm
BDB0126 mmap: Invalid argument
/usr/libexec/cc-shim: EOF received on proxy I/O fd
qemu-lite 2.7.0 docker run fedora:25 dnf install rpm
BDB0126 mmap: Invalid argument
*** Error in `/usr/bin/qemu-lite-system-x86_64': free(): invalid pointer: 0x000055a3d9504893 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x7c89b)[0x7fd21082589b]
/usr/lib64/libc.so.6(+0x86a1a)[0x7fd21082fa1a]
/usr/lib64/libc.so.6(cfree+0x5c)[0x7fd2108336fc]
/usr/bin/qemu-lite-system-x86_64(+0x385de6)[0x55a3d950cde6]
/usr/bin/qemu-lite-system-x86_64(+0x2680de)[0x55a3d93ef0de]
/usr/bin/qemu-lite-system-x86_64(+0x29f2e7)[0x55a3d94262e7]
/usr/bin/qemu-lite-system-x86_64(+0x53d9a7)[0x55a3d96c49a7]
/usr/bin/qemu-lite-system-x86_64(+0x52ac6e)[0x55a3d96b1c6e]
/usr/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x282)[0x7fd210b309e2]
/usr/bin/qemu-lite-system-x86_64(+0x53b2a2)[0x55a3d96c22a2]
/usr/bin/qemu-lite-system-x86_64(+0x1e6464)[0x55a3d936d464]
/usr/lib64/libc.so.6(__libc_start_main+0xf1)[0x7fd2107c95d1]
/usr/bin/qemu-lite-system-x86_64(+0x1e8fda)[0x55a3d936ffda]
======= Memory map: ========
55a3d9187000-55a3d990d000 r-xp 00000000 08:03 5972169 /usr/bin/qemu-lite-system-x86_64
55a3d990e000-55a3d9ac2000 r--p 00786000 08:03 5972169 /usr/bin/qemu-lite-system-x86_64
55a3d9ac2000-55a3d9b1d000 rw-p 0093a000 08:03 5972169 /usr/bin/qemu-lite-system-x86_64
55a3d9b1d000-55a3d9f4e000 rw-p 00000000 00:00 0
55a3da209000-55a3dade8000 rw-p 00000000 00:00 0 [heap]
7fd16c000000-7fd16c021000 rw-p 00000000 00:00 0
7fd16c021000-7fd170000000 ---p 00000000 00:00 0
7fd173e00000-7fd1f3e00000 rw-p 00000000 00:00 0
7fd1f3e00000-7fd1f3e01000 ---p 00000000 00:00 0
7fd1f4000000-7fd1f4021000 rw-p 00000000 00:00 0
7fd1f4021000-7fd1f8000000 ---p 00000000 00:00 0
7fd1f8000000-7fd1f8022000 rw-p 00000000 00:00 0
7fd1f8022000-7fd1fc000000 ---p 00000000 00:00 0
7fd1fc000000-7fd1fc748000 rw-p 00000000 00:00 0
7fd1fc748000-7fd200000000 ---p 00000000 00:00 0
7fd200a00000-7fd200a40000 rw-p 00000000 00:00 0
7fd200a40000-7fd200a41000 ---p 00000000 00:00 0
7fd200bba000-7fd200bda000 rw-s 00000000 00:05 90543 /dev/zero (deleted)
7fd200bda000-7fd200bdb000 ---p 00000000 00:00 0
7fd200bdb000-7fd200d0a000 rw-p 00000000 00:00 0
7fd200d19000-7fd200d59000 rw-s 00000000 00:05 90542 /dev/zero (deleted)
7fd200d59000-7fd200d5a000 ---p 00000000 00:00 0
7fd200d5a000-7fd200d5b000 ---p 00000000 00:00 0
7fd200d5b000-7fd20155b000 rw-p 00000000 00:00 0
7fd20155b000-7fd20155c000 ---p 00000000 00:00 0
7fd20155c000-7fd201e1f000 rw-p 00000000 00:00 0
7fd201e1f000-7fd20ff1f000 rw-p 00000000 08:03 5898297 /usr/share/clear-containers/clear-13280-containers.img
7fd20ff1f000-7fd20ff20000 ---p 00000000 00:00 0
7fd20ff20000-7fd20ff21000 ---p 00000000 00:00 0
7fd20ff21000-7fd210726000 rw-p 00000000 00:00 0
7fd210726000-7fd2107a6000 r-xp 00000000 08:03 5915433 /usr/lib64/libpcre.so.1.2.8
7fd2107a6000-7fd2107a7000 ---p 00080000 08:03 5915433 /usr/lib64/libpcre.so.1.2.8
7fd2107a7000-7fd2107a8000 r--p 00080000 08:03 5915433 /usr/lib64/libpcre.so.1.2.8
7fd2107a8000-7fd2107a9000 rw-p 00081000 08:03 5915433 /usr/lib64/libpcre.so.1.2.8
7fd2107a9000-7fd21096c000 r-xp 00000000 08:03 5927243 /usr/lib64/libc-2.25.so
7fd21096c000-7fd21096d000 ---p 001c3000 08:03 5927243 /usr/lib64/libc-2.25.so
7fd21096d000-7fd210971000 r--p 001c3000 08:03 5927243 /usr/lib64/libc-2.25.so
7fd210971000-7fd210973000 rw-p 001c7000 08:03 5927243 /usr/lib64/libc-2.25.so
7fd210973000-7fd210977000 rw-p 00000000 00:00 0
7fd210977000-7fd210991000 r-xp 00000000 08:03 5927315 /usr/lib64/libpthread-2.25.so
7fd210991000-7fd210992000 r--p 00019000 08:03 5927315 /usr/lib64/libpthread-2.25.so
7fd210992000-7fd210993000 rw-p 0001a000 08:03 5927315 /usr/lib64/libpthread-2.25.so
7fd210993000-7fd210997000 rw-p 00000000 00:00 0
7fd210997000-7fd2109ae000 r-xp 00000000 08:03 5908101 /usr/lib64/libgcc_s.so.1
7fd2109ae000-7fd2109af000 r--p 00016000 08:03 5908101 /usr/lib64/libgcc_s.so.1
7fd2109af000-7fd2109b0000 rw-p 00017000 08:03 5908101 /usr/lib64/libgcc_s.so.1
7fd2109b0000-7fd210ac4000 r-xp 00000000 08:03 5927307 /usr/lib64/libm-2.25.so
7fd210ac4000-7fd210ac5000 r--p 00113000 08:03 5927307 /usr/lib64/libm-2.25.so
7fd210ac5000-7fd210ac6000 rw-p 00114000 08:03 5927307 /usr/lib64/libm-2.25.so
7fd210ac6000-7fd210acd000 r-xp 00000000 08:03 5927266 /usr/lib64/librt-2.25.so
7fd210acd000-7fd210ace000 r--p 00006000 08:03 5927266 /usr/lib64/librt-2.25.so
7fd210ace000-7fd210acf000 rw-p 00007000 08:03 5927266 /usr/lib64/librt-2.25.so
7fd210acf000-7fd210ad1000 rw-p 00000000 00:00 0
7fd210ad1000-7fd210c18000 r-xp 00000000 08:03 5921771 /usr/lib64/libglib-2.0.so.0.5000.3
7fd210c18000-7fd210c19000 r--p 00146000 08:03 5921771 /usr/lib64/libglib-2.0.so.0.5000.3
7fd210c19000-7fd210c1a000 rw-p 00147000 08:03 5921771 /usr/lib64/libglib-2.0.so.0.5000.3
7fd210c1a000-7fd210c24000 r-xp 00000000 08:03 5943262 /usr/lib64/libnuma.so.1.0.0
7fd210c24000-7fd210c25000 ---p 0000a000 08:03 5943262 /usr/lib64/libnuma.so.1.0.0
7fd210c25000-7fd210c26000 r--p 0000a000 08:03 5943262 /usr/lib64/libnuma.so.1.0.0
7fd210c26000-7fd210c27000 rw-p 0000b000 08:03 5943262 /usr/lib64/libnuma.so.1.0.0
7fd210c27000-7fd210c29000 r-xp 00000000 08:03 5927253 /usr/lib64/libutil-2.25.so
7fd210c29000-7fd210c2a000 ---p 00002000 08:03 5927253 /usr/lib64/libutil-2.25.so
7fd210c2a000-7fd210c2b000 r--p 00002000 08:03 5927253 /usr/lib64/libutil-2.25.so
7fd210c2b000-7fd210c2c000 rw-p 00003000 08:03 5927253 /usr/lib64/libutil-2.25.so
7fd210c2c000-7fd210d5a000 r-xp 00000000 08:03 5908613 /usr/lib64/libpixman-1.so.0.34.0
7fd210d5a000-7fd210d5b000 ---p 0012e000 08:03 5908613 /usr/lib64/libpixman-1.so.0.34.0
7fd210d5b000-7fd210d63000 r--p 0012e000 08:03 5908613 /usr/lib64/libpixman-1.so.0.34.0
7fd210d63000-7fd210d64000 rw-p 00136000 08:03 5908613 /usr/lib64/libpixman-1.so.0.34.0
7fd210d64000-7fd210d92000 r-xp 00000000 08:03 5909175 /usr/lib64/libz.so.1.2.8
7fd210d92000-7fd210d93000 r--p 0002d000 08:03 5909175 /usr/lib64/libz.so.1.2.8
7fd210d93000-7fd210d94000 rw-p 0002e000 08:03 5909175 /usr/lib64/libz.so.1.2.8
7fd210da0000-7fd210da1000 rw-p 00000000 00:00 0
7fd210da1000-7fd210da4000 rw-s 00000000 00:0c 12074 anon_inode:kvm-vcpu
7fd210da4000-7fd210da7000 rw-s 00000000 00:0c 12074 anon_inode:kvm-vcpu
7fd210da7000-7fd210da8000 rw-s 00000000 00:05 89707 /dev/zero (deleted)
7fd210da8000-7fd210dab000 rw-s 00000000 00:05 91202 /dev/zero (deleted)
7fd210dab000-7fd210dad000 rw-p 00000000 00:00 0
7fd210dad000-7fd210dd3000 r-xp 00000000 08:03 5927286 /usr/lib64/ld-2.25.so
7fd210dd3000-7fd210dd4000 rw-s 00000000 00:05 89706 /dev/zero (deleted)
7fd210dd4000-7fd210dd5000 r--p 00026000 08:03 5927286 /usr/lib64/ld-2.25.so
7fd210dd5000-7fd210dd7000 rw-p 00027000 08:03 5927286 /usr/lib64/ld-2.25.so
7ffc28d4d000-7ffc28d6e000 rw-p 00000000 00:00 0 [stack]
7ffc28dc8000-7ffc28dca000 r--p 00000000 00:00 0 [vvar]
7ffc28dca000-7ffc28dcc000 r-xp 00000000 00:00 0 [vdso]
/usr/libexec/cc-shim: EOF received on proxy I/O fd
A few more data points:
-
dnf search foo: crash. -
dnf list: crash. -
rpm -qa: OK. -
dnf --version: Error, but no crash.1.1.10 BDB0126 mmap: Invalid argument Failed loading RPMDB. - behaviour is consistent in
fedora:25,fedora:24andfedora:23. - I tried installing strace in a container, committing, then running
strace dnf ...but no longer see the issue.
I suspect this is not the same as the 9p stuff we've seen in the past affecting apt-get - but, just as a test, you could try mounting /tmp onto a ramfs to remove 9p from the equation there - that fixed the 9p issues before... (but @jodh-intel knows that already ;-)
Looks like we are indeed back in 9p territory...
Incorrect behaviour:
$ sudo docker run -ti fedora bash -c 'dnf install foo'
BDB0126 mmap: Invalid argument
/usr/libexec/cc-shim: EOF received on proxy I/O fd
But with a non-9p /tmp, dnf's behaviour is correct (although we still see the mmap error):
$ sudo docker run -ti fedora bash -c 'mount -t ramfs -o size=20M ramfs /tmp && dnf install foo'
BDB0126 mmap: Invalid argument
Fedora 25 - x86_64 - Updates 304 kB/s | 18 MB 01:02
Fedora 25 - x86_64 353 kB/s | 50 MB 02:26
Last metadata expiration check: 0:01:21 ago on Tue Feb 21 08:43:27 2017.
No package foo available.
Error: Unable to find a match.
Anyone knows if this is also happening with debian based distros? what about doing apt-get install ?
I think the P2 label is valid: it should not be possible for an arbitrary command (particularly a commonly-used one like dnf for Fedora) to kill the hypervisor.
It's not a P1 for 2 reasons:
- It's still possible to create and start a Fedora container (with the caveat that you cannot run
dnf install). - There is a workaround for the issue.
@anthonyzxu - this issue has been open for almost 2 months now: could you give an update please?
After merge the support to use devicemapper volumes as storage device , this issue only happen in non-devicemapper rootfs. It is recommended to use devicemapper for a better filesystem compliance.
Sure, though this is still an open 9p issue for all other systems, and likely if we mount a volume still using 9p.
On Jun 22, 2017, at 12:10 PM, Jose Carlos Venegas Munoz <[email protected]mailto:[email protected]> wrote:
After merge the support to use devicemapper volumes as storage device , this issue only happen in non-devicemapper rootfs. It is recommended to use devicemapper for a better filesystem compliance.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/01org/cc-oci-runtime/issues/669#issuecomment-310474286, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATPXWwYlyPaxDIANduxTLKfgyLgDsYuXks5sGrwGgaJpZM4MCKIF.
As pointed out in https://github.com/01org/cc-oci-runtime/issues/949, this also seems an mmap related issue. Please try out with mount -t 9p -o trans=virtio test_mount /tmp/shared/ -oversion=9p2000.L,cache=fscache
Cache=fscache is the main option that needs to be there if you want mmap write to be working.
