docker
fatal error reported: Linux kernel v6.10 crash on virtualization.framework
Description
Docker Desktop consistently crashes while doing Milvus build in a container.
Reproduce
git clone https://github.com/milvus-io/milvus.git
cd milvus
docker run -it -v $(pwd):/root/milvus -v ~/.conan:/root/.conan -w /root/milvus milvusdb/milvus-env:ubuntu22.04-20240805-d8668fe bash
make milvus
Expected behavior
No response
docker version
Client:
Version: 27.1.1
API version: 1.46
Go version: go1.21.12
Git commit: 6312585
Built: Tue Jul 23 19:54:12 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.33.0 (160616)
Engine:
Version: 27.1.1
API version: 1.46 (minimum version 1.24)
Go version: go1.21.12
Git commit: cc13f95
Built: Tue Jul 23 19:57:14 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.19
GitCommit: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc:
Version: 1.7.19
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 27.1.1
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.1-desktop.1
Path: /Users/sanjayk/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.1-desktop.1
Path: /Users/sanjayk/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.34
Path: /Users/sanjayk/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.14
Path: /Users/sanjayk/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/sanjayk/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.25
Path: /Users/sanjayk/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/sanjayk/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.3.0
Path: /Users/sanjayk/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/sanjayk/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.11.0
Path: /Users/sanjayk/.docker/cli-plugins/docker-scout
Server:
Containers: 14
Running: 4
Paused: 0
Stopped: 10
Images: 23
Server Version: 27.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.0-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 10
Total Memory: 31.54GiB
Name: docker-desktop
ID: 7d95379d-2ef7-4818-9f48-e76355b18900
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/sanjayk/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
DA486BA7-7F7A-4F44-B054-B8D52CC67A6C/20240812210143
Additional Info
Docker Desktop Version 4.33.0 (160616)
Running on MacBook Pro (M1 Max), 64GB, macOS 14.5
fatal error reported: Linux kernel v6.10 crash on virtualization.framework
[ 3940.510957] Internal error: Oops: 0000000086000005 [#1] SMP
[ 3940.511011] Modules linked in: shiftfs(O) selfowner(O) rosetta(O) grpcfuse(O) fakeowner(O)
[ 3940.511054] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G O 6.10.0-linuxkit #1
[ 3940.511131] pstate: 814000c5 (Nzcv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 3940.511184] pc : 0x100000000
[ 3940.511224] lr : __hrtimer_run_queues+0x13c/0x308
[ 3940.511267] sp : ffff00081b9afc20
[ 3940.511302] x29: ffff00081b9afc40 x28: ffff00081b99e500 x27: ffff00081b99e4c0
[ 3940.511361] x26: 0000000100000000 x25: ffff800081d3a008 x24: 00000000000000c0
[ 3940.511442] x23: ffff800081d3a008 x22: ffff8000821ca790 x21: ffff00081b99e520
[ 3940.511519] x20: ffff00081b99e4c0 x19: ffff0000c2777d70 x18: 0000000000000000
[ 3940.511601] x17: ffff800799c60000 x16: ffff00081b9abdc0 x15: 0000ffff9c468938
[ 3940.511635] x14: 0000000000000000 x13: 0000000000000000 x12: ffff8000821ca7d0
[ 3940.511703] x11: 0000000000000040 x10: ffff0000c0166468 x9 : ffff800080104bb0
[ 3940.511777] x8 : ffff0000c04004b8 x7 : 0000000000000000 x6 : 0000000000000000
[ 3940.511837] x5 : ffff0000c0400490 x4 : 0000000000000000 x3 : 0000000000000001
[ 3940.511933] x2 : 00000000000000c0 x1 : 0000000000000000 x0 : ffff0000c2777d70
[ 3940.512047] Call trace:
[ 3940.512090] 0x100000000
[ 3940.512113] hrtimer_run_queues+0xd0/0x128
[ 3940.512180] update_process_times+0x3c/0x160
[ 3940.512221] tick_nohz_lowres_handler+0xe4/0x170
[ 3940.512254] arch_timer_handler_virt+0x34/0x58
[ 3940.512308] handle_percpu_devid_irq+0x90/0x1c8
[ 3940.512364] generic_handle_domain_irq+0x34/0x58
[ 3940.512413] gic_handle_irq+0x5c/0xd8
[ 3940.512457] call_on_irq_stack+0x24/0x58
[ 3940.512504] do_interrupt_handler+0x88/0x98
[ 3940.512538] el1_interrupt+0x34/0x58
[ 3940.512577] el1h_64_irq_handler+0x18/0x28
[ 3940.512609] el1h_64_irq+0x64/0x68
[ 3940.512642] do_idle+0x128/0x240
[ 3940.512679] cpu_startup_entry+0x40/0x50
[ 3940.512739] secondary_start_kernel+0x11c/0x148
[ 3940.512821] __secondary_switched+0xb8/0xc0
[ 3940.512849] Code: ???????? ???????? ???????? ???????? (????????)
[ 3940.512915] ---[ end trace 0000000000000000 ]---
I've also started randomly getting this issue, it's impossible to work locally with Docker Desktop for Mac now...
Hi @bsousaa, what's the approx. ETA to have this issue resolved? In the meantime, is there any workaround that you recommend? Thanks!
I had the same issue with Mac M3. Please help resolve this or provide a workaround solution.
The issue was resolved after downgrading Docker Desktop to 4.29.0!
the regression is is in between 4.32.0 (157355) and 4.33.0 (160616)
As a workaround for >4.32 don't use VirtioFS
General -> Choose file sharing implementation for your containers
VirtioFS - regression
gRPC FUSE - ok
osxfs (Legacy) - ok
This started to happen recently to me as well on my Mac M3 today (docker desktop 4.44.3)
fatal error reported: Linux kernel v6.10.14 crash on virtualization.framework
[ 5869.074277] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
[ 5869.074316] Modules linked in: shiftfs(O) selfowner(O) rosetta(O) grpcfuse(O) fakeowner(O)
[ 5869.074411] CPU: 2 PID: 218 Comm: init Tainted: G W O 6.10.14-linuxkit #1
[ 5869.074452] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 5869.074489] pc : generic_shutdown_super+0x100/0x110
[ 5869.074542] lr : generic_shutdown_super+0x100/0x110
[ 5869.074581] sp : ffff0000c2823d30
[ 5869.074609] x29: ffff0000c2823d30 x28: ffff0000c4401040 x27: 0000000000000000
[ 5869.074641] x26: 0000000000000000 x25: 0000000000000000 x24: ffff8000826283b0
[ 5869.074696] x23: 0000000000000000 x22: ffff0000c4401040 x21: 000000000000002e
[ 5869.074746] x20: ffff80007a70e318 x19: ffff0000cc934800 x18: ffffffffffffffff
[ 5869.074782] x17: 20657375206e6920 x16: 6c6c69747320207d x15: 726576726573736c
[ 5869.074826] x14: 0000000000000000 x13: 2972656e776f656b x12: 6166282072656e77
[ 5869.074871] x11: 6f656b616620666f x10: ffff800082406d70 x9 : ffff8000800c18b0
[ 5869.074926] x8 : 00000000ffffefff x7 : ffff8000823f1218 x6 : 0000000000000e79
[ 5869.074979] x5 : 0000000000000e7a x4 : 40000000fffffe79 x3 : 0000000000000000
[ 5869.075019] x2 : 0000000000000000 x1 : ffff0000c4401040 x0 : 0000000000000037
[ 5869.075057] Call trace:
[ 5869.075066] generic_shutdown_super+0x100/0x110
[ 5869.075088] kill_anon_super+0x28/0xf0
[ 5869.075123] deactivate_locked_super+0x50/0x1d8
[ 5869.075156] deactivate_super+0x8c/0xb0
[ 5869.075194] cleanup_mnt+0xa4/0x140
[ 5869.075231] __cleanup_mnt+0x1c/0x30
[ 5869.075255] task_work_run+0x90/0x120
[ 5869.075281] do_notify_resume+0x144/0x168
[ 5869.075314] el0_svc+0x84/0x98
[ 5869.075329] el0t_64_sync_handler+0x120/0x130
[ 5869.075354] el0t_64_sync+0x190/0x198
[ 5869.075378] Code: b0009300 91212000 f9400042 97f601a1 (d4210000)
[ 5869.075412] ---[ end trace 0000000000000000 ]---
I start docker for desktop and run a few containers for some time, when I stop all of them, maybe a minute later I get this error.
it happened to me today after upgrading to latest version of docker desktop
I changed from Apple Virtualization Framework to Docker VMM in General -> Virtual Machine Options and it solved for me at least for now.
Hello,
We're getting the same issue that the moment, and can clearly reproduce it, always. It always happen when running the same program inside the container.
I tried going from Apple to VMM, no change for me.
It happens always when using USB device.
Diag ID : 19EC1F29-78F4-416A-9916-ED802714138A/20251021122411
Getting :
fatal error reported: Linux kernel v6.10.14 crash on virtualization.framework [52371.919487] Unable to handle kernel paging request at virtual address ffffffffc0000000 [52371.919528] Mem abort info: [52371.919545] ESR = 0x0000000096000146 [52371.919561] EC = 0x25: DABT (current EL), IL = 32 bits [52371.919578] SET = 0, FnV = 0 [52371.919608] EA = 0, S1PTW = 0 [52371.919622] FSC = 0x06: level 2 translation fault [52371.919655] Data abort info: [52371.919668] ISV = 0, ISS = 0x00000146, ISS2 = 0x00000000 [52371.919703] CM = 1, WnR = 1, TnD = 0, TagAccess = 0 [52371.919710] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [52371.919729] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000071d28000 [52371.919773] [ffffffffc0000000] pgd=0000000072623003, p4d=0000000072623003, pud=0000000072624003, pmd=0000000000000000
[52371.919845] Internal error: Oops: 00000000960INFO[52370] vhci_hcd: unlink->seqnum 6098 component=kmsg00146 [#1] SMP
[52371.919919] MINFO[52370] vhci_hcd: urb->status -104 component=kmsgodules linked in : selfowner(O) shiftfs(O) rosetta(O) grpcfuse(O) fakeowner(O)
[52371.920033] CPU: 6 PID: 51684 Comm: vhci_rx TINFO[52370] Unable to handle kernel paging request at virtual address ffffffffc0000000 component=kmsgainted: G > O 6.10.14-linuxkit #1 INFO[52370] Mem abort info: component=kmsg [52371.920181] p state: 41400005 (nZcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) [52371.920257] pINFO[52370] ESR = 0x0000000096000146 component=kmsgc : dcache_inval_poc+0x40/0x58 [52371.920368] lr : arch_sync_dmINFO[52370] EC = 0x25: DABT (current EL), IL = 32 bits component=kmsga_for_cpu+0x34/0x50 INFO[52370] SET = 0, FnV = 0 component=kmsg[52371.920578] sp : ffff00027f96bca0 [52371.920628] x29: ffff00027f96bca0 x28: 000000INFO[52370] EA = 0, S1PTW = 0 component=kmsg0000000000 x27: ffff0000c27cac10 [52371.920675] x26: 0000000000000000 x25: ffff00INFO[52370] FSC = 0x06: level 2 translation fault component=kmsg0225d84580 x24: 0000000000000000INFO[52370] Data abort info: component=kmsg [52371.920754] x23: 0000000000000002 x22: 0000000000000020 x21: 0000000000000000INFO[52370] ISV = 0, ISS = 0x00000146, ISS2 = 0x00000000 component=kmsg [52371.920853] x20: 000000000000INFO[52370] CM = 1, WnR = 1, TnD = 0, TagAccess = 0 component=kmsg0004 x19: 0000000000000000 x18: 00000000fffffffe INFO[52370] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 component=kmsg[52371.920949] x 17: 000000000000INFO[52370] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000071d28000 component=kmsg0000 x16: 000000 0000000000 x15: ffff00027f96b990 [52371.921001] xINFO[52370] [ffffffffc0000000] pgd=0000000072623003, p4d=0000000072623003, pud=0000000072624003, pmd=0000000000000000 component=kmsg14: 0000000000000000 x13: 3430312d20737574 x12: 6174733e2d627275 [52371.921076] x11: fffffffffffe0000 x10: ffff80 00823f1218 x9 : ffff8000800eccd4 [52371.921133] x8 : 00000000ffffefff x7 : ffff80INFO[52370] Internal error: Oops: 0000000096000146 [#1] SMP component=kmsg00823f1218 x6 : 80000000fffff000 [52371.921240] x5 : ffff000614cf0688 x4 : 0000000000000000 x3 : 000000000000003f [52371.921289] x2 : 0000000000000040 x1 : ffffffffc0000000 x0 : ffffffffc0000000 [52371.921309] Call trace: [52371.921386] dcache_inval_poc+0x40/0x58 [52371.921413] dma_direct_unmap_sg+0x1ac/0x1d0 [52371.921448] dma_unmap_sg_attrs+0x20/0x38 INFO[52370] Modules linked in: selfowner(O) shiftfs(O) rosetta(O) grpcfuse(O) fakeowner(O) component=kmsg[52371.921476]
usb_hcd_unmap_urb_for_dma+0x9c/0x118 INFO[52370] CPU: 6 PID: 51684 Comm: vhci_rx Tainted: G O 6.10.14-linuxkit #1 component=kmsg[52371.921529] __usb_hcd_giveba ck_urb+0x13c/0x1INFO[52370] pstate: 41400005 (nZcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) component=kmsg60 [52371.921608] usb_hcd_giveback_urb+0x150/0x178 [52371.921659] vhci_rx_loop+0x368/0x650 [52371.921699] kthread+0xe8/0xf8 [52371.921729] ret_from_fork+0xINFO[52370] pc : dcache_inval_poc+0x40/0x58 component=kmsg10/0x20 [52371.921779] Code: 8a230000 54000060 d50b7e20 14000002 (d5087620) [52371.921829] ---[ end trace 0000000000000000 ]---
