vscode-dev-containers
vscode-dev-containers copied to clipboard
microsoft
amd64 emulation with QEmu on Mac OS M1
Issue: When trying to run an x86_64 devcontainer in an M1, the container is running a x86_64 machine but with CPU in 32-bit op-mode, when it should be 64-bit op-mode.
- VSCode Version: 1.69.2
- Local OS Version: Mac OS 12.4
- Local chip architecture: arm64 (M1)
- Reproduces in: Remote - Containers
- Name of Dev Container Definition with Issue:
Steps to Reproduce:
- Find a Mac M1 machine.
-
git clone [email protected]:Findeton/dev-container-mac-issue.git - Open the repo in VS Code Remote - Containers
- Rebuild Container.
-
lscpu
lscpuwill show:
root ➜ /workspaces/dev-container-mac-issue (main) $ lscpu
Architecture: x86_64
**CPU op-mode(s): 32-bit**
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: 0x00
Model: 0
Stepping: 0x0
BogoMIPS: 48.00
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilr
cpc flagm sb paca pacg dcpodp flagm2 frint
The container is running a x86_64 machine but with CPU in 32-bit op-mode, when it should be 64-bit op-mode.
More details
When looking at the Devcontainer logs, I can see that the docker buildx command doesn't include the --platform linux/amd64argument, it's only included later on thedocker runcommand. I've tried including the --platform linux/amd64argument in the Dockerfile and in the runArgs key in devcontainer.json.
This is the start of the Dev Containers log, notice the Warning on the last line. Also notice the platform argument is not on the docker buildx command:
[28 ms] Remote-Containers 0.243.0 in VS Code 1.69.2 (3b889b090b5ad5793f524b5d1d39fda662b96a2a).
[27 ms] Start: Resolving Remote
[33 ms] Setting up container for folder or workspace: /Users/felixrobles/workspace/dev-container-mac-issue
[46 ms] Start: Check Docker is running
[46 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[134 ms] Server API version: 1.41
[134 ms] Start: Run: docker volume ls -q
[216 ms] Start: Run: docker inspect --type container d81088e17e9b50322f376a980d39f17f830a0e9438d78b9d2a5ce6c1c0d219e1
[289 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=/Users/felixrobles/workspace/dev-container-mac-issue --filter label=vsch.quality=stable
[358 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/Users/felixrobles/workspace/dev-container-mac-issue
[424 ms] Start: Run: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Users/felixrobles/.vscode/extensions/ms-vscode-remote.remote-containers-0.243.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/felixrobles/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/felixrobles/workspace/dev-container-mac-issue --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/felixrobles/workspace/dev-container-mac-issue --log-level debug --log-format json --config /Users/felixrobles/workspace/dev-container-mac-issue/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[524 ms] remote-containers 0.243.0.
[524 ms] Start: Run: docker buildx version
[794 ms] Start: Resolving Remote
[796 ms] Start: Run: git rev-parse --show-cdup
[805 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/Users/felixrobles/workspace/dev-container-mac-issue
[877 ms] local container features stored at: /Users/felixrobles/.vscode/extensions/ms-vscode-remote.remote-containers-0.243.0/dist/node_modules/vscode-dev-containers/container-features
[878 ms] Start: Run: tar --no-same-owner -x -f -
[888 ms] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /Users/felixrobles/workspace/dev-container-mac-issue/.devcontainer/Dockerfile -t vsc-dev-container-mac-issue-876c563b5845b91597c09f3cf3de0bff /Users/felixrobles/workspace/dev-container-mac-issue/.devcontainer
[+] Building 0.2s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 31B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/b 0.1s
=> CACHED [1/1] FROM mcr.microsoft.com/vscode/devcontainers/base:bullsey 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:31e7220b4132746a5915f2afccf004ee2a235fdd13acc 0.0s
=> => naming to docker.io/library/vsc-dev-container-mac-issue-876c563b58 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
[1568 ms] Start: Run: docker events --format {{json .}} --filter event=start
[1570 ms] Start: Starting container
[1570 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=/Users/felixrobles/workspace/dev-container-mac-issue,target=/workspaces/dev-container-mac-issue,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=/Users/felixrobles/workspace/dev-container-mac-issue -e PROJECT_DIR=/workspaces/dev-container-mac-issue --entrypoint /bin/sh vsc-dev-container-mac-issue-876c563b5845b91597c09f3cf3de0bff -c echo Container started
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
@Findeton Docker's x86 emulation on Macs is "best effort" according to docs - reporting available op modes may be a limitation of their emulation. Normally you'd see a comma separated list like "CPU op-mode(s): 32-bit, 64-bit" showing what the CPU can do (not the mode it is actually in per-se). By adding --platform linux/amd64 in your Dockerfile, only the arm64 image is pulled, and the image in your sample is not even built for 32-bit. Either way if it was running 32-bit mode, the arm64 binaries would not function.
uname -a should tell you about the host VM including the running architecture.
Are there other issues indicating that it is in 32-bit mode?
@Chuxel So if you open this repo https://github.com/sequentech/strand/ as a dev container from an aarch64 machine with the first line of the Dockerfilemodified to include --platform linux/amd64, it fails while running nix build (which is an automatic step when opening it as a dev container) with this error:
strand-wasm> Compiling wasm-bindgen-shared v0.2.81
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x4d4713)[0x400217f713]
strand-wasm> /nix/store/mhhlymrg2m70r8h94cwhv2d7a0c8l7g6-glibc-2.34-210/lib/libc.so.6(+0x3d0e0)[0x40060aa0e0]
strand-wasm> rustc(+0x32ddf)[0x4000032ddf]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/libLLVM-14-rust-1.62.0-nightly.so(_ZN4llvm9MCContext5resetEv+0x56c)[0x400941fdcc]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/libLLVM-14-rust-1.62.0-nightly.so(_ZN4llvm28MachineModuleInfoWrapperPass14doFinalizationERNS_6ModuleE+0x35)[0x400941f825]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/libLLVM-14-rust-1.62.0-nightly.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0xb0)[0x4009534ed0]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x2524b44)[0x40041cfb44]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x251e4d7)[0x40041c94d7]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x25217a6)[0x40041cc7a6]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x24c0938)[0x400416b938]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x24ba66a)[0x400416566a]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x24a8e5f)[0x4004153e5f]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/librustc_driver-4dc15ebd8072015a.so(+0x250b577)[0x40041b6577]
strand-wasm> /nix/store/529yjy862f6bzlw8jcwyw7s3jbmxq3wb-rust-default-1.62.0-nightly-2022-04-07/lib/libstd-1b64d5fe7a3c3d7f.so(rust_metadata_std_928ba903c221ea94+0xab453)[0x4005d97453]
strand-wasm> /nix/store/mhhlymrg2m70r8h94cwhv2d7a0c8l7g6-glibc-2.34-210/lib/libc.so.6(+0x85ff2)[0x40060f2ff2]
strand-wasm> /nix/store/mhhlymrg2m70r8h94cwhv2d7a0c8l7g6-glibc-2.34-210/lib/libc.so.6(clone+0x40)[0x4006174fb0]
strand-wasm> qemu: uncaught target signal 11 (Segmentation fault) - core dumped
strand-wasm> error: could not compile `typenum`
strand-wasm> Caused by:
strand-wasm> process didn't exit successfully: `rustc --crate-name build_script_main --edition=2018 /tmp/nix-shell.7tMuuG/nix-build-strand-wasm-0.0.1.drv-0/cargo-vendor-dir/typenum-1.15.0/build/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=8d91f4b180671e4b -C extra-filename=-8d91f4b180671e4b --out-dir /tmp/nix-shell.7tMuuG/nix-build-strand-wasm-0.0.1.drv-0/6wwl6ikjfm4ksh6fhcb12iqcf77116xl-source/target/release/build/typenum-8d91f4b180671e4b -C linker=/nix/store/6mx62f4mdv5y1awi7y2mp23a76gg0ba7-gcc-wrapper-11.3.0/bin/cc -L dependency=/tmp/nix-shell.7tMuuG/nix-build-strand-wasm-0.0.1.drv-0/6wwl6ikjfm4ksh6fhcb12iqcf77116xl-source/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
strand-wasm> warning: build failed, waiting for other jobs to finish...
strand-wasm> Error: Compiling your crate to WebAssembly failed
strand-wasm> Caused by: failed to execute `cargo build`: exited with exit status: 101
strand-wasm> full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown" "--features=wasmtest"
error: builder for '/nix/store/87zj3r80ryl20giwbisvlz62ifljxxnj-strand-wasm-0.0.1.drv' failed with exit code 1
This seems to indicate that building libc failed with a SIGSEGV: invalid memory reference, which may indicate trying to use 64bits running on a 32 bit op mode.
I haven't had time to further isolate the issue, so the error might be indicating something else. However the dev container/nix build seems to work when the host system and container are both either x86_64 or arch64, the issue seems to happen when emulating x86_64 from an aarch64 host.
Yeah seg faults are one of the more common issues I've seen with emulation - notably there is an issue in libssl on Ubuntu 20.04, Debian 10, REHL 8, and several others that looks like this. That's why we pushed everyone on arm64 to bullseye.
I am guessing this will repro when you do all this from the command line with docker and use the debian:bullseye base image rather than the one here. Assuming so, raising an issue at https://github.com/docker/for-mac/issues is likely the way to go. The extension does not control the op mode here - Docker Desktop does. I unfortunately do not have access to my M1 mac at the moment, so I can't check (and like you said x86_64 Macs work).
A solution that works for me on a similar problem was to create an X86_64 docker runtime environment using colima, and then run the devcontainers on that docker context.
Related issue in Docker repo: https://github.com/docker/for-mac/issues/6796, no solutions there though.
@codeprefect, sounds interesting, could you elaborate a bit?
Related issue in Docker repo: https://github.com/docker/for-mac/issues/6796, no solutions there though.
@codeprefect, sounds interesting, could you elaborate a bit?
I created a QEMU powered Docker Host using Colima, and then set it to the active Docker context. VS Code dev container chooses the Docker Host OS runtime as the container runtime by default, so it will see it as X86_64 container on X86_64 host. The virtualization is now hidden between the host machine, and the docker host VM instead of the docker host VM and the container.
Thanks @codeprefect it worked :)
I'm on Apple Silicon and Ventura 13.3.1, colima version v0.5.4.
Here's what I did:
$ colima start --arch x86_64 --cpu 4 --memory 16
$ colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/kblazewicz/.colima/default/docker.sock
$ docker run -it --rm debian:bullseye
And the result:
root@66cef63d7ae7:/# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 40 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 15
Model: 107
Model name: QEMU Virtual CPU version 2.5+
Stepping: 1
CPU MHz: 999.929
BogoMIPS: 1999.85
Virtualization: AMD-V
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl cpuid extd_apicid pni cx16 hype
rvisor lahf_lm cmp_legacy svm 3dnowprefetch vmmcall
Just to verify tried colima with virtualization.
Using rosetta:
$ colima start --arch aarch64 --vm-type=vz --vz-rosetta
$ colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/kblazewicz/.colima/default/docker.sock
$ docker run -it --rm --platform=linux/amd64 debian:bullseye
result:
root@e9e644fabd61:/# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: 0x00
Model: 0
Stepping: 0x0
BogoMIPS: 48.00
NUMA node0 CPU(s): 0,1
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ss
bs sb paca pacg dcpodp flagm2 frint
Using QEMU:
$ colima start --cpu 4 --memory 16
$ colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/kblazewicz/.colima/default/docker.sock
$ docker run -it --rm --platform=linux/amd64 debian:bullseye
Same result:
root@a2cee8435958:/# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: 0x00
Model: 0
Stepping: 0x0
BogoMIPS: 48.00
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb
paca pacg dcpodp flagm2 frint
Yes, both seems to work in your case. There is still some obvious feature gaps in the two virtualization modes. I have found some containers to work in X86_64 mode on an aarch64 docker vm too, an example is Microsoft SQL Server, and I have also found some that will only work when the docker vm itself is also X86_64, a good example of that is the Oracle Database servers. My approach is to test on aarch64 first due to the low resource utilization, and only use X86_64 as a last resort.
@codeprefect did you also notice this issue: https://github.com/microsoft/vscode-remote-release/issues/8541 while running x86_64 images on aarch64 vm?
@codeprefect did you also notice this issue: microsoft/vscode-remote-release#8541 while running
x86_64images onaarch64vm?
I did not notice that, so I can't say for sure.