Unable to debug amd64 binaries on apple silicon

[Divix55]

Description

Hi! Im unable to debug binaries in amd64 docker container. Im using Docker Desktop for Mac with enabled rosetta emulation for x86/amd64. Im using centos7-based custom image for compilation & starting my application in C++. Application can be compiled and started within emulated container, but debugging with GDB is not working for some reason. Attempt of debugging result in error message:

Starting program: MyBinary
Couldn't read debug register: Input/output error.
Couldn't get registers: Input/output error.
Couldn't get registers: Input/output error.

Reproduce

1. Create centos7 based image "my-centos7-image" with following Dockerfile:

FROM centos:7

RUN yum install -y deltarpm openssl openssl-devel centos-release-scl && \
    sed -i 's/7/7.6.1810/g; s|^#\s*\(baseurl=http://\)mirror|\1vault|g; /mirrorlist/d' /etc/yum.repos.d/CentOS-SCLo-*.repo && \
    yum install -y gcc-4.8.5 gcc-c++-4.8.5 gdb 

2. Run docker image

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --security-opt apparmor=unconfined -it my-centos7-image

3. Compile any c++11 project which will produce MyBinary binary.
4. Start debugging binary with command gdb MyBinary
5. Use following gdb commands to reproduce issue:

b main
r

Expected behavior

Debugging will start without errors.

docker version

Client:
 Cloud integration: v1.0.35
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.21.1 (114176)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:50:59 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /Users/<username>/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /Users/<username>/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/<username>/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/<username>/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/<username>/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/<username>/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/<username>/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.16.1
    Path:     /Users/<username>/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 6
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: 3bc9cf6b-135c-43a3-bae2-30487b888921
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

F4037DED-224B-4C91-9F8E-B7EEBEFE3816/20230716092324

Additional Info

No response

[joleeee]

Having the same issue, did you find a workaround @Divix55

[Divix55]

@joleeee Unfortunately no, still not able to debug amd64 binaries within container

[loloxwg]

@Divix55 same issue, make me depressed

[Swchexit]

I've encountered similar problems, but not being able to find a workaround.

This ref specify the cause of this, but I cannot find a way to use it with docker.

[tj-oconnor]

Any resolution to this? I was hoping when Rosetta emulation was enabled it would fix it, but I haven’t found a solution yet

[dgageot]

Hello everyone! Do you have a simple repo that I can use to try? Have you tried with Rosetta enabled and with Rosetta disabled?

[mccrearyp]

I have the same issue. Disabling Rosetta just gives a ptrace error:

warning: Could not trace the inferior process. warning: ptrace: Function not implemented During startup program exited with code 127.

From what I read it sounds like disabling Rosetta causes a cutover to Qemu. Apparently there is a work around to the ptrace error here, but I didn't try it as it would over complicate my debugging workflow.

[Novebula]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

[mccrearyp]

Most of my debugging is done on automated test binaries that don't work well with this approach. Unfortunately I really need a workflow like this to work:

gdb ./my_binary
// Set things up like break points
run

[Francesco146]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

After i run target remote localhost:1234, i get:

localhost:1234: Cannot assign requested address.
(gdb) r
Starting program: /path/to/x86bin
warning: linux_ptrace_test_ret_to_nx: Cannot PTRACE_GETREGS: Input/output error
warning: linux_ptrace_test_ret_to_nx: PC 0x9ffffef30 is neither near return address 0x7ffff7669000 nor is the return instruction 0x55555592e021!
Couldn't get CS register: Input/output error.

[tang-hi]

Hope this helps! https://github.com/microsoft/vscode-cpptools/issues/10788#issuecomment-1954363023

[tang-hi]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

After i run target remote localhost:1234, i get:

localhost:1234: Cannot assign requested address.
(gdb) r
Starting program: /path/to/x86bin
warning: linux_ptrace_test_ret_to_nx: Cannot PTRACE_GETREGS: Input/output error
warning: linux_ptrace_test_ret_to_nx: PC 0x9ffffef30 is neither near return address 0x7ffff7669000 nor is the return instruction 0x55555592e021!
Couldn't get CS register: Input/output error.

try continue instead of r

[simo501]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

this works, but it's horrible to use gdb like this

[HamiLemon]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

After i run target remote localhost:1234, i get:

pwndbg> target remote localhost:1234 Remote debugging using localhost:1234 warning: remote target does not support file transfer, attempting to access files from local filesystem. Reading symbols from /lib64/ld-linux-x86-64.so.2... Reading symbols from /usr/lib/debug/.build-id/41/86944c50f8a32b47d74931e3f512b811813b64.debug... rosetta error: no handlers found for <Qqemu.sstepbits>, length=<15>!

Exception occurred: Error: Remote connection closed (<class 'gdb.error'>) For more info invoke set exception-verbose on and rerun the command or debug it by yourself with set exception-debugger on Exception occurred: Error: Remote connection closed (<class 'gdb.error'>) For more info invoke set exception-verbose on and rerun the command or debug it by yourself with set exception-debugger on Python Exception <class 'gdb.error'>: Remote connection closed /build/gdb-Dh0pdX/gdb-12.1/gdb/thread.c:85: internal-error: inferior_thread: Assertion `current_thread_ != nullptr' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. ----- Backtrace ----- 0x5555556efff7 ??? 0x555555a54a44 ??? 0x555555a54c80 ??? 0x555555ba40f4 ??? 0x555555a10aa1 ??? 0x555555a3873c ??? 0x55555585ef4b ??? 0x555555980ccd ??? 0x55555598107c ??? 0x555555981329 ??? 0x555555a0e4c8 ??? 0x555555725644 ??? 0x555555a1b364 ??? 0x5555557f38c4 ??? 0x5555557f3c63 ??? 0x5555557f43e6 ??? 0x7fffff799e0d ??? 0x5555557f29b5 ??? 0x5555557f4294 ??? 0x5555557f25ab ??? 0x555555ba4825 ??? 0x555555ba4cba ??? 0x5555558b02fc ??? 0x5555558b1fe4 ??? 0x55555564810f ??? 0x7ffffe716d8f __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 0x7ffffe716e3f __libc_start_main_impl ../csu/libc-start.c:392 0x55555564dba4 ??? 0xffffffffffffffff ???

This is a bug, please report it. For instructions, see: https://www.gnu.org/software/gdb/bugs/.

[1]+ Stopped ROSETTA_DEBUGSERVER_PORT=1234 ./a.out Aborted (core dumped)

[corneliusroemer]

I confirm, the workaround works with macOS 14.5 ARM:

$ docker run -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v ~/Downloads/Linux-X64-build/:/mnt dbg
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root@0f7df68f0a95:/workspace# ROSETTA_DEBUGSERVER_PORT=1234 /mnt/cmaple & gdb
[1] 9
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) set architecture i386:x86-64
The target architecture is set to "i386:x86-64".
(gdb) file /mnt/cmaple
Reading symbols from /mnt/cmaple...
(No debugging symbols found in /mnt/cmaple)
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: remote target does not support file transfer, attempting to access files from local filesystem.
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/3e/81740f816ee1521a87e439c16ebfde46f147a5.debug...
0x00007ffffffe5540 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) 

[tj-oconnor]

Thanks using gdb-multiarch meets some needs but doesn’t offer a solution for other tools (ltrace, ptrace) that rely on ptrace syscall emulation.I’ve ended up using Colima instead. Basically implements  a very lightweight amd64/linux VM to call docker so amd64/linux containers are emulated with an amd64/linux kernel.abiosoft/colima: Container runtimes on macOS (and Linux) with minimal setupgithub.comOn Jul 18, 2024, at 9:25 AM, Cornelius Roemer @.> wrote: I confirm, the workaround works with macOS 14.5 ARM: $ docker run -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v ~/Downloads/Linux-X64-build/:/mnt dbg WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested @.:/workspace# ROSETTA_DEBUGSERVER_PORT=1234 /mnt/cmaple & gdb [1] 9 GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word". (gdb) set architecture i386:x86-64 The target architecture is set to "i386:x86-64". (gdb) file /mnt/cmaple Reading symbols from /mnt/cmaple... (No debugging symbols found in /mnt/cmaple) (gdb) target remote localhost:1234 Remote debugging using localhost:1234 warning: remote target does not support file transfer, attempting to access files from local filesystem. Reading symbols from /lib64/ld-linux-x86-64.so.2... Reading symbols from /usr/lib/debug/.build-id/3e/81740f816ee1521a87e439c16ebfde46f147a5.debug... 0x00007ffffffe5540 in _start () from /lib64/ld-linux-x86-64.so.2 (gdb)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[1ikeadragon]

@tj-oconnor Hi, hope you're having a food day. Can u elaborate on your setup?

I am trying to set up a container with some CTF tools and gdb (pwndbg to be precise) for x86_64 Linux binaries. I used colima but it's kinda very slow. Even build times are crazy slow.

I'm on M1 Air. 8g ram.

https://github.com/1ikeadragon/pwntainer

This is what I'm using for now.

What tweaks do u think I should make?

[kjcolley7]

My workaround for this, after running container with CAP_PTRACE and unconfined seccomp, is to add this as /usr/local/bin/gdb (which is earlier in the PATH than the real GDB at /usr/bin/gdb):

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

if [ "$#" != "1" ]; then
	echo "Usage: $0 <path/to/program to debug>" >&2
	exit 1
fi

prog="$1"

# Start program in background
ROSETTA_DEBUGSERVER_PORT=1234 "$prog" &

# Run real gdb and tell it to attach
/usr/bin/gdb \
	-iex "set architecture i386:x86-64" \
	-iex "file $prog" \
	-iex "target remote localhost:1234" \
	-iex "set history save on"

It works surprisingly well, just do gdb ./myprog and it works like normal!

[dariomonopoli-dev]

did you find any fixes? I still have the issue.

[tj-oconnor]

I use Colima for building amd64 container runtimes now. Since it’s launching docker through an amd64 VM, all the syscalls (PTRACE, etc) work flawlessly. A little bit slower than running docker directly on apple silicon. But works well nonetheless.

[Divix55]

I use Colima for building amd64 container runtimes now. Since it’s launching docker through an amd64 VM, all the syscalls (PTRACE, etc) work flawlessly. A little bit slower than running docker directly on apple silicon. But works well nonetheless.

Could you share your args for colima start command? I would like to test it with CLion Docker Toolchain

[MuelNova]

Run the docker with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined Inside the container, execute command with env ROSETTA_DEBUGSERVER_PORT=XXXX and gdb with the port, and you'll find it OK.

ROSETTA_DEBUGSERVER_PORT=1234 ./my_binary & gdb
(gdb) set architecture i386:x86-64
(gdb) file my_binary
(gdb) target remote localhost:1234

After i run target remote localhost:1234, i get:

pwndbg> target remote localhost:1234 Remote debugging using localhost:1234 warning: remote target does not support file transfer, attempting to access files from local filesystem. Reading symbols from /lib64/ld-linux-x86-64.so.2... Reading symbols from /usr/lib/debug/.build-id/41/86944c50f8a32b47d74931e3f512b811813b64.debug... rosetta error: no handlers found for <Qqemu.sstepbits>, length=<15>!

Exception occurred: Error: Remote connection closed (<class 'gdb.error'>)

For more info invoke set exception-verbose on and rerun the command or debug it by yourself with set exception-debugger on Exception occurred: Error: Remote connection closed (<class 'gdb.error'>) For more info invoke set exception-verbose on and rerun the command or debug it by yourself with set exception-debugger on Python Exception <class 'gdb.error'>: Remote connection closed /build/gdb-Dh0pdX/gdb-12.1/gdb/thread.c:85: internal-error: inferior_thread: Assertion `current_thread_ != nullptr' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. ----- Backtrace ----- 0x5555556efff7 ??? 0x555555a54a44 ??? 0x555555a54c80 ??? 0x555555ba40f4 ??? 0x555555a10aa1 ??? 0x555555a3873c ??? 0x55555585ef4b ??? 0x555555980ccd ??? 0x55555598107c ??? 0x555555981329 ??? 0x555555a0e4c8 ??? 0x555555725644 ??? 0x555555a1b364 ??? 0x5555557f38c4 ??? 0x5555557f3c63 ??? 0x5555557f43e6 ??? 0x7fffff799e0d ??? 0x5555557f29b5 ??? 0x5555557f4294 ??? 0x5555557f25ab ??? 0x555555ba4825 ??? 0x555555ba4cba ??? 0x5555558b02fc ??? 0x5555558b1fe4 ??? 0x55555564810f ??? 0x7ffffe716d8f __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 0x7ffffe716e3f __libc_start_main_impl ../csu/libc-start.c:392 0x55555564dba4 ??? 0xffffffffffffffff ??? This is a bug, please report it. For instructions, see: https://www.gnu.org/software/gdb/bugs/.

[1]+ Stopped ROSETTA_DEBUGSERVER_PORT=1234 ./a.out Aborted (core dumped)

facing the same issue, it happens using pwndbg or gef, have you ever found the solution?

[tj-oconnor]

Yep. As I mentioned above - I use colima now to launch my amd64 containers that need access to complete syscalls. Since it’s launching docker through an amd64 VM, all the syscalls (PTRACE, etc) work flawlessly. A little bit slower than running docker directly on apple silicon. But works well nonetheless.