box64
box64 copied to clipboard
ptitSeb
Some issues with box64 dynarec
First off I want to say that this project is amazing, thank you for your effort.
I have been experiencing some issues while running box64 in a proot (Ubuntu 22.04 inside Termux using a Pixel 6 with a Google Tensor SoC) and I was wondering if some sort of fix or workaround could be implemented? I also want to add that I am not experienced with emulation so I do not know what is happening here.
From what I understand, it is not a box64 issue but an issue with proot as the binary works with box64 on a Nintendo switch running Ubuntu "natively". The confusing part is that it works with FEX on the same setup as shown below.
This happens with both Dynarec on and off (the libproviders.so library isn't an issue as even with it working the software crashes). Box64:
root@localhost:~/ProjectExabyte# box64 ./ProjectExabyte-linux
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 1ff0a7c built on Jul 25 2022 01:49:57
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402)
Error loading needed lib libproviders.so
Warning: Cannot dlopen("libproviders.so"/0x7e9fa99570, 2)
proot error: POKEDATA workaround stub got signal 11
proot warning: ptrace(POKEDATA): Bad address
node:internal/fs/utils:345
throw err;
^
Error: EFAULT: bad address in system call argument, stat '/root/ProjectExabyte/ProjectExabyte-linux'
at Object.statSync (node:fs:1551:3)
at pkg/prelude/bootstrap.js:96:21
at pkg/prelude/bootstrap.js:2259:3
at readPrelude (node:internal/bootstrap/pkg:31:12)
at node:internal/bootstrap/pkg:36:18
at node:internal/bootstrap/pkg:43:4
at node:internal/bootstrap/pkg:44:2 {
errno: -14,
syscall: 'stat',
code: 'EFAULT',
path: '/root/ProjectExabyte/ProjectExabyte-linux'
}
FEX:
root@localhost:~/ProjectExabyte# FEXInterpreter ./ProjectExabyte-linux
[MISC] mainBot init complete successfully!
The app goes on to work mostly normally with FEX albeit very slowly. It is a private binary made by me however I would be happy to create a fully open source binary that replicates the issue if needed (it is an app written in Node.js and packaged into an executable with https://github.com/vercel/pkg).
If you need more information then please feel free to ask. This is mostly for fun and to see what emulation can do, this isn't really that important as running inside a proot is not a common use case.
EDIT: Sorry for the bad formatting, I wrote this on my phone.
This line Warning, CALL to 0x0 at 0x16dd403 (0x16dd402) is not normal. I'm unsure what is happening but it looks like the relocations didn't work, or some bad native library.
Can't do much with just this log.
You can try to run with BOX64_DUMP=1 (it will generate a lot of data, so redirect to a file) and serach for a relocation near the CALL 0x0 address (0x16dd402 here, but the reloc might be at a slightly different one).
You can also try with BOX64_DLSYM_ERROR=1 to see if it dlopen a library and had issue with it.
I think the binary is fine as it works on a real x86_64 CPU and even in box64 while running natively and not inside of a proot. I would be happy to make an open source binary that replicates the issue if you would like.
BOX64_DLSYM_ERROR=1 Didn't really show anything useful
BOX64_DUMP=1 Produced a lot of data (20+ MB), I redirected it to a file and will look over it when I am at my computer
I think this part of log is not from box64 but proot itself:
proot error: POKEDATA workaround stub got signal 11
proot warning: ptrace(POKEDATA): Bad address
Could the bad address from proot be causing box64 to break? The binary works with box64 on a Nintendo switch running Ubuntu natively.
The think the proot error message are already a consequence of the CALL to 0x0 previous message. The jump to 0x0 will certinly trigger a segfault. If you can easily create a small program that replicate the different behaviour proot / regular, it might be easier I suppose.
It's large as the binary itself is quite big, it is basically a modified Node.js runtime.
I am extremely confused now, this is the log from the nintendo switch:
bartek@bartek-ulnx-27639:~/ProjectExabyte$ ./ProjectExabyte-linux
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.9 bf77789 built on Jul 28 2022 00:22:29
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 57 Env var
BOX64 try to Preload libgtk3-nocsd.so.0
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Error loading needed lib libgtk3-nocsd.so.0
Warning, cannot pre-load the lib (libgtk3-nocsd.so.0)
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402)
[MISC] mainBot init complete successfully!
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402) is there but the binary launches successfully then segfaults later. It used to work and now it doesn't for some reason.
With Dynarec off, the binary works flawlessly (extremely slowly though) on the Nintendo Switch and there is no Warning, CALL to 0x0 at 0x16dd403 (0x16dd402) or anything similar in the log.
On my Pixel 6 it does not work even with Dynarec disabled:
root@localhost:~/ProjectExabyte# BOX64_DYNAREC=0 box64 ./ProjectExabyte-linux
Dynarec is off
Box64 with Dynarec v0.1.9 1ff0a7c built on Jul 25 2022 01:49:57
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./ProjectExabyte-linux Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6 Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Error loading needed lib libproviders.so
Warning: Cannot dlopen("libproviders.so"/0x66bb0950, 2)
proot error: POKEDATA workaround stub got signal 11
proot warning: ptrace(POKEDATA): Bad address
node:internal/fs/utils:345
throw err;
^
Error: EFAULT: bad address in system call argument, stat '/root/ProjectExabyte/ProjectExabyte-linux'
at Object.statSync (node:fs:1551:3)
at pkg/prelude/bootstrap.js:96:21
at pkg/prelude/bootstrap.js:2259:3
at readPrelude (node:internal/bootstrap/pkg:31:12)
at node:internal/bootstrap/pkg:36:18
at node:internal/bootstrap/pkg:43:4
at node:internal/bootstrap/pkg:44:2 {
errno: -14,
syscall: 'stat',
code: 'EFAULT',
path: '/root/ProjectExabyte/ProjectExabyte-linux'
}
Ok, so that means:
- The CALL to 0x0 warning is ok. That call is never executed probably so the message can be ignored.
- There is a bug in the Dynarec (maybe it has been introduced lately?).
- There is an issue with the proot stuff, so more detail about the POKEDATA workaround would be interesting.
Termux maintains their own version of proot (https://github.com/termux/proot), here's what I could find after cloning the repository and recursively grepping and doing some searching for "POKEDATA"/"POKEDATA workaround":
https://github.com/termux/proot/blob/master/src/tracee/mem.c https://github.com/termux/proot/blob/master/src/syscall/syscall.c https://github.com/termux/proot/blob/master/src/tracee/event.c https://github.com/termux/proot/blob/master/src/loader/loader-info.awk
I don't really know what I am doing here but I hope that what I found will be helpful
For point 2, I will make an open source binary so you can test this yourself as we know it is not a proot issue but something with the Dynarec and happens even when running natively.
Edit: I have started work on this and it will be available soon.
box64-dynarec-debugging.tar.gz
Turns out a simple Node.js hello world one-liner causes the same issue. I would have made it a 7z file but GitHub wouldn't allow me to upload that so I just compressed it to .tar.gz.
It has to be built on a x86_64 system (QEMU could also work but might be troublesome) as it needs to generate x86_64 Node.js bytecode
https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js and https://github.com/vercel/pkg-fetch (patched Node.js binaries' source code) may be useful as something in there is causing the whole thing to crash when using box64 dynarec. It may lead you in the right direction
If you have any questions please feel free to ask.
Sorry for the delays, the Nintendo switch is in a remote location so I had to use OpenVPN to SSH in.
On both Termux and the Nintendo Switch, the open-source box64-dynarec-debugging binary works (log is from Termux):
root@localhost:~# box64 box64-dynarec-debugging
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 1 2022 17:29:37
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for box64-dynarec-debugging
Rename process to "box64-dynarec-debugging"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16de043 (0x16de042)
Hello, world!
The Call to 0x0 is still there but it doesn't matter now as it works.
But unfortunately my private binary does not work as it segfaults on both the Nintendo switch and Termux (log is from Termux):
root@localhost:~/ProjectExabyte# box64 ./ProjectExabyte-linux
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 1 2022 17:29:37
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
32592|SIGSEGV @0x798a28f200 (???(/lib/aarch64-linux-gnu/libc.so.6+0x798a28f200)) (x64pc=0xb08d3/???:"???", rsp=0x79892ba468, stack=0x7988abc000:0x79892bc000 own=(nil) fp=0xb), for accessing 0x27db00007f50 (code=-6/prot=0), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0) handler=(nil)
RSP-0x20:0x0000007fc4b9e2d0 RSP-0x18:0x00000079892ba530 RSP-0x10:0x0000000065bd1160 RSP-0x08:0x0000000000000174
RSP+0x00:0x0000000000962b3f RSP+0x08:0x0000000000000000 RSP+0x10:0x0000000000000000 RSP+0x18:0x0000000000000000
Segmentation fault
For this I have 2 ideas:
- I attempt to make a new
box64-dynarec-debuggingopen-source binary that replicates the issue - I could provide more logs about the segfaulting binary
EDIT: Without using the Dynarec the private binary works flawlessly on my Pixel 6 Ubuntu Proot so we now know it is not an issue with Proot.
EDIT 2: I think the other messages from Proot were a consequence of the error in box64? Because they do not appear anymore
Try to run with BOX64_ROLLING_LOG=1 BOX64_SHOWSEGV=1 to have a bit more detail on the segfault (that's some new stuff I added recently to help debugging).
Here (Termux as nintendo switch is irritating to work with because it has very slow SoC):
root@localhost:~/ProjectExabyte# BOX64_ROLLING_LOG=1 BOX64_SHOWSEGV=1 box64 ./ProjectExabyte-linux
Rolling log, showing last 16 function call on signals
Show Segfault signal even if a signal handler is present
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 1 2022 17:29:37
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402)
Error loading needed lib libproviders.so
Warning: Cannot dlopen("libproviders.so"/0x7505cfcf60, 2)
7786|0xd76b8b: Calling my_mprotect (./ProjectExabyte-linux)(0x7509603000, 0x3C000, 0x7, ...) => return 0x0
7788|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF8F0, 0x74F7A00000, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF8F8, 0x74FE200000, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF900, 0x74FB740000, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF908, 0x74F7AC0000, ...) => return 0x0
7789|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF910, 0x74F79C0000, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF918, 0x74FF480000, ...) => return 0x0
7789|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF920, 0x74FF080000, ...) => return 0x0
7786|0xd76b8b: Calling my_mprotect (./ProjectExabyte-linux)(0x7509603000, 0x3C000, 0x5, ...) => return 0x0
7786|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x1, 0x0, ...) => return 0x0
7786|0xd76b8b: Calling my_mprotect (./ProjectExabyte-linux)(0x7509603000, 0x3C000, 0x7, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF928, 0x74FF040000, ...) => return 0x0
7788|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF930, 0x74F7A40000, ...) => return 0x0
7789|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF938, 0x74FB780000, ...) => return 0x0
7786|0xd76b8b: Calling my_mprotect (./ProjectExabyte-linux)(0x7509603000, 0x3C000, 0x5, ...) => return 0x0
7790|0x12c3cb9: Calling my_pthread_cond_broadcast (./ProjectExabyte-linux)(0x6681F4B0, 0x75200BF940, 0x74FEDC0000, ...) => return 0x0
7786|SIGSEGV @0x74f4c3d9fc (my_pthread_cond_broadcast (./ProjectExabyte-linux)) (x64pc=0x75096062e4/???:"???", rsp=0x75332f71e8, stack=0x7532afb000:0x75332fb000 own=(nil) fp=0x75332f7190), for accessing 0x74f366b9 (code=1/prot=0), db=0x74f8cb1af0(0x74f4c3d9f4:0x74f4c3db58/0x75096062e0:0x7509606345/???:clean, hash:62a0b102/fbae5b39) handler=0x962ac0
RAX:0x00000075096062e0 RCX:0x000000059d8b0921 RDX:0x000000059d8b0918 RBX:0x0000000074f366aa
RSP:0x00000075332f71e8 RBP:0x00000075332f7260 RSI:0x0000000000000000 RDI:0x000000059d8b0909
R8:0x00000000667079b0 R9:0x0000000000000002 R10:0x0000000000000000 R11:0x00000075332f7478
R12:0x000000059d8b0909 R13:0x000000059d8b0918 R14:0x000000059d8b0921 R15:0x0000000000000002
RSP-0x20:0x0000000001100dd0 RSP-0x18:0x0000000066833640 RSP-0x10:0x000000059d8b1379 RSP-0x08:0x00000000028df180
RSP+0x00:0x0000000001578188 RSP+0x08:0x0000000066793310 RSP+0x10:0x0000000000000000 RSP+0x18:0x00000000666fc2d0
7786|0x962959: Calling my_fcntl (./ProjectExabyte-linux)(0x1, 0x3, 0x75332F6C90, ...) => return 0x28002
7786|0x96297d: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x75332F6C10, 0x3, 0x0, ...) => return 0x0
7786|0x96298a: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x75332F6C10, 0x16, 0x0, ...) => return 0x0
7786|0x962996: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x75332F6C10, 0x0, ...) => return 0x0
7786|0x9629c7: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x0, 0x2D4FA70, ...) => return 0x0
7786|0x9629de: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x75332F6C10, 0x0, ...) => return 0x0
7786|0x9628f8: Calling my___fxstat64 (./ProjectExabyte-linux)(0x1, 0x2, 0x75332F6C90, ...) => return 0x0
7786|0x962959: Calling my_fcntl (./ProjectExabyte-linux)(0x2, 0x3, 0x75332F6C90, ...) => return 0x28002
7786|0x96297d: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x75332F6C10, 0x3, 0x0, ...) => return 0x0
7786|0x96298a: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x75332F6C10, 0x16, 0x0, ...) => return 0x0
7786|0x962996: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x75332F6C10, 0x0, ...) => return 0x0
7786|0x9629c7: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x2, 0x0, 0x2D4FB48, ...) => return 0x0
7786|0x9629de: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x75332F6C10, 0x0, ...) => return 0x0
PltResolver => return 0x0
7786|0x962b3f: Calling raise (/lib/aarch64-linux-gnu/libc.so.6)(0xB, 0x75332F6C10, 0x0, ...) => return
7786|0x9628f8: Calling my___fxstat64 (./ProjectExabyte-linux)(0x1, 0x1, 0x75332F6C90, ...) => return 0x0
7786|SIGSEGV @0x75342cf200 (???(/lib/aarch64-linux-gnu/libc.so.6+0x75342cf200)) (x64pc=0xb08d3/???:"???", rsp=0x75332f6d58, stack=0x7532afb000:0x75332fb000 own=(nil) fp=0xb), for accessing 0x27db00001e6a (code=-6/prot=0), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0) handler=(nil)
RSP-0x20:0x0000007fe8a8dea0 RSP-0x18:0x00000075332f6e20 RSP-0x10:0x0000000065bd2b60 RSP-0x08:0x0000000000000174
RSP+0x00:0x0000000000962b3f RSP+0x08:0x0000000000000000 RSP+0x10:0x0000000000000000 RSP+0x18:0x0000000000000000
Segmentation fault
As a file: box64.log
EDIT: Having some difficulty reproducing the issue in the other binary, will keep trying if logs are not enough
I believe that I am close to reproducing the issue, I got a segfault once but it seems random for some reason
Edit: I have reproduced the issue but it is not that reliable, however it is the best I can do as I have absolutely no idea what's causing this. At least it reproduces the issue eventually.
box64-dynarec-debugging.tar.gz
Another debugging binary I made, source included like last time.
This one isn't that reliable so if you don't get a segfault within like 20 seconds then interrupt it and execute again.
What this does is it sets up a very tiny local http server and continuously makes requests to itself on 127.0.0.1 every 10 ms. This seems to trigger the segfault sometimes but not always.
Sometimes this also happens with this binary:
root@localhost:~# BOX64_ROLLING_LOG=1 BOX64_SHOWSEGV=1 box64 ./box64-dynarec-debugging
Rolling log, showing last 16 function call on signals
Show Segfault signal even if a signal handler is present
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 1 2022 17:29:37
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 29 Env var
Looking for ./box64-dynarec-debugging
Rename process to "box64-dynarec-debugging"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16de043 (0x16de042)
Listening on port 1337...
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
Got request from: ::ffff:127.0.0.1
Hello, world!
#
# Fatal error in , line 0
# Check failed: fixed_size_above_fp + (stack_slots * kSystemPointerSize) - CommonFrameConstants::kFixedFrameSizeAboveFp + outgoing_size == result.
#
#
#
#FailureMessage Object: 0x79b9965ad0
1: 0xa0c541 node::NodePlatform::PostJob(v8::TaskPriority, std::unique_ptr<v8::JobTask, std::default_delete<v8::JobTask> >) [/root/box64-dynarec-debugging]
2: 0xd76829 V8_Fatal(char const*, ...) [/root/box64-dynarec-debugging]
3: 0x115e9f8 v8::internal::Deoptimizer::Deoptimizer(v8::internal::Isolate*, v8::internal::JSFunction, v8::internal::DeoptimizeKind, unsigned int, unsigned long, int) [/root/box64-dynarec-debugging]
4: 0x79b9965fc0
11097|BOX64: Warning, calling Signal 4 function handler SIG_DFL
Unhandled signal caught, aborting
Aborted
When it segfaults I have noticed that it usually does so right away:
root@localhost:~# BOX64_ROLLING_LOG=1 BOX64_SHOWSEGV=1 box64 ./box64-dynarec-debugging
Rolling log, showing last 16 function call on signals
Show Segfault signal even if a signal handler is present
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 1 2022 17:29:37
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 29 Env var
Looking for ./box64-dynarec-debugging
Rename process to "box64-dynarec-debugging"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
11899|0xb1f090: Calling my_pthread_attr_init (./box64-dynarec-debugging)(0x7E3E2498A0, 0x7E3F27E560, 0x0, ...) => return 0x0
11899|0xb1f0a3: Calling my_pthread_attr_setstacksize (./box64-dynarec-debugging)(0x7E3E2498A0, 0x800000, 0x0, ...) => return 0x0
11899|0xb1f0bc: Calling my_pthread_create (./box64-dynarec-debugging)(0x66658730, 0x7E3E2498A0, 0xA0A0F0, ...) => return 0x0
11901|0xb21802: Calling my_epoll_ctl (./box64-dynarec-debugging)(0x9, 0x1, 0xA, ...) => return 0x0
11899|0xb1f0c6: Calling my_pthread_attr_destroy (./box64-dynarec-debugging)(0x7E3E2498A0, 0x7E3E2498A0, 0xA0A0F0, ...) => return 0x0
11899|0xb1f090: Calling my_pthread_attr_init (./box64-dynarec-debugging)(0x7E3E2498A0, 0x7E3F27E560, 0x0, ...) => return 0x0
11901|0xb21802: Calling my_epoll_ctl (./box64-dynarec-debugging)(0x9, 0x1, 0xC, ...) => return 0x0
11899|0xb1f0a3: Calling my_pthread_attr_setstacksize (./box64-dynarec-debugging)(0x7E3E2498A0, 0x800000, 0x0, ...) => return 0x0
11899|0xb1f0bc: Calling my_pthread_create (./box64-dynarec-debugging)(0x66659460, 0x7E3E2498A0, 0xA0A0F0, ...) => return 0x0
PltResolver => return 0x0
11901|0xb21d93: Calling my_epoll_wait (./box64-dynarec-debugging)(0x9, 0x7E39338EA0, 0x400, ...) => return
11899|0xb1f0c6: Calling my_pthread_attr_destroy (./box64-dynarec-debugging)(0x7E3E2498A0, 0x7E3E2498A0, 0xA0A0F0, ...) => return 0x0
PltResolver => return 0x4
11899|0xa0bbeb: Calling my_pthread_cond_wait (./box64-dynarec-debugging)(0x7E3E2499C0, 0x7E3E249990, 0x0, ...) => return
PltResolver => return
11899|0xb1f0c6: Calling my_pthread_attr_destroy (./box64-dynarec-debugging)(0x7E3E2498A0, 0x7E3E2498A0, 0xA0A0F0, ...) => return 0x0
11902|SIGSEGV @0x649ba19c (???(./box64-dynarec-debugging+0x649ba19c)) (x64pc=0x300b3/???:"???", rsp=0x7e33ffff38, stack=0x7e33800000:0x7e34000000 own=0x7e33800000 fp=0x6664ed30), for accessing 0x70d60 (code=1/prot=81), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0) handler=0x962a00
RSP-0x20:0x0000000066655850 RSP-0x18:0x0000000000000000 RSP-0x10:0x0000000065bd2b00 RSP-0x08:0x0000000000000120
RSP+0x00:0x0000000000a0a207 RSP+0x08:0x0000000000000000 RSP+0x10:0x0000000000000000 RSP+0x18:0x0000000000000000
11902|0x962899: Calling my_fcntl (./box64-dynarec-debugging)(0x1, 0x3, 0x7E33FFF9E0, ...) => return 0x28002
11902|0x9628bd: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x7E33FFF960, 0x3, 0x0, ...) => return 0x0
11902|0x9628ca: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x7E33FFF960, 0x16, 0x0, ...) => return 0x0
11902|0x9628d6: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x7E33FFF960, 0x0, ...) => return 0x0
11902|0x962907: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x0, 0x2D4FA70, ...) => return 0x0
11902|0x96291e: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x7E33FFF960, 0x0, ...) => return 0x0
11902|0x962838: Calling my___fxstat64 (./box64-dynarec-debugging)(0x1, 0x2, 0x7E33FFF9E0, ...) => return 0x0
11902|0x962899: Calling my_fcntl (./box64-dynarec-debugging)(0x2, 0x3, 0x7E33FFF9E0, ...) => return 0x28002
11902|0x9628bd: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x7E33FFF960, 0x3, 0x0, ...) => return 0x0
11902|0x9628ca: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x7E33FFF960, 0x16, 0x0, ...) => return 0x0
11902|0x9628d6: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x7E33FFF960, 0x0, ...) => return 0x0
11902|0x962907: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x2, 0x0, 0x2D4FB48, ...) => return 0x0
11902|0x96291e: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x7E33FFF960, 0x0, ...) => return 0x0
PltResolver => return 0x0
11902|0x962a7f: Calling raise (/lib/aarch64-linux-gnu/libc.so.6)(0xB, 0x7E33FFF960, 0x0, ...) => return
11902|0x962838: Calling my___fxstat64 (./box64-dynarec-debugging)(0x1, 0x1, 0x7E33FFF9E0, ...) => return 0x0
11902|SIGSEGV @0x7e3f21f200 (???(/lib/aarch64-linux-gnu/libc.so.6+0x7e3f21f200)) (x64pc=0xb08d3/???:"???", rsp=0x7e33fffaa8, stack=0x7e33800000:0x7e34000000 own=0x7e33800000 fp=0xb), for accessing 0x27db00002e7b (code=-6/prot=0), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0) handler=(nil)
RSP-0x20:0x0000007e336fcbf0 RSP-0x18:0x0000007e33fffb70 RSP-0x10:0x0000000065bd2b00 RSP-0x08:0x0000000000000174
RSP+0x00:0x0000000000962a7f RSP+0x08:0x0000000000000000 RSP+0x10:0x0000000000000000 RSP+0x18:0x0000000000000000
Segmentation fault
EDIT: In my private binary, I have noticed that segfaults are way more common than the other error for some reason but here the other error is more common than segfaults
EDIT 2: The next day I noticed that sometimes when opening both my private binary and the debugging one that a double free can happen (only with the dynarec on it seems):
root@localhost:~/ProjectExabyte# box64 ./ProjectExabyte-linux
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 d08bd0c built on Aug 2 2022 11:59:27
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 28 Env var
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402)
Error loading needed lib libproviders.so
Warning: Cannot dlopen("libproviders.so"/0x7394f59970, 2)
double free or corruption (fasttop)
Aborted
After the recent commits I am getting the following with my private binary:
root@localhost:~/ProjectExabyte# box64 ./ProjectExabyte-linux
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 642260b built on Aug 7 2022 03:43:13
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./ProjectExabyte-linux
Rename process to "ProjectExabyte-linux"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16dd403 (0x16dd402)
13957|0x1220f72: Unimplemented Opcode (01 00 00 DD) D9 D9 6D 9C DF 7D 90 D9 6D 9E 48 8B 45 90 48
Stop waiting for remaining thread 13958
Stop waiting for remaining thread 13959
Stop waiting for remaining thread 13960
Stop waiting for remaining thread 13961
Stop waiting for remaining thread 13962
Aborted
From what I understand, these op codes have to be implemented in the dynarec?
EDIT: This also happens in the open source debugging binary I provided (previously somewhat worked with the dynarec) which may be a better way of debugging the issue.
EDIT 2: Also happens without the dynarec, it previously worked. Could it be a regression of some sort?
After the recent commit I now get this:
root@localhost:~/box64-dynarec-debugging# BOX64_ROLLING_LOG=1 BOX64_SHOWSEGV=1 box64 ./box64-dynarec-debugging
Rolling log, showing last 16 function call on signals Show Segfault signal even if a signal handler is present
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS PageSize:4096
Box64 with Dynarec v0.1.9 dc5c049 built on Aug 9 2022 21:42:41
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 30 Env var
Looking for ./box64-dynarec-debugging
Rename process to "box64-dynarec-debugging"
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2 Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using emulated /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Using native(wrapped) libm.so.6
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning, CALL to 0x0 at 0x16de043 (0x16de042) 3867|0x962899: Calling my_fcntl (./box64-dynarec-debugging)(0x1, 0x3, 0x78AB776C30, ...) => return 0x28402 3867|0x9628bd: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x78AB776BB0, 0x3, 0x0, ...) => return 0x0
3867|0x9628ca: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x78AB776BB0, 0x16, 0x0, ...) => return 0x0
3867|0x9628d6: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x78AB776BB0, 0x0, ...) => return 0x0
3867|0x962907: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x0, 0x2D4FA70, ...) => return 0x0
3867|0x96291e: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x78AB776BB0, 0x0, ...) => return 0x0
3867|0x962838: Calling my___fxstat64 (./box64-dynarec-debugging)(0x1, 0x2, 0x78AB776C30, ...) => return 0x0
3867|0x962899: Calling my_fcntl (./box64-dynarec-debugging)(0x2, 0x3, 0x78AB776C30, ...) => return 0x28402
3867|0x9628bd: Calling sigemptyset (/lib/aarch64-linux-gnu/libc.so.6)(0x78AB776BB0, 0x3, 0x0, ...) => return 0x0
3867|0x9628ca: Calling sigaddset (/lib/aarch64-linux-gnu/libc.so.6)(0x78AB776BB0, 0x16, 0x0, ...) => return 0x0
3867|0x9628d6: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x0, 0x78AB776BB0, 0x0, ...) => return 0x0
3867|0x962907: Calling tcsetattr (/lib/aarch64-linux-gnu/libc.so.6)(0x2, 0x0, 0x2D4FB48, ...) => return 0x0
3867|0x96291e: Calling pthread_sigmask (/lib/aarch64-linux-gnu/libc.so.6)(0x1, 0x78AB776BB0, 0x0, ...) => return 0x0
PltResolver => return 0x0
3867|0x962a7f: Calling raise (/lib/aarch64-linux-gnu/libc.so.6)(0xB, 0x78AB776BB0, 0x0, ...) => return
3867|0x962838: Calling my___fxstat64 (./box64-dynarec-debugging)(0x1, 0x1, 0x78AB776C30, ...) => return 0x0
3867|SIGSEGV @0x78ac74f200 (???(/lib/aarch64-linux-gnu/libc.so.6+0x78ac74f200)) (x64pc=0xb08d3/???:"???", rsp=0x78ab776cf8, stack=0x78aaf7b000:0x78ab77b000 own=(nil) fp=0xb), for accessing 0x27db00000f1b (code=-6/prot=0), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0) handler=(nil)
RSP-0x20:0x0000007fe2d6e360 RSP-0x18:0x00000078ab776dc0 RSP-0x10:0x0000000065c03be0 RSP-0x08:0x0000000000000174
RSP+0x00:0x0000000000962a7f RSP+0x08:0x0000000000000000 RSP+0x10:0x0000000000000000 RSP+0x18:0x0000000000000000
3867|BOX64: Warning, calling Signal 11 function handler SIG_DFL
Unhandled signal caught, aborting
Aborted
If you require more information then please ask. Both my private binary and the newest debugging binary segfaults, it used to somewhat work before.
I really appreciate that you are working on this btw
Yeah, I know something got broken in a recent commit. I started investigating but I'm not over yet...
Oh alright, is there any way I could help? I don't have a very good understanding of C so I can't really investigate that much.
My binary is quite unusual as it is basically a packaged up Node.js runtime so I'm thinking that's the cause.
Not for now. I have enough information to fix the issue in box64. When it's done, we'll see if that fixed the issue on your side too (or at least, change the issue)
I am making a new debugging binary, while I was testing it I got another unimplemented opcode:
15737|0x7bf805a99b: Unimplemented Opcode (F8 49 8B 40) 3F 48 89 45 90 48 8B 55 D0 48 B9 C1 17 9A F2
box64-dynarec-debugging.tar.gz
The new debugging binary is ready, the only changes are is that this one is compressed and obfuscated. Source (unobfuscated) is included like last time and there is a script to build your own obfuscated binary. The obfuscator I used is open source and can be found here: https://github.com/javascript-obfuscator/javascript-obfuscator
It doesn't always segfault, there are a few other errors too but this seems to replicate the issue that I was experiencing in my private binary
I re-wrote a large chunk of my program in TypeScript and came across some more unimplemented opcodes:
21865|0x620078: Unimplemented Opcode (74 31 5F 6E) 61 6D 65 00 5F 5A 4E 32 76 38 38 69 6E 74 65
22448|0x7c4c0a5218: Unimplemented Opcode (00 00 4D 8B) 67 17 4D 39 65 20 0F 84 15 00 00 00 49 BF E1
10921|0x722a31: Unimplemented Opcode (65 6E 65 72) 61 74 6F 72 32 34 42 75 69 6C 64 50 72 69 76
25582|0x75ad7a: Unimplemented Opcode (6F 72 52 65) 64 75 63 65 72 44 32 45 76 00 5F 5A 33 36 5F
If I come across any more unimplemented opcodes do you want me to post them here?
EDIT: The version of node I packaged in my executable is newer too, that's probably the source of the "new" opcode.
EDIT 2: It sometimes works without the dynarec, when it does it works perfectly but very slowly.
On Oracle Cloud there is no segfaults at all (maybe termux doesn't like the crashing?), the only thing so far preventing me from running it are the unimplemented opcodes.
I do not have a lot of experience in C, would it be hard to implement these opcodes? If not then I would be happy to try if you point me in the right direction
Problem is, the OpCode is not a correct one. There is probably something else going wrong before the 1st "unimplemented opcode" error. I need a binary to reproduce the issue and debug it.
The binary isn't open source but I could provide it if you would like. If not then I can start work on another open source one