qiling
qiling copied to clipboard
qilingframework
Qiling fails to emulate simple binary with a ubuntu-22.04 root fs
*Describe the bug
When trying to emulate simple binaries on a vanilla ubuntu 22.04 qiling fails gloriously with unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)
Sample Code
FROM ubuntu:22.04
RUN apt-get update --fix-missing
RUN apt-get install -y --no-install-recommends python3-minimal python3-pip build-essential
RUN pip install qiling
RUN echo '#include <stdio.h>\nint main() { puts("Hi"); }' > main.c && \
gcc main.c -o main
ENTRYPOINT [ "qltool", "run", "--filename", "main", "--rootfs", "/" ]
Expected behavior The emulations is successful, binary prints "Hi". This works fine on f.e. ubuntu 20.04
Additional context This is the entire output log, but you should be able to reproduce with the given dockerfile:
[=] brk(inp = 0x0) = 0x55555555b000
[!] prctl code 0x3001 not implemented
[=] arch_prctl(code = 0x3001, addr = 0x80000000dcc0) = 0x0
[=] uname(buf = 0x80000000d8a0) = 0x0
[=] access(path = 0x7ffff7e04d90, mode = 0x4) = -0x1 (EPERM)
[=] openat(fd = 0xffffff9c, path = 0x7ffff7e0321b, flags = 0x80000, mode = 0x0) = 0x3
[=] newfstatat(dirfd = 0x3, path = 0x7ffff7e03ee9, buf_ptr = 0x80000000ce10, flags = 0x1000) = 0x0
[=] mmap(addr = 0x0, length = 0x1e89, prot = 0x1, flags = 0x2, fd = 0x3, pgoffset = 0x0) = 0x7fffb7dd6000
[=] close(fd = 0x3) = 0x0
[=] openat(fd = 0xffffff9c, path = 0x7ffff7e10aa0, flags = 0x80000, mode = 0x0) = 0x3
[=] read(fd = 0x3, buf = 0x80000000d048, length = 0x340) = 0x340
[=] pread64(fd = 0x3, buf = 0x80000000cc50, length = 0x310, offt = 0x40) = 0x310
[=] pread64(fd = 0x3, buf = 0x80000000cc10, length = 0x30, offt = 0x350) = 0x30
[=] pread64(fd = 0x3, buf = 0x80000000cbc0, length = 0x44, offt = 0x380) = 0x44
[=] newfstatat(dirfd = 0x3, path = 0x7ffff7e03ee9, buf_ptr = 0x80000000cee0, flags = 0x1000) = 0x0
[=] pread64(fd = 0x3, buf = 0x80000000cb20, length = 0x310, offt = 0x40) = 0x310
[=] mmap(addr = 0x0, length = 0x227e50, prot = 0x1, flags = 0x802, fd = 0x3, pgoffset = 0x0) = 0x7fffb7dd8000
[=] mmap(addr = 0x7fffb7e00000, length = 0x195000, prot = 0x5, flags = 0x812, fd = 0x3, pgoffset = 0x28000) = 0x7fffb7e00000
[=] mmap(addr = 0x7fffb7f95000, length = 0x58000, prot = 0x1, flags = 0x812, fd = 0x3, pgoffset = 0x1bd000) = 0x7fffb7f95000
[=] mmap(addr = 0x7fffb7fed000, length = 0x6000, prot = 0x3, flags = 0x812, fd = 0x3, pgoffset = 0x214000) = 0x7fffb7fed000
[=] mmap(addr = 0x7fffb7ff3000, length = 0xce50, prot = 0x3, flags = 0x32, fd = 0xffffffff, pgoffset = 0x0) = 0x7fffb7ff3000
[=] close(fd = 0x3) = 0x0
[=] mmap(addr = 0x0, length = 0x2000, prot = 0x3, flags = 0x22, fd = 0xffffffff, pgoffset = 0x0) = 0x7fffb8000000
[=] arch_prctl(code = 0x1002, addr = 0x7fffb80010c0) = 0x0
[=] set_tid_address(tidptr = 0x7fffb8001390) = 0x1
[=] set_robust_list(head_ptr = 0x7fffb80013a0, head_len = 0x18) = 0x0
[!] 0x7ffff7dea1cf: syscall ql_syscall_rseq number = 0x14e(334) not implemented
/lib/x86_64-linux-gnu/libc.so.6: CPU ISA level is lower than required
[=] writev(fd = 0x2, vec = 0x80000000d530, vlen = 0x2) = 0x46
[=] exit_group(code = 0x7f) = ?
[x] CPU Context:
[x] ah : 0x0
[x] al : 0xe7
[x] ch : 0x0
[x] cl : 0x0
[x] dh : 0xd9
[x] dl : 0x70
[x] bh : 0x0
[x] bl : 0x0
[x] ax : 0xe7
[x] cx : 0x0
[x] dx : 0xd970
[x] bx : 0x0
[x] sp : 0xd958
[x] bp : 0x0
[x] si : 0x4fc9
[x] di : 0x5149
[x] ip : 0x0
[x] eax : 0xe7
[x] ecx : 0x0
[x] edx : 0xd970
[x] ebx : 0x0
[x] esp : 0xd958
[x] ebp : 0x0
[x] esi : 0xf7de4fc9
[x] edi : 0x55555149
[x] eip : 0x0
[x] rax : 0xe7
[x] rbx : 0x0
[x] rcx : 0x0
[x] rdx : 0x80000000d970
[x] rsi : 0x7ffff7de4fc9
[x] rdi : 0x555555555149
[x] rbp : 0x0
[x] rsp : 0x80000000d958
[x] r8 : 0x0
[x] r9 : 0x3c
[x] r10 : 0xffffffff
[x] r11 : 0x90
[x] r12 : 0x0
[x] r13 : 0x0
[x] r14 : 0x7fffb80010c0
[x] r15 : 0x7ffff7e102e0
[x] rip : 0x0
[x] cr0 : 0x11
[x] cr1 : 0x0
[x] cr2 : 0x0
[x] cr3 : 0x0
[x] cr4 : 0x0
[x] cr8 : 0x0
[x] st0 : 0x0
[x] st1 : 0x0
[x] st2 : 0x0
[x] st3 : 0x0
[x] st4 : 0x0
[x] st5 : 0x0
[x] st6 : 0x0
[x] st7 : 0x0
[x] eflags : 0x46
[x] cs : 0x33
[x] ss : 0x28
[x] ds : 0x0
[x] es : 0x0
[x] fs : 0x0
[x] gs : 0x0
[x] r8b : 0x0
[x] r9b : 0x3c
[x] r10b : 0xff
[x] r11b : 0x90
[x] r12b : 0x0
[x] r13b : 0x0
[x] r14b : 0xc0
[x] r15b : 0xe0
[x] r8w : 0x0
[x] r9w : 0x3c
[x] r10w : 0xffff
[x] r11w : 0x90
[x] r12w : 0x0
[x] r13w : 0x0
[x] r14w : 0x10c0
[x] r15w : 0x2e0
[x] r8d : 0x0
[x] r9d : 0x3c
[x] r10d : 0xffffffff
[x] r11d : 0x90
[x] r12d : 0x0
[x] r13d : 0x0
[x] r14d : 0xb80010c0
[x] r15d : 0xf7e102e0
[x] fsbase : 0x7fffb80010c0
[x] gsbase : 0x6000000
[x] xmm0 : 0x0
[x] xmm1 : 0xff000000000000000000000000000000
[x] xmm2 : 0xffff000000000000ffff0000000000ff
[x] xmm3 : 0xffff000000000000ffff000000000000
[x] xmm4 : 0x0
[x] xmm5 : 0x2815000000000000004b
[x] xmm6 : 0x2815000000000000004b
[x] xmm7 : 0x1
[x] xmm8 : 0x0
[x] xmm9 : 0x0
[x] xmm10 : 0x0
[x] xmm11 : 0x0
[x] xmm12 : 0x0
[x] xmm13 : 0x0
[x] xmm14 : 0x0
[x] xmm15 : 0x0
[x] xmm16 : 0x0
[x] xmm17 : 0x0
[x] xmm18 : 0x0
[x] xmm19 : 0x0
[x] xmm20 : 0x0
[x] xmm21 : 0x0
[x] xmm22 : 0x0
[x] xmm23 : 0x0
[x] xmm24 : 0x0
[x] xmm25 : 0x0
[x] xmm26 : 0x0
[x] xmm27 : 0x0
[x] xmm28 : 0x0
[x] xmm29 : 0x0
[x] xmm30 : 0x0
[x] xmm31 : 0x0
[x] ymm0 : 0x0
[x] ymm1 : 0xff000000000000000000000000000000
[x] ymm2 : 0xffff000000000000ffff0000000000ff
[x] ymm3 : 0xffff000000000000ffff000000000000
[x] ymm4 : 0x0
[x] ymm5 : 0x2815000000000000004b
[x] ymm6 : 0x2815000000000000004b
[x] ymm7 : 0x1
[x] ymm8 : 0x0
[x] ymm9 : 0x0
[x] ymm10 : 0x0
[x] ymm11 : 0x0
[x] ymm12 : 0x0
[x] ymm13 : 0x0
[x] ymm14 : 0x0
[x] ymm15 : 0x0
[x] ymm16 : 0x0
[x] ymm17 : 0x0
[x] ymm18 : 0x0
[x] ymm19 : 0x0
[x] ymm20 : 0x0
[x] ymm21 : 0x0
[x] ymm22 : 0x0
[x] ymm23 : 0x0
[x] ymm24 : 0x0
[x] ymm25 : 0x0
[x] ymm26 : 0x0
[x] ymm27 : 0x0
[x] ymm28 : 0x0
[x] ymm29 : 0x0
[x] ymm30 : 0x0
[x] ymm31 : 0x0
[x] zmm0 : 0x0
[x] zmm1 : 0x0
[x] zmm2 : 0x0
[x] zmm3 : 0x0
[x] zmm4 : 0x0
[x] zmm5 : 0x0
[x] zmm6 : 0x0
[x] zmm7 : 0x0
[x] zmm8 : 0x0
[x] zmm9 : 0x0
[x] zmm10 : 0x0
[x] zmm11 : 0x0
[x] zmm12 : 0x0
[x] zmm13 : 0x0
[x] zmm14 : 0x0
[x] zmm15 : 0x0
[x] zmm16 : 0x0
[x] zmm17 : 0x0
[x] zmm18 : 0x0
[x] zmm19 : 0x0
[x] zmm20 : 0x0
[x] zmm21 : 0x0
[x] zmm22 : 0x0
[x] zmm23 : 0x0
[x] zmm24 : 0x0
[x] zmm25 : 0x0
[x] zmm26 : 0x0
[x] zmm27 : 0x0
[x] zmm28 : 0x0
[x] zmm29 : 0x0
[x] zmm30 : 0x0
[x] zmm31 : 0x0
[x] PC = 0x0000000000000000 (unreachable)
[x] Memory map:
[x] Start End Perm Label Image
[x] 000000000000030000 - 000000000000031000 rwx [GDT]
[x] 000000555555554000 - 000000555555555000 r-- main /main
[x] 000000555555555000 - 000000555555556000 r-x main /main
[x] 000000555555556000 - 000000555555557000 r-- main /main
[x] 000000555555557000 - 000000555555559000 rw- main /main
[x] 000000555555559000 - 00000055555555b000 rwx [hook_mem]
[x] 0000007fffb7dd6000 - 0000007fffb7dd8000 r-- [mmap] ld.so.cache
[x] 0000007fffb7dd8000 - 0000007fffb7e00000 r-- [mmap] libc.so.6
[x] 0000007fffb7e00000 - 0000007fffb7f95000 r-x [mmap] libc.so.6
[x] 0000007fffb7f95000 - 0000007fffb7fed000 r-- [mmap] libc.so.6
[x] 0000007fffb7fed000 - 0000007fffb7ff3000 rw- [mmap] libc.so.6
[x] 0000007fffb7ff3000 - 0000007fffb8000000 rw- [mmap] libc.so.6
[x] 0000007fffb8000000 - 0000007fffb8002000 rw- [syscall_mmap]
[x] 0000007ffff7dd5000 - 0000007ffff7dd7000 r-- ld-linux-x86-64.so.2 //lib64/ld-linux-x86-64.so.2
[x] 0000007ffff7dd7000 - 0000007ffff7e01000 r-x ld-linux-x86-64.so.2 //lib64/ld-linux-x86-64.so.2
[x] 0000007ffff7e01000 - 0000007ffff7e0c000 r-- ld-linux-x86-64.so.2 //lib64/ld-linux-x86-64.so.2
[x] 0000007ffff7e0d000 - 0000007ffff7e11000 rw- ld-linux-x86-64.so.2 //lib64/ld-linux-x86-64.so.2
[x] 0000007ffffffde000 - 00000080000000e000 rwx [stack]
[x] 00ffffffffff600000 - 00ffffffffff601000 rwx [vsyscall]
Traceback (most recent call last):
File "/usr/local/bin/qltool", line 253, in <module>
ql.run(timeout=options.timeout)
File "/usr/local/lib/python3.10/dist-packages/qiling/core.py", line 573, in run
self.os.run()
File "/usr/local/lib/python3.10/dist-packages/qiling/os/linux/linux.py", line 163, in run
self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)
File "/usr/local/lib/python3.10/dist-packages/qiling/core.py", line 706, in emu_start
self.uc.emu_start(begin, end, timeout, count)
File "/usr/local/lib/python3.10/dist-packages/unicorn/unicorn.py", line 548, in emu_start
raise UcError(status)
unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)
Hi there. It looks like your binary fails to run because of the following error message:
/lib/x86_64-linux-gnu/libc.so.6: CPU ISA level is lower than required
That said, we'll look into why Qiling did not fail gracefully.
May this be a direct cause of the preceding syscall failure?
0x7ffff7dea1cf: syscall ql_syscall_rseq number = 0x14e(334) not implemented
That might be the problem.
Can you try again but with the following code added to your script? (somewhere between ql initialization and ql.run)
def null_rseq_impl(ql: Qiling, abi: int, length: int, flags: int, sig: int):
# do nothing
pass
ql.os.set_syscall('rseq', null_rseq_impl, QL_INTERCEPT.CALL)
Notes:
- If the specifying
'rseq'doesn't work, use its numeric id instead:334 - Not sure what this syscall is expected to return, so you could try replacing
passwithreturn 1orreturn 0
I tried that, though it did not help. Neither combination of return values was successful.
@LoverOfSadness just use an older version of libc. If you really need the newer version, the easiest way is probably patching the ISA checks in libc
I found a temporary solution, which is to skip this error through hook.
hook code:
...
def by_pass_isa_check(ql: Qiling) -> None:
print("by_pass_isa_check():")
ql.arch.regs.rip += 0x15
pass
...
ld_so_base = 0x7ffff7dd5000
ql.hook_address(by_pass_isa_check, ld_so_base+0x2389f) # ld.so base + jz offset
Source:
IDA Disassembly:
which glibc version u using
Glibc2.35 in ubuntu 22.04.
I'm not sure whether this offset is appropriate for your version, but the method is the same.
Some background for this:
glibc attempts to determine whether the underlying system supports certain ISA capabilities the program requires. This is done by executing the cpuid instruction using different leafs, and testing for the supported "ISA Level". There are 4 incremental ISA levels: baseline, v2, v3 and v4, where each one represents a set of CPU features / intructions. Supporting an ISA level means all the CPU features included in that level and previous ones are supported.
Emulating a simple cpuid intruction with Unicorn 2.0 shows that neither of the ISA levels are fully supported, which is probably why glib's test is failing. Here is a table that summarizes what is actually supported by Unicorn 2.0:
References:
cc: @wtdcode, in case you want to patch Unicorn to support the mising CPU features (or they are already supported, but that support is not reflected in cpuid output)
Unicorn has a special hook for cpuid: https://github.com/unicorn-engine/unicorn/blob/6c1cbef6ac505d355033aef1176b684d02e1eb3a/include/unicorn/x86.h#L86
Another solution is that the CPU model might be changed by uc_ctl_set_cpu_model however Qiling doesn't offer this feature so far.
I'll work on enabling a CPU selection. Hooking the CPUID may work around this in the short-term, but not really a fix.
any update ..?
I'll work on enabling a CPU selection. Hooking the CPUID may work around this in the short-term, but not really a fix.
Someone actually came out with a workaround.
https://cloud.tencent.com/developer/article/2144036 by @yikesoftware i belive.
can I get the source code and binary for this issue to try it out. Thanks in advance.
It's kinda detrimental if qiling can't run the latest binaries from the most popular linux distro on x8664 (for over a year now!). If binaries run just fine when skipping the check, should we just fake a higher cpuid as standard? If the alternative is fixing unicorn then this issue is gonna stay up forever.
Am i reading @elicn 's chart correctly that unicorn in fact does not even support baseline, and that's why the test fails? I'm kinda surprised Unicorn doesn't support FPU o_O.
Trying this but i cannot get any value to work:
ql = Qiling(argv, rootfs)
print(f"{ql.uc.ctl_get_cpu_model()=}")
ql.uc.ctl_set_cpu_model(UC_CPU_X86_EPYC_ROME)
ql.uc.ctl_get_cpu_model()=0
Traceback (most recent call last):
File "/Users/jole/pro/rebg/tools/ql/run.py", line 331, in <module>
run(rootfs, rest)
File "/Users/jole/pro/rebg/tools/ql/run.py", line 268, in run
ql.uc.ctl_set_cpu_model(UC_CPU_X86_EPYC_ROME)
File "/opt/homebrew/lib/python3.11/site-packages/unicorn/unicorn.py", line 798, in ctl_set_cpu_model
self.__ctl_w_1_arg(uc.UC_CTL_CPU_MODEL, val, ctypes.c_int)
File "/opt/homebrew/lib/python3.11/site-packages/unicorn/unicorn.py", line 748, in __ctl_w_1_arg
self.ctl(self.__ctl_w(ctl, 1), arg)
File "/opt/homebrew/lib/python3.11/site-packages/unicorn/unicorn.py", line 726, in ctl
raise UcError(status)
unicorn.unicorn.UcError: Invalid argument (UC_ERR_ARG)
not even using the value returned from ctl_get_cpu_model() (0 on x8664, 2 on aarch64...)
@joleeee, interacting with the uc object directly is not recommended as it might pull the rug under Qiling's feet in some cases. Here specifically, it is already too late to change the underlying CPU model because Qiling already initialized some things under the hood and Unicorn declines to change CPU model unless it is a fresh object.
Assuming you are using the latest dev version, you may select the CPU model with the method described in #1376. When you do that, please let us know if it worked for you or not.
Thanks for the explanation! Now i can actually set the cputype/model, but I'm still getting the ISA level error.
def run(rootfs, argv):
ql = Qiling(argv, rootfs, cputype=X86_CPU_MODEL.AMD_ATHLON)
print("cpu model", ql.uc.ctl_get_cpu_model())
ql.os.run()
I didn't notice until now but is it actually because of a missing syscall?
[!] 0x7ffff7dea1cf: syscall ql_syscall_rseq number = 0x14e(334) not implemented
/lib/libc.so.6: CPU ISA level is lower than required
I hooked the syscall in gdb, right here:
pwndbg> disassemble 0x7ffff7fd81d1
Dump of assembler code for function __tls_init_tp:
...
0x00007ffff7fd81ca <+218>: mov eax,0x14e
0x00007ffff7fd81cf <+223>: syscall
=> 0x00007ffff7fd81d1 <+225>: cmp eax,0xfffff000
...
End of assembler dump.
pwndbg> reg rax
*RAX 0xffffffffffffffda
pwndbg> print /d $rax
$1 = -38
So on this x8664 vm which can run normal binaries, i'm getting a return value from the rseq syscall -38.
I tried doing like mentioned above, to no avail. Both with the unsigned and signed version.
def run(rootfs, argv):
ql = Qiling(argv, rootfs, cputype=X86_CPU_MODEL.AMD_ATHLON)
print("cpu model", ql.uc.ctl_get_cpu_model())
def null_rseq_impl(ql: Qiling, abi: int, length: int, flags: int, sig: int):
print("hook rseq")
return -38
ql.os.set_syscall('rseq', null_rseq_impl, QL_INTERCEPT.CALL)
ql.os.run()
hook rseq
[=] null_rseq_impl(abi = 0x7fffb7fff9e0, length = 0x20, flags = 0x0, sig = 0x53053053) = -0x26 (ENOSYS)
/lib/libc.so.6: CPU ISA level is lower than required
hook rseq
[=] null_rseq_impl(abi = 0x7fffb7fff9e0, length = 0x20, flags = 0x0, sig = 0x53053053) = 0xffffffffffffffda
/lib/libc.so.6: CPU ISA level is lower than required
It seems the rseq syscall is being called from ld-linux-x86-64.so.2, so it's possibly unrelated to the ISA level check which happens in libc....
Here is a fairly clean workaround for now. No need to set cpumodel.
def try_patch_isa(ql: Qiling):
# 00023881 8b8a28030000 mov ecx, dword [rdx+0x328]
# 00023887 89cf mov edi, ecx
# 00023889 4421c7 and edi, r8d {0x0}
# 0002388c 39f9 cmp ecx, edi
# 0002388e 0f85000b0000 jne 0x24394 // from here
# 00023894 488d50f8 lea rdx, [rax-0x8]
# 00023898 4839f0 cmp rax, rsi
# 0002389b 75c3 jne 0x23860
# 0002389d 4c89ce mov rsi, r9 {0x3010102464c457f} // to here
# 000238a0 4c89ff mov rdi, r15
# 000238a3 e8d872ffff call sub_1ab80
pre = bytes.fromhex("8b8a2803000089cf4421c739f9")
ins = bytes.fromhex("0f85000b0000")
skip = bytes.fromhex("488d50f84839f075c3")
def bypass_isa_check(ql: Qiling) -> None:
print("Bypassing ISA Check...")
ql.arch.regs.rip += len(ins) + len(skip)
for start, end, perm, label, img in ql.mem.get_mapinfo():
if label != "ld-linux-x86-64.so.2":
continue
if "x" not in perm:
continue
adrs = ql.mem.search(pre + ins + skip, begin=start, end=end)
for adr in adrs:
ql.hook_address(bypass_isa_check, adr + len(pre))
ql = Qiling(...)
try_patch_isa(ql)
ql.run()
Indeed, the rseq syscall has nothing to do with this. That should have been resovled by selecting a newer CPU model, but it appears that Unicorn doesn't report compatibility to baseline CPUID features, so the check fails.
In the meantime, this is the workaround I am using. No need to hook the address, as this is just a one-time patching oepration:
ql = Qiling([r'./some_binary'], fr'{QL_HOME}/examples/rootfs/ubuntu-22.04-x64')
def __patch_cpu_isa_check(ql: Qiling) -> None:
ld_img = ql.loader.get_image_by_name('ld-2.31.so')
if ld_img:
ql.mem.write(ld_img.base + 0x2389f, bytes.fromhex('eb'))
__patch_cpu_isa_check(ql)
ql.run()