qiling
qiling copied to clipboard
qilingframework
How can I have buffer overflows (Invalid memory fetch) fail gracefully? I already tried hooking.
I wanted to test using qiling for exploitation development. So I'm passing a cyclic buffer to the program to generate a buffer overflow. A core file gets generated upon the crash, but I can't seem to figure out how to make it fail gracefully so that I can analyze the register states (core file) after the program crashes:
from cyclic import cyclic, cyclic_find
# Qiling
# Full system emulation
from qiling import *
buffer = cyclic(50, n=4)
print(f'generated cyclic buffer: {buffer}')
# pass it to program for an overflow
BIN_PATH = ["exploit_this", buffer]
ENV = "../examples/rootfs/x86_linux"
ql = Qiling(BIN_PATH, ENV)
def dissassem(ql, address, size):
buf = ql.mem.read(address, size)
for i in md.disasm(buf, address):
print("0x%x:\t%s\t%s" %(i.address, i.mnemonic, i.op_str))
def dump_regs(ql, int_code):
print(f'registers:')
for reg_key in ql.reg.register_mapping:
print(f'{reg_key.upper()}: {ql.reg.read(reg_key.upper())}')
# ql.hook_code(dissassem)
ql.hook_intr(dump_regs)
ql.hook_mem_unmapped(dump_regs)
ql.hook_mem_read_invalid(dump_regs)
ql.hook_mem_write_invalid(dump_regs)
ql.hook_mem_fetch_invalid(dump_regs)
ql.hook_mem_invalid(dump_regs)
ql.run()
this_reg_was_tainted = ql.reg.read("EAX")
print(f'EAX: {this_reg_was_tainted}')
offset = cyclic_find(this_reg_was_tainted)
# now we know where to insert a payload
[+] Buffer: aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaama
[=] write(fd = 0x1, buf = 0x804d160, count = 0x3f) = 0x3f
registers:
AH: 0
AL: 63
CH: 209
CL: 96
DH: 0
DL: 63
BH: 0
BL: 1
AX: 63
CX: 53600
DX: 63
BX: 1
SP: 51100
BP: 32128
SI: 53600
DI: 63
IP: 44114
EAX: 63
ECX: 134533472
EDX: 63
EBX: 1
ESP: 2146682780
EBP: 2003402112
ESI: 134533472
EDI: 63
EIP: 75213906
CR0: 17
CR1: 0
CR2: 0
CR3: 0
CR4: 0
CR8: 0
ST0: 0
ST1: 0
ST2: 0
ST3: 0
ST4: 0
ST5: 0
ST6: 0
ST7: 0
EF: 68
CS: 27
SS: 40
DS: 40
ES: 40
FS: 0
GS: 99
[x] CPU Context:
[x] ah : 0x0
[x] al : 0x3f
[x] ch : 0x0
[x] cl : 0x0
[x] dh : 0x88
[x] dl : 0x90
[x] bh : 0x61
[x] bl : 0x61
[x] ax : 0x3f
[x] cx : 0x0
[x] dx : 0x8890
[x] bx : 0x6161
[x] sp : 0xce20
[x] bp : 0x6161
[x] si : 0x7000
[x] di : 0x0
[x] ip : 0x6161
[x] eax : 0x3f
[x] ecx : 0x0
[x] edx : 0x77698890
[x] ebx : 0x61696161
[x] esp : 0x7ff3ce20
[x] ebp : 0x616a6161
[x] esi : 0x77697000
[x] edi : 0x0
[x] eip : 0x616b6161
[x] cr0 : 0x11
[x] cr1 : 0x0
[x] cr2 : 0x0
[x] cr3 : 0x0
[x] cr4 : 0x0
[x] cr8 : 0x0
[x] st0 : 0x0
[x] st1 : 0x0
[x] st2 : 0x0
[x] st3 : 0x0
[x] st4 : 0x0
[x] st5 : 0x0
[x] st6 : 0x0
[x] st7 : 0x0
[x] ef : 0x4
[x] cs : 0x1b
[x] ss : 0x28
[x] ds : 0x28
[x] es : 0x28
[x] fs : 0x0
[x] gs : 0x63
[x] PC = 0x616b6161 (unreachable)
[=] Memory map:
[=] Start End Perm Label Image
[=] 00030000 - 00031000 rwx [GDT] exploit_this
[=] 047ba000 - 047e2000 rwx /home/student/synthesis/examples/rootfs/x86_linux/lib/ld-linux.so.2 exploit_this
[=] 08048000 - 08049000 r-x exploit_this exploit_this
[=] 08049000 - 0804b000 rw- exploit_this exploit_this
[=] 0804b000 - 0804d000 rwx [hook_mem]
[=] 0804d000 - 0806e000 rwx [brk]
[=] 0806e000 - 0806f000 rwx [brk]
[=] 774bf000 - 7769b000 rwx [syscall_mmap2]
[=] 774bf000 - 7769b000 rwx [mmap2] /home/student/synthesis/reverse/../examples/rootfs/x86_linux/lib/libc.so.6
[=] 77695000 - 77698000 rwx [mmap2] /home/student/synthesis/reverse/../examples/rootfs/x86_linux/lib/libc.so.6
[=] 7769b000 - 7769d000 rwx [syscall_mmap2]
[=] 7ff0d000 - 7ff3d000 rwx [stack]
Traceback (most recent call last):
File "reverse.py", line 34, in <module>
ql.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 707, in run
self.os.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/os/linux/linux.py", line 139, in run
self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 867, in emu_start
self.uc.emu_start(begin, end, timeout, count)
File "/usr/local/lib/python3.6/dist-packages/unicorn/unicorn.py", line 465, in emu_start
raise UcError(status)
unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)
You can see in the core file dumped during the crash, the EIP value 0x616b6161 is my buffer overflow, but in the last hooked handler the EIP isnt set to that, so I never have access to a crashed instance of the registers.
Is there something else I need to do to prevent the python code from throwing an error when the emulated program crashes after an invalid memory fetch? I thought just hooking the invalid fetch would work but that hook doesn't even seem to trigger.
https://github.com/unicorn-engine/unicorn/wiki/FAQ#why-do-i-get-a-wrong-pc-after-emulation-stops
@wtdcode "Why do I get a wrong PC after emulation stops? PC is only guaranteed to be correct if you install UC_HOOK_CODE. This is due to the fact that updating PC is a big performance overhead during emulation." It's not that I'm getting an "incorrect" PC, on the contrary I'm getting exactly the EIP that I want. I'm intentionally overflowing the EIP to break the program. It's not that I don't understand why it's "wrong", it's actually exactly what I want it to be. I just want the emulation to fail gracefully after it goes out of bounds and triggers that bad memory fetch so I can then analyze the final state when that EIP got clobbered by my exploit.
@jt0dd Maybe you can try to use ql.hook_mem_fetch_invalid(custom_hook). It will stop the execution before raising the error "Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)". In custom_hook(), you can drop ipython to analyze the final state.
@cq674350529 but look at the code I posted. I'm already using that hook. Do I need to use some function to stop execution?
@cq674350529 but look at the code I posted. I'm already using that hook. Do I need to use some function to stop execution?
/*
Callback function for tracing invalid instructions
@user_data: user data passed to tracing APIs.
@return: return true to continue, or false to stop program (due to invalid
instruction).
*/
typedef bool (*uc_cb_hookinsn_invalid_t)(uc_engine *uc, void *user_data);
Try to return false to stop emulation or call ql.stop()
@wtdcode It doesn't seem to work:
def stop(ql, data):
print('hooked invalid fetch')
ql.stop()
return False
ql.hook_mem_fetch_invalid(stop)
ql.run()
that log in the hook func never prints. The output I originally posted just throws the error without ever seeming to call my hook code.
I think you should return
Trueto prevent it from continuing.
Yeah maybe but I just tried that, the issue is the hook never gets called, at least not before the error gets thrown. I should at least be seeing that print log but it's not executing
Don't use print to print out status or progress; use ql.log.info [or ql.log.warning if you want it to stand out to spot it easily]
Don't use
ql.log.info[orql.log.warningif you want it to stand out to spot it easily]
Thanks, tried this. I see that if I test that out before starting the emulation, it logs with the yellow indicator with my test log, but replacing the print statement in my hook with ql.log.warning('hooked invalid fetch') still results in no log. The hook doesn't appear to work in this case. Let me know how I can help debug. I'm only vaguely familiar with the inner workings of qemu / unicorn so I'm not sure how much I can figure out.
Ping me on TG channel and I'll try to help there.
For starter, I think you hook attempts to fetch from invalid memory rather than from unmapped memory. Since there is no convinient function for it, you have to do this directly:
ql.ql_hook(UC_HOOK_MEM_FETCH_UNMAPPED, stop)
Ping me on TG channel and I'll try to help there.
I will this evening but I'm in an environment where I can't access my mobile phone. Any chance you could throw us a hint for where to start? Edit: Oh I see you added a hint in your prev comment.
@elicn
So I tried ql.ql_hook(UC_HOOK_MEM_FETCH_UNMAPPED, stop) and it didn't do anything so I went into the project code to find where that binding is happening and how its handled, found it here: https://github.com/qilingframework/qiling/blob/master/qiling/core_hooks.py
so I want to debug it a bit and make sure my hook is being applied, so I changed:
if t in (UC_HOOK_MEM_READ_UNMAPPED, UC_HOOK_MEM_WRITE_UNMAPPED, UC_HOOK_MEM_FETCH_UNMAPPED, UC_HOOK_MEM_READ_PROT, UC_HOOK_MEM_WRITE_PROT, UC_HOOK_MEM_FETCH_PROT, UC_HOOK_MEM_READ, UC_HOOK_MEM_WRITE, UC_HOOK_MEM_FETCH, UC_HOOK_MEM_READ_AFTER):
if t not in self._hook_fuc.keys():
self._hook_fuc[t] = self._ql_hook_internal(t, self._hook_mem_cb, t)
if t not in self._hook.keys():
self._hook[t] = []
self._hook[t].append(h)
to
if t in (UC_HOOK_MEM_READ_UNMAPPED, UC_HOOK_MEM_WRITE_UNMAPPED, UC_HOOK_MEM_FETCH_UNMAPPED, UC_HOOK_MEM_READ_PROT, UC_HOOK_MEM_WRITE_PROT, UC_HOOK_MEM_FETCH_PROT, UC_HOOK_MEM_READ, UC_HOOK_MEM_WRITE, UC_HOOK_MEM_FETCH, UC_HOOK_MEM_READ_AFTER):
if t not in self._hook_fuc.keys():
self._hook_fuc[t] = self._ql_hook_internal(t, self._hook_mem_cb, t)
print(f'DEBUG: {t} added to {self._hook_fuc}')
if t not in self._hook.keys():
self._hook[t] = []
self._hook[t].append(h)
print(f'DEBUG: we got this far, appending hook func: {h}, self._hook: {self._hook}')
and got the log after running my script with your suggested hook ql.ql_hook(UC_HOOK_MEM_FETCH_UNMAPPED, stop):
DEBUG: 64 added to {1: 15857872, 64: 17882592}
DEBUG: we got this far, appending hook func: <qiling.core_hooks_types.Hook object at 0x7ffff09a6a90>, self._hook: {1: [<qiling.core_hooks_types.HookIntr object at 0x7ffff0047048>], 64: [<qiling.core_hooks_types.Hook object at 0x7ffff09a6a90>]}
So far, it seems like the hook is being applied (at this level at least)... Digging deeper to see why it's not being triggered. Would be happy to hear any ideas.
Oh no... The next place I have to go is going to be inside qemu isn't it? And it seems to be C code. I haven't learned much C yet :/
Wait, it's simpler than that. Apparently your hook wasn't fired up because the hook callback didn't have all the required arguments list. It looks like Unicorn silently drops it in such case.
This one should work, however I couldn't get to thwart the exception and exit gracefully:
def __hook_ace(ql: Qiling, access: int, address: int, size: int, value: int):
# make sure we got here for the right reason
assert access == UC_MEM_FETCH_UNMAPPED
ql.log.warning(f'Execution flow direvted to {address:#x}')
ql.stop()
# apparently, this doesn't change anything in our case..
return QL_HOOK_BLOCK
# this hook combines a few forms of fetch errors
ql.hook_mem_fetch_invalid(__hook_ace)
ql.run()
I'm going to explore what you just said, but I also arrived at a conclusion of the issue, but a different one, in https://github.com/unicorn-engine/unicorn/blob/master/bindings/python/unicorn/unicorn.py
the code unicorn is using is:
# emulate from @begin, and stop when reaching address @until
def emu_start(self, begin, until, timeout=0, count=0):
status = _uc.uc_emu_start(self._uch, begin, until, timeout, count)
if status != uc.UC_ERR_OK:
raise UcError(status)
if self._hook_exception is not None:
raise self._hook_exception
# stop emulation
def emu_stop(self):
status = _uc.uc_emu_stop(self._uch)
if status != uc.UC_ERR_OK:
raise UcError(status)
Which, they're choosing to raise an error if something bad happened, when we want to actually not raise an error but rather gracefully handle the thing that happened gracefully. I commented out that error raising behavior to test this theory:
def emu_start(self, begin, until, timeout=0, count=0):
print('edited the right file...')
status = _uc.uc_emu_start(self._uch, begin, until, timeout, count)
if self._hook_exception is not None:
print('raising hook exception')
#raise self._hook_exception
elif status != uc.UC_ERR_OK:
print('raising uc err (start)')
#raise UcError(status)
# stop emulation
def emu_stop(self):
status = _uc.uc_emu_stop(self._uch)
if status != uc.UC_ERR_OK:
print('raising uc err (stop)')
#raise UcError(status)
and the logs are showing as expected, but a new error is thrown:
raising uc err (start)
Traceback (most recent call last):
File "reverse.py", line 58, in <module>
ql.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 707, in run
self.os.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/os/linux/linux.py", line 139, in run
self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 870, in emu_start
raise self._internal_exception
File "/usr/local/lib/python3.6/dist-packages/qiling/utils.py", line 163, in wrapper
return func(*args, **kw)
File "/usr/local/lib/python3.6/dist-packages/qiling/core_hooks.py", line 130, in _hook_mem_cb
ret = h.call(ql, access, addr, size, value)
File "/usr/local/lib/python3.6/dist-packages/qiling/core_hooks_types.py", line 23, in call
return self.callback(ql, *args)
TypeError: stop() takes 2 positional arguments but 5 were given
I'm not sure which of us is on the right track here but now I'll look into what you just suggested, thanks.
Your
stopmethod should include 5 arguments, like my__hook_ace
Yes! That works!
However, only with my edit to unicorn. With unicorn being set to its original state, this error is thrown:
raceback (most recent call last):
File "reverse.py", line 59, in <module>
ql.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 707, in run
self.os.run()
File "/usr/local/lib/python3.6/dist-packages/qiling/os/linux/linux.py", line 139, in run
self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)
File "/usr/local/lib/python3.6/dist-packages/qiling/core.py", line 867, in emu_start
self.uc.emu_start(begin, end, timeout, count)
File "/usr/local/lib/python3.6/dist-packages/unicorn/unicorn.py", line 465, in emu_start
raise UcError(status)
unicorn.unicorn.UcError: Invalid memory mapping (UC_ERR_MAP)
So while this might be an intended behavior for unicorn (maybe?), I think depending on how this library is meant to be used (for exploit dev, in this case) we could call this a bug. They're raising errors regardless of the user's hook by default. I propose maybe we patch that unicorn behavior in this library. I will also go raise the issue with them because unicorn after all is also a reverse engineering tool and I see no reason not to support graceful failure.
Unless there's something else that can be done with the current state of the unicorn project to allow the graceful failure as-is?
@jt0dd add import ipdb; ipdb.set_trace() in the custom_hook function of ql.hook_mem_fetch_invalid(custom_hook), then you can stop it before raising error and analyze the context such as register, stack.
Although, as discussed above, after continues, it can't exit gracefully.
@cq674350529 interesting, thanks for the tip. I still think that inability to fail gracefully is not great for a workflow where you might want to crash the program hundreds of thousands of times.
See my comments here.
https://github.com/unicorn-engine/unicorn/issues/1484#issuecomment-970727708
@wtdcode I tested both options and they don't work [or I am missing something here].
Mapping the accessed memory before returning from the handler doesn't prevent the exception from being raised.
Sorrounding ql.run() with try and except UcError does make the except clause trigger, but the CPU context dump is still displayed.
