Encoding problem when using hook_ReadFile

[frozenkp]

Describe the bug My target program contains two continuous API calls, CreateFile then ReadFile. When calling CreateFile, the hooking function opens a file with r. (f = open(filename, 'r')) Since the file, it opens, is a shellcode file, it is stuck in an encoding problem of 'UTF-8' when reading.

I found the problem is that when reading a binary file, it may be stucks since there is no encoding for a binary file. We should modify open mode in kernel32/fileapi.py from r to rb. I'm not sure how many instructions would be influenced if I modified it, so I open this issue before I make any modification.

def _CreateFile(ql, address, params, name):
    ret = INVALID_HANDLE_VALUE
    s_lpFileName = params["lpFileName"]
    dwDesiredAccess = params["dwDesiredAccess"]
    dwShareMode = params["dwShareMode"]
    lpSecurityAttributes = params["lpSecurityAttributes"]
    dwCreationDisposition = params["dwCreationDisposition"]
    dwFlagsAndAttributes = params["dwFlagsAndAttributes"]
    hTemplateFile = params["hTemplateFile"]

    # access mask DesiredAccess
    mode = ""
    if dwDesiredAccess & GENERIC_WRITE:
        mode += "wb"
    else:
        mode += "r"                            # <== here should be 'rb'

    try:
        f = ql.os.fs_mapper.open(s_lpFileName, mode)
    except FileNotFoundError:
        ql.os.last_error = ERROR_FILE_NOT_FOUND
        return INVALID_HANDLE_VALUE

    new_handle = Handle(obj=f)
    ql.os.handle_manager.append(new_handle)
    ret = new_handle.id
    return ret

BTW, I found that in hook_ReadFile. It reads file then write to memory. data here is a string if we use r when opening, while ql.mem.write accepts bytes for data.

f = ql.os.handle_manager.get(hFile).obj
data = f.read(nNumberOfBytesToRead)
ql.mem.write(lpBuffer, data)

Sample Code

Expected behavior Read the file successfully.

Screenshots The calling instruction in 0x4010d3 is call ReadFIle.

Additional context

[xwings]

We need to test to figure out. Maybe you can give us an answer.

[xwings]

Will you be able to try the latest version of Qiling and see if you still face same issue. There is lots of rework since 2021. Feel free to open a new issue if you have any similar problem.