Add x86_64 support

[cube0x8]

The x64 support has started!

1. it is possible to map and link an x64 executable. Basic tests have been created.
2. An mpclient_x64 Makefile target has been created. For now, it's only for development purpose
3. x64 linux and windows calling conventions differ, as volatile and non-volatile registers. Two wrappers for HeapAlloc and HeapFree APIs have been created in order to prove we can effectively call the winapi in x64 mode!

[cube0x8]

I have just replaced libdisasm with Zydis. I did basic tests (only x86) for the insert_function_redirect and redirect_call_within_function functions and they passed but please, @taviso, when/if you have time check and let me know if it breaks your stuff!

Next steps:

1. add x64 support to hook.c
2. create dispatchers to switch linux <=> windows calling conventions

[cube0x8]

I added x86_64 support for the intercept set of libraries:

1. added two assembly (NASM) dispatchers to switch calling convention from x86_64 linux to x86_64 windows and the other way around
2. It is possible to create a HOOK_FUNCTION_REPLACE redirect. It is not possible (yet) to create a HOOK_DEFAULT redirect
3. Added a suite test with a test DLL to prove that the redirect and the calling convention switch is working. The code of the DLL has been added too and it has to be built (VS2019) before running the tests)
4. mpclient_x64 does not work yet (SIGSEGV). I started debugging it to understand what's going on.
5. intercept became a CMake project

[cube0x8]

So,

1. I fixed the x64 assembly dispatchers. It looks like they are working quite well now
2. All the WINAPIs are correctly linked and executed
3. I managed to run a full scan of eicar.com using mpclient_x64.c
4. Tests have been removed

Now, the framework should be able to load, link and execute x86_64 DLLs.

On the other side, I guess I am doing something wrong with one (or more than one) of the mpengine data structures we provide to the __rsignal entry point. It happens that, when EngineScanCallback is called, Scan->Flags randomly changes between executions while it should be always the same as it happens in the x86 version. I probably will investigate this further, even if I reversed multiple time mpengine.dll and all the data structures seem correctly aligned. Also, I've been able to run a scan ONLY passing eicar.com as input. All the other files/binaries I've tested so far raise an exception and terminate the scan.

Next steps:

1. Provide a variadic function which, given a DLL exported function and its arguments as input, proceeds to switch the calling convention and call the user provided entry point.
2. Test more x86_64 DLLs
3. Provide additional hooking capabilities
4. Try to understand what's wrong with mpclient

[cube0x8]

It is possible to call DLL exports using the x86_64_call_exported_function wrapper.

[cube0x8]

Subhook hooks were pushing values on the redzone, which led to some useful values on the stack to be overwritten. I disabled the redzone for now (-mno-red-zone).

I managed to use loadlibrary for other x86_64 DLL - like the BitDefender engine - and I am able to perform scans on different files and binaries. Everything is working quite good I would say.

Still, mpclient just works (more or less, since Scan->Flags is still unreliable) with eicar.com and it fails (uncaught exception) with all the other files I tried so far.

One thing to keep in mind:

• the x86_64 mode works only if peloader is compiled without any optimization (-O0)

[cube0x8]

Update: with the new release of mpengine.dll (Product Version Number: 1.1.18200.2) it does not work also with eicar.com. It correctly boots up the engine but it raises an uncaught exception during the scan. I will try to sort this out as soon as I get some free time.

[taviso]

This is really impressive progress, I think we can try to merge this into main. Let me test it out today to make sure it's not regressing any x86 stuff, I'm a bit nervous about merging the hook and the x64 stuff in one - but maybe it will be easy...

[cube0x8]

Two notes:

1. The framework can't work with selinux enforcing, since mprotect fails to change permission on peloader's functions. I have to find a better solution than patching the APIs at runtime.
2. As I started using it for fuzzing purposes, bugs are mushrooming. For example, on Windows xmm6 and xmm7 are preserved registers and I forgot to store them through the calling convention switch.

I am going to open a couple of issues (assigned to me) to keep track of these bugs. I will provide a patch as soon as I can.

[cube0x8]

So... I have just found out that GCC (and Clang too) implements the function's attribute __attribute__((ms_abi)), which compiles a function with the Windows x86_64 calling convention. This solved more than a half of the workload on the x86_64 support.

Basically, all the heavy lifting with x64 NASM dispatchers and the new hook library is... USELESS.

Next steps:

1. clean the code
2. bring back and test the old good hooking library
3. implement SEH for x86_64 (https://llvm.org/docs/ExceptionHandling.html#wineh)

[cube0x8]

Updates:

1. Lately I've been working on the implementation for the x86_64 version of the SEH unwind support.
2. After 1) is done, I should also fix the regression for the plain x86 version.

@taviso good news: after 1) and 2) are done, I am 99% sure we fully achieved x86_64 native (and reliable) windows execution on Linux, and we would be ready to merge everything into master!

[cube0x8]

Added the support for the x64 Structured Exception Handling. I tested it with few x64 programs that use SEH and it worked.

This said, I also tested the last version of mpengine with different malicious programs: the Scan->Flags parameter is sometimes correct and it returns the right detection but - other times - it gets another value and the user does not get any detection. I am still not able to explain this behaviour, but I guess is due to some bad aligned data structure or incorrect peloader API.

Other AVs engines/x64 programs are correctly loaded and work as expected.

[taviso]

wow! 🥳

I think I don't really understand the flags even for x86, I just guessed from testing them experimentally. I think it probably just needs some reversing of msmpeng to see what it does with the flags, I'll get to it some time!

Do you think we should start trying to merge it into main?

[cube0x8]

I will start fixing the things for the x86 version and I think we can gradually merge them into main.

[cube0x8]

The x86 version of mpclient works. The x64 version is still unreliable.

I think we can start merging into main.

[redmed666]

Hi all! Maybe a stupid question but is "subhook/subhook.h" not missing from your branch @cube0x8 ? Because when trying to compile it, I get this error:

$ make
cc -march=native -ggdb3 -std=gnu99 -fshort-wchar -Wno-multichar -Iinclude -Iintercept/include -Ilog -Ipeloader -mstackrealign -maccumulate-outgoing-args -O3 -m32 -D_GNU_SOURCE -I. -DNDEBUG  -c -o mpclient.o mpclient.c
In file included from mpclient.c:46:
intercept/include/hook.h:4:10: fatal error: ../subhook/subhook.h: No such file or directory
    4 | #include "../subhook/subhook.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [<builtin>: mpclient.o] Error 1

[cube0x8]

Hello @redmed666

subhook is configured as a submodule in the x86/64 version of loadlibrary. Maybe did you forget to clone the submodules?

If you've already cloned the repository and you checked out on the x64 branch, then run: git submodule update --init --recursive

Otherwise, you should clone the repo using the --recurse-submodules switch: git clone --recurse-submodules --branch x64 https://github.com/cube0x8/loadlibrary.git

[redmed666]

It was indeed a stupid question. Thank you!

[shuxin]

Excellent job.

[shuxin]

after a few test， I found many faked api should change from noregmarm to ms_abi. for example crt.c

[cube0x8]

Hello @shuxin,

yeah I definetely forgot about the APIs in crt.c. Anyway, I ran few tests on mpclient_x64.c with different file inputs and never seen them executed. Can you confirm?

I'm planning to get back on this PR as soon as possible to finally make it work reliably for mpclient and merge it.

[shuxin]

Hello @shuxin,

yeah I definetely forgot about the APIs in crt.c. Anyway, I ran few tests on mpclient_x64.c with different file inputs and never seen them executed. Can you confirm?

I'm planning to get back on this PR as soon as possible to finally make it work reliably for mpclient and merge it.

mse mpengine does not depends on msvcrt#.dll . so it should works well. you can merge first. I am trying to load some other dlls, if I have any stable fix, I will make another pull request.

[kh-abd-kh]

Hi and thank you,

in the last step of make i get this error

cc -march=native -ggdb3 -std=gnu99 -fshort-wchar -Wno-multichar -Iinclude -Iintercept/include -Ilog -Ipeloader -mstackrealign -maccumulate-outgoing-args -O3 -g -O0 -fPIC mpclient_x64.o log/log.o -o mpclient_x64 -Wl,--whole-archive peloader/libpeloader.a -Wl,intercept/libhook.a -Wl,intercept/libZydis.a -Wl,intercept/libsubhook.a -Wl,--no-whole-archive -march=native -ggdb3 -std=gnu99 -fshort-wchar -Wno-multichar -Iinclude -Iintercept/include -Ilog -Ipeloader -mstackrealign -maccumulate-outgoing-args -O3 -g -O0 -fPIC -lm -Wl,--dynamic-list=exports.lst -ldl /usr/bin/ld: i386 architecture of input file log/log.o' is incompatible with i386:x86-64 output /usr/bin/ld: i386 architecture of input file peloader/libpeloader.a(Heap.o)' is incompatible with i386:x86-64 output /usr/bin/ld: i386 architecture of input file peloader/libpeloader.a(Files.o)' is incompatible with i386:x86-64 output /usr/bin/ld: i386 architecture of input file peloader/libpeloader.a(IsProcessorFeaturePresent.o)' is incompatible with i386:x86-64 output /usr/bin/ld: i386 architecture of input file `peloader/libpeloader.a(EncodePointer.o)' is incompatible with i386:x86-64 output

....

/usr/bin/ld: i386 architecture of input file intercept/libsubhook.a(subhook_unix.c.o)' is incompatible with i386:x86-64 output /usr/bin/ld: peloader/libpeloader.a(crt.o): in function _alldiv': /home/mabd/loadlibrary64/peloader/crt.c:421: undefined reference to __divdi3' /usr/bin/ld: peloader/libpeloader.a(crt.o): in function _aulldiv': /home/mabd/loadlibrary64/peloader/crt.c:427: undefined reference to __udivdi3' /usr/bin/ld: peloader/libpeloader.a(crt.o): in function _allrem': /home/mabd/loadlibrary64/peloader/crt.c:445: undefined reference to __moddi3' /usr/bin/ld: peloader/libpeloader.a(crt.o): in function _aullrem': /home/mabd/loadlibrary64/peloader/crt.c:451: undefined reference to __umoddi3' /usr/bin/ld: intercept/libZydis.a(String.c.o): in function ZydisStringAppendDecU64': String.c:(.text+0x3bf): undefined reference to `__udivdi3' collect2: error: ld returned 1 exit status make: *** [Makefile:47: mpclient_x64] Error 1

thank you

[kh-abd-kh]

I cloned everything with this, as mentioned above,

git clone --recurse-submodules --branch x64 https://github.com/cube0x8/loadlibrary.git

[cube0x8]

@kh-abd-kh it looks like you have a mix x86 and x86_64 environment. Do you want to execute a 64 or 32 bit DLL?

You might need to make clean first and then try to build your target again.

[ravone]

I leave this comment as yet another confirmation of the x86-64 implementation operability. Creating a binary file to interact with my dll took me less time than I expected. Wow! Thanks to the authors for a great job!

[cube0x8]

@ravone thank you very much for your feedback. I appreciate it :)

[ayoubfaouzi]

@cube0x8 any idea when this will get merged ?

[cube0x8]

It's on my radar. I don't want to promise anything but I could get my hand on it in the next months!

On Tue, 11 Apr 2023, 07:20 Noteworthy, @.***> wrote:

@cube0x8 https://github.com/cube0x8 any idea when this will get merged ?

— Reply to this email directly, view it on GitHub https://github.com/taviso/loadlibrary/pull/97#issuecomment-1502666678, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGW4SPC6AQDJ2FHCWRXH3DXATL2NANCNFSM4YR3A3RA . You are receiving this because you were mentioned.Message ID: @.***>