Resolving issue of single byte buffers causing floating point exception

[Bwall]

Not sure if this is an official repository, but I ran into an issue where a single byte buffer being tested would lead to a floating point exception. While developers using this library could simply not supply such a buffer, I believe it is simpler to update the library to just ignore these buffers.

I'm in the process of fuzzing libemu. This issue was additionally identified in afl-fuzz as well as live testing with the libemu version in the Ubuntu repository.

I'll update with more patches if anymore crashes come up from afl.

[buffer]

This is not an official repository but just a mirror. BTW it is not entirely clear to me the patch to be applied to emu_shellcode.c. If size is 0, everything is already correctly handled because the for loop is not entered at all. But I can't understand why if size is 1, there are issues. Can you provide additional details about the crash?

[Bwall]

Absolutely: In this test, we are running emu_shellcode_test on a buffer that only contains \xe8

We get the fallowing stack trace: Program received signal SIGFPE, Arithmetic exception. 0x00007ffff7afdfb6 in emu_hashtable_search () from /usr/lib/libemu.so.2 (gdb) bt #0 0x00007ffff7afdfb6 in emu_hashtable_search () from /usr/lib/libemu.so.2 #1 0x00007ffff7afe188 in emu_hashtable_insert () from /usr/lib/libemu.so.2 #2 0x00007ffff7aff980 in emu_source_instruction_graph_create () from /usr/lib/libemu.so.2 #3 0x00007ffff7aff5aa in emu_shellcode_test () from /usr/lib/libemu.so.2

In emu_source_instruction_graph_create, a hashtable where the length is defined as size/2 (https://github.com/buffer/libemu/blob/master/src/emu_source.c#L47). If size is 1, the size of that hashtable then becomes 0. In emu_hashtable_search, a value has a modulus applied with the size of the hashtable, as 0, causes SIGFPE.

The patch in emu_hashtable_search is intended to fix the cause of the issue, and the patch in emu_shellcode_test is what I like to refer to as a "fail fast optimization". Given a lack of single byte shellcode, it is better to just fail before continuing forward. It may make sense to apply a brute force methodology to libemu in order to determine the minimum sized buffer that shellcode can be detected in.

[buffer]

Maybe I spotted the bug but can't reproduce it right now. The issue is here

https://github.com/buffer/libemu/blob/master/src/emu_hashtable.c#L96

As you correctly said eh->size ends up to be 0 in your scenario (and shellcode length 1 is the only one which can cause it). The best way to patch is IMHO something like this

uint32_t first_hash = eh->hash(key);

if (eh->size) first_hash %= eh->size;

Can you please test this patch in your scenario? Otherwise I will take care of it in the next hours.

Thanks and apologies for the late reply.