afl-cmin.py does not support Nyx mode

[pyoor]

AFL-cmin.py is currently missing a few checks that would allow it run under nyx mode.

The following blocks should be excluded when running under nyx mode.

https://github.com/AFLplusplus/AFLplusplus/blob/20348a63bd294af0fea4f591bc955bd71972bc37/afl-cmin.py#L230-L232

https://github.com/AFLplusplus/AFLplusplus/blob/20348a63bd294af0fea4f591bc955bd71972bc37/afl-cmin.py#L580C1-L588C67

I would have made a PR to fix these but it looks like there's additional problems that need to be addressed and I don't currently have time to investigate them.

[QEMU-NYX] Loading pre image to start fuzzing...
Process Worker-3:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/srv/repos/AFLplusplus/afl-cmin.py", line 455, in run
    if idx < m[t]:
IndexError: array index out of range

[vanhauser-thc]

@kcwu do you want to take care of that? :-)

[kcwu]

I am not familiar with Nyx. I can try this weekend

[vanhauser-thc]

@kcwu any updates? thank you

[kcwu]

nyx mode doesn't work well for me and I don't have time to dig the issue. Anyway, I created this PR to fix some issues. https://github.com/AFLplusplus/AFLplusplus/pull/2467

I don't know how to run the nyx binary with AFL_DUMP_MAP_SIZE=1, so it cannot detect map size automatically and relay on users to set AFL_MAP_SIZE environment variable.

[vanhauser-thc]

@schumilo any idea how a python script can obtain the map size of a nyx_mode setup?

[pyoor]

The original afl-cmin (awk version) never properly handled this. Checking for AFL_DUMP_MAP_SIZE will fail silently because target_bin is a directory. See here.

We fixed this locally by avoiding this check in nyx mode and allow over-riding AFL_MAP_SIZE. See here.

I don't think there is a way to reliably determine the binary being targeted in QEMU.

[vanhauser-thc]

The original afl-cmin (awk version) never properly handled this. Checking for AFL_DUMP_MAP_SIZE will fail silently because target_bin is a directory. See here.

We fixed this locally by avoiding this check in nyx mode and allow over-riding AFL_MAP_SIZE. See here.

I don't think there is a way to reliably determine the binary being targeted in QEMU.

@pyoor would you mind sending a PR with your afl-cmin* fixes?

IMHO this is not a general fix for nyx, this needs a different solution, but I think this needs something in nyx.

[schumilo]

nyx mode doesn't work well for me and I don't have time to dig the issue. Anyway, I created this PR to fix some issues. #2467

I don't know how to run the nyx binary with AFL_DUMP_MAP_SIZE=1, so it cannot detect map size automatically and relay on users to set AFL_MAP_SIZE environment variable.

The only supported way to obtain the map size in Nyx mode is via nyx_get_bitmap_buffer() (see: https://github.com/AFLplusplus/AFLplusplus/blob/stable/src/afl-forkserver.c#L122C1-L123C1).

While a map size value can be provided per target via the configuration in the package dir, the actual map size is later negotiated by AFL++ and the Nyx backend and therefore can only be retrieved via nyx_get_bitmap_buffer().

So, what we might want to do here is call into libnyx from within Python. In particular, call nyx_new() followed by nyx_get_bitmap_buffer_size(). This shouldn't be too difficult, since we're already using libnyx.so in the fuzzer.

Let me know if I can help with that :)

[schumilo]

This should do the trick:

import ctypes
import uuid
import os

def get_nyx_map_size(target_dir):
    libnyx = ctypes.CDLL('./libnyx.so')

    NYX_ROLE_StandAlone = 0

    target_dir_c = target_dir.encode('utf-8')
        
    dummy_workdir_path = "/tmp/_afl_cmin_nyx_work_dir_%s"%(str(uuid.uuid4()))
    dummy_workdir_path_c = dummy_workdir_path.encode('utf-8')

    # nyx_config_load
    libnyx.nyx_config_load.argtypes = [ctypes.c_char_p]
    libnyx.nyx_config_load.restype = ctypes.c_void_p
    nyx_config = libnyx.nyx_config_load(target_dir_c)

    # nyx_config_set_workdir_path
    libnyx.nyx_config_set_workdir_path.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
    libnyx.nyx_config_set_workdir_path.restype = None 
    libnyx.nyx_config_set_workdir_path(nyx_config, dummy_workdir_path_c)

    # nyx_config_set_process_role
    libnyx.nyx_config_set_process_role.argtypes = [ctypes.c_void_p, ctypes.c_int]
    libnyx.nyx_config_set_process_role.restype = None 
    libnyx.nyx_config_set_process_role(nyx_config, NYX_ROLE_StandAlone)

    # nyx_new
    libnyx.nyx_new.argtypes = [ctypes.c_void_p, ctypes.c_int]
    libnyx.nyx_new.restype = ctypes.c_void_p
    nyx_runner = libnyx.nyx_new(nyx_config, 0)

    # nyx_get_bitmap_buffer_size
    libnyx.nyx_get_bitmap_buffer_size.argtypes = [ctypes.c_void_p]
    libnyx.nyx_new.restype = ctypes.c_int
    map_size = libnyx.nyx_get_bitmap_buffer_size(nyx_runner)

    # nyx_shutdown
    libnyx.nyx_shutdown.argtypes = [ctypes.c_void_p]
    libnyx.nyx_shutdown.restype = None 
    libnyx.nyx_shutdown(nyx_runner)

    # nyx_remove_work_dir
    libnyx.nyx_remove_work_dir.argtypes = [ctypes.c_char_p]
    libnyx.nyx_remove_work_dir.restype = None 
    libnyx.nyx_remove_work_dir(dummy_workdir_path_c)

    return map_size

NYX_TARGET = "/tmp/nyx_libxml2/"
print(get_nyx_map_size(NYX_TARGET))

[kcwu]

@schumilo Thanks for the code. I integrated it to afl-cmin.py

@pyoor Could you please test my PR https://github.com/AFLplusplus/AFLplusplus/pull/2467 ? Does it work for you?

[pyoor]

@kcwu @vanhauser-thc sorry for the late response. I see that #2467 landed. Is there anything you still need from me?