"[QEMU-NYX] Waiting for snapshot to start fuzzing..." but nothing

[leozhao2333]

Hello! I have some issues when using kafl to fuzz windows *.sys file. It seems that QEMU-NYX can't booting VM to starting fuzzing. The terminal output will stuck in "[QEMU-NYX] Waiting for snapshot to start fuzzing..." for minute. The output log shows below, sorry for some confusion about personal information in *

(.venv) *:~/Desktop/kafl/kAFL/kafl/examples/windows_x86_64$ kafl fuzz -p 32

    __                        __  ___    ________
   / /_____  _________  ___  / / /   |  / ____/ /
  / //_/ _ \/ ___/ __ \/ _ \/ / / /| | / /_  / /
 / ,< /  __/ /  / / / /  __/ / / ___ |/ __/ / /___
/_/|_|\___/_/  /_/ /_/\___/_/ /_/  |_/_/   /_____/
===================================================

<< kAFL Fuzzer >>

Warning: Launching without --seed-dir?
No PT trace region defined.
Warning: Requested 32 workers but 0 out of 32 vCPUs seem busy?
00:00:00:     0 exec/s,    0 edges,  0% favs pending, findings: <0, 0, 0>
Worker-00 Launching virtual machine...
*/Desktop/kafl/kAFL/kafl/qemu/x86_64-softmmu/qemu-system-x86_64
	-enable-kvm
	-machine kAFL64-v1
	-cpu kAFL64-Hypervisor-v1,+vmx
	-no-reboot
	-net none
	-display none
	-chardev socket,server,id=nyx_socket,path=/dev/shm/*/interface_0
	-device nyx,chardev=nyx_socket,workdir=/dev/shm/*,worker_id=0,bitmap_size=65536,input_buffer_size=131072
	-device isa-serial,chardev=kafl_serial
	-chardev file,id=kafl_serial,mux=on,path=/dev/shm/*/serial_00.log
	-m 4096
	-drive file=/home/*/.local/share/libvirt/images/windows_x86_64_vagrant-kafl-windows.img
	-monitor unix:/tmp/monitor.sock,server,nowait
	-fast_vm_reload path=/dev/shm/*/snapshot/,load=off
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Booting VM to start fuzzing...
Worker-01 Launching virtual machine...
Worker-02 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-03 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-04 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-05 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-06 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-07 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-08 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-09 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-10 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-11 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-12 Launching virtual machine...
Worker-13 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-14 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-15 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-16 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-17 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-18 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-19 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-20 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-21 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-22 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-23 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-24 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-25 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-26 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-27 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-28 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-29 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-30 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-31 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...

My core version is :

*:~$ uname -r
5.10.75-051075-generic

And my process information is:

*:~$ lscpu
架构：                   x86_64
  CPU 运行模式：         32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  字节序：               Little Endian
CPU:                     32
  在线 CPU 列表：        0-31
厂商 ID：                GenuineIntel
  型号名称：             13th Gen Intel(R) Core(TM) i9-13900HX
    CPU 系列：           6
    型号：               183
    每个核的线程数：     2
    每个座的核数：       24
    座：                 1
    步进：               1
    CPU 最大 MHz：       6900.0000
    CPU 最小 MHz：       800.0000
    BogoMIPS：           4838.40
    标记：               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
                         a cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss 
                         ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art
                          arch_perfmon pebs bts rep_good nopl xtopology nonstop_
                         tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes6
                         4 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr p
                         dcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline
                         _timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefe
                         tch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp
                          ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ep
                         t_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpc
                         id rdseed adx smap clflushopt clwb intel_pt sha_ni xsav
                         eopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp 
                         hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip pku 
                         ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir
                         64b fsrm md_clear serialize arch_lbr flush_l1d arch_cap
                         abilities
Virtualization features: 
  虚拟化：               VT-x
Caches (sum of all):     
  L1d:                   896 KiB (24 instances)
  L1i:                   1.3 MiB (24 instances)
  L2:                    32 MiB (12 instances)
  L3:                    36 MiB (1 instance)
NUMA:                    
  NUMA 节点：            1
  NUMA 节点0 CPU：       0-31
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy ba
                         rriers only; no swapgs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[leozhao2333]

Sorry for the terminal output in Chinese :no_mouth:

[5angjun]

You have to change your kernel version to nyx kernel. Then you can enjoy kAFL

[leozhao2333]

Hello! I do not install the Nyx kernal, but I actually install the NYX-QEMU patch, and I actually successfully started the KAFL before, but after once start up my machine, the components output becomes here 🥹

[5angjun]

It is different between nyx-qemu and nyx-kernel.

You can get nyx-qemu's code coverage if you patch your kernel to nyx kernel modifying hypervisor modules to get qemu coverage instead of host coverage.

[leozhao2333]

Okey I see. I try to change the kernal to nyx-kernal, but do you have any suggestion about how to fix [QEMU-NYX] Booting VM to start fuzzing... but waiting and waiting and get no response Since I config some similar scripts using the kAFL successfully before, but suddenly just once start up my machine, it just into a while true do: waiting infinity loop

[5angjun]

I cannot fully understand your environment and Bugs. Because i don't know your full context.

But the way to resolve these kind of porblems is the set --debug options and observe the hprintf log or other log files from QEMU. Or reset the host Ubuntu and try reinstalling kAFL.

[leozhao2333]

Sorry for my not precise expression. I meet the trouble is that: the kAFL sometimes work, but sometimes do not work even I don't change any config file. The cannot work behavior is: the first thread stuck in [QEMU-NYX] Booting VM to start fuzzing... and the other thread stuck in [QEMU-NYX] Waiting for snapshot to start fuzzing... BUT it in small probability randomly works, the attempts I have tried:

• desytoy the vegrant host machine and up again.
• reboot the host
• rebuild the image
• recompile and remake settings for the virtual machine after restoring from the snapshot

I wander how to fix this situation, if you need more information about my host system, could you please explicate what you need.

[5angjun]

Can you show your agent.cpp or harness? Specifically, Insert hprintf line by line in your harness and observe the output. Through this, We can finally notice which point the fuzz was stucked with and point out what we need to do.

[leozhao2333]

Hello! Thanks for your reply. I have some guess about this situation. If the driver which is been config in the setup_target.yml cannot been installed by the sc.exe in the virtual machine, the frame will believe that the virtual machine don't start yet, so it will stuck and perform as [QEMU-NYX] Booting VM to start fuzzing...

[5angjun]

So it seems you cannot get windows driver handles.

you can check if target driver successfully loaded or installed in your guest OS.

I think you are getting closer to correct answer.🙂

[leozhao2333]

Yes, I find the target driver is not successfully loaded in my virtual OS. Thanks for your patient answer! 👍

[leozhao2333]

Hello! sorry for another question :( I meet such output when starting the kAFL. And I have no idea where to check why the framework meets NO MATCH!

Initiate fuzzer handshake...
	host_config.bitmap_size: 0x10000
	host_config.ijon_bitmap_size: 0x1000
	host_config.payload_buffer_size: 0x20000
Submitting bug check handlers
Worker-00 Guest ABORT: FAIL! NO MATCH!

Worker-00 Failed to connect to Qemu: Guest ABORT: FAIL! NO MATCH!

Worker-00 Shutting down Qemu after 0 execs..

And I upload the setup_target.yml harness vuln_drivers for more information. Additional_file.zip

Thanks for your reply!

[leozhao2333]

Hello! Sorry for bring some inconvenience! I find that I make some typo when setting driver_name. And I successfully enter into fuzz loop!

[ducphanduyagentp]

For those that may be interested, this hanging situation also happens if you don't run the kafl fuzz command from the directory with the Vagrant file in this example. Although this is not the case here, it was what happened to me with the windows userspace target example. Tried to run kafl fuzz from the top directory and it just hung. Hopefully this helps someone.