Add support for VMware port IO backdoor as secondary hypercall interface

[Wenzel]

This ticket explores what's the status of implementing the Nyx hypercall API via the VMware port IO backdoor.

kAFL agent

On the guest side, in libnyx_agent.c, there is a dispatcher already implemented

/**
 * Execute hypercall depending on Nyx CPU type
 */
unsigned long hypercall(unsigned id, uintptr_t arg)
{
	switch (nyx_cpu_type) {
	case nyx_cpu_v1:
		debug_printf("\t# vmcall(0x%x,0x%lx) ..\n", id, arg);
		return kAFL_hypercall(id, arg);
	case nyx_cpu_v2:
	case nyx_cpu_none:
		debug_printf("\t# vmcall(0x%x,0x%lx) skipped..\n", id, arg);
		return 0;
	case nyx_cpu_invalid:
	default:
		fprintf(stderr, "get_nyx_cpu_type() must be called first\n");
		habort_msg("get_nyx_cpu_type() must be called first\n");
		assert(false);
	}
}

depending on the CPU type returned by CPUID.

/**
 * Get Nyx VMM type from CPUID
 */
static nyx_cpu_type_t _get_nyx_cpu_type(void)
{
	uint32_t regs[4];
	char str[17];

	cpuid(KAFL_CPUID_IDENTIFIER, regs[0], regs[1], regs[2], regs[3]);

	memcpy(str, regs, sizeof(regs));
	str[16] = '\0';

	//debug_printf("CPUID string: >>%s<<\n", str);

	if (0 == strncmp(str, "NYX vCPU (PT)", sizeof(str))) {
		return nyx_cpu_v1;
	} else if (0 == strncmp(str, "NYX vCPU (NO-PT)", sizeof(str))) {
		return nyx_cpu_v2;
	} else {
		return nyx_cpu_none;
	}
}

QEMU-Nyx

On the QEMU side, in kvm-all.c:kvm_cpu_exec(), the handler for the VMware port forwarding the Nyx hypercalls seems to already be here

        switch (run->exit_reason) {
        case KVM_EXIT_IO:
            DPRINTF("handle_io\n");
#ifdef QEMU_NYX
            // clang-format on
            if (run->io.port == 0x5658 && run->io.size == 4 &&
                *((uint32_t *)((uint8_t *)run + run->io.data_offset)) == 0x8080801f)
            {
                assert(kvm_state->nyx_no_pt_mode);
                ret = handle_vmware_hypercall(run, cpu);
                break;
            }
// clang-format off
#endif

static int handle_vmware_hypercall(struct kvm_run *run, CPUState *cpu)
{
    kvm_arch_get_registers_fast(cpu);

    X86CPU      *x86_cpu = X86_CPU(cpu);
    CPUX86State *env     = &x86_cpu->env;

    return handle_kafl_hypercall(run, cpu, env->regs[R_EBX] + 100, env->regs[R_ECX]);
}

Related

• on QEMU, the vmport parameter toggles the emulation of the VMware backdoor. (patch)
• on KVM, the vmware backdoor can be toggled with enable_vmware_backdoor=y on the kvm module

@schumilo

• do you happen to have an existing implementation of the glude code necessary in the guest to issue hypercalls with the vmware backdoor ? Since the implementation exists in QEMU, I guest it was already working before ?
• can you detail a bit the existing code in QEMU ? I'm not clear on this line: *((uint32_t *)((uint8_t *)run + run->io.data_offset)) == 0x8080801f). Also handle_kafl_hypercall(run, cpu, env->regs[R_EBX] + 100, env->regs[R_ECX]); why +100 ? 🤔

Thanks !

[schumilo]

The current status regarding vmware backdoor io hypercalls is as follows:

-> We already use this interface in case KVM-Nyx is not installed, and the user does not need Intel PT support (e.g., AFL++ with compile-time instrumentations; thus, the CPU type NYX vCPU (NO-PT)). AFAIK this interface might be disabled if KVM-Nyx is detected by QEMU-Nyx (and consequently, QEMU-Nyx reports the vmcall CPU type) ... but I'm not 100% sure if this is the current state.

-> Btw the reason for using the vmware backdoor interface is that this type of "hypercalls" can be issued from userland and is not limited to CPL0 (like cpuid).

-> So, in case you need a working setup: AFL++ in Nyx mode on a system running a vanilla kernel + vanilla KVM should work just fine (the packer already has an option to either build the agent and all htools with this hypercall type).

-> *((uint32_t *)((uint8_t *)run + run->io.data_offset)) == 0x8080801f) is just a quick hack to avoid collisions with other vmware backdoor based hypercalls (0x8080801f is just a magic number used by Nyx). And the +100 offset is just there to adjust the hypercall number to the KVM exit reason number. Basically, this happens in the kernel in case we use KVM-Nyx (hypercall + 100 -> Nyx exit reason), but it needs to be adjusted in userland in case we use vmware hypercalls.

See: https://github.com/nyx-fuzz/KVM-Nyx/blob/kvm-nyx-5.10.73/arch/x86/kvm/x86.c#L8295C1-L8296C1 (KAFL_EXIT_OFFSET)