Undefined x86 opcodes known to be used for syscall-like functionality

[tremalrik]

There are a number of undefined x86 instruction encodings that have been used to provide syscall-like functionality through the #UD (int 6) invalid opcode exception handler. The ones I've been able to find are:

• C4 C4 imm8/imm16 : "BOP" - when running 16-bit DOS programs under Windows, this opcode is used to provide a mechanism for DOS programs to load and call Windows DLLs, see e.g. https://www.ragestorm.net/tutorial?id=27 or https://sta.c64.org/blog/dosvddaccess.html.
• 0F 3F imm8 imm8 : "VPCEXT" - VirtualPC supervisor call - mostly documented as a virtual machine detection method, e.g. https://shasaurabh.blogspot.com/2017/07/virtual-machine-detection-techniques.html.
• 0F C7 C8 imm8 imm8 : appears to be used by VirtualPC as well, but with no instruction mnemonic given ( https://web.archive.org/web/20110705055615/http://my.opera.com/jaelanicu/blog/just-another-vm-detection-was-vm-detection-combo ); appears to be a virtual machine detection method only, described in a still-active patent ( https://patents.google.com/patent/US7552426 )
• F0 90 : "LOCK NOP" : used by some versions of the L4 microkernel as a system call (see e.g. https://www.l4ka.org/l4ka/l4-x2-r7.pdf, pages 99 and 125)

The question would then be - should Zydis recognize any of these encodings?

[athre0z]

Thanks for the issue! We have considered this before, but decided against it. We only support what is implemented in hardware by Intel, AMD and VIA.

With these software instructions, it's hard to decide where to draw the line. There are hundreds (if not thousands) of hobby-hypervisors. There are very old hypervisors that are mostly dead (e.g. VirtualPC). Some hypervisors even use valid privileged instructions that happen to cause an exit to the hypervisor. All of them would need dedicated decoder modes and filters. We also don't want to just enable them by default, because obfuscators could emit them and confuse users into thinking that a valid instruction is going to be executed. We simply don't think it's worth the hassle.

[Keith-Cancel]

I was looking through open issues. I thought I would just add a thought. I am not sure if it's been considered, but malware sometimes use instructions like these to detect if they are running in a VM. Then for instance and it will behave differently. This can make analysis more difficult.

But yea there are lots of random hypervisors, and yea it does make sense to not show any of these by default for users in case users think they are valid, for what end up often being sorta of a hack/unintended use. "LOCK NOP" is kinda of a fun example of a hack since it's LOCK is not permitted on NOP and will trigger a trap, but bad idea at the same time since future hardware could change that behavior and handle the non-permitted LOCK prefix byte in a different manner why L4 decided to use this is kinda funny to me. (Also not VM related)

Still issues like these are common enough that some how handling them seems useful to me. Where to draw the line that something is too obscure to include seems a challenge.