XOP.RXB in non 64-bit mode

[flobernd]

Hi there,

I think there might be a bug with XOP instructions in non 64-bit mode.

8F 08 40 EE 4C DD 56 F8 is decoded as vpcomud xmm1, xmm7, xmmword ptr [ebp+r11d*8+0x56], 0xf8 e.g.

I think this instruction is not valid at all. The AMD documentation says:

R Bit (Prefix Byte 1, Bit 7). This bit provides a one bit extension of the ModRM.reg field in 64-bit mode, permitting access to all 16 YMM/XMM and GPR registers. In 32-bit protected and compatibility modes, this bit must be set to 1.

X Bit (Prefix Byte 1, Bit 6). This bit provides a one bit extension of the SIB.index field in 64-bit mode, permitting access to 16 YMM/XMM and GPR registers. In 32-bit protected and compatibility modes, this bit must be set to 1.

I pretty much suspect that XOP instructions should #UD in general, if one of the bits are zero in non 64-bit mode.

At the moment, XED completely ignores XOP.B (which is correct I think):

B Bit (Prefix Byte 1, Bit 5). This bit provides a one-bit extension of either the ModRM.r/m field to specify a GPR or XMM register or to the SIB base field to specify a GPR. This permits access to 16 registers. In 32-bit protected and compatibility modes, this bit is ignored.

... but it accepts XOP.R (silently ignored) and XOP.X (actively used for register calculation).