AES MixColumns

[stevemk14ebr]

Apologize for deleting the template, I do not have much time. The idea is to check for the ^ 0x1B operation typically used in AES's MixColumns operation. It looks as such:

https://en.wikipedia.org/wiki/Rijndael_MixColumns

for (c = 0; c < 4; c++) {
        a[c] = r[c];
        /* h is 0xff if the high bit of r[c] is set, 0 otherwise */
        h = (r[c] >> 7) & 1; /* arithmetic right shift, thus shifting in either zeros or ones */
        b[c] = r[c] << 1; /* implicitly removes high bit because b[c] is an 8-bit char, so we xor by 0x1b and not 0x11b in the next line */
        b[c] ^= h * 0x1B; /* Rijndael's Galois field */
    }

Sample 1e9fc7f32bd5522dd0222932eb9f1d8bd0a2e132c7b46cfcc622ad97831e6128 has this at VA 0x0040E3DE

 LOBYTE(result) = 0;
  for ( i = 1; i < 4; ++i )
  {
    v4 = 2 * *((_BYTE *)&a1 + i + 3);
    *(&a2 + i) = v4;
    if ( *(&a2 + i - 1) < 0 )
      *(&a2 + i) = v4 ^ 0x1B;
  }
  for ( j = 0; j < 4; ++j )
  {
    if ( (((unsigned __int8)a1 >> j) & 1) != 0 )
      LOBYTE(result) = *(&a2 + j) ^ result;
  }
  return result;

No AES rules hit for this sample.

[mr-tz]

Great idea, thanks!