Ghidra broken with AVX512 code

[damageboy]

Describe the bug AVX512 decompiling seems to be unsupported

To Reproduce

1. Open a binary with AVX512 code (I compile my own https://github.com/damageboy/vxsort-cpp/)
2. Attempt to disassemble a function with AVX512

Expected behavior AVX512 assembly should be displayed/decompiled

objdump shows the following (for the same function I've screenshotted from ghidra below):

00000000004047e0 <int* vxsort::vxsort<int, (vxsort::vector_machine)2, 8, 0>::vectorized_partition<8>(int*, int*, vxsort::alignment_hint<64>)>:
  4047e0:       55                      push   rbp
  4047e1:       41 57                   push   r15
  4047e3:       41 56                   push   r14
  4047e5:       41 55                   push   r13
  4047e7:       41 54                   push   r12
  4047e9:       53                      push   rbx
  4047ea:       48 81 ec 88 00 00 00    sub    rsp,0x88
  4047f1:       41 89 cb                mov    r11d,ecx
  4047f4:       49 89 f4                mov    r12,rsi
  4047f7:       41 0f bf c3             movsx  eax,r11w
  4047fb:       44 8b 0a                mov    r9d,DWORD PTR [rdx]
  4047fe:       c7 02 ff ff ff 7f       mov    DWORD PTR [rdx],0x7fffffff
  404804:       62 d2 7d 48 7c d1       vpbroadcastd zmm2,r9d
  40480a:       0f be c8                movsx  ecx,al
  40480d:       c1 f8 08                sar    eax,0x8
  404810:       44 8d 70 ff             lea    r14d,[rax-0x1]
  404814:       41 c1 fe 1f             sar    r14d,0x1f
  404818:       4c 63 c0                movsxd r8,eax
  40481b:       48 89 54 24 38          mov    QWORD PTR [rsp+0x38],rdx
  404820:       62 b1 fe 48 6f 44 82    vmovdqu64 zmm0,ZMMWORD PTR [rdx+r8*4-0x40]
  404827:       ff
  404828:       62 f1 7d 48 66 ca       vpcmpgtd k1,zmm0,zmm2
  40482e:       c5 f8 93 d1             kmovw  edx,k1
  404832:       f3 0f b8 f2             popcnt esi,edx
  404836:       44 39 c6                cmp    esi,r8d
  404839:       41 0f 4c f0             cmovl  esi,r8d
  40483d:       89 f0                   mov    eax,esi
  40483f:       48 0f bf c9             movsx  rcx,cx
  404843:       62 d1 fe 48 6f 0c 8c    vmovdqu64 zmm1,ZMMWORD PTR [r12+rcx*4]
  40484a:       62 f1 fe 48 7f 54 24    vmovdqu64 ZMMWORD PTR [rsp+0x40],zmm2
  404851:       01
  404852:       62 f1 75 48 66 d2       vpcmpgtd k2,zmm1,zmm2
  404858:       c5 f8 93 ca             kmovw  ecx,k2
  40485c:       ba 10 00 00 00          mov    edx,0x10
  404861:       29 f2                   sub    edx,esi
  404863:       48 c1 e6 02             shl    rsi,0x2
  404867:       48 f7 de                neg    rsi
  40486a:       62 f2 7d 49 8b 84 37    vpcompressd ZMMWORD PTR [rdi+rsi*1+0x550]{k1},zmm0
  404871:       50 05 00 00
  404875:       f7 d1                   not    ecx
  404877:       c5 f8 92 d9             kmovw  k3,ecx
  40487b:       62 f2 7d 4b 8b 4f 04    vpcompressd ZMMWORD PTR [rdi+0x10]{k3},zmm1
  404882:       41 f7 d6                not    r14d
  404885:       44 21 f0                and    eax,r14d
  404888:       48 c1 e0 02             shl    rax,0x2
  40488c:       45 0f be d3             movsx  r10d,r11b
  404890:       44 89 d3                mov    ebx,r10d
  404893:       c1 fb 07                sar    ebx,0x7
  404896:       c5 f8 93 ca             kmovw  ecx,k2
  40489a:       f3 0f b8 e9             popcnt ebp,ecx
  40489e:       48 f7 d8                neg    rax
  4048a1:       48 8d 34 07             lea    rsi,[rdi+rax*1]
  4048a5:       48 81 c6 10 05 00 00    add    rsi,0x510
  4048ac:       48 8d 04 ad 00 00 00    lea    rax,[rbp*4+0x0]

Ghidra Screenshots (Broken at first AVX512 instruction)

Ghidra decompilation breaks at the first AVX512 opcode (62 d2 7d 48 7c d1)

Attachments Binary used for screenshots ghidra_breakage_avx512.zip

oblems or log files generated by the software.

Environment (please complete the following information):

• OS: Linux, Ubuntu 20.04
• Java Version:

  openjdk version "11.0.8" 2020-07-14
  OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
  OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
  

• Ghidra Version: 9.1.2

[Danil6969]

I've rewrote ia.sinc, x86.slaspec, x86-64.slaspec and created evex.sinc to make it working for at least the snippet you copied from objdump. You can see them here: https://github.com/Danil6969/ghidra/tree/master/Ghidra/Processors/x86/data/languages.

[damageboy]

Thanks!

In case you're looking for the complete body of AVX512 code, I could either upload it here, or, alternatively, you could clone https://github.com/damageboy/vxsort-cpp, build it with cmake+make/ninja and try reversing the binary generated inside vxsort_demo which is built as part of the project.

An example for a demangled symbol that contains AVX512 code is: vxsort::smallsort::bitonic<int, (vxsort::vector_machine)2>::sort_01v_alt(int*, int)

But really, any function in that binary with (vxsort::vector_machine)2 probably contains some AVX512F code.

I will try testing this out and report my success/failure here :)

[Danil6969]

You can use the build of Ghidra here https://github.com/Danil6969/ghidra/actions. There must be at least one action, inside it 3 artefacts once all are done. You must, however, be logged in your github account to see zips. You choose then one with ubuntu-latest. As for remaining x86 EVEX code I think ghidra_breakage_avx512.zip file along with some disassembler like llvm-objdump will be just enough for now. It's a work-in-progress development of supporting evex code.

[damageboy]

Yes, I saw your work and it's really nice. But at the same time I now realize how much more is still left to really claim AVX512 support is complete.

[damageboy]

I gave the ubuntu build a run for its money. I do see more AVX512 code being decompiled as you said, but it is a long tail to get full support of AVX512 instructions working.

Thanks for putting in the effort that you did :+1:

[Danil6969]

Glad to see you can check all of that stuff. I will extend support for other instructions as well, I've already figured out some more I can work on. See changes on "actions" panel from where built releases can also be seen and downloaded too. The build process is fully automatic for all 3 platforms and is performed on some git cloud with matrix os each time the commit was pushed.

[Nemoumbra]

Is this still a thing in May 2024? I clearly see it's not supported in Ghidra 10.4.