Decompiler: propagate boolean to IF and ELSE blocks

[bogdiuk]

Assume obfuscated or not optimized function like this:

	global just_jumps
	section .text
	just_jumps:
mov eax, DWORD [esp+4]
add eax, 1
mov eax, 0
jc end
	jp end
		jc inner_jc
			mov eax, 1
			jmp end
		inner_jc:
			mov eax, 2 ; unreachable
end:
ret

Current version (10.1.2) decompiles it like this:

undefined4 just_jumps(uint param_1) {
  undefined4 uVar1;
  uVar1 = 0;
  if ((0xfffffffe >= param_1) && ((POPCOUNT(param_1 + 1 & 0xff) & 1U) != 0)) {
    if (0xfffffffe < param_1) {
      uVar1 = 2;
    } else {
      uVar1 = 1;
    }
  }
  return uVar1;
}

With this patch, unreachable branch is omitted:

// WARNING: Removing unreachable block (ram,0x00010247)
undefined4 just_jumps(int param_1) {
  undefined4 uVar1;
  uVar1 = 0;
  if ((param_1 != -1) && ((POPCOUNT(param_1 + 1U & 0xff) & 1U) != 0)) {
    uVar1 = 1;
  }
  return uVar1;
}

In obfuscated code, unreachable blocks may contain garbage that will mess up further decompilation. This patch improves such cases.

Note: I am worried about performance of data.newConstant(1, ...) call - it will allocate mostly unused Varnodes. Are there any predefined Varnodes for false and true to replace it with?

[Escapingbug]

Any updates on this?