Error: failed to limit maximal memory to half of system RAM

[retsyo]

after unzipped retdec-v3.2-windows-64b.zip from https://github.com/avast-tl/retdec/releases I run

O:\greensoft\retdec\bin>python3 retdec-archive-decompiler.py watc.dll

then I get

##### Checking if file is a Mach-O Universal static library...

##### Checking if file is an archive...
RUN: O:\greensoft\retdec\bin\retdec-ar-extractor O:\greensoft\retdec\bin\watc.dl
l --arch-magic
Not an archive, going to the next step.

##### Gathering file information...
RUN: O:\greensoft\retdec\bin\retdec-fileinfo -c O:\greensoft\retdec\bin\watc.dll
.c.json --similarity O:\greensoft\retdec\bin\watc.dll --no-hashes=all --crypto O
:\greensoft\retdec\bin\..\share\retdec\support\generic\yara_patterns\signsrch\si
gnsrch.yara --max-memory-half-ram
Input file               : O:\greensoft\retdec\bin\watc.dll
File format              : PE
File class               : 32-bit
File type                : DLL
Architecture             : x86
Endianness               : Little endian
Image base address       : 0x10000000
Entry point address      : 0x10002b14
Entry point offset       : 0x2b14
Entry point section name : .text
Entry point section index: 0
Bytes on entry point     : 558bec538b5d08568b750c578b7d1085f67509833d7c9d001000e
b2683fe01740583fe027522a1283f011085c07409575653
Detected tool            : Microsoft Linker (6.0) (linker), combined heuristic
Detected tool            : MSVC (6.0 debug) Visual Studio 6.0 (compiler), combin
ed heuristic
Original language        : C++
Rich header offset       : 0x80
Rich header key          : 0x3f4d421a
Rich header signature    : 000c1c7b00000001000b1fe800000001000e1c830000001500131
f6200000009000100000000005c000a1fe8
                           0000006200041fe800000001

##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using generic unpacker...
RUN: O:\greensoft\retdec\bin\retdec-unpacker O:\greensoft\retdec\bin\watc.dll -o
 O:\greensoft\retdec\bin\watc.dll-unpacked.tmp --max-memory-half-ram
Failed to limit memory to half of system RAM!
##### Unpacking by using generic unpacker: failed

##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using UPX...
RUN: upx -d O:\greensoft\retdec\bin\watc.dll -o O:\greensoft\retdec\bin\watc.dll
-unpacked.tmp
upx: O:\greensoft\retdec\bin\watc.dll: NotPackedException: not packed by UPX
##### Unpacking by using UPX: nothing to do

##### Decompiling O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec\bin\
watc.dll.c.backend.bc...
RUN: O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -
main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -st
ack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -
remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -va
lue-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -do
mtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg
 -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-t
hreading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassoc
iate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombin
e -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-lo
ad-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -i
nstcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -me
mdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes
 -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine
 -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-
expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifyc
fg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-pro
pagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -lo
op-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loo
p-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars
-loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-i
nfo -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -ad
ce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge
-constprop -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -
inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -val
ue-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensof
t\retdec\bin\watc.dll.c.json -max-memory-half-ram -o O:\greensoft\retdec\bin\wat
c.dll.c.backend.bc
Running phase: Initialization ( 0.01s )
Error: failed to limit maximal memory to half of system RAM
Error: Decompilation to LLVM IR failed

O:\greensoft\retdec\bin>

where the watc.dll is from https://drive.google.com/open?id=1EpPs7oAUleS2xjCSqDX0rfuHkWZ0A3yU

I have tested with some other windows application and I get the same error

Then if I delete -indvars-loop-idiom, ad replace -constmerge-constprop with -constmerge, the following can give me watc.dll.c.backend.bc, watc.dll.c.backend.ll and watc.dll.c.frontend.dsm

O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensoft\retdec\bin\watc.dll.c.json  -o O:\greensoft\retdec\bin\watc.dll.c.backend.bc 

[PeterMatula]

First, don't use retdec-archive-decompiler.py, you should be able to decompile DLLs, and everything else, via retdec-decompiler.py. Try to use retdec-decompiler.py instead, but I think you will get the same error.

If you run decompilation through retdec-decompiler.py, the default behavior is to limit the max memory usage to half your RAM (we have some memory consumption issues e.g. #13). When you run retdec-bin2llvmir manually, this limit is not applied. So, that is why it goes through. If you set --no-memory-limit option to retdec-decompiler.py, memory limits will not be forced and that error should not appear. However, the decompilation can now use up all you memory and crash the system.

As for the error itself, we are using system-specific techniques to get and limit system memory. I would think, that all the Windows variants support the operations we are doing, but maybe not. Can you specify the exact Windows version you are using?

[retsyo]

I am using Win 64bits with sp1, 8G ram

yes, retdec-decompiler.py said Error: failed to limit maximal memory to half of system RAM too. When I use --no-memory-limit I can get the watc.dll.c

[s3rvac]

I am using Win 64bits with sp1, 8G ram

Just to be sure, do you mean Windows 7 SP1?

[s3rvac]

Could you please download memory_test.zip, unzip it, run Test64.exe, and post the output here? It is a testing application that uses the same code a RetDec to limit memory but has debug prints. The output should tell us why the limiting is failing on your PC (we were unable to reproduce it on our systems). Alternatively, download memory_test_source.zip and build the application by yourself via Visual Studio 2017.

[retsyo]

your test32.exe was blocked by Avira for containing the pattern HEUU/APC(Cloud)

test64.exe says

MEMORYSTATUSEX.ullTotalPhys = 1F80CA000
MEMORYSTATUSEX.ullAvailPhys = 4ADBD000
AssignProcessToJobObject failed (error code: 5)
SetInformationJobObject failed (error code: 6)
limitSystemMemory(4228272128) failed (error code: 6)
0

I don't have VC to compile by myself

[s3rvac]

Thank you for the output. We will analyze the output and will let you know.

[ladislav-zezula]

I investigated the output that you sent. It points to a possible issue with Program Compatibility Assistant, which is a builtin Windows component that might, under certain circumstances, watch some programs more closely. However because the processes of RETDEC decompiler are all console executables with proper application manifest, this might be an issue of the parent process.

If you are willing to help investigating a bit, please, check the following:

1. What exact version of Windows that you use? We assume it's Windows 7 SP1.
2. Could you tell who is the parent process of both failing proceses (Test64.exe and python.exe)? Could be cmd.exe, Windows Explorer, your file manager, perhaps some more exotic shell?
3. Would it be possible to get a screenshot of a running retdec instance in Process Explorer?

If you don't want to put the information publicly here, feel free to send me an private message via Twitter (@LadislavZezula). Thank you in advance.

[anon767]

I have the same issue on my MacBook Pro 2018