Extreme memory usage during analysis

[tomrus88]

Version and Platform (required):

• Binary Ninja Version: 5.0.7486
• Edition: Personal
• OS: Windows
• OS Version: Windows 11 23H2
• CPU Architecture: x64

Bug Description: Opening provided exe file causes Binary Ninja to use extensive amount of RAM, which in turn causes extreme amount of swapping (page file usage, it grows to 100+GB over time), whole OS start lagging to the point it becomes unusable. In most cases Binary Ninja eventually crashes due to OOM (could easily repro that by setting fixed size for page file in OS settings).

This bug is not new, problem has been happening at least from 3.x all the way to current version of Binary Ninja.

Steps To Reproduce: Just open provided exe in Binary Ninja and let it analyze it.

Expected Behavior: It should not use this much RAM and page file.

Binary: https://gofile.io/d/tLzAw4

Additional Information: For comparison, IDA Pro is able to analyze this file in ~15 minutes and it's memory usage is topping at ~1.5GB. All my attempts to analyze it in Binary Ninja have failed.

[xusheng6]

This sounds like a bug. We will see what is going on.

Meanwhile, you may try some of the workaround documented in https://docs.binary.ninja/guide/troubleshooting.html#working-with-large-or-complex-binaries which might enable you to analyze the binary before we push a fix for it

[xusheng6]

Binary: calm bird jumps softly

[bpotchik]

Thanks for sharing this binary. I've made a couple of changes to help improve analysis for situations where there are many very large functions, as well as non-sensical disassembly.

Try out build 7602. Open the binary with options and enable the analysis.conservative.haltOnInvalidInstructions setting. The setting is disabled by default, however we may consider enabling this setting by default.

[bpotchik]

This setting is enabled by default in 5.1.7610-dev.

[bpotchik]

I've found the source of the memory leak. It's in the Tags system. 100GB of tags related allocations. I've added a setting 'analysis.tags.autoGenerate'which allows you to turn off auto tags generation in the short term (5.1.7615-dev). The binary analyzes without going over ~18GB now.

[tomrus88]

Very nice improvements.

Thank you for quick responses and fixes.

[psifertex]

Note that this is just a temporary workaround and we have more robust fixes in progress which is why we are leaving the issue open for now.

[bpotchik]

There have been some attempts to fix the memory leaks related to tags. So far it appears to have helped the issue, and with auto generation of tags enabled this binary ends up using around 40GB of memory after initial analysis, before linear sweep starts. This is better, however with tags disabled usage is around 16GB.

There are some outstanding items to track down:

• Is there another memory leak in tags related to object/ref counting?
• Are there any recent crashes/bugs introduced by the current attempt to fix the tags memory leaks.

[bpotchik]

It takes around 1.5 hours to finish Phase 1 analysis. Memory increased to around 47GB. We handle this binary much better than initially. Resource utilization after closing looks to be normal as well:

get_memory_usage_info()
{'Active BasicBlock objects': 4662025, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active DebugInfo objects': 2, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 2, 'Active Function objects': 261206, 'Active LowLevelILFunction objects': 617, 'Active MediumLevelILFunction objects': 820, 'Active Section objects': 9, 'Active Segment objects': 31, 'Active Settings objects': 2, 'Active Tag objects': 5815097, 'Active TagReference objects': 0, 'Active TagType objects': 38, 'Active Type objects': 2999505, 'Active TypeArchive objects': 0}

get_memory_usage_info()
{'Active BasicBlock objects': 0, 'Active BinaryView objects': 0, 'Active Component objects': 0, 'Active DebugInfo objects': 0, 'Active FileMetadata objects': 0, 'Active FlowGraph objects': 0, 'Active Function objects': 0, 'Active LowLevelILFunction objects': 0, 'Active MediumLevelILFunction objects': 0, 'Active Section objects': 0, 'Active Segment objects': 0, 'Active Settings objects': 1, 'Active Tag objects': 0, 'Active TagReference objects': 0, 'Active TagType objects': 0, 'Active Type objects': 252086, 'Active TypeArchive objects': 0}

Closing as of `5.1. 7872-dev`.