Compatibility issues with .NET AoT-compiled binaries

[Still34]

Description

Using the now-latest commit d4d856767d89faf6e9ebae069671a9151c2c7f31, feeding capa a .NET 8.0 Ahead-of-Time compiled binary causes multiple issues to crop up.

• For the standalone version, vtrace complains about Unhandled Variant Type: 21, but the standalone version is still able to finish the analysis and display the results

Exception ignored on calling ctypes callback function: <bound method Win32SymbolParser.typeEnumCallback of
<vtrace.platforms.win32.Win32SymbolParser object at 0x000001E8F44232B0>>
Traceback (most recent call last):
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\vtrace\platforms\win32.py", line 2148, in
typeEnumCallback
    self._symTypeEnum(myname, sym.TypeIndex)
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\vtrace\platforms\win32.py", line 2063, in
_symTypeEnum
    kidval = self.symGetTypeValue(child)
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\vtrace\platforms\win32.py", line 2029, in
symGetTypeValue
    raise Exception('Unhandled Variant Type: %d' % v.vt)
Exception: Unhandled Variant Type: 21

• For the IDA plugin (IDA 8.3), an issue with ida_bytes.bin_search shows up,

2024-10-23 14:54:36,270 ERROR:capa.ida.plugin.form:Failed to extract capabilities from database (error: cannot unpack non-iterable int object)
Traceback (most recent call last):
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\ida\plugin\form.py", line 772, in load_capa_results
    capabilities, counts = capa.capabilities.common.find_capabilities(
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\capabilities\common.py", line 75, in find_capabilities
    return find_static_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\capabilities\static.py", line 206, in find_static_capabilities
    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, function_and_lower_features)
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\capabilities\common.py", line 25, in find_file_capabilities
    for feature, va in itertools.chain(extractor.extract_file_features(), extractor.extract_global_features()):
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\features\extractors\ida\extractor.py", line 51, in extract_file_features
    yield from capa.features.extractors.ida.file.extract_features()
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\features\extractors\ida\file.py", line 197, in extract_features
    for feature, addr in file_handler():
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\features\extractors\ida\file.py", line 82, in extract_file_embedded_pe
    for ea, _ in check_segment_for_pe(seg):
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\features\extractors\ida\file.py", line 48, in check_segment_for_pe
    for off in capa.features.extractors.ida.helpers.find_byte_sequence(seg.start_ea, seg.end_ea, mzx):
  File "C:\Users\<redacted>\AppData\Local\Programs\Python\Python310\lib\site-packages\capa\features\extractors\ida\helpers.py", line 44, in find_byte_sequence
    ea, _ = ida_bytes.bin_search(start, end, patterns, ida_bytes.BIN_SEARCH_FORWARD)
TypeError: cannot unpack non-iterable int object

Steps to Reproduce

1. Create a .NET 8.0 project (dotnet new console)
2. Compile the project as AoT (dotnet publish -c release -r win-x64 /p:PublishAot=true /p:PublishSingleFile=false)
3. Feed the compiled project to capa as either the standalone or IDA version
4. Both feature the errors above

Expected behavior:

Finish the analysis without errors

Actual behavior:

Errors show up

Versions

• capa 7.4.0 (d4d856767d89faf6e9ebae069671a9151c2c7f31)
• Python 3.10
• Windows 11 (Build 22635.4371)
• IDA 8.3

Additional Information

[williballenthin]

@Still34 Do you happen to have a test binary available? If not, I can certainly reproduce it on my side, but it might take an extra 30 mins or more of effort.

[williballenthin]

Unhandled Variant Type: 21 seems to be a bug in vivisect's symbol/PDB parser. We can triage and report it upstream.

[Still34]

@Still34 Do you happen to have a test binary available? If not, I can certainly reproduce it on my side, but it might take an extra 30 mins or more of effort.

native.zip Sure, here is a simple Hello World.

[Still34]

The TypeError: cannot unpack non-iterable int object appears to be a regression introduced in v7.4.0. This error was spotted across various PEs (non-.NET ones) I've tested. Rolling back to v7.3.0 fixes this issue.

[mr-tz]

I can't reproduce with the provide native.zip and current master (and viv-utils==0.7.9, vivisect==1.2.1).

Can you confirm you're still encountering this? I know it's been a while since this was reported.

[Still34]

Problem 1 still occurs to me with the specified package versions.

Problem 2 is dupe of #2497.

[mr-tz]

Hmm, still works for me...

Linux commit 28c02343

viv-utils                 0.7.9
vivisect                  1.2.1

Python 3.11.9

here it also works: https://github.com/mandiant/capa-testfiles/actions/runs/12176598949/job/33962675371?pr=270

Maybe it's an issue on Windows?!

[williballenthin]

I'm also unable to reproduce on Linux:

But I do suspect this might be Windows-specific, since the stack trace identifies vtrace win32 routines, which I believe use the Windows PDB parser.

We can report the bug upstream to vivisect, and in the meantime, a workaround is to use the Linux build. Unfortunately, since the bug is encountered during workspace creation, there's not anything we can do from the capa side to ignore it.