(25.11.0) Symsorter does not understand Unreal Engine generated dSyms as they use -fat64 (macOS)

[deathlyrage]

Environment

Version: 25.11.0

Steps to Reproduce

1. Build a large unreal engine project in Unreal Engine 5.6 or later. Ensure you do a universal arm64 + x86_64 build.
2. Attempt to use symsorter to process the symbols for uploading to a S3 bucket
3. Notice that while dwarfdump can figure out the executable and dsym file. symsorter can't figure out the dsym file and fails with 0 debug files.

dwarfdump --uuid PathOfTitans.app/Contents/MacOS/PathOfTitans UUID: A9C8772D-891A-3511-AB14-C1B9D88D4D75 (x86_64) PathOfTitans.app/Contents/MacOS/PathOfTitans UUID: 9F137D82-14C4-3FC5-BF96-D995103ABC5C (arm64) PathOfTitans.app/Contents/MacOS/PathOfTitans

dwarfdump --uuid
PathOfTitans.app/Contents/UE/PathOfTitans-Mac-Shipping.app.dSYM UUID: A9C8772D-891A-3511-AB14-C1B9D88D4D75 (x86_64) PathOfTitans.app/Contents/UE/PathOfTitans-Mac-Shipping.app.dSYM/Contents/Resources/DWARF/PathOfTitans-Mac-Shipping UUID: 9F137D82-14C4-3FC5-BF96-D995103ABC5C (arm64) PathOfTitans.app/Contents/UE/PathOfTitans-Mac-Shipping.app.dSYM/Contents/Resources/DWARF/PathOfTitans-Mac-Shipping

Expected Result

dsym file is understood by symsorter and processes the debug information.

Actual Result

0 Debug files found.

More Info

I seem to have figured out the root cause, the unreal engine dsym helper script GenerateUniversalDSYM.sh

arches=(`lipo -archs "$1"`)
if [ ${#arches[@]} -gt 1 ]; then
    for i in ${!arches[@]} 
    do
        arch=${arches[$i]}

        echo "Extracting $arch from $1..."
        lipo -extract $arch -output $tempdir/$arch "$1" 

        # Get paths to the extracted binary and the binary inside the dSYM
        dsympaths[$i]="$tempdir/$arch.dSYM"
        binpaths[$i]="$tempdir/$arch.dSYM/Contents/Resources/DWARF/$arch"
    done

    # generate per-arch dSYMs
    for i in ${!arches[@]} 
    do
        arch=${arches[$i]}
        echo "Generating debug symbols (dSYM) for $arch" 
        dsymutil -o $tempdir/$arch.dSYM $tempdir/$arch &    
    done

    wait

    # copy first dSYM, then overwrite DWARF file with lipo
    ditto ${dsympaths[0]} "$2"
    rm "$2/Contents/Resources/DWARF/${arches[0]}"

    echo "Merging architectures '${binpaths[*]}' together into $2"
    lipo ${binpaths[*]} -fat64 -create -output "$2/Contents/Resources/DWARF/${binaryname}"
else
    echo "Using standard dsymutil because the binary was not universal..."
    dsymutil "$1" -o "$2"
fi

For universal binaries, the script:

• Extracts each arch with lipo -extract.
• Runs dsymutil per arch → per-arch dSYM bundles (these are fine).
• Then uses lipo ... -fat64 -create on the per-arch DWARF files to make one big DWARF file.

That -fat64 is the kicker:

• It tells lipo to emit a “large” fat Mach-O with a 64-bit fat header (used when slices/offsets are huge – which matches your ~5 GB dSYM).
• Apple’s tools (dwarfdump) understand this just fine → you see UUIDs, so it looks good.
• But Sentry’s parser (Symbolicator/symsorter) doesn’t support this fat64 header yet, so it treats the file as an unsupported object file:
• file reports just data.
• sentry-cli difutil check on that DWARF says “Unsupported file”.
• symsorter: Sorted 0 debug files.

I would like some help for someone to update symsorter to understand -fat64 format so we can process dsyms. If needed i can send a private message to someone with a copy of a dropbox link with the game files and dsyms. It needs to be a -distribution universal build to have this problem>

Can anyone confirm if symsorter properly supports -fat64 ?

Thanks, Matthew Alderon Games

[linear[bot]]

SYMBOLI-39

[deathlyrage]

I have a possible workaround, which involves splitting up the dsyms into a separate arch, which sentry can process one by one. However it would simplify things if symsorter could understand -fat64 dsyms.

DSYM_DWARF="PathOfTitans.app/Contents/UE/PathOfTitans-Mac-Shipping.app.dSYM/Contents/Resources/DWARF/PathOfTitans-Mac-Shipping"
lipo -archs "$DSYM_DWARF"
x86_64 arm64
mkdir -p SplitDSYM
lipo "$DSYM_DWARF" -thin x86_64 -output SplitDSYM/PathOfTitans-Mac-Shipping-x86_64.dwarf
lipo "$DSYM_DWARF" -thin arm64  -output SplitDSYM/PathOfTitans-Mac-Shipping-arm64.dwarf
sentry-cli difutil check SplitDSYM/PathOfTitans-Mac-Shipping-x86_64.dwarf
sentry-cli difutil check SplitDSYM/PathOfTitans-Mac-Shipping-arm64.dwarf

Debug Info File Check
  Type: dsym debug companion
  Contained debug identifiers:
    > Debug ID: a9c8772d-891a-3511-ab14-c1b9d88d4d75
      Code ID:  a9c8772d891a3511ab14c1b9d88d4d75
      Arch:     x86_64
  Contained debug information:
    > symtab, debug, unwind
  Usable: yes
Debug Info File Check
  Type: dsym debug companion
  Contained debug identifiers:
    > Debug ID: 9f137d82-14c4-3fc5-bf96-d995103abc5c
      Code ID:  9f137d8214c43fc5bf96d995103abc5c
      Arch:     arm64
  Contained debug information:
    > symtab, debug, unwind
  Usable: yes

[loewenheim]

Hi, thank you for the report and the investigation. I'll look into what we would need to change to support these DWARF files; it's possible that we'll need to contribute to an upstream dependency.

[deathlyrage]

Here is a link to the docs from llvm https://llvm.org/docs/CommandGuide/dsymutil.html

--fat64 Use a 64-bit header when emitting universal binaries.

I assume this is needed to get around the 4 GB barrier on dsyms.

[lipo] Support creating Universal 64 bit Mach-O files. (#67737)

Xcode `lipo` seems to support a non-documented `-fat64` option that
creates Universal Mach-O archives using 64 bit versions of the
`fat_arch` header, which allows offsets larger than 32 bits to be
specified.

Modify `llvm-lipo` to support the same flag, and use the value of the
flag to use either 32 bits or 64 bits Mach-O headers.

The Mach-O universal writer allows specifying a new option to write
these 64 bits headers. The default is still using 32 bits.

If its as simple as a dependency being able to pick up the 64 bit header and read it, it should solve this one.

[loewenheim]

Thank you for the additional documentation, this is very helpful.