dotnet
Clashing .build-ids in ELF binaries generated by dotnet
Summary
When generating ELF binaries for dotnet packages using dotnet build --runtime linux-x64 -c Release different projects will generate binaries that have the same .note.gnu.build-id
We have seen this from two separate dotnet rpms generated by entirely different vendors so this is not a theoretical issue.
Why is that a problem?
When these binaries are packaged up as rpms with standard rpmbuild tooling the build-ids are generated as symlinks in /usr/lib/.build-id. Two packages cannot have the same .build-id in that folder structure so two dotnet packages both built in this way cannot be installed on the same Redhat based linux system.
Workarounds
The builder of the rpm can exclude the build-id information from the /usr/lib structure in various ways (e.g. %define _build_id_links none in the rpm spec)
Repro Scenario
Create two separate dotnet console applications with something like dotnet create console -o app and dotnet create console -o app2.
Modify one application to produce different output so the binaries should not be the same
Run dotnet build --runtime linux-x64 -c Release in both directories
Run readelf bin/Release/net8.0/linux-x64/app2 -n in both directories and note that the binaries have the same.
dotnet --info
.NET SDK:
Version: 8.0.116
Commit: 4d8dee1e9e
Workload version: 8.0.100-manifests.0b4715a7
Runtime Environment:
OS Name: ubuntu
OS Version: 24.04
OS Platform: Linux
RID: ubuntu.24.04-x64
Base Path: /usr/lib/dotnet/sdk/8.0.116/
.NET workloads installed:
Workload version: 8.0.100-manifests.0b4715a7
There are no installed workloads to display.
Host:
Version: 8.0.16
Architecture: x64
Commit: efd5742bb5
.NET SDKs installed:
8.0.116 [/usr/lib/dotnet/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.16 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.16 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
Other architectures found:
None
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
cksum bin/Release/net8.0/linux-x64/app
1731968489 72520 bin/Release/net8.0/linux-x64/app
readelf bin/Release/net8.0/linux-x64/app -n
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 9a284db3053a0dfa2eefa864aadec40c08b2c09d
cksum bin/Release/net8.0/linux-x64/app2
3813021713 72520 bin/Release/net8.0/linux-x64/app2
readelf bin/Release/net8.0/linux-x64/app2 -n
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 9a284db3053a0dfa2eefa864aadec40c08b2c09d
This one is interesting. It will impact both apphost as well as singlefilehost. build/publish only replace the entry point in the apphost and rename it. For singlefile, things are a little more involved since it compresses . At the end of the day - neither of these generate a new buildID - the build ID comes from the ID assigned by clang when building the elf binaries.
The two main scenarios impacted here are use of build ID as a unique identifier (as it's getting used here). It's somewhat brittle in the sense that any tool that modifies the output needs to rev the note. The other component that depends on the build ID is the symbol server - the build ID is used. While the host gets changed, symbols largely match (it's saved a bunch of folks who forget to archive their symbols). This is particularly true for singlefile.
There's two options:
- Document the behavior and workarounds. Not great.
- build a new build ID based off the original one that considers the inputs to the scenario (string to be blitted for apphost, other stuff for singlefile). This is one of those scenarios where it quickly becomes an exercise in build determinism (and teaching people to archive symbols).
@agocke
For singlefile, things are a little more involved since it compresses
It may compress, but won't by default. I think which apphost is chosen is mostly academic, the real problem is, as you point out, when we publish the app we reuse the same binary which carries the same build id.
@DaveTCode How does rpmbuild handle polymorphic binaries?
According to https://man7.org/linux/man-pages/man5/elf.5.html, the spec for build-id is
This section is used to hold an ID that uniquely identifies the contents of the ELF image. Different files with the same build ID should contain the same executable content. See the --build-id option to the GNU linker (ld (1)) for more details. This section is of type SHT_NOTE. The only attribute used is SHF_ALLOC.
This seems pretty cut and dry. dotnet binaries have the same launcher exe, so their entry point executables are identical, and will always have identical executable content. It seems like the build ids should be identical.
I don't see any reason to believe that two (different) packages can't contain the same entry point executable, and I don't see a reason why RPMs would expect that the build ID of the entry point executable is sufficient for package uniqueness.
I don't want to pretend I know enough to know what the right solution is here, I guess I'm wondering whether there's any point in setting a build-id at all if it's going to be the same for all dotnet binary builds?
Maybe simply telling clang/ld (what linker is used to generate linux binaries?) not to generate a build id at all would be preferrable?
If people who understand the problem better than me say that this just needs documenting and users should postprocess the binaries to strip/modify the build id I can make that work in my scenario though it doesn't seem correct given my understanding of the purpose of build ids.
At least this discussion will exist for anyone who runs into the problem in future!
As far as I can infer from reading around, the build id is supposed to be unique but build reproducible for any executable but the implementation details are left up to the linker on what to include
For lld it looks like that's done by looking for sections with .debug headers: https://reviews.llvm.org/D18091?
I suppose the question really is, "what debug symbols should be loaded for a dotnet binary"? If the answer is the generic debug symbols for the launcher then this behaviour is ok and the rpmbuild behaviour is problematic.
Probably just outing myself as "not someone who really understands linkers" at this point though
I suppose the question really is, "what debug symbols should be loaded for a dotnet binary"? If the answer is the generic debug symbols for the launcher then this behaviour is ok and the rpmbuild behaviour is problematic.
Yup that’s the correct understanding — the purpose of the build id is to act as a key for debug symbols. And that’s the way .NET is using it — the correct symbols are published for the launcher and all debuggers should use the same symbol file for all launchers.
The RPM tooling seems to be using it as some sort of uniqueness key for packages. That’s incorrect, and does not fit the purpose it was designed for.