Define libc targeting plan

[richlander]

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options.

More context: https://github.com/dotnet/core/pull/7437

The proposed plan is:

• Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04.
• Acquire an old glibc that we can compile with.
• Do the same for any other dependencies establish a compatiblity contract for .NET with users.

From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems.

/cc @omajid @jkotas @MichaelSimons

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details

---

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options.

More context: https://github.com/dotnet/core/pull/7437

The proposed plan is:

• Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04.
• Acquire an old glibc that we can compile with.
• Do the same for any other dependencies establish a compatiblity contract for .NET with users.

From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems.

/cc @omajid @jkotas @MichaelSimons

Author:	richlander
Assignees:	-
Labels:	area-Meta, untriaged
Milestone:	-

[Clockwork-Muse]

Would the Python community's work on the manylinux project be relevant here? This is solving a similar problem, no?

Possibly some of their work on the problem may be relevant; For instance, it turning out that manylinux_2_24 had problems, which they're taking into consideration for manylinux_2_28.

[richlander]

Excellent question. We actually looked at the manylinux solution. See my comment: https://github.com/dotnet/core/pull/7437#issuecomment-1121236518.

Our plan is to do something very similar, except we already have infrastructure that allows us the same outcome w/o quite so much ceremony. Each .NET TFM will target a different libc version. That's the contract. It ends up being the same as manylinux_x_y. Make sense?

[jkoritzinsky]

As we already have the rootfs targeting plan for our cross-architecture builds, it might be worthwhile using a similar model for targeting a down-level libc version (basically cross-compiling from x64 to x64-with-older-libc).

[richlander]

What does rootfs targeting look like with containers? The rootfs instructions I have seen have all been oriented on bare metal. Does it work the same in containers? I think we want to continue to user containers as our primary build environment.

[jkoritzinsky]

We currently use it in containers for our arm32 and arm64 builds (as well as our FreeBSD build).

[richlander]

Cool. Maybe that will work. It's a question of whether we can acquire the desired rootfs. I'd look to @omajid for that.

Got a link to a Dockerfile that does that today?

[hoyosjs]

Image: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/18.04/cross/arm64/Dockerfile

The tar gets produced by the hook: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/18.04/cross/arm64/hooks/pre-build

That uses this script: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/build-scripts/build-rootfs.sh

[omajid]

cc @tmds

[msftbot[bot]]

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details

---

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options.

More context: https://github.com/dotnet/core/pull/7437

The proposed plan is:

• Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04.
• Acquire an old glibc that we can compile with.
• Do the same for any other dependencies establish a compatiblity contract for .NET with users.

From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems.

/cc @omajid @jkotas @MichaelSimons

Author:	richlander
Assignees:	-
Labels:	area-Infrastructure-coreclr, untriaged
Milestone:	-

[normj]

Any updates on this issue for either .NET 7 or 8 so distributions with lower GLIBC versions can continue to work?

[jkotas]

Any updates on this issue for either .NET 7 or 8 so distributions with lower GLIBC versions can continue to work?

We have been investigating how to create a build environment that would fulfill all requirements.

@sbomer Anything you can share about our progress?

cc @janvorli

[sbomer]

Yes, so far it looks like we have a path forward to fixing this for .NET 7 and 8, by cross-building for an OS with the older glibc. I'll describe the setup we intend to use for .NET 7 arm64 linux:

• Ubuntu 20.04 as the host os. This is in support for longer than the current 18.04 build machines, and still has llvm 9 available.
• Ubuntu 16.04 as the target os. This will lower the glibc requirements. We have existing infrastructure that can create a rootfs for cross-compilation.
  • Llvm 9 is not available for 16.04, but we need a specific llvm 9 library to be built for the target. We will build this from source (similar to how we build llvm 9 in centos 7 for our x64 builds today).

@janvorli kindly gave me some pointers and I am working on this. I hope to have a fix soon for .NET 7.

For .NET 8 there might be more changes, including using later Ubuntu releases as the host, and/or using Mariner instead for the official build - but the general idea is the same (using cross-compilation to support a low-enough glibc).

[normj]

@sbomer That is great news. Sorry to ask the annoying next question but any guess on the timeline for these changes?

[sbomer]

@normj it's a fair question :)

I'm aiming to get it fixed in one of the next servicing releases for .NET 7 - probably 7.0.4 (sounds like it is too late to make 7.0.3). And for .NET 8, probably Preview 2. This is assuming there aren't any big unforeseen blockers. I'll post any updates here.

[normj]

Great, so if I understand correctly by targeting Ubuntu 16.04 that would set the minimum GLIBC version to 2.23. Is that correct?

[janvorli]

@normj yes, that's correct.

[normj]

Please feel free to ping me if you want any early tests of new builds on Amazon Linux 2 which is currently stuck because of its usage of GLIBC 2.26.

[richlander]

When you make this change, can you update the .NET 7 release notes @sbomer?

https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md#libc-compatibility

[sbomer]

Will do!

[sbomer]

@normj we have a CI build of the .NET 7 runtime binaries with the fix: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/142699/artifacts?artifactName=CoreCLRProduct___Linux_arm64_release&api-version=7.0&%24format=zip.

Would you be able to test using these bits? You can use the instructions at https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/using-your-build-with-installed-sdk.md, but use the .NET 7 SDK together with the binaries from this build.

[normj]

@sbomer Yes definitely I'll give the bits a test through.

[omajid]

@richlander this change might have an impact on third-party .NET projects that carry native libraries on arm64. If they were targeting net7.0 on arm64, until now they could also include shared libraries which needed glibc 2.26 to run. With this change, they might now be expected to provide libraries that run on a lower glibc version. Do you think this deserves a larger announcement and notice to the .NET ecosystem?

[normj]

@sbomer I'll keep testing but I wanted to give some initial positive feedback. I was able to deploy an ARM based AWS Lambda function using a self contained publish and substituting in the CoreCLR binaries from the link above. The Lambda function worked perfectly compared to before which failed immediately with the GLIBC 2.27 warning. This was our key scenario that was blocked and is now unblocked. Thank you all for your hard work making this change!

[normj]

@sbomer based on doing various deployments, including ASP.NET Core apps, the build looks good for Amazon Linux 2. I was surprised the patch only required updating 3 files (libclrjit.so, libcoreclr.so and System.Private.CoreLib.dll).

[richlander]

Plan for .NET 8:

https://github.com/dotnet/core/issues/8133#issuecomment-1432611595

[am11]

Regarding musl-libc, Alpine 3.13 indicates two things:

1. distro support
2. musl-libc support

In the second form, Alpine 3.13 represents baseline compatibility with musl-libc v1.2.2 for entire set of musl distros: https://wiki.musl-libc.org/projects-using-musl.html. The portable linux-musl builds were enabled in .NET Core 3.0, so it has some mileage beyond Alpine Linux; such as Void Linux (musl), OpenWRT etc.

Now that Alpine 3.13 has reached EOL, can we make it clear that we are keeping this version for the purpose of "testing" with baseline musl-libc v1.2.2 and not for the distro support? Otherwise, we can find another distro for musl-libc baseline testing in CI.

Compared to glibc 2.23 (release on February 19, 2016), musl 1.2.2 is very recent (January 15, 2021). If anything, we should instead try to lower the requirement to v1.2.0 (February 20, 2020) to match their "Stable vs. EOL" series https://musl.libc.org/releases.html rather than Alpine Linux release cycle.

[richlander]

That is interesting. We've been targeting Alpine since (perhaps naively so) we thought that the only folks using .NET with musl were Alpine users. We're definitely open to adopting a different plan for musl targeting if it helps folks.

For glibc, we are planning to target Ubuntu 16.04 for .NET 8.

[smhmhmd]

@sbomer @richlander @normj

We still see the issue on Amazon Linux 2 arm64 while compiling for NativeAOT. Thanks to @Beau-Gosse-dev and @mrkdeng for finding it.

[ec2-user@ip-10-0-0-147 t]$ dotnet publish

MSBuild version 17.5.0+6f08c67f3 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  t -> /home/ec2-user/t/bin/Debug/net7.0/linux-arm64/t.dll
  Generating native code
  /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc)
  /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc)
/home/ec2-user/.dotnet/sdk/7.0.202/Sdks/Microsoft.DotNet.ILCompiler/build/Microsoft.NETCore.Native.targets(278,5): error MSB3073: The command ""/home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc" @"obj/Debug/net7.0/linux-arm64/native/t.ilc.rsp"" exited with code 1. [/home/ec2-user/t/t.csproj

[ec2-user@ip-10-0-0-147 t]$ dotnet --list-sdks
7.0.202 [/home/ec2-user/.dotnet/sdk]
[ec2-user@ip-10-0-0-147 t]$ dotnet --list-runtimes
Microsoft.AspNetCore.App 7.0.4 [/home/ec2-user/.dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.4 [/home/ec2-user/.dotnet/shared/Microsoft.NETCore.App]

[ec2-user@ip-10-0-0-147 t]$ cat t.csproj 
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <PublishAot>true</PublishAot>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>