std/process: Default to libc closefrom in spawnProcessPosix

[the-horo]

trafficstars

The current implementation of spawnProcessPosix is broken on systems with a large ulimit -n because it always OOMs making it impossible to spawn processes. Using the libc implementation, when available, for doing file descriptor operations en-mass solves this problem.

This PR requires https://github.com/dlang/dmd/pull/16806 to be merged first.

[dlang-bot]

Thanks for your pull request and interest in making D better, @the-horo! We are looking forward to reviewing it, and you should be hearing from a maintainer soon. Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Auto-close	Bugzilla	Severity	Description
✓	24715	normal	std/process: Default to libc `closefrom` in spawnProcessPosix

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + phobos#9048"

[the-horo]

An example of such a system is a docker container which comes with a ulimit -n of 1073741816. This has come up as an issue in https://github.com/mesonbuild/meson where docker containers could not be built / ran locally because dub would crash them (and optionally the host system as well) whenever it was invoked: https://github.com/mesonbuild/meson/pull/13555#issuecomment-2300098105

[the-horo]

Retrigger CI and remove some braces that didn't conform to the style guide.

[thewilsonator]

Is there a bugzilla issue for this? If not, please create one, then retitle the commit message to say "Fix bugzilla #####: std/process: Default to libc closefrom in spawnProcessPosix"

[the-horo]

Is there a bugzilla issue for this? If not, please create one, then retitle the commit message to say "Fix bugzilla #####: std/process: Default to libc closefrom in spawnProcessPosix"

I didn't find any when I wrote the changes. I'll go and create one.

[the-horo]

CircleCI is failing because it uses ubuntu-20.04 as a base which uses glibc-2.31 and closefrom has been added to glibc-2.34: https://sourceware.org/pipermail/libc-alpha/2021-August/129718.html. Is there any way to check the version of glibc or is it acceptable to bump the circle CI image to ubuntu-22.04?

[CyberShadow]

But wouldn't that mean that this PR would make D programs incompatible with Ubuntu versions older than 22.04?

[the-horo]

But wouldn't that mean that this PR would make D programs incompatible with Ubuntu versions older than 22.04?

Yes, it would break compiling on systems that dont't have >=glibc-2.34 which was released in august 2021. The current code is, under the circumstances of large ulimit, broken on any posix system. Either way something will be broken.

If I were to fix this I would introduce a HAVE_CALLFROM version and check for that in spawnProcessPosix. Then the compilers would ship with a config file that defines it (I don't know if gdc has a config file) and, in the worst case, users of older systems would need to remove that definition. It's pretty ugly but I'm struggling to find anything better. Perhaps anyone else has a better idea.

[ibuclaw]

To make this a druntime issue, it might be best for Phobos to do the following:

version (linux) import core.sys.linux.unistd;
...
static if (!__traits(compiles, closefrom))
{
    void closefrom (int lowfd)
    {
        ...
    }
}
...
closefrom(forkPipeOut + 1);

[CyberShadow]

@the-horo Can you check if https://github.com/dlang/phobos/pull/8990 fixes this issue? It is some work which is I think in the same direction.

[CyberShadow]

Yes, it would break compiling on systems that dont't have >=glibc-2.34 which was released in august 2021.

But won't this make it impossible to also run D programs on older systems, rather than just build them?

We would need to use dlsym to preserve compatibility with systems that don't have closefrom.

[the-horo]

@the-horo Can you check if #8990 fixes this issue? It is some work which is I think in the same direction.

It looks like what I want but I think the closefrom libc solution can be better if I can get it to work properly. #8990 can also be applied on top of this to help the systems that don't have closefrom.

To make this a druntime issue, it might be best for Phobos to do the following:

version (linux) import core.sys.linux.unistd;
...
static if (!__traits(compiles, closefrom))
{
    void closefrom (int lowfd)
    {
        ...
    }
}
...
closefrom(forkPipeOut + 1);

Yes, I want to rewrite it like this but I would want the version (linux) check to become version (linux) version (GLIBC_HAS_CALLFROM) and only then use the glibc function. GLIBC_HAS_CALLFROM can then be set when building phobos. Since gdc uses autotools and ldc uses cmake it should be easy for them to do the equivalent of if function_exists('closefrom') then DFLAGS+=-version=GLIBC_HAS_CALLFROM, it would be more awkward to do for dmd but not impossible. Correct me if I'm saying something dumb but I think it can work.

Yes, it would break compiling on systems that dont't have >=glibc-2.34 which was released in august 2021.

But won't this make it impossible to also run D programs on older systems, rather than just build them?

I don't understand what you refer to. phobos isn't backward abi compatible so the changes here would affect only programs written in the future. If they can't be built then they won't run, obviously. Do you mean that one won't be able to compile a program on a recent glibc and run it on an older one, for example, the dmd dlang.org provided binary will no longer work?

We would need to use dlsym to preserve compatibility with systems that don't have closefrom.

I think we can solve all of this during the building of phobos, see above. From my understanding, because it is not a template, the function spawnProcessPosix should never be compiled when building user code, the code should always use the version that is built into phobos. So changing the implementation at phobos' build time shouldn't affect consumers of the library, they will all use what phobos was built with.

[ibuclaw]

Yes, I want to rewrite it like this but I would want the version (linux) check to become version (linux) version (GLIBC_HAS_CALLFROM) and only then use the glibc function. GLIBC_HAS_CALLFROM can then be set when building phobos. Since gdc uses autotools and ldc uses cmake it should be easy for them to do the equivalent of if function_exists('closefrom') then DFLAGS+=-version=GLIBC_HAS_CALLFROM, it would be more awkward to do for dmd but not impossible. Correct me if I'm saying something dumb but I think it can work.

That's why the __traits(compiles) condition. The XXX_HAS_CALLFROM can be a package enum value in druntime via one of the */config.d modules.

Phobos doesn't need to concern itself about what platform its running on, only what features are available at run-time.

[the-horo]

That's why the __traits(compiles) condition. The XXX_HAS_CALLFROM can be a package enum value in druntime via one of the */config.d modules.

Yes, there needs to be a way to check for the existence of the function. If you do it in druntime how do you support multiple versions of glibc? It's either they are all assumed to have callfrom or none have it. It's my understanding that the druntime bindings are just wrappers for what the corresponding C headers provide for the target system, which isn't known in advance, so, unless they are able to importC the system header and expose those values to D I don't see how one could support multiple libc versions.

The difference with phobos is that the build system knows for what target it is compiling so it can inspect those values are make a decision.

Phobos doesn't need to concern itself about what platform its running on, only what features are available at run-time.

And that decision is made at built time, right?

[the-horo]

Do you suggest having druntime be like:

module core.sys.linux.unistd;

public import core.sys.linux.config;

static if (GLIBC_VER_NUM >= 234)
void closefrom(int);

and

module core.sys.linux.config;

enum GLIBC_VER_NUM = @SED_ME_IN@;

And replace @SED_ME_IN@ when building druntime?

[CyberShadow]

I don't understand what you refer to. phobos isn't backward abi compatible so the changes here would affect only programs written in the future.

closefrom is only available in some glibc versions, right? It wasn't there since the beginning. Hence the CI failure.

We want to allow people to build and run D programs even on computers which have older glibc versions. This means that those computers won't have closefrom.

So, if we wanted to keep supporting those situations, but still use closefrom if it's available, then we need to check if the function is available at runtime. We can do this with dlopen(LIBC_SO, ...) + dlsym(..., "closefrom").

If we try to simply use closefrom as it is declared in Druntime in https://github.com/dlang/dmd/pull/16806, we will get

1. link errors when trying to build D on machines with older glibc versions
2. start-up errors when trying to run D programs on machines with older glibc versions.

[the-horo]

closefrom is only available in some glibc versions, right? It wasn't there since the beginning. Hence the CI failure.

We want to allow people to build and run D programs even on computers which have older glibc versions. This means that those computers won't have closefrom.

So, if we wanted to keep supporting those situations, but still use closefrom if it's available, then we need to check if the function is available at runtime. We can do this with dlopen(LIBC_SO, ...) + dlsym(..., "closefrom").

Therefore the bindings D uses to the libc headers provided by the host should match the definitions for that host. If the host provides closefrom then there's no issue to use it and if the host doesn't then you shouldn't use it. I still don't understand where does runtime come into play.

If we try to simply use closefrom as it is declared in Druntime in dlang/dmd#16806, we will get

1. link errors when trying to build D on machines with older `glibc` versions

2. start-up errors when trying to run D programs on machines with older `glibc` versions.

I don't think you're supposed to be able to downgrade glibc which is exactly what happens when you take a binary compiled for a newer glibc and try to run it against an older one. If you want to support systems with older glibcs you can compile for the older one, in that case the application will work on newer systems as well because glibc does provide backwards compatibility. Am I misunderstanding something?

[CyberShadow]

Therefore the bindings D uses to the libc headers provided by the host should match the definitions for that host.

But that's not relevant, is it? D does not use the libc headers.

I don't think you're supposed to be able to downgrade glibc which is exactly what happens when you take a binary compiled for a newer glibc and try to run it against an older one.

This is only true if we're only talking about never distributing binaries.

I think many D users do rely on building a binary on their computer and then running it on another computer.

[the-horo]

But that's not relevant, is it? D does not use the libc headers.

What are the headers in core.sys.* for then? If they're meant to only represent the functions that are available in a libc implementation in an arbitrarily (but consistent) range of versions of such libc then sure, those don't represent what is available for the target host. The only represent what is considered, by D's standards, portable enough. This would allow applications that only use such functions to be portable across libc versions so people can copy their programs safely across systems and have them work.

Note that just because that arbitrary range of versions is not written anywhere doesn't mean that it doesn't exist. Functions that are either soon to be removed or too recent shouldn't be part of those headers in that situation, and this should be documented somewhere.

Now, in phobos' case, in does need to perform system calls so it would need those prototypes for those functions somewhere. It makes the most sense that those prototypes be under core.sys. If phobos needs to be portable to some degree it would basically mean that every time a system call is introduced it must be checked that it exists on all versions of libc that phobos needs to support. Again, if this desired, it should be clearly stated and enforced. This is why I suggested having the core.sys modules represent the portability basis so that one can code in phobos without worrying about the portability of the final binary.

The second approach for the headers is have them represent the interface that is provided by libc for the target system. This is what, and I think @ibuclaw suggested. In this situation they should match what the C system headers define so we might as well call them that.

This is only true if we're only talking about never distributing binaries.

Distributing binaries can be done with any of the approached above, you just need to make sure that you compile the programs properly. This does mean obvious things like not compiling for a different architecture nor os but it also includes not using CPU extensions that not all target machines support and not making calls to external functions that may not exist.

It all comes down to what are the actual systems that you want to support. In phobos' case, if we only resort to what CI says, portable enough means being able to run on ubuntu 20.04. If portability is desired then, in the first situation in which core.sys headers represent the portability basis no work needs to be done in phobos, include everything that you need and rewrite anything that isn't available.

In the second case in which the core.sys headers are properly versioned, phobos would then declare that it wants to see the interface provided by, say, glibc-2.30 and use only those functions in order to be portable. The only requirement is that functions that are removed in later versions aren't used to there is a little bit more work when deciding if something can be used.

The fundamental difference in the second case is that the core.sys files can then be used in any D program and they will represent what the host environment offers and not some phobos subset of it. I think you should be able to also do this with importC but then, what would be the point of the core.sys headers?

I think many D users do rely on building a binary on their computer and then running it on another computer.

If people do want to do this then they should do it willingly, by writing code that is portable. This implies that libraries that they link statically (like phobos by default) should also be portable. If we want to support this than having a minimum version of libc that we target is more helpful than "if CI is green it's portable".

If we take this approach we should also consider the users who would prefer that the phobos code uses the efficient functions provided by their system rather then doing its own inefficient thing.

At least I understand now what you mean by doing the check at runtime. libphobos can be compiled for an older glibc and check if the one that it loaded into memory happens to provided the function that it needs. In this situation I would highlight that if the code is bad enough that it would rather be replaced at runtime maybe it's worth considering bringing the minimum supported version higher. In this particular case I agree with you, depending on a version of glibc from 3 years ago is a little bit much.

All in all, it think it would be nice if:

1. core.sys became versioned
2. phobos was explicit about which libc function are allowed and which aren't
3. phobos' build system could be configured to either target a "portable" libc or the current one.

What I really want to see changed it having phobos by in this limbo of portability where there's no documentation about what's allowed and what isn't and the enforcement is a single CI run.

[CyberShadow]

What are the headers in core.sys.* for then? If they're meant to only represent the functions that are available in a libc implementation in an arbitrarily (but consistent) range of versions of such libc then sure, those don't represent what is available for the target host. The only represent what is considered, by D's standards, portable enough. This would allow applications that only use such functions to be portable across libc versions so people can copy their programs safely across systems and have them work.

Note that just because that arbitrary range of versions is not written anywhere doesn't mean that it doesn't exist. Functions that are either soon to be removed or too recent shouldn't be part of those headers in that situation, and this should be documented somewhere.

Now, in phobos' case, in does need to perform system calls so it would need those prototypes for those functions somewhere. It makes the most sense that those prototypes be under core.sys. If phobos needs to be portable to some degree it would basically mean that every time a system call is introduced it must be checked that it exists on all versions of libc that phobos needs to support. Again, if this desired, it should be clearly stated and enforced. This is why I suggested having the core.sys modules represent the portability basis so that one can code in phobos without worrying about the portability of the final binary.

Agreed, and this makes sense to me. I can't speak authoritatively but I think contributions which implement the above would be welcome.

The second approach for the headers is have them represent the interface that is provided by libc for the target system. This is what, and I think @ibuclaw suggested. In this situation they should match what the C system headers define so we might as well call them that.

There must have been a misunderstanding because I have no idea how this would be implemented in practice. We distribute Phobos+Druntime as precompiled libraries. The decision whether to use a certain function or not in this way would need to be done at compile time, so it clearly cannot be done in that way.

Distributing binaries can be done with any of the approached above, you just need to make sure that you compile the programs properly. This does mean obvious things like not compiling for a different architecture nor os but it also includes not using CPU extensions that not all target machines support and not making calls to external functions that may not exist.

True in theory, but in practice, compilers already generally produce fairly portable binaries at their default settings, which even run well across multiple Linux distributions. Yes, this is not codified in any way right now (except said CI), but I don't think that's an excuse to drastically worsen the situation.

It all comes down to what are the actual systems that you want to support.

I don't know if it reflects the current situation or not, but to the best of my knowledge, the DFL's position is that D targets all OS versions which are supported by their vendor. So, for Windows, that would be Windows 10 or newer. Looking at what Canonical supports, I guess that would be Ubuntu 14.04 and newer.

[eli-schwartz]

This is only true if we're only talking about never distributing binaries.

I think many D users do rely on building a binary on their computer and then running it on another computer.

True in theory, but in practice, compilers already generally produce fairly portable binaries at their default settings, which even run well across multiple Linux distributions. Yes, this is not codified in any way right now (except said CI), but I don't think that's an excuse to drastically worsen the situation.

In theory you "can" do this, in practice this is incredibly dangerous.

This is NOT about compilers, at all, period, end of story. This is about glibc, a library which provides a library API and a library ABI. And glibc also has a strict backwards compatibility story. You can compile programs using an old glibc and it shall continue working with new glibc. The reverse is not true.

glibc's ABI is versioned. Every time they add a new symbol to the ABI, they version it by the release it was added in. So the minimum version of glibc you need to run a binary you have built, is the newest ABI your binary ended up including when it was compiled. This may be older than the glibc you compiled with, but it depends on glibc private implementation details. They do sometimes improve the codebase, providing newer versions of an older symbol that are an ABI break and therefore require providing two copies of the symbol: one is symname@GLIBC_2.2.5 or whatever, and the other is symname@GLIBC_2.41.

Your suggestion is to avoid binding to the ABI at compile time for closefrom, because there is no closefrom older than closefrom@GLIBC_2.34. Using dlsym is a workaround for this, but not for other symbols.

e.g. dlsym@GLIBC_2.34 overrides dlsym@GLIBC_2.2.5, so I dearly hope you're not depending on that.

[schveiguy]

Wow, closefrom is nice. That's exactly what we need for this very long standing problem! There have been several PRs to try and fix this, the most recent being #8990 with accompanying bug report https://issues.dlang.org/show_bug.cgi?id=24524

So in response to the discussion about using directly closefrom or using dlsym to look it up, I prefer the dlsym because it's very robust, and does not require synchronized compile-time versioning (a chronic problem). If you have the symbol, it's there, and if not, you do some fallback.

[the-horo]

True in theory, but in practice, compilers already generally produce fairly portable binaries at their default settings, which even run well across multiple Linux distributions. Yes, this is not codified in any way right now (except said CI), but I don't think that's an excuse to drastically worsen the situation.

I compiled this code on my Gentoo machine (glibc-2.39):

void main () {
	import core.stdc.stdio;
	puts("Hello, world!");
}

with dmd-2.109 a.d and got a binary. Here's what it links:

$ lddtree a
a => ./a (interpreter => /lib64/ld-linux-x86-64.so.2)
    libm.so.6 => /usr/lib64/libm.so.6
    libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libgcc_s.so.1
    libc.so.6 => /usr/lib64/libc.so.6
    ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

Here's what I get if I try to run this program on ubuntu-20.04:

circleci@5e25cb3e6e83:~/project$ /tmp/file
/tmp/file: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/file)
/tmp/file: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/file)

This is because, when building my program I built it against the glibc version on my machine. If phobos uses any function that has been changed (even internally) since glibc-2.31 which is what the host system (the ubuntu-20.04 container) has then the produced binary will not work. Because there is no way for phobos to control what glibc functions get updated one concludes that, in order to provide a portable binary, one must build against the minimum version of glibc they want to support.

Here's a script, courtesy of @eli-schwartz, that shows in which glibc versions the functions a binary uses were added:

det_glibc ()
{
    nm -D "$1" | grep -o 'GLIBC.*' | sort -u
}

Using it on my a file it gives:

$ det_glibc a
GLIBC_2.14
GLIBC_2.17
GLIBC_2.2.5
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.32
GLIBC_2.34
GLIBC_2.7

As you can see, because my binary uses functions that were changed in GLIBC_2.32 and GLIBC_2.34 I can not run it on a system that doesn't have, at least, glibc-2.34.

Let's see how one can build against a different version of glibc if that's what it takes to build portable applications.

The first option is to have a setup similar to cross-compiling. Even if your target system has the same CPU architecture and the same operating system, the principles still apply.

Let's start with a D compiler setup that is under the folder CBUILD. That's just a normal installation of a D compiler, it can be used to compile D binaries for the build system (the one that will perform the build) but there is no guarantee that those binaries will run on our intended host, denoted by CHOST. To make binaries compatible for CHOST one just needs to be able to build against libraries, as they would be available on CHOST, in this case we will restrict to glibc. Building against a library requires header files + linking. Since D leaves the linking to the C compiler we will assume that there is already a cross C compiler, which will be appropriately named HOST_CC. The only thing we need to set up are the header files. The most straight forward solution is to write new headers. That's how it's done for C and C++ code so it will work for D as well. The final step is to create a HOST_DC. One solution is writing a dmd.conf in which we put:

[Environment]
CC=<HOST_CC>
DFLAGS=-I<CHOST>/import/d

And have HOST_DC="dmd -conf=<CHOST>/dmd.conf". Another approach is putting the flags straight on the command line: HOST_DC="ldc -conf= -I<CHOST>/import/d -gcc=<HOST_CC>". With gdc you can even get a final binary that compiles for the desired platform and looks in the appropriate places for headers and libs, they are typically names CHOST-gdc so, for example, aarch64-unknown-linux-gnu-gdc.

What am I trying to prove by this tangent? The fact that this problem has already been solved for the more complicated use case and there is only one thing that requires work: having proper headers for the libraries we intend to use.

There must have been a misunderstanding because I have no idea how this would be implemented in practice. We distribute Phobos+Druntime as precompiled libraries. The decision whether to use a certain function or not in this way would need to be done at compile time, so it clearly cannot be done in that way.

I said above that we need proper headers and by that I mean that those headers should, as accurately, describe the API provided by the library. There is nothing wrong with dropping some functions, nothing will break in that case but things will be suboptimal. There is a big issue, however, if a function is wrongly added, as it can lead to build failures.

It doesn't matter for portability how many functions from glibc phobos uses. So long as it uses at least one any application that uses phobos will require, at runtime, a version of glibc that is at least as big as the one that was used for building. Technically you may be able to do with a few versions less, just like in my first example, it may just happen that the functions you currently use haven't been changed in a few releases but in this case just say that you build against that older version of glibc. There is no indication, however, when or whether any of the functions you use will change so you can't depend on things working in the future just because they work now.

As the naive approach above I suggested having different headers for the CHOST. This can be used but it will mean that, from all the systems under core.sys it would only be meaningful to use core.sys.<your_target>, all other files will be redundant. So, if you're building on linux, in your CBUILD/import/d/core/sys folder you would only need linux and posix, all other files would be useless. If you're targeting the same OS but with a slightly changed libc then CHOST will need a copy of the same files but with that little changed applied to them. I think this would lead to pointless duplication and useful information becoming split across different places so I wanted a better solution.

Let's take as an example that we wanted to use the glibc closefrom function. We could have the core.sys.linux.unistd module be:

module core.sys.linux.unistd;

public import core.sys.linux.config;

static if (GLIBC_VER >= 234)
    pragma(msg, "closefrom available");
else
    pragma(msg, "closefrom not available");

In this case the header correctly specified that that function exists only if the glibc is at least 2.34. The trick is having GLIBC_VER being picked up correctly.

Let's turn back to our CBUILD, example. Let's say that CBUILD uses glibc-2.30. In the case <CBUILD>/import/d/core/sys/linux/config.d should contain enum GLIBC_VER = 230;. If CHOST has a more recent glibc then it can have <CHOST>/import/d/core/sys/linux/config.d contain enum GLIBC_VER = 236;. The next step is getting import core.sys.linux.config to resolve to the CHOST value when building for it but keep the CBUILD/import/d/core/sys/linux/unistd.d header intact. That's not that hard, both -I and -mv can be used.

Here's a full example:

/////////////////// CBUILD/unistd.d ////////////////////////
module unistd;

import config;

static if (VER >= 234)
	pragma(msg, "closefrom available");
else
	pragma(msg, "closefrom not available");
/////////////////// CBUILD/config.d ////////////////////////
module config;

enum VER = 230;
/////////////////// CHOST/config.d ////////////////////////
module config;

enum VER = 236;

And when compiling:

$ dmd -ICBUILD CBUILD/unistd.d -o- # build for CBUILD
closefrom not available
$ dmd -ICHOST -ICBUILD CBUILD/unistd.d -o- # build for CHOST
closefrom available
$ dmd -mv=config=CHOST/config -ICBUILD CBUILD/unistd.d -o- # build for CHOST
closefrom available

This allows to keep all the headers in one place, CBUILD/import/d/core/sys, for every platform, and only have to change the configured versions if we target a different system. The only downside is that enum VER = ... needs to be automatically populated somewhere. The solution for this is, when compiling druntime, to check for the version of glibc and embed that into the config.d file. This doesn't lower to portability of the code, the same glibc gets linked and the same version requirements end up imposed by using phobos, the only difference is that we would be able to use all the features that are provided by the glibc we are using instead of ignoring them because the core.sys modules don't expose the appropriate functions.

In short, thing change would allow phobos code to do:

import core.sys.linux.unistd;
import core.sys.freebsd.unistd;
import core.sys.openbsd.unistd;

static if (!__traits(compiles, closefrom))
       void closefrom (int lowfd) { ... }

closefrom(4);

Without any worry for compatibility.

I don't know if it reflects the current situation or not, but to the best of my knowledge, the DFL's position is that D targets all OS versions which are supported by their vendor. So, for Windows, that would be Windows 10 or newer. Looking at what Canonical supports, I guess that would be Ubuntu 14.04 and newer.

At the beginning I implied that there are more ways to build against an older glibc in order make portable applications which is what the DFL wants their dmd*.linux.tar.xz archives to contain. I presented first the complicated approach to highlight why changes in core.sys can be useful but for strict portability the easiest way is to spin up a container that contains your oldest supported glibc and build against that. So long as the druntime code correctly picks up it's version all should be good.

Indeed, this wouldn't fix anything for older targets, but there's nothing stopping us from changing how the closefrom fallback implementation works. We can use dlsym to try to check at runtime for closefrom's availability and, if it's not found, there's also the other PR that tries to fix this. I think it's best that the changes of making the fallback closefrom are kept to https://github.com/dlang/phobos/pull/8990 and leave this PR for being able to link against closefrom if it's available.

[CyberShadow]

This is because, when building my program I built it against the glibc version on my machine.

...

Here's what I get if I try to run this program on ubuntu-20.04:

Well, that's a little unfair. A more fair attempt would be to use Ubuntu 24.04 :)

Yes, you're right in general. But, realistically, probably most users won't concern themselves for why this doesn't work; rather, they will put the error message in Google, find the advice to use a container, or GitHub Actions runner with an older Ubuntu version, or generally take the shortest path towards making it work (e.g. try -static), try it on a few computers, and go with that if their program runs, whether or not anyone considers anything to be "incredibly dangerous" or not.

I think that if D had some kind of mechanism where it adapts the built runtime to the host environment would be helpful in situation where D is packaged by distributions. Then, there would be some guarantee that if you installed D using a package manager, it and the program it builds are going to be compatible with the rest of the system, and maybe use the versions of things in its environment to their full potential. But, I don't think such a mechanism exists right now, and it would indeed only help in those situations. Such situations are also a little outside DFL's responsibility - after all, distributions can apply whatever patches they want to improve compatibility in whatever way they see fit.

We can use dlsym to try to check at runtime for closefrom's availability

I'm not sure if that's what you meant or not, but we can't just use dlsym to check if closefrom is available. We have to also call it via the returned pointer, and only via the returned pointer - meaning, we can't use the declaration in https://github.com/dlang/dmd/pull/16806 - not unless it's guarded by a static if that effectively means "D and all D programs built by it shall be executed by a system which has closefrom".

[eli-schwartz]

I have already pointed out that you cannot use dlsym at all, period, end of story, no ifs ands or buts.

If you use dlsym then it adds a dependency on newer glibc, which you already said you don't want to do.

Many things add a dependency on newer glibc.

Well, that's a little unfair. A more fair attempt would be to use Ubuntu 24.04 :)

This is rules lawyering. Ubuntu 24.04 is newer than Ubuntu 20.04, ipso facto its glibc is newer and ipso facto running the D compiler on Ubuntu 24.04 produces binaries that don't run on Ubuntu 20.04

In this case the example code when compiled on Gentoo with glibc 2.39 produces binaries that require a minimum of glibc 2.34. If Ubuntu 24.04 has at least glibc 2.34 then those results will replicate to compiling on Ubuntu 24.04.

By sheer coincidence, Ubuntu 24.04 also has glibc 2.39, just like Gentoo. So it's a perfect test, actually.

...

"That test is unfair, you didn't compile using the supported distro" is Oracle style support responses. Endlessly depressing. The rules of how glibc works have been clearly laid out, and the reasons why they apply regardless of distro have been explained. To pull an Oracle and say you simply will not take the discussion into consideration in the first place and don't care about how technology works because the test case didn't use the Officially Supported Platform Image Of Officialness is... well.

[the-horo]

Yes, you're right in general. But, realistically, probably most users won't concern themselves for why this doesn't work; rather, they will put the error message in Google, find the advice to use a container, or GitHub Actions runner with an older Ubuntu version, or generally take the shortest path towards making it work (e.g. try -static), try it on a few computers, and go with that if their program runs, whether or not anyone considers anything to be "incredibly dangerous" or not.

I am not trying to say how people should be allowed to use phobos in their code. If people want to compile their programs on recent machines and then run then on older ones then all the power to them. If it works, it works, if it doesn't they get to deal with the errors. Phobos can not do anything outside of providing the same guarantees as glibc which are that, once compiled, you are able to run the program on any systems with a glibc version higher or equal then what was used during the build. This is how it has always been and will be, I am just stating clearly what the rules are, I'm not trying to change them, they have always been like this.

I think that if D had some kind of mechanism where it adapts the built runtime to the host environment would be helpful in situation where D is packaged by distributions.

It is the exact opposite. Having D programs adapt at runtime to the environment is useful only for distributors that target multiple versions of certain platforms. Linux distribution don't target multiple platforms, they target their own distribution, and the versions of libraries that they support is already know at build time.

Then, there would be some guarantee that if you installed D using a package manager, it and the program it builds are going to be compatible with the rest of the system, and maybe use the versions of things in its environment to their full potential. But, I don't think such a mechanism exists right now,

You are describing the configure step in a build. Most respectable build systems (dmd's makefile and dub don't qualify) run a step in which they check the status of the system they will run on. You can verify that a library exists, or that it is a specific version. For C and C++ programs you can also check that a function exists in a library or that there is a #define for a constant you are interested it. You typically then transform that information into a #define, or in D's case, version, and then in the library code you do version (HAVE_LIB_FOO) import foo.thing etc.

and it would indeed only help in those situations. Such situations are also a little outside DFL's responsibility - after all, distributions can apply whatever patches they want to improve compatibility in whatever way they see fit.

I am trying to provide a solution that is both friendly for linux distributions, by having druntime's build system automatically pick up the version of glibc they are building against and have it so that core.sys.linux.* will offer bindings for that version, being equivalent to doing #import <unistd.h> in a C program on the same system; and friendly for the DFL by being able to build on an older systems to provide portable binaries that run on any linux distribution.

It seems that we don't really understand each other so let start again.

Issue: some D projects depend on glibc functions so they would need the prototypes for those functions. A popular example of such a project is phobos. How we can fix it: Provide declarations for the glibc functions in a header, let's have it be under core.sys.linux.

How to we implement the solution: We needs to write bindings for a library. There's plenty of projects on core.dlang.org that do just that so the steps needed are already known:

1. Pick a version of that library
2. Check the headers of that version of the library and write associated D declarations.

The accent is that the bindings are for a version of the library.

There is one more question needed to be answered, what version of glibc the bindings will wrap. Again there are two possibilities:

1. A random version, probably older, let's say 2.30.
2. The same version that's available on the target system.

All I am arguing is that option 2 is way better than option 1. It is not right to require that anyone who wants to use glibc functions only uses the ones in glibc-2.30, this is the opposite of offering portability (both for running and building) which is how this whole conversation got started.

Option 2 is better in any way possible outside of requiring that extra step during the build. It is not a complicated step:

############### Makefile #################
unistd.d: unistd.d.in includefeatures.c
	cp $< $@
	$(CC) -E -dM includefeatures.c | sed -nE '/^[[:space:]]*#define[[:space:]]+(__GLIBC__|__GLIBC_MINOR__)/ { s/^.*(__GLIBC__|__GLIBC_MINOR__)[[:space:]]*(.*)$$/enum \1 = \2\;/; p}' >> $@

//////////////////////// includefeatures.c ////////////////////
#include <features.h>

////////////////////////// unistd.d.in ///////////////////////////
module unistd;

version (CRuntime_Glibc):

bool GLIBC_PREREQ(int major, int minor) {
	alias combine = (a, b) => (a << 16) + b;
	return combine(__GLIBC__, __GLIBC_MINOR__) >= combine(major, minor);
}

extern(C):
nothrow:
@nogc:

static if (GLIBC_PREREQ(2, 34))
void closefrom();

// GLIBC version constants should follow:

This even works when cross compiling because $(CC) is only used to do macro processing, not code generation and it will generate appropriate unistd.d headers. On my glibc-2.39 Gentoo:

module unistd;

version (CRuntime_Glibc):

bool GLIBC_PREREQ(int major, int minor) {
	alias combine = (a, b) => (a << 16) + b;
	return combine(__GLIBC__, __GLIBC_MINOR__) >= combine(major, minor);
}

extern(C):
nothrow:
@nogc:

static if (GLIBC_PREREQ(2, 34))
void closefrom();

// GLIBC version constants should follow:
enum __GLIBC__ = 2;
enum __GLIBC_MINOR__ = 39;

allowing me to see the closefrom declaration. On the circleci ubuntu-20.04 container:

module unistd;

version (CRuntime_Glibc):

bool GLIBC_PREREQ(int major, int minor) {
	alias combine = (a, b) => (a << 16) + b;
	return combine(__GLIBC__, __GLIBC_MINOR__) >= combine(major, minor);
}

extern(C):
nothrow:
@nogc:

static if (GLIBC_PREREQ(2, 34))
void closefrom();

// GLIBC version constants should follow:
enum __GLIBC__ = 2;
enum __GLIBC_MINOR__ = 31;

Hiding the declaration because the glibc version is too old.

The only way to get this code to fail to link or expose invalid symbols is by manually modifying the version constants or by copy-pasting the header to a system with an older glibc. Nothing can fix the former but the later can be resolved by just replacing the version constants appropriately by the person who copy pasted the files.

That is, what I believe, an insignificant cost to pay compared to the reward of having functioning, consistent, and, up to date, libc wrappers.

[the-horo]

The only way to get better from this would be if we depended on importC to include <features.h> and extract the information from that whenever someone imports core.sys.posix.*. I don't seem why this couldn't work, so long importC can handle whatever weird preprocessor macros it may encounter in those headers. I don't know how good it is because I don't use it.

Either way, I'm fine with both approaches, embedding the version of libc or getting it through importC, I only want the wrappers to become properly versioned.

[CyberShadow]

I have already pointed out that you cannot use dlsym at all, period, end of story, no ifs ands or buts.

We already use it to load libcurl at runtime for std.net.curl. So, I think that problem is solved for Phobos.

I'm guessing D binary tarballs are built against an older glibc, so this would avoid that particular problem.

It is the exact opposite. Having D programs adapt at runtime to the environment is useful only for distributors that target multiple versions of certain platforms. Linux distribution don't target multiple platforms, they target their own distribution, and the versions of libraries that they support is already know at build time.

No, that is what I meant to say. ("built runtime" as in the runtime that was built when the package was built)

Apologies, this discussion is a little overwhelming with respect to how much time I can dedicate to it. I'm aware of many things that have been written above and have done a poor job communicating this, so we're going in circles somewhat. It sounds like you're trying to convince someone of something, but I'm not the correct person who needs convincing. In any case, discussing a mechanism that allows Druntime to make some declarations available depending on the glbc version it was built against seems a little out of scope for this PR.

[eli-schwartz]

I have already pointed out that you cannot use dlsym at all, period, end of story, no ifs ands or buts.

We already use it to load libcurl at runtime for std.net.curl. So, I think that problem is solved for Phobos.

I'm guessing D binary tarballs are built against an older glibc, so this would avoid that particular problem.

It is possible to use dlsym for the intended purpose of dlsym, which is to access libraries which are optionally dlopened (e.g. libcurl).

It is not possible to use it for the use case described in this PR, which is "allow compiling against a new glibc, and running against an old glibc".

The D binary tarballs don't avoid any particular problem by being built against an older glibc -- the D binary tarballs simply do the correct thing, which all software is expected to do for portability and reliability, and build against the oldest version of glibc they intend to support.

It sounds like you're trying to convince someone of something, but I'm not the correct person who needs convincing.

Well, you're also the person that originally suggested that dlsym is necessary in the first place...

[CyberShadow]

The D binary tarballs don't avoid any particular problem by being built against an older glibc -- the D binary tarballs simply do the correct thing, which all software is expected to do for portability and reliability, and build against the oldest version of glibc they intend to support.

Err, that's what I meant?

It is not possible to use it for the use case described in this PR, which is "allow compiling against a new glibc, and running against an old glibc".

Sorry, why would this not work?

import std.exception;
import std.stdio;

import core.sys.posix.dlfcn;

extern(C) int closefrom(int);

void main()
{
	auto glibc = dlopen("libc.so", 0);
	auto closefrom = cast(typeof(&closefrom))dlsym(glibc, "closefrom");
	if (closefrom)
	{
		writeln("Available");
		auto f = File("/dev/stdout", "wb");
		closefrom(f.fileno);
		f.close().assertThrown;
		writeln("OK");
	}
	else
		writeln("Not available");
}