Encourage using feature detection instead of distro name/version

[encukou]

Hello! I'm the person responsible for dropping platform.linux_distrubution from Python 3.8. We're now recommending distro instead. Thank you for all the work going into this module – it's certainly useful in certain cases, like finding out what to put in a log or problem report.

However, think that distro's docs should include a warning against using it for decisions, because a platform's name is not a good indicator of what is possible on it.

Once people apply custom workarounds for Ubuntu (and use distro info for that), less popular derivatives like Linux Mint will need to report themselves as Ubuntu to get those workarounds. That then leads to the need for better distro reporting tools, like this one... which then leads to more distro-specific workarounds. It's a vicious circle.

It's the same problem as "browser sniffing" in the web world, where modern “user agent” strings contain mostly lies:

Mozilla/5.0 (Linux; Android 4.4.2); Nexus 5 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Mobile Safari/537.36 OPR/20.0.1396.72047

You'll find warnings against using browser-agent in any respectable recent documentation on the topic.

[doko42]

completely agreed with that. Use it carefully if you need to. But better try to make decisions on features found.

[nir0s]

While I definitely agree we should add a warning, I disagree that we should state that it shouldn't be used to make decisions. Rather, I think it makes sense to state that "It should be used to make decisions only if it makes sense".

A for instance:

Cloudify uses Wagon to build their plugin packages. Packages that contain C extensions are compiled for a specific distribution and is expected to run on that distribution and only if the distribution contains the exact glibc (etc) as the build environment. You could say that they should start checking ABI's, features, glibc versions, etc.. for every possible distribution out there. What made more sense to them, is to check whether the destination distribution's id and version are the same as the distro the package was build on and document the culprits. Someone else might decide otherwise, but that doesn't mean that it's a bad decision.

What I do agree with, is that "Using the distribution info to decide on things reduces precision. If you care about precision, go for a lower-level decision-making algorithm".

WDYT?

[encukou]

What made more sense to them, is to check whether the destination distribution's id and version are the same as the distro the package was build on and document the culprits. Someone else might decide otherwise, but that doesn't mean that it's a bad decision.

That is exactly what forces Ubuntu derivatives to identify as Ubuntu (and optionally publishing their “true name” in a non-standard location, which will in turn force distro to also consider that non-standard location, making it de-facto standard, and so on).

[basak]

The particular derivative case is solved by Debian by using dpkg-vendor --derives-from. So a conditional check would be "Is this Ubuntu or derived from Ubuntu?" instead of "Is this Ubuntu?"

[encukou]

That looks like a distro-specific solution, which requires a special case in the detection code :slightly_frowning_face:

[basak]

I just mentioned it as an example of how the problem is solved today. I didn't intend to suggest that you used that as-is. But os-release does have ID_LIKE, which provides the information you need to implement something similar. I think you could implement a "am I like X" function using that data.

[nir0s]

@encukou, I'd appreciate it if you proposed the wording for the wording. Also, where do you think the warning should go?

[encukou]

Sorry for the delay. Would the following work for the documentation, under Overview and motivation?

Preferring feature detection

Information from this package is useful for logging and problem reporting.

However, it is usually a bad idea to base automated decisions on the information. Instead, try to use feature detection. If you need to install packages, look for yum vs. apt-get commands, if you need a specific version of glibc, query the library version directly, and so on.

If people use distro to apply custom workarounds for, say, Ubuntu, less popular derivatives like Linux Mint will need to report themselves as Ubuntu to get those workarounds. That then leads to the need for more powerful distro reporting tools, which then leads to more distro-specific workarounds, and more distros needing to pose as other distros to get the workarounds. Please, do not feed the vicious circle.

[encukou]

@basak

[...] But os-release does have ID_LIKE, which provides the information you need to implement something similar. I think you could implement a "am I like X" function using that data.

But what does ID_LIKE mean? In what respect is the distro like another distro? If I take Ubuntu, re-package it all into RPM, and put in yum instead of apt-get, should I label it as “like Ubuntu”?

I believe this scope problem is also the main reason for removing platform.linux_distribution() from Python: We don't know the questions we are trying to answer. Or, we're trying to give a single answer to too many questions at once.

[basak]

On Fri, Apr 12, 2019 at 07:30:32AM -0700, Petr Viktorin wrote:

@basak

[...] But os-release does have ID_LIKE, which provides the information you need to implement something similar. I think you could implement a "am I like X" function using that data.

But what does ID_LIKE mean? In what respect is the distro like another distro? If I take Ubuntu, re-package it all into RPM, and put in yum instead of apt-get, should I label it as “like Ubuntu”?

I don't think it needs to specify crazy situations and cover all cases before it can be useful. If you're going to make such invasive changes in your derivative, then you'll need to patch things all over the place to accomodate differing behaviour anyway. At that point, you might as well add specific behaviour for the new scenario in those patches anyway (and send them upstream etc).

In the majority of cases, something like "like Debian" will do, and automatically incorporate Ubuntu and Mint.

FWIW, the specification of ID_LIKE is in here: https://www.freedesktop.org/software/systemd/man/os-release.html

I believe this scope problem is also the main reason for removing platform.linux_distribution() from Python: We don't know the questions we are trying to answer. Or, we're trying to give a single answer to too many questions at once.

Related are my comments and their replies at https://bugs.python.org/issue28167. This isn't a problem that the Python world needs to solve alone. The Free Software community has converged on os-release, and it would be very useful to at least pass that through in a Pythonic way. Extra smarts is fine, but it only really needs to be added on as an opt-in now, or used where (presumably only obscure distributions now when) os-release isn't available.

I'm not suggesting that feature detection is wrong; it certainly is the most granular and future-proof way of solving the general problem. I'm only saying that it can be difficult to do right in some cases, and in those cases would be overkill when ID_LIKE would suffice - if only there were a library that allowed Python developers to do that test easily in one line.

[encukou]

@basak

[...] I'm not suggesting that feature detection is wrong; it certainly is the most granular and future-proof way of solving the general problem. I'm only saying that it can be difficult to do right in some cases, and in those cases would be overkill when ID_LIKE would suffice - if only there were a library that allowed Python developers to do that test easily in one line.

Oh, nothing against quick hacks! But I'd like to make sure people know what the proper solution is. Instead of having my new Linux masquerade as something it's not, I should be able to send a PR to the offending project to enable feature detection, and link to distro's official docs to support my case :)