stderr duplication failed in console scripts

[pierreluctg]

Description

I use a console_scripts entry point with a test function. If I do pip install . or use setup.py to make a binary dist then install it, under some shell like Git Bash, running the created test-script.exe. stdin, stdout and stderr should all be set to a TextIOWrapper however, stderr is set to None.

The stderr handle is apparently not correctly set.

This is probably related to https://github.com/pypa/pip/issues/10444 and the fix introduced in Distlib 0.3.4 for a similar issue.

Expected behavior

$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>

pip version

22.0

Python version

Python 3.7

OS

Windows

How to Reproduce

# setup.py
from setuptools import setup

setup(
    name="test",
    packages=["testing"],
    entry_points={
        "console_scripts": [
            "test-script=testing:main"
        ]
    }
)

# testing/__init__.py
import sys


def main():
    print(f"stdin={sys.stdin}")
    print(f"stdout={sys.stdout}")
    print(f"stderr={sys.stderr}")

Output

# With pip 21.*
$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>

# With pip 22.0, 22.0.1 and 22.0.2 
$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=None

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[pradyunsg]

/cc @vsajip, since they're the expert here. :)

[pradyunsg]

x-ref #10874, which also seems like something that's related to the Windows launchers.

[pierreluctg]

The following Java code calling test-script.exe (the console script) is also producing the issue:

package com.example.testScript;

import java.io.IOException;

public class TestScript {
    public static void main(String[] args) throws IOException, InterruptedException {
        String cmd = "test-script.exe";
        ProcessBuilder pb = new ProcessBuilder().command(cmd);

        // This is apparently exposing the issue
        pb.redirectErrorStream(true);

        Process iostat = pb.inheritIO().start();
        int exitCode = iostat.waitFor();
        System.out.println("exitCode = " + exitCode);
    }
}

[pierreluctg]

@vsajip can you please have a look at this distlib regression?

Switching the distlib binaries back to 0.3.3 (from pip 21.x) or moving back to pip 21.x is not showing the issue.

In the affected effected environment this is a major issue.

thank you

[carlkl]

Right now I can only recommend to install an older pip version on Windows: python -m pip install -U pip==21.3.1 After that all packages with broken launchers also should be reinstalled.

[vsajip]

@pierreluctg I'm aware of this issue, but I probably won't have the bandwidth to spend much (if any) time on it until the last week of March 2022, due to other commitments 😞

[pierreluctg]

Thank you @vsajip for the update. Is there any additional information that you need from us in the meantime to help you fix the issue when you will have free time?

[carlkl]

Right now my solution is to install an older pip version on Windows: python -m pip install -U pip==21.3.1. After that all packages with faulty launchers should also be reinstalled. I've seen issues with meson as well as with tqdm.

[pradyunsg]

Is this bad enough to justify a bug fix release?

[shireenrao]

I have started running my console scripts with Python to get around this for now

$ python -m pytest

[vsajip]

Is there any additional information that you need from us in the meantime to help you fix the issue when you will have free time?

Does the problem occur for GUI scripts as well as console scripts? (Might be harder to ascertain, as there's no console to display diagnostic output). Other than that, no other things I can think of.

[aminya]

Right now my solution is to install an older pip version on Windows: python -m pip install -U pip==21.3.1. After that all packages with faulty launchers should also be reinstalled. I've seen issues with meson as well as with tqdm.

@carlkl

For meson as an example, it is enough to run the following? or should it be specifically installed via the pip executable?

python -m pip install -U pip==21.3.1
python -m pip install meson

[carlkl]

@aminya,

For meson as an example, it is enough to run the following? or should it be specifically installed via the pip executable? python -m pip install -U pip==21.3.1 python -m pip install meson

I reinstalled meson in excact that way.

[pfmoore]

Is this bad enough to justify a bug fix release?

It doesn't seem to happen all the time. The following reproducer builds a simple console executable, and when I run that it works fine (shows stderr as a file, not as None, and writes output to stderr).

So I think someone needs to be more precise about how this issue gets triggered, before we can assess the severity of the problem. (On the other hand, if pip install meson is installing a broken application, that's fairly severe for meson users, so maybe that's enough on its own? How broken is meson as a result of this bug? Does it not work at all, or is it only particular functionality that's affected?)

py -m venv .venv
.venv/scripts/python -m pip install -U pip setuptools wheel
mkdir test_bug
set-content test_bug/test_bug.py -encoding UTF8 @"
import sys
def main():
    print(repr(sys.stdin))
    print(repr(sys.stdout))
    print(repr(sys.stderr))
    print('Writing to stdout')
    print('Writing to stderr', file=sys.stderr)
"@
set-content test_bug/setup.py -encoding UTF8 @"
from setuptools import setup
setup(
    name='test_bug',
    version='1.0',
    py_modules=['test_bug'],
    entry_points = {
        'console_scripts': ['test_bug=test_bug:main'],
    },
)
"@
.venv/scripts/pip install ./test_bug
.venv/scripts/test_bug.exe

[carlkl]

@pfmoore, I don't know how more more precise, please take a look at: https://github.com/pypa/pip/issues/10444#issuecomment-1030805082. Other projects now have problems with pip 22, not only meson. To be even more precise: The problem is due to the change to distlib-0.34.

On windows you have to use the meson launcher, as pip -m meson dosn't work. If meson was install with pip 22 meson doesn't work anymore due to this issue.

[pfmoore]

@carlkl I understand what the problem is, and I understand what we'd need to do to fix it (revert to an older distlib, or wait for a distlib fix). What I don't know is how many people are affected, nor do I know whether this issue is more disruptive than the issue that the distlib upgrade fixed, or whether it will be more disruptive than the cost of making a bugfix release.

The comment you linked to says it is a "severe" bug. It's not severe for me, I've never seen it happen. So if you can be more precise about what circumstances cause it to occur, that would make it easier to assess the severity in a way that doesn't just boil down to "it doesn't happen for me" vs "it happens for a program someone else uses a lot". You also say "Now sys.stderr is None if you start launchers created by pip 22". My reproducer above demonstrates that this isn't always true. So the question is how often is that the case, and under what circumstances. I can't tell, because as yet I haven't ever had it happen for me.

🤷 I'm not really that bothered. Ultimately someone needs to make the case that this is important enough for us to produce a bugfix release with a reverted distlib. I'm only trying to help people make that case - if they don't, then this will likely just wait for a fix in distlib, and the next pip release.

[carlkl]

main issue: https://github.com/pypa/pip/issues/10444

Here is a list of affected projects I'm aware of:

• https://github.com/mesonbuild/meson/pull/9918
• https://github.com/takluyver/pynsist/issues/243 is somehow related
• also tqdm needs sys.stderr during installation
• and another project somewhere I forgot to bookmark,

As I understand, it will happen if (and only if):

• you installed pip 22 first
• you install meson or any other project (which depends on sys.stderr) afterwards
• you use the binary launcher instead oy python -m whatever

IMHO it is a severe problem that will most likely hit other project sooner or later. In case it pops up you have absolutely no clue what happens. It took me hours to come to that conclusion with pip 22 and distutils.

[pfmoore]

• #10444 is about GUI script wrappers, isn't it?
• A brief skim of the nsist issue also seems to be about GUI executables.
• I don't get anything failing when I install tqdm, or run the tqdm executable.

As I understand, it will happen if (and only if):

• you installed pip 22 first - I did that
• you install meson or any other project (which depends on sys.stderr) afterwards - I did that, assuming my test project counts
• you use the binary launcher instead oy python -m whatever - I did that

As I say, I can't reproduce this.

If it's only about GUI executables, my view is that it's (a) not demonstrably worse than reverting, as I believe the distlib change was to fix some aspect of how GUI scripts were handled, and (b) not major, demonstrated by the fact that the original distlib issue had gone for so long without being reported.

The meson case worries me more, as it seems to be related to a console script. But until someone can provide a reproducer which doesn't involve meson, I remain unconvinced it's a general issue.

Note: I'm not the RM for 22.0. If @pradyunsg decides to do a bugfix release, that's fine. I'm mostly interested here as I want to improve the process whereby we gather information about post-release issues, and in particular establish a principle that we should be looking for more objective measures of the impact rather than just reacting to generalised "it's broken" messages.

[eli-schwartz]

The Meson issue as reported by one of our users apparently only manifests in CI, not on local dev machines, and only happens when meson.exe's symbolextractor internal command is launched by ninja, and it works okay when launched by hand.

Ninja does do some playing around with file descriptors, because stdout/stderr get buffered (failing isatty() checks) in order to print all output when ninja finds best, particularly to avoid mixing output from multiple build rules running simultaneously in a parallel build.

[pradyunsg]

It seems like a fairly convoluted problem to debug, and I'm fine with reverting the distlib upgrade in a bugfix -- I'll cut it sometime next week tho.

If someone wants to file a PR downgrading to distlib 0.3.3 (edit vendor.txt to say distlib==0.3.3 + run nox -s vendoring + write news fragments (news/distlib.vendor.txt + news/10875.bugfix.txt) + commit this whole thing), that'll reduce the work I'd need to do for cutting a bugfix; which I'd appreciate. :)

It's still not clear to me how pervasive this issue is -- I see a few folks pinning to older version of pip because of this bug tho.

[carlkl]

Unfortunately the old distlib also has problems: see https://github.com/pypa/pip/issues/10444#issuecomment-1030837832. I can reproduce the problem with mne @cbrnr described and pip 21 on my local machine. Simply going back to an older distlib would open another can of worms again. Too bad.

[pfmoore]

Simply going back to an older distlib would open another can of worms again.

So what we're saying is that there's no-one actually requesting a bugfix release that reverts the distlib change?

ping @pradyunsg as not doing a bugfix release is even less work 😉

[carlkl]

Even if a quick solution would be desirable: yes, a bugfix release with distlib-0.33 is probably not the best idea right now.

[pradyunsg]

Neato. Looks like this needs a bugfix on distlib's end that we'll include in our next release then!

[carlkl]

+1

[cbrnr]

The problem with pip install mnelab prior to v0.22 was related to its gui_scripts entry point. Maybe the recent changes fixed that but created a problem with the console_scripts entry point?

[pierreluctg]

@carlkl I understand what the problem is, and I understand what we'd need to do to fix it (revert to an older distlib, or wait for a distlib fix). What I don't know is how many people are affected, nor do I know whether this issue is more disruptive than the issue that the distlib upgrade fixed, or whether it will be more disruptive than the cost of making a bugfix release.

The comment you linked to says it is a "severe" bug. It's not severe for me, I've never seen it happen. So if you can be more precise about what circumstances cause it to occur, that would make it easier to assess the severity in a way that doesn't just boil down to "it doesn't happen for me" vs "it happens for a program someone else uses a lot". You also say "Now sys.stderr is None if you start launchers created by pip 22". My reproducer above demonstrates that this isn't always true. So the question is how often is that the case, and under what circumstances. I can't tell, because as yet I haven't ever had it happen for me.

🤷 I'm not really that bothered. Ultimately someone needs to make the case that this is important enough for us to produce a bugfix release with a reverted distlib. I'm only trying to help people make that case - if they don't, then this will likely just wait for a fix in distlib, and the next pip release.

I have an example of a very common use case affected by this bug.

Imagine a python developer using Jenkins to run his unit tests, using pytest, on a Windows host.

Now because he start using pip 22.x (he updated because pip warned him that he was not using the latest version), running pytest will not produce any output on the console, tests will not run and there is no reporting.

Now how is this user supposed to root cause this issue? There is no output at all since stderr is None.

And this is not a intermittent issue, in the affected environment (like Jenkins on Windows in this example), stderr will always be None.

@pfmoore, I hope this helps you understand the importance of getting this bug fixed.

[carlkl]

https://github.com/pypa/pip/issues/10444#issuecomment-1031092176 - hopefully it helps

[jeremyd2019]

# test-script
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
stderr=None

# cp ../testlauncher/* /c/hostedtoolcache/windows/Python/3.10.1/x64/Lib/site-packages/pip/_vendor/distlib/

# rm -rf build/ test.egg-info/

# pip install .

# test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>

testlauncher from https://github.com/jeremyd2019/simple_launcher/actions/runs/1808357504 (all-in-one artifact, thanks to @lazka for the CI setup), change to launcher code https://github.com/jeremyd2019/simple_launcher/commit/948b4e72c6cb10eb23a0909bb887586fc34a34f3

[WenjieZ]

Imagine a python developer using Jenkins to run his unit tests, using pytest, on a Windows host.

Now because he stared using pip 22.x (he updated because pip warned him that he was not using the latest version), running pytest will not produce any output on the console, tests will not run and there is no reporting.

Now how is this user supposed to root cause this issue? There is no output at all since stderr is None.

Hi @pierreluctg, I had a similar issue (https://github.com/pytest-dev/pytest/issues/9627) with AppVeyor, Windows, and pytest. The troubleshooting cost me an afternoon.

BTW, I found a walk-around by installing pytest from conda-forge.