Subprocess termination failed due to Fastly CDN downtime and therefore pypi downtime

[kristang]

Issue description

During the Fastly CDN downtime (https://status.fastly.com/incidents/vpk0ssybt3bj), pypi.org was affected. Before realizing there was an issue I was trying install a new pipenv, but was encountering an issue that was not clearly identifiable as being related to pypi being down.

When no response was received from pypi.org pipenv seems to keep starting subprocesses for each individual package install, and thus spawn new python processes that starts eating memory.

Using python 3.6, pipenv 2020.11.15 and pip 21.1.2. I also experienced the issue on python 3.9 but the attached results are for 3.6

Expected result

pipenv install to work as it had done earlier

Actual result

A lot of "An error occured while installing" and then:

Installing initially failed dependencies...
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\cli\command.py", line 253, in install
[InstallError]:       site_packages=state.site_packages
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 2063, in do_install
[InstallError]:       keep_outdated=keep_outdated
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 1312, in do_init
[InstallError]:       pypi_mirror=pypi_mirror,
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 900, in do_install_dependencies
[InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Looking in indexes: https://pypi.org/simple
[pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement pytz==2021.1
[pipenv.exceptions.InstallError]: ERROR: No matching distribution found for pytz==2021.1
ERROR: Couldn't install package: pytz
 Package installation failed...
     ================================ 23/29 - 00:00:37
c:\program files\python36\lib\subprocess.py:786: ResourceWarning: subprocess 28840 is still running
  ResourceWarning, source=self)
sys:1: ResourceWarning: unclosed file <_io.FileIO name=9 mode='wb' closefd=True>
sys:1: ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True>
c:\program files\python36\lib\subprocess.py:786: ResourceWarning: subprocess 38960 is still running
  ResourceWarning, source=self)
sys:1: ResourceWarning: unclosed file <_io.FileIO name=11 mode='wb' closefd=True>
sys:1: ResourceWarning: unclosed file <_io.FileIO name=12 mode='rb' closefd=True>
c:\program files\python36\lib\subprocess.py:786: ResourceWarning: subprocess 37896 is still running
  ResourceWarning, source=self)
sys:1: ResourceWarning: unclosed file <_io.FileIO name=15 mode='wb' closefd=True>
sys:1: ResourceWarning: unclosed file <_io.FileIO name=16 mode='rb' closefd=True>
c:\program files\python36\lib\subprocess.py:786: ResourceWarning: subprocess 13660 is still running
  ResourceWarning, source=self)
sys:1: ResourceWarning: unclosed file <_io.FileIO name=17 mode='wb' closefd=True>
sys:1: ResourceWarning: unclosed file <_io.FileIO name=18 mode='rb' closefd=True>

Steps to replicate

I have not found a way to replicate this issue.

[GPHemsley-RELX]

We are having trouble authenticating into our internal package index, and so pipenv is not able to obtain the packages it's looking for. The warnings themselves occur during shutdown, but it's unclear what code is spawning the subprocesses in the first place.

Perhaps the code that bails on package installation error is not properly killing its subprocesses?

[matteius]

@kristang or @GPHemsley-RELX Is this still an issue with 2022.1.8 or the main branch? The code that is spawning the subprocess is pip itself calling call_subprocess which uses subprocess.Popen for each package installation. If there is a memory leak from this, it may have been in the pip internals. If so, we have a number of things to have you test including a new vendoring of pip 22.0.4 to determine if the bug is still present.

[kristang]

Hello Matt,

In my case I can't really replicate the issue. It was in connection to a internet-wide breakdown at Fastly, and I just stumbled across this sub-process issue while debugging.

[GPHemsley-RELX]

@matteius I think @kristang was indicating that the situation is difficult to reproduce artificially, as it involves network problems, not that the issue no longer exists.

[matteius]

@GPHemsley-RELX well its not entirely clear from the report how to proceed with the investigation or what the desired outcome is. I can open it back up, but I fear it would just sit open for another long while without more details. It is well known that pip and pipenv use subprocesses and it seems clear from the output it had trouble with ERROR: Couldn't install package: pytz which was caused by networking issues. What really could be done here?

[kristang]

This was not really a request for specific changes. This was more of a "Hey I noticed this separate thing happened because of another issue"

My very narrow understanding would suggest making some sort of limit to the number of processes that is allowed to run, but I have no knowledge about pipenv or pip to comment on where something like that would be best implemented.

[matteius]

Most all of the subprocess usage is within pip and other vendor code. I am compelled to say this is something that is not fixable -- that many subprocesses would have been running for a normal install anyway you just wouldn't have noticed it because of no errors. Ref: https://github.com/pypa/pipenv/search?q=subprocess

However I will re-open it for now in case someone else has some insights here for you, but if we don't hear anything by later this year I will recommend close the issue again.

[matteius]

Well with the new optimization to pipenv install we no longer invoke many sub processes: https://dev.to/matteius/pipenv-install-performance-improved-5cdn