pipenv tests failing using requirementslib when offline

[jayvdb]

The following five pipenv v2018.11.26 tests pass when offline using requirementslib v1.3.3, but try to use the internet in requirementslib v1.4.2

• tests/unit/test_utils.py::test_convert_deps_to_pip[deps7-git+git://github.com/pinax/[email protected]#egg=pinax]
• tests/unit/test_utils.py::test_convert_deps_to_pip[deps8-hg+http://hg.myproject.org/MyProject@da39a3ee5e6b#egg=MyProject]
• tests/unit/test_utils.py::test_convert_deps_to_pip[deps10-https://github.com/Rapptz/discord.py/archive/rewrite.zip#egg=discord.py[voice]]
• tests/unit/test_utils.py::test_convert_deps_to_pip[deps11-git+https://github.com/requests/requests.git@master#egg=requests[security]]
• tests/unit/test_utils.py::test_convert_deps_to_pip_one_way[deps3-git+https://github.com/requests/requests.git@master#egg=requests[security]]

It wouldnt surprise me if it was necessary to fetch from the repos in order to resolve those URLs accurately, at least some of them, but I am hoping the folks here can confirm that this internet access is now absolutely necessary for all of them. It wasnt necessary in pipenv v2018.11.26 which vendored requirementslib v1.3.3.

This underscores the need for https://github.com/sarugaku/requirementslib/issues/145 , so that changes to the 'online-ness' of any function is clearly understood before merging, as such changes can have significant effects on users of the library if they were not expecting requirementslib to use the network for what was previously an offline invocation.

[jayvdb]

It also seems strange that the same requirement with "editable": True doesnt need the internet, and that svn doesnt fail.

[techalchemy]

@jayvdb pipenv doesn't currently actually have the released version of requirementslib vendored on master, it has a dev release of it. I would suggest trying the master branch (not saying it would work but I'm not sure)

[techalchemy]

My goal is to delay the network call as long as possible so we should be able to hold off on the network call until we need to do dependency resolution, which is slow and should be delayed as long as possible

[jayvdb]

pipenv doesn't currently actually have the released version of requirementslib vendored on master

Been there .. done that .. makes no difference. The problems arose after v1.3.3 , and havent been fixed since. The changes here were good changes, which improved the resolution, but I suspect they were over-eager in resorting to using the internet, and need to be slightly restructured so they dont use the internet when not necessary, and the caller needs to be more explicit about how much information it needs, so that internet hits can be skipped (which may be very expensive) when the caller doesnt need the extra info (and may still not find the missing extra info in the online resource).

[jayvdb]

In a very freaky accident, I chanced upon some very similar errors in pipenv CI.

I removed unused vendored library blindspin, and running the CI shows the first four failures mentioned in this issue.

https://github.com/pypa/pipenv/pull/3675

Errors shown at https://dev.azure.com/pypa/pipenv/_build/results?buildId=6410 . The logs are different from the ones I was encountering without internet, but it seems to be a very closely related problem to what I was seeing.

The same occurs when removing shutilwhich.

https://github.com/pypa/pipenv/pull/3676

I cant imagine how blindspin and shutilwhich are related to these failures.

[jayvdb]

And I havent been able to reproduce it locally when removing blindspin or shutilwhich. Very curious.

[jayvdb]

And I rebased https://github.com/pypa/pipenv/pull/3670 and suddenly these errors appear again.

[jayvdb]

At https://github.com/pypa/pipenv/pull/3679 @frostming noted that the pipenv tests are also breaking.

[techalchemy]

hrm I haven't merged this code in recently but I did notice that it is once again making network calls which is why I am sitting on the current cleanup PR

[jayvdb]

Could it be the new setuptools, released yesterday? I notice that notpip has a vendored pkg_resources, but not setuptools, so maybe there is some inconsistency now causing this failure.

[techalchemy]

It’s possible but requirementslib doesn’t actually use setuptools in any way that would be affected. I think this is actually my fault and I am making something less lazy somewhere when I instantiate it

[matteius]

The test runner current checks if you have internet available, and adjusts which tests cases run accordingly. Except for these failing:

SKIPPED [16] tests/conftest.py:86: requires internet access, skipping...
FAILED tests/unit/test_requirements.py::test_convert_from_pipfile[requirement10-https://github.com/Rapptz/discord.py/archive/async.zip#egg=discord.py[voice]] - pip._vendor.requests.exceptions.Connecti...
FAILED tests/unit/test_requirements.py::test_get_requirements - AttributeError: 'NoneType' object has no attribute 'url'
FAILED tests/unit/test_setup_info.py::test_local_req[test_artifact0] - AttributeError: 'NoneType' object has no attribute 'keys'
FAILED tests/unit/test_setup_info.py::test_local_req[test_artifact1] - AttributeError: 'NoneType' object has no attribute 'replace'
FAILED tests/unit/test_setup_info.py::test_no_duplicate_egg_info - AssertionError: assert (None or None or None)
FAILED tests/unit/test_setup_info.py::test_parse_function_call_as_name - AssertionError: assert None == 'package-with-function-call-as-name'
FAILED tests/unit/test_setup_info.py::test_ast_parser_handles_repeated_assignments - KeyError: 'name'
FAILED tests/unit/test_setup_info.py::test_ast_parser_handles_exceptions - KeyError: 'requires'
FAILED tests/unit/test_setup_info.py::test_read_requirements_with_list_comp - KeyError: 'requires'

but I think we can track that with: https://github.com/sarugaku/requirementslib/issues/145