pip
pip copied to clipboard
Add option to bypass http proxy
- Pip version: 10.0.1
- Python version: 2.7.8 & 3.6.3
- Operating system: Windows 7
Description:
I'm trying to install packages from a local (SVN) repository. Since I'm behind a proxy I have configured the proxy in the pip.ini. For local resources pip should bypass the proxy.
Please add an option to specify a list of IP address ranges, domains and domain wildcards to connect directly.
What I've run:
prompt> pip install https://svnsrv.domain.net/svn/MyRepo/trunk/some-package-1.0.0.zip
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out',))': /svn/MyRepo/trunk/some-package-1.0.0.zip
Hey @lordyavin! Thanks for filing this issue.
As a workaround, if you pass --proxy=""
, pip will not use the proxy for that run.
I'm not sure how simple/difficult it would be to provide this support. Do you happen to know how other package managers handle this? That might be a good reference to gain insight from.
Why looking at other package managers? Just look at your frontend to the internet. Every browser is capable to bypass a proxy as I requested.
Because pip is a package manager, not a browser?
Regardless, as far as I know, pip respects the usual http_proxy
, https_proxy
and no_proxy
environment variables, so why not just use those?
This information is new to me. Didn't find that in the manual. I'll give it a try, but I would prefer to configure the no_proxy side by side with the proxy server settings in the ini file.
You're referring to the user guide? We could improve the documentation for this.
Agreed we probably should have docs for this. It's a "standard" feature, probably implemented by requests or one of our dependencies rather than by pip itself, so it's not entirely clear where we'd put it in the pip docs, but it would make sense to do so. (When I say "standard" feature, it's standard on Unix, but not on Windows, so that makes documenting it as a cross-platform feature is even mode relevant).
@pradyunsg , the workaround option didn't work for me to disable proxy
` $ pip install flask-socketio --proxy="" Collecting flask-socketio
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000004313BA8>, 'Connection to <IP_ADDRESS> timed out. (connect timeout=15)')': `
Can we list the currently used proxy in pip?
Because pip is a package manager, not a browser?
Regardless, as far as I know, pip respects the usual
http_proxy
,https_proxy
andno_proxy
environment variables, so why not just use those?
Because real life configurations are much more complex then this. Proxy/No-Proxy can change depending on application context. Also those proxy ENV vars are probably the most heavily used and yet most uncouth defined pseudo-standard in IT. The syntax is not defined in a standard leading to multiple competing / supported syntaxes. One app may use "*.domain" while another one will use ".domain". One App expects domains to be seperated by , another one expects ; as seperator. One allows to ignore subnets by using CIDR style adressing another doesn't support IP ranges at all. Great fun if you try to actually run more then one application on a system.
GitHub issues for hundrets of projects are full of issues around this not defined pseudo standard syntax. Most refer to this documentation when asked for the formal definition of the expected syntax. Note how loosely defined it is. Note also how this is not a standard or something worth to be called definition at all since it leaves so much stuff open to interpretation. It's simply how wget
decided to handle it.
And still people write software around this crap and expect it to work. Sorry: this is designed to break.
In addition command line args should always have higher precedence then ENV var imho since they are more explicit. However: when those ENV vars are set the suggested --proxy=""
option did not work in my tests (Windows platform). Also it seems to be possible to set that param in the pip.ini. It's ignored just like the commandline arg regardless.
IMHO using the http_proxy ENV vars is a nice to have default. But because of the lack of definition and app-level control really every app should at least define a app-specific way to control proxy settings that always should have precedence over the http_proxy system config. At least until we someday will maybe finally have an OS independed, strictly defined proxy setting that can provide those often needed level of control.
In the absence of a --no-proxy
option I can confirm that neither --no-proxy=""
is available. Like any other CLI tools, the is a rule that a CLI option always take precedence over ENV or config-file options.
This does not work with pip, as pip refuses to disable proxy when that is defined using http_proxy env vars.
Some woud wonder why is such a big deal? Try to remember that the HTTP proxy may do SSL rewriting and pip does not respect the system installed certificates.
So that chain of bugs contribute to providing a poor experience.
Because pip is a package manager, not a browser?
Anything that uses HTTP is, in the end, a browser.
Regardless, as far as I know, pip respects the usual
http_proxy
,https_proxy
andno_proxy
environment variables, so why not just use those?
Because your proxy may depend on the index url. For example, I have to use different proxies depending on which URLs I am connecting to. The browser generally handles this through a proxy.pac, that computes dynamically the proxy depending on the URL, but neither pip nor using http_proxy does that.
Is there prior art for other package managers to have a --no-proxy
option? That would be a fairly strong indicator. :)
Is there prior art for other package managers to have a
--no-proxy
option? That would be a fairly strong indicator. :)
Package managers generally have fairly flexible configuration files. Pip is not that smart on that respect.
$ wget --help | grep proxy --no-proxy explicitly turn off proxy --proxy-user=USER set USER as proxy username --proxy-password=PASS set PASS as proxy password
npm has it (can be passed as --option)
https://docs.npmjs.com/misc/config#noproxy
On windows you can also use the following commands to set the proxy to nothing and then use pip without issues: set https_proxy= set http_proxy=
Execute the above two commands. It works for me that way.
Okie; I think it's perfectly reasonable to add a --no-proxy
flag for this case. If someone wants to file a PR adding a --no-proxy
flag, they're welcome to.
Hi @pradyunsg
I wanted to work on this PR. From what I could understand from looking at the code on how to approach it is as follows.
We check if the --proxy
option is set at https://github.com/pypa/pip/blob/master/src/pip/_internal/cli/req_command.py#L118 and set the session.proxies
with the values provided.
I assume that we can do a similar thing with checking if the --noproxy
option is provided, and if it is, set the session.proxies['no_proxy]
to True which will be checked at https://github.com/pypa/pip/blob/master/src/pip/_vendor/requests/sessions.py#L293 and the proxy won't be used in further requests.
I am assuming that's all there is to it, but. feel free to add more to what is needed to be done.
Too bad this seems to have stalled, I could really use that --noproxy option today.
Contributions are always welcomed.
Too bad this seems to have stalled, I could really use that --noproxy option today.
it seems that the no-proxy feature was added but not in style of CLI argument. please refer to https://pip.pypa.io/en/stable/user_guide/#using-a-proxy-server
in my case(windows powershell), i can do:
>$Env:no_proxy = 1
>$Env:no_proxy
1
>pip list -o
Package Version Latest Type
---------- ------- ------ -----
idna 2.10 3.2 wheel
requests 2.25.1 2.26.0 wheel
setuptools 56.0.0 57.4.0 wheel
given that my pip version is 21.1.3 and the issue #9972 , it work fine. considered that the no-proxy is not a usual feature, i think hiding this on environment variable is a elegant design. thanks for everyone's work.
I still can't install any package with PIP
I need help!
PS C:\Users\AXA> pip install pyOpenSSL --proxy=off
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D3D0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3DA60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D670>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D3A0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D79220>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
ERROR: Operation cancelled by user
Traceback (most recent call last):
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\util\connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "c:\users\axa\appdata\local\programs\python\python39\lib\socket.py", line 953, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\axa\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\axa\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\AXA\AppData\Local\Programs\Python\Python39\Scripts\pip.exe\__main__.py", line 7, in <module>
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\main.py", line 71, in main
return command.main(cmd_args)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\base_command.py", line 104, in main
return self._main(args)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\base_command.py", line 221, in _main
self.handle_pip_version_check(options)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\req_command.py", line 147, in handle_pip_version_check
pip_self_version_check(session, options)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\self_outdated_check.py", line 152, in pip_self_version_check
best_candidate = finder.find_best_candidate("pip").best_candidate
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 879, in find_best_candidate
candidates = self.find_all_candidates(project_name)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 824, in find_all_candidates
page_candidates = list(page_candidates_it)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\sources.py", line 134, in page_candidates
yield from self._candidates_from_page(self._link)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 783, in process_project_url
html_page = self._link_collector.fetch_page(project_url)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 512, in fetch_page
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 422, in _get_html_page
resp = _get_html_response(url, session=session)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 120, in _get_html_response
resp = session.get(
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\network\session.py", line 449, in request
return super().request(method, url, *args, **kwargs)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 53, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 353, in connect
conn = self._new_conn()
File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 169, in _new_conn
conn = connection.create_connection(
KeyboardInterrupt
any updates on this request?
The maintainers have said that a PR would be welcome, and no one has filed a PR for this so far.
I've labelled this, to reflect that.
Hi all - I dont mind taking a crack at implementing this. My plan is:
- define
--no-proxy
option incmdoptions.py
- process
--no-proxy
option inreq_command.py
-- from here its not super clear since it seems likepip
is using therequests
library.
-- I plan on settingPipSession(requests.Session).proxies
toproxies={"http": "", "https": ""}
which seems to be the way to bypass proxies per here and here.
Any thoughts? Just go ahead and submit PR and solicit feedback?
Any thoughts? Just go ahead and submit PR and solicit feedback?
I would carefully read the documentation, especially the warnings, and what the resquest devs have said is the supported way to do this, rather than relying on stack overflow answers:
- https://requests.readthedocs.io/en/latest/user/advanced/#proxies
- https://github.com/psf/requests/issues/6153#issuecomment-1148827852
- https://github.com/psf/requests/issues/6153#issuecomment-1158017813
- https://requests.readthedocs.io/en/latest/api/#requests.Session.trust_env
I suspect any PR would face significant pushback if it's using an undocumented way to achieve this behavior.
@notatallshaw - thanks!
Yes, I agree on not relying on stack overflow answers. I'll take a look at your pointers and then generate a PR for review. I ran into the same issues you link when researching this issue.
Specifically, I need to be ensure any http_proxy
and https_proxy
enviromental variables are not used with --no-proxy
option. Looks like its not clear how this can be accomplished.
The request
docs are a bit confusing on this issues.
OK, I'll dig around some. Thanks again.
I've submitted a PR for this - https://github.com/pypa/pip/pull/12011 . Not sure what the process is for getting a review.