Setting http_proxy breaks HTTPretty

[gregnavis]

Save the code below in proxy_test.py:

import httpretty

import requests


httpretty.enable()
httpretty.register_uri(httpretty.GET,
                       'http://www.github.com',
                       body='It works!')
print requests.get('http://www.github.com').text

The code works when run without http_proxy set but breaks when that variable is set. In the later case a real HTTP request is sent via the proxy.

[deontologician]

I just ran into this problem as well. Looks like requests does the proxy handling so it might be hard to work around it at the socket level.

Maybe httpretty could detect if a proxy is being used and rewrite requests to the proxy url to the url given in the http metadata?

[ncjones]

I also encountered this problem and found that I can workaround it by ignoring the host name in the registered URL. For example, change the full URL:

httpretty.register_uri(httpretty.GET, 'http://myhost/path/to/resource/')

to the following regular expression:

httpretty.register_uri(httpretty.GET, re.compile('.*/path/to/resource/'))

[gabrielfalcao]

Interesting, looks like a nice bug to kill, would you give me example code of how to reproduce?

[ncjones]

Given the following simple unit test:

from unittest import TestCase
import httpretty
import requests

class HttprettyTest(TestCase):
    @httpretty.activate
    def test_httpretty_intercepts_request(self):
        httpretty.register_uri(httpretty.GET, "http://google.com/humans.txt")
        response = requests.get("http://google.com/humans.txt")
        self.assertEqual("HTTPretty :)", response.text)

we can see that it passes when the http proxy environment variable is not set:

$ export http_proxy=
$ nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.103s

OK

but it fails when the http proxy environment variable is set to a valid proxy server:

$ export http_proxy=http://localhost:8888
$ nosetests
F
======================================================================
FAIL: test_httpretty_intercepts_request (proxy_test.HttprettyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nathan/.virtualenvs/httpretty_test/local/lib/python2.7/site-packages/httpretty/core.py", line 1006, in wrapper
    return test(*args, **kw)
  File "/home/nathan/Development/Code/httpretty_test/proxy_test.py", line 11, in test_httpretty_intercepts_request
    self.assertEqual("HTTPretty :)", response.text)
AssertionError: 'HTTPretty :)' != u"Google is built by a large team of engineers, designers, researchers, robots, and others in many different sites across the globe. It is updated continuously, and built with more tools and technologies than we can shake a stick at. If you'd like to help us out, see google.com/jobs.\n"
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): localhost
requests.packages.urllib3.connectionpool: DEBUG: "GET http://google.com/humans.txt HTTP/1.1" 301 229
requests.packages.urllib3.connectionpool: INFO: Resetting dropped connection: localhost
requests.packages.urllib3.connectionpool: DEBUG: "GET http://www.google.com/humans.txt HTTP/1.1" 200 None
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.345s

FAILED (failures=1)

The test has failed because the real HTTP request has completed successfully instead of being intercepted by httpretty.

Note that this example requires a local proxy server running on port 8888. I am using OWASP ZAP as an HTTP debugging proxy.

[demianbrecht]

I just ran into a similar issue. It /seems/ like issues related to proxies can be encountered with at least a couple different paths:

1. Hostnames This can happen when an HTTP proxy is used. Connections are established to the proxy rather than the intended host. This can be resolved by either using the proxy hostname when registering a mocked response or by using regex with wildcards for the hostname as found above.
2. HTTP CONNECT This one's a little more finicky: When the SSL tunnel is established, httplib will send a CONNECT request over multiple calls to sendall(): https://hg.python.org/cpython/file/2.7/Lib/httplib.py#l804. Because there is no body present (and in fact, only a single CRLF), it fails during unpacking here: https://github.com/gabrielfalcao/HTTPretty/blob/eb3b7edfbfb30e809fc3aa92318b311c0e20c42d/httpretty/core.py#L417. The data in this case looks like this: 'CONNECT example.com:443 HTTP/1.0\r\n'

Unfortunately, the only not-so-ugly workaround that I can think of offhand to get around this is to simply not use proxies when tunneling. An ugly workaround might be to mock out httplib's HTTPConnect.sendall, intercepting CONNECT calls and appending an additional CRLF to them.

[gregnavis]

Have we made any progress on the issue? If not, we could at least make the docs mention that proxies should not be used. What do you think?

[amites]

A work around is to disable proxy within the process

import os

os.environ['http_proxy'] = ''

os.environ is isolated to the current process so it will not affect the external system

from what I found when http_proxy is set then the override is not applied correctly, even the built in functional tests start to fail

happy to share more detailed notes with anyone that wants to tackle this -- most of my investigation revolved around boto but much will probably still apply