connector.py file doesn't correctly set CONNECT method

[rookiemann]

Long story short

My problem is I really want to use aiohttp with asyncio to get Bing search engine results. My biggest problem is I've been trying to figure this out for days now. I've looked over many, MANY different discussions on the topic here (mostly here) and elsewhere.

Expected behaviour

The expected behavior is - using publicly available proxies (which can be many different type like Socks and regular CONNECT proxies etc. I do have a internal proxy server (http://127.0.0.1:8080) I'm using now to test, it gets public proxies, tests them and sets it on the server proxy. This server works fine.

I think for just a moment this morning I had it, I was testing against whatsmyip.org and I believe I did see I was 'broadcasting' from a different IP, but just like that I can't get it to work. I don't know what's going on. I am a novice, that is a problem as well too.

Actual behaviour

At first, I tried connecting using both http and https methods I was using for just regular 'request' method ... which was working out for me. When I wanted to switch to asyncio/aiohttp method I see I can't use https and need to use CONNECT method to tunnel in (I guess on :443?) using the ProxyConnector and setting up the proxy in the ClientSession.

Steps to reproduce

So, I'm using this guys code here which I like and it works great without proxies. I would like to use public proxies with this code here: https://fadeevab.com/18-lines-of-the-powerful-request-generator-with-python-asyncio-aiohttp/

import aiohttp
import asyncio

MAXREQ = 500
MAXTHREAD = 50
URL = 'https://google.com'

g_thread_limit = asyncio.Semaphore(MAXTHREAD)


async def worker(session):
    async with session.get(URL) as response:
        await response.read()


async def run(worker, *argv):
    async with g_thread_limit:
        await worker(*argv)


async def main():
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[run(worker, session) for _ in range(MAXREQ)])


if __name__ == '__main__':
    # Don't use asyncio.run() - it produces a lot of errors on exit.
    asyncio.get_event_loop().run_until_complete(main())

OK, so with this code above works fine, I need to connect public proxies to it.

Your environment

This here is the code I'm trying, it doesn't connect and I get back my own IP address each time. This is the code I was using that seems to have gotten me the closest to what I'm trying to accomplish ... but it's still telling me MY IP and not an IP from my proxy server. I have to put the code conn = None to get going, that seems sloppy to me though, not sure.

import aiohttp
import asyncio
import re
from bs4 import BeautifulSoup
from aiohttp_proxy import ProxyConnector, ProxyType

MAXREQ = 1
MAXTHREAD = 1
URL = 'https://www.whatismybrowser.com/detect/ip-address-location'

g_thread_limit = asyncio.Semaphore(MAXTHREAD)
conn = None

async def worker(session):
    conn == ProxyConnector("http://127.0.0.1:8080")

    async with session.get(URL) as response:
        
        await response.read()
        response = await response.read()
        soup = BeautifulSoup(response, "html.parser")
        print(soup)


async def run(worker, *argv):
    async with g_thread_limit:
        await worker(*argv)


async def main():
    async with aiohttp.ClientSession(connector=conn) as session:
        await asyncio.gather(*[run(worker, session) for _ in range(MAXREQ)])



if __name__ == '__main__':
    # Don't use asyncio.run() - it produces a lot of errors on exit.
    asyncio.get_event_loop().run_until_complete(main())

Sincerest apologies!

It took me days to reach out for help, I know I needed to really put some work in first before contacting, 3 days was enough frustration to bring me here. I hope I'm not irratating anyone with my noobness, I just think aiohttp is fantastic and want to use it.

Thanks for any help, it is really REALLY appreciated.