idandaniel
Unable to work inside async functions
Describe the bug Can't run properly in async functions.
To Reproduce Steps to reproduce the behavior:
- Create an async function which would try to fetch proxies via ballyregan.
- Call the async function.
- See error.
Expected behavior Normal operation, as with synchronous calling.
Screenshots
Desktop (please complete the following information):
- OS: N/A
- Browser N/A
- Version N/A
Smartphone (please complete the following information):
- Device: Poco F3
- OS: Android 12
- Browser N/A
- Version N/A
Additional context The library seems to work fine with simple synchronous calls, but it's always spitting out an error whenever I'm trying to use it in async functions.
Here's (one of the) the errors I get:
Traceback (most recent call last): File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/storage/emulated/0/tmp/apihost-tts-proxy.py", line 37, in handle_request fetch_proxies() File "/storage/emulated/0/tmp/apihost-tts-proxy.py", line 18, in fetch_proxies raw_proxies = fetcher.get( ^^^^^^^^^^^^ File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/ballyregan/fetcher.py", line 134, in get proxies = self._gather( ^^^^^^^^^^^^^ File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/ballyregan/fetcher.py", line 95, in _gather valid_proxies = self._proxy_validator.filter_valid_proxies( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/ballyregan/validator.py", line 129, in filter_valid_proxies return self.loop.run_until_complete( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 629, in run_until_complete self._check_running() File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 590, in _check_running raise RuntimeError( RuntimeError: Cannot run the event loop while another loop is running /data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py:1907: RuntimeWarning: coroutine 'ProxyValidator._async_filter_valid_proxies' was never awaited handle = self._ready.popleft() RuntimeWarning: Enable tracemalloc to get the object allocation traceback
I even tried using it as normal synchronous, blocking function, but it still throws an error.
Here's ~~my~~ gpt-4's code (lol) I was trying to get working with it:
import random
from aiohttp import web, ClientSession
from ballyregan import ProxyFetcher
from ballyregan.models import Protocols
# A list to hold our proxies and number of uses
proxies = []
MAX_USES = 10 # Maximum number of uses per proxy
REFETCH_NO = 20 # Number of proxies you want to fetch initially and when the list is exhausted
# Prepare the proxy fetcher
fetcher = ProxyFetcher(debug=True)
# Fetch and format proxies
def fetch_proxies():
proxies.clear()
raw_proxies = fetcher.get(
limit=REFETCH_NO,
protocols=[Protocols.HTTPS, Protocols.SOCKS4, Protocols.SOCKS5]
)
for p in raw_proxies:
proxies.append({
"proxy": f'{p.ip}:{p.port}',
"type": {'https': 'HTTPS', 'socks4': 'SOCKS4', 'socks5': 'SOCKS5'}.get(p.protocol.lower()),
"count": 0,
})
print(f'Fetched {len(proxies)} proxies. Proxy list: {proxies}')
async def handle_request(request):
# Prepare headers for forward request
headers = {k: v for k, v in request.headers.items()}
url = 'https://hidden.due/to/privacy/reasons'
while True:
if not proxies: # If no proxies left
fetch_proxies()
cur_proxy = random.choice(proxies)
count = cur_proxy["count"]
if count >= MAX_USES: # If proxy use count exceed the maximum use limit
proxies.remove(cur_proxy) # If so, remove the proxy from the list
else:
try:
cur_proxy["count"] += 1
async with ClientSession() as session:
# Correctly format the proxy URL depending on the type
proxy_url = {'HTTP': 'http://', 'HTTPS': 'https://', 'SOCKS4': 'socks4://', 'SOCKS5': 'socks5://'}.get(cur_proxy['type'], 'http://') + cur_proxy["proxy"]
print(f'Proxy url is {proxy_url}')
async with session.request(
request.method,
url,
headers=headers,
allow_redirects=False,
data=await request.read(),
proxy=proxy_url
) as resp:
resp_headers = {k: v for k, v in resp.headers.items()}
body = await resp.text()
break
except:
proxies.remove(cur_proxy) # If the proxy is not working, remove it from the list
return web.Response(
headers=resp_headers,
status=resp.status,
body=body
)
app = web.Application()
app.router.add_route('*', '/{tail:.*}', handle_request)
fetch_proxies() # Fetch proxies initially
try:
web.run_app(app, port=1337)
except KeyboardInterrupt:
print("KeyboardInterrupt received, exiting...")
As expected, it works for me at the start, but throws an error every time the script tries to fetch new proxies afterwards.
Hi, thank you for opening an issue! Will investigate it and let you know.
tl;dr Create the ProxyFetcher instance inside an async scope for now, I'll fix it in the next few days.
Explanation: The issue is caused by the fact that 'ProxyValidator' initiates it's event loop once in the class init. That means that when you create the 'ProxyFetcher' instance outside an async scope, the 'ProxtValidator' does not recognize any existing event loop and creates one. Then when you run in inside an async scope there will be 2 event loops running, which is not allowed.