ish icon indicating copy to clipboard operation
ish copied to clipboard

Python hangs fetching data with urllib in parallel

Open Rukario opened this issue 3 years ago • 3 comments

import time
from urllib import request
from queue import Queue
from threading import Thread

slots = 4

def dl(q):
    while True:
        url = q.get()
        if isinstance(url, int):
            time.sleep(1)
            print(url)
        else:
            print("Fetching...")
            print(f"{url} size is: {len(request.urlopen(request.Request(url)).read())}")
            print("SUCCESS")
        q.task_done()

q = Queue()
for i in range(slots):
    t = Thread(target=dl, args=(q,))
    t.daemon = True
    t.start()

while True:
    for n in range(4):
        if n == 0:
            q.put("https://example.com/")
        elif n == 2:
            q.put("https://example.net/")
        else:
            q.put(n)
    q.join()
    input("Enter to continue")

There are data fetching and 1s timers arranged in odd/even pattern. Success with slots = 1 (no parallel), success with slots = 2 if data fetch can complete before 1s timer.

Resorting to Requests same result.

Rukario avatar Feb 25 '22 08:02 Rukario

Does anyone have any insight on this? If I recall correctly, it used to work on iOS 14.8 but not any more after I upgraded to one of the iOS 15.4 betas and it still does not work on iOS 15.4 RC that's released just a few hours ago. Don't really want to reinstall to find out but I might.

Rukario avatar Mar 08 '22 21:03 Rukario

It appears the TestFlight version of this app has been having success fetching data with urllib in parallel. Not sure what was the fix but I hope it won't be too long to see an update for the App Store version!

Rukario avatar Mar 09 '22 08:03 Rukario

The App Store version of this app is still having this problem running under iOS 16 (public beta 3).

It's been a while and the problem remains. I've re-opened this issue for visibility. To put it simply, the TestFlight version of this app has had it resolved, it's just that the App Store version is in need of an update.

Rukario avatar Aug 12 '22 00:08 Rukario