conda
Minor: flaky test_concurrent_cached_download_file
On a few occasions I have noticed test_concurrent_cached_download_file failing with the following error:
> assert (
thread_names_calling_requests_get.qsize()
== 1
== len(set(thread_names_calling_requests_get.queue))
== mock_get.call_count
), f"{thread_names_calling_requests_get.queue=}"
E AssertionError: thread_names_calling_requests_get.queue=deque(['CachedDownloadFileThread-1', 'CachedDownloadFileThread-0'])
E assert 2 == 1
E + where 2 = qsize()
E + where qsize = <queue.Queue object at 0x10babbef0>.qsize
tests/test_lookup_cache.py:308: AssertionError
I want to start collecting occurrences of this failure here so that I can, for example, see if the issue is operating-system specific.
Failures:
- MacOS: https://github.com/conda/conda-lock/actions/runs/14116344346/job/39547362653
What's happening with this test failure is that I have a FileLock intended to ensure that only one concurrent process (for testing I'm using threads instead of processes) downloads the lookup table. Somehow the lock is failing and two threads download the lookup table.
This is not particularly serious, because it works most of the time, and when it fails it only leads to an extra unnecessary download.
What's especially tricky is that this is an occasional flake so I don't have a reliable reproducer.
Failed attempt
I was thinking that maybe I need to set thread_local=False to make the FileLock global across threads. However, I now doubt that this is the correct fix:
As far as I can tell, what thread_local=False does is to make it so that, if the same FileLock instance (as a Python object) is shared between multiple threads, then each thread can simultaneously acquire the lock. If each thread creates its own FileLock instance then I don't think thread_local has any effect.
Based on my belief that thread_local has no effect, I don't think it's justified to introduce it.
Here's a script to demonstrate the thread_local behavior:
import threading
import time
from filelock import FileLock
def worker(name: str, lock: FileLock):
print(f"Thread {name} attempting to acquire the lock.")
with lock:
print(f"Thread {name} acquired the lock!")
time.sleep(2)
print(f"Thread {name} releasing the lock.")
print(f"Thread {name} finished.")
if __name__ == "__main__":
for thread_local in [True, False]:
for same_lock_object in [True, False]:
lock_a = FileLock("test.lock", thread_local=thread_local)
if same_lock_object:
lock_b = lock_a
else:
lock_b = FileLock("test.lock", thread_local=thread_local)
print(f"\nUsing thread_local={thread_local} and same_lock_object={same_lock_object}")
t1 = threading.Thread(target=worker, args=("A", lock_a))
t2 = threading.Thread(target=worker, args=("B", lock_b))
t1.start()
time.sleep(0.5) # Stagger to see the effect
t2.start()
t1.join()
t2.join()
The lock is only acquired concurrently when both thread_local=False and same_lock_object=True:
$ python script.py
Using thread_local=True and same_lock_object=True
Thread A attempting to acquire the lock.
Thread A acquired the lock!
Thread B attempting to acquire the lock.
Thread A releasing the lock.
Thread A finished.
Thread B acquired the lock!
Thread B releasing the lock.
Thread B finished.
Using thread_local=True and same_lock_object=False
Thread A attempting to acquire the lock.
Thread A acquired the lock!
Thread B attempting to acquire the lock.
Thread A releasing the lock.
Thread A finished.
Thread B acquired the lock!
Thread B releasing the lock.
Thread B finished.
Using thread_local=False and same_lock_object=True
Thread A attempting to acquire the lock.
Thread A acquired the lock!
Thread B attempting to acquire the lock.
Thread B acquired the lock!
Thread A releasing the lock.
Thread A finished.
Thread B releasing the lock.
Thread B finished.
Using thread_local=False and same_lock_object=False
Thread A attempting to acquire the lock.
Thread A acquired the lock!
Thread B attempting to acquire the lock.
Thread A releasing the lock.
Thread A finished.
Thread B acquired the lock!
Thread B releasing the lock.
Thread B finished.
