microsoft
This SDK isn't thread-safe (or even multi-process safe) because of OPTIONS cache
This SDK is crashing some of my build agents occasionally with this error:
[Errno 13] Permission denied: '/Users/USER_NAME/.azure-devops/python-sdk/cache/options.json'
Based on reading the code, it appears to be coming from here: https://github.com/microsoft/azure-devops-python-api/blob/dev/azure-devops/azure/devops/_file_cache.py
This code doesn't appear to be at all thread, or even process, safe, as it just asserts handles to the file. If it can't be made thread/process safe, some option to just not use the cache at all, and just fetch it directly when needed for the connection would work. As is now, I've had to implement a machine level mutex and lock all my calls to this API behind it, which is horribly inefficient, but I can't have my builds crashing with random IO errors.
The cache is also a problem with read-only file systems like serverless functions, can we please add a no-cache option
I was finally able to work around it with some incredily hacky code, just in case anyone elses servers are exploding with problems around this like mine were
from azure.devops._file_cache import OPTIONS_CACHE, RESOURCE_CACHE
def no_op():
pass
OPTIONS_CACHE.load = no_op
OPTIONS_CACHE.save = no_op
OPTIONS_CACHE.clear = no_op
RESOURCE_CACHE.load = no_op
RESOURCE_CACHE.save = no_op
RESOURCE_CACHE.clear = no_op
The "file_cache" is in fact an in memory cache that only reads from a "file" at startup (but writes to the file on every set call), so was useless to my program anyway (it tend to run for days at a time, and the cache is only good for 12 hours). But the in memory stuff works even without the load/save/clear methods.
It's pretty messy, and you need to do it really early in the program (before importing anything else in the azure.devops package).
The OPTIONS_CACHE and RESOURCE_CACHE are module level variables that many other modules will pull in as part of importing, so it has to happen really early to avoid them interacting with the harddrive before you mess around with them.
I could workaround the read-only file systems in serverless functions by setting an environment variable AZURE_DEVOPS_CACHE_DIR pointing to /tmp folder (e.g. AZURE_DEVOPS_CACHE_DIR=/tmp/.azure-devops-cache). It worked fine.
Does it need to be a different file for each process? My problem was locking, so even if I pointed it to a different file, it still failed, because all the processes attempt to mess with the same file at the same time, causing some of them to fail when one of the other processes had the lock.