win32com.client race condition on first import

[vsohler]

Context:

win32com.client seems to requires the loading/creation of a cache file dicts.dat located in %USER%/AppData/Local/Temp/gen_py/ If the file does not exist, it is created along with an __init__.py upon first import of win32com.client. However if you just installed pywin32, the file is not there yet and running multiple functions requiring win32com.client imports in parallel processes will potentially fail due to a race condition; some process creating the folder and the dicts.dat and the other trying to read the yet still empty file.

Reproduction steps:

# coding: utf-8

import multiprocessing as mp
import os
import shutil
import traceback

# Path to the gen_py folder as of Python 3.8.8
GEN_PY_PATH = os.path.join(os.getenv("LOCALAPPDATA"), "Temp", "gen_py")
ITERATIONS = 1000


# Target to test parallel import of win32com.client
def importFct(event, queue):
    event.wait()
    try:
        import win32com.client
    except EOFError:
        ret = queue.get()
        ret.append(traceback.format_exc())
        queue.put(ret)


if __name__ == "__main__":
    for i in range(ITERATIONS):
        # Use events to synchronize peocesses
        e = mp.Event()
        # Remove gen_py folder first
        if os.path.exists(GEN_PY_PATH):
            print(f"Run: {i}, removing {GEN_PY_PATH}")
            shutil.rmtree(GEN_PY_PATH)
        assert not os.path.exists(GEN_PY_PATH)

        # Use a queue to store exceptions raised in processes
        ret = []
        queue = mp.Queue()
        queue.put(ret)

        # We start 8 processes
        procList = []
        for i in range(8):
            procList.append(mp.Process(target=importFct, args=(e, queue)))
        for i in range(8):
            procList[i].start()
        # Trigger the event to synchronously launch the target
        e.set()
        # Wait for processes to return
        for i in range(8):
            procList[i].join()

        # Display exception raised
        res = queue.get()
        if len(res):
            raise Exception(res[0])

Software version

• Python version: Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)]
• pywin32 version: 300
• OS version: Windows 10 Version 2004 (Build 19041.867)

[mhammond]

Can you please see if the problem is fixed via https://github.com/mhammond/pywin32/commit/7b2f81b1b4a8fa61520e5da1f7ffda84749121f7?

[vsohler]

Tested with the last artifacts wheel from e0a7d99, the same error still occurs.

[vsohler]

I used a workaround that consists of importing win32com.client before running parallel tasks which need win32com.client, thus initalizing the content of the gen_py folder. Also, modules that use win32com.client.gencache.EnsureModule will encounter the same problem because this function dumps custom module infos into dicts.dat. For example, the xlwings module calls this at import time, thus if you import it in multiple parallel processes, there will again be a race condition on dicts.dat Workaround for anyone in this specific case is to roll back to version 0.20.6