Race Condition with Dynamic Import

[ceisenach]

When using liburing inside of a multiprocessing environment (e.g. multiple ray workers ) the dynamic importer has a race condition where multiple workers may simultaneously trigger deletion of files from the cache (https://github.com/YoSTEALTH/Dynamic-Import/blob/master/dynamic_import/importer.py#L77).

Is there any reason not to set cache=False in the dynamic import call in the init (https://github.com/YoSTEALTH/Liburing/blob/master/src/liburing/init.py#L7)? Or is there a better way to use liburing in a multiprocessing setup that doesnt cause a race?

Happy to make the changes or provide more info on the issue if desired.

[YoSTEALTH]

Thanks @ceisenach for the report. Dynamic Import wasn't tested in multiprocessor environment, I should add an lock while scanning.

Cached info is what makes it dynamic since it remember all the .py files and var/func/class names thus load time is dynamically based on whats being called.

Its totally fine to set cache=False this is normally how using import would load .py files.

I actually rewrote this whole library in Zig & cPython while ago, I just haven't had the time to finish and upload it. I wonder if I should just do that, it doesn't use dynamic import, just 1 .so file is created.

[ceisenach]

Thanks for the explanation -- that makes sense! We've found your library to be very useful for reading memmapped tensors (hence the multiprocessing reads) :)

If your plan is to push those changes sometime in not too distant future, I will just wait for that and continue patching the import on my end.

[YoSTEALTH]

I normally work on this library over the winter time, most likely that's when I will do the upgrade.

O, btw Liburing is not really designed to be used in multiprocessing/threads directly. The principle is you use 1 ring + thread as the main event manager and it can control/communicate with multiple rings + threads, this way it can all work asynchronously and you don't need to use internal locks in your application, since you are at any given time only dealing with 1 event at a time. Since Liburing can handle millions of events per second using 1 main ring + thread to manage isn't an issue.