Mikubill
[Bug]: module file conflict
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
What happened?
i've been fumbling around this, but i cant find solution. problem is if two different extensions have exactly the same internal folders/files.
for example, both controlnet and kohya additional networks perform same import
from scripts import xyz_grid_support
those are different files that are local to each extension. but python loader in its wisdom decides that module is already loaded so what ends up is that whatever extension loads second will try to use module from first loaded extension - and that will of course fail.
i'm really out of ideas.
i'd love to solve this in core, but for the moment easy solution would be to have both extensions have unique module names - can you do that?
anyone with ideas?
link to issue https://github.com/vladmandic/automatic/issues/297
Steps to reproduce the problem
- Go to ....
- Press ....
- ...
What should have happened?
multiple extensions should not create module namespace conflict.
Commit where the problem happens
N/A
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
N/A
Console logs
Module load: /home/vlado/dev/automatic/extensions/sd-webui-additional-networks/scripts/additional_networks.py: AttributeError
Traceback (most recent call last) │ /home/vlado/dev/automatic/modules/script_loading.py:10 in load_module
│ 9 │ try: │ ❱ 10 │ │ module_spec.loader.exec_module(module) │ 11 │ except Exception as e: │ in exec_module:883 │ in _call_with_frames_removed:241 │
│ /home/vlado/dev/automatic/extensions/sd-webui-additional-networks/scripts/additional_networks.py:394 in <module> │
│ │ 393 │ ❱ 394 xyz_grid_support.initialize(Script) │ 395 AttributeError: module 'scripts.xyz_grid_support' has no attribute 'initialize'
when adding `print(xyz_grid_support.__file__)`, you can clearly see its the wrong module:
> /home/vlado/dev/automatic/extensions-builtin/sd-webui-controlnet/scripts/xyz_grid_support.py
Additional information
No response
This and #940 are caused by our module paths not being unique. We should prefix the names with lib_controlnet_ or something like that.
Yes, by manipulating sys.path I presume. I am not sure if this could break something else unexpectedly again though,and if it can actually work. Going with the unique module names is what most extensions have been doing AFAIK. For example see unprompted's source: https://github.com/ThereforeGames/unprompted/tree/main/lib_unprompted
It's a bit unfortunate. It feels like every time I commit something, something else unexpectedly breaks. In this case the underlying issue is not even something I wrote code for. The pointer was already null, I just happened to dereference it.
Yes sys.path looks for imports in the order the paths are listed. i.e. we could try reordering. Again, I am not convinced this is the way to go and if it can actually fix the problem.
wait, if all extensions share the same name "scripts" then how those people use "from scripts import X"
why I can use "from scripts import api" without read other extensions' api?
I don't know, is it related to the values in sys.path? Put a breakpoint or print its value when loading controlnet.py.
Are there extensions that import the annotator module? I was under the impression most of the dependencies should be inside scripts.
for example when we
from scripts import global_state, hook, external_code, processor, xyz_grid_support
why this "scripts" is correct?
why it is not other extensions' "scripts". note that all extensions will have "scripts" folder
Look at the body of the issue here. It actually has the same rules as outside scripts. i.e. we cannot import xyz_grid_support.
but if we use something like lib_controlnet_script then A1111 cannot read this. is this problem unsolveable?
No we cannot rename scripts. We can try renaming conflicting files inside of it instead.
Maybe for now, don't import and reload annotator_path and xyz_grid_support inside controlnet.py. If we don't import them, we should not get these error like before I presume.
Unfortunately this means having users reatart the webui from the ground up every time we change these files.
We only started getting reports of annotator being a problematic name when importing it from controlnet.py. I meant to refer to the imports inside controlnet.py.
what about I remove the importlib.reload(annotator_path) and importlib.reload(xyz_grid_support) for the time now and we consider these later
how you think about this? https://github.com/Mikubill/sd-webui-controlnet/pull/951 @ljleb
Yes this should resolve this issue. Now we have to make sure to never import these modules directly from controlnet.py, unless we rename them.
In fact, a possible solution is to replace annotator and xyz_grid_support with forwarding modules:
xyz_grid_support.py:
from lib_controlnet.xyz_grid_support import *
And then just copy the current xyz_grid_support.py file to lib_controlnet, adjusting imports as necessary.
Then, in controlnet.py we can do:
from lib_controlnet import xyz_grid_support
importlib.reload(xyz_grid_support)
annotator has super complicated self import in those controlnet's mmcv and controlnet's detectron2
it cannot be changed, perhaps ask users to uninstall any package named annotator
We can otherwise move the models_path variable to global_state instead. This would allow the settings to be changed on ui reload instead of complete restart.