[Discussion] Should we implement an interface for loading python files for models?

[lllyasviel]

Since many users may want to add their own preprocessors, perhaps we can just trigger their .py files in model folder

for example, if I train a

control_XXX_XXX_XXX.pth

and I have a

control_XXX_XXX_XXX.yaml

perhaps i can just write a

control_XXX_XXX_XXX.py

and put it in model folder

and controlnet will just run those py files and people do whatever they want.

This may not be safe enough - but considering that in A1111 everyone is just using those unsafe codes from all those unsafe plugins and scripts, perhaps just let user to decide whether to trust code providers.

[ljleb]

Good idea. Maybe prepend literally "(user) " to the preprocessor alias name to make it clear in the ui when a preprocessor is user-defined.

I suggest these functions for user-defined preprocessors:

• process_image(img: np.ndarray, res, thr_a, thr_b) -> Tuple[np.ndarray | ???, bool] : process image function, runs the custom preprocessor. return a tuple of array and bool. the bool is true if the result is an np.ndarray and otherwise I am not sure... seems to be a torch tensor in controlnet.py currently?
• detail() -> List[dict] : return the properties of the preprocessor. it is the same type of list as the elements in processor.preprocessor_sliders_config
• install(launch) : executed on start-up. Can be used to install dependencies. launch is the launch module from the webui, which can be used to invoke pip.

These function names are just for illustrative purposes. As of right now our code directly passes res, thr_a and thr_b to preprocessor callbacks, we need to make a couple changes in controlnet.py to support arbitrary parameters.

Do you see any issues with this design? Something we might encounter is that with this, we may need to load user-defined scripts twice: once for the install function, and once again to load the other 2. There may be a way to let users specify additional pip dependencies instead inside a different file like control_XXX_XXX_XXX_requirements.txt?

The models folder may start to be a little bit cluttered, I suggest we put these files in their own directory, i.e. <controlnet-directory>/extensions. Alternatively, we could convert the code of our own annotators into the specified code api as a proof of concept. If we are using it, there is way less chance that some key features are missing or that it does not work well for other users.

Another idea is to provide a way to add a new preprocessor through the external_code api. Then, extensions can use importlib to import controlnet and then register their preprocessor. This would allow people to phrase preprocessors code as a dedicated webui extension, or to include these preprocessors inside existing extensions when it is appropriate to do so.