BUG/QUESTION: Problem with order of operation of batch_hijack

[marcsyp]

Hi all,

As part of a personal project involving an AlwaysOn script I am developing, a few months ago I made some edits to ControlNet to add a "textinfo" field to the ControlNetUnit class, so that I can access the geninfo of the files pulled by the controlnet units. My previous setup was the following:

ControlNet 1.1.415 (I think) ControlNet Prompter (my extension) Dynamic Prompts

The naming is intentional so that the load order would be CN, CNP, and then DP. It was set up this way because in my previous installation, the batch_hijack of CN would always run first and process the batches, adding the image to the p object for each process run. However, because choose_input_image function delivers an numpy ndarray image (which has no metadata), I was unable to extract metadata from the batch images even though my extension's code executed after the CN batch hijack process. So I simply added some code to CN's batch hijack to grab the metadata using the unit.image filename and store it in a new property on the unit:

`@staticmethod def choose_input_image( p: processing.StableDiffusionProcessing, unit: external_code.ControlNetUnit, idx: int ) -> Tuple[np.ndarray, bool]: """ Choose input image from following sources with descending priority: - p.image_control: [Deprecated] Lagacy way to pass image to controlnet. - p.control_net_input_image: [Deprecated] Lagacy way to pass image to controlnet. - unit.image: - ControlNet tab input image. - Input image from API call. - p.init_images: A1111 img2img tab input image.

    Returns:
        - The input image in ndarray form.
        - Whether input image is from A1111.
    """
    image_from_a1111 = False

    p_input_image = Script.get_remote_call(p, "control_net_input_image", None, idx)
    image = image_dict_from_any(unit.image)

    if batch_hijack.instance.is_batch and getattr(p, "image_control", None) is not None:
        logger.warning("Warn: Using legacy field 'p.image_control'.")
        input_image = HWC3(np.asarray(p.image_control))
    elif p_input_image is not None:
        logger.warning("Warn: Using legacy field 'p.controlnet_input_image'")
        if isinstance(p_input_image, dict) and "mask" in p_input_image and "image" in p_input_image:
            color = HWC3(np.asarray(p_input_image['image']))
            alpha = np.asarray(p_input_image['mask'])[..., None]
            input_image = np.concatenate([color, alpha], axis=2)
        else:
            input_image = HWC3(np.asarray(p_input_image))
    elif image is not None:
    ##################################################################
        # Set textinfo from file
        try:
            image_object = Image.open(unit.image)
            textinfo, _ = read_info_from_image(image_object)
            unit.textinfo = textinfo    
        except Exception:
            pass
        
        #################################################################
        while len(image['mask'].shape) < 3:
            image['mask'] = image['mask'][..., np.newaxis]`

After some problems with my installation, I reverted my plugins and lost the code that I wrote for the modifications to CN. I tried to recreate them (as shown above) but behavior is very different this time around. I believe that everything is roughly the same (inlcuding the naming and loading order of the extensions, which I have double and triple checked with the debugger and logging messages), but I'm getting different execution order now.

Previously, the order was this: CN batch_hijack.py functions CN controlnet.py choose_input_image() CNP process()

Now I am getting this: CN batch_hijack.py functions CNP process() CN controlnet.py choose_input_image()

That means that when my script process fires for each batch, the unit object does not have my textinfo field populated but it does have the unit.image as the source path of the image, so I can access the file and extract the metadata directly from the file in my extension (and would not require any updates to CN code). So, this seems like a good thing for me -- but I am baffled. When/why/how did the order of execution change? Is this a fluke or something particular about my setup? Any insight?

Thanks, Marc

[huchenlei]

I think the callstack is always: CN's Script.process calls choose_input_image. See following console logs:

2023-12-17 20:40:32,410 - ControlNet - DEBUG - parse_remote_call ran in: 0.001 sec
2023-12-17 20:40:32,411 - ControlNet - DEBUG - get_enabled_units ran in: 0.002 sec
2023-12-17 20:40:32,411 - ControlNet - INFO - unit_separate = False, style_align = False
2023-12-17 20:40:32,871 - ControlNet - DEBUG - clear_control_model_cache ran in: 0.460 sec
2023-12-17 20:40:32,872 - ControlNet - INFO - Loading model: control_sd15_animal_openpose_fp16 [293713ec]
2023-12-17 20:40:33,522 - ControlNet - INFO - Loaded state_dict from [D:\stable-diffusion-webui\models\ControlNet\control_sd15_animal_openpose_fp16.pth]
2023-12-17 20:40:33,525 - ControlNet - INFO - controlnet_default_config
2023-12-17 20:40:36,627 - ControlNet - INFO - ControlNet model control_sd15_animal_openpose_fp16 [293713ec] loaded.
2023-12-17 20:40:36,702 - ControlNet - DEBUG - build_control_model ran in: 3.829 sec
2023-12-17 20:40:36,703 - ControlNet - DEBUG - load_control_model ran in: 3.830 sec
2023-12-17 20:40:36,706 - ControlNet - DEBUG - choose_input_image ran in: 0.003 sec
2023-12-17 20:40:36,707 - ControlNet - DEBUG - Safe numpy convertion START
2023-12-17 20:40:36,712 - ControlNet - DEBUG - Safe numpy convertion END
2023-12-17 20:40:36,712 - ControlNet - INFO - Loading preprocessor: animal_openpose
2023-12-17 20:40:36,713 - ControlNet - INFO - preprocessor resolution = 512
2023-12-17 20:40:36,713 - ControlNet - DEBUG - Use numpy seed 1824187546.
2023-12-17 20:40:36,719 - ControlNet - DEBUG - Calling preprocessor animal_openpose outside of cache.
2023-12-17 20:40:43,880 - ControlNet - DEBUG - detectmap_proc ran in: 0.012 sec
2023-12-17 20:40:43,996 - ControlNet - DEBUG - controlnet_main_entry ran in: 11.601 sec
2023-12-17 20:40:43,996 - ControlNet - INFO - ControlNet Hooked - Time = 11.600558042526245
2023-12-17 20:40:43,997 - ControlNet - DEBUG - controlnet_hack ran in: 11.602 sec
2023-12-17 20:40:43,997 - ControlNet - DEBUG - process ran in: 11.602 sec

I think there is a alphabetical priority for A1111 to call each extension's Script.process. See https://github.com/Bing-su/adetailer/wiki#why-is-there-a--at-the-beginning-of-the-name.

[ljleb]

As a side note, since webui==1.7.0 you can put a metadata.ini file at the root of your extension to determine script order: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Developing-extensions#metadataini