Using SDXL Refiner

[anujnayyar1]

Is there a method for enabling / using the SD XL refiner. Ideally using passing over the latent rather than using img2img.

[anujnayyar1]

This is the traceback when trying to load the refiner using this command

onediffusion start stable-diffusion-xl --model-id stabilityai/stable-diffusion-xl-refiner-1.0

It seems to be expecting the 'scheduler', 'tokenizer'. which are part of the main SDXL model file but not the refiner.

Traceback:

  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 686, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/server/base_app.py", line 75, in lifespan
    on_startup()
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/runner/runner.py", line 317, in init_local
    raise e
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
    self._set_handle(LocalRunnerRef)
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
    runner_handle = handle_class(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
    self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/frameworks/diffusers.py", line 749, in __init__
    self.pipeline: diffusers.DiffusionPipeline = load_model(
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/frameworks/diffusers.py", line 339, in load_model
    pipeline: diffusers.DiffusionPipeline = pipeline_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py", line 1165, in from_pretrained
    raise ValueError(
ValueError: Pipeline <class 'diffusers.pipelines.stable_diffusion_xl.pipeline_stable_diffusion_xl.StableDiffusionXLPipeline'> expected {'unet', 'text_encoder', 'vae', 'text_encoder_2', 'tokenizer_2', 'scheduler', 'tokenizer'}, but only {'unet', 'vae', 'text_encoder_2', 'tokenizer_2', 'scheduler'} were passed.

2023-08-25T05:31:48+0000 [ERROR] [runner:pt-sdxl-stabilityai--stable-diffusion-xl-refiner-1.0:1] Application startup failed. Exiting.```

[larme]

Eventually we want to support this. But I need to consider the best API/cli switch UX to adapt refiner and other image modifiers/enhancers.