Image output tiling for seamless textures with Stable Diffusion

[torrinworx]

Is your feature request related to a problem? Please describe. Currently there is no way to create seamless textures with Stable Diffusion, a crucial feature that is missing.

Describe the solution you'd like Something similar to this pull on the sd-webui repo: https://github.com/sd-webui/stable-diffusion-webui/pull/911

A simple argument in the StableDiffusionPipeline that would enable seamless texture generation for 3D applications.

[anton-l]

Hi @torrinworx! As the https://github.com/sd-webui/stable-diffusion-webui/pull/911 PR suggests, you can make the Stable Diffusion models tile-able by patching the torch.nn.Conv2d before loading the pipeline:

# add global options to models
def patch_conv(**patch):
    cls = torch.nn.Conv2d
    init = cls.__init__
    def __init__(self, *args, **kwargs):
        return init(self, *args, **kwargs, **patch)
    cls.__init__ = __init__

patch_conv(padding_mode='circular')
print("patched for tiling")

Native support for tiling in diffusers is unlikely to come, as it would either be hacky, or complicate the model the model design with additional arguments :)

[shirayu]

FYI: In my app, I toggle to be tile-able or not after pipe loading by change padding_mode in each layer.

1. First, save conv layers and their original padding modes
2. Change padding mode on each generation

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Could we make this a community pipeline maybe? :-)

[camenduru]

is it possible for flax maybe like this: https://flax.readthedocs.io/en/latest/api_reference/_autosummary/flax.linen.Conv.html?highlight=circular

update: I test it working 🎉

[uyo9ko]

Hi @torrinworx! As the sd-webui/stable-diffusion-webui#911 PR suggests, you can make the Stable Diffusion models tile-able by patching the torch.nn.Conv2d before loading the pipeline:

# add global options to models
def patch_conv(**patch):
    cls = torch.nn.Conv2d
    init = cls.__init__
    def __init__(self, *args, **kwargs):
        return init(self, *args, **kwargs, **patch)
    cls.__init__ = __init__

patch_conv(padding_mode='circular')
print("patched for tiling")

Native support for tiling in diffusers is unlikely to come, as it would either be hacky, or complicate the model the model design with additional arguments :)

hello, i follow your advice , add this snippet before loading the pipeline, but there is an error when loading pipeline:

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    model_path,
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-29-796abcd4f547>](https://localhost:8080/#) in <module>
      6     revision="fp16",
      7     torch_dtype=torch.float16,
----> 8     use_auth_token=True
      9 )
     10 pipe = pipe.to(device)

8 frames
[<ipython-input-19-17c9e93293f0>](https://localhost:8080/#) in __init__(self, *args, **kwargs)
      4     init = cls.__init__
      5     def __init__(self, *args, **kwargs):
----> 6         return init(self, *args, **kwargs, **patch)
      7     cls.__init__ = __init__
      8 

TypeError: __init__() got multiple values for keyword argument 'padding_mode'

can you tell me why?

[patrickvonplaten]

@anton-l, I think we can allow to adapt the conv mode with a nice diffusers API here no?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Linking some first PRs here:

• https://github.com/huggingface/diffusers/pull/1521
• https://github.com/huggingface/diffusers/pull/1441

[patrickvonplaten]

cc @patil-suraj @anton-l does any of you have time to look into tiling?

[keturn]

Note this must be applied to both the diffusion model and the decoder (VAE).

[patil-suraj]

That's the plan @keturn, still experimenting with it.

[keturn]

begone, stalebot!

[patrickvonplaten]

Ok it does not look like @patil-suraj you'll have time for this anytime soon no? Is someone else maybe interested in picking this one up (might be a bit difficult as a first PR though): @pcuenca @williamberman @yiyixuxu maybe?

[camenduru]

up

[patil-suraj]

I don't have much bandwidth for it this week, but I will try to look into https://github.com/huggingface/diffusers/pull/1521 next week

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[keturn]

it's still an open issue, stalebot

[AmanKishore]

This seems like an interesting add-on to the WebUI: https://github.com/tjm35/asymmetric-tiling-sd-webui Any insight on how to add X and Y padding into the current diffusers implementation?

[patrickvonplaten]

Since we have this: https://github.com/huggingface/diffusers/pull/1441 added now think we can close this one no?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[jtoy]

I’m interested in this as well. Is there at least a workable hack? Then we can look into making into usable code.

[shamelesslyAI]

I’m interested in this as well. Is there at least a workable hack? Then we can look into making into usable code.

Implementation in colab notebook: (Inspect the setup cell and copy paste) https://colab.research.google.com/drive/1E5Fa2Tu04g3kb443WnrhbWNhoMzcijoj?authuser=3#scrollTo=Z_pZ9zpFJvbY

[psychedelicious]

@patrickvonplaten I think this one was closed prematurely - this is to create seamless, tiling outputs - not tiled VAE decode. Can we re-open?

[patrickvonplaten]

Sure!

[pietrobolcato]

this is to create seamless, tiling outputs

Would also be super interested in this one!

[icech]

is there any progress about creating seamless, tiling outputs in diffusers？ @patrickvonplaten

[patrickvonplaten]

Are there any pointers of a working implementation?

[jtoy]

Automatic1111 had has it for a long time. — Sent from my mobileOn Aug 4, 2023, at 5:47 AM, Patrick von Platen @.***> wrote: Are there any pointers of a working implementation?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[icech]

Are there any pointers of a working implementation?

I want to know if it's possible to implement a tiled mode control flag, just like enable_vae_tiled(), so that everyone can use the tiled mode more conveniently. Shirayu provided a feasible approach, but it's not elegant enough. Link to the discussion