Feature request: Tiled Image support

[anonderpling]

I'm interested in creating tiled images.

https://github.com/invoke-ai/InvokeAI/pull/339 seems to implement it in InvokeAI. TomMoore515/material_stable_diffusion seems to implement it in SD. It's available in a bunch of places, including A1111 and a bunch of HF spaces. According @TomMoore515's modifications, it seems impossibly easy.

[comfyanonymous]

Is this supposed to actually work well? I tried it out and it worked very poorly for me which is why I didn't implement it.

[sliterok]

Is this supposed to actually work well? I tried it out and it worked very poorly for me which is why I didn't implement it.

Yes, tiling works very well!

Here's a couple of examples

[sliterok]

Any chance this gets implemented?

[JordanPowell]

I had a go and managed to get tileable images but something internally is broken and I'm not knowledgeable enough about the inner workings of the model to know what's going on. It basically loses coherency and even sort of inverts some colours. If I prompt the unaltered model for a pattern of cartoon dogs it gives something reasonable, and if I ask my doctored version then it's close (and tileable) but the anatomy and general structure of the output is totally wonky, and the background is often a dark colour rather than light.

My changes just involved changing the nn.Conv2D padding mode to circular - initially I did it globally and then I tried it just in ResBlock but it seems to produce the same result either way. I'm guessing that there are some Conv2D that should not have padding_mode circular and that's what's causing the problem but I'm not sure exactly what.

If anyone knows conceptually what should be happening I can probably translate it to the code.

[MoonRide303]

I've looked at A1111 code, and it's seems "Tiling" implementation goes down to 2 places:

1. processing.py, process_images_inner() - there is a call to modules.sd_hijack.model_hijack.apply_circular(p.tiling)
2. sd_hijack.py, where this short function is implemented:

    def apply_circular(self, enable):
        if self.circular_enabled == enable:
            return

        self.circular_enabled = enable

        for layer in [layer for layer in self.layers if type(layer) == torch.nn.Conv2d]:
            layer.padding_mode = 'circular' if enable else 'zeros'

@comfyanonymous any chance we could have something like that added to ComfyUI? It works well in A1111 - one click to enable "tiling" in generation parameters, and it's producing seamlessly tiling images, then. Works with SDXL, too.

[BoosterCore]

really need this option.

[LantosIstvan]

+1

[BoosterCore]

Is this supposed to actually work well? I tried it out and it worked very poorly for me which is why I didn't implement it.

This feature is really useful, for example, you can use it to make tile maps for 3D models. This feature of the WebUI works well, but overall I prefer the ComfyUI experience. Also, DreamTexture for blender is good too. https://github.com/carson-katri/dream-textures However, it is not ideal for model import, and it currently does not support SDXL. Therefore, I really hope ComfyUI can integrate this feature and at least give users a choice.

[LantosIstvan]

I only found this plugin for ComfyUI which provides asymmetric tiling support: https://github.com/FlyingFireCo/tiled_ksampler

The generated images are really seamless (also offering a Circular VAE Decoder, which also implements the padding_mode='circular' pytorch option), but in my test, even if pipe the output from the SDXL refiner, the images that it generates are wayyy different from the latent source. Almost like a new generation from scratch.

I think the implementation needs more work than throwing padding_mode='circular' everywhere in the Ksampler and VAE decoder definitions. Maybe creating a latent mask automatically around the x% edges of the frame? Just an idea.

[BoosterCore]

I only found this plugin for ComfyUI which provides asymmetric tiling support: https://github.com/FlyingFireCo/tiled_ksampler

The generated images are really seamless (also offering a Circular VAE Decoder, which also implements the padding_mode='circular' pytorch option), but in my test, even if pipe the output from the SDXL refiner, the images that it generates are wayyy different from the latent source. Almost like a new generation from scratch.

I think the implementation needs more work than throwing padding_mode='circular' everywhere in the Ksampler and VAE decoder definitions. Maybe creating a latent mask automatically around the x% edges of the frame? Just an idea.

Thank you very much. The repository you mentioned has been used in WebUI before. I didn't expect that there is also a version adapted to ComfyUI. Although the results it provides have a very small deviation of a few pixels, the overall performance is very close to the Tiling performance of WebUI itself. I hope ComfyUI can achieve the same perfect tiling effect as Dream Texture in the future.

[jn-jairo]

I made a custom node that I use to create circular images (tiling), it is very simple, I didn't tested it with advanced workflows but it works fine with a basic workflow and SD 1.5, here is a example:

And the texture applied in Blender:

If someone want it here is the node code, just save it as a .py file inside the custom_nodes folder like custom_nodes/conv2d.py.

import torch

class Conv2dSettings:
    PADDING_MODES = ['zeros', 'circular']
    
    @classmethod
    def INPUT_TYPES(s):
        return {
            "optional": {
                "model": ("MODEL",),
                "vae": ("VAE",),
            },
            "required": {
                "padding_mode": (s.PADDING_MODES, ),
            },
        }

    RETURN_TYPES = ("MODEL", "VAE")

    FUNCTION = "apply_settings"

    CATEGORY = "_for_testing"

    def apply_settings(self, model=None, vae=None, padding_mode=None):
        if model:
            self._apply_settings(model.model, padding_mode)
        if vae:
            self._apply_settings(vae.first_stage_model, padding_mode)

        return (model, vae)

    def _apply_settings(self, model, padding_mode):
        def flatten(el):
            flattened = [flatten(children) for children in el.children()]
            res = [el]
            for c in flattened:
                res += c
            return res

        layers = flatten(model)

        for layer in [layer for layer in layers if isinstance(layer, torch.nn.Conv2d)]:
            layer.padding_mode = padding_mode if padding_mode else 'zeros'

        return model

NODE_CLASS_MAPPINGS = {
    "Conv2dSettings": Conv2dSettings
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "Conv2dSettings": "Conv2d Settings"
}

[n00mkrad]

I made a custom node that I use to create circular images (tiling), it is very simple, I didn't tested it with advanced workflows but it works fine with a basic workflow and SD 1.5, here is a example:

And the texture applied in Blender:

If someone want it here is the node code, just save it as a .py file inside the custom_nodes folder like custom_nodes/conv2d.py.

import torch

class Conv2dSettings:
    PADDING_MODES = ['zeros', 'circular']
    
    @classmethod
    def INPUT_TYPES(s):
        return {
            "optional": {
                "model": ("MODEL",),
                "vae": ("VAE",),
            },
            "required": {
                "padding_mode": (s.PADDING_MODES, ),
            },
        }

    RETURN_TYPES = ("MODEL", "VAE")

    FUNCTION = "apply_settings"

    CATEGORY = "_for_testing"

    def apply_settings(self, model=None, vae=None, padding_mode=None):
        if model:
            self._apply_settings(model.model, padding_mode)
        if vae:
            self._apply_settings(vae.first_stage_model, padding_mode)

        return (model, vae)

    def _apply_settings(self, model, padding_mode):
        def flatten(el):
            flattened = [flatten(children) for children in el.children()]
            res = [el]
            for c in flattened:
                res += c
            return res

        layers = flatten(model)

        for layer in [layer for layer in layers if isinstance(layer, torch.nn.Conv2d)]:
            layer.padding_mode = padding_mode if padding_mode else 'zeros'

        return model

NODE_CLASS_MAPPINGS = {
    "Conv2dSettings": Conv2dSettings
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "Conv2dSettings": "Conv2d Settings"
}

Huge thanks, this works!

One question though, how would one have to modify this in order to make only the sides or top/bottom seamless? (Think 2D sidescroller games where only the sides are repeating).

InvokeAI does this, here's the relevant code:

import torch.nn as nn

def _conv_forward_asymmetric(self, input, weight, bias):
    """
    Patch for Conv2d._conv_forward that supports asymmetric padding
    """
    working = nn.functional.pad(input, self.asymmetric_padding['x'], mode=self.asymmetric_padding_mode['x'])
    working = nn.functional.pad(working, self.asymmetric_padding['y'], mode=self.asymmetric_padding_mode['y'])
    return nn.functional.conv2d(working, weight, bias, self.stride, nn.modules.utils._pair(0), self.dilation, self.groups)

def configure_model_padding(model, seamless, seamless_axes):
    """
    Modifies the 2D convolution layers to use a circular padding mode based on the `seamless` and `seamless_axes` options.
    """
    for m in model.modules():
        if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
            if seamless:
                m.asymmetric_padding_mode = {}
                m.asymmetric_padding = {}
                m.asymmetric_padding_mode['x'] = 'circular' if ('x' in seamless_axes) else 'constant'
                m.asymmetric_padding['x'] = (m._reversed_padding_repeated_twice[0], m._reversed_padding_repeated_twice[1], 0, 0)
                m.asymmetric_padding_mode['y'] = 'circular' if ('y' in seamless_axes) else 'constant'
                m.asymmetric_padding['y'] = (0, 0, m._reversed_padding_repeated_twice[2], m._reversed_padding_repeated_twice[3])
                m._conv_forward = _conv_forward_asymmetric.__get__(m, nn.Conv2d)
            else:
                m._conv_forward = nn.Conv2d._conv_forward.__get__(m, nn.Conv2d)
                if hasattr(m, 'asymmetric_padding_mode'):
                    del m.asymmetric_padding_mode
                if hasattr(m, 'asymmetric_padding'):
                    del m.asymmetric_padding

[jn-jairo]

Huge thanks, this works!

One question though, how would one have to modify this in order to make only the sides or top/bottom seamless? (Think 2D sidescroller games where only the sides are repeating).

InvokeAI does this, here's the relevant code:

import torch.nn as nn

def _conv_forward_asymmetric(self, input, weight, bias):
    """
    Patch for Conv2d._conv_forward that supports asymmetric padding
    """
    working = nn.functional.pad(input, self.asymmetric_padding['x'], mode=self.asymmetric_padding_mode['x'])
    working = nn.functional.pad(working, self.asymmetric_padding['y'], mode=self.asymmetric_padding_mode['y'])
    return nn.functional.conv2d(working, weight, bias, self.stride, nn.modules.utils._pair(0), self.dilation, self.groups)

def configure_model_padding(model, seamless, seamless_axes):
    """
    Modifies the 2D convolution layers to use a circular padding mode based on the `seamless` and `seamless_axes` options.
    """
    for m in model.modules():
        if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
            if seamless:
                m.asymmetric_padding_mode = {}
                m.asymmetric_padding = {}
                m.asymmetric_padding_mode['x'] = 'circular' if ('x' in seamless_axes) else 'constant'
                m.asymmetric_padding['x'] = (m._reversed_padding_repeated_twice[0], m._reversed_padding_repeated_twice[1], 0, 0)
                m.asymmetric_padding_mode['y'] = 'circular' if ('y' in seamless_axes) else 'constant'
                m.asymmetric_padding['y'] = (0, 0, m._reversed_padding_repeated_twice[2], m._reversed_padding_repeated_twice[3])
                m._conv_forward = _conv_forward_asymmetric.__get__(m, nn.Conv2d)
            else:
                m._conv_forward = nn.Conv2d._conv_forward.__get__(m, nn.Conv2d)
                if hasattr(m, 'asymmetric_padding_mode'):
                    del m.asymmetric_padding_mode
                if hasattr(m, 'asymmetric_padding'):
                    del m.asymmetric_padding

@n00mkrad I already did it, my version of this node is more complex now, I also created other nodes for myself, I'll try to take some time to make a repository on github to save my nodes and I post it here so you can see how I made it. It has some workarounds to avoid editing the core of the comfyui and be reliable in complex workflows, so it is not pretty but it works.

[jn-jairo]

@n00mkrad I already did it, my version of this node is more complex now, I also created other nodes for myself, I'll try to take some time to make a repository on github to save my nodes and I post it here so you can see how I made it. It has some workarounds to avoid editing the core of the comfyui and be reliable in complex workflows, so it is not pretty but it works.

I finally had time to commit it jn_node_suite_comfyui those are my nodes, it is a work in progress, but you can see how the Seamless node works if you want to.

My nodes use the beta branch of my fork of the ComfyUI that has some PR that are not merged to the original ComfyUI, so if you use it with the original ComfyUI you may get some errors.