refiners
refiners copied to clipboard
finegrain-ai
feature: create seamless tileable images in x or y dimension
Adds a seamless tile mode but does so in a way you guys probably won't like :-) Opening the pull request to have the discussion.
Tile mode can be set to tile in both directions or only "x" or "y". Set an image as desktop background in tile mode to really get the full effect.
I suspect this will work with SDXL as well but haven't tried it yet.
Examples:
Also FYI diffusers open feature request for the same feature: https://github.com/huggingface/diffusers/issues/556
Awesome idea! To make sure we don't mess with the existing implementation of UNet, we usually work with the Adapter pattern.
Here's how it works:
- We'll start by creating a TilingConv2dAdapter that will target a fl.Conv2d. This will wrap the original conv and modify it as needed.
- Next, we'll create a TilingAdapter that will target UNets. This will inject the TilingConv2dAdapter into all the conv layers.
Don't worry too much about the naming - it's just a suggestion!
Let me know if you have any questions or need anything else.
Was revisiting this. Should TilingConv2dAdapter inherit from Adapter? I ask because this assert seems to imply that an Adapter cannot wrap fl.Conv2d since it's not a chain
@brycedrennan Adapters can wrap Conv2d, they cannot be a Conv2d (they must subclass Chain). self is the adapter here, the conv would be the adaptee (the generic T). This is similar to SingleLoraAdapter which wraps a linear.
Adapters can wrap
Conv2d
@brycedrennan FYI, you have such an example part of the UNet: https://github.com/finegrain-ai/refiners/blob/e7892254eb85b06212515b5482a4fa3f699650f8/src/refiners/foundationals/latent_diffusion/stable_diffusion_1/unet.py#L263-L270
Well I'm pretty sure this is wrong in multiple ways, but it does work. I wasn't quite sure what the "right" way was to wrap the conv2d.
Additionally, it runs roughly .6 it/s slower than the method in the first commit that does not use adapters
Well I'm pretty sure this is wrong in multiple ways, but it does work.
Hey @brycedrennan, do you consider it "ready for review" or not?
Sorry to have let this sit so long. The proposed changes work but are slower than the implementation that directly patches conv2d. I'm not sure why that is but doesn't feel right to propose merging as is.
Since the feature is already working in imaginairy there isn't much urgency on my part. I'd be okay with this pull request being closed or alternatively if you have feedback on it I can continue iterating on it.
Maybe what could be done is just having an Adapter on UNet that would patch all Conv2d as a whole instead of wrapping each of them; that would work, too. I have yet to test it myself.