keras-cv icon indicating copy to clipboard operation
keras-cv copied to clipboard

Add a neural style transfer model with AdaIN

Open chunxiangzheng opened this issue 3 years ago • 2 comments

chunxiangzheng avatar Jun 21 '22 17:06 chunxiangzheng

@LukeWood

chunxiangzheng avatar Jun 21 '22 17:06 chunxiangzheng

Notes from Luke/Jake 1:1:

  • get_adain_encoder(), get_adain_decoder() returns a tuple of encoder and decoder
  • Internally, we can use AdaIn as a custom layer
  • AdaIn(include_rescaling=..., weights=”mine.h5”) -> this is a functional model that just calls encoder()
  • AdaIn() layer decoder()
  • Internal that creates an AdaIn identical model from an encoder and decoder
  • model = produceAdaIn(encoder, decoder) -> functional model
  • Then we can call model.save(“mymodel.h5”)
  • examples/models/training/style_transfer/adain.py -> define the model trainer the same way we have defined it before

LukeWood avatar Jun 21 '22 18:06 LukeWood

As far as the API is concerned; lets wrap the inference logic in a top level class that loads the proper weights to the models, and implements inference in a style_transfer() method. You can see an example of this here: https://github.com/keras-team/keras-cv/blob/master/keras_cv/models/generative/stable_diffusion/stable_diffusion.py#L41

LukeWood avatar Oct 25 '22 17:10 LukeWood

Thanks for the PR! I'm not sure about the context here. It's not aligned with our immediate roadmap, so if this is important, please go ahead and address the comments so we can merge it. Or we can close it for now, and if this becomes needed we can re-open it later.

tanzhenyu avatar Nov 11 '22 17:11 tanzhenyu

Hey @chunxiangzheng -- where did we land on this PR? If you're interested in finishing it up, please feel free to address comments and re-request a review from Luke.

I'm going to close this PR for now, but just re-open it if you'd like to keep it going!

ianstenbit avatar Feb 06 '23 21:02 ianstenbit