During Example-guided Image Translation extracting style from domain A works, but from domain B breaks

[MInner]

Hi!

For some reason, after 500k iterations translations between domains with random styles look very good (test.py <...> --a2b 1 outputs), whereas an attempt to add --style <image_from_the_domain_b> flag yields a complete mess that looks like translations from very early epochs - it preserves general contours/layout of an image, but is extremely blurry, and does not look anything like the domain b. But, if I add the --style <image_from_the_domain_A> flag - it yields reasonable images. The image_from_the_domain_b was in the training set for the domain A.

Any suggestions about why that might be happening?

I am using the same set of network hyperparameters as in the edges2handbags config on 128x128 images. Translations with random styles, as I mentioned, look good.

Ben

[gcosne]

Did you find anything on this side ? I have the same issue training on my own dataset: Here are somes results after 250K epochs Guided style:

Random style:

[MInner]

I tried generating all combinations of "style from domain X, content from Y, decode as Z" and some of them worked well, while others failed miserably; fortunately, the one I needed was among those combinations that worked

[gcosne]

I have an answer to that guidance issue: to perform guided translation you have to train the network accordingly, which means changing the gen.update & discriminator update to perform the translation a-to-b using a guided style. In gen_update (trainer.py) you could add a if statement saying guided or not:

            # encode
            c_a, s_a_prime = self.gen_a.encode(x_a)
            c_b, s_b_prime = self.gen_b.encode(x_b)
            # decode (within domain)
            x_a_recon = self.gen_a.decode(c_a, s_a_prime)
            x_b_recon = self.gen_b.decode(c_b, s_b_prime)
            # decode (cross domain)
            if self.guided == 0 :
                x_ba = self.gen_a.decode(c_b, s_a)
                x_ab = self.gen_b.decode(c_a, s_b)
            elif self.guided ==1:
                x_ba = self.gen_a.decode(c_b, s_a_prime)
                x_ab = self.gen_b.decode(c_a, s_b_prime)

Same in the dis_update function.

[yingxingde]

I have an answer to that guidance issue: to perform guided translation you have to train the network accordingly, which means changing the gen.update & discriminator update to perform the translation a-to-b using a guided style. In gen_update (trainer.py) you could add a if statement saying guided or not:

            # encode
            c_a, s_a_prime = self.gen_a.encode(x_a)
            c_b, s_b_prime = self.gen_b.encode(x_b)
            # decode (within domain)
            x_a_recon = self.gen_a.decode(c_a, s_a_prime)
            x_b_recon = self.gen_b.decode(c_b, s_b_prime)
            # decode (cross domain)
            if self.guided == 0 :
                x_ba = self.gen_a.decode(c_b, s_a)
                x_ab = self.gen_b.decode(c_a, s_b)
            elif self.guided ==1:
                x_ba = self.gen_a.decode(c_b, s_a_prime)
                x_ab = self.gen_b.decode(c_a, s_b_prime)

Same in the dis_update function.

I encounter ther same problem when I use own dataset for training. Do you have any solution to it Now?