Permuter not applied to condition indices

[nihalsid]

trafficstars

In file taming/models/cond_transformer in function encode_to_c, permute is not used, while it is used for z. Is this intentional? Default permuter is Identity so i guess it doesn't affect much right now.

[adeptflax]

I'm trying do to image 2 image translation on with transformers. I seemed to have to add it into encode_to_z to get it to stop throwing error. Though It still doesn't output any sane results.