Question about the Shortcut of FFN in AutoFormer?

Open pprp opened this issue 3 years ago • 1 comments

Thank you for your great job! I have some questions about the shortcut design of FFN in AutoFormer.

https://github.com/microsoft/Cream/blob/83a154beb6f85dd6141853b4b7c0738eeec628ba/AutoFormer/model/supernet_transformer.py#L245-L246

To match the dimension setting of FFN, sample_embed_dim should be the same as sample_out_dim.

I wonder why it is not set up like:

self.sample_output_dim = self.sample_embed_dim

but like:

https://github.com/microsoft/Cream/blob/83a154beb6f85dd6141853b4b7c0738eeec628ba/AutoFormer/model/supernet_transformer.py#L109

Jun 19 '22 01:06 pprp

Hi, thanks for your interest in our work. We set it up like this because the output_dim should be the same as the input dimensions of the next transformer block.

Aug 11 '22 06:08 nbl97