downstream tasks in `norm_layer` for EfficientFormerV2

[deepkyu]

Hi,

First of all, congrats for this awesome research 🎉

I have a simple question while reading your EfficientFormerV2 codes.

In your backbone codes for detection and segmentation, I found that norm_layers are not applied in forward_token: https://github.com/snap-research/EfficientFormer/blob/2c0e950dc269b7f0229a3917fd54661b964554e0/segmentation/backbonev2.py#L650-L653

However, for your backbone in classifcation, it forwards with the norm_layer: https://github.com/snap-research/EfficientFormer/blob/2c0e950dc269b7f0229a3917fd54661b964554e0/models/efficientformer_v2.py#L622-L625

Actually, it seems that the difference in the above between classification and other tasks does not occur in your EfficientFormer code. I tried my best to find the detail in both your code and paper, but I couldn't.

So, I kindly ask you if you could explain why this should be different.

Thank you in advance.

+) To clarify my question, I added the corresponding code lines from EfficientFormer used in segmentation:

https://github.com/snap-research/EfficientFormer/blob/2c0e950dc269b7f0229a3917fd54661b964554e0/segmentation/backbone.py#L478-L483

Those seems to be the outputs from each norm layer.