[FEATURE] Implement SPACH models (A Battle of Network Structures)

[ardeal]

Hi,

are the following networks implemented in this repo? The network in the red curve

In this paper: A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP

[rwightman]

@ardeal unlikely any time soon, you're probably already aware that they are here, and were implemented using some parts / code from timm via deit/swin https://github.com/microsoft/SPACH

they would be a bit of a chore to reformat and bring into timm in a style I'm happy with for future models, and the results are decent but nothing special compared to existing hybrid or hierarchical transformer models...

[ardeal]

@rwightman ， Thank you!

The data in Transformer is: Swin-T [21] 29M 4.5G 81.3 but the data in Ours is Hybrid-MS-XS 28M 4.5G 82.4 The results are not decent, but a little bit ascent(1 percent).