nos icon indicating copy to clipboard operation
nos copied to clipboard

Add Layer Replacement to the ModelReshaper

Open diegofiori opened this issue 2 years ago • 0 comments

We can replace target layers with similar but "cheaper" implementations in terms of FLOPs.

Examples are:

  • Transformers layer with ALiBi
  • Resnet implementation with Resnet-RS
  • Linear layers with sparse Linear layers when preceded by an aggressive dropout

diegofiori avatar Jun 06 '22 21:06 diegofiori