wenet icon indicating copy to clipboard operation
wenet copied to clipboard

Squeezeformer

Open yygle opened this issue 2 years ago • 0 comments

Develop Record

squeezeformer
├── attention.py                        # reltive multi-head attention module  
├── conv2d.py                           # self defined conv2d valid padding module
├── convolution.py                      # convolution module in squeezeformer block
├── encoder_layer.py                    # squeezeformer encoder layer
├── encoder.py                          # squeezeformer encoder class 
├── positionwise_feed_forward.py        # feed forward layer 
├── subsampling.py                      # sub-sampling layer, time reduction layer
└── utils.py                            # residual connection module
  • Implementation Details
    • Squeezeformer Encoder
      • [x] add pre layer norm before squeezeformer block
      • [x] derive time reduction layer from tensorflow version
      • [x] enable adaptive scale operation
      • [x] enable init weights for deep model training
      • [x] enable training config and results
      • [x] enable dynamic chunk and JIT export
    • Training
      • [x] enable NoamHoldAnnealing schedular

yygle avatar Sep 15 '22 06:09 yygle