wenet
wenet copied to clipboard
Squeezeformer
Develop Record
squeezeformer
├── attention.py # reltive multi-head attention module
├── conv2d.py # self defined conv2d valid padding module
├── convolution.py # convolution module in squeezeformer block
├── encoder_layer.py # squeezeformer encoder layer
├── encoder.py # squeezeformer encoder class
├── positionwise_feed_forward.py # feed forward layer
├── subsampling.py # sub-sampling layer, time reduction layer
└── utils.py # residual connection module
- Implementation Details
- Squeezeformer Encoder
- [x] add pre layer norm before squeezeformer block
- [x] derive time reduction layer from tensorflow version
- [x] enable adaptive scale operation
- [x] enable init weights for deep model training
- [x] enable training config and results
- [x] enable dynamic chunk and JIT export
- Training
- [x] enable NoamHoldAnnealing schedular
- Squeezeformer Encoder