Simultaneous Speech Translation

Code base for simultaneous speech translation experiments. It is based on fairseq.

Implemented

Encoder

• Transformers with convolutional context for ASR
• Emformer

Streaming Models

• Wait-k [example]
• Monotonic Multihead Attention [example]
• Continuous Integrate-and-Fire [example]

Setup

1. Install fairseq

git clone https://github.com/pytorch/fairseq.git
cd fairseq
git checkout 4a7835b
python setup.py build_ext --inplace
pip install .

2. (Optional) Install apex for faster mixed precision (fp16) training.
3. Install dependencies

pip install -r requirements.txt

4. Update submodules

git submodule update --init --recursive

Pre-trained model

ASR model with Emformer encoder and Transformer decoder. Pre-trained with joint CTC cross-entropy loss.

Sequence-level Knowledge Distillation

Citation

Please consider citing our paper:

@inproceedings{chang22f_interspeech,
  author={Chih-Chiang Chang and Hung-yi Lee},
  title={{Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={5175--5179},
  doi={10.21437/Interspeech.2022-10627}
}