Support Transformer LM

[marcoyang1998]

This PR support transformer LM training in icefall.

Initial testing: an RNNLM and a transformer LM trained on LibriSpeech 960h text and their best perplexities are shown below.

LM	Num params	Perplexity
RNN LM	38.5M	17.94
Transformer LM	41.5M	15.13

TODO:

1. Finish training a larger version
2. Implement decoding methods supporting transformer LM

[marcoyang1998]

Some updates:

I am training a larger transformer LM using librispeech-lm-corpus. The results are here:

Model	Num params	Perplexity
RNNLM	102.8M	9.97
Transformer LM	98.6M	9.73

The perplexity is still going down. I will upload it to huggingface and paste the link here once it's done.

[marcoyang1998]

Update of transformer LM decoding:

Model	LM	test-clean	test-other
lstm_transducer_stateless2	RNNLM	2.42	6.46
lstm_transducer_stateless2	TransformerLM	2.37	6.48

Will update the decoding results for LODR.

[marcoyang1998]

Upload the pre-trained model to huggingface: https://huggingface.co/marcoyang/icefall-librispeech-transformer-lm/tree/main

[marcoyang1998]

Some LODR results:

Model	LM	test-clean	test-other
lstm_transducer_stateless2	RNNLM	2.24	5.89
lstm_transducer_stateless2	TransformerLM	2.19	5.90
pruned_transducer_stateless3 + Giga	RNNLM	1.77	3.99
pruned_transducer_stateless3 + Giga	TransformerLM	1.75	3.94
pruned_transducer_stateless7	RNNLM	1.91	4.57
pruned_transducer_stateless7	TransformerLM	1.91	4.51

[csukuangfj]

Could you also update the CI? (Decoding test-clean/test-other with transformer LM)

[marcoyang1998]

Could you also update the CI? (Decoding test-clean/test-other with transformer LM)

Will do.