Device confusion in ConformerLstmModel

[Stasiche]

Environment info

• Platform: Ubuntu 20.04
• Python version: 3.9
• PyTorch version (GPU?): pytorch-lightning==1.5.2
• Using GPU in script?: No

Information

Model I am using (ListenAttendSpell, Transformer, Conformer ...): ConformerLstmModel

I hadn't got enough memory to use GPU, but I wanted to try one thing. So I set "trainer=cpu". I got "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)" on https://github.com/openspeech-team/openspeech/blob/main/openspeech/decoders/lstm_attention_decoder.py#L140

As I can see the reason is that it's not safe to do this this way: https://github.com/openspeech-team/openspeech/blob/main/openspeech/decoders/lstm_attention_decoder.py#L243 Because my card is still installed, so torch.cuda.is_available() == True. But my model is on CPU.

[sooftware]

Thank you for letting us know. We'll check. cc. @upskyy