Using ELMo with TorchText

[littleflow3r]

Hi Keita, thanks for the tutorial. I have one question, though. Assume this is a sentiment analysis task, and suppose I already have ELMo representation for every sentence in the shape (seq_len, elmo_dimension). I want to either:

1. concat this representation with the embedding from embedding layer or,
2. use this representation before passing it to RNN/CNN.

Do you have any idea on how to use this with Torchtext? I am not sure how to add the elmo sentence representation to the batch and pass it to my model together with the input (which has been converted to indices) ?

Any advice/pointer would be greatly appreciated.

[zjxs1997]

Hi, I've been working on this recently, and here's my solution:

• first build dataset, and make sure the examples in the dataset contain the list ELMo representation shaped (seq len, hidden dim), for example: ex.elmo = torch.rand((len(sequence), hidden_dim))
• then define the corresponding field:

elmo_field = Field(
  sequential=True, 
  use_vocab=False, 
  batch_first=True, 
  pad_token = [0 for _ in range(hidden_dim)], 
  dtype=torch.float64
)

• finally build iterator with BucketIterator or whatever, the elmo attribute of the batch from iterator should be shaped (batch size, seq len, hidden dim)