Seq2seq multiple input features

[iamsiva11]

Is there a way to pass extra feature along with the existing word tokens as input and feed it to the encoder RNN?

Lets consider the NMT problem , say I have 2 more feature columns for the corresponding source vocabulary( Feature1 here ). For example, consider this below:

Feature1 Feature2 Feature3 word1 x a word2 y b word3 y c . .

Would be great of anyone tells how to practically implement/get this done in pytorch. Thanks in advance.

[spro]

The simplest way is to concatenate features into a single input vector. However this only works if your RNN takes vector input, not discrete inputs (LongTensor) through an embedding layer. In that case you would want to concatenate your extra features after the input is embedded. This can most easily be achieved by using an extra input argument in your encoder and using torch.cat (see the category input in https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb for a simple example, though that does not use an embedding layer).

If your features are also discrete you would want multiple embedding layers, one for each, and concatenate all the results.

The quoted section is more about input and target sequence length, not feature size.

[spro]

For the initializer you would need to add arguments for your feature sizes, and create a new Embedding layer for each discrete feature. In the forward() method you would add arguments for the extra features, forward them through the relevant embedding layers, and concatenate them all into one vector for the RNN. The RNN also would have an input size of 3 * hidden_size because of the three embeddings added together (or you could create different embedding sizes for each feature).

Overall it should look something like this:

class EncoderRNN(nn.Module):
    def __init__(..., word_size, feature2_size, feature3_size, hidden_size, ...):
        
        ...
        
        self.word_embedding = nn.Embedding(word_size, hidden_size)
        self.feature2_embedding = nn.Embedding(feature2_size, hidden_size)
        self.feature3_embedding = nn.Embedding(feature3_size, hidden_size)
        # Note: * 3 because the above 3 embeddings will be concatenated
        self.gru = nn.GRU(hidden_size * 3, hidden_size, n_layers, dropout=self.dropout, bidirectional=True)
        
    def forward(self, word_seqs, feature2_seqs, feature3_seqs, input_lengths, hidden=None):
        # Note: we run this all at once (over multiple batches of multiple sequences)
        word_embedded = self.word_embedding(word_seqs)
        feature2_embedded = self.feature2_embedding(feature2_seqs)
        feature3_embedded = self.feature3_embedding(feature3_seqs)
        combined = torch.cat((word_embedded, feature2_embedded, feature3_embedded), 2)
        packed = torch.nn.utils.rnn.pack_padded_sequence(combined, input_lengths)
        ...

[iamsiva11]

@spro Thanks much for your inputs. I have written a blog post on the same - https://iamsiva11.github.io/extra-features-seq2seq/. Hope it'll be useful for many others.