Transformer Networks for Sentiment Analysis

Implements a simple binary classifier for sentiment analysis, embedding sentences using a Transformer network. Transformer networks were introduced in the paper All You Need is Attention, where the authors achieve state of the art performance on several NLP tasks.

Usage

Run python train.py, to train a model on the IMDB reviews dataset (it will be downloaded automatically through torchtext if it's not present). This uses trained positional embeddings for the transformer networks, as opposed to the sinusoidal positional encodings introduced in the paper.

To use the Transformer module in another project, be sure to add some sort of positional encoding to the input before passing it to the module, as these are not automatically added.

Options

python train.py --help
usage: train.py [-h] [--max_length MAX_LENGTH] [--model_size MODEL_SIZE]
                [--epochs EPOCHS] [--learning_rate LEARNING_RATE]
                [--device DEVICE] [--num_heads NUM_HEADS]
                [--num_blocks NUM_BLOCKS] [--dropout DROPOUT]
                [--train_word_embeddings TRAIN_WORD_EMBEDDINGS]
                [--batch_size BATCH_SIZE]

Train a Transformer network for sentiment analysis

optional arguments:
  -h, --help            show this help message and exit
  --max_length MAX_LENGTH
                        Maximum sequence length, sequences longer than this
                        are truncated
  --model_size MODEL_SIZE
                        Hidden size for all hidden layers of the model
  --epochs EPOCHS       Number of epochs to train for
  --learning_rate LEARNING_RATE
                        Learning rate for optimizer
  --device DEVICE       Device to use for training and evaluation e.g. (cpu,
                        cuda:0)
  --num_heads NUM_HEADS
                        Number of attention heads in the Transformer network
  --num_blocks NUM_BLOCKS
                        Number of blocks in the Transformer network
  --dropout DROPOUT     Dropout (not keep_prob, but probability of ZEROING
                        during training, i.e. keep_prob = 1 - dropout)
  --train_word_embeddings TRAIN_WORD_EMBEDDINGS
                        Train GloVE word embeddings
  --batch_size BATCH_SIZE
                        Batch size

Requirements

• Python 2.7
• PyTorch 4.1
• TorchText
• NumPy
• tqdm (optional)

Acknowledgements

The Transformer networks were introduced by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin in All You Need is Attention.