ConvNets-for-Sentence-Classification

ConvNets-for-Sentence-Classification copied to clipboard

Sentence Classification with Convolution Neural Networks

This repository contains experiments done with Keras implementations of the Convolutional Neural Networks (ConvNets) to perform sentence classification based on the paper of Kim (2014).

The idea was not to replicate the results reported on the paper, but to have a first hands-on experiment with Keras, and some understanding of the method for classification proposed in the paper.

This code is also the result of a blog post I wrote about the idea of using ConvNets to classify sentences:

• http://www.davidsbatista.net/blog/2018/03/31/SentenceClassificationConvNets/

Experiments

• TREC (http://cogcomp.cs.illinois.edu/Data/QA/QC/)
• Stanford Sentiment Treebank (https://nlp.stanford.edu/sentiment/) (fine-grained-labels)

Some considerations:

• I used GloVe vectors of dimensionality 50
• The random embeddings have a dimensionality of 300
• I ran the training for 10 epochs

Accuracy

The results are in line with the ones reported in the paper, a bit lower, but within the expected.

Precision / Recall / F1

Since this is a multi-label dataset I also report the results per class for each of the different models.

TREC

CNN w/ random word embeddings

         precision    recall  f1-score   support

    ABBR       1.00      0.22      0.36         9
    DESC       0.74      0.22      0.34       138
    ENTY       0.55      0.88      0.68        94
     HUM       0.84      0.86      0.85        65
     LOC       0.91      0.93      0.92        81
     NUM       0.61      0.84      0.70       113

CNN w/ pre-trained static word embeddings

      precision    recall  f1-score   support

    ABBR       1.00      0.56      0.71         9
    DESC       0.57      0.09      0.15       138
    ENTY       0.32      0.89      0.47        94
     HUM       0.94      0.69      0.80        65
     LOC       0.75      0.73      0.74        81
     NUM       0.98      0.71      0.82       113

CNN w/ pre-trained dynamic word embeddings

     precision    recall  f1-score   support

    ABBR       0.75      0.67      0.71         9
    DESC       0.76      0.22      0.35       138
    ENTY       0.39      0.82      0.52        94
     HUM       0.94      0.77      0.85        65
     LOC       0.74      0.90      0.82        81
     NUM       0.96      0.85      0.90       113

CNN multichannel dynamic and static word embeddings

           precision    recall  f1-score   support

    ABBR       0.88      0.78      0.82         9
    DESC       0.64      0.26      0.37       138
    ENTY       0.38      0.79      0.51        94
     HUM       0.84      0.83      0.84        65
     LOC       0.87      0.89      0.88        81
     NUM       1.00      0.81      0.90       113

SST-1

CNN w/ random word embeddings

               precision    recall  f1-score   support

     negative       0.22      0.19      0.20       405
      neutral       0.53      0.72      0.61      1155
     positive       0.19      0.11      0.14       424
very positive       0.07      0.02      0.03       112
very_negative       0.05      0.01      0.01       114

CNN w/ pre-trained static word embeddings

               precision    recall  f1-score   support

     negative       0.13      0.03      0.05       405
      neutral       0.52      0.91      0.66      1155
     positive       0.13      0.03      0.05       424
very positive       0.00      0.00      0.00       112
very_negative       0.00      0.00      0.00       114

CNN w/ pre-trained dynamic word embeddings

               precision    recall  f1-score   support

     negative       0.15      0.10      0.12       405
      neutral       0.52      0.76      0.61      1155
     positive       0.18      0.08      0.11       424
very positive       0.12      0.03      0.04       112
very_negative       0.04      0.01      0.01       114

CNN multichannel dynamic and static word embeddings

               precision    recall  f1-score   support

     negative       0.16      0.11      0.13       405
      neutral       0.52      0.65      0.58      1155
     positive       0.23      0.22      0.22       424
very positive       0.02      0.01      0.01       112
very_negative       0.12      0.03      0.04       114