VDCNN

Tensorflow Implementation of Very Deep Convolutional Neural Network for Text Classification, proposed by Conneau et al.

Archiecture for VDCNN is now correctly re-implemented with Tensorflow 2 and tf.keras support. A simple training interface is implemented following Tensorflow 2 Expert Tutorial. Feel free to contribute additional utilities like TensorBoard support.

Side Note, if you are a newcomer for NLP text classification:

• Please checkout new SOTA NLP methods like transformers or Bert.

• Check out PyTorch for MUCH BETTER dynamic graphing and dataset object support.

  • Current VDCNN implementation is also extremely easy to be ported onto PyTorch.

Prerequisites

• Python3
• Tensorflow >= 2.0
• tensorflow-datasets
• numpy

Datasets

The original paper tests several NLP datasets, including DBPedia, AG's News, Sogou News and etc.

tensorflow-datasets is used to support AG's News dataset.

Downloads of those NLP text classification datasets can be found here (Many thanks to ArdalanM):

Parameters Setting

The original paper suggests the following details for training:

• SGD optimizer with lr 1e-2, decay 0.9.
• 10 - 15 epochs for convergence.
• He Initialization.

Some additional parameter settings for this repo:

• Gradient clipping with norm_value of 7.0, to stablize the training.

Skip connections and pooling are correctly implemented now:

• k-maxpooling.
• maxpooling with kernel size of 3 and strides 2.
• conv pooling with K_i convolutional layer.

For dotted skip connections:

• Identity with zero padding.
• Conv1D with kernel size of 1.

Please refer to Conneau et al for their methodology and experiment section in more detail.

Experiments

Results are reported as follows: (i) / (ii)

• (i): Test set accuracy reported by the paper (acc = 100% - error_rate)
• (ii): Test set accuracy reproduced by this Keras implementation

TODO: Feel free to report your own experimental results in the following format:

Results for "Identity" Shortcut, "k-max" Pooling:

Reference

Original preprocessing codes and VDCNN Implementation By geduo15

Train Script and data iterator from Convolutional Neural Network for Text Classification

NLP Datasets Gathered by ArdalanM and Others