Bag of Tricks for Efficient Text Classification, fastText

Unofficial PyTorch Implementation of "Bag of Tricks for Efficient Text Classification", 2016, A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (https://arxiv.org/pdf/1607.01759.pdf)

• The original model

  • fastText, h=10, bigram (See Table 1 of the paper)

• Dataset

  • We use preprocessed data (See Xiang Zhang's folder)
    • AG's news, Sogou, DBpedia, Yelp P., Yelp F., Yahoo A., Amazon F., Amazon P.

• Experiment

  # Download a spacy "en_core_web_lg" model
  $ python3 -m spacy download en_core_web_lg --user
  
  # Download datasets (select your os (mac or ubuntu))
  $ sh download_datasets_mac.sh
  

  • AG

  # Create a pickle file: data/ag_news_csv/ag.pkl
  $ python3 dataset.py --data_dir ./data/ag_news_csv --pickle_name ag.pkl --num_classes 4 --max_len 467
  
  # Run
  $ python3 main.py --data_path ./data/ag_news_csv/ag.pkl --batch_size 2048 --lr 0.5 --log_interval 20
  

  • Sogou

  # Create a pickle file: data/sogou_news_csv/sogou.pkl
  $ python3 dataset.py --data_dir ./data/sogou_news_csv --pickle_name sogou.pkl --num_classes 5 --max_len 90064
  
  # Run
  $ python3 main.py --data_path ./data/sogou_news_csv/sogou.pkl --batch_size 1024 --lr 0.1 --log_interval 40
  

  • DBpedia

  # Create a pickle file: data/dbpedia_csv/dbp.pkl
  $ python3 dataset.py --data_dir ./data/dbpedia_csv --pickle_name dbp.pkl --num_classes 14 --max_len 3013
  
  # Run
  $ python3 main.py --data_path ./data/dbpedia_csv/dbp.pkl --batch_size 2048 --lr 0.1 --log_interval 20
  

  • Yelp P.

  # Create a pickle file: data/yelp_review_polarity_csv/yelp_p.pkl
  $ python3 dataset.py --data_dir ./data/yelp_review_polarity_csv --pickle_name yelp_p.pkl --num_classes 2 --max_len 2955
  
  # Run
  $ python3 main.py --data_path ./data/yelp_review_polarity_csv/yelp_p.pkl --batch_size 1024 --lr 0.1 --log_interval 40
  

  • Yelp F.

  # Create a pickle file: data/yelp_review_full_csv/yelp_f.pkl
  $ python3 dataset.py --data_dir ./data/yelp_review_full_csv --pickle_name yelp_f.pkl --num_classes 5 --max_len 2955
  
  # Run
  $ python3 main.py --data_path ./data/yelp_review_full_csv/yelp_f.pkl --batch_size 1024 --lr 0.05 --log_interval 40
  

  • Yahoo A.

  # Create a pickle file: data/yahoo_answers_csv/yahoo_a.pkl
  $ python3 dataset.py --data_dir ./data/yahoo_answers_csv --pickle_name yahoo_a.pkl --num_classes 10 --max_len 8024
  
  # Run
  $ python3 main.py --data_path ./data/yahoo_answers_csv/yahoo_a.pkl --batch_size 1024 --lr 0.05 --log_interval 40
  

  • Amazon F.

  # Create a pickle file: data/amazon_review_full_csv/amazon_f.pkl
  $ python3 dataset.py --data_dir ./data/amazon_review_full_csv --pickle_name amazon_f.pkl --num_classes 5 --max_len 1214
  
  # Run
  $ python3 main.py --data_path ./data/amazon_review_full_csv/amazon_f.pkl --batch_size 4096 --lr 0.25 --log_interval 10
  

  • Amazon P.

  # Create a pickle file: data/amazon_review_polarity_csv/amazon_p.pkl
  $ python3 dataset.py --data_dir ./data/amazon_review_polarity_csv --pickle_name amazon_p.pkl --num_classes 2 --max_len 1318
  
  # Run
  $ python3 main.py --data_path ./data/amazon_review_polarity_csv/yahoo_a.pkl --batch_size 4096 --lr 0.25 --log_interval 10
  

• Performance (accuracy %)

  • Results may vary slightly depending on your experimental environment.

• Training time for an epoch (CPU)
  • Results may vary slightly depending on your experimental environment.

• Dictionary size & data size

• Embeddings are used instead of binary encoding (=multi-hot).
• Diff. from the paper
  • Adam optimizer instead of SGD (stochastic gradient descent)
  • No Hashing Trick for AG, Sogou, DBP, Yelp P. and Yelp F. because the vocabulary size (1.4M ~ 7.1M) is less than 10M
  • No Hierarchical softmax because the number of classes is only 2 ~ 14
• Reference
  • https://github.com/poliglot/fasttext
  • https://github.com/bentrevett/pytorch-sentiment-analysis