PCPM
PCPM copied to clipboard
rusiaaman
Presenting Collection of Pretrained Models. Links to pretrained models in NLP and voice.
PCPM
Presenting Corpus of Pretrained Models. Links to pretrained models in NLP and voice with training script.
With rapid progress in NLP it is becoming easier to bootstrap a machine learning project involving text. Instead of starting with a base code, one can now start with a base pretrained model and within a few iterations get SOTA performance. This repository is made with the view that pretrained models minimizes collective human effort and cost of resources, thus accelerating development in the field.
Models listed are curated for either pytorch or tensorflow because of their wide usage.
Note: pytorch-transofmers is an awesome library which can be used to quickly infer/fine-tune from many pre-trained models in NLP. The pre-trained models from those are not included here.
Contents
- Text ML models
- Speech to text models
- Datasets
- Hall of Shame
- Non english models
- Other Collections
Text ML
Language Models
| Name | Link | Trained On | Training script |
|---|---|---|---|
| Transformer-xl | https://github.com/kimiyoung/transformer-xl/tree/master/tf#obtain-and-evaluate-pretrained-sota-models | enwik8, lm1b, wt103, text8 |
https://github.com/kimiyoung/transformer-xl |
| GPT-2 | https://github.com/openai/gpt-2/blob/master/download_model.py | webtext |
https://github.com/nshepperd/gpt-2/ |
| Adaptive Inputs (fairseq) | https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.md#pre-trained-models | lm1b |
https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.md |
Permutation lanugage modelling Based - XLNet
| Name | Link | Trained On | Training script |
|---|---|---|---|
| XLnet | https://github.com/zihangdai/xlnet/#released-models | booksCorpus+English Wikipedia+Giga5+ClueWeb 2012-B+Common Crawl |
https://github.com/zihangdai/xlnet/ |
Masked Language Modelling Based - Bert
| Name | Link | Trained On | Training script |
|---|---|---|---|
| RoBERTa | https://github.com/pytorch/fairseq/tree/master/examples/roberta#pre-trained-models | booksCorpus+CC-N EWS+OpenWebText+CommonCrawl-Stories | https://github.com/huggingface/transformers |
| BERT | https://github.com/google-research/bert/ | booksCorpus+English Wikipedia | https://github.com/huggingface/transformers |
| MT-DNN | https://mrc.blob.core.windows.net/mt-dnn-model/mt_dnn_base.pt (https://github.com/namisan/mt-dnn/blob/master/download.sh) | glue | https://github.com/namisan/mt-dnn |
Machine Translation
| Name | Link | Trained On | Training script |
|---|---|---|---|
| OpenNMT | http://opennmt.net/Models-py/ (pytorch) http://opennmt.net/Models-tf/ (tensorflow) | English-German | https://github.com/OpenNMT/OpenNMT-py |
| Fairseq (multiple models) | https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md#pre-trained-models | WMT14 English-French, WMT16 English-German | https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md |
Sentiment
| Name | Link | Trained On | Training script |
|---|---|---|---|
| Nvidia sentiment-discovery | https://github.com/NVIDIA/sentiment-discovery#pretrained-models | SST, imdb, Semeval-2018-tweet-emotion | https://github.com/NVIDIA/sentiment-discovery |
| MT-DNN Sentiment | https://drive.google.com/open?id=1-ld8_WpdQVDjPeYhb3AK8XYLGlZEbs-l | SST | https://github.com/namisan/mt-dnn |
Reading Comprehension
SQUAD 1.1
| Rank | Name | Link | Training script |
|---|---|---|---|
| 49 | BiDaf | https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz | https://github.com/allenai/allennlp |
Summarization
Model for English summarization
| Name | Link | Trained On | Training script |
|---|---|---|---|
| OpenNMT | http://opennmt.net/Models-py/ | Gigaword standard | https://github.com/OpenNMT/OpenNMT-py |
Speech to Text
| Name | Link | Trained On | Training script |
|---|---|---|---|
| NeMo-quartznet | https://ngc.nvidia.com/catalog/models/nvidia:quartznet15x5 | librispeech,mozilla-common-voice | https://github.com/NVIDIA/NeMo |
| OpenSeq2Seq-Jasper | https://nvidia.github.io/OpenSeq2Seq/html/speech-recognition.html#models | librispeech | https://github.com/NVIDIA/OpenSeq2Seq |
| Espnet | https://github.com/espnet/espnet#asr-results | librispeech,Aishell,HKUST,TEDLIUM2 | https://github.com/espnet/espnet |
| wav2letter++ | https://talonvoice.com/research/ | librispeech | https://github.com/facebookresearch/wav2letter |
| Deepspeech2 pytorch | https://github.com/SeanNaren/deepspeech.pytorch/issues/299#issuecomment-394658265 | librispeech | https://github.com/SeanNaren/deepspeech.pytorch |
| Deepspeech | https://github.com/mozilla/DeepSpeech#getting-the-pre-trained-model | mozilla-common-voice, librispeech, fisher, switchboard | https://github.com/mozilla/DeepSpeech |
| speech-to-text-wavenet | https://github.com/buriburisuri/speech-to-text-wavenet#pre-trained-models | vctk | https://github.com/buriburisuri/speech-to-text-wavenet |
| at16k | https://github.com/at16k/at16k#download-models | NA | NA |
Datasets
Datasets referenced in this document
Language Model data
Common crawl
http://commoncrawl.org/
enwik8
Wikipedia data dump (Large text compression benchmark) http://mattmahoney.net/dc/textdata.html
text8
Wikipedia cleaned text (Large text compression benchmark) http://mattmahoney.net/dc/textdata.html
lm1b
1 Billion Word Language Model Benchmark https://www.statmt.org/lm-benchmark/
wt103
Wikitext 103 https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
webtext
Original dataset not released by the authors. An open source collection is available at https://skylion007.github.io/OpenWebTextCorpus/
English wikipedia
https://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia
BooksCorpus
https://yknzhu.wixsite.com/mbweb https://github.com/soskek/bookcorpus
Sentiment
SST
Stanford sentiment tree bank https://nlp.stanford.edu/sentiment/index.html. One of the Glue tasks.
IMDB
IMDB movie review dataset used for sentiment classification http://ai.stanford.edu/~amaas/data/sentiment
Semeval2018te
Semeval 2018 tweet emotion dataset https://competitions.codalab.org/competitions/17751
Glue
Glue is a collection of resources for benchmarking natural language systems. https://gluebenchmark.com/ Contains datasets on natural language inference, sentiment classification, paraphrase detection, similarity matching and lingusitc acceptability.
Speech to text data
fisher
https://pdfs.semanticscholar.org/a723/97679079439b075de815553c7b687ccfa886.pdf
librispeech
www.danielpovey.com/files/2015_icassp_librispeech.pdf
switchboard
https://ieeexplore.ieee.org/document/225858/
Mozilla common voice
https://github.com/mozilla/voice-web
vctk
https://datashare.is.ed.ac.uk/handle/10283/2651
Hall of Shame
High quality research which doesn't include pretrained models and/or code for public use.
- KERMIT https://arxiv.org/abs/1906.01604 Generative Insertion-Based Modeling for Sequences. No code.
Non English
Other Collections
Allen NLP
Built on pytorch, allen nlp has produced SOTA models and open sourced them. https://github.com/allenai/allennlp/blob/master/MODELS.md
They have neat interactive demo on various tasks at https://demo.allennlp.org/
GluonNLP
Based on MXNet this library has extensive list of pretrained models on various tasks in NLP. http://gluon-nlp.mxnet.io/master/index.html#model-zoo
rusiaaman