🤗 + ⚛️ Fine-tuned Transformers compatible BERT models for Sequence Tagging

This repository contains fine-tuned Transformers compatible BERT models for sequence tagging tasks like NER or PoS tagging.

We use the (coming soon) fine-tuning NER example from the awesome Transformers repository.

Changelog

• 23.09.2019: PoS tagging results on German HDT added
• 21.09.2019: Initial release of this repo. Publicly available models are coming soon!

CoNLL datasets

For NER tasks, the CoNLL 2002 and 2003 datasets are used. The following languages are covered: German, English, Dutch and Spanish.

These models are trained with a max. sequence length of 128.

❔ But what happens, if a sentence in the dataset is longer than 128 subtokens? We simply split longer sentences into smaller ones. We do not ✂️ sentences, which would be bad for evaluation!

❔ What is current state-of-the-art for fine-tuned models? Well, we mainly compare our models to the following two papers:

• "How multilingual is Multilingual BERT?" by Pires, Schlinger and Garrette (2019)

• "Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT" by Wu and Dredze (2019)

❔ We use a max. sequence length of 128 and a batch size of 32. These are the same parameters as used by Pires, Schlinger and Garrette (2019).

Results

We report an averaged F1-score of 5 different runs. We use different seeds for each run. The official CoNLL evaluation script from here is used.

English

Pires, Schlinger and Garrette (2019) report a F1-score of 91.07% using the EN-BERT model.

German

Notice: For this experiment we use the original CoNLL-2003 data. The dataset also includes an 2006 update that fixes various MISC annotations. However, it turns out that most papers are not using the updated 2006 version of the dataset. So in this experiment, we use the "old" dataset in order to achieve a better comparison to other papers. See a more detailed discussion about this dataset in here.

We use three BERT models:

• Multilingual BERT (base and cased), further called mBERT
• German BERT (base and cased) from here
• An own trained German BERT (base and cased), currently unreleased

Pires, Schlinger and Garrette (2019) report a F1-score of 82.00% for their fine-tuned multilingual BERT model, whereas Wu and Dredze (2019) report a F1-score of 82.82%.

For German DistilBERT a learning rate of 6e-5 was used.

Dutch

Pires, Schlinger and Garrette (2019) report a F1-score of 89.86% for their fine-tuned multilingual BERT model, whereas Wu and Dredze (2019) report a F1-score of 90.94%.

Spanish

Pires, Schlinger and Garrette (2019) report a F1-score of 87.18% for their fine-tuned multilingual BERT model, whereas Wu and Dredze (2019) report a F1-score of 87.38%.

GermEval 2014

We train three models for the GermEval 2014 shared task: mBERT, German BERT and our own German BERT model.

Notice: the original dataset includes some strange character control sequences (all labeled with "O" tag). We remove them in a pre-processing step (for train/dev/test sets).

Results

We report averaged F1-score over 5 different runs (with different seeds). The official evaluation script is used for evaluation. We report the Strict, Combined Evaluation (official) metric here.

For German DistilBERT a learning rate of 6e-5 was used.

Universal Depedencies

German HDT

We train three models (mBERT, German BERT and our own German BERT model) on the recently released German HDT Universal Dependencies corpus. It contains over 200K annotated sentences, resulting in one of the largest UD corpora. We use the latest data from the dev branch on the German HDT repository.

Results

We report averaged accuracy over 5 different runs (with different seeds).

Italian

We've trained BERT models from scratch for Italian (both cased and uncased).

WikiNER

The WikiNER dataset from here for Italian is pre-processed into a 80/10/10 (training, development, test) split. We train for 10 epochs using the default parameters from the run_ner.py script.

Results

EVALITA 2009

We use the NER data provided by the EVALITA 2009 shared task. We train for 20 epochs using the default parameters from the run_ner.py script.

Results

PoSTWITA

We use the dataset for the PoSTWITA shared task to report results for PoS tagging (Twitter). The dataset is used from this repository. We report results both for cased and uncased BERT models. We train for 20 epochs using the default parameters from the run_ner.py script.

ToDo

• [ ] Release models (incl. BertTokenizer and BertModel example usage)
• [ ] Provide more details about training parameters, and prediction script
• [ ] Release trained BERT model: German DistilBERT
• [ ] Add link to Italian BERT models (cased and uncased)