Wiki Summary

This paper introduced the opportunity to use available pre-trained LMs (BERT, GPT-2, and RoBERTa) to achieve state-of-the-art results on some interesting NLP tasks like Text Summaziration. So, I trained an EncoderDecoder model based on parsBERT on WikiSummary (A summarization dataset extracted from Wikipedia). The model achieved an 8.47 ROUGE-2 score.

Dataset information

The dataset extracted from Persian Wikipedia into the form of articles and highlights. I cleaned the dataset into pairs of articles and highlights and reduced the articles' length (only version 1.0.0) and highlights' length to a maximum of 512 and 128, respectively, suitable for parsBERT. Also, for making things simple, converted the dataset into the HuggingFace' datasets, which is available in the repo. Or you can access the csv format here:

Version 1.0.0

• Train Set
• Dev Set
• Test Set

Version 2.0.0

• Train Set
• Dev Set
• Test Set

Example

An example ID is c291771f3b7d6dbc229fda944b7c03a7a1834263. These are heximal formated SHA1 hashes of the urls where the stories were retrieved from. For each ID, there is a string for the article, a string for the highlights, and a string for the id.

Evaluation

The following table summarizes the ROUGE scores obtained by the Bert2Bert model.

Version 1.0.0

Version 2.0.0 (coming soon)

How to use

HuggingFace Transformers

from transformers import (
    BertTokenizerFast,
    EncoderDecoderConfig,
    EncoderDecoderModel
)

model_name = 'm3hrdadfi/bert2bert-fa-wiki-summary'
tokenizer = BertTokenizerFast.from_pretrained(model_name)
config = EncoderDecoderConfig.from_pretrained(model_name)
model = EncoderDecoderModel.from_pretrained(model_name, config=config)

sequence = """YOUR TEXT COMES HERE"""
inputs = tokenizer([sequence], padding="max_length", truncation=True, max_length=512, return_tensors="pt")
input_ids = inputs.input_ids
attention_mask = inputs.attention_mask

outputs = model.generate(input_ids, attention_mask=attention_mask)
generated = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(f'YOUR GENERATED TEXT: {generated}')

Streamlit Application on Google-Colab

Follow the block steps on this Colab notebook.

Cite

I didn't publish any paper about this work. Please cite this repository in publications as the following:

@misc{Bert2BertWikiSummaryPersian,
  author = {Mehrdad Farahani},
  title = {Summarization using Bert2Bert model on WikiSummary dataset},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {https://github.com/m3hrdadfi/wiki-summary},
}

Releases

v1.0.0: Hello World!

Available by: m3hrdadfi/bert2bert-fa-wiki-summary

License

Apache License 2.0