Glyce: Glyph-vectors for Chinese Character Representations

Glyce is an open-source toolkit built on top of PyTorch and is developed by Shannon.AI.

Citation

To appear in NeurIPS 2019.

Glyce: Glyph-vectors for Chinese Character Representations

(Yuxian Meng*, Wei Wu*, Fei Wang*, Xiaoya Li*, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun and Jiwei Li, 2019)

@article{meng2019glyce,
  title={Glyce: Glyph-vectors for Chinese Character Representations},
  author={Meng, Yuxian and Wu, Wei and Wang, Fei and Li, Xiaoya and Nie, Ping and Yin, Fan and Li, Muyu and Han, Qinghong and Sun, Xiaofei and Li, Jiwei},
  journal={arXiv preprint arXiv:1901.10125},
  year={2019}
}

Table of Contents

• What is Glyce ?
• Experiment Results
  • Sequence Labeling Tasks
  • Sentence Pair Classification
  • Single Sentence Classification
  • Chinese Semantic Role Labeling
  • Chinese Dependency Parsing
• Requirements
• Installation
• Quick Start of Glyce
  • Usage of Glyce Char/Word Embedding
  • Usage of Glyce-BERT
• Folder Description
• Implementation
• Download Task Data
• Welcome Contributions to Glyce Open Source Project
• Acknowledgement
• Contact

What is Glyce ?

Glyce is a Chinese char representation based on Chinese glyph information. Glyce Chinese char embeddings are composed by two parts: (1) glyph-embeddings and (2) char-ID embeddings. The two parts are combined using concatenation, a highway network or a fully connected layer. Glyce word embeddings are also composed by two parts: (1) word glyph-embeddings and (2) word-ID embeddings. Glyph word embedding is the output of forwarding glyph embeddings for each char in the word through the max-pooling layer. Then glyph word embedding combine word-ID embedding using concatenation to get glyce word embedding.

Here are the some highlights in glyce:

1. Utilize Chinese Logographic Information

Glyce utilizes useful logographic information by encoding images of historical and contemporary scripts, along with the scripts of different writing styles.

2. Combine Glyce with Chinese Pre-trained BERT Model

We combine Glyce with Pre-trained Chinese BERT model and adopt specific layer to downstream tasks. The Glyce-BERT model outperforms BERT and sets new SOTA results for tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks.

3. Propose Tianzige-CNN(田字格) to Model Chinese Char

We propose the Tianzige-CNN(田字格) structure, which is tailored to Chinese character modeling. Tianzige-CNN(田字格) tackle the issue of small number of Chinese characters and small sacle of image compared to Imagenet.

4. Auxiliary Task Performs As a Regularizer

During training process, image-classification loss performs as an auxiliary training objective with the purpose of preventing overfitting and promiting the model's ability to generalize.

Experiment Results

1. Sequence Labeling Tasks

Named Entity Recognition (NER)

MSRA(Levow, 2006), OntoNotes 4.0(Weischedel et al., 2011), Resume(Zhang et al., 2018).

Chinese Part-Of-Speech Tagging (POS)

CTB5, CTB6, CTB9 and UD1

Chinese Word Segmentation (CWS)

PKU, CITYU, MSR and AS.

2. Sentence Pair Classification

Dataset Description

The BQ corpus, XNLI, LCQMC, NLPCC-DBQA

3.Single Sentence Classification

Dataset Description

ChnSentiCorp, Fudan and Ifeng.

4. Chinese Semantic Role Labeling

Dataset Description

CoNLL-2009

Model Performance

5. Chinese Dependency Parsing

Dataset Description

Chinese Penn TreeBank 5.1. Dataset splits follows (Dozat and Manning, 2016).

Model Performance

Requirements

• Python Version >= 3.6
• GPU, Use NVIDIA TITAN Xp with 12G RAM
• Chinese scripts could be found in Google Drive. Please refer to the description and download scripts files to glyce/glyce/fonts/.

NOTE: Some experimental results are obtained by training on multi-GPU machines. May use DIFFERENT PyTorch versions refer to previous open-source SOTA models. Experiment environment for Glyce-BERT is Python 3.6 and PyTorch 1.10.

Installation

# Clone glyce 
git clone [email protected]:ShannonAI/glyce.git
cd glyce 
python3.6 setup.py develop 

# Install package dependency 
pip install -r requirements.txt

Quick start of Glyce

Usage of Glyce Char/Word Embedding

import torch 
import torch.nn as nn 
from torch.nn import CrossEntropyLoss 


from glyce import GlyceConfig 
from glyce import CharGlyceEmbedding, WordGlyceEmbedding 


# load glyce hyper-params
glyce_config = GlyceConfig()


# if input ids is char-level
glyce_embedding = CharGlyceEmbedding(glyce_config)
# elif input ids is word-level 
glyce_embedding = WordGlyceEmbedding(glyce_config)


# forward input_ids into embedding layer 
glyce_embedding, glyph_loss = glyce_embedding(input_ids)


# utilize image classifiation loss as the auxiliary loss 
glyph_decay = 0.1 
glyph_ratio = 0.01 # the proportion of image classification loss to total loss 
current_epoch = 3 


# compute loss 
loss_fct = CrossEntropyLoss()
task_loss = loss_fct(logits, labels)
total_loss = task_loss * (1 - glyce_ratio) + glyph_ratio * glyph_decay ** (current_epoch+1) * glyph_loss 

Usage of Glyce-BERT

1. Preparation

• Download and unzip BERT-Base, Chinese pretrained model.

• Install PyTorch pretrained bert by pip as follows:

pip install pytorch-pretrained-bert

• Convert TF checkpoint into PyTorch

export BERT_BASE_DIR=/path/to/bert/chinese_L-12_H-768_A-12

pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
$BERT_BASE_DIR/bert_model.ckpt \
$BERT_BASE_DIR/bert_config.json \
$BERT_BASE_DIR/pytorch_model.bin

2. Train/Dev/Test DataFormat

• Sequence Labeling Task

模型的输入: [CLS] 我 爱 北 京 天 安 门
模型的输出: ‘我爱北京天安门' 对应的命名实体标签 O  O  B-LOC M-LOC M-LOC M-LOC E-LOC 

Model Input: [CLS] I like Bei Jing Tian An Men
Model Output: labels corresponds to 'I like BeiJing Tian An men' are O  O  B-LOC M-LOC M-LOC M-LOC E-LOC 

• Sentence Pair Classification

模型的输入: [CLS] 我 爱 北 京 天 安 门 [SEP] 北 京 欢 迎 您
模型的输出: [CLS] 位置对应的输出,假如是语意匹配任务则为 0,代表两句话的语意不相同

Model Input: [CLS] I like Bei Jing Tian An Men [SEP] Bei Jing Wel ##come you 
Model Output: the output in [CLS] position. 0 if task sign is semantic matching. It means that the semantic of two sentences are different. 

• Single Sentence Classification

模型的输入: [CLS] 我 爱 北 京 天 安 门
模型的输出: [CLS] 位置对应的输出,假如是情感分类任务则为1,代表情感为积极

Model Input: [CLS] I like Bei Jing Tian An Men. 
Model Output: the output in [CLS] position. 1 if task is sentiment analysis. It means the sentiment polarity is positive. 

3. Start Train and Evaluate Glyce-BERT

• scritps/*_bert.sh are the commands we used to finetune BERT.
• scripts/*_glyce_bert.sh are the commands we used to obtained the results of Glyce-BERT.
• scripts/ctb5_binaffine.sh is the command that we used to reimplement PREVIOUS SOTA result on CTB5 for dependency parsing.
• scripts/ctb5_glyce_binaffine.sh is the command that we used to obtain the SOTA result on CTB5 for dependency parsing.

For example, training command of Glyce-BERT for sentence pair dataset BQ is included in scripts/glyce_bert/bq_glyce_bert.sh. Start train and evaluate Glyce-BERT on BQ by,

bash scripts/glyce_bert/bq_glyce_bert.sh

Notes:

• repo_path refer to work directory of glyce.
• config_path refer to the path of configuration file.
• bert_model refer to the the directory of the pre-trained Chinese BERT model.
• task_name refer to the task signiature.
• data_dir and output_dir refer to the directories of the "raw data" and "intermediate checkpoints" respectively.

Folder Description

Implementation

Glyce toolkit provides implementations of previous SOTA models incorporated with Glyce embeddings.

• Glyce: Glyph-vectors for Chinese Character Representations. Refer to ./glyce/models/glyce_bert
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Refer to ./glyce/models/bert
• Chinese NER Using Lattice LSTM. Refer to ./glyce/models/latticeLSTM
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF. Refer to ./glyce/models/latticeLSTM/model
• Syntax for Semantic Role Labeling, To Be, Or Not To Be. Refer to ./glyce/models/srl
• Bilateral Multi-Perspective Matching for Natural Language Sentences. Refer to ./glyce/models/bimpm
• Deep Biaffine Attention for Neural Dependency Parsing. Refer to ./glyce/models/biaffine

Download Task Data

Download Task Data

Welcome Contributions to Glyce Open Source Project

We actively welcome researchers and practitioners to contribute to Glyce open source project. Please read this Guide and submit your Pull Request.

Acknowledgement

Acknowledgement. Vanilla Glyce is developed based on the previous SOTA model. Glyce-BERT is developed based on PyTorch implementation by HuggingFace. And pretrained BERT model is released by Google's pre-trained models.

Contact

Please feel free to discuss paper/code through issues or emails.

License

Apache License 2.0