AiSpace

AiSpace provides highly configurable framework for deep learning model development, deployment and conveniently use of pre-trained models (bert, albert, opt, etc.).

Table of Contents

• Features
• Requirements
• Instructions
• Configuration
• Pretrained
• Experiments
• Todos
• Refs

Features

• Highly configurable, we manage all hyperparameters with inheritable Configuration files.
• All modules are registerable, including models, dataset, losses, optimizers, metrics, callbacks, etc.
• Standardized process
• Multi-GPU Training
• K-fold cross validation training
• Integrate lr finder
• Integrate multiple pre-trained models, including chinese
• Simple and fast deployment using BentoML
• Integrated Chinese benchmarks CLUE

Requirements

git clone https://github.com/yingyuankai/AiSpace.git
cd AiSpace && pip install -r requirements

Instructions

Training

python -u aispace/trainer.py \
    --schedule train_and_eval \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

Output file structure

The default output path is save, which may has multiple output directories under name as:

{experiment_name}_{model_name}_{dataset_name}_{random_seed}_{id}

Where id indicates the sequence number of the experiment for the same task, increasing from 0.

Take the text classification task as an example, the output file structure is similar to the following:

experiment_name: test

model_name: bert_for_classification

dataset_name: glue_zh/tnews

random_seed: 119

id: 0

test_bert_for_classification_glue_zh__tnews_119_0
├── checkpoint                  # 1. checkpoints
│   ├── checkpoint
│   ├── ckpt_1.data-00000-of-00002
│   ├── ckpt_1.data-00001-of-00002
│   ├── ckpt_1.index
|   ...
├── deploy                      # 2. Bentoml depolyment directory
│   └── BertTextClassificationService
│       └── 20191208180211_B6FC81
├── hparams.json                # 3. Json file of all hyperparameters
├── logs                        # 4. general or tensorboard log directory
│   ├── errors.log              # error log file
│   ├── info.log                # info log file
│   ├── train                
│   │   ├── events.out.tfevents.1574839601.jshd-60-31.179552.14276.v2
│   │   ├── events.out.tfevents.1574839753.jshd-60-31.profile-empty
│   └── validation
│       └── events.out.tfevents.1574839787.jshd-60-31.179552.151385.v2
├── model_saved                 # 5. last model saved
│   ├── checkpoint
│   ├── model.data-00000-of-00002
│   ├── model.data-00001-of-00002
│   └── model.index
└── reports                     # 6. Eval reports for every output or task
    └── output_1_classlabel     # For example, text classification task
        ├── confusion_matrix.txt
        ├── per_class_stats.json
        └── stats.json

Training with resumed model

python -u aispace/trainer.py \
    --schedule train_and_eval \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --model_resume_path MODEL_RESUME_PATH \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

--model_resume_path is a path to initialization model.

lr finder

Firstly, use optimizer adma and open lr_finder callback.

policy:
    name: "base"
        
optimizer:
  name: adam
    
callbacks:
    lr_finder:
      switch: true

Then run training policy as base.

Lastly, you can find lr_finder.jpg in you workspace.

K-fold cross validation training

Firstly, Replace training default policy form base to:

training:
  policy:
    name: "k-fold"
    config:
      k: 5

The k is the number of fold. Your can refer to the configuration file in:

./confis/glue_zh/tnews_k_fold.yml

Then run training script as usual.

Average checkpoints

python -u aispace/trainer.py \
    --schedule avg_checkpoints \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --prefix_or_checkpoints PREFIX_OR_CHECKPOINGS \
    [--ckpt_weights CKPT_WEIGHTS] \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

--prefix_or_checkpoints is paths to multiple checkpoints separated by comma.

--ckpt_weights is weights same order as the prefix_or_checkpoints.

Deployment

Generate deployment files before deployment, you need to specify the model path (--model_resume_path) to be deployed like following.

python -u aispace/trainer.py \
    --schedule deploy \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --model_resume_path MODEL_RESUME_PATH \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

We use BentoML as deploy tool, so your must implement the deploy function in your model class.

Configuration

All the configurations are in configs, in which base (./configs/default/base.yml) is the most basic, any configuration downstream includes this configuration directly or indirectly. Before you start, it is best to read this configuration carefully.

Your can use includes field to load other configurations, then the current configuration inherits the configurations and overrides the same configuration fields. Just like class inheritance, a function of the same name in a subclass overrides a function of the parent class.

The syntax is like this:

merge configuration of bert_huggingface into current.

includes:
  - "../pretrain/bert_huggingface.yml"     # relative path

Datasets

Pretrained

We have integrated multiple pre-trained language models and are constantly expanding。

For those models that need to be downloaded manually, download, unzip them and modify the path in the corresponding configuration.

Some pre-trained models don't have tensorflow versions, I converted them and made them available for download。

Examples

We have implemented some tasks in the CLUE (The Chinese General Language Understanding Evaluation (GLUE) benchmark).

Please refer to Examples Doc.

Take glue_zh/tnews as an example:

Tnews is a task of Chinese GLUE, which is a short text classification task from ByteDance.

Run Tnews classification

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name tnews \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 2 3  \

Specify different pretrained model, please change includes and pretrained.name in config file.

NOTE: The hyper-parameters used here have not been fine-tuned.

Todos

• More complete and detailed documentation;
• More pretrained models;
• More evaluations of CLUE;
• More Chinese dataset;
• Support Pytorch;
• Improve the tokenizer to make it more versatile;
• Build AiSpace server, make it can train and configure using UI.

Contact Author

Mail: [email protected]

Wechat: woshimoming1991

Citing

@misc{AiSpace,
  author = {yuankai ying},
  title = {AiSpace: Highly configurable framework for deep learning model development and deployment},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/yingyuankai/AiSpace}},
}

Refs

• transformers

• UER-py

• Huawei pretrained models