iswc2020_prodcls
iswc2020_prodcls copied to clipboard
ISWC2020 Semantic Web Challenge - Product Classification Top1 Solution
iswc2020_prodcls
ISWC2020 Semantic Web Challenge - Product Classification Top1 Solution
Solution
Overall Framework
- Train 17 different BERT base models with dynamic masked softmax
- Adopt a two-level ensemble strategy to combine the single models
- Utilize pseudo labeling for data augmentation
Base Model Construction
- BERT-PO uses the pooler output of BERT as the product representation.
- BERT-K-hidden concatenates the first hidden state from the last $K$ hidden layers of BERT as the product representation.
- BERT-K-hidden-PO concatenates the first hidden state from the last $K$ hidden layers as well as the pooler output of BERT as the product representation.
- BERT-seq uses the hidden states from the last hidden layer of BERT as the input of another sequence layer, and then concatenates the pooler output of BERT, with the last hidden output as well as the max-pooling and mean-pooling over the hidden states of sequence layer, as the final product representation.
Dynamic Masked Softmax
- Devise a mask matrix for each sub-level based on the category hierarchy
$$ M^{l} \in{0,1}^{N^{l-1} * N^{l}} $$ - Adopt dynamic masked softmax to filter out the unrelated categories
$$ P\left(y_{v}^{l} \mid s, \theta\right)=\frac{\exp \left(O_{v}^{l}\right) * M_{u, v}^{l}+\exp (-8)}{\sum_{v^{\prime}=1}^{N} \exp \left(O_{v^{\prime}}^{l}\right) * M_{u, v^{\prime}}^{l}+\exp (-8)} $$
Model Ensemble
- Averaging ensemble to the single models with the same architecture but trained on different folds of data
- Voting ensemble to 17 different single models
Pseudo Labeling
Experiment
Environment Preparation
pip install virtualenv
virtualenv tf2
source tf2/bin/activate
pip install -r requirements.txt
Data Preparation
-
dataset
Download the dataset below, put them in theraw_data
dir: -
pre-trained BERT model Download the files of BERT model, put them in the
raw_data/embeddings/bert-base-uncased
dir:
Pre-processing
python preprocess.py
(1st) Training
python train.py
(1st) Ensemble and Pseudo Labeling
python ensemble.py
(2nd) Re-training with Pseudo Labels
python train_pseudo_label.py
(2nd) Re-ensemble
python ensemble_pseudo_label.py