Key error 0 on evaluation set

[bollampa]

Hello,

I built a sample model with only 18 records. The tokenization used was from spacy and the format is BILUO instead of BIO and I only have one class which is Org. I modified the get labels function and also verified the model config json file.

My labels looks like this,

["O", "B-ORG", "I-ORG", "L-ORG", "U-ORG", "[CLS]", "[SEP]"]

The training goes fine, but validation fails with the following error.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 06/12/2020 14:07:59 - INFO - main - ***** Running evaluation ***** 06/12/2020 14:07:59 - INFO - main - Num examples = 18 06/12/2020 14:07:59 - INFO - main - Batch size = 8 Evaluating: 0%| | 0/3 [00:00<?, ?it/s] Traceback (most recent call last): File "run_ner.py", line 594, in main() File "run_ner.py", line 582, in main temp_2.append(label_map[logits[i][j]]) KeyError: 0

I have attached the train and valid files. Please point me in the right direction.

train.txt valid.txt

[heraclex12]

I have the same issue. Have you resolved it?

I think you can treat 0's index as PAD tag/or O tag. And then compute scores exclude PAD/O tag.