keras-nlp
keras-nlp copied to clipboard
Make an eval script for SQuaD
From a high level it is just a classification task, but there are some details to handle. The whole workflow can be described as:
- Data
- We can use the SQuaD dataset from tensorflow dataset: link.
- Preferably just use V2?
- Data preprocessing
- Add [CLS] token at the start, and [SEP] token between context and answer.
- From the given
answer_start
field, calculate theanswer_end
value, representing the index of start and end in the context. - Calculate the start and end index in the tokenized context.
- Set the three labels: start token index, end token index, impossible (means impossible to find the answer).
- Classification Head
- Takes in the pretrained model output, and outputs 3 things: start token logits/prob (shape=[sequence_length, ]), end token logits/prob (shape=[sequence_length, ]), impossible logits/prob (shape=[1,]).
- The structure should be one simple dense layer.
- Training config
- Optimizer: TBD
- Learning rate: TBD
- Should we use KerasTuner? TBD
Is this issue specifically for the BERT example?
It's for general purpose, I am thinking we should have some components for eval purposes.
It might be good to build the script specifically for BERT right now, and then shuffle of the components as we get a better understanding of what we need.
Assigning this to @aflah02 as I think you are the one actively working on this.
yup thanks!