Make an eval script for SQuaD

[chenmoneygithub]

From a high level it is just a classification task, but there are some details to handle. The whole workflow can be described as:

• Data
  1. We can use the SQuaD dataset from tensorflow dataset: link.
  2. Preferably just use V2?
• Data preprocessing
  1. Add [CLS] token at the start, and [SEP] token between context and answer.
  2. From the given answer_start field, calculate the answer_end value, representing the index of start and end in the context.
  3. Calculate the start and end index in the tokenized context.
  4. Set the three labels: start token index, end token index, impossible (means impossible to find the answer).
• Classification Head
  1. Takes in the pretrained model output, and outputs 3 things: start token logits/prob (shape=[sequence_length, ]), end token logits/prob (shape=[sequence_length, ]), impossible logits/prob (shape=[1,]).
  2. The structure should be one simple dense layer.
• Training config
  1. Optimizer: TBD
  2. Learning rate: TBD
  3. Should we use KerasTuner? TBD

[mattdangerw]

Is this issue specifically for the BERT example?

[chenmoneygithub]

It's for general purpose, I am thinking we should have some components for eval purposes.

[mattdangerw]

It might be good to build the script specifically for BERT right now, and then shuffle of the components as we get a better understanding of what we need.

[mattdangerw]

Assigning this to @aflah02 as I think you are the one actively working on this.

[aflah02]

yup thanks!