Reproducibility of results

[jurader]

Hi.

I have several questions about reproducibility of the code.

1 Pretraining part1

python run_pretraining.py \
  --input_file=../preprocessed_data/tf_examples_128.tfrecord \
  --output_dir=../output_pretrained/pretraining_output_128 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=bert_config.json \
  --init_checkpoint=bert_model.ckpt \
  --train_batch_size=64 \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5

This command produced the following result (../output_pretrained/pretraining_output_128/eval_results.txt). Is this result acceptable?

global_step = 100000 loss = 1.0482247 masked_lm_accuracy = 0.79015565 masked_lm_loss = 0.90652496 next_sentence_accuracy = 0.9425 next_sentence_loss = 0.14315148

2 Pretraining part2

python run_pretraining.py \
  --input_file=../preprocessed_data/tf_examples_512.tfrecord \
  --output_dir=../output_pretrained/pretraining_output_512 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=bert_config.json \
  --init_checkpoint=../output_pretrained//pretraining_output_128/model.ckpt-100000 \
  --train_batch_size=16 \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5

This command produced the following result (../output_pretrained/pretraining_output_512/eval_results.txt). Is this result acceptable?

global_step = 100000 loss = 0.64307 masked_lm_accuracy = 0.85223603 masked_lm_loss = 0.6104435 next_sentence_accuracy = 0.99 next_sentence_loss = 0.029400254

3 Early Notes Prediction

python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_eval \
  --data_dir ./data/3days/ \
  --bert_model ./model/early_readmission/ \
  --max_seq_length 512 \
  --output_dir ./result_early

"./data/3days/" was obtained by [preprocess.py] (https://github.com/kexinhuang12345/clinicalBERT/blob/master/preprocess.py). "./model/early_readmission/" was obtained from the link of ClinicalBERT Weights.

This command produced the following result. Is this result acceptable?

08/29/2022 08:17:52 - INFO - main - ***** Eval results ***** 08/29/2022 08:17:52 - INFO - main - RP80 = 0.596958174904943 08/29/2022 08:17:52 - INFO - main - eval_accuracy = 0.7395699319977945 08/29/2022 08:17:52 - INFO - main - eval_loss = 0.4970936025972028 08/29/2022 08:17:52 - INFO - main - global_step = 0 08/29/2022 08:17:52 - INFO - main - training loss = 100000.0 Recall at Precision of 80 is {} 0.596958174904943

4 Training readmission prediction model from pretraining ClinicalBERT

python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_train \
  --do_eval \
  --data_dir ./data/3days/ \
  --bert_model ./model/pretraining \
  --max_seq_length 512 \
  --train_batch_size 16 \
  --learning_rate 2e-5 \
  --num_train_epochs 1 \
  --output_dir ./result_finetuning

"./data/3days/" was obtained by preprocess.py. "./model/pretraining" was obtained from the link of ClinicalBERT Weights.

This command produced the following result. For me, this result is not acceptable. If there is something wrong, please let me know the way to finetune the model.

08/29/2022 08:43:39 - INFO - main - ***** Eval results ***** 08/29/2022 08:43:39 - INFO - main - RP80 = 0.12547528517110265 08/29/2022 08:43:39 - INFO - main - eval_accuracy = 0.6173497518838449 08/29/2022 08:43:39 - INFO - main - eval_loss = 0.6674913585799357 08/29/2022 08:43:39 - INFO - main - global_step = 2988 08/29/2022 08:43:39 - INFO - main - training loss = 0.6257770110864237 Recall at Precision of 80 is {} 0.12547528517110265