Ying-Jia Lin

CoLA reproducibility

2

Hi, I cannot reproduce the CoLA score as same as the one on paper. I followed the HuggingFace's repo to train a predecessor model with Matthew correlation score of 55.76....

https://github.com/ncbi-nlp/BLUE_Benchmark/blob/37cc4f7fff6b1a313353cbd70afde5e6c5732071/blue/bert/create_mednli_bert.py#L14 https://github.com/ncbi-nlp/BLUE_Benchmark/blob/37cc4f7fff6b1a313353cbd70afde5e6c5732071/blue/bert/create_mednli_bert.py#L18 These two lines do not match the MedNLIProcessor in run_bluebert.py and will cause KeyError after running run_bluebert.py. I suggest to modify the sequence of column indices.

AssertionError: compute_gradients() on the differentially private optimizer was not called. Which means that the training is not differentially private. It happens for example in Keras training in TensorFlow 2.0+.

3

Dear Developers, Thank you for implementing and maintaining such a great repository. I tried to train my model without eager execution in `TensorFlow version 2.3.1`. Some code snippets are below:...