model isn't learning

[raviolli]

Hi there, I have loaded up an albert model and loaded the weights just fine no errors or warning. but when I go to train it the loss never goes lower it stays around 0.7 and the accuracy stays around 0.5

I'm trying to perform a fine-tuning with a dense layer of size 2 classes. Any thoughts?

I've checked the data X and Y values and I think their correct. ids is tokenized+encoded. the mask is 1 for ids and 0 for padding. the type is 0 for all tokens. I'm using SpareCategoricalCrossEntropy so the label is 0 or 1 and not OHE.

I'm so stuck. Any help is greatly appreciated.

[Danny-Google]

Could you post more info like training info? Also, you may want to start with our colab tutorial.