ALBERT model not learning

[raviolli]

Hi There, I am having some issues getting the model to finetune.

I'm sort of confused and could use some help. Is there a forum I could ask for help?

The issue is that the model doesn't learn, it just stays at ~ 0.5 accuracy. (N.B. the output is 2 class dense)

Here's a sample output:

input_word_ids (InputLayer) [(None, 35)] input_mask (InputLayer) [(None, 35)] input_type_ids (InputLayer) [(None, 35)]

albert_model (AlbertModel) [(None, 1024)], (None 17683968)

                                                             input_word_ids[0][0]             
                                                             input_mask[0][0]                 
                                                             input_type_ids[0][0]             

dropout (Dropout) (None, 1024) 0 albert_model[0][0]

output (Dense) (None, 2) 2050 dropout[0][0]

Total params: 17,686,018 Trainable params: 17,686,018 Non-trainable params: 0

I0416 20:14:06.850114 140122845333248 finetune.py:186] ***** Running training ***** I0416 20:14:06.850288 140122845333248 finetune.py:187] Num examples = 52500 I0416 20:14:06.850376 140122845333248 finetune.py:188] Batch size = 32 I0416 20:14:06.850451 140122845333248 finetune.py:189] Num steps = 32812 Train on 47261 samples, validate on 5252 samples Epoch 1/20 2020-04-16 20:14:41.742967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 4064/47261 [=>............................] - ETA: 25:16 - loss: 0.8179 - sparse_categorical_accuracy: 0.4783