qm9_nn_conv.py gives unexpected large MAE on target 7

[DHPO]

❓ Questions & Help

When I run the example code qm9_nn_conv.py , I found that the MAE of target 7 is much larger than it should be.

The expected MAE of target 7($\U_0$) is 0.0194, and of target 0($\mu$) is 0.030, as reported in this paper.

However, the result of the example code is 5.226 and 0.061, accordingly.

The training logs are as follows. Target 7:

Epoch: 001, LR: 0.001000, Loss: 0.1442482, Validation MAE: 199.7960085, Test MAE: 198.4628370
Epoch: 002, LR: 0.001000, Loss: 0.0269572, Validation MAE: 103.6007244, Test MAE: 104.6817957
Epoch: 003, LR: 0.001000, Loss: 0.0129499, Validation MAE: 143.3599425, Test MAE: 104.6817957
Epoch: 004, LR: 0.001000, Loss: 0.0470941, Validation MAE: 89.2011484, Test MAE: 88.6568295
Epoch: 005, LR: 0.001000, Loss: 0.0113325, Validation MAE: 63.9636422, Test MAE: 63.2541860
...
Epoch: 296, LR: 0.000010, Loss: 0.0000911, Validation MAE: 5.0655050, Test MAE: 5.3100406
Epoch: 297, LR: 0.000010, Loss: 0.0000909, Validation MAE: 5.1383144, Test MAE: 5.3100406
Epoch: 298, LR: 0.000010, Loss: 0.0000907, Validation MAE: 5.1324929, Test MAE: 5.3100406
Epoch: 299, LR: 0.000010, Loss: 0.0000909, Validation MAE: 5.0166510, Test MAE: 5.2263104
Epoch: 300, LR: 0.000010, Loss: 0.0000907, Validation MAE: 5.3334611, Test MAE: 5.2263104

Target 0:

Epoch: 001, LR: 0.001000, Loss: 0.4338519, Validation MAE: 0.5790149, Test MAE: 0.5825766
Epoch: 002, LR: 0.001000, Loss: 0.2424187, Validation MAE: 0.4568612, Test MAE: 0.4578967
Epoch: 003, LR: 0.001000, Loss: 0.1836276, Validation MAE: 0.3748054, Test MAE: 0.3785959
Epoch: 004, LR: 0.001000, Loss: 0.1361738, Validation MAE: 0.3760909, Test MAE: 0.3785959
Epoch: 005, LR: 0.001000, Loss: 0.1016096, Validation MAE: 0.2895398, Test MAE: 0.2939654
...
Epoch: 296, LR: 0.000010, Loss: 0.0015362, Validation MAE: 0.0599686, Test MAE: 0.0610195
Epoch: 297, LR: 0.000010, Loss: 0.0015365, Validation MAE: 0.0599787, Test MAE: 0.0610195
Epoch: 298, LR: 0.000010, Loss: 0.0015354, Validation MAE: 0.0599160, Test MAE: 0.0611275
Epoch: 299, LR: 0.000010, Loss: 0.0015351, Validation MAE: 0.0599488, Test MAE: 0.0611275
Epoch: 300, LR: 0.000010, Loss: 0.0015351, Validation MAE: 0.0599262, Test MAE: 0.0611275

Since the MAE of target 0 is acceptable, I think the model and training process is correct, and the problem may be in the dataset. I have checked the unit, both of them are "eV".

I cannot figure out what the problem is, so I ask for your help. Thanks in advance.

[Powerd0g]

i got the same problem as you. Do you solve it？

[rusty1s]

Can you also test target 13? I am not totally sure but I remember that for example DimeNet uses target 12 instead of target 7, see https://github.com/pyg-team/pytorch_geometric/blob/master/examples/qm9_pretrained_dimenet.py.

[whyydsforever]

I got the same issue, I got large MAE loss using Schnet model. However, the pretrained model can achieve the performance as reported in the paper. I followed the instruction of qm9_nn_conv.py too.

Running experiment for SchNetModel, training on 110000 samples for 500 epochs.

Model architecture: SchNetModel(hidden_channels=128, num_filters=128, num_interactions=6, num_gaussians=50, cutoff=10) Total parameters: 455809

Start training: Epoch: 010, LR: 0.000100, Loss: 0.0010925, Val MAE: 23.2841089, Test MAE: 16.5220145 Epoch: 020, LR: 0.000100, Loss: 0.0008374, Val MAE: 17.8200987, Test MAE: 8.5787781 Epoch: 030, LR: 0.000100, Loss: 0.0004474, Val MAE: 12.1935564, Test MAE: 5.7325794 Epoch: 040, LR: 0.000100, Loss: 0.0003067, Val MAE: 8.1412306, Test MAE: 3.9658791 Epoch: 050, LR: 0.000100, Loss: 0.0001964, Val MAE: 3.2376000, Test MAE: 3.1256540 Epoch: 060, LR: 0.000100, Loss: 0.0002505, Val MAE: 4.6266191, Test MAE: 3.1256540 Epoch: 070, LR: 0.000100, Loss: 0.0001756, Val MAE: 19.2558215, Test MAE: 3.1256540 Epoch: 080, LR: 0.000080, Loss: 0.0001542, Val MAE: 3.5565595, Test MAE: 3.1256540 Epoch: 090, LR: 0.000080, Loss: 0.0001435, Val MAE: 2.6551781, Test MAE: 2.1585150 Epoch: 100, LR: 0.000080, Loss: 0.0001710, Val MAE: 3.0050470, Test MAE: 1.9153853 Epoch: 110, LR: 0.000080, Loss: 0.0001386, Val MAE: 7.1951441, Test MAE: 1.9153853 Epoch: 120, LR: 0.000080, Loss: 0.0001058, Val MAE: 4.7333981, Test MAE: 1.9153853 Epoch: 130, LR: 0.000064, Loss: 0.0000726, Val MAE: 2.7598097, Test MAE: 1.8128454 Epoch: 140, LR: 0.000064, Loss: 0.0000598, Val MAE: 3.0954379, Test MAE: 1.7436448 Epoch: 150, LR: 0.000064, Loss: 0.0000621, Val MAE: 2.7002962, Test MAE: 1.7436448 Epoch: 160, LR: 0.000051, Loss: 0.0000587, Val MAE: 2.3588643, Test MAE: 1.7436448 Epoch: 170, LR: 0.000051, Loss: 0.0000362, Val MAE: 4.6948277, Test MAE: 1.5140756 Epoch: 180, LR: 0.000051, Loss: 0.0000441, Val MAE: 5.9637215, Test MAE: 1.5140756 Epoch: 190, LR: 0.000051, Loss: 0.0000402, Val MAE: 1.6962807, Test MAE: 1.2457897 Epoch: 200, LR: 0.000051, Loss: 0.0000554, Val MAE: 2.1899763, Test MAE: 1.2457897 Epoch: 210, LR: 0.000051, Loss: 0.0000451, Val MAE: 1.3153362, Test MAE: 1.2457897 Epoch: 220, LR: 0.000041, Loss: 0.0000271, Val MAE: 1.6840457, Test MAE: 1.2457897 Epoch: 230, LR: 0.000041, Loss: 0.0000286, Val MAE: 4.3280940, Test MAE: 1.2457897 Epoch: 240, LR: 0.000033, Loss: 0.0000176, Val MAE: 1.4835783, Test MAE: 1.2457897 Epoch: 250, LR: 0.000033, Loss: 0.0000157, Val MAE: 2.5439449, Test MAE: 1.2457897 Epoch: 260, LR: 0.000033, Loss: 0.0000200, Val MAE: 2.5516019, Test MAE: 0.9603749 Epoch: 270, LR: 0.000033, Loss: 0.0000204, Val MAE: 2.6618340, Test MAE: 0.9603749 Epoch: 280, LR: 0.000033, Loss: 0.0000175, Val MAE: 1.9810213, Test MAE: 0.9603749 Epoch: 290, LR: 0.000026, Loss: 0.0000094, Val MAE: 2.7670695, Test MAE: 0.8689007 Epoch: 300, LR: 0.000026, Loss: 0.0000117, Val MAE: 1.6332636, Test MAE: 0.8689007 Epoch: 310, LR: 0.000026, Loss: 0.0000099, Val MAE: 1.2873657, Test MAE: 0.8484793 Epoch: 320, LR: 0.000026, Loss: 0.0000122, Val MAE: 0.9789019, Test MAE: 0.8484793 Epoch: 330, LR: 0.000026, Loss: 0.0000140, Val MAE: 2.1227426, Test MAE: 0.8484793 Epoch: 340, LR: 0.000026, Loss: 0.0000121, Val MAE: 1.0345773, Test MAE: 0.6861685 Epoch: 350, LR: 0.000026, Loss: 0.0000139, Val MAE: 0.9370375, Test MAE: 0.6861685 Epoch: 360, LR: 0.000021, Loss: 0.0000098, Val MAE: 1.3877939, Test MAE: 0.6861685 Epoch: 370, LR: 0.000021, Loss: 0.0000075, Val MAE: 1.3967029, Test MAE: 0.6861685 Epoch: 380, LR: 0.000021, Loss: 0.0000079, Val MAE: 1.3510868, Test MAE: 0.6861685 Epoch: 390, LR: 0.000017, Loss: 0.0000061, Val MAE: 0.9012446, Test MAE: 0.6861685 Epoch: 400, LR: 0.000017, Loss: 0.0000065, Val MAE: 1.7884143, Test MAE: 0.6861685 Epoch: 410, LR: 0.000017, Loss: 0.0000062, Val MAE: 0.7813370, Test MAE: 0.6511383 Epoch: 420, LR: 0.000017, Loss: 0.0000058, Val MAE: 0.8835795, Test MAE: 0.6511383 Epoch: 430, LR: 0.000013, Loss: 0.0000034, Val MAE: 0.6644465, Test MAE: 0.6511383 Epoch: 440, LR: 0.000013, Loss: 0.0000040, Val MAE: 1.7867371, Test MAE: 0.6511383 Epoch: 450, LR: 0.000013, Loss: 0.0000038, Val MAE: 2.7047916, Test MAE: 0.6511383 Epoch: 460, LR: 0.000011, Loss: 0.0000035, Val MAE: 0.5817563, Test MAE: 0.5104253 Epoch: 470, LR: 0.000011, Loss: 0.0000032, Val MAE: 1.5761049, Test MAE: 0.5104253 Epoch: 480, LR: 0.000011, Loss: 0.0000029, Val MAE: 3.9288855, Test MAE: 0.5104253 Epoch: 490, LR: 0.000010, Loss: 0.0000032, Val MAE: 1.1278518, Test MAE: 0.5104253 Epoch: 500, LR: 0.000010, Loss: 0.0000018, Val MAE: 4.1070732, Test MAE: 0.5104253