Question about the performance of ARML

[peterdarkdarkgogo]

I tried to run the ARML training and inference code, and my result on plainmulti dataset seems not match with the result on paper.

In my experiment, the acc of plainmulti dataset 0 to 3 are 0.56, 0.33, 0.55, 0.43, which are quite different from the paper result, which are 62:33 ± 1:47% 35:65 ± 1:40% 58:56 ± 1:41% 44:82 ± 1:38. I used the trained ARML model49999 as the inference model.

>>> arr = [0.20019874, 0.5173999 , 0.57059884, 0.5827988 , 0.5869986 ,
...        0.5877987 , 0.5879988 , 0.5873988 , 0.5883988 , 0.58859867,
...        0.5877988 , 0.5869989 , 0.58699894, 0.58679897, 0.5875988 ,
...        0.5877989 , 0.5889989 , 0.5881989 , 0.5885988 , 0.5877989 ,
...        0.58819896]
>>> sum(arr) / len(arr)
0.564951258095238
>>> arr = [0.19799837, 0.30459967, 0.32159966, 0.34019986, 0.33839968,
...        0.33979967, 0.33959967, 0.3409997 , 0.34059966, 0.3403997 ,
...        0.33979973, 0.3399998 , 0.3399998 , 0.34019983, 0.33979982,
...        0.34079984, 0.34119982, 0.34099984, 0.34079984, 0.34079984,
...        0.34119982]
>>> sum(arr) / len(arr)
0.33094255333333333
>>> arr = [0.18879882, 0.48420003, 0.54219973, 0.5623991 , 0.572999  ,
...        0.5753989 , 0.5757989 , 0.575399  , 0.57559896, 0.5773989 ,
...        0.57719886, 0.57699883, 0.5773987 , 0.5785988 , 0.5781988 ,
...        0.57819885, 0.57819885, 0.5785988 , 0.5781988 , 0.57839876,
...        0.57819873]
>>> sum(arr) / len(arr)
0.5518275295238095
>>> arr = [0.19559856, 0.38499972, 0.41160023, 0.44420016, 0.4516001 ,
...        0.45320007, 0.45400003, 0.45319998, 0.45199996, 0.4528    ,
...        0.4526    , 0.452     , 0.4526    , 0.45340002, 0.4534    ,
...        0.45399997, 0.45419994, 0.45379996, 0.45360002, 0.45319998,
...        0.453     ]
>>> sum(arr) / len(arr)
0.4351904142857143

Would you mind sharing some advice ? Thank you. @huaxiuyao

[prantikhowlader]

HI @huaxiuyao I am getting very similar results. The validation results for CUB: Mean validation accuracy/loss, stddev, and confidence intervals (array([0.19819896, 0.44620022, 0.5184002 , 0.53419966, 0.53259957, 0.5337997 , 0.53319967, 0.5355997 , 0.5349997 , 0.5359998 , 0.53719974, 0.5373997 , 0.5379998 , 0.5375998 , 0.53719985, 0.5369998 , 0.5369998 , 0.5377998 , 0.53819984, 0.53859985, 0.5377998 ], dtype=float32), array([0.16768113, 0.2099184 , 0.23677287, 0.23330276, 0.23472783, 0.23181376, 0.23034328, 0.2305049 , 0.23077096, 0.2309638 , 0.23146595, 0.23125988, 0.23116252, 0.23174632, 0.23129322, 0.2308043 , 0.23063095, 0.23050198, 0.23095587, 0.23054318, 0.23050196], dtype=float32), array([0.01039298, 0.01301088, 0.01467534, 0.01446026, 0.01454858, 0.01436797, 0.01427682, 0.01428684, 0.01430333, 0.01431528, 0.01434641, 0.01433364, 0.0143276 , 0.01436379, 0.0143357 , 0.0143054 , 0.01429466, 0.01428666, 0.01431479, 0.01428921, 0.01428666], dtype=float32)) Can you please tell us how to calculate the actual accuracy, from the above