Bug:I just can not get the same result after running get_pred func!!!!

[yifeiSunny]

I just can not get the same result after running get_pred func!!!! https://colab.research.google.com/github/timeseriesAI/tsai/blob/master/tutorial_nbs/01_Intro_to_Time_Series_Classification.ipynb I get the 0.9278 after running learn.get_preds first time , but i get 0.7111 then i run learn.get_preds again?????

[oguiza]

Hi @yifeiSunny, The issue seems to be that you are using an old version of tsai (0.2.15) when we are in 0.3.1. You should try updating it an running it again. You can use this code at the top of the notebook you are using:

# **************** UNCOMMENT AND RUN THIS CELL IF YOU NEED TO INSTALL/ UPGRADE TSAI ****************
stable = True # Set to True for latest pip version or False for main branch in GitHub
!pip install {"tsai -U" if stable else "git+https://github.com/timeseriesAI/tsai.git"} >> /dev/null

[yifeiSunny]

Hi @oguiza I really appreciate that you could offer help for me! But I am very sure that i am using the 0.3.1 tsai since i just check it.

Maybe I shoule express my question more clearly. when i run the code after(the code is from tutorial and the link is in my first comment):

learn = load_learner_all(path='export', dls_fname='dls', model_fname='model', learner_fname='learner')
dls = learn.dls
valid_dl = dls.valid
b = next(iter(valid_dl))
b

then i run the code after:

valid_probas, valid_targets, valid_preds = learn.get_preds(dl=dls.valid,reorder=False,with_decoded=True)
valid_probas, valid_targets, valid_preds
(targets == preds).float().mean()

we get the result 0.922 **However!**when i run the code above again

valid_probas, valid_targets, valid_preds = learn.get_preds(dl=dls.valid,reorder=False,with_decoded=True)
valid_probas, valid_targets, valid_preds
(targets == preds).float().mean()

I GET 0.7889 and the valid_probas is also different. Then i run it third time and i get the 0.7889 which will be stable in the following times i run the code above. This happen no matter in the tutorial or in my own computer. Hoping for your reply and best wishes!

[oguiza]

I cannot see the image you posted. But the notebook you shared with me shows this:

tsai       : 0.2.15
fastai     : 2.2.5
fastcore   : 1.3.19
torch      : 1.7.0+cu101

[yifeiSunny]

I add the code

# **************** UNCOMMENT AND RUN THIS CELL IF YOU NEED TO INSTALL/ UPGRADE TSAI ****************
stable = True # Set to True for latest pip version or False for main branch in GitHub
!pip install {"tsai -U" if stable else "git+https://github.com/timeseriesAI/tsai.git"} >> /dev/null

and i get the environment

os             : Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
python         : 3.7.13
tsai           : 0.3.1
fastai         : 2.6.3
fastcore       : 1.4.4
torch          : 1.11.0+cu113
device         : cpu
cpu cores      : 2
RAM            : 12.68 GB
GPU memory     : N/A

and i run the tutorial again But i still get the bug i posted.

[lwachowiak]

Hi, I am having the same problem as @yifeiSunny The first reported performance (0.8514) is the same as reported for the last training epoch. After running get_preds a second time, however, this performance changes (0.8283). If I at this point run get_preds again and again, I will always get 0.8283. The results from the confusion matrix at that point also give me 0.8283.

Any idea what is happening?

[zmce2018]

Hello @lwachowiak

Have you tried get_X_preds function？

[lwachowiak]

Hi @zmce2018, Thanks for the input. If I run get_X_preds() multiple times, the accuracy stays the same, i.e., it works! As soon as I run get_preds a single time (it is correct on the first trial), the next iteration of either get_X_preds() or get_preds() gives me a wrong, much lower accuracy (see 1st screenshot).

So overall, get_X_preds() would be a good work around. However, I still have the problem that the code from the ClassificationInterpretation gives me a confusion matrix with the wrong accuracy that I also get after running get_preds() --- even if I create the confusion matrix before running get_preds() myself (see 2nd screenshot). I am creating my own confusion matrix for now, but there seems to be some bug in the library.

[lwachowiak]

So here is some further info in case you are working on fixing that bug. I reproduced the same error in Google Colab, so it does not seem to be machine specific. Moreover, even get_X_preds() does not work in the case that you save everything beforehand! However, after you reload your saved model and dataset, the predictions are correct again. You can see an example in this screenshot where the accuracy suddenly changes after saving and then is correct again as in the report of the last epoch after reloading:

[zmce2018]

Hi Lennart

I am just a active user of TSAI. I am sure some one will fix this bug shortly.

Regards

Lennart Wachowiak @.***>于2022年6月29日 周三20:26写道：

So here is some further info in case you are working on fixing that bug. I reproduced the same error in Google Colab, so it does not seem to be machine specific. Moreover, even get_X_preds() does not work in the case that you save everything beforehand! However, after you reload your saved model and dataset, the predictions are correct again. You can see an example in this screenshot where the accuracy suddenly changes after saving and then is correct again as in the report of the last epoch after reloading: [image: image] https://user-images.githubusercontent.com/71786387/176414477-c8b49e07-776e-41fe-b32a-846b2b43df6b.png

— Reply to this email directly, view it on GitHub https://github.com/timeseriesAI/tsai/issues/540#issuecomment-1169809889, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXFFFLBKABOFURH6CKUNY3VRQQELANCNFSM5YIQL27A . You are receiving this because you were mentioned.Message ID: @.***>

-- Chunrui Liu Email address: @.***

[huameichen0523]

If I use Learner.save() and Learner.load() to save and load the saved model, even get_X_preds() outputs the wrong values. But Learner.export() and load_learner() work correctly.

[ejokhan]

Hi, Did anyone solve this problem? I am getting the same issue... when I run it the first time, I get the same accuracy as the training shows. But re-running it drops it to a meager value.

[huameichen0523]

1. You need to save the model using learn.export() and load it with load_leaner()
2. Use learn.get_X_preds() for inference.

Hope it works for you.

On Thu, Sep 15, 2022 at 2:20 PM ejokhan @.***> wrote:

Hi, Did anyone solve this problem? I am getting the same issue... when I run it the first time, I get the same accuracy as the training shows. But re-running it drops it to a meager value.

— Reply to this email directly, view it on GitHub https://github.com/timeseriesAI/tsai/issues/540#issuecomment-1248445776, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG2APCYZKR4MXGV5V64MBVDV6NSG7ANCNFSM5YIQL27A . You are receiving this because you commented.Message ID: @.***>

[oguiza]

Hi all, I've been investigating and there was indeed a bug. It should be fixed now. I'd appreciate it if you could confirm it's working well.

[oguiza]

Closed due to lack of response.

[alitirmizi23]

when loading using load_learner_all, why does the function expect EXACT SAME directory name and path as when we saved it? If i rename the export folder to lets say "export1", it doesn't work. Same with the model folders inside the export folder. If you could help with this pls

[oguiza]

when loading using load_learner_all, why does the function expect EXACT SAME directory name and path as when we saved it? If i rename the export folder to lets say "export1", it doesn't work. Same with the model folders inside the export folder. If you could help with this pls

Please, do not create a new issue within a closed one. Otherwise, it'll be disregarded. If you have an issue, create a new one and document it properly.