transformers

transformers copied to clipboard

Hello community, I am having the same problem described to save and load a fine tuned model using transformers and tensorflow. I have used save_pretrained, save_weights and model.save with save_format=tf. I have been able to load the model with from_pretrained but it loads no weights and when I perform evaluation performance is too low compared when training while the fine tuned model is in memory. You could check my code at GitHub in LeninGF/clasificaion_robos_fge in model_train_huggingface.ipynb and evaluate Notebook.ipynb

cc @Rocketknight1 @gante

Hi @LeninGF 👋 I had a look into your notebooks but they are very long, which makes it very hard to pin the problem. Would you be able to share a short notebook (as short as possible) where the problem can be reproduced? Thanks :)

Hi Huggingface/Transformers I will do it . The only problem is that I will use other dataset different from the one I am working because if privacy policy... Give some hours to upload it

On Mon, Aug 1, 2022, 4:18 PM Joao Gante @.***> wrote:

Hi @LeninGF https://github.com/LeninGF 👋 I had a look into your notebooks but they are very long, which makes it very hard to pin the problem. Would you be able to share a short notebook (as short as possible) where the problem can be reproduced? Thanks :)

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/18214#issuecomment-1201732303, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TWKIUSR6BG5KVBRGMBO3VXA5KPANCNFSM54DTBWPA . You are receiving this because you were mentioned.Message ID: @.***>

Hi Joao Please you can check my training code here: https://colab.research.google.com/gist/LeninGF/89234ab4ba45147d34b8e8657caff761/model_train_huggingface_gitnew.ipynb

I think that the problem should happen with any dataset used. I am using a multi labelled dataset. For company reasons I am not yet able to share it. Please let me know if you would need a sample of it to reproduce the problem.

The following colab shows how I am trying to train the model https://colab.research.google.com/gist/LeninGF/89234ab4ba45147d34b8e8657caff761/model_train_huggingface_gitnew.ipynb

The following colab shows that I am trying to load the weights of the trained model to test it again with the test set by using a new colab notebook

https://colab.research.google.com/gist/LeninGF/08b2824b73692134ec27979a7e6011ea/testingsavedfthfmodel.ipynb

you can reach me at @.*** too

The problem is as follows: it does not matter how I train the model. While the notebook where it was trained is active, you can see that the model.evaluate(test_dataset) achieves a satisfactory 0.8 accuracy (even though there is some overfitting) However, once I saved the model and I try to load it again, it does not work and you can see that repeating the weights load, model compile and model evaluate gives me an accuracy off 0.08

thanks for your kind help. If It is not to bother you a lot I am trying to replicate this problem using the tweet emotion dataset that huggingface has, I can send you the gist if you agree. I have already trained the model and I am about to test if the downloaded model will be working

Atentamente,

Lenin Falconí Estrada

El lun, 1 ago 2022 a las 16:18, Joao Gante @.***>) escribió:

Hi @LeninGF https://github.com/LeninGF 👋 I had a look into your notebooks but they are very long, which makes it very hard to pin the problem. Would you be able to share a short notebook (as short as possible) where the problem can be reproduced? Thanks :)

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/18214#issuecomment-1201732303, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TWKIUSR6BG5KVBRGMBO3VXA5KPANCNFSM54DTBWPA . You are receiving this because you were mentioned.Message ID: @.***>

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.