Can't upload custom TF models e.g. TFBERT, TFRoBERTa due to model.save() failing

[ck37]

Hello,

I'm interested in sharing models on the HF Hub but it appears that custom TF models can't be shared because their model.save() doesn't work (appears to be a known issue: https://github.com/huggingface/transformers/issues/13610#issuecomment-921134250).

I've uploaded a Jupyter notebook to colab which shows the error for simple RoBERTA and BERT models. Note: I ran this in jupyter rather than Colab so it would probably require a little editing to work directly in Colab.

Is it possible to also use save_pretrained() for this use case as well? Even if so it would be good to resolve the model.save() issue for the TF user base, e.g. others are struggling with this https://discuss.tensorflow.org/t/list-index-out-of-range-while-saving-a-trained-model/6901

cc: @Rocketknight1 @LysandreJik

Thanks, Chris

[LysandreJik]

Hey @ck37, @Rocketknight1 can validate but I believe this was fixed recently, and model.save now works correctly.

The TF models from transformers have a push_to_hub method which you may leverage to push your models!

[ck37]

Hmm I tried reinstalling transformers from the main branch head (4.16.0.dev0) but I'm getting the same error - sorry if I'm missing something. I'm using TF 2.6.2 currently.

[Rocketknight1]

Investigating now, pinging @nateraw and @gante on this one too! I've confirmed that there is an underlying issue here where model.save is failing on Keras sequential/functional models containing HF models as layers. This is an issue I'd hoped was fixed already (see #14361 in transformers), but obviously there are some rough edges here that still aren't working correctly.

[Rocketknight1]

Actually, wait, I think the issue may be with some of the example code! Replacing hf_result = transformer_model([input_word_ids, input_mask]) with hf_result = transformer_model(input_word_ids, input_mask) causes one of the examples to save correctly.

This is still quite cryptic for users, though, as the model runs correctly either way.

[ck37]

Interesting, confirmed that it resolves the problem for me - appreciate it 🙏 .

And yep, the documentation says that inputs can be passed as a list in the first argument, so it would be good to support that form of calling the model (or at least try to catch if that form has been used and give a clearer notice to the user): https://huggingface.co/docs/transformers/model_doc/roberta#transformers.TFRobertaModel

But on my end I'm happy to just change this in the short term, thanks for the fast investigation!

[Rocketknight1]

Yeah, this is something we should either support fully, or be very clear that we don't support - something that works half the time but breaks when you try to save is very messy for everyone. I'll make a note in our internal TF roadmap and see if we can come up with a proper solution.

[Wauplin]

Hi @ck37 @Rocketknight1 , is this issue fully solved by the PR mentioned above ? https://github.com/huggingface/transformers/pull/15074 If yes, we can close this issue.

[Rocketknight1]

I believe so! I'm working on documenting the pitfalls of model.save() with functional models, so hopefully we can avoid issues like this in future.