[Feature request] Add the Noisy Students pretrained weights

[Cyril9227]

Hello @lukemelas ,

Thank's for your amazing work. Do you have any plans to incorporate the pretrained weights using noisy students scheme? Weights are available for the TPU / Tensorflow version, not really sure how to port these into Pytorch tho.

Thank's !

[chenshen03]

You can try the conversion script in the tf_to_pytorch directory.

[lukemelas]

Yes, I'll add the noisy students version soon. I'm a bit swamped with work at the moment, but I'll convert the weights, run the tests, and make them available as soon as I get a chance.

[Cyril9227]

Yes, I'll add the noisy students version soon. I'm a bit swamped with work at the moment, but I'll convert the weights, run the tests, and make them available as soon as I get a chance.

Nice ! Looking forward to it, thanks

[Selimonder]

Yes, I'll add the noisy students version soon. I'm a bit swamped with work at the moment, but I'll convert the weights, run the tests, and make them available as soon as I get a chance.

+1 looking forward. Thanks!

[Soonhwan-Kwon]

You can try the conversion script in the tf_to_pytorch directory.

It seems that leads to error that it can't find the missing keys

[authman]

I tried porting it too. Most of the errors I got were patched using tf.compat.v1 wherever it complained—except tf.train.Saver(), which apparently doesn't exist.

EDIT: was able to get around that too with:

import tensorflow.compat.v1 as tf1
tf1.disable_v2_behavior()

And then using tf1 for the .Saver methods, and using regular tf (2) for all other calls. For reference, here is the noisy student b3 pytorch model. Hasn't been tested but this is what was spit out of the conversion script w/o error: https://drive.google.com/file/d/1bOYh7KkhQfW1HQwzSzyqOuxdITaRpCzq/view?usp=sharing

[chenshen03]

You can try the conversion script in the tf_to_pytorch directory.

It seems that leads to error that it can't find the missing keys

The network structure of NoisyStudent and EfficientNet is the same, so you can add key fields similar to EfficientNet and other conditional judgments to the conversion codes.

[Soonhwan-Kwon]

@authman @chenshen03 I finally make it work with tensorflow==1.14.0, and it works perfectly Thank you

[Cyril9227]

I tried porting it too. Most of the errors I got were patched using tf.compat.v1 wherever it complained—except tf.train.Saver(), which apparently doesn't exist.

EDIT: was able to get around that too with:

import tensorflow.compat.v1 as tf1
tf1.disable_v2_behavior()

And then using tf1 for the .Saver methods, and using regular tf (2) for all other calls. For reference, here is the noisy student b3 pytorch model. Hasn't been tested but this is what was spit out of the conversion script w/o error: https://drive.google.com/file/d/1bOYh7KkhQfW1HQwzSzyqOuxdITaRpCzq/view?usp=sharing

Has someone tested it already ?

Thank's tho !

[authman]

I've tested it and it works great. I've also converted b4 if anyone is interested.

[Cyril9227]

I've tested it and it works great. I've also converted b4 if anyone is interested.

Cool, thank's for the feedback, will give it a try unless @lukemelas releases an official version for them

[seefun]

I've tested it and it works great. I've also converted b4 if anyone is interested.

Cool! Could you share your weight file?

[laurentdillard]

I've tested it and it works great. I've also converted b4 if anyone is interested.

Is it possible to get the converted weights for b4?

[authman]

https://drive.google.com/file/d/1hORyc4b8_QPs1O67br5T_a9XeB92ICkF/view?usp=sharing B4 noisy student pth.

[ngessert]

Any updates on this? Did anyone convert b5 and the larger ones already?

[authman]

You guys can easily convert the files yourselves. There is a script in this very repo:

https://github.com/lukemelas/EfficientNet-PyTorch/tree/master/tf_to_pytorch

Use it to download tf weights and convert to .pth. For example, I navigated to https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet, and downloaded NoisyStudent + RA EfficientNet's B2 tensorflow .ckpt. Untar'd the file into the pretrained_tensorflow directory.

Then ran this:

python load_tf_weights.py --model_name efficientnet-b2 --tf_checkpoint ../pretrained_tensorflow/noisy-student-efficientnet-b2/ --output_file ../pretrained_pytorch/noisy-student-efficientnet-b2.pth which resulted in a .pth file ready to rock and roll.

If you have errors with load_tf_weights.py due to it or .any of the other scripts complaining about being written in tf1.x, here's a copy of the code I've altered such that it runs just fine with tensorflow-2.2.0: https://drive.google.com/file/d/11jDyfRKoIhVpuNsFSeR_SAFe0e0R0fA1/view?usp=sharing

Maybe some less lazy than I can take the code and turn it into a PR?

[jaideepmurkute]

Hello @authman , Thanks for sharing your script for TF-2.2.0. When I try the same procedure you mentioned above, I get an error reading the checkpoint. I get the same error with the original script and TF-1.15.0 as well. I have verified that directories and extracted checkpoints do exist at the mentioned paths. The function tf.train.latest_checkpoint(checkpoint_dir) is returning None for some reason. Any suggestions?

Script that I run:

python3 load_tf_weights.py \
--model_name efficientnet-b4 \
--tf_checkpoint ../pretrained_tensorflow/noisy-student-efficientnet-b4/ \
--output_file ../pretrained_pytorch/noisy-student-efficientnet-b4.pth

Traceback (most recent call last):
  File "load_tf_weights_tf2.py", line 167, in <module>
    load_and_save_temporary_tensorflow_model(args.model_name, args.tf_checkpoint)
  File "load_tf_weights_tf2.py", line 138, in load_and_save_temporary_tensorflow_model
    eval_ckpt_driver.restore_model(sess, model_ckpt)
  File "original_tf/eval_ckpt_main.py", line 92, in restore_model
    saver.restore(sess, checkpoint)
  File "/home/jvm6526/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.