supervised-reptile
supervised-reptile copied to clipboard
openai
Cannot reproduce the results for 1-shot 5-way Mini-ImageNet
I used the command below to perform the experiment (environment: Python 3.6 / TensorFlow 1.10):
# 1-shot 5-way Mini-ImageNet.
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 15 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 5 --checkpoint ckpt_m15t
The last several lines of output:
batch 99950: train=0.200000 test=0.200000
batch 99960: train=0.000000 test=0.000000
batch 99970: train=0.200000 test=0.000000
batch 99980: train=0.200000 test=0.200000
batch 99990: train=0.200000 test=0.400000
Evaluating...
Train accuracy: 0.28016
Validation accuracy: 0.26512
Test accuracy: 0.24262
TensorBoard output:
However, in the paper, the test accuracy is around 47%.
Is there a chance the data you downloaded is incomplete or corrupted somehow? There should be 600 images per class, and there should be 100 classes in total. If you find that the data is in fact invalid (perhaps the download script is broken?), I can send you the data directly.
On Fri, Oct 26, 2018 at 12:51 PM Weijian Xu [email protected] wrote:
I used the command below to perform the experiment (environment: Python 3.6 / TensorFlow 1.10):
1-shot 5-way Mini-ImageNet.
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 15 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 5 --checkpoint ckpt_m15t
The last several lines of output:
batch 99950: train=0.200000 test=0.200000 batch 99960: train=0.000000 test=0.000000 batch 99970: train=0.200000 test=0.000000 batch 99980: train=0.200000 test=0.200000 batch 99990: train=0.200000 test=0.400000 Evaluating... Train accuracy: 0.28016 Validation accuracy: 0.26512 Test accuracy: 0.24262
TensorBoard output: [image: 1-shot 5-way reptile] https://user-images.githubusercontent.com/16413829/47580731-7a509c80-d904-11e8-9424-dd755f26241d.png
However, in the paper, the test accuracy is around 47%.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openai/supervised-reptile/issues/16, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYyBTtPkZn8J8Zf7f_sCD1Q3tGi3Lggks5uoz2TgaJpZM4X8mce .
It seems the dataset is indeed incomplete (see the output below). However, it does not lose too many images (38392/38400, 9593/9600, 11996/12000). Thus, I wonder if there are some other reasons behind the performance.
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./train -name '*.JPEG' | wc -l
38392
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./val -name '*.JPEG' | wc -l
9593
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./test -name '*.JPEG' | wc -l
11996
By the way, could you send me a copy of the data? Thank you very much!
See if 7f815bca856f96a04dc5519b3ad87d99b86d3104 fixes your problem. The command in the README didn't quite match the hyperparameters in the paper. The correct command is:
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 5 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 15 --checkpoint ckpt_m15t
@unixpickle By the way, 7f815bc changes --eval-batch 15 --train-shots 5 to --eval-batch 5 --train-shots 15. I could understand the change of eval-batch, which behaves the same as reducing the learning rate (to 1/3 of original one) in the inner loop during the evaluation stage. However, I wonder why train-shots should be increased. In fact, it seems train-shots=1 matches the best when considering the similarity between the training and the evaluation stage.
@xwjabc for whatever reason, we found that training on more "shots" helped Reptile's performance, probably because it allows you to take more diverse gradient steps during each inner-loop. Table 4 of Appendix A in the paper specifies the hyper-parameters, this included.
@unixpickle @xwjabc I had the same problem with the incomplete miniimagenet data downloaded from fetch_data.sh,most of folder is empty images. Could you send me a complete data？My email address is [email protected] ,thanks.
I used to have the dataset on my OpenAI machine and in the cloud. Unfortunately, I no longer have access to either copy. I'll see if I can find it sitting anywhere else, but I doubt I can.