Phil Wang

@mark-selyaeff Hey Mark! So the LR in the paper is actually specific to LARS I believe For Adam, try using a smaller learning rate (3e-4)

@SilverUnicorn Hi! thanks for reporting this! would you like to give 0.5.3 a try? I implemented the solution that @Vurkty linked to!

https://github.com/lucidrains/byol-pytorch/commit/8b0be4859305c6635a18b36e17d7bec85c1c9c9e

hmm I'm not too sure actually, does it work if you restrict it to 1 gpu? `CUDA_VISIBLE_DEVICES=0 python train.py`?

@jlindsey15 ok, i think i know why. https://github.com/lucidrains/byol-pytorch/commit/e3c245311bcfee58982d1a63e66587f9b302ba31 can you try again?

@jlindsey15 ahh, I believe you must have some greyscale images (mixed in with colored images) in your folder

@jlindsey15 ok, try again, this should fix the greyscale image problem, but it's probably better that you make sure it isn't auto-including some image you don't want pretraining on https://github.com/lucidrains/byol-pytorch/commit/87232dc89bd6b51fdf24d5174c20f806bad5b98e

https://github.com/lucidrains/byol-pytorch/commit/6b6a167b4f154861c55bc8a06ede5c71e46c6e0e ok, put in another fix lol

@jlindsey15 yea, i'm not sure about that one doing a quick google brings me to https://github.com/pytorch/fairseq/issues/1608

do you want to give the accepted solution on that page a try?