Phil Wang comments

Results 1516 comments of


                                            Phil Wang

Trained on small dataset with pre-trained weight, don't have good result.

Hi James, attention excels in the regime of big data, as shown in the paper. However, I am curious why fine tuning did not work. Are you using Ross' model?...

Trained on small dataset with pre-trained weight, don't have good result.

@JamesQFreeman I think fine-tuning from a pretrained model should generally work well. maybe you should raise the issue with him

Trained on small dataset with pre-trained weight, don't have good result.

@JamesQFreeman ohh... well, I think I spot the error, your learning rate is way too high `1e-2`, try Karpathy's favorite LR, `3e-4`

Train result on my own dataset. A Big gap between Train and Valid dataset

@Erichen911 1200 is not enough! Off by 3 orders of magnitude at least!

Train result on my own dataset. A Big gap between Train and Valid dataset

@Erichen911 I would recommend getting a huge amount of images, preferrably a million at least, and then doing self-supervised learning with BYOL, before training on your tiny training set otherwise,...

Model doesn't converge

@liberbey Hey Ahmet! One of pitfalls of transformers is having settings that result in the dimension per head to be too small. The dimension per head should be at least...

Model doesn't converge

@liberbey depth should be at a minimum of 6

Model doesn't converge

@liberbey I think the only option is to get a bunch of unlabelled images (in the millions) and do self-supervised learning with BYOL before fine-tuning on your dataset. Transformers only...

Model doesn't converge

@SuX97 @liberbey well, there's been a new development, you two should try https://github.com/lucidrains/vit-pytorch#distillation

Model doesn't converge

you'll both still need at least a million images... haha