fwdgrad icon indicating copy to clipboard operation
fwdgrad copied to clipboard
verified publisher icon orobix

Use FGD to fine-tune the transformer

Open cyl943123 opened this issue 2 years ago • 2 comments

Hi, Cool Work!

I'm curious about the performance of using the FGD to fine-tune the transformer on GLUE task do you have done it before?

Thanks!!

cyl943123 avatar Jul 14 '23 09:07 cyl943123

Hi @cyl943123, nope I haven't tried. I think this will be difficult: even for the simple MNIST a subtle change in the hyperparameters, the learning rate for example, led to instabilities. It will still be nice to see generalization to other tasks. Have you tried something in this regard?

belerico avatar Sep 16 '23 20:09 belerico