layer-to-layer-pytorch icon indicating copy to clipboard operation
layer-to-layer-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon TezRomacH

GPT-2 training example

Open loretoparisi opened this issue 4 years ago • 3 comments

Checklist

  • [X] I've searched the project's issues.

❓ Question

Is it possible to provide an example of training GPT-2 with L2L?

📎 Additional context

I'm currently using the HuggingFace training script for train the GPT-2 model here.

loretoparisi avatar Sep 16 '20 07:09 loretoparisi

Hello @loretoparisi, thank you for your interest in our work!

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

github-actions[bot] avatar Sep 16 '20 07:09 github-actions[bot]

Hi! In HuggingFace models, layers are not in one ModuleList as required by L2L (and, for example, DeepSpeed).

But I will try to make such an example with GPT-2 👍

TezRomacH avatar Sep 16 '20 07:09 TezRomacH

Hi! In HuggingFace models, layers are not in one ModuleList as required by L2L (and, for example, DeepSpeed).

But I will try to make such an example with GPT-2 👍

Will be good to make a translator for HuggingFace models into the L2L ModuleList format.

drpatrickkaggle avatar Oct 23 '20 03:10 drpatrickkaggle