aitextgen

aitextgen copied to clipboard

Huggingface is adding PyTorch-based GPT-Neo support via https://github.com/huggingface/transformers/pull/10848

That's just the superlarge models (1.3B and 2.7B). If performance/support is good (since this is the only practical way to get a GPT-3 analogous architecture), I am open to doing the necessary work to add it to aitextgen. (it shouldn't be too much though since the defaults between GPT-2 and GPT-Neo are similar, but will have to add some config metadata)

Also depends on DeepSpeed and ONNX support, which won't be automatic.

Since that PR is now merged and there's already blog posts talking about finetuning on GPT Neo, I supposed I'll have to add it at some point.

The 1.5B Neo might be fussy; Ideally someone will train a smaller GPT-Neo for testing.

Now in released Transformers so can test.

There is a released 125M model comparable to GPT-2's 124M model. Will test if finetuning works out of the box. (it should)

Due to me being stupid, I hardcoded a lot of GPT2LMHeadModel which unfortunately causes this to not work out of the box.

I probably need to go back to AutoConfig so transformers can infer the correct model.

just curious whether this still support to train GPT-Neo from scratch? like GPT2 in aitextgen does. specifically can it be trained on a nvidia GPU with 8G memory (like 3060Ti)

So it appears there's a slightly increased memory overhead for training GPT Neo (could also be a function of that it's new and less optimized)

When finetuning the 125M model in Colab it hit about 10GB VRAM so that may not work well on a 8GB VRAM GPU. (although a 3060Ti should support fp16 so it might work with that)