OLMo icon indicating copy to clipboard operation
OLMo copied to clipboard
verified publisher icon allenai

Gradient Checkpointing

Open fakerybakery opened this issue 1 year ago • 2 comments

Hi, I'm trying to finetune OLMo but running into the error ValueError: OLMoForCausalLM does not support gradient checkpointing. Is this planned in the future?

Thanks for releasing OLMo!

fakerybakery avatar Apr 18 '24 02:04 fakerybakery

We just released OLMo integration into the transformers library (v4.40.0 and up), with corresponding -hf checkpoints on Huggingface Hub (e.g. https://huggingface.co/allenai/OLMo-1.7-7B-hf). I haven't tried gradient checkpointing there, but it may work.

2015aroras avatar Apr 19 '24 17:04 2015aroras

I confirmed it does not work. This would a great addition.

bdytx5 avatar May 20 '24 20:05 bdytx5