NanoCode012

Hey, on off chance, have you tried with 2 (or even number) of GPUs?

Thanks for sharing your experience. Please let us know if you manage to narrow down the cause as I'm quite confused too. By resume training, do you mean `resume_checkpoint: `...

@nicoboss , thanks for looking into this. Just to clarify, the blog didn't have an attached solution, correct?

That's an interesting find. Unfortunately, I'm not sure how we could repro this to look into it 😕 Are you running bare metal or from our docker image?

Your request is to be able to load (and freeze) an adapter while training a separate one right? I'm open to a PR on this as PEFT does support multi-adapter....