Matthew Stallone

icon indicating a website link stallone.io

icon company MIT-IBM Watson AI lab icon location Boston, Massachusetts icon location AI Architect @ MIT-IBM Watson AI Lab. Striving to create the best open AI models, yet

Results 2 comments of Matthew Stallone

Mixtral branch: What option should I choose when I want to do some finetuning after the merge?

Thanks for sharing, @cg123! Agreed, that is promising. Do you mind sharing your scripts you used for fine tuning? I am also very interested in this aspect of LLM research,...

[big-refactor] Accelerate launch FSDP Runtime Error

Same issue here as well with `TRANSFORMER_BASED_WRAP`: `RuntimeError: 'weight' must be 2-D` `SIZED_BASED_WRAP` seems to work but then NCCL timeouts (30minutes) on the last request batch. It is hanging on...