Philip Bontrager

Currently our batch size is a local batch size. This means with a bs=4, if you launch on 4 gpus then each gpu gets 4 data points and your real...

Add LoRA and QLoRA Components, Models, and Configs for Phi3 Mini

2

#### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [ ] update tests and/or...

Flamingo Model Components

2

#### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [ ] update tests and/or...

[RFC] Fusion Models

1

# [RFC] Fusion Models **TLDR** - Fused Models are two+ pre-trained models joined together and further tuned to work as one model. This is the approach used for most SOTA...

QLoRA with Llama 3.1 405B

7

#### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [ ] update tests and/or...

[RFC] TransformerDecoderLayer Refactor

4

# [RFC] TransformerDecoderLayer Refactor Refactor TransformerDecoder so it can be used for multimodal architectures. **TLDR** - Replace TransformerDecoderLayer with TransformerSelfAttention and TransformerCrossAttention - Replace CausalSelfAttention with GroupedQueryAttention - Support legacy...

Not saving Phi3 Mini LoRA Adapter Checkpoint

3

In the HF Checkpointer, we warn the user that the adapter weights can't be converted to the PEFT format and will be converted to a torchtune format, but then we...

Assert LLama Vision Image size divides by 14

3

Image size must be divisible by ViT patches in the CLIP encoder, 14.

With the addition of multimodal and dpo, collation is getting more varied and complicated. Either depending on the recipe or the model and type of data in the model. To...