salman

[RFC] Proximal Policy Optimisation

10

# Implementing Proximal Policy Optimisation I've used some of the [PyTorch RFC](https://github.com/pytorch/rfcs/blob/master/README.md) template here for clarity. **Authors:** * @salmanmohammadi ## **Summary** I'd like to add support for fine-tuning models using...

Transformer classifier

10

Context/Changelog See https://github.com/pytorch/torchtune/issues/812 This PR adds a TransformerClassifier layer which extends the TransformerDecoder functionality to classification tasks. Exemplar component and model builders have been implemented for the base mistral model....

adding model builders for code-llama2 7b, 13b, and 70b

9

#### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [ ] update tests and/or...

I would have found it helpful to have tests for mistral while I was working on https://github.com/pytorch/torchtune/pull/840, and I noticed it was mentioned in https://github.com/pytorch/torchtune/issues/691, so I thought I'd work...

Mistral testing

5

#### Context What is the purpose of this PR? Is it to - [ ] add a new feature - [ ] fix a bug - [x] update tests and/or...

Loading mistral reward model checkpoints

1

See https://github.com/pytorch/torchtune/issues/812#issuecomment-2067790583 for context. #### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [...

Updating generation to use custom_generate_next_token for initial token generation

2

#### Context What is the purpose of this PR? Is it to - [ ] add a new feature - [x] fix a bug - [ ] update tests and/or...

[RFC] TransformerDecoder refactor

1

# `TransformerDecoder` Refactor **Authors:** * @SalmanMohammadi with input from: * @kartikayk * @ebsmothers * @pbontrager ## **Summary** Refactoring `TransformerDecoder` to offer additional flexibility for new use-cases. ## **Motivation/Prior art** -...

HF Checkpointer doesn't support loading quantized models

1

Trying ``` quantizer: _component_: torchtune.utils.quantization.Int8WeightOnlyQuantizer ``` In a config for a `generate` recipe using HF checkpoints gives ``` TypeError: load_checkpoint() got an unexpected keyword argument 'weights_only' ``` Since loading quantized...

RLHF with PPO

1

#### Context What is the purpose of this PR? Is it to - [x] add a new feature - [ ] fix a bug - [ ] update tests and/or...