Joe Cummings

Remove `datasets` requirement and instead rely on download from `huggingface_hub`

6

We currently handle all dataset manipulation through our own dataset classes, all we are using `datasets` for is a convenient wrapper for downloading the dataset to disk. We should be...

Update nightly build instructions

1

#### Context - Our nightly builds previously needed an option ``--extra-index-url``, which allowed pip to look both in PyPI and the PyTorch Index. After @atalman runs the new changes in...

Add documentation information

1

#### Context - [ ] add a new feature - [ ] fix a bug - [x] update tests and/or documentation - [ ] other (please add here) #### Changelog...

Unify example dataset in configs

3

We use alpaca cleaned and alpaca. Should just use one for our example datasets.

Fix decode for Phi-3 Mini

1

#### Context What is the purpose of this PR? Is it to - [ ] add a new feature - [x] fix a bug - [ ] update tests and/or...

[WIP] Free generation for evals

1

#### Context What is the purpose of this PR? Is it to - [ ] add a new feature - [ ] fix a bug - [ ] update tests...

### Summary Adds diverse beam search from [this paper](https://arxiv.org/pdf/1610.02424.pdf). #### Helpful Diagrams Figure 1: Example of how diversity factor (Hamming) is applied at timestep *t* Figure 2: General algo ###...

Add batched inference

2

#### Context This PR adds batched inference capabilities to our generation utility. **Why is this needed?** In general, this isn't super important for our current generation recipe. We expect users...