Sebastian Raschka

Should we automatically infer and download the tokenizer config when no `--tokenizer_dir` is provided in `litgpt pretrain`? I.e., ```bash litgpt download \ --repo_id EleutherAI/pythia-14m \ --tokenizer_only true litgpt pretrain \...

Print initial validation loss + final training and validation loss

1

These PR does 2 things 1) Users were confused about the initial n/a in the validation loss. One idea I had was that we can actually use the initial validation...

Meaningful error if no validation split fraction is provided in custom JSON data module

1

If a user doesn't set `--data.val_split_fraction` in ```bash litgpt finetune lora \ --data JSON \ --data.json_path ....json \ --checkpoint_dir checkpoints/$REPO_NAME ``` It raises an error ```bash train_data, test_data = self.get_splits()...

Categorize SFT and Pretraining data

2

Ideally we only want the SFT finetuning datasets to show up in `litgpt finetune --help` and only the pretraining datasets to show up in `litgpt pretrain --help`. I believe we...

Defaults fail on small block size for some models

1

I think that all models should all work out of the box with the defaults. Maybe we need to do some automatic clipping on the inputs and raise a warning...

LLMs are commonly pretrained using gradient clipping, which we also implement for pretraining here in LitGPT. In addition, it might be useful to add a gradient clipping option to the...

I talked to some users who were a bit confused / surprised that the validation loss didn't get calculated during a run. I think that's a valid issue if you...

Python 3.12

3

Is there anything that speaks against listing Python 3.12 support on the Readme? 

Add TinyStories to the pretraining docs

3

As far as I know, one can pretrain fine using TinyStories: ```python litgpt/pretrain.py --data litgpt.data.TinyStories ``` Should we add this to the documentation? Right now, we only have a `pretrain_tinyllama.md`...