llm-foundry
llm-foundry copied to clipboard
mosaicml
LLM training code for Databricks foundation models
Tried setting max_seq_len to 5k and added alibi to attn config with Triton flash attn. Otherwise using: scripts/train/yamls/finetune/mpt-7b_dolly_sft.yaml. Finetuning seems to be working but it does emit this sketchy log...
`formatted_example = {'prompt': , 'response': }` looks like it only applies to a single round of conversation
Finetune command: ```shell cd scripts composer train/train.py \ ./train/yamls/finetune/mpt-7b_dolly_sft.yaml ``` Got error: ```txt Traceback (most recent call last): ... .../llm-foundry/llmfoundry/data/finetuning/dataloader.py", line 215, in _validate_config raise ValueError( ValueError: In the dataset...
Hi Mosaicml folks, this is because of a lack of knowledge on my end, and I apologize for this, I'd be extremely greatful if you could give me example code...
when I run "python ../../llmfoundry/data/text_data.py --local_path ./my-copy-c4 --split val_small", I get the error ------------------------------------------------------------- (mpt-env) root@autodl-container-645911b4fa-161dd8b6:~/autodl-tmp/llm-foundry/scripts/train# python ../../llmfoundry/data/text_data.py --local_path ./my-copy-c4 --split val_small Traceback (most recent call last): File "../../llmfoundry/data/text_data.py", line...
When using pre-trained Chinese language corpus, which tokenizer should I use? EleutherAI/gpt-neox-20b does not have Chinese vocabulary.
My server has 8 GPUs. I want to test it on single GPU. I set the num_workers=1 in the yaml file. But every time I run the command 'composer train.py...
At the moment your readme says: > This repository contains code for training, finetuning, evaluating, and deploying LLMs for inference with Composer and the [MosaicML platform](https://forms.mosaicml.com/demo?utm_source=github.com&utm_medium=referral&utm_campaign=llm-foundry). and doesn't provide a...
I tried fine tuning mpt7b using dolly dataset. Using below command: composer train.py yamls/finetune/mpt-7b_dolly_sft.yaml yaml file: https://github.com/mosaicml/llm-foundry/blob/main/scripts/train/yamls/finetune/mpt-7b_dolly_sft.yaml Before strating training i am getting below error: [Eval batch=321/321] Eval on eval...
