llm-foundry
llm-foundry copied to clipboard
mosaicml
LLM training code for Databricks foundation models
I think eval dataloaders should not shuffle data for consistent evaluation results
Edits needed to support a combo of composer with hf/peft. Pipeline is: 1. load a hf model e.g., mpt-7b 2. use hf/peft to add lora modules or adapter modules. 3....
I'm trying to convert c4 dataset from your `convert_hf` code [here](https://huggingface.co/datasets/allenai/c4) they say `en` subset is 305 Gb but if I'm give `c4` and `en` as arguments, it looks like...
## 🚀 Feature Request I believe that, to infer using a longer context, I can set max_seq_len to something longer when starting the huggingface based inference driver. However, I don't...
## Environment I'm using the Docker images for llm-foundry, training on 8xA100. mosaicml/llm-foundry:1.13.1_cu117-latest mosaicml/llm-foundry:2.0.1_cu118-latest Collecting system information... --------------------------------- System Environment Report Created: 2023-06-24 17:52:44 UTC --------------------------------- PyTorch information ------------------- PyTorch...
Hello! In many examples including this one (https://github.com/mosaicml/llm-foundry/blob/90795f37c16c008aae954df55fc4f3323bc581e4/scripts/train/yamls/finetune/mpt-7b_dolly_sft.yaml#L1), the max_seq_length doesn't affect model configuration implicitly. That means the configuration of the model has to be overriden explicitly: ``` model: config_overrides:...
Created model gauntlet. This PR makes a number of significant changes. It checks in 38 datasets, it adds a callback which can compute model gauntlet scores from a large number...
Hi, I'm working on fine-tuning the MPT-30B for function calling. Currently still preparing the fine-tuning dataset. AFAIK there is no open-source fine-tuned model for function support(let me know if you...
## ❓ Question Fine tuning of MPT-7B failing on A10 GPUs... Can you help with a script for the same
I am seeing random results after converting model to FT. I used the conversion and inference scripts included in this repo by @dskhudia. To reproduce the issue, 1. Downloaded mpt-7b-instruct...
