DeepSpeedExamples

DeepSpeedExamples copied to clipboard

run deepspeed_chat example code error

3

when I run code bash training_scripts/single_node/run_1.3b.sh , meet error ```shell ninja: no work to do. Loading extension module fused_adam... Time to load fused_adam op: 0.0961456298828125 seconds Loading extension module fused_adam......

bestpredicts

when I am running RLHF script, I encountered a error

1

  My environments setting: deepspeed==0.9.0, torch==2.0.0+cu117 CUDA Version: 11.0 pretrained model is facebook/opt-350m Who can help me solve this problem? Thanks

liuzhiyong01

DeepSpeed op builder

4

Thanks for helping to solve this problem 

wangshuo6699

stuck running >>bash training_scripts/single_gpu/run_1.3b.sh

1

I run it on Ubunu 20.04 with 2 3090 cards, it always get stuck, py-spy dump shows: Process 46930: /home/qi/anaconda3/envs/deepspeed/bin/python -u main.py --local_rank=0 --model_name_or_path facebook/opt-1.3b --gradient_accumulation_steps 2 --lora_dim 128 --zero_stage...

woshialex

Single node multi card training failed

1

(deepspeed) [menkeyi@gpu1 DeepSpeed-Chat]$ python train.py --actor-model facebook/opt-13b --reward-model facebook/opt-350m --deployment-type single_node ---=== Running Step 1 ===--- Running: bash /home/menkeyi/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/training_scripts/single_node/run_13b.sh /home/menkeyi/DeepSpeedExamples/applications/DeepSpeed-Chat/output/actor-models/13b GPU usage rate: (deepspeed) [menkeyi@gpu1 DeepSpeed-Chat]$ nvidia-smi Sat Apr 15...

menkeyi

It seems that lora does not work

7

When I run step1_supervised_finetuning, I found that the memory of the opt-1.3B model occupies about 15G, as if lora does not work, but when I print the trainable parameters, it...

blldd

https://github.com/microsoft/DeepSpeedExamples/blob/cd19b3bf1e5b60dd73b09c7463da4eedada1eed7/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py#L234 there are two parameters in DistributedSampler ```python num_replicas (int, optional): Number of processes participating in distributed training. By default, :attr:`world_size` is retrieved from the current distributed group. rank (int,...

nickyi1990

step1-sft use lora failed

4

env ``` gpu: 4*A100 80G pytorch: 1.13.1 cuda version: 11.7 deepspeed: 0.9.0 transformers: 4.28.0.dev ``` run script ``` OUTPUT=$1 ZERO_STAGE=3 if [ "$OUTPUT" == "" ]; then OUTPUT=./output fi if...

bytes-lost

Is there a way to log experiments to wandb? e.g., loss, lr and customized metrics.

puyuanOT

Does the model in this framework have to be trained to have conversations?

1

I am new to using LLM, and is it possible to just download the open source LM and have multiple conversations without fine-tuning?

code-isnot-cold