LMFlow

LMFlow copied to clipboard

Make lots of packages optional

wheresmyhair

[BUG]TypeError: Field.__init__() missing 1 required positional argument: 'kw_only'

2

**To Reproduce** bash scripts/run_finetune_with_qlora_9_24_4.sh #!/bin/bash # Please run this script under ${project_id} in project directory of # Parses arguments #model_name_or_path=LLM-Research/Meta-Llama-3-70B-Instruct model_name_or_path=/data/hf_cache/hub/models--meta-llama--Meta-Llama-3-70B-Instruct/snapshots/5fcb2901844dde3111159f24205b71c25900ffbd #lora_model_path=/data/midreal/rio/lora/opening_0923 dataset_path=/data/midreal/rio/LMFlow-main/data/opening conversation_template=llama3 output_dir=/data/midreal/rio/LMFlow-main/output_models/qlora_finetuned_llama3_70b_opening_model_9_27 deepspeed_args="--master_port=13001 --include localhost:1" # Safety...

ORGRUI

tianshijing

# Description LISA now supports multi GPU. Key points: 1. When initialize models, ds will wrap all model params with optimizer, which uses huge amount of gpu mem. However, according...

wheresmyhair

Some question on code of LISA

1

I have known that LISA's core code in src\lmflow\pipeline\finetuner.py, mainly in class DynamicLayerActivationCallback. I read it with Algorithm 1 Layerwise Importance Sampling AdamW (LISA) in paper aside. So where is...

milong26

How can I use other optimizers in LISA?

1

I tried to pass --optim , nothing happens. How can I use optimizers such as adamw_8bit or adafactor in LISA? They are not in the custom_optimizers either.

greatzane

Can lisa training support MiniCPM-V?

1

Can lisa training support MiniCPM-V?

orderer0001

Specific methods about lisa

1

Excuse me, I want to know what layers Lisa is frozen in and his freezing strategy. Which file code should I look at?

orderer0001

Does it support qwen2?

2

Does it support qwen2-1.5b?

yangxue-1

Is there any Lisa that supports llama3.1 yet?

1

Is there any Lisa that supports llama3.1 yet?

orderer0001