humphery

[Feature] 使用internvl_chat_llava/scripts_internvl/pretrain_internvit6b_448_vicuna7b.sh预训练时train/grad_norm值0.5左右loss在0.5与2.4之间振荡

1

### Motivation 如题，grad_norm很快降为0.5左右，是什么参数不对吗？ 资源：1*A100 80G 训练参数： deepspeed --include localhost:2 \ llava/train/train_mem.py \ --deepspeed ./scripts/zero2.json \ --model_name_or_path $DATA_HOME/pretrained_mm_projector/vicuna-7b-v1.5 \ --version plain \ --data_path $DATA_HOME/LLaVA-Pretrain/enhanced_llava_pretrain_data_708K.json \ --image_folder $DATA_HOME/LLaVA-Pretrain/images \ --vision_tower $DATA_HOME/pretrained_mm_projector/InternViT-300M-448px \...