zzx528

> 问题解决了么？同+1 请问解决了吗，我也有同样的问题

> 单机多卡或者多机多卡都支持的 请问这个怎么用？ 这是我的预训练脚本 lr=2e-4 lora_rank=8 lora_alpha=32 lora_trainable="q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj" modules_to_save="embed_tokens,lm_head" lora_dropout=0.05 pretrained_model=/zzx/Chinese-LLaMA-Alpaca/model/merge_chinese_alpaca_plus_13b chinese_tokenizer_path=/zzx/Chinese-LLaMA-Alpaca/model/merge_chinese_alpaca_plus_13b dataset_dir=/zzx/Chinese-LLaMA-Alpaca/data data_cache=/zzx/Chinese-LLaMA-Alpaca/data/data_tmp per_device_train_batch_size=1 per_device_eval_batch_size=1 training_steps=10 # 默认100 gradient_accumulation_steps=1 output_dir=/zzx/Chinese-LLaMA-Alpaca/output deepspeed_config_file=ds_zero2_no_offload.json torchrun --nnodes 1 --nproc_per_node 8 run_clm_pt_with_peft.py \ --deepspeed...

> > @sszyl 朋友你微调用了多少GPU显存，我24G显存OOM > > 8卡V100 32G*8 你好，请问一下怎么多卡微调

> 1T内存 肯定是够了 请问怎么一机多卡训练？

We had a lot of problems reproducing the paper, can you provide NAGA-II search code?

> > We had a lot of problems reproducing the paper, can you provide NAGA-II search code? > > My reimplementation is very far from perfect, but if you like,...

> Sorry, there is no plan to open the search code till now. > > You can refer to [pymoo](https://github.com/msu-coinlab/pymoo) and [MixPath](https://github.com/CuriousCat-7/MixPath) to implement the NAGA-II search. If you have...