多卡finetune launch_agent run报错

[yueshiyi]

hello，使用accelerate多卡finetune时launch_agent报错

报错信息如下：

INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 459723) of binary: /data/shiyi_yue/miniconda3/envs/pt113/bin/python
Traceback (most recent call last):
  File "/data/shiyi_yue/miniconda3/envs/pt113/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/accelerate/commands/launch.py", line 900, in launch_command
    deepspeed_launcher(args)
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in deepspeed_launcher
    distrib_run.run(args)
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data/shiyi_yue/miniconda3/envs/pt113/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
========================================================
finetune_moss.py FAILED

请问可以帮助确定是哪里的问题吗？

[lukaswangbk]

是怎么指定多卡的

[Yiwen-Yang-666]

解决了吗 @yueshiyi 求分享