多卡训练 时长问题

[Murphy-MML]

我使用python -m paddle.distributed.launch train.py --config configs/pspnet/pspnet_resnet101_os8_cityscapes_1024x512_80k.yml --do_eval --save_dir /mnt/Workdir/pspnet/pspnet_resnet101_os8_cityscapes_1024x512_80k/ --num_workers 12 为例 为什么 时长会到达40小时

2022-06-28 07:04:06 [INFO] [TRAIN] epoch: 1, iter: 20/80000, loss: 2.1961, lr: 0.009998, batch_cost: 1.7653, reader_cost: 0.00016, ips: 1.1330 samples/sec | ETA 39:13:06 2022-06-28 07:04:26 [INFO] [TRAIN] epoch: 1, iter: 30/80000, loss: 2.0358, lr: 0.009997, batch_cost: 1.9892, reader_cost: 0.00033, ips: 1.0054 samples/sec | ETA 44:11:13

硬件信息如下 ------------Environment Information------------- platform: Linux-4.19.0-19-amd64-x86_64-with-debian-buster-sid Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0] Paddle compiled with cuda: True NVCC: Build cuda_11.2.r11.2/compiler.29618528_0 cudnn: 8.1 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: NVIDIA GeForce', 'GPU 1: NVIDIA GeForce', 'GPU 2: NVIDIA GeForce', 'GPU 3: NVIDIA GeForce'] GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PaddleSeg: 2.5.0 PaddlePaddle: 2.3.0 OpenCV: 4.6.0

2022-06-28 07:02:52 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 loss: coef:

• 1
• 0.4 types:
• ignore_index: 255 type: CrossEntropyLoss lr_scheduler: end_lr: 1.0e-05 learning_rate: 0.01 power: 0.9 type: PolynomialDecay model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd enable_auxiliary_loss: true pretrained: null type: PSPNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms:
• max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling
• crop_size:
  • 1024
  • 512 type: RandomPaddingCrop
• type: RandomHorizontalFlip
• brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort
• type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms:
• type: Normalize type: Cityscapes

---

[wuyefeilin]

你是用什么显卡训练的呢

[Murphy-MML]

你是用什么显卡训练的呢

4 张 3090

[wuyefeilin]

如果您追求速度的话，可以切换到ppliteseg或者bisenetv2等轻量级模型

[Murphy-MML]

你好 我的意思是相同的batchsize等设置下 这个模型在四张显卡的训练时长接近3倍在单张3090的表现。我想询问的是造成这种现象的原因是什么，这种情况是否正常。很感谢你的回答

[wuyefeilin]

麻烦给一下单卡和多卡的训练日志哈，另外确保所有的卡没有额外的程序在运行

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.