[Bug] finetune the PP-HumanSeg General Human Segmentation model

[fschvart]

Hi,

I've been trying to finetune the PP-HumanSeg General Human Segmentation model with annotated images. I'm using the PP-HumanSegV1-Server yml file and retrained model.

First, there seems to be two bugs in the YML file: Row 74, "pretrained:" appears twice, one of these has to be deleted Row 60, there's actually one mixed_loss, so the list should only have 1 loss, so [1] instead of [1,1,1]. At least that's how I was able to debug the code.

Next when I try to run the code I get the following error:

OSError: (External) CUDA error(700), an illegal memory access was encountered. [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:258)

The code seems to run OK on CPU (at least, I didn't get that message after 10-15 minutes)

When I run the code in PyCharm debug mode I get this message in several variables: "Unable to get repr for <class 'paddle.Tensor'>

I ran the file with different batch files to make sure it's not a OOM issue.

Here's my environment and the full error message:

RTX 3090

1. PaddleSeg version: (please specify the branch as well，e.g. PaddleSeg release/2.3). - 2.6.0
2. PaddlePaddle version: (e.g. PaddlePaddle 2.1.0). - PaddlePaddle-gpu 2.3.2
3. Operation system: (e.g. Linux/Windows/MacOS). - Windows
4. Python version: (e.g. Python3.6/7/8) - 3.10
5. CUDA/cuDNN version: (e.g. CUDA10.2/cuDNN 7.6.5) - 11.6/8.4
6. Full codes: (if you modify any original code，please show the comparison of the codes before and after) - I mentioned the only changes
8. Detailed error information, releated running log: (if using multi-gpus，the log is saved to log/worklog.0 by default)

W0919 18:21:22.095626 12976 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.7, Runtime API Version: 11.6 W0919 18:21:22.096624 12976 gpu_resources.cc:91] device: 0, cuDNN Version: 8.4. 2022-09-19 18:21:22 [INFO] Loading pretrained model from .\model\1\model.pdparams 2022-09-19 18:21:23 [WARNING] conv1_1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_3._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_3._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_3._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_3._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] conv1_3._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.short._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.short._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.short._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.short._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_0.short._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_1.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_0_2.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.short._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.short._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.short._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.short._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_0.short._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_1.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_2.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_1_3.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.short._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.short._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.short._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.short._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_0.short._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_1.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_2.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_3.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_4.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_2_5.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.short._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.short._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.short._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.short._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_0.short._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_1.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv0._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv0._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv0._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv0._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv0._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv1._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv1._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv1._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv1._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv1._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv2._conv.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv2._batch_norm.weight is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv2._batch_norm.bias is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv2._batch_norm._mean is not in pretrained model 2022-09-19 18:21:23 [WARNING] bb_3_2.conv2._batch_norm._variance is not in pretrained model 2022-09-19 18:21:23 [INFO] There are 0/275 variables loaded into ResNet_vd. 2022-09-19 18:21:23 [INFO] Loading pretrained model from .\model\1\model.pdparams 2022-09-19 18:21:24 [INFO] There are 360/360 variables loaded into DeepLabV3P. C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\nn\layer\norm.py:653: UserWarning: When training, we now always track global mean and variance. warnings.warn( C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\math_op_patch.py:276: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.int64, the right dtype will convert to paddle.float32 warnings.warn( Traceback (most recent call last): File "E:\videoseg\PaddleSeg\train.py", line 229, in main(args) File "E:\videoseg\PaddleSeg\train.py", line 205, in main train( File "E:\videoseg\PaddleSeg\paddleseg\core\train.py", line 207, in train loss_list = loss_computation( File "E:\videoseg\PaddleSeg\paddleseg\core\train.py", line 48, in loss_computation mixed_loss_list = loss_i(logits, labels) File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "E:\videoseg\PaddleSeg\paddleseg\models\losses\mixed_loss.py", line 55, in forward output = loss(logits, labels) File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "E:\videoseg\PaddleSeg\paddleseg\models\losses\lovasz_loss.py", line 53, in forward loss = lovasz_softmax_flat(vprobas, vlabels, classes=self.classes) File "E:\videoseg\PaddleSeg\paddleseg\models\losses\lovasz_loss.py", line 165, in lovasz_softmax_flat if probas.numel() == 0: File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\varbase_patch_methods.py", line 669, in bool return self.nonzero() File "C:\Users\FabionDual2\miniconda3\envs\pytorch\lib\site-packages\paddle\fluid\dygraph\varbase_patch_methods.py", line 666, in nonzero return bool(np.all(tensor.array() > 0)) OSError: (External) CUDA error(700), an illegal memory access was encountered. [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:258)

[juncaipeng]

Thank you for pointing out the bugs. We'll fix it right away.

OSError: (External) CUDA error(700), an illegal memory access was encountered.

I think the reason of the above error is that PaddlePaddle is no good compatible for CUDA on Windows. You can remove the LovaszSoftmaxLoss in config or run the training on another environment.