[Bug] ValueError: Target -4844520860993308645 is out of lower bound

[kejom-ou]

在教程：20分钟快速上手PaddleSeg 中训练时loss报错

1. PaddleSeg版本：PaddleSeg release/2.6

2. PaddlePaddle版本：PaddlePaddle 2.3.1

3. 操作系统信息：Linux

4. Python版本号：Python3.7

5. CUDA/cuDNN版本：CUDA10.2/cuDNN 7

6. 详细的错误信息 Traceback (most recent call last): File "train.py", line 230, in main(args) File "train.py", line 225, in main to_static_training=cfg.to_static_training) File "/imggroup/kongou/PaddleSeg/paddleseg/core/train.py", line 211, in train losses=losses) File "/imggroup/kongou/PaddleSeg/paddleseg/core/train.py", line 55, in loss_computation loss_list.append(coef_i * loss_i(logits, labels)) File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/imggroup/kongou/PaddleSeg/paddleseg/models/losses/cross_entropy_loss.py", line 86, in forward weight=self.weight) File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/nn/functional/loss.py", line 1720, in cross_entropy label_min.item())) ValueError: Target -4844520860993308645 is out of lower bound.

7. 运行指令或复现步骤： CUDA_VISIBLE_DEVICES=2 python train.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --do_eval --save_interval 50 --save_dir output

[Bobholamovic]

本地无法复现此问题。请问多次重新执行都会出现此问题吗？

[kejom-ou]

是的

[Bobholamovic]

请尝试重新下载数据，以及用--seed选项更换随机种子。

[Copilot-X]

是的

我也遇到这个bug，目前有解吗

[Copilot-X]

请尝试重新下载数据，以及用--seed选项更换随机种子。

bug解决了吗

[kejom-ou]

换了一台服务器，重新跟着教程跑，就没有问题了~~~

[Vinbert0203]

有谁知道是什么问题么？我也遇到了

[Vinbert0203]

File "train.py", line 230, in main(args) File "train.py", line 206, in main train( File "/nas/liwenbo/PaddleSeg/paddleseg/core/train.py", line 207, in train loss_list = loss_computation( File "/nas/liwenbo/PaddleSeg/paddleseg/core/train.py", line 55, in loss_computation loss_list.append(coef_i * loss_i(logits, labels)) File "/usr/local/miniconda3/envs/paddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/miniconda3/envs/paddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/nas/liwenbo/PaddleSeg/paddleseg/models/losses/cross_entropy_loss.py", line 81, in forward loss = F.cross_entropy( File "/usr/local/miniconda3/envs/paddle/lib/python3.8/site-packages/paddle/nn/functional/loss.py", line 1720, in cross_entropy raise ValueError("lwb Target {} is out of lower bound.".format( ValueError: lwb Target 4872363901336821031 is out of lower bound.