PaddleSeg icon indicating copy to clipboard operation
PaddleSeg copied to clipboard

[General Issue] Error when trying to train in batch size > 1

Open cspearl opened this issue 3 years ago • 2 comments

  1. PaddleSeg version: PaddleSeg release/2.6

  2. Operation system: Linux - Google Colab

  3. Python version: 3.7

  4. CUDA/cuDNN version: CUDA11.1/cuDNN 7.6.5

  5. Additional context: There is no error in training when I am setting batch size to 1. But mIOU is very poor I dont know why. And whenever I am setting batch size to 2 or 4 the following error pops up - (All my images and labels have the same shape - 512x512x3 for images and 512x512x1 for laebels)

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 218, in _thread_loop self._thread_done_event) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/fetcher.py", line 134, in fetch data = self.collate_fn(data) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/collate.py", line 70, in default_collate_fn for key in sample File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/collate.py", line 70, in for key in sample File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/collate.py", line 58, in default_collate_fn batch = np.stack(batch, axis=0) File "<array_function internals>", line 6, in stack File "/usr/local/lib/python3.7/dist-packages/numpy/core/shape_base.py", line 426, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape

cspearl avatar Jul 30 '22 07:07 cspearl

在使用dataloader时候,输入图片需要变成相同的尺寸才能压入一个batch中进行训练,如果此过程是训练过程,建议检查数据增强中是否有reshape到统一的大小的操作。

haoyuying avatar Aug 01 '22 02:08 haoyuying

在使用dataloader时候,输入图片需要变成相同的尺寸才能压入一个batch中进行训练,如果此过程是训练过程,建议检查数据增强中是否有reshape到统一的大小的操作。

Yes it is a training process but all my images are already of the same size - 512x512 RGB images and labels are 512x512 greyscale

cspearl avatar Aug 02 '22 13:08 cspearl

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 11 '22 17:12 github-actions[bot]