Issue during training for multiple classes

[NNtamp]

Hi Sir. We faced an issue during our tries for training the model in multiple classes. We modified the sfd.yaml file based on the voxel_rcnn (please find attached the sfd.yaml file in text format) We received the following error message:

Traceback (most recent call last): File "train.py", line 200, in main() File "train.py", line 155, in main train_model( File "/workspace/SFD/tools/train_utils/train_utils.py", line 86, in train_model accumulated_iter = train_one_epoch( File "/workspace/SFD/tools/train_utils/train_utils.py", line 19, in train_one_epoch batch = next(dataloader_iter) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 345, in next data = self._next_data() File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg) TypeError: Caught TypeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/workspace/SFD/pcdet/datasets/kitti/kitti_dataset_sfd.py", line 517, in collate_batch ret[key] = np.stack(val, axis=0) File "<array_function internals>", line 5, in stack File "/usr/local/lib/python3.8/dist-packages/numpy/core/shape_base.py", line 427, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape

Do you have any solution please? We tried with batch size of 1 in the beggining but the model couldn't perform the batch normalization so we increased the batch size to 2. Also we have one single gpu.

sfd_yaml_file.txt

[LittlePey]

Hi, the current version of our code seems only support 1 sample on each GPU, could you show the error information after you train with batch size of 1.

[NNtamp]

Thank you for your answer. Attached you can find the traceback error after trying training with batch size of 1. message (6).txt

[NNtamp]

@LittlePey Does the above traceback error help you to understand the issue? Is there a solution? Thank you in advance.

[NNtamp]

@LittlePey Any idea?

[LittlePey]

Hi, it is a bug in SFD code when there is no pseudo point in any ROI. Sometimes, we just resume the latest checkpoint and the error will disappear.

[NNtamp]

Hi @LittlePey and thank you for your answer. To be honest I didn't understand your answer. I will try to repeat the issue. We tried to train the SFD for multiple classes and with batch size of 1 in a single gpu machine. We received the attached error. (We tried also with greater batch sizes but as you mentioned the current version of our code seems only support 1 sample on each GPU). How can we resume the latest checkpoint if the training doesn't start at all? Is there a solution? What do you think? Thank you in advance. message.6.txt

[LittlePey]

Hi, we didn't encounter your problem that the training doesn't start at all, maybe you can skip forward and backward when this situation happened.

[NNtamp]

Hi again @LittlePey . Any update on this? Is there a way to configure the training procedure with batch size of 2?

[Dowe-dong]

This bug is appear before the first epoch finished. So how can I resume the latest checkpoint?