RuntimeError: Index put requires the source and destination dtypes match, got BFloat16 for the destination and Float for the source.

[kawabe-jiw]

Search before asking

• [x] I have searched the RF-DETR issues and found no similar bug report.

Bug

An error occurred when running training with use_position_supervised_loss set to True. This issue arises when training on an NVIDIA L4.

File "/home/ubuntu/rf_deter_train/rf_detr_train/train.py", line 102, in train
    model.train(**train_params)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/detr.py", line 83, in train
    self.train_from_config(config, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/detr.py", line 191, in train_from_config
    self.model.train(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/main.py", line 341, in train
    train_stats = train_one_epoch(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/engine.py", line 130, in train_one_epoch
    loss_dict = criterion(outputs, new_targets)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/models/lwdetr.py", line 551, in forward
    losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes))
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/models/lwdetr.py", line 524, in get_loss
    return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/rf-detr-train-vhWH2Pc8-py3.10/lib/python3.10/site-packages/rfdetr/models/lwdetr.py", line 372, in loss_labels
    cls_iou_func_targets[pos_ind] = pos_ious_func
RuntimeError: Index put requires the source and destination dtypes match, got BFloat16 for the destination and Float for the source.

Environment

• RF-DETR: 1.3.0
• OS: Ubuntu 22.04.5 LTS
• Python: 3.10.12
• PyTorch: 2.9.0
• CUDA: 12.8
• GPU: NVIDIA L4

Minimal Reproducible Example

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = rfdetr.RFDETRMedium(
    num_classes = 2,
    pretrained = True,
    device = device
)
train_params ={
  "dataset_dir": "/home/ubuntu/rf_deter_train/datasets",
  "epochs": 2,
  "batch_size": 4,
  "lr": 0.0001,
  "output_dir": "/tmp/train_result/train_result_1",
  "early_stopping": True,
  "patience": 50,
  "resolution": 640,
  "checkpoint_interval": 2,
  "use_position_supervised_loss": True,
  "use_varifocal_loss": False,
  "sum_group_losses": False,
  "ia_bce_loss": False
}
model.train(**train_params)

Additional

This error did not occur when running in an environment with an NVIDIA GeForce RTX 3080 Laptop GPU.

Are you willing to submit a PR?

• [x] Yes, I'd like to help by submitting a PR!

[kawabe-jiw]

This error did not occur when running in an environment with an This error did not occur when running in an environment with an NVIDIA GeForce RTX 3080 Laptop GPU

When training on an NVIDIA GeForce RTX 3080 Laptop GPU, the parameter ia_bce_loss was set to True. When I changed ia_bce_loss to False, the same error occurred.