YOLOX icon indicating copy to clipboard operation
YOLOX copied to clipboard

RuntimeError: CUDA error: device-side assert triggered

Open xauat-liushipeng opened this issue 2 years ago • 2 comments

2022-05-23 10:33:27.312 | INFO | yolox.core.trainer:before_train:126 - args: Namespace(batch_size=4, ckpt='model_data/last_epoch_ckpt.pth', devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/yolox_l.py', experiment_name='yolox_l', fp16=False, machine_rank=0, name=None, num_machines=1, occupy=True, opts=[], resume=False, start_epoch=None) 2022-05-23 10:33:27.314 | INFO | yolox.core.trainer:before_train:127 - exp value: ╒══════════════════╤════════════════════════════════════════════╕ │ keys │ values │ ╞══════════════════╪════════════════════════════════════════════╡ │ seed │ None │ ├──────────────────┼────────────────────────────────────────────┤ │ output_dir │ '/home/axs-liushipeng/YOLOX/YOLOX_outputs' │ ├──────────────────┼────────────────────────────────────────────┤ │ print_interval │ 10 │ ├──────────────────┼────────────────────────────────────────────┤ │ eval_interval │ 1 │ ├──────────────────┼────────────────────────────────────────────┤ │ num_classes │ 1 │ ├──────────────────┼────────────────────────────────────────────┤ │ depth │ 1.0 │ ├──────────────────┼────────────────────────────────────────────┤ │ width │ 1.0 │ ├──────────────────┼────────────────────────────────────────────┤ │ data_num_workers │ 8 │ ├──────────────────┼────────────────────────────────────────────┤ │ input_size │ (640, 640) │ ├──────────────────┼────────────────────────────────────────────┤ │ random_size │ (14, 26) │ ├──────────────────┼────────────────────────────────────────────┤ │ data_dir │ 'datasets/COCO' │ ├──────────────────┼────────────────────────────────────────────┤ │ train_ann │ 'instances_train2017.json' │ ├──────────────────┼────────────────────────────────────────────┤ │ val_ann │ 'instances_val2017.json' │ ├──────────────────┼────────────────────────────────────────────┤ │ degrees │ 10.0 │ ├──────────────────┼────────────────────────────────────────────┤ │ translate │ 0.1 │ ├──────────────────┼────────────────────────────────────────────┤ │ scale │ (0.1, 2) │ ├──────────────────┼────────────────────────────────────────────┤ │ mscale │ (0.8, 1.6) │ ├──────────────────┼────────────────────────────────────────────┤ │ shear │ 2.0 │ ├──────────────────┼────────────────────────────────────────────┤ │ perspective │ 0.0 │ ├──────────────────┼────────────────────────────────────────────┤ │ enable_mixup │ False │ ├──────────────────┼────────────────────────────────────────────┤ │ warmup_epochs │ 5 │ ├──────────────────┼────────────────────────────────────────────┤ │ max_epoch │ 300 │ ├──────────────────┼────────────────────────────────────────────┤ │ warmup_lr │ 0 │ ├──────────────────┼────────────────────────────────────────────┤ │ basic_lr_per_img │ 0.00015625 │ ├──────────────────┼────────────────────────────────────────────┤ │ scheduler │ 'yoloxwarmcos' │ ├──────────────────┼────────────────────────────────────────────┤ │ no_aug_epochs │ 15 │ ├──────────────────┼────────────────────────────────────────────┤ │ min_lr_ratio │ 0.05 │ ├──────────────────┼────────────────────────────────────────────┤ │ ema │ True │ ├──────────────────┼────────────────────────────────────────────┤ │ weight_decay │ 0.0005 │ ├──────────────────┼────────────────────────────────────────────┤ │ momentum │ 0.9 │ ├──────────────────┼────────────────────────────────────────────┤ │ exp_name │ 'yolox_l' │ ├──────────────────┼────────────────────────────────────────────┤ │ test_size │ (640, 640) │ ├──────────────────┼────────────────────────────────────────────┤ │ test_conf │ 0.01 │ ├──────────────────┼────────────────────────────────────────────┤ │ nmsthre │ 0.65 │ ╘══════════════════╧════════════════════════════════════════════╛ 2022-05-23 10:33:27.743 | INFO | yolox.core.trainer:before_train:133 - Model Summary: Params: 57.52M, Gflops: 297.18 2022-05-23 10:33:29.304 | INFO | yolox.core.trainer:resume_train:292 - loading checkpoint for fine tuning 2022-05-23 10:33:29.728 | INFO | yolox.data.datasets.coco:init:43 - loading annotations into memory... 2022-05-23 10:33:29.843 | INFO | yolox.data.datasets.coco:init:43 - Done (t=0.11s) 2022-05-23 10:33:29.843 | INFO | pycocotools.coco:init:89 - creating index... 2022-05-23 10:33:29.855 | INFO | pycocotools.coco:init:89 - index created! 2022-05-23 10:33:30.692 | INFO | yolox.core.trainer:before_train:153 - init prefetcher, this might take one minute or less... 2022-05-23 10:33:36.484 | INFO | yolox.data.datasets.coco:init:43 - loading annotations into memory... 2022-05-23 10:33:36.490 | INFO | yolox.data.datasets.coco:init:43 - Done (t=0.01s) 2022-05-23 10:33:36.490 | INFO | pycocotools.coco:init:89 - creating index... 2022-05-23 10:33:36.491 | INFO | pycocotools.coco:init:89 - index created! 2022-05-23 10:33:36.538 | INFO | yolox.core.trainer:before_train:183 - Training start... 2022-05-23 10:33:36.541 | INFO | yolox.core.trainer:before_train:184 - YOLOX( (backbone): YOLOPAFPN( (backbone): CSPDarknet( (stem): Focus( (conv): BaseConv( (conv): Conv2d(12, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (dark2): Sequential( (0): BaseConv( (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): CSPLayer( (conv1): BaseConv( (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) ) (dark3): Sequential( (0): BaseConv( (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): CSPLayer( (conv1): BaseConv( (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (3): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (4): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (5): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (6): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (7): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (8): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) ) (dark4): Sequential( (0): BaseConv( (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): CSPLayer( (conv1): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (3): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (4): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (5): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (6): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (7): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (8): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) ) (dark5): Sequential( (0): BaseConv( (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): SPPBottleneck( (conv1): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): ModuleList( (0): MaxPool2d(kernel_size=5, stride=1, padding=2, dilation=1, ceil_mode=False) (1): MaxPool2d(kernel_size=9, stride=1, padding=4, dilation=1, ceil_mode=False) (2): MaxPool2d(kernel_size=13, stride=1, padding=6, dilation=1, ceil_mode=False) ) (conv2): BaseConv( (conv): Conv2d(2048, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): CSPLayer( (conv1): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) ) ) (upsample): Upsample(scale_factor=2.0, mode=nearest) (lateral_conv0): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_p4): CSPLayer( (conv1): BaseConv( (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (reduce_conv1): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_p3): CSPLayer( (conv1): BaseConv( (conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (reduce_conv2): BaseConv( (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_p2): CSPLayer( (conv1): BaseConv( (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (bu_conv3): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_n2): CSPLayer( (conv1): BaseConv( (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (bu_conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_n3): CSPLayer( (conv1): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (bu_conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (C3_n4): CSPLayer( (conv1): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv3): BaseConv( (conv): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): Sequential( (0): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Bottleneck( (conv1): BaseConv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (conv2): BaseConv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) ) (head): YOLOXHead( (cls_convs): ModuleList( (0): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (3): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) (reg_convs): ModuleList( (0): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (1): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (2): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (3): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) (cls_preds): ModuleList( (0): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (1): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (2): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (3): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) ) (reg_preds): ModuleList( (0): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1)) (1): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1)) (2): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1)) (3): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1)) ) (obj_preds): ModuleList( (0): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (1): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (2): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) (3): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)) ) (stems): ModuleList( (0): BaseConv( (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): BaseConv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (2): BaseConv( (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (3): BaseConv( (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) (l1_loss): L1Loss() (bcewithlog_loss): BCEWithLogitsLoss() (iou_loss): IOUloss() ) ) 2022-05-23 10:33:36.542 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch1 2022-05-23 10:33:42.574 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 10/1442, mem: 19817Mb, iter_time: 0.603s, data_time: 0.000s, total_loss: 1.8, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.4, cls_loss: 0.3, lr: 1.202e-09, size: 640, ETA: 3 days, 0:26:39 2022-05-23 10:33:46.840 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 20/1442, mem: 19817Mb, iter_time: 0.426s, data_time: 0.000s, total_loss: 2.2, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 4.809e-09, size: 608, ETA: 2 days, 13:49:58 2022-05-23 10:33:52.881 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 30/1442, mem: 19817Mb, iter_time: 0.604s, data_time: 0.000s, total_loss: 2.4, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.082e-08, size: 608, ETA: 2 days, 17:24:09 2022-05-23 10:33:56.988 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 40/1442, mem: 19817Mb, iter_time: 0.410s, data_time: 0.000s, total_loss: 2.0, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.4, lr: 1.924e-08, size: 640, ETA: 2 days, 13:22:43 2022-05-23 10:34:01.127 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 50/1442, mem: 19817Mb, iter_time: 0.413s, data_time: 0.000s, total_loss: 1.8, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.3, lr: 3.006e-08, size: 480, ETA: 2 days, 11:02:17 2022-05-23 10:34:06.098 | INFO | yolox.core.trainer:after_iter:250 - epoch: 1/300, iter: 60/1442, mem: 19817Mb, iter_time: 0.497s, data_time: 0.000s, total_loss: 1.4, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 4.328e-08, size: 736, ETA: 2 days, 11:08:42 2022-05-23 10:34:11.576 | ERROR | yolox.models.yolo_head:get_losses:333 - OOM RuntimeError is raised due to the huge memory cost during label assignment. CPU mode is applied in this batch. If you want to avoid this issue, try to reduce the batch size or image size. 2022-05-23 10:34:11.737 | INFO | yolox.core.trainer:after_train:188 - Training of experiment is done and the best AP is 0.00 2022-05-23 10:34:11.737 | ERROR | yolox.core.launch:launch:78 - An error has been caught in function 'launch', process 'MainProcess' (9744), thread 'MainThread' (140289474713408): Traceback (most recent call last):

File "/home/axs-liushipeng/YOLOX/yolox/models/yolo_head.py", line 327, in get_losses imgs, └ <unprintable Tensor object>

File "/home/axs-gaoman/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context return func(*args, **kwargs) │ │ └ {} │ └ └ <function YOLOXHead.get_assignments at 0x7f9707e0b440>

File "/home/axs-liushipeng/YOLOX/yolox/models/yolo_head.py", line 494, in get_assignments cls_preds_.sqrt_(), gt_cls_per_image, reduction="none" │ │ └ <unprintable Tensor object> │ └ <method 'sqrt_' of 'torch._C._TensorBase' objects> └ <unprintable Tensor object>

File "/home/axs-gaoman/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2526, in binary_cross_entropy input, target, weight, reduction_enum) │ │ │ └ 0 │ │ └ None │ └ <unprintable Tensor object> └ <unprintable Tensor object>

RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "tools/train.py", line 127, in args=(exp, args), │ └ Namespace(batch_size=4, ckpt='model_data/last_epoch_ckpt.pth', devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/... └ ╒══════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════...

File "/home/axs-liushipeng/YOLOX/yolox/core/launch.py", line 78, in launch main_func(*args) │ └ (╒══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════... └ <function main at 0x7f96dd849d40>

File "tools/train.py", line 105, in main trainer.train() │ └ <function Trainer.train at 0x7f96d46f6c20> └ <yolox.core.trainer.Trainer object at 0x7f96d269add0>

File "/home/axs-liushipeng/YOLOX/yolox/core/trainer.py", line 72, in train self.train_in_epoch() │ └ <function Trainer.train_in_epoch at 0x7f96d3390320> └ <yolox.core.trainer.Trainer object at 0x7f96d269add0>

File "/home/axs-liushipeng/YOLOX/yolox/core/trainer.py", line 81, in train_in_epoch self.train_in_iter() │ └ <function Trainer.train_in_iter at 0x7f96d3397560> └ <yolox.core.trainer.Trainer object at 0x7f96d269add0>

File "/home/axs-liushipeng/YOLOX/yolox/core/trainer.py", line 87, in train_in_iter self.train_one_iter() │ └ <function Trainer.train_one_iter at 0x7f96d1cb3dd0> └ <yolox.core.trainer.Trainer object at 0x7f96d269add0>

File "/home/axs-liushipeng/YOLOX/yolox/core/trainer.py", line 99, in train_one_iter outputs = self.model(inps, targets) │ │ │ └ <unprintable Tensor object> │ │ └ <unprintable Tensor object> │ └ YOLOX( │ (backbone): YOLOPAFPN( │ (backbone): CSPDarknet( │ (stem): Focus( │ (conv): BaseConv( │ (conv): ... └ <yolox.core.trainer.Trainer object at 0x7f96d269add0>

File "/home/axs-gaoman/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) │ │ │ └ {} │ │ └ │ └ <function YOLOX.forward at 0x7f9707e0bc20> └ YOLOX( (backbone): YOLOPAFPN( (backbone): CSPDarknet( (stem): Focus( (conv): BaseConv( (conv): ...

File "/home/axs-liushipeng/YOLOX/yolox/models/yolox.py", line 35, in forward fpn_outs, targets, x │ │ └ <unprintable Tensor object> │ └ <unprintable Tensor object> └

File "/home/axs-gaoman/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) │ │ │ └ {} │ │ └ │ └ <function YOLOXHead.forward at 0x7f9707e0b170> └ YOLOXHead( (cls_convs): ModuleList( (0): Sequential( (0): BaseConv( (conv): Conv2d(256, 256, kernel_size=...

File "/home/axs-liushipeng/YOLOX/yolox/models/yolo_head.py", line 203, in forward dtype=xin[0].dtype, └

File "/home/axs-liushipeng/YOLOX/yolox/models/yolo_head.py", line 357, in get_losses "cpu",

File "/home/axs-gaoman/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context return func(*args, **kwargs) │ │ └ {} │ └ └ <function YOLOXHead.get_assignments at 0x7f9707e0b440>

File "/home/axs-liushipeng/YOLOX/yolox/models/yolo_head.py", line 451, in get_assignments gt_bboxes_per_image = gt_bboxes_per_image.cpu().float() │ └ <method 'cpu' of 'torch._C._TensorBase' objects> └ <unprintable Tensor object>

RuntimeError: CUDA error: device-side assert triggered

xauat-liushipeng avatar May 23 '22 02:05 xauat-liushipeng

From your log here:

File "/home/axs/YOLOX/yolox/models/yolox.py", line 35, in forward
fpn_outs, targets, x
│ │ └
│ └
└

Are your input empty? Did you paste the full log?

FateScript avatar May 23 '22 03:05 FateScript

Did it happen during the training process? Not at the begining of training?

fengkh avatar Sep 10 '23 14:09 fengkh