CUDA error: an illegal memory access was encountered CUDA kernel

[nazib]

HI There, I tried to run your old training code given in tools dir. When I tried to run your training code with single gpu and with coco data set I get this error: CUDA error: an illegal memory access was encountered CUDA kernel . In your requirements file you didn't mention the pytorch version you used. Is it because of the the pytorch version. By the way I am using T4 gpu and cudnn version is 11.3 python=3.20.4 pytorch=1.12.1. for trainning is used config this configfile: MODEL: META_ARCHITECTURE: "panoptic_deeplab" BN_MOMENTUM: 0.01 BACKBONE: NAME: "resnet50" DILATION: (False, False, False) PRETRAINED: True DECODER: IN_CHANNELS: 2048 FEATURE_KEY: "res5" DECODER_CHANNELS: 256 ATROUS_RATES: (3, 6, 9) PANOPTIC_DEEPLAB: LOW_LEVEL_CHANNELS: (1024, 512, 256) LOW_LEVEL_KEY: ["res4", "res3", "res2"] LOW_LEVEL_CHANNELS_PROJECT: (128, 64, 32) INSTANCE: ENABLE: True LOW_LEVEL_CHANNELS_PROJECT: (64, 32, 16) DECODER_CHANNELS: 128 HEAD_CHANNELS: 32 ASPP_CHANNELS: 256 NUM_CLASSES: (1, 2) CLASS_KEY: ["center", "offset"] DATASET: ROOT: "/home/nazib/Data/coco/" DATASET: "coco_panoptic" NUM_CLASSES: 133 TRAIN_SPLIT: 'train2017' TEST_SPLIT: 'val2017' CROP_SIZE: (1025, 2049) MIRROR: True MIN_SCALE: 0.5 MAX_SCALE: 2.0 SCALE_STEP_SIZE: 0.1 MEAN: (0.485, 0.456, 0.406) STD: (0.229, 0.224, 0.225) SEMANTIC_ONLY: False IGNORE_STUFF_IN_OFFSET: True SMALL_INSTANCE_AREA: 4096 SMALL_INSTANCE_WEIGHT: 3 SOLVER: BASE_LR: 0.00005 WEIGHT_DECAY: 0.0 WEIGHT_DECAY_NORM: 0.0 BIAS_LR_FACTOR: 1.0 WEIGHT_DECAY_BIAS: 0.0 OPTIMIZER: "adam" LR_SCHEDULER_NAME: "WarmupPolyLR" WARMUP_ITERS: 0 LOSS: SEMANTIC: NAME: "hard_pixel_mining" IGNORE: 255 TOP_K_PERCENT: 0.2 WEIGHT: 1.0 CENTER: NAME: "mse" WEIGHT: 200.0 OFFSET: NAME: "l1" WEIGHT: 0.01 TRAIN: IMS_PER_BATCH: 8 MAX_ITER: 90000 DEBUG: DEBUG: True DEBUG_FREQ: 100 TEST: EVAL_INSTANCE: True EVAL_PANOPTIC: True POST_PROCESSING: CENTER_THRESHOLD: 0.1 NMS_KERNEL: 7 TOP_K_INSTANCE: 200 STUFF_AREA: 2048 OUTPUT_DIR: "./output/panoptic_deeplab_R50_os32_cityscapes" GPUS: (0, 1, 2, 3, 4, 5, 6, 7) WORKERS: 1