[Bug] MMDataParallel loads very slowly

Open clanguagesh opened this issue 1 year ago • 1 comments

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

/hdc/miniconda3/envs/LSKNet/lib/python3.8/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. warnings.warn( fatal: not a git repository (or any of the parent directories): .git sys.platform: linux Python: 3.8.18 | packaged by conda-forge | (default, Oct 10 2023, 15:44:36) [GCC 12.3.0] CUDA available: True GPU 0,1,2,3: NVIDIA A40-48Q CUDA_HOME: None GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with:

GCC 9.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.7
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.5
Magma 2.6.1
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.14.1+cu117 OpenCV: 4.8.0 MMCV: 1.7.1 MMCV Compiler: GCC 9.3 MMCV CUDA Compiler: 11.6 MMRotate: 0.3.4+

Reproduces the problem - code sample

Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=0 python /home/shenhui/semi-objetc-detection/SOOD-main/train.py /home/shenhui/semi-objetc-detection/SOOD-main/configs/ssad_fcos/dense_teacher_fcos_dota15_10per.py --work-dir /hdc/workspace_sh/debug

Reproduces the problem - error message

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.neck.fpn_convs.4.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.neck.fpn_convs.4.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.0.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.0.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.1.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.1.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.2.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.2.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.3.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.cls_convs.3.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.0.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.0.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.1.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.1.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.2.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.2.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.3.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,734 - mmcv - INFO - teacher.bbox_head.reg_convs.3.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_cls.weight - torch.Size([16, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=-4.59511985013459

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_cls.bias - torch.Size([16]): NormalInit: mean=0, std=0.01, bias=-4.59511985013459

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_reg.weight - torch.Size([4, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_reg.bias - torch.Size([4]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_centerness.weight - torch.Size([1, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_centerness.bias - torch.Size([1]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_angle.weight - torch.Size([1, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.conv_angle.bias - torch.Size([1]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scales.0.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scales.1.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scales.2.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scales.3.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scales.4.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - teacher.bbox_head.scale_angle.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.conv1.weight - torch.Size([64, 3, 7, 7]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.bn1.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.bn1.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.conv1.weight - torch.Size([64, 64, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn1.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn1.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.conv2.weight - torch.Size([64, 64, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn2.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn2.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.conv3.weight - torch.Size([256, 64, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn3.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.bn3.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.downsample.0.weight - torch.Size([256, 64, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,735 - mmcv - INFO - student.backbone.layer1.0.downsample.1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.0.downsample.1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.conv1.weight - torch.Size([64, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn1.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn1.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.conv2.weight - torch.Size([64, 64, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn2.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn2.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.conv3.weight - torch.Size([256, 64, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn3.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.1.bn3.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.conv1.weight - torch.Size([64, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn1.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn1.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.conv2.weight - torch.Size([64, 64, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn2.weight - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn2.bias - torch.Size([64]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.conv3.weight - torch.Size([256, 64, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn3.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer1.2.bn3.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.conv1.weight - torch.Size([128, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn1.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn1.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.conv2.weight - torch.Size([128, 128, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn2.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn2.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.conv3.weight - torch.Size([512, 128, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn3.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,736 - mmcv - INFO - student.backbone.layer2.0.bn3.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.0.downsample.0.weight - torch.Size([512, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.0.downsample.1.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.0.downsample.1.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.conv1.weight - torch.Size([128, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn1.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn1.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.conv2.weight - torch.Size([128, 128, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn2.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn2.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.conv3.weight - torch.Size([512, 128, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn3.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.1.bn3.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.conv1.weight - torch.Size([128, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn1.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn1.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.conv2.weight - torch.Size([128, 128, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn2.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn2.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn3.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.2.bn3.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.conv1.weight - torch.Size([128, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.bn1.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.bn1.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.conv2.weight - torch.Size([128, 128, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.bn2.weight - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.bn2.bias - torch.Size([128]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,737 - mmcv - INFO - student.backbone.layer2.3.conv3.weight - torch.Size([512, 128, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer2.3.bn3.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer2.3.bn3.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.conv1.weight - torch.Size([256, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.downsample.0.weight - torch.Size([1024, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.downsample.1.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.0.downsample.1.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.conv1.weight - torch.Size([256, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.1.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.2.conv1.weight - torch.Size([256, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.2.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.2.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,738 - mmcv - INFO - student.backbone.layer3.2.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.2.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.2.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.2.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.2.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.2.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.conv1.weight - torch.Size([256, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.3.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.conv1.weight - torch.Size([256, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.4.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.conv1.weight - torch.Size([256, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.bn1.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.bn1.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.conv2.weight - torch.Size([256, 256, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.bn2.weight - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,739 - mmcv - INFO - student.backbone.layer3.5.bn2.bias - torch.Size([256]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer3.5.conv3.weight - torch.Size([1024, 256, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer3.5.bn3.weight - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer3.5.bn3.bias - torch.Size([1024]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.conv1.weight - torch.Size([512, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn1.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn1.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.conv2.weight - torch.Size([512, 512, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn2.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn2.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.conv3.weight - torch.Size([2048, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn3.weight - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.bn3.bias - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.downsample.0.weight - torch.Size([2048, 1024, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.downsample.1.weight - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.0.downsample.1.bias - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.conv1.weight - torch.Size([512, 2048, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn1.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn1.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.conv2.weight - torch.Size([512, 512, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn2.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn2.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.conv3.weight - torch.Size([2048, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn3.weight - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.1.bn3.bias - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.2.conv1.weight - torch.Size([512, 2048, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.2.bn1.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,740 - mmcv - INFO - student.backbone.layer4.2.bn1.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.conv2.weight - torch.Size([512, 512, 3, 3]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.bn2.weight - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.bn2.bias - torch.Size([512]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.conv3.weight - torch.Size([2048, 512, 1, 1]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.bn3.weight - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.backbone.layer4.2.bn3.bias - torch.Size([2048]): PretrainedInit: load from torchvision://resnet50

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.0.conv.weight - torch.Size([256, 512, 1, 1]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.0.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.1.conv.weight - torch.Size([256, 1024, 1, 1]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.1.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.2.conv.weight - torch.Size([256, 2048, 1, 1]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.lateral_convs.2.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.0.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.1.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.2.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.3.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.4.conv.weight - torch.Size([256, 256, 3, 3]): XavierInit: gain=1, distribution=uniform, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.neck.fpn_convs.4.conv.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.bbox_head.cls_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.bbox_head.cls_convs.0.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.bbox_head.cls_convs.0.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,741 - mmcv - INFO - student.bbox_head.cls_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,741 - mmcv - INFO - student.bbox_head.cls_convs.1.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.1.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.2.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.2.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.3.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.cls_convs.3.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.0.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.0.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.1.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.1.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.2.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.2.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.3.gn.weight - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.reg_convs.3.gn.bias - torch.Size([256]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_cls.weight - torch.Size([16, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=-4.59511985013459

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_cls.bias - torch.Size([16]): NormalInit: mean=0, std=0.01, bias=-4.59511985013459

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_reg.weight - torch.Size([4, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_reg.bias - torch.Size([4]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_centerness.weight - torch.Size([1, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_centerness.bias - torch.Size([1]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_angle.weight - torch.Size([1, 256, 3, 3]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.conv_angle.bias - torch.Size([1]): NormalInit: mean=0, std=0.01, bias=0

2024-09-23 11:46:08,742 - mmcv - INFO - student.bbox_head.scales.0.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,743 - mmcv - INFO - student.bbox_head.scales.1.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,743 - mmcv - INFO - student.bbox_head.scales.2.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,743 - mmcv - INFO - student.bbox_head.scales.3.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,743 - mmcv - INFO - student.bbox_head.scales.4.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

2024-09-23 11:46:08,743 - mmcv - INFO - student.bbox_head.scale_angle.scale - torch.Size([]): The value is the same before and after calling init_weights of RotatedSSTGDenseTeacher

fatal: not a git repository (or any of the parent directories): .git

It was stuck here for a long time for more than half an hour and I found out by debugging that it's stuck here model = MMDataParallel( model.cuda(cfg.gpu_ids[0]), device_ids=cfg.gpu_ids) and The rest of the training became very slow， Both the GPU and the CPU have 0 usage

Additional information

No response

Sep 23 '24 11:09 clanguagesh

have you solved this ?

Nov 07 '24 09:11 Leoeeeeeeea